Compare commits

..

194 Commits

Author SHA1 Message Date
cxymds 0517566fad chore: sync main for scanner statistics review 2026-11-15 15:56:57 +08:00
Hauser Zhi 8f361e3a6f perf(ecstore): share rename fanout metadata (#9432)
* perf(ecstore): share rename fanout metadata

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(storage): stabilize async regression fixtures

Run the multipart refusal matrix on the shared large-stack harness, wait for scanner metadata commit tails before restart discovery, and observe io_uring owner release alongside in-flight retirement.

Keep the production paths unchanged while preserving each original failure oracle.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
2026-11-15 15:05:20 +08:00
Hauser Zhi 3cee9c154b perf(grpc): attribute rebuilds and bound bulk responses (#9420)
* perf(grpc): attribute rebuilds and bound bulk responses

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(grpc): attribute rebuild success after publication

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* feat(obs): add exact HTTP/2 ping timeout diagnostics

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(multipart): run refusal matrix on large stack

Run the real-store completion refusal matrix through the shared 32 MiB test harness so its deep async storage future does not overflow libtest's default stack.

Keep the original test name and all refusal-order assertions intact.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* chore(deps): pin nixl-sys workspace source

Manage the nixl-sys Git revision from the workspace dependency table and point the optional Linux transport dependency at houseme/nixl commit cc636c83f13d692e85b67e55aed092413d16fb41.

Allow the reviewed source in cargo-deny and refresh the lockfile, replacing the unmaintained bincode dependency with the fork's wincode graph.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* feat(rpc): authenticate msgpack peer capability

Add an optional boot-epoch-bound HMAC capability proof and cache the verified per-peer observation without enabling any msgpack send policy.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* feat(grpc): bind peer capability to connection generations (#9433)

* feat(grpc): track physical connection generations

Observe every tonic physical reconnect through a stable custom connector while preserving existing TCP, TLS, HTTP/2, cache, and public channel behavior.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* feat(rpc): bind msgpack capability to connections

Scope authenticated msgpack observations to channel identity and physical connection generation so transparent reconnects require a fresh proof without enabling the send policy.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-15 15:05:09 +08:00
Ramakrishna Chilaka 504cc06d0c perf(ecstore): stage inline rollback backups and fuse backup reclaim (#9445)
Stage inline overwrite rollback backups (xl.meta.bkp) directly relative
to the pinned destination directory descriptor in RenameCommitGuard
instead of staging in .rustfs.sys/tmp and walking both paths a second
time. Also fuse post-commit xl.meta.bkp removal and empty-ancestor
directory pruning into a single blocking dispatch while stopping at live
object roots.
2026-11-15 15:04:18 +08:00
Hauser Zhi 44145ef327 chore(deps): update flake.lock (#9441)
Flake lock file updates:

• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/4fe25e1' (2026-11-06)
  → 'github:NixOS/nixpkgs/2074cc6' (2026-11-12)
• Updated input 'rust-overlay':
    'github:oxalica/rust-overlay/d3fe742' (2026-11-07)
  → 'github:oxalica/rust-overlay/5b87ee0' (2026-11-14)
2026-11-15 15:04:11 +08:00
Hauser Zhi ba9e8342f8 test(storage): add fail-closed quiescence gate (#9436)
* test(storage): add fail-closed quiescence gate

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(storage): gate benchmark rounds on quiescence

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(bench): bound quiescence Docker calls

Start the storage-quiescence deadline before Docker discovery, pass the remaining budget to every subprocess, and reject samples completed after the deadline.

Cover delayed initial lookup and delayed exec through the supported Docker binary override.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
2026-11-15 15:04:04 +08:00
Hauser Zhi a41019802d test(storage): expose direct-write validation controls (#9430)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-15 15:03:53 +08:00
Hauser Zhi 4313812e24 perf(ecstore): keep inline commits fused with file grouping (#9427)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-15 15:03:45 +08:00
hector 5c58a56c4c fix: recognize gVisor directfs bind mounts in the FS guard (#9426)
gVisor presents a host bind mount as 9p, so statfs reports V9FS.
When the mountinfo options include both trans=fd and directfs, the
startup filesystem guard treats that endpoint as local POSIX storage
and logs that the host type is not visible. Real 9p and virtio shares
still warn.

Co-authored-by: hector <majinghe@users.noreply.github.com>
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
2026-11-15 15:03:37 +08:00
GatewayJ 8196603a76 fix(iam): authorize delegated service account creation (#9408)
* fix(iam): authorize delegated service account creation

* docs(iam): clarify explicit grants for custom administrator policies

* fix(iam): include service account delegation in admin wildcards

* style(policy): format delegated authorization test cases

* fix(iam): keep delegated authorization and replication consistent

* fix(ci): include delegated account test in E2E selections
2026-11-15 15:03:24 +08:00
Hauser Zhi 1e62cbaeaf perf(ecstore): integrate safe sync and allocator diagnostics (#9428)
* feat(diagnostics): restore allocator and runtime metrics

Port the missing live-byte accounting and serving-runtime observations from 13a30abca. Reuse the current MiMalloc GlobalAlloc wrapper, expose explicit metric availability, and cover allocation operations and feature selection.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(ecstore): detach queued fdatasync followers safely

Adapt the follower timeout experiment from 138bc69f6 with exclusive queue ownership transfer. Move queued paths and resource units into independent sync, retain claimed batch ownership through completion, and record observed queue time. Cover publication before the original batch resumes, sync failures, claimed batches, and queue accounting.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(ecstore): index sync followers and isolate missing sources

Use reusable slab slots and monotonic identities for constant-time queued follower extraction. Avoid retaining follower group references when the timeout is disabled. Preserve claimed-batch ownership and reject stale tickets after slot reuse.

Keep missing-source results local to their requests so healthy requests still synchronize their own files. Preserve fail-fast behavior for other disk I/O failures. Repair failure-probe synchronization and exercise source disappearance, reused slots, disabled tracking, cancellation and bounded admission on Linux.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ci): apply storage stack to multipart refusal fixture

Route the real-store absent-binding refusal-order fixture through the existing bounded 4 MiB storage setup in both default and CI profiles. The Linux CI failure reproduced with the default 2 MiB stack. Quota, legal-hold, error, cause, audit and notification assertions execute through the established storage fixture budget.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): make sync worker idle handoff atomic

Move batch claim, in-flight accounting and completion onto the existing worker guard. Publish the idle state under the queue lock so an enqueue during cleanup establishes successor-worker ownership. Keep stale cleanup conditional on current ownership.

Add a controlled interleaving regression that detaches the final queued follower and enqueues a successor after the empty claim. The old worker leaves that request without an owner; the refactored worker hands it off and completes one physical sync.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): await grouped sync resource release

Wait for native producer admission units to reach zero after the blocking sync finishes. This removes a scheduler-dependent assertion window exposed by the latest main integration while preserving the cancellation and ownership checks.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-15 14:58:54 +08:00
Hauser Zhi da1817ae01 test(rpc): expose msgpack fleet confirmation (#9429) 2026-11-15 14:54:23 +08:00
cxymds 73895b37e8 test(admin): bind snapshot fixtures to current bucket identities 2026-11-15 14:49:54 +08:00
overtrue c2b5fbb138 feat(gateway): serve CreateMultipartUpload through the owned path
Share authorization between both CreateMultipartUpload edges. Keep the
decoded legacy fields intact, preserve the replication destination
version and apply the captured bucket incarnation before creating the
upload. Leave checksum, encryption and retention metadata on the same
storage path.

Verify the frozen legacy behavior on 2c2af51a before the migration and
exercise both stacks with 66 refusal cases and 22 success controls.
Read back sessions, version, retention and encryption metadata.

Fault probes made authorization tests fail on both edges when an action
or the resource binding was lost. Storage tests detected a different
destination version and unexpected sessions when generation, quota, KMS
or SSE checks were removed. The wire and
helper tests failed for early name rejection, lost timestamp precision,
metadata, checksum, SSE response, the operation count and unexpected
notification.

On 77da6339, 185 scoped tests passed with a 32 MiB test stack. The default
stack run passed 184 tests and reproduced the existing overflow in
completion_absent_binding_refusal_order_baseline. Both all-target
Clippy variants, test wiring, script tests and prepush passed. After
integration with 97a7639f, gateway all-target check and prepush passed.
Whole-workspace CI and rio-v2 were outside the local verification.
2026-11-15 14:08:38 +08:00
overtrue 97a7639f59 feat(s3): serve UploadPart through the gateway
Share the streaming multipart write path and staged authorization across both edges. Resolve the native upload ID against the active storage record at the existing lookup. Carry checksums, trailers, SSE values and the stored ETag through each response projection.

Add seven tests for application lookup, route authorization, framing and durable part state. Exercise 62 refusals and 10 successful requests across those fixtures. The existing SSE-C completion test also covers the migrated UploadPart core.

Verification on 1467d32695: fixed legacy contract, 147 gateway-feature tests and 119 default tests, both all-target Clippy legs, script tests, test wiring and prepush checks. The test scopes include the entire access module. The gateway scope includes PutObject and both refusal matrices.

Twenty-two local fault controls exercised edge authorization, policy and generation, resource and upload ID projection, owner binding, lookup count and error ordering, logical length, trailer publication, checksum validation and typed producer errors. Each fault failed its intended test after the same artifact passed with the selector unset.

Linux RDMA and a native KMS denial through UploadPart remain unverified.
2026-11-15 13:41:36 +08:00
overtrue 77da63397c feat(s3): serve PutBucketLifecycleConfiguration through the gateway
Validate lifecycle status across both request adapters and preserve mutation error priority, ordered rule IDs, and stored dates. Forward the original bucket and request envelope at the legacy edge. Pin the gateway lifecycle date carrier.

The priority fixture covers seven refusal cases and one success case. All 39 scoped tests passed after source restoration. Fifteen fault switches triggered 16 expected failures covering priority, early conversion, metrics and audit, action binding, transition writes, date precision, offsets, and rule indices.

The tests and both Clippy modes passed on 0567078 with gateway 2a6b8d1. The gateway all-target check and prepush passed after rebasing onto 1467d32. Full-workspace and remote CI results remain unverified.
2026-11-15 13:08:53 +08:00
cxymds 00243c7e71 chore: sync main for scanner validation
Resolve the shared checkpoint fixture update by retaining production drain and the stronger carried-value, certificate, budget and acknowledgement assertions. Preserve upstream functionality and the validated scanner implementation.
2026-11-15 12:02:04 +08:00
overtrue 1467d32695 feat(gateway): serve PutObject through the owned write path
Register PutObject and share write authorization with legacy PUT and
browser POST. Carry authenticated trailer declarations and captured
frames into the owned write, preserve typed eager-read errors, and keep
the table-catalog guard tied to the actual write call.

Verified on RustFS 0567078f5 with gateway d75b6ce on macOS. All 1022
affected tests passed with a 4 MiB test stack, along with both
all-target Clippy configurations, script tests, test wiring, and
prepush checks. A separate gateway wire run stored and read an empty
object using the rebuilt server binary.

Fourteen single-fault checks covered trailer selection and capture,
typed producer failures, tag policies, the shared write grants, and
POST policy projection. The reserved-prefix source guard also rejected
a fixture whose owned write call was replaced.
2026-11-15 11:51:26 +08:00
overtrue 2c2af51a46 feat(gateway): serve PutBucketVersioning
Route bucket versioning writes through the shared authorization and audited use case. Preserve legacy configuration preparation, the incarnation fence, refusal order, persisted XML, and the reload, replication, and scanner callbacks. Count each admitted write once at its edge.

Verified against RustFS 0567078f5 and gateway d75b6ceb. The signed wire cases cover six accepted writes and nine refusals per stack. The affected suite passed 29 tests, the typed-error suite passed 26, and both Clippy configurations, script tests, wiring, and prepush checks passed. The wire cases and both refusal matrices passed again after boxing the test fixture's SDK error.

Eight fault controls went red for both authorization hooks, the legacy counter, an omitted write, a dropped empty excluded prefix, Object Lock precedence, replication refusal, and the custom-status error. CI, the Linux reference comparison, and live multi-node callback delivery remain unverified.
2026-11-15 11:33:25 +08:00
cxymds 36bc1057ac fix(ci): give the multipart refusal fixture its storage stack 2026-11-15 11:25:22 +08:00
cxymds 32f7f70a29 feat(scanner): observe bucket reuse and production proof gates 2026-11-15 11:25:22 +08:00
cxymds 5d32827af5 fix(usage): validate cached statistics against bucket identity 2026-11-15 11:25:06 +08:00
cxymds 04ef836aa3 fix(scanner): cap superseded retries at the nominal cycle 2026-11-15 11:25:05 +08:00
cxymds 03850c1cd6 fix(scanner): preserve baselines during backup catchup
Require decoded-primary provenance before accepting an unchanged baseline after its backup catches up. Keep current-revision CAS and publication fences, and align the real checkpoint fixture with the bounded-cycle carry contract.
2026-11-15 11:24:42 +08:00
overtrue 0567078f5b feat(gateway): serve GetObject through the owned read path
Register GetObject and share its authorization and native response facts
with the legacy edge. Preserve conditional headers, range completion,
stream errors, body ownership, SSE reads and request accounting. Pin the
gateway 304 response-override fix without unrelated dependency changes.
Keep the incoming native Put path and update its profiling guard source.

Validation on b7e7a6156:
- Normal and mutation-disabled controls: 9/9 each.
- Affected suite: 754/755; the profiling guard still read the old Put file.
  After correcting that source path, its targeted rerun passed 1/1.
- All-target Clippy passed with gateway-stack and default features.
- Formatting, script-tests, test wiring and prepush-quick passed.
- Authorization matrix: 40 refusals and 8 allowances across both edges.

Mutations: swapped version action, redirected resource, ignored denial,
and bypassed Served authorization failed the permission or HTTP checks.
Earlier effective mutations covered body copying, premature owner drop,
range status, missing 304 headers, a 304 body, wrapped stream error causes
and duplicate accounting. No assertions were removed or weakened.

Linux RDMA paths were not compiled or exercised. GetObjectAttributes,
PutObject gateway registration and the default-stack switch are not part
of this increment.
2026-11-15 07:40:48 +08:00
overtrue b7e7a6156c feat(gateway): serve bucket notification configuration reads
Use native gateway DTOs for stored notification reads, register the handler,
and keep both edges on the shared bucket authorizer. The legacy edge now
uses the compatibility seam. Notification writes and the default stack are
unchanged. Non-test changes: +53/-36 lines.

Add three refusal-oriented tests and two configuration-read tests, including
successful IAM controls. Cover absent and stored configurations, reserved
bucket shapes, unknown stored XML, event/filter values, bucket-bound IAM
allow/implicit deny/explicit deny, and one handled-operation count.

Mutation controls on da5414c: bypassing each edge's authorization made its
notification_authorization_binds_the_bucket_on_the_*_stack test fail because
the wrong bucket was served. Dropping the stored queue made both
*_stack_answers_the_stored_notification_configuration tests fail. Restore
hashes were checked and sources were touched before a real rebuild.

Validation: 279 scoped tests passed on da5414c; 70 scoped tests passed after
rebasing to fd8dc41. Both rustfs all-target Clippy builds passed on fd8dc41,
as did fmt, test wiring and prepush-quick. Script tests passed on da5414c.
The earlier broad run reported one existing CORS test as leaky with exit 0.
CI and the full workspace suite were not run.
2026-11-15 06:08:43 +08:00
overtrue df4c4916c2 refactor(storage): own the PutObject write core
Use RequestEnvelope, a native upload body, raw write fields and owned results in the PutObject core. Legacy and RDMA ingress project once; internal writes share the core. Preserve archive batching, shared SSE and object-lock validation, completion ownership, audit events and response facts. Keep the merged RDMA length, cache lifetime and response-header behavior.

No gateway registration or UploadPart migration is included. Source outside test files/modules: +820/-383 (including conditional test imports).

Validation on fd8dc41f0c: fmt; default and gateway-stack all-target Clippy; 903 scoped tests with RUST_MIN_STACK=4194304; script-tests; test wiring; prepush quick checks. The archive authorization matrix has six refusal cases and three allowed cases, with persisted object/metadata checks. A fixed a279 legacy TCP baseline has 48 matching response records; the two retained socket tests cover PUT and POST behavior. The recorder observes drained counter deltas alongside the official cumulative 0/1/2/3 operation totals.

Mutation evidence: 44 distinct faults, grouped below by the test that rejected them. The counter and archive authorization additions each passed an unflagged control before their five mutations. The other fault controls and failures were retained from the same unchanged implementation.

- app::object::native_body::tests::legacy_body_preserves_demand_pending_and_length_validation_location: body-eager, body-length
- app::object::native_body::tests::legacy_producer_error_keeps_its_concrete_type_and_is_not_eof: producer-source
- app::object::native_body::tests::internal_producer_error_preserves_io_kind_and_nested_source_after_data: producer-source
- app::object::native_body::tests::native_trailers_are_published_only_at_eof: trailer-pending, trailer-value
- app::object::native_body::tests::native_framing_and_classified_errors_are_not_unwrapped: native-byte-count, native-class, native-kind
- storage::ecfs::legacy_edge::object_write::native_put_tests::legacy_projection_defers_invalid_raw_values_to_the_existing_validators: input-bucket, input-key, input-storage_class, input-server_side_encryption, input-object_lock_mode, input-object_lock_legal_hold_status
- storage::ecfs::legacy_edge::object_write::native_put_tests::legacy_projection_keeps_each_checksum_and_unknown_length: input-checksum_crc32, input-checksum_crc32c, input-checksum_sha1, input-checksum_sha256, input-checksum_crc64nvme, input-length, input-trailer
- storage::ecfs::legacy_edge::object_write::native_put_tests::owned_response_projection_preserves_status_headers_and_all_checksums: output-e_tag, output-checksum_crc32, output-checksum_crc32c, output-checksum_sha1, output-checksum_sha256, output-checksum_crc64nvme, output-status, output-headers
- app::object::put::tests::put_read_quorum_failures_are_normalized_at_each_stage: error-status, error-message, error-source
- app::object::put::tests::put_commit_lock_failure_keeps_request_operation_in_spawned_owner: error-headers
- app::object::put::tests::cancelled_put_request_completes_post_commit_publication: completion
- app::object::shared::tests::build_put_like_object_lock_metadata_rejects_mode_without_retain_until_date: lock-pairing
- app::object::shared::tests::build_put_like_object_lock_metadata_rejects_retain_until_date_without_mode: lock-pairing
- storage::ecfs_test::tests::test_parse_object_lock_retention: lock-presence
- app::object::extract::tests::snowball_member_mtime_requires_member_replication_authorization: archive-replication
- table_catalog::tests::put_object_reserved_prefix_guard_rejects_broken_native_delegation: guard-reason
- object_write::legacy_owned_put_preserves_refusal_order_checksums_and_short_body: refusal-priority, output-headers, output-checksum_crc32
- app::object::put::tests::put_edge_counts_polled_requests_once_before_usecase_refusal: count-missing, count-double
- app::object::extract::tests::snowball_owned_member_privileges_gate_storage: archive-tagging, archive-retention, archive-legal_hold

Existing expected values are unchanged.

Not verified: CI, Linux RDMA execution, performance, and gateway native-frame trailer consumption through this business core. The native trailer adapter itself is covered; gateway integration is a separate increment. The local suite uses a 4 MiB stack and makes no claim about default-stack behavior.
2026-11-15 01:20:53 +08:00
Hauser Zhi fd8dc41f0c feat(ecstore): add file-sync durability tier (#9422)
* refactor(ecstore): split durability sync capabilities

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(ecstore): classify durability barriers

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* feat(ecstore): add file-sync durability tier

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(storage): harden durability docker validation

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(storage): isolate durability PUT benchmarks

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(storage): expose file sync grouping controls

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-15 00:17:34 +08:00
Chris An 645bb63809 feat: S3-over-RDMA with NIXL and cuObject transports (#9394)
* feat: S3-over-RDMA with NIXL and cuObject transports

A signed S3 request with an empty body names a registered client buffer; the server moves the object bytes with one-sided RDMA (NIXL/UCX, or NVIDIA cuObject through the cuobject crate) instead of the HTTP body. Internode shard reads and writes can also move over NIXL.

Applied onto main from #9305 (head 0f3c19a0f); the stacked #9302-#9304
commits are already on main as eca95f297. Conflicts were resolved where
this change met newer main: crates/ecstore/src/api/mod.rs (kept the
internode-transport exports, added the nixl module),
crates/rio/src/http_reader.rs (internode_http_request uses the pooled
get_http_client::<false>), rustfs/src/storage/rpc/http_service.rs
(write_authenticated_put_file stays a plain fn returning a future, now
pub(super)), rustfs/src/app/multipart_usecase.rs (the RDMA UploadPart
branch moved into multipart_usecase/upload_part.rs after the
per-operation split), Cargo.lock (main's lock plus only this change's
new packages).

* fix(rdma): build the RDMA crates without vendor SDKs and pass guards

Maintainer changes on top of the contributed S3-over-RDMA work:

- rustfs-cuobject: cuobject is an optional, default-features = false
  dependency behind a new `server` feature (enabled by the rustfs
  `cuobject` feature); the crate root is #![cfg(feature = "server")], so
  without the feature it exports nothing and a workspace build needs no
  NVIDIA cuObject SDK.
- The NIXL agent handle (OnceLock, install_nixl_server, nixl_server)
  moved from app/object/cuobject.rs to storage/rpc/nixl_read.rs; the app
  init installs it. The layer-dependency guard rejects infra -> app.
- New code reaches ECStore through a cfg-gated `ecstore_rdma` facade
  module and s3s types through the existing re-export hubs
  (architecture-migration and s3s-footprint guards).
- SAFETY notes at the two #[allow(unsafe_code)] sites, inside the window
  the unsafe-allowance guard checks.
- transport_config_known_backends_are_current_oss_values loops over
  KNOWN_INTERNODE_DATA_TRANSPORT_BACKENDS again, expecting nixl to name
  itself.
- New non-required workflow rdma-feature-check.yml runs
  `cargo check --locked -p rustfs --features nixl` on Linux (paths
  filter, weekly, manual), registered in scheduled-validations.json.

* fix(ecstore): keep NIXL internode error messages constant

The three io::Error::other(format!(..)) sites in nixl_internode.rs
(read refused, write refused, write byte-count mismatch) now return
constant messages; the status, response body length and byte counts go
to a debug! line in the same place, the level this file already uses for
a failed NIXL peer call. No response body is logged, as on the HTTP
internode path.

The error-format ratchet exists because per-disk formatted messages make
otherwise identical errors compare unequal and split the quorum
reduction; constant messages keep them in one bucket. A typed transport
error would carry the detail without that problem; this takes the
mechanical route.

* fix(ecstore): flush staged NIXL writes before waiting for buffers

* fix(rdma): build rustfs-nixl only on Linux

nixl-sys's stub build links -lstdc++, which macOS does not ship, so
`cargo test -p rustfs-nixl` (and a workspace test run) failed to link on
macOS. rustfs-nixl's dependencies are now Linux target dependencies and
its crate root is #![cfg(target_os = "linux")]; the rustfs `nixl`
feature code that was not already Linux-gated (the storage/rpc nixl_read
module, open_shard_contained, the internode route arms and the facade
module) is now cfg(all(feature = "nixl", target_os = "linux")). Linux
builds are unchanged: the workspace tests still build and run the
crate's tests through the stub.

* ci(rdma): alert on scheduled failures of the RDMA feature check

The test-wiring guard requires every scheduled validation workflow to carry the shared alert-on-failure job; a failed weekly run now opens or updates the tracking issue like the other scheduled lanes.

* fix(rdma): put rustfs-nixl behind a sys feature

nixl-sys needs a C++ compiler and libclang (bindgen) to build, which the shared Linux runners do not have, so Workspace Clippy and Lint and Build failed once the crate compiled there. Its dependencies are now optional behind a 'sys' feature that the rustfs 'nixl' feature enables; without it (and off Linux) the crate is empty, so no default or workspace build touches vendor C/C++ tooling. The RDMA feature check now also runs the crate's unit tests with the feature on; they use a fake sysfs and host memory, no device.

---------

Signed-off-by: Hauser Zhi <housemecn@gmail.com>
Co-authored-by: Rain Jiang <rain-jiang@outlook.com>
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
2026-11-15 00:10:10 +08:00
Hauser Zhi 5f9e5d62df feat(obs): measure remote rename field bytes (#9417)
* feat(obs): split PUT commit and remote rename timing

Measure the app commit-owner scheduling gap and split remote rename client
preparation into compatibility encoding, client acquisition, and request digest
stages. Keep the timers behind the existing detailed PUT diagnostics gate.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* bench(filemeta): measure inline rename RPC encoding

Compare JSON compatibility, msgpack, and dual encoding for metadata-only and
representative inline FileInfo payloads used by remote rename requests.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(bench): use exact inline rename shard fixture

Model the 100 KiB EC12 inline data as one rounded data shard plus its
HighwayHash header, and report metadata-only throughput as operations.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* feat(obs): measure remote rename field bytes

Record detail-gated JSON and msgpack field lengths after successful remote
rename request encoding so msgpack-only rehearsals can prove sender behavior.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(scanner): preserve partial carry drain window

Use the production drain window for real partial-bucket checkpoints so CI storage latency cannot truncate source-cache persistence.

Align the certified partial-cycle assertions with carried-value semantics while retaining mixed-coverage and final convergence checks.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-15 00:00:09 +08:00
cxymds 654be96186 feat(scanner): expose scan work and reuse observations 2026-11-14 21:25:19 +08:00
overtrue da5414c345 fix(protocols): preserve HeadBucket storage root compatibility
Resolve empty-bucket requests through the existing storage lookup at the
legacy edge. Keep authorization, operation counting and nonempty bucket
conversion unchanged. No gateway bucket-name contract changes.

Add two tests covering two positive and two negative cases: protocol root,
ordinary bucket, missing bucket and uninitialized storage. Check response
status/headers and one count per request. Disabling the compatibility
branch makes both tests fail; restoring it passes both after recompilation.

Verified on 4182d55d1: focused tests, gateway Clippy, and a rebuilt binary
with libunftp 0.25 using FTP_TLS: login 230, root and ordinary CWD 250,
missing bucket 550. Default Clippy and script-tests passed on 925e5d0c0.
After integrating a279b4bb6, gateway Clippy and prepush-quick passed.
The wire probe and focused tests were not rerun after that integration.
2026-11-14 20:57:53 +08:00
overtrue a279b4bb63 feat(gateway): serve owned multipart completion
Use shared request authorization and native completion responses across both S3 stacks. Keep storage responsible for final commit and replay decisions.
2026-11-14 20:42:23 +08:00
Hauser Zhi 4182d55d11 chore(deps): upgrade JWT and FTP dependencies (#9419)
Update jsonwebtoken to 12.0.0 and libunftp to 0.25.0, including the JWT key construction API changes required by the new release. Refresh the lockfile through two requested cargo update and cargo upgrade rounds.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-14 09:11:54 +00:00
Hauser Zhi d4af2377fe perf(grpc): prewarm isolated bulk channels (#9418)
perf(grpc): prewarm isolated bulk channel pools

Extend the existing opt-in peer prewarm to the configured bulk channel pool. Control prewarm remains one channel, isolated bulk prewarm fills the per-peer pool, and legacy shared mode reuses the control cache. Add a cache-reuse regression for the legacy mode.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-14 09:02:49 +00:00
Ramakrishna Chilaka f5e8f17461 perf(ecstore): coalesce shard, integrity index, and commit syncs (#9407)
* perf(ecstore): coalesce shard, integrity index, and commit syncs

- Enable sync-on-close (WriteMode::TruncateSync with sync_parent_dir: false)
  in LocalDisk::create_file for multipart staging writes so part.N and
  part.N.integrity.<gen> run fdatasync before POSIX_FADV_DONTNEED page-cache
  reclaim and register in PRESYNCED_TMP_FILES without polluting
  PRESYNCED_TMP_DATA_DIRS.
- Keep LazyLocalFileWriter open across write_bitrot_shard when shard integrity
  is enabled and coalesce the .idx sidecar creation, write, and fdatasync into
  poll_shutdown via set_pending_integrity_index + MultiWriter::finish_and_shutdown,
  skipping duplicate PreparedIntegrity::write file creations for prewritten disks.
- In fused_attempt (commit.rs), sync prepared_metadata_source (xl.meta) before
  os::fsync_dir_std(src_data_path) when src_data_path is presynced so the
  active journal transaction commits both xl.meta data extents and the staged
  data directory entry in one flush instead of two.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 21d8304f-0f49-48eb-8c28-26f3689b081b

* perf(ecstore): skip redundant .idx parent creation and header re-read when presynced

- Mark parent_exists = true in StdBackend::open_write for temporary .integrity.
  and .proof. sidecars so open_lazy_local_file_std opens the file directly in
  its existing staging directory before falling back to create_dir_all_std.
- Skip reopening and re-reading the 32-byte .idx header in
  LocalDisk::prepare_part_transaction when take_presynced_tmp_file confirms the
  sidecar was just written and synced by LazyLocalFileWriter.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 21d8304f-0f49-48eb-8c28-26f3689b081b

* fix(ecstore): validate presynced multipart integrity indexes

Validate the proof file type, length, and header before preparing a multipart transaction. Use a presync receipt to skip the redundant file sync after validation.

Cover valid, truncated, and mismatched-header indexes carrying presync receipts.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): wait for tail rename drain in inline_put_direct_commit_handles_post_encode_rename_failures

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 21d8304f-0f49-48eb-8c28-26f3689b081b

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Copilot-Session: 21d8304f-0f49-48eb-8c28-26f3689b081b
2026-11-14 18:03:56 +08:00
overtrue 925e5d0c0c feat(gateway): serve HeadObject through native response facts
Share HEAD authorization and its use case across both request edges. Keep
stored and proxy ETag parsing distinct, preserve conditional headers and
version bytes, and finish auditing before terminal header encoding. Pin
gateway to 1afff6a for the measured 304 response profile.

Verified 785 affected and embedded tests, then 12 tests after test-only
module/import cleanup. Both all-target Clippy configurations, script tests,
wiring and prepush checks passed. Tests cover malformed stored tags,
separate replication denial, conditional status conflicts and ordinary
success controls; the proxy oracle checks 779 spellings.

Observed 39 effective isolated faults: authorization/delegation, execution,
metadata exposure, stored/legacy/proxy tags, raw header escaping/repetition,
framing conflicts, length/counting, range boundaries, checksum and storage
class, ordinary completion, terminal code/status/default resolution, and
each of the seven 304 fields. An ineffective fallback mutation was retained
as a failed experiment and replaced with an explicit-status-only fault.

The 785-test base was c13232996; final module tests and Clippy ran on
fc22c11d1. No GetObject migration added. Cross-binary pinned acceptance,
remote proxy/ODM integration, SSE-C real-storage wire and performance were
not verified. Existing assertions were not weakened.
2026-11-14 17:54:44 +08:00
Hauser Zhi 247aecf7b8 fix(health): bound quorum probes and readiness observations (#9410)
* fix(health): bound quorum probes and record timeout observations

Collect fresh local and remote observations concurrently and finish once
each configured set has sufficient storage and lock quorum evidence.
Validate configured peer identities and disk slots before counting votes.

Apply the cluster deadline to cache and singleflight waiting, retain flight
ownership through timeout publication, and record timeout observations once.
Cover slow peers, unavailable quorum, identity validation, cancellation,
cache ownership and compatibility with focused regressions.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(health): coalesce probes and count quorum incrementally

Share the completion generation with overlapping uncached requests while
retaining fresh observations for later requests and bounded flight ownership.
Build validated topology thresholds once and count unique disk and lock
votes incrementally instead of rescanning collected inventories.

Honor the existing inventory metrics flag through RPC and storage owners.
Health probes omit operation metric snapshots while retaining fresh disk
identity, capacity and runtime checks. Move owned inventories across local
aggregation layers and preserve the full-metrics administrative APIs.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(health): cover failed uncached flight recovery

Cover a later waiter sharing its overlapping failed round before an independent request starts a fresh successful observation.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(health): group lock membership in one endpoint pass

Group configured lock members by set in one pass and reject impossible set counts before allocation. Retain per-set thresholds and add a bounded malformed-topology regression.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(health): publish startup topology in gating fixture

Publish the shared four-disk fixture's endpoint topology before initializing disks, matching storage startup. Cluster health probes can then validate fresh quorum observations while the readiness test exercises S3 admission publication.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-14 06:24:21 +00:00
Hauser Zhi fc22c11d1a feat(obs): split PUT commit and remote rename timing (#9416)
* feat(obs): split PUT commit and remote rename timing

Measure the app commit-owner scheduling gap and split remote rename client
preparation into compatibility encoding, client acquisition, and request digest
stages. Keep the timers behind the existing detailed PUT diagnostics gate.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* bench(filemeta): measure inline rename RPC encoding

Compare JSON compatibility, msgpack, and dual encoding for metadata-only and
representative inline FileInfo payloads used by remote rename requests.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(bench): use exact inline rename shard fixture

Model the 100 KiB EC12 inline data as one rounded data shard plus its
HighwayHash header, and report metadata-only throughput as operations.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-14 16:19:00 +08:00
overtrue 0eec25e4c0 fix(ci): compile gateway features in nightly matrix legs
Select gateway-stack for replication and protocol gateway legs and use matching build and run features. Legacy feature selections remain unchanged.

Verified on 0460ad63a: 20 e2e binary tests, full make script-tests, actionlint with shellcheck, CI workflow check, and prepush quick checks. The workflow contract exercises four passing build/run selections, four propagated failures and four mismatched-feature rejections. Removing either gateway feature, mismatching either run selection, and emptying the matrix each made the contract test fail.

No Rust compilation or live nightly execution was performed; actual gateway startup and suite outcomes remain to be observed in CI.
2026-11-14 15:28:03 +08:00
cxymds c132329961 fix(scanner): preserve usage convergence across bounded cycles (#9397)
* feat(usage): record bucket provenance and complete observations

* fix(scanner): preserve usage progress across bounded scan cycles

* docs(scanner): explain usage provenance and bounded convergence

* chore(scanner): correct requeuing spelling in runtime guidance

* fix(scanner): route usage contracts through domain facades

* fix(usage): keep admin projection futures sendable

* fix(scanner): expose endpoint layout in production facade

* fix(scanner): preserve checkpoint ownership during budget drain

* test(scanner): remove redundant fixture clones

* test(ecstore): update Windows rename publication calls

* fix(ci): restore scanner test exports and lint assertions

* test(scanner): preserve production drain for real checkpoints

* fix(scanner): retire obsolete work without clearing responsibility

* test(table-catalog): follow DeleteObject guard delegation

---------

Co-authored-by: Hauser Zhi <housemecn@gmail.com>
2026-11-14 15:07:51 +08:00
cxymds 0f503fc613 feat(admin): expose caller-bound effective permissions (#9414)
* feat(admin): add caller-bound effective permission queries

Expose bounded advisory checks through the existing IAM, bucket-policy, table, and KMS authorization paths. Preserve three-state decisions for incomplete request facts and bypass publication guards only for internal probes while real operations continue to authorize independently.

* fix(auth): preserve effective permission probe boundaries

Use canonical action scopes and verified signing facts, keep unavailable conditions unknown across table metadata gates, and resolve table resources without catalog repair. Route admin test dependencies through existing facades and keep legacy S3 error usage within its migration baseline.
2026-11-14 15:07:24 +08:00
overtrue 0460ad63a0 feat(ecstore): add read-only multipart completion bindings
Observe active, completed and retained multipart ownership without replay, recovery, acknowledgement or reclamation. Only active uploads supply preflight metadata. Completed and retained records require validated physical evidence; completion still owns publication.

Preserve native inspection errors before reporting absence. Completed quorum retains precedence, and UnsupportedDisk keeps its compatibility behavior.

Validation: 17 binding tests and all 7,453 ecstore tests passed with four workers and no skips. Both default and internode-transport,test-util all-target Clippy passed with warnings denied. Formatting, script tests, test wiring and prepush checks passed.

Combined mutations of inspection error propagation, read quorum selection and UnsupportedDisk handling produced four intended runtime failures while two controls passed. Sources were restored by hash and actually recompiled before the full final run.

Application and CompleteMultipartUpload integration are not included.
2026-11-14 14:01:10 +08:00
Nonot 28da2e12a8 feat(initContainer): add resource configuration for init containers (#9400)
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
2026-11-14 13:01:38 +08:00
overtrue 41225da343 fix(ci): provision runtime tools for both full nightly stacks
Reuse the full CI job's pinned mc and Vault provisioning, version checks,
and iptables capability warning before both e2e-full nightly stack runs.
Keep existing test filters, profile selection and report-only policy.

Validation on 4053e694c: actionlint with shellcheck, YAML step equivalence,
test wiring, CI workflow contract, prepush-quick and full script-tests.
The first script-tests attempt hit an unchanged scanner fixture JSON read;
the fixture control and final full rerun passed without source changes.
This does not establish the cause or repair of that fixture failure.

Executed the fixed Linux mc and Vault downloads, SHA-256 checks and version
commands in a private prefix. No test assertions were added or changed
(negative/positive 0/0); no mutations or Rust builds were needed.
The complete nightly workflow and E2E cases were not run for this change.
2026-11-14 12:50:34 +08:00
overtrue 4053e694cb feat(storage): carry native values through shared SSE
Keep SSE configuration, requests, materials and read response fields on
native strings. Project legacy enums at the DTO boundaries and retain the
existing validation order, bucket defaults and KMS authorization behavior.
Carrier diagnostics omit customer keys and encryption key material.

Production lines: +186/-166. Add 5 negative diagnostic tests, 0 positives.
Do not implement or register an owned PutObject operation.

Validation: fmt, default and gateway-stack all-target Clippy, 777 affected
unit tests, script-tests with aligned Python, test wiring and prepush-quick.
Tests ran on 9ea8e8d37f9670f8da3dc5cb6ed5fe6e6e9bb592; rebased onto
a9dd5f0fd1 with no overlapping files or pin change.

Mutation witnesses (8 sites, 9 failing cases): restore derived Debug for
SseTypeV2, EncryptionRequest, DecryptionRequest, EncryptionMaterial and
DecryptionMaterial -> all five native_*debug_withholds* tests fail; change
unknown default mapping -> shared and copy unknown-default tests fail;
admit an unsupported algorithm -> rejects_invalid_sse_algorithm fails;
project AES reads as KMS -> classification_withholds_the_wrapping_key_for_sse_s3_reads fails.
Restore exact source hashes, touch the SSE source and rebuild: 777/777 pass.

Not verified: full workspace, optional rio-v2/FIPS builds, or CI for this commit.
No intended legacy protocol change; diagnostics intentionally omit key fields.
2026-11-14 12:25:38 +08:00
overtrue a9dd5f0fd1 test(cache): observe entry destruction before claim release
Moka maintenance can logically evict an entry while epoch reclamation still
owns its allocation. Check a cache miss, tear down the cache, and observe
entry destruction through Weak before asserting response ownership and the
final reservation release. Keep the original five-byte and zero-byte checks;
only their physical-owner precondition changes. No runtime behavior changes.

Validated on 4bb272ed6 with default cache/runtime-memory features on macOS:
140 package tests passed, all-target Clippy, formatting, test wiring, and
prepush quick checks. Linux has one extra cgroup-only test, not run locally.

Mutation checks: copying response bytes breaks the live claim assertion;
skipping reservation release breaks the final zero assertion; retaining an
entry breaks the Weak assertion; skipping cache removal breaks the miss
assertion; detaching the response after eviction breaks the post-teardown
five-byte assertion. All failed through their intended runtime assertions.
No tests added or removed. No cache policy or reclamation timing changes.
2026-11-14 11:58:09 +08:00
overtrue 4bb272ed64 fix(gateway): honor the SSE-C transport setting
Reuse the outer transport layer's deployment setting in the gateway builder. The existing false default allows customer keys over plaintext with the gateway's explicit acknowledgement; true keeps both key sides strict. Transport facts remain per request.

The shared signer fixture now accepts headers before signing. The new test exercises 45 scenarios (38 refusals, 7 test-handler successes), checks signed SSE headers and raw-key handoff, and runs the inner plaintext gate separately from the outer layer.

Validation on ebe31dfdb5: fmt, all-target Clippy with and without gateway-stack, test wiring, and 63 gateway-stack/handler/transport tests passed. Production diff: +29/-14 lines.

Fault controls fail ssec_deployment_policy_and_key_handoff: forced strict/allow, false secure transport, re-signed key/input rewrites, skipped handler, response-key echo and wrong refusal code. Separate fixture checks reject anonymous allowance and unsigned SSE headers. The original strict selector fails a valid signed plaintext request with InvalidRequest after a TLS handler control succeeds.

Not verified: copy-source key wire handoff, storage use cases, real Served IAM authorization, network TLS termination, cutover acceptance or CI. No production operation was added.
2026-11-14 10:46:32 +08:00
overtrue 9ea8e8d37f fix(deps): align bindgen and prettyplease on syn 3
Restore only bindgen 0.73.2's syn edge to the already locked syn 3.0.10.
Its >=2,<4 range permits this version and prettyplease 0.3 requires syn 3.
No package version, checksum, manifest, feature or formatter changes.

Regression evidence: the frozen lock failed the FIPS tls-runtime test
build with E0277 in bindgen's prettyplease::unparse call. The corrected
locked graph builds and runs provider_policy_isolated in both modes.
No new tests or assertions; no behavior mutation was needed for this
compile failure. Product source lines changed: 0; lock lines: +1/-1.

Verified: tls-runtime all-targets Clippy and provider_policy_isolated
(1 passed in each mode), default on ebd4bb2ba and FIPS on f2a43376d.
The final rebase only adds two test-guard fixes. cargo tree --locked,
fmt, diff whitespace and prepush-quick passed on the final base.

Not verified: full RustFS/workspace suites, the remaining nightly FIPS
consumer matrix, or a new Linux CI run. No runtime TLS policy changed.
2026-11-14 10:35:45 +08:00
overtrue f2a43376de test: adapt owned entrypoint guard fixtures
Follow DeleteObject's typed and legacy entrypoints through the shared fields adapter into its owned implementation. Preserve the existing mutation guards and key/prefix/store ordering checks.

Let the TRACE mutation probe locate either ListObjectsV2 parameter shape exactly once, then require the real logging guard to reject DEBUG.

Validation on ebd4bb2ba8: default and gateway-stack all-target Clippy, three focused nextest tests, full make script-tests, split probe (61 checks), test wiring, shell syntax, and prepush-quick passed. Product lines: +0/-0.

Source-guard evidence: one valid source control and 11 negative fixtures; missing methods, bypassed delegation, missing guards, and reversed ordering fail with the intended diagnostic. The old checker fails on current source. Disabling each typed, legacy, or fields delegation check makes the fixture test fail. TRACE marker evidence: two accepted shapes and five rejected missing/debug/duplicate shapes; TRACE-to-DEBUG mutation is rejected by the real logging guard.

No runtime code or logging policy changes. Full workspace and Linux CI were not run locally; source-guard mutations do not prove runtime authorization.
2026-11-14 10:12:26 +08:00
overtrue ebd4bb2ba8 feat(gateway): serve object and multipart-upload listings
Move ListObjects, ListObjectsV2, ListObjectVersions and ListMultipartUploads
to owned use cases and gateway handlers. Keep legacy encoding, stored ETags,
optional facts and cursor boundaries at their existing edges. ListParts and
multipart create/complete/abort are outside this increment.

Non-test changed lines: +973/-659, excluding test modules and test-only
imports/reexports; comments and whitespace are included.

Validation: 282 focused tests passed on ed3a9e670d,
including all storage::access::tests and both stack refusal matrices. The four
listing matrix rows exercise 32 refusals and 8 allowed requests across both
stacks; these are migrated coverage, not 40 new tests. Stored-key, entity-tag,
encoding and pagination tables provide additional compatibility coverage.
The 1024-byte stored-key positive runs on Linux; macOS attempts the write and
asserts FileNameTooLong and absence because of its full-path limit.

Final RustFS all-target Clippy passed with and without gateway-stack on ed3a9e670.
The affected e2e_test package Clippy passed on e3858df4fa.
After rebasing over the adjacent response-helper change, one gateway-stack
all-target cargo check passed on ebe31dfdb5.
Formatting, test wiring, seam validation and prepush-quick passed.

Ten effective local mutations were reviewed on d5f640243952b571111bb3c97a98850f04a92172:
- Count: key_count_includes_objects_and_prefixes_and_a_keyless_entry_is_not_an_object.
- Marker: list_objects_next_marker_follows_the_cursor_then_the_last_raw_entry.
- Key floor: every_key_the_store_can_hold_is_represented_unchanged.
- Nil version: versions_delete_markers_and_markers_are_answered_raw.
- Upload fields: list_multipart_uploads_output_maps_uploads_and_prefixes_raw.
- XML escaping: listing_entries_are_encoded_per_segment_only_for_url_and_tags_are_escaped
  and the_legacy_stack_lists_every_stored_entity_tag.
- Three direct authorization mutations (ListObjectsV2, ListObjects,
  ListObjectVersions): the_refusal_matrix_holds_on_the_gateway_stack.
- Shared multipart authorization: both stack refusal matrices; this also covers
  ListMultipartUploads delegation. Its separate direct early-return experiment
  failed for a root request and is excluded from authorization evidence.
All temporary experiment code was removed before the final product build.

Not verified: full dual-stack differential run, full workspace CI, or Linux
runtime execution locally. Bootstrapping operations for the differential suite
are not all migrated yet. Historical wire baselines cover a narrower surface.
2026-11-14 09:23:04 +08:00
overtrue ebe31dfdb5 feat(app): carry answered status and headers across both edges
Add Answered<Output> for use cases that return a status override or extra
response headers. The legacy seam preserves those fields on S3Response;
the gateway helper passes them to Resp through its supported API. Bare
outputs share the seam implementation and retain their default response.
No operation consumes the carrier yet. Non-test changes: +87/-4 lines.

Validated on ed3a9e670d:
- cargo clippy --locked -p rustfs --all-targets -- -D warnings, both
  default and --features gateway-stack.
- cargo nextest run --locked -p rustfs --features gateway-stack --lib
  --no-tests fail -E 'test(storage::ecfs::legacy_edge::tests) |
  test(server::gateway_handlers::tests)': 23 passed.
- cargo fmt --all --check, prepush-quick, test wiring, CI workflow
  contract and validation registry passed.

Three added tests cover carrier values and defaults (0 refusal / 3
positive tests). Representative mutations discarded status and headers
on both edges: an_answered_status_and_headers_reach_the_legacy_response_unchanged
failed its status assertion; the corresponding gateway answer test
failed its status/header tuple. Both tests ran and failed. Restore hashes
matched and sources were touched before the passing ordinary rebuild.

Not verified: wire header delivery or encoder-owned/header-collision
refusals; these belong with the first HeadObject/GetObject consumers.
The legacy header assertion was not independently mutated. No new
operation, authorization path, wire claim, or legacy operation behavior
change is included. Full workspace tests and CI were not run locally.

Refs: rustfs/backlog#2767, rustfs/backlog#2734
2026-11-14 09:07:52 +08:00
overtrue ed3a9e670d feat(gateway): serve DeleteObject on both stacks
Move DeleteObject to an owned use case and share authorization between the legacy and gateway edges. Keep the legacy request fields at the entry so validation and audit run in order. Give the refusal matrix its own bucket.

Tests: 168 passed on e97dc74d9; the restored mutation control passed 9 tests before an equivalent test-only Clippy fix. RustFS Clippy passed in both configurations; e2e_test Clippy and test wiring passed. The recursive authorization table covers 24 refusals and 16 allows. The header and caller tables add three refusals and five allows across both request forms; the wire cases add key, version and caller refusals.

Mutations: bypassing the primary, force, authenticated-caller, or governance decisions turned their refusal cases into allows. Skipping descendant delete, replication, or bypass decisions turned the respective refusal rows into allows. Removing context restoration failed both allowed and refused rows. Disabling the helper removed the invalid-key audit. The gateway root matrix stayed green for each mutation except the primary decision, where its root row passed and the unauthorized rows were served.

Not verified: the full dual-stack differential scenario requires PutObject and bucket versioning on the gateway stack. CI has not run for this candidate.
2026-11-14 07:55:37 +08:00
overtrue e3858df4fa feat(gateway): serve object metadata operations
Serve object ACL reads and writes, legal hold and retention reads, tagging reads and deletes, and torrent requests through shared use cases and authorization.

On base af93bf466, 131 object metadata and access tests, four refusal matrix tests, and five targeted behavior controls passed. Mutations covered each authorization, version actions, retention UTC conversion and range, ACL responses, tags, lock checks, missing objects, version parsing, and store initialization. Both package clippy configurations passed.

After rebasing onto e97dc74d9, the gateway-stack all-target check and all four refusal matrix tests passed. Full workspace tests and differential scenarios were not run.
2026-11-14 07:36:21 +08:00
overtrue 8e64a3aa2f feat(server): configure gateway bucket owner source
Use the owner ID RustFS reports for gateway expected-owner checks. A missing bucket is still resolved by its handler; no per-bucket owner lookup is added.

The gateway wire table covers six refusals and four successes across ordinary, missing, and reserved-shaped buckets and ListBuckets. The same table runs against the legacy stack. The source unit test covers a reported owner and an unavailable owner.

Mutations: removing the source makes the gateway table fail; returning the wrong owner makes the gateway table and owner unit test fail; returning the wrong owner only for the missing bucket changes its matching-owner response from 404 to 403; returning an owner when none exists makes the owner unit test fail.

Verified against base e97dc74d9: both RustFS clippy configurations, three focused tests, test wiring, and prepush quick checks. Successful SDK responses are checked as success, without asserting their HTTP status. Full workspace tests and CI were not run locally.
2026-11-14 07:33:47 +08:00
overtrue 092c8c272d test: accept owned bucket in metadata reload guard
Verified both bucket guards, default and gateway-stack Clippy, formatting, test wiring, and prepush checks. Removing the CORS notification and flipping its maintenance flag each failed the guard; the restored control passed.

Tests ran on e97dc74d9 before rebasing over unrelated KMS changes at 3c59e4f30. Full workspace tests were not run locally.
2026-11-14 07:32:45 +08:00
唐小鸭 3c59e4f306 fix(kms): tolerate Vault reads that lag the client's own writes (#9406)
* fix(kms): tolerate Vault reads that lag the client's own writes

A Vault or OpenBao node that answers reads from its own copy (an OpenBao standby, a Vault Enterprise performance standby) can serve the state from before a write for a short while after acknowledging it, and can refuse a write it is able to reject from that copy without forwarding it.

- Create a KV2 key in one create-only write that carries usage, description and tags, instead of writing the record and reading it back to attach them.
- Pause with jittered backoff before re-reading after a lost check-and-set write, and allow five attempts instead of three, in both backends.
- Remember for five seconds what this client last wrote or deleted at each record and re-read an answer that is older. The last answer is returned once the attempts are spent, so this only delays a read.
- Retry a Transit key delete that is refused because the deletion switch is not visible yet.

* docs(kms): describe running the Vault backends against OpenBao

* fix(kms): keep the remembered write version from moving backwards

Two writes of one record by the same client can have their acknowledgements processed out of order. Recording the lower version last made it the floor again, so a read served from that older version was accepted. The marker is now updated atomically and only raised within a record's lifetime; a write after this client's own delete starts the count again.
2026-11-14 06:59:29 +08:00
overtrue e97dc74d93 feat(gateway): serve GetBucketLocation on both stacks
Route the operation through the owned use case and shared authorization.
The tests cover two bucket names and a missing bucket on both stacks.

Validation on 20ea63e27: both clippy configurations, targeted nextest
(113 passed), and test wiring. Bypassing location authorization made the
gateway refusal matrix fail for anonymous and no-policy callers. The main
legacy comparison passed. Non-default regions were not verified.
2026-11-14 05:21:23 +08:00
overtrue e83b6815ed feat(gateway): serve DeleteBucket on both stacks
Share the use case and authorization across both edges. Preserve the
force-delete header rules and operation accounting.

Validation on 20ea63e27: both clippy configurations, targeted nextest
(113 passed), test wiring and main legacy comparison. The force-delete table
covers eight refusals and six successful cases; additional tests cover header
precedence and anonymous invalid input. Bypassing DeleteBucket authorization
made the gateway refusal matrix fail for anonymous and no-policy callers;
bypassing ForceDeleteBucket made the false-header authorization test fail.
Site replication and multi-node deletion were not verified.
2026-11-14 05:21:23 +08:00
overtrue 20ea63e27e feat(gateway): serve bucket CORS mutations
Route PutBucketCors and DeleteBucketCors through owned use cases and shared authorization, preserving audit, storage, and legacy edge behavior.

Coverage includes eight refusal and two root matrix scenarios per stack, a wrong-resource refusal, missing-context refusal, and CORS persistence cases. The scoped suite passed 125 tests on 05554bee0. Bypassing Put or Delete authorization failed the gateway refusal matrix; skipping each storage mutation failed the CORS persistence test; allowing a missing context failed the access unit test.

Not verified: full workspace CI and the missing-bucket plus invalid-configuration error ordering. The delete audit status remains 200 while the response is 204.
2026-11-14 04:00:05 +08:00
唐小鸭 a984f42b0b fix(replication): stop re-sending existing objects every scan cycle (#9412)
* fix(replication): stop re-sending existing objects every scan cycle

Existing objects on a site-replication source were re-queued and
re-sent on every scanner cycle, so the replica kept rewriting the same
versions and its reported bucket size grew while the object count stayed
flat.

Two defects combined:

- The per-target resync stamp was computed by the worker but never
  written back to the source object, so the resync decision kept
  returning "replicate" for every object older than the reset date. The
  status writeback now persists the stamp under both internal metadata
  prefixes, and a successful full-object send of an existing object is
  stamped as well.
- The target HEAD only reported the tag count, so the tag comparison
  always saw a difference for tagged objects and fell back to a full
  re-send. The replication HEAD now sends
  x-amz-tagging-directive: ACCESS and the server answers with the tag
  set under the same authorization as the tag count. Targets that do not
  return the tag set still compare as different, which costs one extra
  send instead of a loop.

Also drop replicate_all_stat_options, which built the directive but was
never passed to the HEAD call.

Refs #9354

* test(replication): use relative paths in the resync stamp test
2026-11-14 03:59:46 +08:00
overtrue f9f52f9fad fix(app): preserve stored entity tag spellings
Read stored entity tags with the existing legacy spelling rules before
constructing gateway values. Keep the original control-character coverage.
This prepares the shared helper without adding an operation caller.

Tests: four focused tests passed on 05554bee0, covering eleven spellings,
weak empty tags, control characters, and invalid header values. The table
covers seven malformed or boundary spellings and four ordinary spellings.

Mutations: weak, quotes, bare weak prefix, nested quotes and plain values
failed the spelling table; control characters failed the preservation and
header tests; empty weak tags failed the weak-tag test; removing the version
header failed the header test.

Not verified: operation-level wire responses and full workspace tests.
2026-11-14 03:50:33 +08:00
overtrue af93bf4663 feat(auth): share write-path authorization checks
Share replication header, write ETag, and staged multipart authorization across request forms, preserving the existing decisions and marker cleanup.

The parity table covers eight denials and six allowed cases. Mutations of each authorization call, replication marker and header source, ETag header source, and staged marker cleanup on success and failure all failed the parity test.

Tests and both Clippy configurations passed on 05554bee0. The gateway handlers are not included in this increment. The script test suite did not pass locally: four unchanged scanner fixture tests failed twice with expired fixture deadlines.
2026-11-14 03:43:04 +08:00
clangenb 4db694f1c4 docs(readme): add Docker Swarm install via the SwarmCLI rustfs chart (#9405)
* docs: add Docker Swarm with the SwarmCLI chart to the README quickstart

Option 7 links to the Docker Swarm guide on docs.rustfs.com and to the
community rustfs chart, noting that Eldara Tech maintains both.

* docs: point README Option 7 at the SwarmCLI chart guide's new path

The docs guide moved from Container to Cloud Native as SwarmCLI Chart.

* docs: name the SwarmCLI chart's key features in README Option 7

Secrets, versioned upgrades and rollbacks, Traefik HTTPS; and that its
Console is off and rustfsadmin refused, unlike Accessing RustFS below.

* docs: call the SwarmCLI chart open-source rather than community in README

It is maintained by Eldara Tech, which 'community' contradicted.

* docs: add the SwarmCLI chart's distributed mode to README Option 7

The chart guide now covers distributed, erasure-coded clusters as well
as a single-node service, so Option 7 no longer says single-node only.
2026-11-13 14:53:55 +00:00
Adrián Pérez 0e8fb73e57 perf(lifecycle): skip immediate-expiry lookup without lifecycle rules (#9404)
After every successful write, enqueue_immediate_expiry took the bucket
metadata-transaction read lock and read the bucket incarnation before
checking the cached bucket metadata that decides the outcome. A cached
entry without any lifecycle document can never enqueue an expiry.

Return early in that case. Buckets with lifecycle rules, unreadable
lifecycle documents, a pending creation commit or no cached entry keep
the authoritative path.

Signed-off-by: AdrianPerez3 <70107702+AdrianPerez3@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:52:54 +00:00
Hauser Zhi 05554bee00 chore(deps): clean up unused dependency declarations (#9411)
Move test helpers to dev-dependencies and clean unused build
requirements and workspace metadata.

Use profiling compatibility entrypoints for probe-free crates and
verify the application's hotpath, allocation, and CPU feature graphs.

Validate the affected libraries, FTPS/SFTP test targets, and 1729
existing library tests.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-14 02:04:31 +08:00
Hauser Zhi 8d5faa5750 fix(ecstore): persist restartable format bootstrap authority (#9359)
* fix(ecstore): persist restartable format bootstrap authority

Persist all-drive creator plans before format publication and resume exact deployment and drive slots after interrupted fresh formation. Consume authority durably before startup publication and validate regular control files on full and streamed reads.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(ecstore): finalize format bootstrap once

Order bootstrap retirement before initialized identity publication, preserve established expansion preflight repair, and require followers to observe committed topology and complete identities.

Publish bootstrap controls through one authenticated atomic RPC on the dedicated fsync lane. Keep cancellation-safe cache ownership in the native operation and retain interoperable replacement healing.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): fence anchored format publication retries

Select the durable first-slot plan before fanout and complete immutable anchored plans across retries. Retain physical-root namespace ownership through native publication, retirement and cache invalidation, rejecting conflicting or retired control records.

Reuse main's unary RPC lifecycle recorder for validated outcomes and exact request, byte and cancellation accounting.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(test): align Windows and test-util compilation gates

Match physical publication-hook tests to their non-Windows provider. Compile pool-capacity overrides with their test-util consumers, and keep alias-list mutation inside the Unix branch.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): target native format publication paths

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* style(rpc): format merged request imports

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Chris An <anzhengchao@gmail.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
2026-11-14 01:04:03 +08:00
Hauser Zhi a9f308b075 test(ilm): stabilize manual transition accounting and pagination (#9409)
* test(ilm): account for completed immediate transitions

PUT and lifecycle configuration updates can enqueue transitions before a manual scan. Assert complete per-object accounting and verify a repeated scan skips the committed transition without duplicating the remote object or changing GET content.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ilm): stabilize continuation eligibility across restart

Wait for configuration-triggered transitions to commit before testing opaque continuation tokens. Assert both pages skip exactly one completed object, never enqueue work and preserve the two remote objects across restart.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi <qiuzgang@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
2026-11-14 01:02:11 +08:00
Adrián Pérez 2b0d74027a fix(ecstore): keep delete activation off the metadata read lane (#9402)
Strict recoverable-delete activation renamed the prepared transaction
directory on the disk's single-worker foreground lane, which also serves
metadata reads, and then queued its directory fsyncs as a separate
fsync-lane job. A stalled activation rename blocked unrelated metadata
reads on the same disk.

Run the rename, the fsync of the pinned object directory and, for a
missing original, the ancestor fsyncs in one fsync-lane closure. The set
and order of fsyncs are unchanged; relaxed and none modes keep the
previous rename path.

Signed-off-by: AdrianPerez3 <70107702+AdrianPerez3@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:51:38 +00:00
Hauser Zhi 964cb463a2 chore(deps): refresh Prost and WebPKI dependencies (#9392)
* chore(deps): refresh Prost and WebPKI dependencies

Run cargo update --verbose and cargo upgrade --verbose twice from main.
Upgrade the Prost family to 0.14.6 and rustls-webpki to 0.103.17.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ci): document shard buffer unsafe allowance

Document the arena block validity and exclusivity contract beside the
unsafe_code allowance to satisfy the existing guard.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test: avoid typo false positive in PUT body chunks

Use hell and o as fixture chunks that concatenate to hello.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* chore(deps): upgrade rustfs-erasure-codec to 9.1.0

Run cargo update --verbose and cargo upgrade --verbose again.
Resolve rustfs-erasure-codec 9.1.0 and refresh the workspace lockfile.

Validate the integrated legacy GET producer stack fix with all 28
reported scenarios at a 2 MiB stack in both nextest profiles, plus
codec compatibility cases and the producer future-size regression.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(storage): follow moved request denial audit calls

Inspect both legacy access and owned object authorization production
sources when checking request-level denial audit wiring.

Add regression coverage for a moved call, removal of that call, and
audit calls confined to test code.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-13 14:15:42 +00:00
cxymds ef6af77026 feat(heal): add bounded MRF management and streaming audits (#9399)
* feat(heal): add bounded MRF management and streaming audits

Publish lightweight responsibility summaries and expose authenticated, identity-bound cluster administration without reading object payloads. Stream audit-only reads while retaining staged verified bytes and full target readback for migration.

* test(admin): keep MRF coverage within gateway boundaries

Exercise owned MRF handlers through the shared admin test seam and keep wire-level authentication assertions in the router tests, without introducing another direct s3s consumer.

* test(e2e): complete MRF lifecycle lock fixture RPC

* docs: clarify integrity job resume refusals

Explain that a failed lock acquisition does not prove a live worker, and require job-status reconciliation before retrying an unknown request outcome.

* test(admin): include MRF cluster routes in route matrix

Keep the exhaustive admin route matrix aligned with the four registered cluster management endpoints, including the concrete node and responsibility ID probe.
2026-11-13 23:03:02 +08:00
Hauser Zhi 707694c097 fix(ecstore): make retired part settlement idempotent (#9393)
* fix(ecstore): make retired part settlement idempotent

Resolve a failed source admission against the canonical transaction while
retaining SourceRW ownership. Treat proven absence as already settled when
cached directory descriptors still refer to an upload retired into trash.

Cover a real pinned part transaction through certified Abort, existing closed
transactions and dangling aliases. Preserve active settlement barriers and
rooted no-follow path validation.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(e2e): bound offline OPA fixture future stack usage

Construct offline storage futures in a separate noninlined frame before
boxing them. Keep the bucket-tag fixture within the normal test thread
stack while retaining real metadata writes and authorization assertions.

The affected debug test and the full STS/OPA module pass with a 2 MiB
thread stack after the original CI smoke stack overflow.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(e2e): isolate OPA contract on a dedicated stack

Run the bucket-tag OPA contract on a dedicated 32 MiB thread with a current-thread Tokio runtime. Nested ECStore initialization poll frames can overflow the default test stack even when the outer future is boxed.

Remove the superseded future-boxing helper. Validate all eight STS/OPA tests and targeted runs with 1 MiB and 512 KiB entry stacks on Linux ARM64, plus all-target Clippy and formatting checks.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-13 08:25:50 +00:00
Hauser Zhi f621f28d01 fix(lock): lengthen contended writer retries and report holders (#9390)
* fix(lock): extend contended writer retries within caller budget

Keep the initial probe, shared locks and batch waits at their existing peer limit. Allow later exclusive contention rounds to use a bounded longer wait so read holders outlasting one short round do not repeatedly erase writer progress. Preserve the caller deadline, quorum and grant cleanup, and cover the original failure with real LocalClient grants.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* feat(admin): expose physical holder counts in top locks

Serialize the node-local physical holder count from the existing fast-lock snapshot for each resource/version/mode/owner cohort. Keep row totals and lease-metadata validation, and cover repeated owners, stale leases and the actual JSON field.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* docs(admin): clarify node-local holder sampling

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-13 07:23:24 +00:00
overtrue d57e781375 feat(gateway): serve the policy, status and ACL reads on the gateway stack
GetBucketPolicy, GetBucketPolicyStatus and GetBucketAcl are served by the
gateway stack and by the legacy stack through one owned use case each. The
legacy access hooks call authorize_bucket_request, the function the gateway
edge calls; the owner's bypass of a bucket-policy Deny for GetBucketPolicy stays
in the shared authorization core. BucketAcl gets its use case in
bucket_usecase/acl.rs: the answer (one owner, full control, a grantee carrying
no identity) is unchanged.

Not built: GetBucketNotificationConfiguration, the four mutations and
PutBucketAcl (they wait for shared helpers).

Removed code: build_get_bucket_acl_output in storage/s3_api/acl.rs, replaced by
the use case.

Removed tests: test_build_get_bucket_acl_output_contains_full_control_grant_and_owner,
covered by the_gateway_stack_answers_the_bucket_acl and
the_legacy_stack_answers_the_bucket_acl (one owner, full control, grantee with
no id and no display name, NoSuchBucket for a missing bucket) in
tests/embedded_gateway_stack_test/bucket_access.rs.

Tests: refusal-matrix rows for the three reads on both stacks; NoSuchBucketPolicy,
the verbatim policy document, the owner reading a policy that denies everyone,
IsPublic following the stored policy, and the ACL answer, on both stacks;
use-case tests ported to the owned signatures.

Mutations: the gateway authorize answering Ok(()) in each of the three reads
turns that operation's anonymous and no-policy rows of the refusal matrix red.

Legacy answers compared with main: see the previous commit (40 rows, identical,
including the missing content-type of the stored-policy answer).
2026-11-13 20:32:56 +08:00
overtrue da07ad6a08 feat(gateway): serve GetBucketVersioning on the gateway stack
GetBucketVersioning is served by the gateway stack, and by the legacy stack
through the same owned use case (gateway input and output types). The legacy
access hook now calls authorize_bucket_request, the function the gateway edge
calls. The stored status is carried verbatim as the gateway status enumeration,
so a stored value is never dropped. As before, MFADelete is not answered.

Not built: PutBucketVersioning (it waits for the shared bucket-configuration
mutation wrapper and the owned audit call).

Tests: refusal-matrix row on both stacks (anonymous and a user without a policy
are refused, root is served and counted once); GetBucketVersioning answers every
stored bucket name the SDK reaches (a-s3alias, sthree-x, x--ol-s3), a stored
Enabled status, NoSuchBucket for a missing bucket, and the 400 InvalidBucketName
both stacks give xn--abc, on both stacks; the ported use-case test.
Differential cases: an unconfigured and a suspended bucket.

Mutations: the gateway authorize answering Ok(()) turns the anonymous and
no-policy rows of the refusal matrix red; dropping the status from the use case
turns the stored-status assertion red (left None, right Enabled).

Legacy answers compared with main (status, headers and body; the SDK output for
successes): 40 rows, for plain-bucket, sthree-x, a-s3alias, x--ol-s3 and a
missing bucket, each unconfigured and configured, for GetBucketVersioning and
the three reads of the next commit. All identical.
2026-11-13 20:32:56 +08:00
overtrue 17081a6d23 feat(server): serve the lifecycle, replication and lock reads
Batch A3 of rustfs/backlog#2765, second increment: the gateway stack serves
GetBucketLifecycleConfiguration, GetBucketReplication and
GetObjectLockConfiguration through the owned use cases from the previous
commit.

- `impl Served` for the three operations and their rows in the A3 family. Each
  `authorize` calls `authorize_bucket_request` with the action its legacy hook
  asks (GetBucketLifecycle, GetReplicationConfiguration,
  GetBucketObjectLockConfiguration) on the decoded bucket.
- The three rows of the refusal matrix. Each row stores its configuration
  through the bucket metadata first, so root is served without depending on the
  Put operations.
- The behaviour test from the previous commit also runs on the gateway stack,
  for an ordinary and a reserved-shaped bucket name.

Tests (negative / positive): refusal matrix 3 rows x 4 refused callers (12
negative, 3 positive; each is counted once under its legacy label);
gateway-stack behaviour test: NoSuchBucket 404 (3), each read's own 404 (6),
wrong-resource AccessDenied 403 (6), stored contents (6 positive).

Mutations: the three `authorize` bodies allow everything. Red:
the_refusal_matrix_holds_on_the_gateway_stack (each of the three reads serves
the anonymous and the no-policy caller), and
the_gateway_stack_answers_the_configuration_reads_as_stored (the wrong-resource
caller is served). The legacy-stack variants stay green.

Not built: the five mutations.

Not verified: the differential's gateway side (CreateBucket, PutObject and the
Put configuration operations are not all served yet; Linux reference only).
storage::access::tests::denial_exits_without_a_context_call_the_audit_function
fails on this base: the source guard looks for the force-delete exit in
access.rs, and that exit now lives in access_owned/object_misc.rs. It is not
this change.
2026-11-13 20:17:47 +08:00
overtrue d36c3595ad feat(gateway): serve CreateBucket on the gateway stack
CreateBucket is served end to end by the gateway stack; the legacy stack
serves it through the same owned use case.

- Use case: execute_create_bucket(envelope, gateway input) -> gateway
  output, on the fresh admitted task as before (the text guard names the
  new inner call). The audit entry and event come from the owned helper.
- Authorization: authorize_create_bucket in storage/access_owned/
  bucket_core.rs is the one composite hook both edges call: s3:CreateBucket
  with the body's LocationConstraint as a condition value (first decision
  without the presigned-query hoist), then PutBucketObjectLockConfiguration
  and PutBucketVersioning when object lock is requested.
- Legacy edge: record_s3_op, then through_gateway_seam.
- Gateway edge: impl Served and one ROWS entry.

Not built: no change to what CreateBucket answers; DeleteBucket and
GetBucketLocation follow in their own commits.

Tests (negatives 9, positives 8):
- embedded, both stacks: the_{gateway,legacy}_stack_creates_buckets_as_before
  (fresh, idempotent re-create, foreign and EU constraint refused, the four
  reserved name families created, xn-- refused, x--x-s3 refused by the SDK
  on the client); refusal-matrix row (anonymous, wrong secret, unknown key,
  no policy refused; root served, counted once).
- parity: owned_and_legacy_create_bucket_authorization_use_location_constraint
  gains six composite cases (location, object lock) comparing decision and
  handed-on bucket, legacy hook against owned function.
- use case: tests ported to the owned signature; empty LocationConstraint
  reads as none.
- differential: CreateBucket cases for a foreign constraint, EU and the
  reserved name families.

Mutation: Served::authorize returns Ok(()) ->
the_refusal_matrix_holds_on_the_gateway_stack red ("CreateBucket serves the
anonymous caller", "... the no policy caller").

Changed assertion: guards.rs names the inner call
execute_create_bucket_inner(envelope, input, mutation_permit) instead of
(req, mutation_permit); same check, new signature.

Metric (ledger candidate, no wire change): the legacy edge now counts every
CreateBucket call, including one its own validation refuses.

Not verified: the differential run against 1.0.5 (Linux CI).
2026-11-13 19:15:24 +08:00
overtrue db5a910b69 fix(s3-types): keep a typed cause across the s3s round trip
A typed cause now survives the s3s -> owned -> s3s round trip when the
code is unchanged. Before, an s3s error that crossed into
`rustfs_s3_types::S3Error` carried its whole self as a `LegacyCause`
wrapper, and crossing back set that wrapper as the cause, so a caller
downcasting the cause after an owned use case (a legacy edge's
diagnostics, a test asserting `StorageError::ErasureReadQuorum`) found a
wrapper. Crossing back now hands the original s3s error on and applies
the message, request id, status and headers on top; an error our side
re-coded keeps the wrapper, which shows the cause's text, as before.

Test (1 added, both directions):
`a_round_trip_hands_the_typed_cause_back_unless_the_code_changed` - an
untouched round trip is exactly the original (code, message, explicit
status, headers, typed cause); a changed message is applied with the
typed cause kept; a re-coded error keeps only the cause's text.

Mutation: always keeping the wrapper (the hand-back arm dropped) makes
the new test red.
2026-11-13 19:08:01 +08:00
overtrue 77ce76fde9 test(storage): read the force-delete denial exit where it now lives
The source-level guard that every denial exit without a context calls
audit_request_denial read only storage/access.rs. 8eb89dd54 moved
DeleteObject's decision, with its recursive-force-delete exit, into
access_owned/object_misc.rs and left the guard behind, so the guard went
red although the exit never stopped auditing. The guard now reads each
exit in the file that holds it.

Tests: the storage::access lib module, run whole: 113 passed.
Mutation: removing the audit_request_denial call from the exit in
access_owned/object_misc.rs turns the guard red with
"\"recursive_force_delete_requires_authenticated_caller\" exit does not
call audit_request_denial".
2026-11-13 18:59:43 +08:00
overtrue d645456a90 refactor(storage): carry body-sourced policy inputs in owned form
The policy inputs some hooks take from a request body are now owned
types that both request forms carry and install through
`AuthorizationRequest`, so the gateway edge no longer needs s3s DTOs:

- `RequestedObjectLockRetention { mode, retain_until }`.
- `RequestedObjectLegalHold(Option<String>)`, the status.
- `RequestedObjectTags(Vec<(String, String)>)`. It keeps only complete
  key/value pairs; those are the only tags a condition ever named.

Each type converts from its s3s DTO once, at the legacy insert site
(`From<&ObjectLockRetention>`, `From<&ObjectLockLegalHold>`,
`From<&Tagging>`). The conditions built from them are unchanged.

New trait installers: `install_requested_retention`,
`install_requested_retain_until`, `install_requested_legal_hold` and
`install_requested_object_tags`. The legacy hooks use them.

`set_requested_object_lock_retain_until` is now generic and takes an
owned `OffsetDateTime`.

`RequestEnvelope::fork_for_authorization` copies a request for one
more authorization, such as an archive member. Every field is copied,
the access phase's output included. The table publication guards stay
one shared set, so a lock retained through the copy is held for the
original's write. Anything else installed on the copy does not show on
the original.

Tests:
- A new parity test installs body inputs on both forms. The installed
  bodies, not conflicting headers, must describe the request, and an
  incomplete tag must name nothing.
- A fork test checks that the copy reads like the original and that
  what is installed on the copy stays off the original.
- The object-shape column of the decision table now runs on a forked
  envelope.
2026-11-13 18:09:11 +08:00
overtrue fa13764bb7 feat(bucket): answer lifecycle, replication and lock reads as owned
Batch A3 of rustfs/backlog#2765, first increment: GetBucketLifecycleConfiguration,
GetBucketReplication and GetObjectLockConfiguration are owned use cases
(gateway DTO in, gateway DTO out). The legacy stack serves them through
`through_gateway_seam`. Nothing is registered on the gateway stack yet; that
comes in the next commit, once the gateway error mapping answers a missing
bucket 404.

What changes:
- `app::dto_values::stored_config(&T: BucketConfigXml, parse)` is the one way a
  stored bucket configuration becomes a gateway DTO: its persisted bytes, read
  by the gateway's DTO bridge (`parse_*_dto`). A refusal is InternalError.
- `execute_get_bucket_lifecycle_configuration` and `execute_get_bucket_replication`
  take and answer gateway DTOs. `execute_get_object_lock_configuration` is new
  in `bucket_usecase/object_lock.rs`, moved from the inline legacy edge body.
  Codes and messages are unchanged.
- The legacy edges of the three reads count the operation and go through the
  seam. The input members the seam converts are `bucket` and
  `expected_bucket_owner` for each.
- The three legacy access hooks call `authorize_bucket_request` with their
  legacy action (GetBucketLifecycle, GetReplicationConfiguration,
  GetBucketObjectLockConfiguration).
- `storage/s3_compat/{lifecycle,replication}.rs`: the reverse (stored -> s3s)
  direction is no longer on the request path and is compiled only under
  `cfg(test)`, as the oracle of the parity tests below and of replication's
  existing round-trip tests. It goes, with retirement entries, when the Put
  edges cross the seam.

Tests (negative / positive):
- Legacy answer parity against main's own conversion: lifecycle (XML bytes,
  with DelMarkerExpiration, ExpiredObjectAllVersions, Transition,
  NoncurrentVersionTransition, an And/Tag filter and dates, one with a +08:00
  offset), replication (XML bytes, with DeleteReplication,
  ExistingObjectReplication, ReplicaModifications) and object lock (value; it
  holds no timestamp). 3 positive.
- dto_values: a stored configuration round-trips byte for byte through the
  bridge (1 positive); a bridge refusal is InternalError (1 negative).
- Use cases: store not initialized is InternalError for each read (3 negative;
  the replication one is ported to the owned signature).
- Embedded, legacy stack, for an ordinary bucket and a reserved-shaped one
  (`sthree-a3-stored`, `a3-empty-s3alias`): missing bucket is NoSuchBucket 404
  (3), no configuration gives each read's own 404 code (6), a caller allowed on
  other buckets only is AccessDenied 403 (6), the stored contents are answered
  (6 positive). 15 negative, 6 positive.
- Differential (`scenarios/extra/bucket_replication_lifecycle.rs`): lifecycle
  with every member and a dated rule on a reserved-shaped bucket, lock reads
  before and after a Years rule, absent-configuration reads.

Mutations (run on this tree before the rebase onto ea56aa923):
- M1, `stored_config` rewrites a stored member (WARM -> COLD) before the
  bridge: red in s3_compat::lifecycle::tests::the_legacy_answer_through_the_seam_is_the_answer_of_the_reverse_direction,
  dto_values::tests::a_stored_configuration_is_answered_with_every_stored_member,
  and embedded the_legacy_stack_answers_the_configuration_reads_as_stored.
- M2, the GetBucketLifecycleConfiguration legacy hook allows everything: red in
  embedded the_legacy_stack_answers_the_configuration_reads_as_stored (the
  wrong-resource caller was served).

Not built: the gateway edge (`impl Served`, rows, refusal-matrix rows, the
gateway-stack variant of the behaviour test), the five mutations.

Not verified: the differential (needs the Linux reference binary, so it is
CI's); the e2e_test crate built only by clippy.
2026-11-13 17:57:35 +08:00
overtrue c221fb62a3 test(storage): audit each mutation without an event at exactly one place
The audit coverage guard required the legacy wrapper for every mutating
operation without a notification event. A migrated operation writes its
audit entry in the use case instead, so both stacks record the same entry,
and its legacy edge drops the wrapper. The guard now accepts exactly one
audit site per operation: the legacy wrapper in ecfs.rs / legacy_edge,
or the owned call under src/app. Both sites at once or neither is a failure.
Calls are matched with whitespace ignored, so rustfmt wrapping and trailing
commas still match.

Accepted owned shapes under src/app:
  audit_only_operation(envelope, S3Operation::X, ...)
  OperationHelper::audit_only_from_view(&envelope.audit_view(), S3Operation::X)

SourceSet gains count_ignoring_whitespace for this.

Tests: the production guard (all 22 operations, still on the legacy
wrapper); fixtures accepted (legacy only, owned helper wrapped by rustfmt
with a trailing comma, owned envelope wrapper wrapped); fixtures refused
(neither, legacy plus each owned form, an owned call only under #[cfg(test)],
a wrapper and an owned call for UploadPartCopy, which must not count for
UploadPart). Negatives 6, positives 4.

Mutation: owned helper needle renamed so it never matches; both fixture
tests went red (the_guard_accepts_exactly_one_audit_place,
the_guard_refuses_an_operation_audited_nowhere_or_twice), the production
guard stayed green. An earlier run of the fixtures caught a needle that
missed rustfmt's trailing comma; fixed before this commit.

Not verified: nothing beyond the audit_coverage_tests module was run.
2026-11-13 17:36:03 +08:00
overtrue ad94f6914a feat(gateway): answer RustFS refusal facts in their legacy form
The gateway stack's one error mapping now answers every code the
gateway refuses without a context (its is_contextual set: NoSuchKey,
NoSuchVersion, NoSuchBucket, PermanentRedirect, TemporaryRedirect,
NotModified, AuthorizationHeaderMalformed, MethodNotAllowed,
BucketAlreadyOwnedByYou, AccessForbidden) and InvalidRange through
the gateway's LegacyRustfsRefusal, stating the facts the error's own
response headers carry:
- NotModified: ETag, Last-Modified, Cache-Control, Content-Location,
  Expires, Vary and the evaluated x-amz-version-id;
- InvalidRange: Content-Range: bytes */<length>;
- NoSuchKey / MethodNotAllowed on a delete marker:
  x-amz-delete-marker: true, x-amz-version-id, Last-Modified.
s3s takes no part. A fact a code may not state, an unreadable or
repeated fact, or a refusal the gateway cannot state is answered 500
InternalError rather than sent with a member missing. Before this, each
of these codes failed the gateway's context check, so for example
HeadBucket on a missing bucket could not answer NoSuchBucket on the
gateway stack. The list is pinned against the gateway's own behaviour
in both directions.

PermanentRedirect and TemporaryRedirect are answered 500: the legacy
refusal states only 4xx, 5xx and the 304 of NotModified. RustFS raises
neither.

Adds crate::app::dto_values::stored_etag(&str) -> ETag, the one
conversion of a stored entity tag for every answer that carries one:
a tag the gateway reader accepts is read as it reads it, any other
(empty, a control character) is carried byte for byte
(ETag::from_stored). Nothing calls it yet; batch C's reads and B2's
listings are its first callers.

Not built: no operation of batch C is registered here; the
embedded-server tests of these answers come with HeadObject.

Tests: 12 new in gateway_handlers::error with 18 refused rows (facts a
code may not state, unreadable or incomplete facts, redirects, a hidden
missing key) and the list checked against the gateway both ways, versus
15 answered rows; 2 in dto_values (1 carried-verbatim, 1 spellings).

Changed assertions: every_header_of_the_closed_set_is_carried uses
SlowDown instead of InvalidRange, because InvalidRange is now answered
in the legacy form; it still covers every header of the ordinary
closed set.

Mutations: run right after this commit lands, reported on
rustfs/backlog#2767; a mutation that stays green is fixed by the next
commit.

Refs: rustfs/backlog#2767
2026-11-13 17:24:44 +08:00
overtrue 190f3686f4 refactor(storage): own the object-listing authorization for both edges
Move the listing authorization of batch B2 into
`storage/access_owned/listing_multipart.rs`, generic over
`AuthorizationRequest`, so the legacy access hooks and the gateway edge
call one copy:

- `authorize_bucket_listing` (ListObjects, ListObjectsV2,
  ListObjectVersions, the metadata listings): the bucket on `ReqInfo`, the
  listing action, `ListBucketVersions` as well for `include-deleted: true`
  on a `ListBucket` listing, then the table snapshot. The repeated-query
  `InvalidRequest` and the anonymous `ListBucket` fallback stay inside the
  policy decision.
- `authorize_list_objects_v2`: the on-demand migration generation captured
  before the decision and installed after an allow.
- `prepare_table_data_plane_list_access` and its fingerprint/cursor-key
  helpers move here unchanged, now generic; the snapshot reaches the use
  case through the new `AuthorizationRequest::install_table_list_access`.

The ListObjects, ListObjectsV2 and ListObjectVersions hooks reduce to one
call each; ListMultipartUploads keeps its generation guard and calls the
moved snapshot. Same actions, same order, same refusals.

Tests (1 added, 1 extended; negatives in the extended table outnumber the
allows): `table_list_access_reaches_the_use_case_from_either_edge`; the
include-deleted table (`include_deleted_listing_enforces_version_permissions`)
now decides every case on an envelope too and requires the same answer.

Mutations: dropping the `include-deleted` follow-up makes the extended
table red; an envelope installer that installs nothing makes the new test
red.

Not built here: the gateway rows and the owned use cases (they wait for
the gateway re-pin).

Verified: the touched tests (19, listing/table access/metadata route)
ran on f33271a74 (before re-pin 1); feature and default clippy
--all-targets ran there too; feature clippy --all-targets ran again on
941fb4fc1, the final base.
2026-11-13 17:13:13 +08:00
overtrue 941fb4fc11 refactor(access): one owned copy-source authorization for both copies
CopyObject and UploadPartCopy authorized their copy source with two
identical blocks in their S3Access hooks. Both hooks now call
`access_owned::object_commit::authorize_copy_source`, generic over
`AuthorizationRequest`, which the gateway stack's `Served::authorize`
for both operations will call too (batch E1, rustfs/backlog#2769). The
order is the hooks' order, unchanged: an access-point or Outposts ARN
source is NotImplemented before any policy is read; the source is
recorded for the `x-amz-copy-source` condition; its bucket incarnation
is captured; reading it is authorized (s3:GetObjectVersion when a
version is named); a failed capture is surfaced only after the allow,
and the incarnation is handed on. The destination is decided afterwards
by each hook, as before. `AuthorizationRequest` gains the two installers
the shape needs, implemented for `S3Request` and `RequestEnvelope`.

Measurement for the batch (recorded here as the coordinator asked):
none of the three E1 operations commits a 200 before its work ends, so
the gateway edges will answer with `Resp::new`, not `Resp::commit`.
The pinned 1.0.5 reference binary could not be run on this host (the
asset is Linux x86-64 only and no container runtime was usable), so the
current legacy stack was measured instead: the debug binary at e8847f6f1,
default stack, on 127.0.0.1, driven by `curl --aws-sigv4`:
- CompleteMultipartUpload with a wrong part ETag: 400 InvalidPart with
  Content-Length; success: 200 with Content-Length, not chunked.
- CopyObject, missing source: 404 NoSuchKey; success: 200 with
  Content-Length.
- UploadPartCopy, missing source: 404 NoSuchKey; range beyond the
  source: 416 InvalidRange; success: 200 with Content-Length.
The 1.0.5 source agrees (tag 1.0.5, s3s 5761ddfe): s3s writes a
keep-alive body only for CompleteMultipartUpload and only when the
output's `future` member is set (crates/s3s/src/ops/multipart.rs:27);
RustFS never sets it, and CopyObject and UploadPartCopy have no such
path.

Not built: the destination half of either hook, the gateway edges
(they arrive with each operation's own increment), and the copy-source
read guard (task H's generic form is used as is).

Lines outside tests: +105 / -59 (access.rs, envelope.rs, the new
access_owned/object_commit.rs).

Tests (6 negative cases, 2 positive in the parity table; the hook test
has 2 negative cases and 1 control per operation): copy-source parity
on both edges, comparing decision,
recorded source, handed-on incarnation and ReqInfo for an ARN source,
a source without a read policy, a named version under s3:GetObject,
a deny with a failed capture, a failed capture after an allow, a
missing ReqInfo, and the two allows; and both legacy hooks refuse a
denied source and an ARN source without ReqInfo ever naming the
destination.

Mutations (two grouped runs of the two new tests, each reverted):
- ARN source answered Ok: parity "an ARN source is NotImplemented..."
  and hook "CopyObject: a denied source"/ARN case red.
- source decision skipped: parity "a source without a read policy..."
  and "a deny wins over a failed incarnation capture" red.
- version swap dropped (always s3:GetObject): parity "a named version
  needs s3:GetObjectVersion" red.
- capture failure surfaced before the decision: parity "a deny wins
  over a failed incarnation capture" and "a failed incarnation capture
  is surfaced after the allow" red.
- destination check before the source (CopyObject hook): hook
  "CopyObject: a denied source" red.

Not verified: UploadPartCopy's source-before-destination order is not
distinguishable by an answer on the legacy hook (both halves refuse
AccessDenied); it is pinned on the gateway edge with that operation.
The gateway stack is unchanged by this commit.
2026-11-13 16:58:20 +08:00
overtrue 0e44453639 refactor(helper): make run_owned_mutation generic over its result
`run_owned_mutation` owned a mutation on its own task only when the
mutation answered an s3s `S3Response`. The owned use cases of batch E1
(CompleteMultipartUpload first) answer a gateway DTO with the owned
error type, so the helper is now generic over the result:
`run_owned_mutation<R, E, F>(operation: F) -> Result<R, E>` where
`F: Future<Output = Result<R, E>> + Send + 'static`, `R: Send + 'static`
and `E: From<rustfs_s3_types::S3Error> + Send + 'static`. A lost owner
task is still InternalError "object mutation owner task failed"; the
s3s error is reached through the existing `From` conversion, so the
legacy callers (DeleteObject, DeleteObjects, CompleteMultipartUpload)
compile and answer unchanged. The task-locals it carries into the
spawned task (Keystone credentials, the S3 HTTP operation context) are
unchanged.

Not built: no new caller in this commit; the owned
CompleteMultipartUpload use case is the first.

Tests: none added; the existing callers' tests ran unchanged (71 run
in the touched modules, all passed). Not mutation-checked: a pure
signature generalisation with no new assertion.
2026-11-13 16:58:19 +08:00
overtrue 0fde0f14a0 Revert "fix(ecstore): isolate unavailable local drives during startup (#9353)"
This reverts commit f6d64bc1f.

The change was merged outside the merge queue, so the full end-to-end
merge gate never ran on it. On main that gate fails 25 multi-node
cluster tests since f6d64bc1f (distributed_startup_regression_test,
inline_fast_path_cluster_test, heal_erasure_disk_rebuild_test,
tier_stats_cluster_test, upgrade_compatibility_test): every node answers
503 and never becomes ready, logging "Local endpoint has no format on
the system filesystem; restore its data mount" for its fresh data
directories. The run before it (c43e415b0) passed all of them.

Reverting restores the previous startup behaviour so the change can be
re-landed through the queue together with a decision on how fresh
URL-style clusters on a system filesystem (and the cluster test
harness) are meant to start.

The stack-overflow fix c83f47871 in set_disk/ops/object.rs is kept.
2026-11-13 16:27:06 +08:00
overtrue aa5f145983 build: re-pin the gateway to 1bbef575
Brings rustfs/gateway#1613 (listing members a stored entry may lack are
written as no element), #1614 (ListMultipartUploads url encoding under
the RustFS profile) and #1616 (PutObject and UploadPart carry the legacy
content length). No RustFS caller uses the changed converters yet.
2026-11-13 16:06:54 +08:00
overtrue ea56aa9238 refactor(storage): authorize bucket configuration changes on both edges
`authorize_bucket_config_mutation(server_ctx, req, bucket, action)` is
now generic over `AuthorizationRequest`, so the legacy hooks and the
gateway edge run the same shape. The shape records the bucket on
`ReqInfo` and snapshots the bucket incarnation under the lifecycle
namespace read lock before the decision. It surfaces a snapshot failure
only after an Allow, and on Allow it installs the snapshot (new trait
installer `install_bucket_config_mutation`) for
`bucket_config_mutation_incarnation`.

The 22 legacy hooks of the shape now call it, with no other change:

- Delete: BucketCors, BucketEncryption, BucketLifecycle, BucketPolicy,
  BucketReplication, BucketTagging, BucketWebsite and
  PublicAccessBlock.
- Put: BucketAccelerateConfiguration, BucketCors, BucketEncryption,
  BucketLifecycleConfiguration, BucketLogging,
  BucketNotificationConfiguration, BucketPolicy, BucketReplication,
  BucketRequestPayment, BucketTagging, BucketVersioning, BucketWebsite,
  ObjectLockConfiguration and PublicAccessBlock.

DeleteBucketTagging keeps its `with_bucket_tag_deletion` scope at the
call site.

Tests: the decision fixture of the owned-versus-legacy parity tests is
now one shared function. A new parity test runs eight cases through
both request forms and compares the decision and the snapshot handed
on. Three are allowed: IAM user, anonymous via bucket policy, owner.
Five are refused: wrong resource, no policy for the action, anonymous
on a private bucket, anonymous on a missing bucket, owner on a missing
bucket.

Not built here: the gateway edges of these operations (batches A1-A4).
2026-11-13 15:51:23 +08:00
overtrue 8eb89dd542 refactor(storage): decide DeleteObject in one owned function
DeleteObject's composite access decision moves out of the legacy S3Access
hook into access_owned::object_misc::authorize_delete_object, generic over
AuthorizationRequest, so the legacy hook and the gateway edge call the same
code. The legacy hook loads the bucket generation, calls it, and installs
the guard it returns; its order and answers do not change: NoSuchBucket
before policy, DeleteObject[Version] for the version storage deletes (the
source-version header for a replica delete), ForceDeleteObject, the
authenticated-caller rule for a recursive force delete, and
BypassGovernanceRetention.

Not built here: the gateway edge of DeleteObject and its use case on the
owned DTOs (they route the legacy body through the seam and wait for the
re-pin).

Tests (access_owned/object_misc_test.rs): one table test runs both
request forms (s3s request and owned envelope) through the decision and
compares the answer and the state handed on (ReqInfo bucket, key, version;
guard ready or failed): 5 refused (missing bucket before policy,
anonymous, caller without a policy, no caller installed, invalid
force-delete value) and 3 allowed (replica delete naming no version gets
its source version, any other delete ignores that header, an unavailable
generation is handed on as a failed guard).

Mutations (each run collects every failing case):
- NoSuchBucket check removed, invalid force-delete read as false, replica
  version swap removed (one build): the missing-bucket, invalid
  force-delete and replica cases went red, in both request forms
- policy decision skipped: the anonymous and no-policy cases went red,
  in both request forms

Verified locally: clippy -D warnings with and without gateway-stack (on
the base before the last rebase, which touched none of these files); 64
delete-related lib tests (app::object::delete, the lifecycle delete tests,
storage::access delete tests) passed, and the parity test passed in its
own run.

Not mutation-checked here: ForceDeleteObject, the authenticated-caller rule
and BypassGovernanceRetention need a caller a policy partly allows; the
existing use-case tests cover the predicate.
2026-11-13 15:26:49 +08:00
overtrue 4a1d542f2d build: re-pin the gateway to 66193a80 and map seam name refusals once
The pin carries floor-only bucket names across the s3s seam and a typed
ConversionError::name_kind (rustfs/gateway#1610), so HeadBucket's legacy
edge goes through the generated conversion again. seam_error now lives in
crate::app::dto_values and ecfs.rs delegates to it: a bucket name below
the floor stays a server fault, because s3s admits only names the floor
accepts; any other refusal is a server fault as before. handler_error
carries the four stored headers the pin adds to ErrorHeader.

One legacy difference is observable: a key the seam refuses (empty,
longer than 1024 bytes, NUL) is answered 400 InvalidArgument with the
fixed message "Invalid object key", where the message text previously
came from the store or the key validator.
2026-11-13 14:30:24 +08:00
overtrue 75f53303e1 fix(storage): count an operation once, at its edge, not in the owned helper
An operation is counted once per request by its edge, after authorization
and before the use case: the gateway's blanket handler already does so, and
a migrated operation's legacy edge calls record_s3_op at the top of its
impl S3 body. OperationHelper::from_view therefore no longer counts; the
legacy constructor OperationHelper::new keeps counting exactly as before for
operations that are not migrated yet. Without this, every owned mutation
that builds its helper with from_view would count twice on the gateway stack.

Not built: no caller of from_view changes here; the first one is the owned
CreateBucket use case.

Tests (1 negative, 1 positive in one test):
only_the_legacy_helper_counts_the_operation - the owned helper counts
nothing; the legacy one counts once.

Mutation: record_s3_op(op) put back into from_view ->
only_the_legacy_helper_counts_the_operation red ("the owned helper counts
nothing").

Local gate: fmt; clippy --all-targets -D warnings with gateway-stack and in
the default build; nextest storage::helper + app::owned_audit_test, 28 passed.
2026-11-13 14:25:35 +08:00
overtrue f33271a743 test(storage): compare resource-free conditions without the clock keys
`resource_free_conditions_read_the_same_client_through_both_forms`
compared two condition maps read at different instants, so
`CurrentTime` (and, across a second boundary, `EpochTime`) differed
between the legacy and the owned read, and the test failed. Both clock
keys are now removed before the comparison. Every request-derived key
must still match, and `SourceIp` is still pinned to its expected value.

This corrects a test introduced in 9c394af48.
2026-11-13 14:04:54 +08:00
overtrue c83f47871d fix(ecstore): box the legacy GET producer read pipeline
Since eca95f297 (#9304) every GET through the legacy duplex pipeline
built `run_legacy_producer_segment` futures that embedded the whole
`get_object_with_fileinfo` read future inline, wrapped again in an
`async move` block and the shard-emitter scope. `get_object_reader`
builds one such producer per downstream arm and per segment, so its
debug poll frame grew to 641,888 bytes. Any test that read an object
back through a real store on a 2 MiB thread overflowed the stack and
aborted (28 workspace tests and one e2e test on main, e.g.
rustfs-heal mrf_partial_write_test, heal_integration_test, and
rustfs app::object::internal_put::tests).

The read future is now boxed inside `run_legacy_producer_segment`, so
the producer future is pointer-sized. No behaviour change.

Measured (debug, aarch64-apple-darwin, line-tables-only):
- run_legacy_producer_segment future: 110,200 -> 320 bytes
- <SetDisks as ObjectIO>::get_object_reader poll frame:
  641,888 -> 202,352 bytes (prologue stack allocation)

Regression test (negative): set_disk::ops::object::
legacy_producer_stack_tests::legacy_producer_future_keeps_the_read_pipeline_off_the_stack
asserts the producer future is at most 1024 bytes. Mutation: removing
the Box::pin makes it fail with "got 110200 bytes".

Verified locally: the 7 previously aborting rustfs-heal tests pass;
the rustfs lib tests that aborted on main (app::object::internal_put,
on_demand_migration_put, put sse_c, owned_authorization_test,
storage::access, object_zip_download, directory_index, data_usage
snapshot gating, snowball extract) pass, 94 of 94 selected.
clippy -p rustfs-ecstore --all-targets reports only the existing
unused PoolSpaceForTest items in store/rebalance.rs, not this change.
Not run locally: the full workspace, the e2e test
sts_query_compat_test::test_bucket_tags_opa_contract, and release-build
frame sizes.
2026-11-13 13:52:41 +08:00
Hauser Zhi f6d64bc1fb fix(ecstore): isolate unavailable local drives during startup (#9353)
* fix(ecstore): isolate unavailable local drives during startup

Keep missing local endpoints in their configured slots and initialize healthy
local drives before applying the existing format and service quorums.
Protect unformatted Linux URL endpoints from writes to the system filesystem,
validate legacy admission, and retry missing local slots during format waits.

Refs rustfs/rustfs#9350
Refs rustfs/backlog#3174

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(ecstore): prepare local drives before registry publication

Prepare local disks with bounded concurrency outside inventory locks and publish validated batches under per-instance writer coordination. Preserve existing registrations on failed or cancelled full initialization, reuse current-root handles, and withdraw stale reopened slots.

Unify initial-format and reconnect publication, retain raw and explicit-context constructor contracts, and move filesystem queries into owned native jobs. Centralize format validation and reject special format files before startup writes.

Refs rustfs/rustfs#9350
Refs rustfs/backlog#3174

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* chore(ecstore): tighten formatted-error ratchet after refactor

Drop the three removed runtime registration call sites from the shrink-only baseline.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test: align drive fixtures with startup admission

Expect the disk-scoped missing-root error and enable the existing directory test mode for the synthetic URL-based receiver fixture.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(ecstore): unify local drive admission and publication

Share bounded preparation and pre-I/O membership snapshots across local
startup and retry paths, with coherent registry and topology publication.
Coalesce native startup preflight and preserve root and buffer ownership.
Keep reconnect ID caches instance-scoped and retain admitted health monitors.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): pin multipart index queue aliases to owner lifetime

Derive the Linux index alias from the existing identity descriptor so a
retired caller descriptor cannot redirect rebuild or reconnect queue lookup.
Retain the original descriptor assertion and cover cross-disk alias reuse.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-13 00:04:54 -05:00
Hauser Zhi d37640fbb9 fix(grpc): record TLS setup failures and cancelled dial attempts (#9388)
* perf(protos): balance peer channels and synchronize TLS eviction

Use bounded per-peer bulk cursors and shared TLS generation reads. Keep eviction and generation removal coherent with concurrent publication, and expose opt-in bulk transport tuning with checked capacities.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* feat(metrics): observe internode unary request lifetimes

Record client attempts through errors and cancellation, preserve the published server-duration family, and add gated activity and completion metrics. Cache enabled instruments after recorder publication while retaining startup duration recording.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(perf): make internode gRPC experiments reproducible

Reset phase-owned settings before applying overrides, separate lifecycle experiments, and provide a bounded loopback workload with independently reported control and bulk latency.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(perf): isolate transport knobs and redact dry runs

Add single-parameter P0 presets with separate artifacts while preserving the legacy combined default. Mask credential arguments in dry-run output without changing actual dispatch, and document fleet stop barriers and measurement boundaries.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* feat(observability): qualify gRPC windows and trace shutdown

Reject stale, incomplete or maintenance-active measurement evidence using
producer timestamps and independent process identities. Preserve raw inputs,
bind evidence hashes, and exercise failure paths through the script entry point.

Record HTTP/2 graceful, child-join and TLS reloader retirement phases without
changing shared deadlines, join ownership or unclean-shutdown marker policy.
Document RPC coverage, fixed-window behavior and separate maintenance workloads.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(perf): add authenticated ReadAll and fixed-rate Ping probe

Exercise production remote-disk and signed control clients against task-owned temporary fixtures. Preserve all offered control arrivals, dispatch delay, payload validation and bounded raw request samples, with optional external measurement markers.

Register probe regressions with the default package test discovery and document its telemetry, retry and storage-acceptance boundaries.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(perf): resolve RPC probe disks by explicit UUID

Allow a complete non-nil disk UUID mapping to be installed through the existing remote-disk client API before fixture preflight. This supports test services that register local paths instead of the probe's transport URLs, without modifying server metadata.

Validate mapping shape and assert the actual wire target in the loopback regression. Keep preflight diagnostics limited to the numeric disk-error classification.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* docs(perf): reseed temporary RPC fixtures after restart

Separate startup-cleaned RPC temporary files from confirmed S3 durability fixtures. Revalidate readiness and disk identity before reseeding the probe after recovery.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(server): cancel unfinished TLS handshakes during shutdown

Cancel pre-request TLS acceptance when the listener retires so silent
peers cannot retain graceful watchers beyond the shared drain budget.
Keep established TLS/H2 response draining and original child joins intact.

Add real TCP/TLS shutdown regressions and document fresh native OTLP
observation fences and exec-confirmed diagnostic test units.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(grpc): reuse bulk keys and fix IPv6 TLS verification

Reuse bounded per-peer slot keys and retain the existing cache identities for larger pools. Delegate custom-CA certificate names to tonic's URI parser and cover real IPv6 control/bulk TLS connections with a wrong-name rejection regression.

Classify ReadVersion failure boundaries while preserving error totals, raise the bounded raw probe budget, and align the Linux-only rename test helper with its caller's platform condition.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(grpc): keep eviction race behind cache helpers

Pause eviction on the owned TLS-generation lock and inspect the channel through the common cache API. Assert lock ordering while both mutations are queued, then retain the replacement-generation assertion after they complete.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(grpc): load native trust and preserve mTLS client identity

Select explicit private trust or platform roots, then attach the published client identity for either mode. Build the TLS connector on the blocking pool under the existing acquisition deadline.

Cover real control/bulk TLS connections, required client authentication, private-root scope and queued configuration timeout without late connection establishment.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(grpc): account for TLS setup failures and cancelled dials

Own explicit dial metrics from validated endpoint setup through connection completion. Record TLS preparation failures and acquisition cancellation once while keeping peer-health decisions in the connect-result branches.

Cover malformed client identity, queued configuration cancellation and successful coalesced dials, and document the metric scope.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-13 00:04:41 -05:00
overtrue 9c394af489 refactor(storage): read and install authorization state on both edges
The state an access hook hands to a use case is now read and installed
through the `AuthorizationRequest` trait. Its two implementors are the
s3s request and `RequestEnvelope`, so the legacy hooks and the gateway
edge call the same functions:

- These readers are now generic over `AuthorizationRequest`:
  `apply_bucket_generation_guard`,
  `apply_copy_source_bucket_generation_guard` (both on one shared
  fence), `odm_read_generation`, `prepare_odm_read_generation`,
  `bucket_config_mutation_incarnation`,
  `replication_request_authorized`,
  `retained_table_data_plane_publication_guards`,
  `request_context_from_req`, `resource_free_condition_values`,
  `req_info_ref`, `req_info_mut`, `load_bucket_generation_from_store`
  and `SseKmsPrincipal::from_request`.
- `load_bucket_generation` and `capture_odm_read_generation` now take
  the server context explicitly. The legacy hooks pass
  `FS::server_ctx`.
- New trait accessors: request context; the bucket, pending-delete,
  copy-source and ODM generation guards; the bucket-config mutation
  snapshot.
- New trait installers: the bucket, pending-delete and ODM generation
  guards. The legacy hooks now install those guards through them.
- New: `authorize_object_request(req, bucket, key, version_id,
  action)`, the object twin of `authorize_bucket_request`, with no
  version swap. `versioned_read_action` and `versioned_tagging_action`
  become crate-visible so callers can pick the action first. The
  GetObjectAcl, PutObjectAcl, GetObjectLegalHold and GetObjectRetention
  hooks now call it.
- Removed: `request_context_from_extensions`, which had no caller left.

Each reader keeps the same answers and the same error codes and
messages.

Not built here: the bucket-configuration mutation shape, the
body-input newtypes, and the replication-only, write-ETag and
staged-part checks.

Tests: `storage/access_readers_test.rs` covers each reader through
both forms with the state present and absent (12 refusal
cases, 11 accepted or absent ones), plus the installers on the envelope. The
existing owned-versus-legacy decision table now also runs every case
through `authorize_object_request` on the envelope.
2026-11-13 12:38:19 +08:00
overtrue be2d3ca2cf refactor(storage): add one owned-authorization module per batch
Add `storage/access_owned/` with one empty module for each migration
batch stem: bucket_core, bucket_configs, bucket_access,
bucket_replication_lifecycle, object_meta, listing_multipart,
object_read, object_write, object_commit and object_misc. A batch puts
an authorization shape that only its own operations use into its own
file, so parallel batches never edit the same `mod` list. Shapes that
several batches share stay in `storage/access.rs`.

The directory is mounted as a child module of `access.rs`
(`#[path = "access_owned/mod.rs"]`), so the shapes can use the guard
state that `access.rs` keeps private without widening its visibility.
The gateway edge reaches the modules through
`storage_api::server::http::access::access_owned`, feature-gated in the
same way as `authorize_bucket_request`. The re-export carries
`#[expect(unused_imports)]` until the first batch uses it.

No behaviour change. No tests, since the modules are empty.
2026-11-13 12:38:19 +08:00
overtrue 1ce2e51358 test(server): pin legacy HeadBucket name answers as measured on main
xn--abc is refused by the edge (400) and the SDK sends nothing for a
--x-s3 name, as on main before the use case moved onto the gateway's
types (measured with an embedded-server probe on main). The corrected
test was not run locally: the build host had no disk for it.
2026-11-13 12:12:13 +08:00
overtrue 95c7e4218e build(rustfs): add rustfs-gateway-stream behind gateway-stack
The gateway stack's request-body bridge (batch D: PutObject, UploadPart,
PostObject) reads the gateway's streamed body type, so the rustfs crate
needs rustfs-gateway-stream as a dependency. Added as an optional
dependency enabled only by the gateway-stack feature, without the
tokio-io feature: the bridge does not need it. The workspace entry
already existed (rustfs-protocols depends on it), so the lock gains one
edge and no package.

Not built: the bridge itself; batch D writes it with its first caller.
No tests: a dependency edge has no behaviour of its own.
Verified: check_connect_release_boundary.py --features-only and
--self-test, ci_gate.py --self-test and --check-workflow.
Not compiled locally: the host is short of disk; the crate already
builds in the workspace through rustfs-protocols.
2026-11-13 12:04:29 +08:00
overtrue c43e415b09 feat(server): serve ListBuckets and HeadBucket on the gateway stack (#9296)
The walking skeleton of the gateway edge, behind the gateway-stack cargo
feature and RUSTFS_S3_STACK=gateway. rustfs-gateway inside the RustFS binary
serves ListBuckets and HeadBucket end to end and answers every other S3
operation 501 NotImplemented. A default build carries neither the stack
selection nor the wrapper: its request path is unchanged.

- One per-operation unit: impl Served for dto::Op with a metric label, an
  authorize without a default body, and an execute. The only impl Handler is
  the blanket one over Served (access phase, authorize, count, execute, one
  error mapping). authorize calls the function the legacy access hook calls.
- One table laid out per family (ten family files named after the migration
  batches); it registers the handlers and is the whole early gate.
- A shared access phase used by both edges, fed on the gateway edge from the
  gateway's verdict.
- Stored bucket names are materialised under the gateway floor only, so names
  the AWS grammar refuses are listed as before; HeadBucket's legacy edge builds
  its input the same way until the gateway re-pin (rustfs/gateway#1610).
- With the gateway selected the legacy service receives exactly what the
  legacy router claims (extension routes, object lambda, control plane).
- A start-up line names the stack the mounted service answers with.
- The release-boundary guard reads feature closures from cargo metadata.

Squashed from gw/T2.5-gateway-stack at aca434fdf onto ce9d0b153, with two
changes made while landing: the new CI job reads the ci-dev cache under its
current digest key, and the cache-reader count in scripts/test_ci_plan.py is
7 (this job is the seventh reader).

Checked: cargo clippy --locked -p rustfs --all-targets -- -D warnings in both
builds on aca434fdf (main up to eca95f297); on the squashed tree the Quick
Checks guards, the spell check, ci_gate.py --self-test and --check-workflow,
and the release-boundary tests. Not run on the squashed tree: a compile of
the rustfs crate (six small commits landed on main since the clippy run) and
any test. The legacy HeadBucket name table carries four provisional statuses
for names no store holds; they are corrected by a follow-up when the probe
of main reports.
2026-11-13 11:31:50 +08:00
cxymds ce9d0b153e fix(heal): retain legacy MRF responsibility after physical repair (#9391) 2026-11-13 11:03:53 +08:00
overtrue ef79dcbe19 ci: conformance and client-matrix lanes against both S3 stacks
Acceptance lanes T3.4 (rustfs/backlog#2775) and T3.5 (rustfs/backlog#2776):
.github/workflows/gateway-acceptance.yml runs the rustfs/gateway conformance
corpus (--profile rustfs) and client matrix (ci/compat/run_matrix.sh
--external) against the candidate binary on RUSTFS_S3_STACK=legacy and
=gateway, one server per stack. The suites are checked out at the gateway rev
Cargo.toml pins. Nightly schedule and workflow_dispatch only.

- Each stack must be observed: the start step writes `ready` only when
  /health/ready answers while its own process is alive and the server's
  server_starting event names the requested stack (scripts/dual_stack.py
  stack); otherwise `did not start: ...`. Summaries print the observed stack.
- Verdicts come from files the run leaves on disk (startup record, stack
  evidence, liveness after the corpus, exit codes, report.json, matrix.json),
  never from an earlier step's exit status.
- Conformance: the rustfs profile describes the gateway edge, so only the
  gateway stack is judged (scripts/check_conformance_rulings.py against
  .config/conformance-rulings.toml). Red: an unruled failure, a ruled case
  failing differently from its ruling's `failure`, a stale ruling, and a skip
  on the server's own answer (endpoint not answering, refused fixture). The
  legacy run is an unjudged reference: it must serve, stay up and complete.
- Client matrix (scripts/compat_matrix_report.py): per stack, red on a `fail`
  or `sut-unregistered` cell, a client that could not be installed, or an
  unfinished matrix; legacy is the baseline and every legacy pass must pass on
  gateway.
- The gateway leg and the stack comparison are continue-on-error, report-only
  until T2.10.

Not built: no re-pin (write cases need a gateway rev with
--allow-external-writes plus that flag on the lane's run line); no legacy
rulings; no use of the runner's own --rulings ledger (stale rulings do not
block there); no call to the gateway's check_compat_matrix.sh (it does not
fail `fail` cells or clients that never ran); no JUnit (the pinned runner
writes a raw U+0001); no install cache; toolchain versions copied from the
gateway's client-matrix.yml at 7915d069.

Tests: --self-test of both checkers in make script-tests, 40 tests, 31
negative. Mutations, each red: unruled failure ignored
(test_an_unruled_failure_is_red); stale only when passed
(test_a_ruling_on_a_skipped_or_absent_case_is_stale); did-not-start green
(test_a_server_that_did_not_start_is_red_and_says_why); swallowed runner exit
(test_a_swallowed_exit_cannot_hide_a_failed_case); exit 3 accepted
(test_an_environment_exit_is_red); legacy judged or tallied
(test_the_legacy_reference_is_counted_never_judged); dead server green
(test_a_server_that_died_during_the_corpus_is_red); endpoint and fixture
skips not red (test_a_skip_on_the_endpoint_not_answering_is_red,
test_a_skip_on_a_refused_fixture_is_red); ruling on id only
(test_a_ruled_case_failing_differently_is_red); unknown verdict
(test_an_unknown_verdict_is_refused); stack not compared
(test_a_server_of_the_other_stack_is_red); no stack evidence
(test_ready_without_stack_evidence_is_red); failed cell, skipped client,
swallowed matrix exit (test_a_failed_cell_is_red,
test_a_client_that_could_not_be_installed_is_skipped_never_passed,
test_a_missing_manifest_is_red_even_when_the_exit_was_swallowed); superset
gate (test_a_legacy_pass_lost_or_absent_on_gateway_is_red,
test_a_gateway_superset_of_the_baseline_is_clean_and_its_extra_passes_are_listed,
test_diff_without_both_manifests_is_red); diffs
(test_diff_lists_changed_and_absent_cases_only,
test_diff_without_both_reports_compares_nothing).

Trial runs on rustfs/rustfs#9368: 43264289490, 43325101234. Gateway stack did
not start (gateway stack is not assembled yet, T2.5); legacy reference
completed, 140 of 890 cases executable against an external endpoint; legacy
matrix 13 x 14: pass 121, fail 0, client-unsupported 61, sut-unregistered 0.

Not verified: the stack observation and liveness record in CI (exercised
locally under bash -eo pipefail with a stand-in server and main's
dual_stack.py, both directions); a scheduled run; a diff with gateway content.
2026-11-13 10:31:46 +08:00
overtrue dea9ac7fe2 refactor(admin): read target subscriptions from the stored configuration
ListTargetSubscriptions rendered each bucket's stored notification
configuration as the s3s DTO (an XML round trip) and walked it as JSON.
It now reads the routes of the stored PersistedNotificationConfiguration
directly through its behavior() projection: same order (queue, topic,
lambda), same exact destination match, same filter-rule reading
("Prefix" before "prefix", a rule without a value supplies none). The
conversion step and its two internal-error exits are gone; the function no
longer fails. admin/handlers/event.rs no longer references s3s.

Tests: the existing subscription test now starts from stored XML, the way
the handler loads it; a new test pins rule-name precedence, valueless and
nameless rules, and exact (case- and whitespace-sensitive) destination
matching. Both were run against the old code first and passed unchanged,
then against the new code. Mutation: not run yet (the build was interrupted); it follows separately.

Not verified: the workspace suite (only the touched modules' tests ran).
2026-11-13 10:27:53 +08:00
overtrue ca5ac98c6c refactor(rustfs): use owned error and header names in three files
Part of the s3s removal tail (rustfs/backlog TAIL, feeding T4.2):

- storage/options.rs: the empty Object Lock value check compares against
  the lowercase rustfs_utils names the file already imports instead of the
  s3s header constants; the comparison stays case-insensitive.
- admin/handlers/pools.rs tests: compare ApiError codes through the owned
  code (the idiom the file already uses) and read headers and status from
  the owned S3Error conversion, from which the s3s one is derived.
- init.rs: the error logged when a bucket's notification rules cannot be
  registered is built with rustfs_s3_types::s3_error!; code and message are
  unchanged, the Debug spelling in that log line is the owned type's.

Not changed: init.rs still builds s3s notification DTOs, because the notify
rule conversion it calls takes them.

Tests: empty_object_lock_values_are_not_user_metadata (4 skipped keys, 1
kept), run against the old code first and passed. Mutation: not run yet (the build was interrupted); it follows separately.

Not verified: the workspace suite (only the touched modules' tests ran).
2026-11-13 10:27:53 +08:00
overtrue 34be0a802f fix(ci): satisfy the unsafe-allowance and spelling checks on main
Two quick checks failed on main after eca95f297:

- scripts/check_unsafe_code_allowances.sh requires a SAFETY: comment next
  to an unsafe_code allowance. The comment above the allowance in
  crates/ecstore/src/erasure/codec/shard_buf.rs already stated the
  contract; it now carries the marker the guard looks for.
- The spell checker reads the byte literal b"hel" in a put.rs test as a
  misspelling. The two frames are now b"he" and b"llo"; the test still
  joins them into "hello".

No behaviour change. Checked locally: the unsafe-allowance guard, typos
over the whole tree, cargo fmt --check. Not run: clippy or tests (a
comment and a test literal).
2026-11-13 10:27:10 +08:00
Hauser Zhi 7e93d41c5d test(ecstore): reserve capacity for namespace ownership probes (#9382) 2026-11-12 21:14:42 -05:00
Rain Jiang eca95f2973 perf(ecstore): erasure read path for large GETs (#9304)
Lands the contributor's pull requests #9302 (opt-in lockless KV cache
buckets), #9303 (per-bucket XXH3 ETags; MD5 off the request task) and
#9304 (erasure read path for large GETs) as one change, because #9304
was stacked on the other two.

Maintainer changes on top of the contributed work:
- Merged main. crates/ecstore/src/erasure/coding/bitrot.rs: kept the
  PR's verify_chunked_frame/append_chunked_data restructuring and routed
  its hash-mismatch log through main's throttled log_hash_mismatch.
- rustfs/src/app/multipart_usecase.rs: took main's per-operation split
  and re-applied the PR's changes in place: multipart_etag_algorithm and
  its test in the parent module, the XXH3 pin in create.rs, the stage
  timer and from_stream_with_etag in upload_part.rs.
- Reworded one doc comment in crates/config/src/constants/object.rs that
  spelled an ecstore import path, which the architecture migration guard
  counts as a direct import.
- fix(config): made the MD5 offload opt-in. Each offloaded upload holds
  one blocking-pool thread until its body ends, and that pool also
  serves disk I/O, so many concurrent large uploads could starve disk
  writes. The default threshold is now 0 (disabled); operators enable
  it with RUSTFS_OBJECT_ETAG_MD5_OFFLOAD_MIN_BYTES (for example 8388608).
2026-11-13 10:05:26 +08:00
hector 5a26862e2e ci: build macOS and Windows on a weekly schedule (#9286)
* ci: build macOS and Windows on a weekly schedule

Keep the 01:13 UTC cron on Linux and add a Sunday 16:00 UTC cron
for macOS and Windows. Select that subset from the firing cron so a
delayed runner cannot guess the weekday.

Co-authored-by: hector <majinghe@users.noreply.github.com>

* ci: move the weekly desktop cron off minute zero

---------

Co-authored-by: hector <majinghe@users.noreply.github.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-11-12 20:51:08 -05:00
hector b8efcb8dfa feat(release): package manual Linux builds from a git tag (#9332)
* feat(release): package manual Linux builds from a git tag

Let operators name an existing tag, branch, or commit. An empty ref
still packages the tip of origin/main, and a tag can supply the
package version by dropping one leading v.

Co-authored-by: hector <majinghe@users.noreply.github.com>

* fix(release): confirm historical tags with the tooling helper

The confirm step was calling the script from the tagged commit, which
exits 127 when that commit predates the script and cannot confirm when
it only has an older copy. Use a helper that implements confirm-source.

Co-authored-by: hector <majinghe@users.noreply.github.com>

---------

Co-authored-by: hector <majinghe@users.noreply.github.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-11-12 20:51:00 -05:00
overtrue 704b984c1f ci: both-stack e2e diff in the source workflows and a strict mint stack gate
T3.1/T3.3 (rustfs/backlog#2772, #2774). The existing nightly lanes get a stack
dimension and their legacy leg is the legacy side: e2e-replication-nightly (repl,
cluster, protocols), e2e-distributed and the ODM minio-source job. A gateway leg
runs scripts/dual_stack.py preflight after its build and stops there when no gateway
start is proven; the summary says 'skipped: gateway stack refuses startup: <why>'.
One summary job per workflow diffs the legs of the same run with junit_diff.py.
dual-stack-nightly.yml covers only e2e-smoke and e2e-full: one build, both stacks
from that binary, gateway legs only when the preflight proved a start.
junit_diff.py reports a missing, empty or case-less side as 'no result for <label>'.
mint.yml stays weekly, builds the image once and runs it per stack; mint_stack_diff.py
gates that every case legacy passes is present and passing on gateway with no new
failure (missing, NA, unknown status and truncated lines never pass). Gateway legs and
the compare job are report-only until T2.10.

Not built: a shared-binary preflight job in the three source workflows (one preflight
step per gateway leg); harness-only filtering for their five profiles; mc/Vault/iptables
in the e2e-full leg; an s3s-e2e both-stack run; rustfs_gateway=debug.
Tests: dual_stack 30 (preflight with fake binaries), mint_stack_diff 16, junit_diff 27;
negatives outnumber positives. Mutations (red test named) are in the PR 9367 description.
Not verified: the workflows have not run in CI in this shape (composites, stack matrices,
image artifact round trip, needs result of a red continue-on-error leg).
2026-11-13 09:21:25 +08:00
Chris An 598282bc63 refactor(admin): move replication and listen routes to owned requests (#9373)
The replication and listen extension routes take owned requests instead of s3s requests: handlers under admin/handlers/s3_extension, one authorization entry per route, the listen subscriber holds a body-less request envelope. Route-by-caller tests cover the seven routes (served and refused), including the empty events value.
2026-11-12 20:20:50 -05:00
Hauser Zhi 45156f313e perf(ecstore): reduce default fsync worker idle spin budget (#9386)
Use 256 idle iterations by default and preserve the configurable 1024
iteration ceiling. Update fallback regression cases and tuning guidance.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-12 20:17:38 -05:00
Joe C a6cd90b6fb feat(s3): accept a comma-separated region list in RUSTFS_REGION (#9338)
* feat(s3): accept a comma-separated region list in RUSTFS_REGION

Since #8739 CreateBucket rejects any LocationConstraint other than the
server's region. A single deployment that serves clients configured for
several regions can no longer create buckets for all of them.

RUSTFS_REGION / --region now accepts a comma-separated list. The first
entry stays the server's own region, so request signing and
GetBucketLocation are unchanged; every entry is accepted as a
CreateBucket LocationConstraint, and anything else is still rejected
with InvalidLocationConstraint.

The list is parsed once at startup and the extra regions are stored
write-once on the per-instance InstanceContext next to the region,
reaching the use-case through RegionInterface (default: none).

A comma was never a valid region (s3s accepts [a-z0-9-] only), so every
existing value keeps its meaning.

* test(e2e): pin e2e-smoke selection for the listed-regions test

---------

Co-authored-by: Chris An <anzhengchao@gmail.com>
2026-11-12 20:17:30 -05:00
Ramakrishna Chilaka 248c56ec1f perf(ecstore): use pinned directory FDs for rename_part and delete_recovery (#9375)
- Reuse pinned txn_parent_fd and txn_dir_fd in LocalDisk::rename_part to perform FD-relative renameat and a single parent directory fsync for part data and publish.meta, avoiding two full-path prepare_rename_with_retry walks per drive (32 openat2 + 32 fstat + 32 close syscalls per UploadPart/UploadPartCopy across 8 drives).
- Probe open_part_dir_fd on transaction_parent before calling create_dir_all in prepare_part_transaction so Part 2..N uploads reuse the existing <upload_id>/<data_dir> directory with a single openat2.
- Remove the redundant file.metadata() (fstat) pre-check in read_child_regular_bytes and delegate directly to File::read_to_end, which already performs its own single fstat and buffer reservation.
- Use FD-relative openat, renameat, unlinkat, and unlinkat(AT_REMOVEDIR) via file_path_dir for <object>/<old_data_dir> staging and finalization in delete_recovery, and drop file_path_dir at the end of Stage 1 for inline unversioned deletes when no old_data_dir or surviving target exists.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 21d8304f-0f49-48eb-8c28-26f3689b081b
2026-11-12 20:17:22 -05:00
overtrue cb75405d8c ci: log the selected S3 stack at start-up and prove it from server logs
The both-stack lanes need the stack a server actually selected, not the variable
a workflow set. The start-up line gains the s3_stack field (below the stack match,
so a refused gateway request never logs it), and scripts/dual_stack.py stack reads
that line from server logs: it fails on a server of the other stack, on no server of
the requested stack, and on no server output; older-release servers count as unstamped.

Not built: the preflight and summary subcommands, the workflows and the mint gate.
Tests: 10 in scripts/test_dual_stack.py (7 negative, 3 positive), in make script-tests.
Mutation: ignoring foreign-stack starts turns test_a_server_of_the_other_stack_fails_the_proof
and test_the_other_direction_fails_too red.
Not verified: clippy with --features gateway-stack; rustfs nextest.
2026-11-13 09:08:23 +08:00
Chris An a09b0d2d2c docs(filemeta): stop spelling s3s paths in comments (#9383)
Four doc comments in crates/filemeta spelled s3s paths although the crate has no s3s dependency; reworded as prose so the s3s footprint counts 60 files instead of 63. No code change.
2026-11-12 19:40:34 -05:00
Chris An 1d47ad5b61 ci: let a push pass when the s3s footprint is below its baseline (#9387)
Without --base, a footprint counter below its baseline line passes and prints the lowered line; a counter above it still fails. Pull requests and merge groups are still held to their base revision, so the footprint cannot rise.
2026-11-12 19:40:26 -05:00
Chris An 8632458d92 refactor(storage): build audit entries from the owned request envelope (#9374)
One implementation of the audit and notification helper, built from the owned request envelope and completed with an owned result; the s3s constructors become callers of it. Legacy behaviour unchanged.
2026-11-12 19:37:00 -05:00
Chris An 10103c824f ci: trim the merge-group proof and keep a proven plan honest (#9381)
Cleanups from the review of the merge-group proof change: the proof lookup no longer special-cases the event in the workflow, a proven plan carries connect=false and full_validation=false and the plan validator requires it, two redundant input guards are removed, plan contract 7.
2026-11-12 19:36:52 -05:00
Chris An c2cd9c2d22 ci(e2e): refresh the protocols selection digest (#9384)
The e2e-protocols selection digest was stale since a WebDAV test was added without regenerating it; the nightly lane failed on the selection check. Regenerated with the repository's own tool. The linux digest equals the one the failing CI run reported.
2026-11-12 19:36:44 -05:00
Chris An 02f77faaf5 feat(gateway-adapter): Keystone header-token verifier (#9358)
The RustFS side of the gateway's host-verified header token claim for OpenStack Keystone, in crates/gateway-adapter, plus the validation proof it is built from in crates/keystone. Nothing in the server calls it yet; the wiring is a separate change.

- rustfs-keystone returns a ValidatedKeystoneToken proof with the legacy credentials; only that crate can build one.
- keystone_header_token_claim(): x-auth-token, refusing x-storage-token.
- KeystoneVerifier answers keystone:<user_id> from the established value and the request clock.
- The SigV4 credential provider and the policy source refuse keystone: principals.
2026-11-12 19:35:41 -05:00
Chris An 9e3d4cbcbb chore(app): localize use-case imports and add slice seams (#9372)
Preparation for the parallel operation slices of the s3s to gateway migration. No behaviour change.

- Use-case modules import the s3s names they use instead of reaching them through a glob from the parent.
- Per-slice test retirement tables and scripts/check_test_retirements.py.
- Per-slice extension points in the dual-stack differential script.
- The s3s footprint check compares a pull request with its base revision; the baseline file is a ceiling there.
2026-11-12 19:21:41 -05:00
Chris An b42f444166 ci: stop rebuilding what the cache already built (#9371)
* ci: stop rebuilding what the cache already built

cache-warm.yml built `--workspace --all-targets`, a selection no ci-dev
reader builds. Feature resolution differs per selection, so the test
archive recompiled 123 external units and the E2E archive about 290 on
every exact cache hit. warm-ci-dev now runs the readers' own builds:
`ci_test_artifact.py cargo warm` (the Cargo build of `nextest archive`)
for both archive configurations, the hooks server and the lint command.
Rust-cache never saves over an exact key match, so the warm jobs skip
their builds after one; ci_gate.py --check-workflow pins both rules.

The manual shard rehearsal ran only after a full execution, saved
nothing, and could not run since #9071 renamed the archive it downloads.
It is removed with its plan field, jobs, script and tests; the plan
contract moves to 5.

* ci: key ci-dev by its warm commands and close the warm guard's gaps

The ci-dev key now ends in a digest of warm-ci-dev's build steps, so the
reader selections this branch warms are written under a new key instead of
being skipped as an exact hit of the old one. ci_gate.py --check-workflow
derives the digest, requires the writer and every reader in ci.yml and
reference-parity.yml to name that key, and prints the key a change needs.

The presence check now reads only the steps that run whenever the key is
written, so a selection moved into the on-request --timings step no longer
counts. Configuration names are compared as sets in both directions and
accept either option spelling.

The plan contract moves to 6: #9369 already used 5 for a different plan.
2026-11-12 18:29:34 -05:00
Hauser Zhi 307fb13d32 chore(deps): upgrade object, dav-server, and zstd (#9364)
Pin object to 0.41.0 and upgrade dav-server to 0.12.0.
Align the zstd adapter with the 0.14.4 registry implementation and
update its dependency graph assertion and documentation.

Implement the required WebDAV symlink metadata methods and cover
regular entries and backend authorization with regression tests.
Run two complete cargo update and cargo upgrade rounds after
resolving the initial zstd version conflict.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-12 18:28:18 -05:00
Hauser Zhi 70525a9698 feat(ecstore): expose fsync worker policy experiments (#9357)
* feat(ecstore): expose fsync worker policy experiments

Add independent idle-spin controls and gated worker diagnostics for controlled performance comparisons.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(perf): forward Docker telemetry in ABBA runs

Validate node mappings and record per-leg container resources for isolated fsync policy experiments.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(perf): reject baseline drift in either direction

Apply absolute delta budgets to A2 baseline checks while preserving directional candidate checks and their exemptions.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(perf): stop ABBA after a load driver failure

Preserve the load driver exit status through command substitution and stop before later deployment legs while retaining cleanup traps.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-12 18:28:01 -05:00
Rain Jiang 2a0a9819d0 fix(lock): allow more than 255 concurrent shared lock holders (#9301)
The fast-lock state packed the reader count into 8 bits, so the 256th concurrent shared holder of one key was refused. Bucket-scoped fences are read-locked by every in-flight request into the bucket, so busy buckets hit the cap. The count now has 16 bits.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
2026-11-12 18:27:45 -05:00
Chris An 0f904a170e fix(clippy): satisfy lints introduced in Rust 1.100 (#9377)
* fix(clippy): satisfy lints introduced in Rust 1.100

* fix(clippy): cover Linux-only and feature-gated Rust 1.100 lints

Restore the StorageError clone grouping check in the migration test,
assert collection lengths before the rewritten test loops, and fix the
needless_bool in the Linux-only heal readiness test and the
redundant_clone in the swift formpost path.
2026-11-12 17:57:00 -05:00
Chris An 6b27fa4756 ci: accept the proving merge-group run as release evidence (#9376)
* ci: accept the proving merge-group run as release evidence

A commit that lands through the merge queue gets a push run of ci.yml in
mode proven: Quick Checks and Test and Lint succeed, and every validation
job, End-to-End Tests (full merge gate) included, is skipped. source_ci
refused such a commit, so no release candidate could be cut from it.

source_ci now accepts that push run when Quick Checks and Test and Lint
succeeded and a successful merge_group run of ci.yml on that branch's
queue has the same head tree and passed all three required jobs itself;
that run is recorded as the candidate's exact-source CI identity. The
match is ci_merge_group_proof.proves, the same predicate the push plan
uses.

Also: a test that a full merge_group run requires every job a full
integration push requires; the sibling queue-branch case
(gh-readonly-queue/mainline/...); select_plan's proof is keyword-only;
the release, CI gate and workflow docs describe proven pushes.

* test(release): pin the proving run's position, attempt and branch

The release test now lists the matching merge-group run among a run of
another tree and a run of another branch's queue, in both orders; gives
it attempt 3 and reads jobs by run and attempt; and accepts a release
branch's own queue run while refusing a main queue run for it.
2026-11-12 16:14:04 -05:00
GatewayJ b753f69300 fix(sts): accept Id in inline session policies (#9253)
* fix(sts): accept Id in inline session policies

* test(sts): register the session policy regression in e2e profiles

---------

Co-authored-by: Hauser Zhi <housemecn@gmail.com>
2026-11-13 04:02:38 +08:00
Hauser Zhi 3d2664152f test(rpc): verify TLS peer identity across DNS and IP channels (#9360)
* fix(rpc): derive TLS peer identity from the parsed endpoint

Use Tonic URI hostname handling for custom-root TLS connections. Manual colon splitting turns an IPv6 literal into an invalid DNS name and prevents internode channels from connecting. Preserve CA and mTLS identity configuration and cover DNS, IPv4, IPv6 and mapped addresses through the production TLS configuration boundary.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(rpc): reconcile TLS merge with the main channel builder

Restore the upstream TLS configuration path after the conflicting merge removed its tls binding. Expand the existing real-handshake regression to DNS, IPv4 and IPv6 control/bulk channels, require an RPC response for positive cases, and verify restoration of TLS material and generation.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Signed-off-by: Hauser Zhi <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-13 04:01:12 +08:00
Chris An 4a34e512cd ci: skip the push run for a tree the merge queue already proved (#9369)
A merge-queue landing fast-forwards main to the group's commit, so the
push run re-validates a tree a successful merge_group run of ci.yml has
just passed. Select CI scope now looks that run up with the job's
read-only Actions token (scripts/ci_merge_group_proof.py) and plans
mode `proven`: Select CI scope, Typos and Quick Checks run, every
validation job is skipped, and Test and Lint requires exactly that.
The plan names the proving run, its not-applicable reasons and the
aggregate's summary say the jobs were skipped as proven.

Any doubt runs everything: no token, an API or network error, a rate
limit, an unexpected payload, no successful queue run of the same tree,
or a proof on another event. A direct push has no such run. The plan
contract moves to 5.
2026-11-13 03:58:53 +08:00
zjuncal 78325ce491 fix(s3-client): retry transient remote S3 requests (#9361)
Signed-off-by: zjuncal <naclzju@outlook.com>
2026-11-12 13:10:28 -05:00
Chris An 755cca3336 test(audit): assert event-name expansion by count, not wall clock (#9363) 2026-11-12 13:08:46 -05:00
Ramakrishna Chilaka 24b667e738 perf(ecstore): use pinned directory FDs for delete and part txns (#9366)
- Stream canonical version identity JSON directly into Sha256 via a stack-allocated Sha256Writer in delete_recovery::generation(), avoiding per-version Vec<u8> allocations during Stage 1 preparation.
- Open and cache pinned directory FDs (O_RDONLY | O_DIRECTORY | O_CLOEXEC) for the object directory and rollback transaction directory across fused recoverable DELETE Stage 1/2/3, replacing repeated full-path lookups with FD-relative openat, statat, linkat, renameat, unlinkat, fsync, and unlinkat(AT_REMOVEDIR).
- Open and cache pinned directory FDs for the part transaction parent and staging directories across prepare_part_transaction, rename_part, and settle_part_transaction, replacing repeated path walks with FD-relative statat, linkat, openat, renameat, fsync, and unlinkat.
- Purge cached delete and part transaction receipts under both io_root and root in LocalDisk::close and Drop for LocalDisk, and insert receipts in the async callers after .await so cancelled futures cannot insert FDs after disk teardown.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 21d8304f-0f49-48eb-8c28-26f3689b081b
2026-11-12 13:06:29 -05:00
Hauser Zhi 5f460c2383 perf(grpc): optimize peer pools and unary RPC observability (#9355)
* perf(protos): balance peer channels and synchronize TLS eviction

Use bounded per-peer bulk cursors and shared TLS generation reads. Keep eviction and generation removal coherent with concurrent publication, and expose opt-in bulk transport tuning with checked capacities.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* feat(metrics): observe internode unary request lifetimes

Record client attempts through errors and cancellation, preserve the published server-duration family, and add gated activity and completion metrics. Cache enabled instruments after recorder publication while retaining startup duration recording.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(perf): make internode gRPC experiments reproducible

Reset phase-owned settings before applying overrides, separate lifecycle experiments, and provide a bounded loopback workload with independently reported control and bulk latency.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(perf): isolate transport knobs and redact dry runs

Add single-parameter P0 presets with separate artifacts while preserving the legacy combined default. Mask credential arguments in dry-run output without changing actual dispatch, and document fleet stop barriers and measurement boundaries.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* feat(observability): qualify gRPC windows and trace shutdown

Reject stale, incomplete or maintenance-active measurement evidence using
producer timestamps and independent process identities. Preserve raw inputs,
bind evidence hashes, and exercise failure paths through the script entry point.

Record HTTP/2 graceful, child-join and TLS reloader retirement phases without
changing shared deadlines, join ownership or unclean-shutdown marker policy.
Document RPC coverage, fixed-window behavior and separate maintenance workloads.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(perf): add authenticated ReadAll and fixed-rate Ping probe

Exercise production remote-disk and signed control clients against task-owned temporary fixtures. Preserve all offered control arrivals, dispatch delay, payload validation and bounded raw request samples, with optional external measurement markers.

Register probe regressions with the default package test discovery and document its telemetry, retry and storage-acceptance boundaries.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(perf): resolve RPC probe disks by explicit UUID

Allow a complete non-nil disk UUID mapping to be installed through the existing remote-disk client API before fixture preflight. This supports test services that register local paths instead of the probe's transport URLs, without modifying server metadata.

Validate mapping shape and assert the actual wire target in the loopback regression. Keep preflight diagnostics limited to the numeric disk-error classification.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* docs(perf): reseed temporary RPC fixtures after restart

Separate startup-cleaned RPC temporary files from confirmed S3 durability fixtures. Revalidate readiness and disk identity before reseeding the probe after recovery.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(server): cancel unfinished TLS handshakes during shutdown

Cancel pre-request TLS acceptance when the listener retires so silent
peers cannot retain graceful watchers beyond the shared drain budget.
Keep established TLS/H2 response draining and original child joins intact.

Add real TCP/TLS shutdown regressions and document fresh native OTLP
observation fences and exec-confirmed diagnostic test units.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(grpc): reuse bulk keys and fix IPv6 TLS verification

Reuse bounded per-peer slot keys and retain the existing cache identities for larger pools. Delegate custom-CA certificate names to tonic's URI parser and cover real IPv6 control/bulk TLS connections with a wrong-name rejection regression.

Classify ReadVersion failure boundaries while preserving error totals, raise the bounded raw probe budget, and align the Linux-only rename test helper with its caller's platform condition.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(grpc): keep eviction race behind cache helpers

Pause eviction on the owned TLS-generation lock and inspect the channel through the common cache API. Assert lock ordering while both mutations are queued, then retain the replacement-generation assertion after they complete.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(grpc): load native trust and preserve mTLS client identity

Select explicit private trust or platform roots, then attach the published client identity for either mode. Build the TLS connector on the blocking pool under the existing acquisition deadline.

Cover real control/bulk TLS connections, required client authentication, private-root scope and queued configuration timeout without late connection establishment.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-12 11:21:40 -05:00
Hauser Zhi 078fc47e9c perf(lock): avoid scanning live leases on every acquisition (#9356)
* fix(lock): allow more than 255 concurrent shared lock holders

The fast-lock state packed the reader count into 8 bits, so the 256th concurrent shared holder of one key was refused. Bucket-scoped fences are read-locked by every in-flight request into the bucket, so busy buckets hit the cap. The count now has 16 bits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(lock): cover shared-reader capacity through namespace quorum

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(lock): skip lease scans until a shard deadline is due

Keep a conservative earliest-expiry hint with the guard table. Healthy acquisitions and reaper passes avoid walking live leases, while due sweeps recompute the hint and release expired guards outside the registry lock. Preserve resource-specific retry counts and cover refresh, replacement, removal and cross-resource expiration.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: Rain Jiang <rain-jiang@outlook.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-12 10:21:21 -05:00
Chris An 55d9bf08a2 ci: run workspace clippy beside the test binary build (#9345)
Clippy ran in front of the test-binary archive inside Workspace Lint and Build, so every workspace test waited 13 to 17 minutes for a lint that shares no compiler output with the archive.

Move the unchanged Clippy step into its own workspace-clippy job with the same condition, needs, runner and cache key. Workspace Lint and Build keeps its ID and name as the archive producer. The aggregate and the scheduled alert depend on the new job, and the plan contract moves to 4 so a base planner that predates the job set falls back to the full matrix.

ci_gate.py now rejects a workflow where the lint is conditional, duplicated, outside the aggregate, scoped differently from the archive producer, or back on the archive consumers' dependency chain, and where the workflow's plan-contract marker differs from the planner's.
2026-11-12 10:09:05 -05:00
Chris An cf3e359794 refactor(storage): split impl S3 for FS bodies into legacy_edge files (#9319)
* feat(server): add the RUSTFS_S3_STACK selector and dual-stack e2e lane (#8411)

* refactor(server): group the s3s-compat tower layers under legacy_compat (#8410)

Move the ten tower layers that patch s3s behaviour out of
rustfs/src/server/layer.rs into rustfs/src/server/legacy_compat/, one file
per layer with its tests. Each file header names the s3s behaviour it
patches, the gateway mechanism that replaces it (Replaced by:), and the task
that removes it (Removed by: T2.12, rustfs/backlog#2771). The layers keep
their code, their order in external_service_stack!, and their behaviour; no
layer is deleted.

Add server::http::tests::stack_census, which renders the ordered service
chain of the real external_service_stack! expansion and compares it with a
committed constant, so a reorder, insertion or removal is a visible diff.

Route the two s3s error types the moved layers use through a
storage_api::server::legacy_compat shim so the split adds no s3s-importing
file, and lower the s3s footprint baselines to the measured values (files 209
to 208, s3_error! lines 1586 to 1585). Keep the server runtime-source bypass
guard covering the moved code.

Refs rustfs/backlog#2787, rustfs/backlog#2734

* feat(s3-types): add the RustFS-owned S3 error type and s3_error! macro (#8406)

`rustfs-s3-types` now provides `S3ErrorCode` (generated by build.rs from
src/codes.txt, the grep-derived call surface, plus `Custom`), `S3Error`,
`S3Result`, `StdError` and the `s3_error` macro with the same call syntax
as s3s, so the T1.2 codemod is a `use` swap. Statuses are hand-written and
pinned to the s3s behaviour RustFS ships today by an oracle test that uses
s3s as a dev-dependency only. The `From` conversions for the legacy edge
live behind the off-by-default `compat-s3s` feature (the orphan rule
forbids them in rustfs/src/error.rs); `rustfs` enables it.

Refs rustfs/backlog#2742, rustfs/backlog#2734

* ci: add the per-crate test-count ratchet (#8404)

Quick Checks now fails any PR that lowers the number of test attributes
in a workspace crate or adds #[ignore], measured per crate against the
exact rows of .config/test-count-baseline.txt (rustfs/backlog#2734
section 1 rule 3). The counter widens the spec's literal
#[(tokio::)?test] to the tokio attributes that carry arguments and both
spellings of #[test_case(...)], and anchors at line start so
commented-out attributes do not count. The initial baseline covers 52
crates, 21077 tests and 164 ignored.

Refs rustfs/backlog#2738
Refs rustfs/backlog#2734

* test(ecstore): pin the persisted bucket-config XML bytes for 13 families (#8408)

Before any DTO or codec is replaced, pin the exact bytes that
deserialize followed by serialize produces for every persisted
bucket-configuration XML family in its three source shapes (AWS
documentation, MinIO-written, RustFS-written), and the bytes RustFS
writes today for a typed value. Byte equality only: a structural
comparison would let a codec replacement change the persisted encoding
unnoticed.

The generator script refuses to overwrite existing goldens without
--force, refuses on dirty codec paths, cannot report success from its
write mode, and records the generating commit in the fixtures README.

Refs rustfs/backlog#2744, rustfs/backlog#2734

* ci: raise the test-count baseline to the integration tree (#8426)

* ci: make the s3s footprint baseline exact and lower-only (#8403)

* ci: make the s3s footprint baseline exact and lower-only

The s3s footprint guard compared three counters against ceilings held in
the script itself, so a ceiling could sit above the tree and a drift was
fixable by raising it (rustfs/rustfs#5964 raised the s3_error! ceiling
to go green). The three values now live in
.config/s3s-footprint-baseline.txt as files, s3_error_lines and
ecstore_files, the script reads only that file, and a measurement that
differs from its baseline in either direction fails: growth prints
"s3s footprint grew: <measured> > <baseline>", shrinkage prints
"lower the baseline to <measured> in this PR". The counting method is
unchanged; the baseline is the current measurement of
gateway/integration (209 / 1585 / 36). The header records the end state:
once all three reach 0, T4.2 switches the script to failing on any
occurrence.

Refs rustfs/backlog#2739, rustfs/backlog#2734

* ci: add the s3s edge allowlist mode to the footprint guard (#8405)

`scripts/check_s3s_footprint.sh --mode allowlist` fails on every Rust
file that references s3s (`s3s::` or `s3s_sigv4`) and is not matched by
an entry of the new `.config/s3s-edge-allowlist.txt`; `--dry-run` lists
those files and exits 0. The allowlist carries the initial edge from the
spec, and the script rejects an entry that names or matches anything
under crates/ other than crates/e2e_test. The baseline mode and the
argument-less CI invocation are unchanged.

Refs rustfs/backlog#2741, rustfs/backlog#2734

* test(ecstore): reach the bucket-config codec through the test facade (#8428)

* refactor(s3-types): own the HTTP Body and drop s3s from three crates (#8431)

Add rustfs_s3_types::Body, a BoxBody<Bytes, StdError> wrapper with
empty() and From<Bytes | Vec<u8> | String | &'static str>, and the
legacy From conversions to and from s3s::Body behind compat-s3s.

The signer is now generic over the request body: it only reads and
writes the request head, so it needs no body type and its callers keep
theirs. s3-client sends its requests with the owned Body, madmin signs a
bodiless head, and targets drops its two S3Request/S3Response header
wrappers in favour of extract_params_header.

rustfs-signer, rustfs-madmin and rustfs-targets no longer depend on
s3s; the s3s footprint baseline drops from 209 to 202 files.

Refs rustfs/backlog#2747, rustfs/backlog#2734

* refactor(ecstore): move S3ErrorCode to rustfs-s3-types by codemod (#8432)

* test(ecstore): pin the InvalidRange responses of new_getobjectreader

Both error branches of new_getobjectreader build an ErrorResponse with S3ErrorCode::InvalidRange, and no test observed either code. Pin the code, its wire name and the location fields of each branch before the code type moves to rustfs-s3-types (rustfs/backlog#2743), so a changed code on either branch fails a test. The ecstore test-count row rises by the two tests.

* chore(scripts): add the s3s-to-rustfs-s3-types error codemod

Task T1.2 of rustfs/backlog#2734 (rustfs/backlog#2743) moves S3Error, S3ErrorCode, S3Result and s3_error! from s3s to rustfs-s3-types by script, one crate closure per PR. The codemod rewrites use trees (splitting mixed s3s imports) and qualified code paths, never comments or literals, adds the rustfs-s3-types dependency, drops s3s only from a crate that no longer names it, refuses every file of the s3s edge allowlist, takes --hold for a file whose type is fixed outside the closure, and prints a per-crate census. A second run changes nothing. Its self-test runs under make script-tests.

* refactor(ecstore): take S3ErrorCode from rustfs-s3-types

Output of scripts/codemods/s3_error_to_s3_types.py over crates/ecstore, crates/lifecycle, crates/scanner and crates/replication (rustfs/backlog#2743 batches 1 and 2). The ErrorResponse code field, the two InvalidRange constructions in new_getobjectreader and an unused tier import now use rustfs_s3_types::S3ErrorCode; the variant names and wire names are unchanged. lifecycle, scanner and replication reference none of the four error names. warm_backend.rs and warm_backend_s3.rs are held: they compare against rustfs_s3_client::ErrorResponse.code, which stays the s3s type until s3-client moves.

* ci(footprint): count only s3_error! lines still on s3s's macro

s3_error_lines counted every line holding the text `s3_error!`, which
also matches the RustFS-owned `rustfs_s3_types::s3_error!`. A codemod
that moves call sites onto rustfs-s3-types therefore could not lower it
(rustfs/backlog#2743, decision 3; rustfs/rustfs#8432, gotcha 2).

The counter now applies a textual, per-file and per-line rule that errs
on the side of counting (full rule in the script header):
- occurrences spelled `rustfs_s3_types::s3_error!` never count;
- a line holding `s3s::s3_error!` always counts;
- any other `s3_error!` line counts unless its file is cleared, that is,
  it imports the macro from rustfs_s3_types by name on a line starting
  with the use item (or defines it with `macro_rules! s3_error`) and does
  not also bring s3s's macro in (`use s3s::s3_error`, a `use s3s::{..}`
  tree naming it or a crate-root `*`, `use s3s::*`, or
  `#[macro_use] extern crate s3s`);
- fail closed: a file matching neither form (`use super::*`, a
  crate-local re-export, a renamed or glob rustfs_s3_types import, a
  commented-out import, a comment-only mention) counts.

Census of the 1585 lines on 96fc06776: 1478 in 64 files that import the
macro from s3s; 107 in 15 files matching neither form (105 through
`use super::*`, 1 qualified `s3s::s3_error!`, 1 comment line); none in
files that import it from rustfs_s3_types or define it. Every line stays
counted, so the baseline remains s3_error_lines=1585 (files=202 and
ecstore_files=36 unchanged). Rewriting rustfs/src/auth.rs's import to
rustfs_s3_types lowers the new counter to 1567; the old one stayed 1585.

`--self-test` (wired into `make script-tests`) copies the script into
22 fixture trees: 6 positive cases and 16 negative ones (lines that must
not count, forms that must not clear a file, and a missing baseline,
missing key or failing classification search that must fail). Each tree
has a control line that must count, and each run gets s3s text on stdin.
Seven cases were red against the old counter. 19 mutants each turn their
case red: old definition, flipped import test, dropped fail-closed
branch, dropped e2e_test exclusion, dropped s3s veto, dropped qualified
s3s rule, dropped rustfs-qualified strip, unanchored rustfs import,
renamed import accepted, nested glob taken as root glob, single-line
s3s trees only, dropped s3s glob forms, dropped macro_use form, dropped
definition clearing, ignored classification rg errors, error-line search
without its path, classification searches without their path, missing
baseline skipped, missing key skipped.

* test(e2e): compare two RUSTFS_S3_STACK servers over every S3 operation (#8435)

Add dual_stack_differential to e2e_test. It starts two rustfs processes
from the same binary, stack A on RUSTFS_S3_STACK=legacy and stack B on
RUSTFS_E2E_STACK_B (default legacy until rustfs/backlog#2734 T2.5),
sends both the same aws-sdk-s3 script covering the 73 methods of
impl S3 for FS (a normal request and a 4xx negative each), and compares
status, error code, an allowlist of response headers and a few typed
body fields. Error message text is never compared. Each run writes
<target>/dual-stack/<run>.json with compared_ops and the (op, field,
a, b) differences; any difference fails the test.

differential_detects_an_injected_difference starts B with
RUSTFS_REGION=eu-west-1 and requires GetBucketLocation to be reported.
The comparator is pure and unit-tested without a server.

Wire the module into the e2e-smoke filter and update the e2e-smoke and
e2e-full selection digests and the e2e_test test-count row.

Refs rustfs/backlog#2740

* feat(app): add the RustFS-owned RequestEnvelope and its s3s constructor (#8437)

* feat(app): add the RustFS-owned RequestEnvelope and its s3s constructor

* ci: classify the s3s edge adapters in storage/s3_compat as interface

* feat(s3-types): add Body::from_stream and move Swift off s3s::Body (#8438)

rustfs/backlog#2747 batch 3. `Body::from_stream` wraps a chunk stream of
unknown length over `StreamBody`/`BoxBody`: one data frame per chunk, the
first error ends the body, the size hint stays unknown, and dropping the
body drops the stream. Parity tests pin it to the s3s `StreamingBlob::wrap`
it replaces, at the `http_body` level and byte for byte through hyper.

The Swift handlers in rustfs-protocols build their responses as the owned
`Body`. `SwiftService` converts them into the wrapped S3 service's response
body type through `From<Body>`, so rustfs keeps serving `s3s::Body` at the
legacy edge unchanged. Request bodies stay Send-only (`UnsyncBoxBody`); the
two storage writers and the DLO/SLO segment streams get `Sync` from
`sync_wrapper::SyncStream`, which is what s3s's `http_body_unsync` mutex
provided before. The DTO and error uses of s3s in protocols are untouched
(T1.5 and T1.2).

* refactor(protocols): move error types to rustfs-s3-types by codemod (#8439)

T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and
the error items of protocols move off s3s by scripts/codemods, run with
--hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it
downcasts the backend error, whose runtime type is still s3s::S3Error.

- rustfs/src/protocols/client.rs: the two webdav session methods now
  return rustfs_s3_types::S3Result and convert the s3s error with From.
- codemod: a child module that calls s3_error! through `use super::*`
  gets its own `use rustfs_s3_types::s3_error;` once its parent imports
  that macro by name; otherwise it is listed as GLOB.
- s3-types: a table test pins code, status and the s3s error the legacy
  edge renders for every variant a moved call site names.
- s3_error_lines 1585 -> 1584 (a lifecycle comment reworded).

* test(connect): restore one-second drive request budgets

Apply the exact test-only repair already landed on main in 5972a09cc3. The filesystem probe and production CLI success test request one second and keep their existing one-second duration and parameter checks. All outcome, I/O, error, cleanup and resource-limit assertions remain.

Scope: rustfs/tests/connect_perf_drive.rs only. Patch-id: cb0a1dfc8f68a67727525adea7458e2cee7d09f8.

Validation environment: LANG=en_US.UTF-8, CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0, exclusive APFS-cloned CARGO_TARGET_DIR.

Quick Checks: every command from .github/actions/quick-checks/action.yml, with captured output and exit status below.

$ shellcheck --version && actionlint
ShellCheck - shell script analysis tool
version: 0.11.0
license: GNU General Public License, version 3
website: https://www.shellcheck.net
exit_code=0

$ cargo fmt --all --check
exit_code=0

$ ./scripts/check_unsafe_code_allowances.sh
exit_code=0

$ ./scripts/check_layer_dependencies.sh
Layer dependency guard passed (no new reverse dependencies/cycles).
exit_code=0

$ ./scripts/check_architecture_migration_rules.sh
Architecture migration rules passed.
exit_code=0

$ ./scripts/check_logging_guardrails.sh
✅ Logging guardrails check passed
exit_code=0

$ ./scripts/check_error_other_format_ratchet.sh
error(format!) ratchet passed (total call sites: 582).
exit_code=0

$ ./scripts/check_no_tokio_io_uring.sh
exit_code=0

$ ./scripts/check_extension_schema_boundaries.sh
Extension schema boundaries passed.
exit_code=0

$ ./scripts/check_body_cache_whitelist.sh
Body-cache whitelist guard passed.
exit_code=0

$ ./scripts/check_s3s_footprint.sh
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)
✅ s3s footprint baseline check passed
exit_code=0

$ ./scripts/check_test_count_ratchet.sh
test-count ratchet OK: 52 crates, 21222 tests, 164 ignored match .config/test-count-baseline.txt
exit_code=0

$ ./scripts/check_fips_wording.sh
FIPS wording guard passed (README.md and CHANGELOG.md contain no forbidden claims).
KMS wording guard passed (crates/kms claims no Vault KV2 Transit wrapping).
exit_code=0

$ ./scripts/check_embedded_secrets.sh
Embedded secret guard passed (no private key material or provider credential literal in the tree).
exit_code=0

$ make script-tests
Running script tests...
entrypoint credential policy tests passed
internode grpc A/B bench env tests passed
object batch benchmark enhanced tests passed
PASS package service script contract
hotpath warp A/B gate tests passed
hotpath warp ABBA tests passed
Scanner validation artifacts written to /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.1pFe6Kl1Yu/out
Scanner validation artifacts written to /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.1pFe6Kl1Yu/out-missing-pid
.............................
----------------------------------------------------------------------
Ran 29 tests in 181.286s

OK
{"decision":"verified","pending_gates":["G01","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G02"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-E"}
{"decision":"verified","pending_gates":["G01","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G02"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-E"}
{"decision":"verified","pending_gates":["G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G01"}
{"decision":"verified","pending_gates":["G01","G02","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G03"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-L"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G14"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G14"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G10"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P1"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P2"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P3"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G05"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G06"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-D"}
Status-and-outcome release descriptor verified: /private/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp8rwn1pe0/descriptor/release-bundle-status-outcome.json
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G05"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G06"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-D"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G07"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G08"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P4"}
exact 1MiB handoff ABBA tests passed
pinned paired ABBA bench tests passed
==> /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/bin/path_containment (-max_total_time=7, -seed=123456789, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/explicit-artifacts/path_containment/, corpus=/private/tmp/gw-orchestrator-2734/ci-drive/fuzz/corpus/path_containment)
-seed=123456789
target=path_containment
seed=123456789
max_total_time=7
git_revision=4d15b816236630b2c42f321ffc9ed3654edc94e3
git_dirty=true
runner_mode=prebuilt
==> /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/bin/path_containment (-max_total_time=1, -seed=2322617706, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/auto-artifacts/path_containment/, corpus=/private/tmp/gw-orchestrator-2734/ci-drive/fuzz/corpus/path_containment)
-seed=2322617706
==> cargo +nightly fuzz run path_containment (-max_total_time=9, -seed=987654321, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/cargo-artifacts/path_containment/)
+nightly
fuzz
run
path_containment
-seed=987654321
seed=987654321
runner_mode=cargo-fuzz
fuzz runner seed manifest tests passed
✅ scripts/python_bin.sh resolver checks passed
self-test ok: clean fixture with every exemption present passes
self-test ok: all 11 pattern families fire
self-test ok: moving a pinned source is reported
Embedded secret guard self-test passed.
self-test ok: p1-s3s-import: 'use s3s::s3_error;' counts every line, the rustfs_s3_types module's too (s3_error_lines=3)
self-test ok: p2-s3s-mixed-tree: a multi-line s3s use tree naming s3_error counts (s3_error_lines=3)
self-test ok: p3-s3s-glob: a crate-root glob of s3s, bare or inside a tree, counts (s3_error_lines=5)
self-test ok: p4-macro-use: '#[macro_use] extern crate s3s' counts (s3_error_lines=3)
self-test ok: p5-qualified-s3s: a qualified s3s::s3_error! line counts in a cleared file (s3_error_lines=2)
self-test ok: p6-exact-pass: a baseline equal to the tree passes (exit 0)
self-test ok: n1-rustfs-import: 'use rustfs_s3_types::s3_error;' alone counts nothing (s3_error_lines=1)
self-test ok: n2-rustfs-mixed-tree: a rustfs_s3_types tree plus an s3s tree without the macro counts nothing (s3_error_lines=1)
self-test ok: n3-defines: the file defining the macro counts nothing (s3_error_lines=1)
self-test ok: n4-qualified-rustfs: a qualified rustfs_s3_types::s3_error! line counts nothing (s3_error_lines=1)
self-test ok: n5-renamed-call: a call through a renamed macro is not text-matched (out of scope) (s3_error_lines=1)
self-test ok: n6-comment-string: a comment or string mentioning s3_error! in a cleared file counts nothing (s3_error_lines=1)
self-test ok: n7-e2e-excluded: crates/e2e_test is excluded (s3_error_lines=1)
self-test ok: n8-super-glob: an unrecognised import form ('use super::*;') counts (s3_error_lines=3)
self-test ok: n9-facade: a crate-local re-export counts (s3_error_lines=2)
self-test ok: n10-renamed-import: a renamed rustfs_s3_types import does not clear s3_error! lines (s3_error_lines=2)
self-test ok: n11-commented-import: a commented-out rustfs_s3_types import does not clear (s3_error_lines=2)
self-test ok: n12-rustfs-glob: a glob import of rustfs_s3_types does not clear (s3_error_lines=2)
self-test ok: n13-comment-only: a comment mentioning s3_error! in a file with no import counts (s3_error_lines=2)
self-test ok: f1-baseline-missing: a missing baseline file fails (exit 1)
self-test ok: f2-key-missing: a baseline without s3_error_lines fails (exit 1)
self-test ok: f3-rg-fails: a failing classification search fails the guard (exit 1)
✅ s3s footprint self-test passed
test_core_gate_rejects_missing_ignored_filtered_and_corrupt_inputs (__main__.SelfTests.test_core_gate_rejects_missing_ignored_filtered_and_corrupt_inputs) ... ok
test_e2e_requires_registration (__main__.SelfTests.test_e2e_requires_registration) ... ok
test_fuzz_runtime_uses_matrix_target (__main__.SelfTests.test_fuzz_runtime_uses_matrix_target) ... ok
test_ilm_lane_keeps_the_measured_cargo_build_budget (__main__.SelfTests.test_ilm_lane_keeps_the_measured_cargo_build_budget) ... ok
test_profile_listing_binds_platform_digest (__main__.SelfTests.test_profile_listing_binds_platform_digest) ... ok
test_profile_listing_enforces_selection (__main__.SelfTests.test_profile_listing_enforces_selection) ... ok
test_quick_checks_commands_propagate_failure (__main__.SelfTests.test_quick_checks_commands_propagate_failure) ... ok
test_quick_checks_rejects_caller_and_execution_bypasses (__main__.SelfTests.test_quick_checks_rejects_caller_and_execution_bypasses) ... ok
test_runner_readiness_fails_closed (__main__.SelfTests.test_runner_readiness_fails_closed) ... ok
test_s3_runner_rejects_unbounded_failure_locals (__main__.SelfTests.test_s3_runner_rejects_unbounded_failure_locals) ... ok
test_scanner_heal_begin_requires_embedded_source_provenance (__main__.SelfTests.test_scanner_heal_begin_requires_embedded_source_provenance) ... ok
test_scanner_heal_case_does_not_approve_pending_release (__main__.SelfTests.test_scanner_heal_case_does_not_approve_pending_release) ... ok
test_scanner_heal_case_lane_is_required_for_runner_profile (__main__.SelfTests.test_scanner_heal_case_lane_is_required_for_runner_profile) ... ok
test_scanner_heal_case_only_schema_cannot_approve_release (__main__.SelfTests.test_scanner_heal_case_only_schema_cannot_approve_release) ... ok
test_scanner_heal_crash_case_rejects_restart_oracle (__main__.SelfTests.test_scanner_heal_crash_case_rejects_restart_oracle) ... ok
test_scanner_heal_ec84_case_rejects_wrong_erasure_geometry (__main__.SelfTests.test_scanner_heal_ec84_case_rejects_wrong_erasure_geometry) ... ok
test_scanner_heal_finish_collects_oracles_from_registry (__main__.SelfTests.test_scanner_heal_finish_collects_oracles_from_registry) ... ok
test_scanner_heal_multipool_case_accepts_deferred_optional_outage_write (__main__.SelfTests.test_scanner_heal_multipool_case_accepts_deferred_optional_outage_write) ... ok
test_scanner_heal_pending_gate_cannot_map_to_implemented_lane (__main__.SelfTests.test_scanner_heal_pending_gate_cannot_map_to_implemented_lane) ... ok
test_scanner_heal_receipts_reject_reuse_and_missing_builds (__main__.SelfTests.test_scanner_heal_receipts_reject_reuse_and_missing_builds) ... ok
test_scanner_heal_rejects_boolean_fractional_and_out_of_geometry_integers (__main__.SelfTests.test_scanner_heal_rejects_boolean_fractional_and_out_of_geometry_integers) ... ok
test_scanner_heal_rejects_broken_execution_and_artifacts (__main__.SelfTests.test_scanner_heal_rejects_broken_execution_and_artifacts) ... ok
test_scanner_heal_rejects_copied_junit_and_wrong_suite_build (__main__.SelfTests.test_scanner_heal_rejects_copied_junit_and_wrong_suite_build) ... ok
test_scanner_heal_release_bundle_accepts_complete_measured_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_accepts_complete_measured_evidence) ... ok
test_scanner_heal_release_bundle_assembler_copies_measured_artifacts (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_copies_measured_artifacts) ... ok
test_scanner_heal_release_bundle_assembler_merges_measured_descriptors (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_merges_measured_descriptors) ... ok
test_scanner_heal_release_bundle_assembler_rejects_duplicate_descriptor_gate (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_duplicate_descriptor_gate) ... ok
test_scanner_heal_release_bundle_assembler_rejects_fixture_descriptor (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_fixture_descriptor) ... ok
test_scanner_heal_release_bundle_assembler_rejects_hash_mismatch (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_hash_mismatch) ... ok
test_scanner_heal_release_bundle_assembler_rejects_mixed_descriptor_revisions (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_mixed_descriptor_revisions) ... ok
test_scanner_heal_release_bundle_enforces_topology_duration_profile_and_versions (__main__.SelfTests.test_scanner_heal_release_bundle_enforces_topology_duration_profile_and_versions) ... ok
test_scanner_heal_release_bundle_fixture_writer_is_not_release_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_fixture_writer_is_not_release_evidence) ... ok
test_scanner_heal_release_bundle_gate_accepts_single_measured_gate (__main__.SelfTests.test_scanner_heal_release_bundle_gate_accepts_single_measured_gate) ... ok
test_scanner_heal_release_bundle_gate_rejects_missing_field (__main__.SelfTests.test_scanner_heal_release_bundle_gate_rejects_missing_field) ... ok
test_scanner_heal_release_bundle_gate_rejects_unknown_gate (__main__.SelfTests.test_scanner_heal_release_bundle_gate_rejects_unknown_gate) ... ok
test_scanner_heal_release_bundle_rejects_synthetic_or_missing_fields (__main__.SelfTests.test_scanner_heal_release_bundle_rejects_synthetic_or_missing_fields) ... ok
test_scanner_heal_release_bundle_requires_domain_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_requires_domain_evidence) ... ok
test_scanner_heal_release_bundle_requires_field_provenance (__main__.SelfTests.test_scanner_heal_release_bundle_requires_field_provenance) ... ok
test_scanner_heal_release_bundle_requires_json_artifact_provenance (__main__.SelfTests.test_scanner_heal_release_bundle_requires_json_artifact_provenance) ... ok
test_scanner_heal_release_bundle_requires_mixed_version_field_roles (__main__.SelfTests.test_scanner_heal_release_bundle_requires_mixed_version_field_roles) ... ok
test_scanner_heal_release_bundle_requires_same_gate_measurement_window (__main__.SelfTests.test_scanner_heal_release_bundle_requires_same_gate_measurement_window) ... ok
test_scanner_heal_release_consumes_multiple_registry_oracles (__main__.SelfTests.test_scanner_heal_release_consumes_multiple_registry_oracles) ... ok
test_scanner_heal_release_matrix_lanes_remain_pending_without_real_evidence (__main__.SelfTests.test_scanner_heal_release_matrix_lanes_remain_pending_without_real_evidence) ... ok
test_scanner_heal_release_requirements_cannot_be_cleared (__main__.SelfTests.test_scanner_heal_release_requirements_cannot_be_cleared) ... ok
test_scanner_heal_release_status_rejects_focused_case_run (__main__.SelfTests.test_scanner_heal_release_status_rejects_focused_case_run) ... ok
test_scanner_heal_release_status_rejects_synthetic_case (__main__.SelfTests.test_scanner_heal_release_status_rejects_synthetic_case) ... ok
test_scanner_heal_required_evidence_fields_cannot_be_removed (__main__.SelfTests.test_scanner_heal_required_evidence_fields_cannot_be_removed) ... ok
test_scheduled_alerts_require_completion_watchdog (__main__.SelfTests.test_scheduled_alerts_require_completion_watchdog) ... ok
test_update_profile_selection_changes_only_requested_platform (__main__.SelfTests.test_update_profile_selection_changes_only_requested_platform) ... ok
test_validate_includes_quick_checks (__main__.SelfTests.test_validate_includes_quick_checks) ... ok
test_vault_tests_require_cross_process_serialization (__main__.SelfTests.test_vault_tests_require_cross_process_serialization) ... ok
test_workflow_readiness_requires_dependency_probe (__main__.SelfTests.test_workflow_readiness_requires_dependency_probe) ... ok

----------------------------------------------------------------------
Ran 52 tests in 14.301s

OK
e2e-full selection OK: 1 tests, sha256=29e30e6ed2ed38d9e5b0a7cdd9536abf5c719f7021a02b578ae1381563a3c273
...................................
----------------------------------------------------------------------
Ran 35 tests in 1.161s

OK
...................
----------------------------------------------------------------------
Ran 19 tests in 17.397s

OK
..........
----------------------------------------------------------------------
Ran 10 tests in 0.183s

OK
test_actual_selector_bootstrap_uses_base_policy_and_fails_closed (__main__.SelfTests.test_actual_selector_bootstrap_uses_base_policy_and_fails_closed) ... ok
test_documentation_paths_do_not_hide_build_or_fixture_changes (__main__.SelfTests.test_documentation_paths_do_not_hide_build_or_fixture_changes) ... ok
test_e2e_scope_keeps_production_and_shared_configuration_full (__main__.SelfTests.test_e2e_scope_keeps_production_and_shared_configuration_full) ... ok
test_e2e_shortcut_rejects_new_direct_renamed_and_target_dependencies (__main__.SelfTests.test_e2e_shortcut_rejects_new_direct_renamed_and_target_dependencies) ... ok
test_e2e_shortcut_traverses_implicit_path_dependencies (__main__.SelfTests.test_e2e_shortcut_traverses_implicit_path_dependencies) ... ok
test_event_contract_requires_complete_candidate_and_optional_lanes (__main__.SelfTests.test_event_contract_requires_complete_candidate_and_optional_lanes) ... ok
test_every_wrong_result_missing_job_or_selection_fails_closed (__main__.SelfTests.test_every_wrong_result_missing_job_or_selection_fails_closed) ... ok
test_full_e2e_gate_preserves_workflow_branch_and_event_scope (__main__.SelfTests.test_full_e2e_gate_preserves_workflow_branch_and_event_scope) ... ok
test_git_range_includes_deleted_source_and_rename_origins (__main__.SelfTests.test_git_range_includes_deleted_source_and_rename_origins) ... ok
test_job_ids_and_display_names_cannot_hide_validation (__main__.SelfTests.test_job_ids_and_display_names_cannot_hide_validation) ... ok
test_report_only_stack_lane_mirrors_e2e_smoke_outside_the_aggregate (__main__.SelfTests.test_report_only_stack_lane_mirrors_e2e_smoke_outside_the_aggregate) ... ok
test_repository_wiring_and_missing_dependency_regression (__main__.SelfTests.test_repository_wiring_and_missing_dependency_regression) ... ok
test_verify_command_preserves_failures (__main__.SelfTests.test_verify_command_preserves_failures) ... ok

----------------------------------------------------------------------
Ran 13 tests in 3.280s

OK
security coverage self-test passed
test_api_failures_preserve_other_evidence_and_never_enter_grace (__main__.SelfTests.test_api_failures_preserve_other_evidence_and_never_enter_grace) ... ok
test_attempt_outcome_does_not_replace_recent_success (__main__.SelfTests.test_attempt_outcome_does_not_replace_recent_success) ... ok
test_cancelled_attempt_cannot_refresh_expired_success (__main__.SelfTests.test_cancelled_attempt_cannot_refresh_expired_success) ... ok
test_cli_requires_the_repository_default_branch (__main__.SelfTests.test_cli_requires_the_repository_default_branch) ... ok
test_config_rejects_duplicate_and_invalid_entries (__main__.SelfTests.test_config_rejects_duplicate_and_invalid_entries) ... ok
test_freshness_boundaries (__main__.SelfTests.test_freshness_boundaries) ... ok
test_grace_requires_two_successful_queries_with_no_history (__main__.SelfTests.test_grace_requires_two_successful_queries_with_no_history) ... ok
test_invalid_api_evidence_fails_closed (__main__.SelfTests.test_invalid_api_evidence_fails_closed) ... ok
test_report_retains_every_workflow (__main__.SelfTests.test_report_retains_every_workflow) ... ok
test_requests_filter_schedule_default_branch_and_success_on_server (__main__.SelfTests.test_requests_filter_schedule_default_branch_and_success_on_server) ... ok

----------------------------------------------------------------------
Ran 10 tests in 0.051s

OK
..............................
----------------------------------------------------------------------
Ran 30 tests in 22.354s

OK
..........
----------------------------------------------------------------------
Ran 10 tests in 2.969s

OK
.................................
----------------------------------------------------------------------
Ran 33 tests in 0.160s

OK
Online functional runner labels: pf-testing
.......
----------------------------------------------------------------------
Ran 7 tests in 0.002s

OK
.......
----------------------------------------------------------------------
Ran 7 tests in 0.001s

OK
....
----------------------------------------------------------------------
Ran 4 tests in 0.128s

OK
...
----------------------------------------------------------------------
Ran 3 tests in 0.547s

OK
PASS: cold stampede script self-test
exit_code=0

$ python3 ./scripts/test_object_generation_protocol_model.py
test_a_recovery_barrier_cannot_choose_an_empty_value (__main__.ObjectGenerationProtocolModel.test_a_recovery_barrier_cannot_choose_an_empty_value) ... ok
test_all_bounded_two_proposer_interleavings_choose_at_most_one_value (__main__.ObjectGenerationProtocolModel.test_all_bounded_two_proposer_interleavings_choose_at_most_one_value) ... ok
test_an_unavailable_decision_quorum_cannot_choose_a_candidate (__main__.ObjectGenerationProtocolModel.test_an_unavailable_decision_quorum_cannot_choose_a_candidate) ... ok
test_counter_allocation_requires_persistence_and_is_consumed_once (__main__.ObjectGenerationProtocolModel.test_counter_allocation_requires_persistence_and_is_consumed_once) ... ok
test_counter_restart_burns_persisted_allocations_before_or_after_issue (__main__.ObjectGenerationProtocolModel.test_counter_restart_burns_persisted_allocations_before_or_after_issue) ... ok
test_crash_after_promise_before_reply_does_not_restore_an_older_ballot (__main__.ObjectGenerationProtocolModel.test_crash_after_promise_before_reply_does_not_restore_an_older_ballot) ... ok
test_duplicate_or_unconfigured_voters_cannot_form_a_quorum (__main__.ObjectGenerationProtocolModel.test_duplicate_or_unconfigured_voters_cannot_form_a_quorum) ... ok
test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision (__main__.ObjectGenerationProtocolModel.test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision) ... ok
test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected (__main__.ObjectGenerationProtocolModel.test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected) ... ok
test_forgetting_tombstone_authority_allows_an_aba_create (__main__.ObjectGenerationProtocolModel.test_forgetting_tombstone_authority_allows_an_aba_create) ... ok
test_issuing_before_persistence_reuses_a_ballot_after_restart (__main__.ObjectGenerationProtocolModel.test_issuing_before_persistence_reuses_a_ballot_after_restart) ... ok
test_lost_ack_and_restart_preserve_the_chosen_operation (__main__.ObjectGenerationProtocolModel.test_lost_ack_and_restart_preserve_the_chosen_operation) ... ok
test_minority_accepted_work_is_adopted_and_finished (__main__.ObjectGenerationProtocolModel.test_minority_accepted_work_is_adopted_and_finished) ... ok
test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample (__main__.ObjectGenerationProtocolModel.test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample) ... ok
test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it (__main__.ObjectGenerationProtocolModel.test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it) ... ok
test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor (__main__.ObjectGenerationProtocolModel.test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor) ... ok
test_replacing_voters_with_empty_state_can_choose_a_conflicting_value (__main__.ObjectGenerationProtocolModel.test_replacing_voters_with_empty_state_can_choose_a_conflicting_value) ... ok
test_skipping_predecessor_recovery_chooses_a_broken_revision_chain (__main__.ObjectGenerationProtocolModel.test_skipping_predecessor_recovery_chooses_a_broken_revision_chain) ... ok
test_stale_publication_rollback_and_gc_cannot_change_the_winner (__main__.ObjectGenerationProtocolModel.test_stale_publication_rollback_and_gc_cannot_change_the_winner) ... ok
test_superseded_operation_identity_cannot_be_reused_for_different_input (__main__.ObjectGenerationProtocolModel.test_superseded_operation_identity_cannot_be_reused_for_different_input) ... ok
test_superseded_operation_retry_returns_its_original_successor (__main__.ObjectGenerationProtocolModel.test_superseded_operation_retry_returns_its_original_successor) ... ok
test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision (__main__.ObjectGenerationProtocolModel.test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision) ... ok
test_tombstone_rejects_a_delayed_never_present_create (__main__.ObjectGenerationProtocolModel.test_tombstone_rejects_a_delayed_never_present_create) ... ok
test_unlearned_predecessor_is_recovered_before_the_second_slot (__main__.ObjectGenerationProtocolModel.test_unlearned_predecessor_is_recovered_before_the_second_slot) ... ok

----------------------------------------------------------------------
Ran 24 tests in 0.958s

OK
model: states=51023, transitions=167520, terminal=732, chosen_candidates=2
negative control: double-choice trace=((0, 'prepare', 0), (0, 'prepare', 1), (0, 'prepare', 2), (0, 'accept', 0), (0, 'accept', 1), (0, 'accept', 2), (1, 'prepare', 0), (1, 'prepare', 1), (1, 'prepare', 2), (1, 'accept', 0), (1, 'accept', 1), (1, 'accept', 2))
multi-slot round schedules: 640
exit_code=0

$ python3 ./scripts/check_test_wiring.py
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired
exit_code=0

$ ./scripts/check_no_planning_docs.sh
exit_code=0

$ ./scripts/check_uring_lane_lib_only.sh
OK: no 'uring_'-matching test functions under crates/ecstore/tests; --lib is safe
exit_code=0

Initial complete gate
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
 Nextest run ID e07fc5b9-c6c8-4cdd-ae61-54839bf24437 with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.016s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.011s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.009s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.009s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  16.950s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   4.025s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.008s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.007s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.035s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  21.087s] 10 tests run: 10 passed, 0 skipped
exit_code=0

Restored complete gate
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
 Nextest run ID 233b7307-5361-4eb7-94d5-b384f17d2d6a with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.009s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.008s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.008s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.008s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  19.518s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   4.146s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.009s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.018s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.009s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.054s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  23.787s] 10 tests run: 10 passed, 0 skipped
exit_code=0

Mutation M1: success_measurement reports max(actual elapsed, requested budget + 1), forcing 1001 ms for the one-second request. The retained duration assertion at rustfs/tests/connect_perf_drive.rs:105 fails.
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast -E 'test(real_local_write_read_is_measured_and_scratch_is_removed)'
info: experimental features enabled: setup-scripts
   Compiling rustfs v1.0.1 (/private/tmp/gw-orchestrator-2734/ci-drive/rustfs)
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs") generated 1 warning
warning: `rustfs` (bin "rustfs-cli") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 7m 06s
────────────
 Nextest run ID 9f618777-7bf6-4cbc-8803-dd7f50f449ed with nextest profile: ci
    Starting 1 test across 1 binary (9 tests skipped)
        FAIL [   0.016s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
  stdout ───

    running 1 test
    test real_local_write_read_is_measured_and_scratch_is_removed ... FAILED

    failures:

    failures:
        real_local_write_read_is_measured_and_scratch_is_removed

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 0.01s

  stderr ───

    thread 'real_local_write_read_is_measured_and_scratch_is_removed' (106986846) panicked at rustfs/tests/connect_perf_drive.rs:105:5:
    assertion failed: (1..=1_000).contains(&data.duration_millis)
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

────────────
     Summary [   0.017s] 1 test run: 0 passed, 1 failed, 9 skipped
        FAIL [   0.016s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
error: test run failed
exit_code=100

Mutation M2: target_parameters reports requested duration + 1. The in-process assertion at rustfs/tests/connect_perf_drive.rs:111 and CLI JSON assertion at rustfs/tests/connect_perf_drive.rs:527 both reject 1001 against 1000.
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast -E 'test(real_local_write_read_is_measured_and_scratch_is_removed) | test(production_cli_measures_and_signs_exact_binary_provenance)'
info: experimental features enabled: setup-scripts
   Compiling rustfs v1.0.1 (/private/tmp/gw-orchestrator-2734/ci-drive/rustfs)
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs") generated 1 warning
warning: `rustfs` (bin "rustfs-cli") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 7m 20s
────────────
 Nextest run ID b73d46ab-c910-4e1e-8063-7fc7e332ec15 with nextest profile: ci
    Starting 2 tests across 1 binary (8 tests skipped)
        FAIL [  13.968s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
  stdout ───

    running 1 test
    test production_cli_measures_and_signs_exact_binary_provenance ... FAILED

    failures:

    failures:
        production_cli_measures_and_signs_exact_binary_provenance

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 13.96s

  stderr ───

    thread 'production_cli_measures_and_signs_exact_binary_provenance' (107015462) panicked at rustfs/tests/connect_perf_drive.rs:527:5:
    assertion `left == right` failed
      left: Number(1001)
     right: 1000
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

        FAIL [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
  stdout ───

    running 1 test
    test real_local_write_read_is_measured_and_scratch_is_removed ... FAILED

    failures:

    failures:
        real_local_write_read_is_measured_and_scratch_is_removed

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 0.01s

  stderr ───

    thread 'real_local_write_read_is_measured_and_scratch_is_removed' (107016667) panicked at rustfs/tests/connect_perf_drive.rs:111:5:
    assertion `left == right` failed
      left: 1001
     right: 1000
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

────────────
     Summary [  13.985s] 2 tests run: 0 passed, 2 failed, 8 skipped
        FAIL [  13.968s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        FAIL [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
error: test run failed
exit_code=100

Transient production mutations were restored byte-for-byte before the complete green rerun. Original/restored SHA256: 8d389e6d8836c041aab8c6a93e56addd65ba3f70b33264e1c74f639d0572a2da.

$ git diff --check

exit_code=0

$ git diff --cached --check

exit_code=0

Independent orchestrator acceptance
Tested code tree: e67c72fb01

Tested tree: 16fbbd2ea4
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

$ LANG=en_US.UTF-8 CARGO_BUILD_JOBS=1 CARGO_INCREMENTAL=0 cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
info: experimental features enabled: setup-scripts
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs-cli") generated 1 warning
warning: `rustfs` (bin "rustfs") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.69s
────────────
 Nextest run ID cd1edf4c-f1c0-456d-ad1a-76cc00ce05c9 with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.012s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.010s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.010s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.009s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  20.328s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   5.527s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.013s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.019s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.011s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.049s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  25.991s] 10 tests run: 10 passed, 0 skipped

exit_code=0

All 54 tracked Cargo.toml files parse without duplicate keys. git diff --check passed. The production source remains unchanged; the two deliberate implementation mutations above were restored before this complete green run.

* fix(test): compare legacy body wire parity as raw bytes

Lossy UTF-8 comparison concealed a changed binary response byte.
Compare the already captured Vec<u8> values directly.

Validation:
- Binary mutant: migrated payload 0x84 vs legacy 0x83; old comparator
  passed 1/1, raw comparator failed 0/1 (exit 100); restored green.
- Non-Swift body-drain mutant: untouched-body delegation test failed
  with empty vs payload (exit 100); restored handler without a diff.
- cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols
  --features rustfs-protocols/swift --no-fail-fast:
  543 tests run, 543 passed, 0 skipped after both mutants restored.
- cargo fmt --all --check and git diff --check passed.

Orchestrator acceptance (LANG=en_US.UTF-8):
cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
Summary [2.445s] 543 tests run: 543 passed, 0 skipped.

All shared quick checks on the final code tree:
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

Final independent acceptance after the drive-test CI repair
Parent: a99d8f4f89
Tested code tree: 93cbd96bf0

Tested tree: 075ae5161c2b62a96812bc1e3c904a7663b4f2f8
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

$ LANG=en_US.UTF-8 CARGO_BUILD_JOBS=1 CARGO_INCREMENTAL=0 cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
    Finished `test` profile [unoptimized + debuginfo] target(s) in 10m 08s
 Nextest run ID 5c7ff102-ec6d-479a-86e0-810859b28bbf with nextest profile: ci
    Starting 543 tests across 8 binaries
     Summary [   6.729s] 543 tests run: 543 passed, 0 skipped
exit_code=0

The only source change is the raw-byte wire comparator. Cargo manifests and the ratchet baselines remain unchanged from the independently validated parent. The invalid-byte and consumed-request mutations remain recorded above; their implementation changes were restored.

* fix: repair integration checks and IPC test readiness

Wait for the RPC/locks test sockets to pass the existing owner and mode
checks inside the original 5s readiness window and 10ms poll interval.

Use the existing admin S3 carrier for the new recovery and quota error
calls. Regenerate the test-count baseline for main's 118 existing tests
across seven crates. Pair the private pagination helper's raw-scan inputs
and remove the redundant Copy clone from the retirement event.

Keep the original assertions, responses, authorization, locks and storage
operations. The pagination body and raw-scan expressions at all five calls
are byte-identical; reinserting only the Copy clone restores the entire
base heal file. All ignored baseline values and s3s counts stay unchanged.

Validation base: c47432e353
Validation tree: f081084bc5
Patch SHA-256: 2dcf0028691bfb2772edfb38e262829b04f4151832dcce823c9dd9f7434b52de
Trace production-prefix SHA-256: fd994f218a21429d62866f4efb8eee1ee57b58f4a51865aa35d05a54ef1ccb45

All commands used LANG=en_US.UTF-8, CARGO_BUILD_JOBS=1,
CARGO_INCREMENTAL=0 and CARGO_NET_OFFLINE=true in one exclusive target.
Every gate retained the same tree and six source hashes before and after.
54 Cargo.toml files parse without duplicate keys.

cargo clippy -p rustfs --all-targets -- -D warnings
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 12m 50s
Exit: 0. Wall time: 770.855s.

The original 19 quick-check steps:
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
Wall time: 286.337s.

cargo nextest run --profile ci -p rustfs --no-fail-fast
     Summary [ 217.017s] 5382 tests run: 5382 passed (4 leaky), 40 skipped
Exit: 0. Wall time: 1324.414s.

cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
     Summary [   2.189s] 543 tests run: 543 passed, 0 skipped
Exit: 0. Wall time: 123.192s.

cargo nextest run --profile ci -p rustfs-ecstore list_objects_paginate --no-fail-fast
     Summary [   0.170s] 8 tests run: 8 passed, 5921 skipped
Exit: 0. Wall time: 261.324s.

cargo nextest run --profile ci -p rustfs-heal durable_mrf_anchor_retires_only_after_exact_bucket_generation_is_deleted --no-fail-fast
     Summary [   0.805s] 1 test run: 1 passed, 835 skipped
Exit: 0. Wall time: 199.801s.

Baseline regeneration:
./scripts/check_test_count_ratchet.sh --update
wrote .config/test-count-baseline.txt: 52 crates, 21340 tests, 164 ignored
Test-only increases: ecstore48, heal3, iam14, io-metrics2, obs3, policy14,
rustfs34. Existing ignored values and all other rows are byte-identical.
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)

Historical runtime red controls on f622 (same retained two assertions):
cargo nextest run --profile ci -p rustfs --lib -E 'test(=connect::diagnostics::trace_runtime::tests::local_top_rpc_reads_service_http_completions_and_signs_offline) | test(=connect::diagnostics::trace_runtime::tests::local_top_locks_reads_service_manager_and_signs_offline)' --no-fail-fast
Original path readiness with a temporary 500ms bind/seal gap:
     Summary [   0.863s] 2 tests run: 0 passed, 2 failed, 5113 skipped
Exit: 100.
Sealed readiness in that same gap:
     Summary [   5.621s] 2 tests run: 2 passed, 5113 skipped
Exit: 0.
Compiled M1: replace the helper predicate with path.exists():
     Summary [   0.870s] 2 tests run: 0 passed, 2 failed, 5113 skipped
Exit: 100.
Restore the sealed predicate byte-for-byte:
     Summary [   5.640s] 2 tests run: 2 passed, 5113 skipped
Exit: 0.
Both reds observe mode755 with owner_matches=true and fail at the original
RPC/locks unwraps with StateSecurity. M1 failure locations: trace_runtime.rs
2547:29 and 2713:29. No new assertions were introduced.

cargo nextest run --profile ci -p rustfs --lib -E 'test(connect::diagnostics::trace_runtime::tests::)' --no-fail-fast
     Summary [  20.748s] 28 tests run: 28 passed (1 leaky), 5087 skipped
Exit: 0. This security sweep is historical; the full final unit gate above
covers the actual c474 candidate.
The first disk-stopped compile ran no tests and is excluded from red credit.
All temporary seal instrumentation is removed from the final source.

Final unit cleanup markers: 4 leaky, 40 original skips (exit0). The ld
__eh_frame linker warning also appears verbatim in the original f622 run
and the earlier c474 run; there are no new source warnings.

* fix: remove redundant pending closure in disk test

Pass the capture-free pending future factory as a function item in the existing slow-operation test. This fixes the redundant_closure error at disk_store.rs:2914 that stopped strict Clippy before the feature tests could run.

The single-line test edit preserves the factory's result, warning and timeout behavior. Reversing the replacement reproduces the parent file byte for byte. All 220 assertion lines and the production prefix are unchanged; no tests, assertions, guards, profiles or baselines were added or relaxed.

Original Linux CI evidence (run 37888552268 at 53245be9f):
- RIO job113685483992: strict Clippy exit101, redundant closure at disk_store.rs:2914; feature tests skipped.
- Workspace Test and Lint job113685484058: same strict Clippy exit101; nextest and doc tests skipped.
These are lint failures, not runtime mutation kills or known flakes.

Native macOS arm64 validation on tree7a5bf47e7533b29479bc85207416102f1337c566, LANG=en_US.UTF-8, one build job, CARGO_INCREMENTAL=0 and offline dependencies:

shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
All 19 checks: exit0, 312.161s.

$ cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 20m 11s
exit0, 1213.210s total.

$ cargo nextest run --profile ci -p rustfs-ecstore --features rio-v2 slow_disk_operation_warns_once_while_pending_and_preserves_result --no-fail-fast
Finished `test` profile [unoptimized + debuginfo] target(s) in 4m 34s
PASS [0.026s] rustfs-ecstore disk::disk_store::tests::slow_disk_operation_warns_once_while_pending_and_preserves_result
Summary [   0.170s] 1 test run: 1 passed, 5943 skipped
exit0, 281.920s total; 5943 existing filter skips.

warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
This linker warning is retained; flags, profiles and allowances are unchanged.

54 tracked Cargo.toml files parsed without duplicate keys. git diff --check passed. No new assertions were introduced, so no new runtime mutation is claimed. Native compilation and the existing helper test do not substitute for actual Linux RIO execution; the new integration CI run still needs to recover before the RPC merge.

Preservation and output hashes:
parent file SHA256: 8e59ee745e3aaaa22848b943d3b10f6353b515956282cde1f3af2c394200c9ac
final file SHA256: b69ba3237061b161448ae8b62b615885d5770b98a5b4a2479bcfbd70f1bd1f17
unchanged production prefix SHA256: 7903a6e288252c420231242d7b58a25fa2a7140ea3dd430cbe7f0dde11369dcb
RIO Linux failure log SHA256:421addb9868fd88e4109a922311e54f91f0dfd70862f8ba7fcaba193482ed90b
Workspace Linux failure log SHA256:e0aea8d0f014c022252a9534968e083c98cfc1fd760da445ecd04ec3f522e26a
19 output SHA256:35a54f28d707956ed1edb29d492c4dfe8a9fe753d103b111b5a2f1b5c41c2b98
RIO Clippy output SHA256:d6046e3868f2b3992fe11cdd04e9e98eb50a87d1234d9a70dbfeb154ff61638d
slow-operation output SHA256:605b7b258d2fd09c6ef2ab2ffc90d433d0509480f1e45125d20c823759e83608

Refs rustfs/backlog#2734.

* test(ecstore): isolate tier-free recovery fixture startup

The fixture constructor can take the production recovery lock and wait for 60 seconds before this test installs its queue and seeds metadata. Disable its background recovery only while creating this fixture, then restore the environment before calling the existing production recovery entrypoint.

Only the existing test setup changes. Its production entrypoint, default lock parameters, original assertions, 30-second timeout, cancellation and cleanup remain intact. No tests or assertions were added. Production and store/init.rs are byte-identical to the parent.

Independent acceptance ran on merge tree 8ec358cb42ba9ebb3bcfeafcef048a3f0366355b; the runner header reports parent HEAD, while before/after index and source hashes confirm this candidate. Worker task branch d8055766b8 has the same tree.

Every command used LANG=en_US.UTF-8. Task compilation also used CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0, CARGO_NET_OFFLINE=true, an independent target, and empty RUSTC_WRAPPER/RUSTC_WORKSPACE_WRAPPER to keep compilers in its owned process group without changing repository configuration.

Command:
shellcheck --version && actionlint
Exit 0; observed output excerpt:
version: 0.11.0
Log SHA-256: fc9eec16b4191d058a295813ca9afbe9d2b37b59975da860170726be162a3924

Command:
cargo fmt --all --check
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_unsafe_code_allowances.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_layer_dependencies.sh
Exit 0; observed output excerpt:
Layer dependency guard passed (no new reverse dependencies/cycles).
Log SHA-256: 69c6cb1896c0586b28d1167380d3260d99d8017a4265c8df5a2348f289e93a12

Command:
./scripts/check_architecture_migration_rules.sh
Exit 0; observed output excerpt:
Architecture migration rules passed.
Log SHA-256: 28790e70d070607ed4de0704f292a65030be92ec224ef106d4b72fd8296f896e

Command:
./scripts/check_logging_guardrails.sh
Exit 0; observed output excerpt:
✅ Logging guardrails check passed
Log SHA-256: 7adf41abff8ed8f7617decf86498888ddde78937099dd9ce47f6d359dfc2d317

Command:
./scripts/check_error_other_format_ratchet.sh
Exit 0; observed output excerpt:
error(format!) ratchet passed (total call sites: 579).
Log SHA-256: 66172aef18f150ba0605af122e26ffcf91f6dd86f43e38b05e2e982ce8d783ee

Command:
./scripts/check_no_tokio_io_uring.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_extension_schema_boundaries.sh
Exit 0; observed output excerpt:
Extension schema boundaries passed.
Log SHA-256: f13d0acd0e2ce5d9815241b12b4dab7d01cc11b9b65bcf6dfb7812c878532a51

Command:
./scripts/check_body_cache_whitelist.sh
Exit 0; observed output excerpt:
Body-cache whitelist guard passed.
Log SHA-256: 6b828847b0383950ee6eef9cd846d0c322c945e47efd4c7d1ad881178810d6bb

Command:
./scripts/check_s3s_footprint.sh
Exit 0; observed output excerpt:
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)
✅ s3s footprint baseline check passed
Log SHA-256: 47f53d04ef645618563fe1f22abc64667f19078f1974feedcca46e4f2c016844

Command:
./scripts/check_test_count_ratchet.sh
Exit 0; observed output excerpt:
test-count ratchet OK: 52 crates, 21340 tests, 164 ignored match .config/test-count-baseline.txt
Log SHA-256: 618a5e330e933de0c6ff484546df401457024a88bb5986d11d588794e33dab60

Command:
./scripts/check_fips_wording.sh
Exit 0; observed output excerpt:
FIPS wording guard passed (README.md and CHANGELOG.md contain no forbidden claims).
KMS wording guard passed (crates/kms claims no Vault KV2 Transit wrapping).
Log SHA-256: 99221cc9979c480ec4f46f50d76c52b18c72fe499f12e15e4f4a22a1b8bfb140

Command:
./scripts/check_embedded_secrets.sh
Exit 0; observed output excerpt:
Embedded secret guard passed (no private key material or provider credential literal in the tree).
Log SHA-256: 437a3ceba32094d2901889965ad191e411aad32cdb18bf0b8659e818011c406b

Command:
make script-tests
Exit 0; observed output excerpt:
test_cli_requires_the_repository_default_branch (__main__.SelfTests.test_cli_requires_the_repository_default_branch) ... ok
test_config_rejects_duplicate_and_invalid_entries (__main__.SelfTests.test_config_rejects_duplicate_and_invalid_entries) ... ok
test_freshness_boundaries (__main__.SelfTests.test_freshness_boundaries) ... ok
test_grace_requires_two_successful_queries_with_no_history (__main__.SelfTests.test_grace_requires_two_successful_queries_with_no_history) ... ok
test_invalid_api_evidence_fails_closed (__main__.SelfTests.test_invalid_api_evidence_fails_closed) ... ok
test_report_retains_every_workflow (__main__.SelfTests.test_report_retains_every_workflow) ... ok
test_requests_filter_schedule_default_branch_and_success_on_server (__main__.SelfTests.test_requests_filter_schedule_default_branch_and_success_on_server) ... ok
Ran 10 tests in 0.042s
OK
Ran 30 tests in 20.627s
OK
Ran 10 tests in 2.268s
OK
Ran 33 tests in 0.127s
OK
Ran 7 tests in 0.002s
OK
Ran 7 tests in 0.001s
OK
Ran 4 tests in 0.108s
OK
Ran 3 tests in 0.564s
OK
PASS: cold stampede script self-test
Log SHA-256: 75fda2d069a373c63667b908baaf99e6591aab9099a739b0c4c22ceb6da8d68c

Command:
python3 ./scripts/test_object_generation_protocol_model.py
Exit 0; observed output excerpt:
test_an_unavailable_decision_quorum_cannot_choose_a_candidate (__main__.ObjectGenerationProtocolModel.test_an_unavailable_decision_quorum_cannot_choose_a_candidate) ... ok
test_counter_allocation_requires_persistence_and_is_consumed_once (__main__.ObjectGenerationProtocolModel.test_counter_allocation_requires_persistence_and_is_consumed_once) ... ok
test_counter_restart_burns_persisted_allocations_before_or_after_issue (__main__.ObjectGenerationProtocolModel.test_counter_restart_burns_persisted_allocations_before_or_after_issue) ... ok
test_crash_after_promise_before_reply_does_not_restore_an_older_ballot (__main__.ObjectGenerationProtocolModel.test_crash_after_promise_before_reply_does_not_restore_an_older_ballot) ... ok
test_duplicate_or_unconfigured_voters_cannot_form_a_quorum (__main__.ObjectGenerationProtocolModel.test_duplicate_or_unconfigured_voters_cannot_form_a_quorum) ... ok
test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision (__main__.ObjectGenerationProtocolModel.test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision) ... ok
test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected (__main__.ObjectGenerationProtocolModel.test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected) ... ok
test_forgetting_tombstone_authority_allows_an_aba_create (__main__.ObjectGenerationProtocolModel.test_forgetting_tombstone_authority_allows_an_aba_create) ... ok
test_issuing_before_persistence_reuses_a_ballot_after_restart (__main__.ObjectGenerationProtocolModel.test_issuing_before_persistence_reuses_a_ballot_after_restart) ... ok
test_lost_ack_and_restart_preserve_the_chosen_operation (__main__.ObjectGenerationProtocolModel.test_lost_ack_and_restart_preserve_the_chosen_operation) ... ok
test_minority_accepted_work_is_adopted_and_finished (__main__.ObjectGenerationProtocolModel.test_minority_accepted_work_is_adopted_and_finished) ... ok
test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample (__main__.ObjectGenerationProtocolModel.test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample) ... ok
test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it (__main__.ObjectGenerationProtocolModel.test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it) ... ok
test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor (__main__.ObjectGenerationProtocolModel.test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor) ... ok
test_replacing_voters_with_empty_state_can_choose_a_conflicting_value (__main__.ObjectGenerationProtocolModel.test_replacing_voters_with_empty_state_can_choose_a_conflicting_value) ... ok
test_skipping_predecessor_recovery_chooses_a_broken_revision_chain (__main__.ObjectGenerationProtocolModel.test_skipping_predecessor_recovery_chooses_a_broken_revision_chain) ... ok
test_stale_publication_rollback_and_gc_cannot_change_the_winner (__main__.ObjectGenerationProtocolModel.test_stale_publication_rollback_and_gc_cannot_change_the_winner) ... ok
test_superseded_operation_identity_cannot_be_reused_for_different_input (__main__.ObjectGenerationProtocolModel.test_superseded_operation_identity_cannot_be_reused_for_different_input) ... ok
test_superseded_operation_retry_returns_its_original_successor (__main__.ObjectGenerationProtocolModel.test_superseded_operation_retry_returns_its_original_successor) ... ok
test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision (__main__.ObjectGenerationProtocolModel.test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision) ... ok
test_tombstone_rejects_a_delayed_never_present_create (__main__.ObjectGenerationProtocolModel.test_tombstone_rejects_a_delayed_never_present_create) ... ok
test_unlearned_predecessor_is_recovered_before_the_second_slot (__main__.ObjectGenerationProtocolModel.test_unlearned_predecessor_is_recovered_before_the_second_slot) ... ok
Ran 24 tests in 0.929s
OK
Log SHA-256: 0b4be926efb0609d497ae58930340153f01957b606206cbbdb3e0d47eb29d3a2

Command:
python3 ./scripts/check_test_wiring.py
Exit 0; observed output excerpt:
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired
Log SHA-256: 21f676f458451525535ab0ef73d6a9c0d9774e16d76a4d19e396ad34936dd39c

Command:
./scripts/check_no_planning_docs.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_uring_lane_lib_only.sh
Exit 0; observed output excerpt:
OK: no 'uring_'-matching test functions under crates/ecstore/tests; --lib is safe
Log SHA-256: 43f3f851e27ec2ab9e34d6b05fefdede38af992699c473904bfe6f53a46d52e3

Command:
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast
Exit 0; wall 234.399s.
Observed output:
    Finished `test` profile [unoptimized + debuginfo] target(s) in 3m 49s
    Starting 1 test across 1 binary (5871 tests skipped)
  SETUP PASS [   0.006s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"'
        PASS [   0.440s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item
     Summary [   0.586s] 1 test run: 1 passed, 5871 skipped
Log SHA-256: 34e973f86e8dc2102c1a0e483e0ac802ce7df3d1b40a603932f62c0805c4de27

Command:
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over) | test(=store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence) | test(=store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable)' --no-fail-fast
Exit 0; wall 4.010s.
Observed output:
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.43s
    Starting 3 tests across 1 binary (5869 tests skipped)
  SETUP PASS [   0.005s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"'
        PASS [   0.068s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over
        PASS [   0.637s] rustfs-ecstore store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence
        PASS [   0.846s] rustfs-ecstore store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable
     Summary [   0.987s] 3 tests run: 3 passed, 5869 skipped
Log SHA-256: b5da2a927ff18ca9aa217762f7186a6d4371bd9e8feedd7b77940e10b64dffac

Command:
cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
Exit 0; wall 1084.508s.
Observed output:
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 18m 03s
Log SHA-256: 28f98be7970faaa71cd118da1da4e6e2e3d23b9dda7e6449eaa7a37e741260b5

Compiled mutation command (unchanged for each red and restored run):
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast

M1: reverse only this test-local constructor flag from false to true. The mutant compiled, then failed the original recovered-item queue assertion after 30.481s. Restoring the entire observed candidate passed in 0.460s. Red log 55bf3b1535529ac0bcfb8071f9b56f9f40631007a85c315367e7e93786d65293; restored log 204d5fbe4574a7917f5e740a795d864fe28a7e7fb5e33d27b0a9cf68e2f17f52.

M2: replace only the actual awaited enqueue with false, consuming its original inputs. The mutant compiled, then failed the same original queue assertion after 30.516s. Restoring the entire observed candidate passed in 0.457s. Red log 89186330413215e231e2b06bf013cc31097ea8f4a106d91421a36d6c8986a7fd; restored log e848b42d2302401afa5dbcd9e37bf76d2e56a96ab6de5a488513b997f53c5e9b.

All temporary observation and mutation code was removed before final acceptance. Historical baseline and diagnostic failures remain recorded; they are not mutation credit. The separate Object Lock intent-retry diagnostic still fails, and these native macOS checks do not establish Linux CI recovery.

* refactor(rpc): serve internode rpc bodies with the owned Body (#8441)

* test(rpc): pin the wire bytes of every internode rpc response body

rustfs/backlog#2747 batch 4, step 1 of 2, on the s3s bodies the rpc paths
serve today. Each response body builder in storage/rpc/http_service.rs is
served through hyper's HTTP/1.1 server and the raw bytes the peer reads are
compared with goldens captured from this commit: status, content-length or
chunked framing, chunk boundaries, the terminating chunk, HEAD, and the cut
connection a failure after data produces. The capability responses carry a
per-process proof, so their length header and payload are checked against
the body they encode instead.

The read_file_stream response builder moves into read_file_stream_response
so the test reaches the same code the handler runs; behaviour is unchanged.

* refactor(rpc): serve internode rpc bodies with the owned Body

rustfs/backlog#2747 batch 4, step 2 of 2. The internode rpc HTTP paths
build their responses with rustfs_s3_types::Body: Body::from_stream
replaces Body::from(StreamingBlob::wrap(..)) at the six stream sites, and
the in-memory and empty bodies keep their constructors. InternodeRpcService
converts its responses into the wrapped service's response body through
From<Body>, so the server keeps composing it with the s3s service as
before. The wire-byte goldens from the previous commit pass unchanged;
storage/rpc no longer names s3s.

* fix(rpc): simplify wire test response builder type

Name the boxed response builder used by the capability wire test so
all-target Clippy can check it without a type-complexity warning.

Validation (LANG=en_US.UTF-8, CARGO_INCREMENTAL=0):
- cargo clippy -p rustfs --all-targets -- -D warnings: exit 0.
- cargo nextest run --profile ci -p rustfs --lib -E 'test(/storage::rpc::/)': 279 tests run, 279 passed, 4842 skipped.
- All 19 quick-check commands, including script-tests, both Python scripts and cargo fmt --all --check: exit 0.
- Footprint baseline regenerated: files=194, s3_error_lines=1584, ecstore_files=36; test-count baseline regenerated: 52 crates, 21228 tests, 164 ignored.
- All 54 Cargo.toml files parse without duplicate keys.

No assertion was added or changed by this lint fix. R1-R8 are retained
in the existing PR verification record.

* fix(ecstore): resume fenced Object Lock creation intents

* fix(ecstore): bound creation retry futures on the stack

* fix(ecstore): require native proof for creation retries

* fix(ecstore): bound bucket creation stack usage (#8466)

* fix(ecstore): retain creation fences through rollback

Keep lifecycle, metadata-transaction, and namespace guards around rollback and metadata cleanup. Preserve the original publication error when a guard is lost. Build the timestamp retry fixture through the real interrupted producer so it checks a valid pending creation.

Validated source tree: 4e8e207999.
Two independent source reviews found no blocking regression.

Validation (LANG=en_US.UTF-8):
- bash -c 'shellcheck --version && actionlint' — exit 0
- cargo fmt --all --check — exit 0
- ./scripts/check_unsafe_code_allowances.sh — exit 0
- ./scripts/check_layer_dependencies.sh — exit 0
- ./scripts/check_architecture_migration_rules.sh — exit 0
- ./scripts/check_logging_guardrails.sh — exit 0
- ./scripts/check_error_other_format_ratchet.sh — exit 0
- ./scripts/check_no_tokio_io_uring.sh — exit 0
- ./scripts/check_extension_schema_boundaries.sh — exit 0
- ./scripts/check_body_cache_whitelist.sh — exit 0
- ./scripts/check_s3s_footprint.sh — exit 0
- ./scripts/check_test_count_ratchet.sh — exit 0
- ./scripts/check_fips_wording.sh — exit 0
- ./scripts/check_embedded_secrets.sh — exit 0
- make script-tests — exit 0
- python3 ./scripts/test_object_generation_protocol_model.py — exit 0
- python3 ./scripts/check_test_wiring.py — exit 0
- ./scripts/check_no_planning_docs.sh — exit 0
- ./scripts/check_uring_lane_lib_only.sh — exit 0
- cargo nextest run --profile ci -p rustfs-ecstore --lib --locked --no-fail-fast — 5894 passed, 0 skipped
- cargo check -p rustfs-ecstore --all-targets --locked — exit 0
- cargo nextest run --profile ci -p rustfs --lib --locked --no-fail-fast -E 'test(=admin::handlers::table_catalog::tests::cold_strong_registration_completes_while_other_bucket_backup_waits) | test(=admin::handlers::table_catalog::tests::commit_backend_borrowed_put_preserves_owner_authorization_and_precondition) | test(=admin::handlers::table_catalog::tests::disable_table_bucket_preserves_objects_and_can_be_reenabled) | test(=admin::handlers::table_catalog::tests::load_table_handler_negotiates_vended_credentials_without_breaking_metadata_only_callers) | test(=admin::handlers::table_catalog::tests::namespace_property_handler_updates_object_backed_catalog_and_maps_errors) | test(=app::object::get::tests::execute_get_object_resumes_from_relocated_pool_without_splicing_body) | test(=table_catalog::store::tests::catalog_writer_loss_prevents_recovery_storage_commit)' — 7 passed
- cargo nextest run --profile ci -p rustfs --lib --locked --no-fail-fast -E 'test(unsupported_remote_disk_)' — 2 passed

Additional fault-injection evidence remains incomplete. Timestamp M1 reached its original assertion; its restore run is being collected. M2–M5 and the three planned rollback faults have not run. An automated risk check interrupted the timestamp proof task; no assertions or validation settings were weakened.

* fix(ecstore): box nested bucket rollback state

Keep the guarded rollback future behind a heap allocation so bucket creation does not store its nested state inline. The rollback body, all three fence references and pending-proof revocation order are unchanged.

The same three original Linux tests overflowed on both e611 CI attempts. Local format, ecstore test compilation and all 19 quick-check steps pass; the original Linux feature lanes still need to validate this candidate.

* test(ecstore): isolate rollback guard loss fixtures

Run the three rollback metadata-cleanup cases in separate test processes so reused /proc/self/fd aliases from earlier cases cannot invalidate their creation witnesses. Keep the original fixture, guards, assertions and io_uring test selection, and verify that each child runs the intended case successfully.

Update the generated test-count baseline for the added macro arm.

Validation: cargo check -p rustfs-ecstore --tests; cargo fmt --all --check; all 19 quick-check commands, including make script-tests and the two Python checks. Linux io_uring acceptance and fault controls remain pending.

* refactor(replication): move the replication config onto gateway types (#8443)

* wip: T1.4 replication family

* refactor(replication): move the replication config onto gateway types

The replication configuration family leaves s3s: rustfs-replication no longer depends on s3s, and ECStore bucket metadata, the scanner and the rustfs app/admin layers hold rustfs_gateway_types::persistence::PersistedReplicationConfiguration. The persisted XML goes through the gateway persistence codec behind a new BucketConfigXml trait (the 39 byte goldens pass with unchanged fixtures); the legacy s3s edge converts once in storage/ecfs.rs through storage/s3_compat/replication.rs; the scanner data usage cache keeps its MessagePack bytes through a serde mirror pinned by bytes recorded from s3s 5761ddfe. rustfs-gateway-types 0.32.0 is pinned once (rev 772d3385). Refs rustfs/backlog#2745.

* test: use S3 serializer for list response tests

* fix(ci): restore integration test compilation

Gate the diagnostic test import with its feature, initialize the conditional
MetadataWriteStore fields explicitly, and decode the scanner replication fixture
through its existing cache boundary. Preserve the original fixture and assertions.

Validation was run on tree 544acb80c0 at 6003d4b8.
LANG=en_US.UTF-8; no source changes follow validation.

Quick checks: 19/19 passed.
- shellcheck --version && actionlint: exit 0
- cargo fmt --all --check: exit 0
- ./scripts/check_unsafe_code_allowances.sh: exit 0
- ./scripts/check_layer_dependencies.sh: exit 0
- ./scripts/check_architecture_migration_rules.sh: exit 0
- ./scripts/check_logging_guardrails.sh: exit 0
- ./scripts/check_error_other_format_ratchet.sh: exit 0
- ./scripts/check_no_tokio_io_uring.sh: exit 0
- ./scripts/check_extension_schema_boundaries.sh: exit 0
- ./scripts/check_body_cache_whitelist.sh: exit 0
- ./scripts/check_s3s_footprint.sh: exit 0
- ./scripts/check_test_count_ratchet.sh: exit 0
- ./scripts/check_fips_wording.sh: exit 0
- ./scripts/check_embedded_secrets.sh: exit 0
- make script-tests: exit 0
- python3 ./scripts/test_object_generation_protocol_model.py: exit 0
- python3 ./scripts/check_test_wiring.py: exit 0
- ./scripts/check_no_planning_docs.sh: exit 0
- ./scripts/check_uring_lane_lib_only.sh: exit 0

Darwin validation:
- cargo clippy --locked -p rustfs-ecstore -p rustfs-scanner --all-targets -- -D warnings: exit 0
- cargo nextest run --locked -p rustfs-ecstore -p rustfs-scanner --lib -E 'test(config::com::tests::save_config_preserves_retryable_system_volume_errors) | test(scanner_io::tests::replication_maintenance_refreshes_compacted_stats_across_cycles_and_retries)' --test-threads 1 --no-fail-fast: exit 0, 2/2 passed
- cargo clippy --locked -p rustfs-ecstore -p rustfs-scanner --all-targets --features rustfs-ecstore/diagnostic-read-version-caller,rustfs-scanner/diagnostic-read-version-caller -- -D warnings: exit 101
- cargo nextest run --locked -p rustfs-ecstore -p rustfs-scanner --lib --features rustfs-ecstore/diagnostic-read-version-caller,rustfs-scanner/diagnostic-read-version-caller -E 'test(config::com::tests::save_config_preserves_retryable_system_volume_errors) | test(scanner_io::tests::replication_maintenance_refreshes_compacted_stats_across_cycles_and_retries) | test(cluster::rpc::runtime_sources::tests::) | test(config::com::tests::diagnostic_config_write_retains_origin_and_api_for_results_and_cancellation)' --test-threads 1 --no-fail-fast: exit 0, 14/14 passed

Diagnostic strict Clippy remains red on four unchanged production lints:
runtime_sources.rs if_same_then_else and three redundant_locals reports in
set_disk/core/metadata_read.rs and set_disk/read.rs. Their source ranges and
feature wiring match the base; the base was not compiled separately.

Linux aarch64 validation (Rust/Clippy 1.99.0, UID 501, jobs=1):
- cargo clippy --all-targets -- -D warnings: exit 0; Finished dev profile in 68m 12s
- cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings: exit 0; Finished dev profile in 49m 45s
Source HEAD/tree and clean status matched before and after both commands.
No compiler or stack overrides were used. protoc 35.1 was installed from its
verified ARM64 release; flatc was absent and not invoked by the workspace build scripts.
The local environment differs from hosted CI.

* fix(ecstore): recover delete batches with missing versions

Treat an explicit non-null version as a no-op when it is absent from both recovery sources. Keep rejecting missing null versions, delete-marker mismatches and conflicting local generations.

Add returning-disk and partial-receipt regressions, extend journal boundary checks, and regenerate the ecstore test-count baseline. Production faults exercise the 24 added assertion sites and 7 semantic checks: 33 faults failed at the expected sites and passed after full restoration.

Validation (LANG=en_US.UTF-8, one build job):

cargo nextest run --locked -p rustfs-ecstore --lib -E 'test(delete_recovery) | test(quorum_delete_cleanup_coordinator_binds_single_and_batch) | test(crash_delete_retry_batch_reclaims_inline_and_reserved_external)' --test-threads 1 --no-fail-fast
Summary [  34.580s] 69 tests run: 69 passed, 6058 skipped; exit 0

cargo clippy --locked -p rustfs-ecstore --all-targets -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 5m 37s; exit 0

All 19 final Quick Checks completed with exit 0:

- shellcheck --version && actionlint
- cargo fmt --all --check
- ./scripts/check_unsafe_code_allowances.sh
- ./scripts/check_layer_dependencies.sh
- ./scripts/check_architecture_migration_rules.sh
- ./scripts/check_logging_guardrails.sh
- ./scripts/check_error_other_format_ratchet.sh
- ./scripts/check_no_tokio_io_uring.sh
- ./scripts/check_extension_schema_boundaries.sh
- ./scripts/check_body_cache_whitelist.sh
- ./scripts/check_s3s_footprint.sh
- ./scripts/check_test_count_ratchet.sh
- ./scripts/check_fips_wording.sh
- ./scripts/check_embedded_secrets.sh
- make script-tests
- python3 ./scripts/test_object_generation_protocol_model.py
- python3 ./scripts/check_test_wiring.py
- ./scripts/check_no_planning_docs.sh
- ./scripts/check_uring_lane_lib_only.sh

test-count ratchet OK: 52 crates, 22009 tests, 165 ignored match .config/test-count-baseline.txt
PASS: cold stampede script self-test
model: states=51023, transitions=167520, terminal=732, chosen_candidates=2
multi-slot round schedules: 640
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired

* fix(ecstore): retry tier commit intent lock contention

Retry transient namespace lock timeouts within the existing three-attempt
CAS budget. Reload and verify the original mutation identity after
contention, and preserve the final typed error when attempts are exhausted.
Cover save/read contention, exhausted retries, replacement identities and
CAS reconciliation without changing the existing timeout or lock policy.

Validation:
- cargo nextest run --locked -p rustfs-ecstore --features test-util --lib -E 'test(tier_mutation)' --test-threads 1 --no-fail-fast: 52 passed.
- cargo nextest run --locked -p rustfs-ecstore --features test-util --lib -E 'test(intent_advance_) | test(=services::tier::tier::tests::caller_cancellation_after_durable_prepare_does_not_hide_the_mutation)' --test-threads 1 --no-fail-fast: 5 passed.
- cargo clippy --locked -p rustfs-ecstore --all-targets -- -D warnings: passed.
- cargo clippy --locked -p rustfs-ecstore --all-targets --features test-util -- -D warnings: passed.
- 27 mutation pairs failed at the intended assertions and passed after restoring the production code.
- bash scripts/check_test_count_ratchet.sh --update, then bash scripts/check_test_count_ratchet.sh: 52 crates, 22013 tests, 165 ignored; ecstore increases from 6581 to 6585.

* refactor(lifecycle): replace s3s domain types with Gateway DTOs

Keep lifecycle rules, restore requests, Object Lock, and scanner cached tags in domain types. Convert at the existing HTTP boundary while preserving legacy XML, scanner cache bytes, date ordering, and tombstones.

Linux test validation before the final import-path-only adjustment:

cargo nextest run -p rustfs-ecstore bucket_config_goldens
39 passed; 6211 skipped by selector.

cargo nextest run --lib -p rustfs -E 'test(/test_zero_rule_lifecycle_tombstone_recognition/) | test(/test_local_lifecycle_staleness_axis_selection/) | test(/test_lifecycle_staleness_preserves_legacy_submillisecond_axis/) | test(/test_merge_incoming_lifecycle/) | test(/test_lifecycle_expiry_statement_matrix/) | test(/test_lifecycle_expiry_subset_xml/) | test(/put_bucket_lifecycle_validation_errors_keep_their_s3_code/) | test(/execute_put_bucket_lifecycle_configuration_rejects_missing_configuration/) | test(/execute_restore_object_rejects_missing_restore_request/) | test(/execute_restore_object_rejects_select_type/) | test(/execute_restore_object_maps_failures_to_typed_s3_errors/) | test(/execute_restore_object_returns_internal_error_when_store_uninitialized/) | test(/restore_request_conversion_preserves_legacy_output_values/) | test(/restore_request_conversion_preserves_presence_and_raw_enums/) | test(/restore_request_conversion_distinguishes_absent_and_empty_containers/)'
25 passed; 5318 skipped by selector.

cargo nextest run --profile ci -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-replication -p rustfs-scanner
7872 passed; 21 skipped under the existing CI profile.

Previously verified mutation controls: 39 production faults, 41 RED/GREEN pairs.

cargo clippy --locked -p rustfs -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-scanner --all-targets -- -D warnings
Passed in 40m 45s before the import-path adjustment, then in 1m 59s on
the final source with warnings denied. The final adjustment imports the
existing helper through the object domain facade; fmt and the architecture
check also passed.

Part of rustfs/backlog#2745. The remaining ecstore configuration families follow separately.

* refactor(admin): move error types to rustfs-s3-types (#8440)

* refactor(protocols): move error types to rustfs-s3-types by codemod

T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and
the error items of protocols move off s3s by scripts/codemods, run with
--hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it
downcasts the backend error, whose runtime type is still s3s::S3Error.

- rustfs/src/protocols/client.rs: the two webdav session methods now
  return rustfs_s3_types::S3Result and convert the s3s error with From.
- codemod: a child module that calls s3_error! through `use super::*`
  gets its own `use rustfs_s3_types::s3_error;` once its parent imports
  that macro by name; otherwise it is listed as GLOB.
- s3-types: a table test pins code, status and the s3s error the legacy
  edge renders for every variant a moved call site names.
- s3_error_lines 1585 -> 1584 (a lifecycle comment reworded).

* refactor(admin): move error types to rustfs-s3-types (WIP, no build)

T1.2 batch 4 (rustfs/backlog#2743), work in progress: rustfs check
still fails with 16 errors (listed in the PR body).

- codemod: --facade PATH treats a crate-local re-export of s3s's error
  items like s3s for the four names (9 new self-tests, 9 mutants red).
- codemod output for rustfs/src/admin and rustfs/src/site_replication
  (--facade crate::storage_api::site_replication::s3); 6 glob children
  get an explicit rustfs_s3_types::s3_error import.
- admin/router.rs: the Operation trait returns rustfs_s3_types::S3Result.
- error.rs: From<ApiError> builds the rustfs-s3-types error; the s3s
  conversion is derived from it.

* fix(admin): convert error types at the s3s boundary

Batch 4 of T1.2 left the admin and site_replication modules returning
rustfs_s3_types::S3Error while their callers and callees on the storage
and auth side still speak s3s::S3Error. Convert at each edge with the
existing From impls in rustfs-s3-types (compat_s3s.rs) instead of
changing any code, message or status:

- admin/auth.rs, admin/handlers/object_zip_download.rs,
  admin/handlers/table_catalog/mod.rs: map the s3s result returned by
  check_key_valid / authorize_request / authorize_internal_object_request
  into the admin error type with .map_err(Into::into).
- storage/ecfs.rs: the s3s-facing side maps the admin site_replication
  gate results the other way.
- admin/handlers/pools.rs: ApiError.code stays s3s::S3ErrorCode, so the
  rustfs code is converted with .into() when the struct is built, in
  production and in the two tests that build or compare it.
- admin/handlers/heal.rs, admin/runtime_sources.rs: the router returns
  s3s errors (S3Route impl); the expected rustfs code is converted with
  .into() before the comparison.
- admin/handlers/table_catalog/mod.rs: iceberg_rest_error takes
  &'static str so the Cow<'static, str> custom code borrows it; every
  caller already passes an ICEBERG_ERROR_* constant or a &'static str.
- admin/handlers/quota.rs: drop a now same-type .into() after s3_error!.
- storage_api.rs, site_replication/mod.rs: drop re-exports and an import
  that no longer have users (clippy -D warnings).

cargo check -p rustfs --all-targets and cargo clippy -p rustfs
--all-targets -- -D warnings are clean; the s3s footprint baseline is
unchanged at files=172, s3_error_lines=303, ecstore_files=28.

Refs rustfs/backlog#2743, rustfs/backlog#2734.

* refactor(ecstore): move bucket configs onto gateway persisted types (#8709)

* refactor(ecstore): move bucket configs onto gateway persisted types

Finish the ECStore batch of T1.4 (rustfs/backlog#2745). The remaining
eleven bucket configuration families (Website, CORS, Tagging, Versioning,
ObjectLock, Encryption, Notification, Logging, Accelerate, RequestPayment,
PublicAccessBlock) now read and write their stored XML through the
rustfs-gateway-types persistence and cors_tagging codecs instead of the
s3s codec, so the legacy s3s bucket-config codec and its
s3gate-metadata-xml compat marker are removed.

- ecstore: BucketConfigXml is implemented by the gateway codec for every
  family; metadata, metadata_sys, versioning, object lock, replication and
  lifecycle consumers take the persisted types. Region moves to
  rustfs_s3_types::Region, and the ETag conversion to s3s moves to the
  rustfs edge with its tests. Only warm_backend.rs and warm_backend_s3.rs
  still reference s3s, held for T1.5 (rustfs/backlog#2746).
- scanner: object lock cache, item actions and data usage use the
  persisted types.
- rustfs: the legacy S3 edge converts each family between s3s DTOs and the
  persisted types in storage/storage_api.rs by writing one codec's XML and
  reading it with the other; Tagging converts field by field.
- Bump the rustfs-gateway-types pin to 21283e87.

The 39 persisted-byte goldens are unchanged and pass, so a document the
old codec wrote reads back and re-encodes to the same bytes. s3s footprint:
files 181 -> 146, ecstore_files 28 -> 2, s3_error_lines unchanged.

* test(app): prepare the PutBucketEncryption config from the request

The uninitialized-store test passed an empty persisted configuration, which the use case now validates before it reaches the store check, so it answered MalformedXML instead of InternalError. Convert the request's configuration the way the S3 edge does.

* style(app): format the PutBucketEncryption test

* refactor(s3-client): move remote error codes and DTO enums off s3s (#8783)

Part of T1.5 (rustfs/backlog#2746), batch 1.

* refactor(s3select): run queries on gateway DTO request shapes (#8794)

* refactor(s3select): run queries on an engine request built from gateway DTO shapes

Part of T1.5 (rustfs/backlog#2746), batch 2.

* fix(s3select): keep the select request conversion in the app layer

The layer guard forbids app -> storage; the conversion moves from storage/s3_compat/select.rs to app/select_request.rs unchanged.

* fix(heal): retry transient metadata conflicts (#8812)

* fix(heal): retry transient metadata conflicts

* chore: update heal test count baseline

* test: reserve nextest capacity for progress probes (#8818)

* test(e2e): use a non-default region for the differential negative control

Since main's region-aware CreateBucket and GetBucketLocation (#8739, #8743), us-east-1 answers an empty LocationConstraint and 200 for an owned re-create, so the control no longer reported its own region. Also refresh the e2e-full linux selection from the CI listing (682 tests).

* refactor(protocols): move StorageBackend to gateway types (#8825)

* refactor(protocols): move storage backend to gateway types

* test: update e2e test count baseline

* fix: preserve copy source keys and unknown lengths

* test: update protocols test count baseline

* fix(protocols): preserve typed SFTP backend errors

* test(protocols): use gateway list input in policy tests

* fix(protocols): clear clippy failures after T1.5 merge (#8831)

fix(protocols): clear clippy failures after merge

* fix(ecstore): finalize restore metadata after local copy-back (#8836)

* fix(ecstore): finalize restore metadata after local copy-back

* test(ecstore): ratchet added restore finalization cases

* fix(ecstore): keep restore copyback compatible with older peers

* fix(ecstore): gate restore copyback on fleet compatibility

* fix(ecstore): retain transition identity during restore copyback

* fix(ecstore): route committed restore copyback locally

* fix restore copyback cleanup

* fix(ecstore): honor restore probe test fallback

* refactor(storage): migrate option errors to domain types (#8839)

* refactor(s3): migrate private multipart validation errors (#8844)

* refactor(auth): derive policy conditions from AuthzFacts (#8442)

* refactor(auth): extract authorization facts

* test(e2e): update smoke selection baseline

* test(e2e): update full selection baseline

* refactor(admin): migrate health and prefix usage to owned requests (#8887)

* refactor(admin): add owned health router boundary

* refactor(admin): migrate prefix usage to owned request

* refactor(admin): complete the owned admin request boundary (#8949)

* refactor(admin): complete the owned admin request boundary

Every type implementing the owned admin `Operation` now serves the legacy
router through one blanket impl that converts the s3s request and response
at that single edge, so handlers register with `AdminOperation(&Handler {})`
and need no adapter statics; `LegacyAdminAdapter` is removed.

`AdminRequest` gains the accessors handlers read today (`remote_addr`,
`client_info`, `request_context`, `req_info`, `req_info_mut`,
`install_req_info`, `envelope_mut`, `take_body`), the request helpers gain
envelope forms (`authenticate_envelope_credentials`,
`object_store_from_envelope`, the table-catalog `*_from_envelope` helpers,
`KmsAdminAudit::from_envelope`, `AccountAuditContext::from_envelope`,
`CallerIdentity::resolve_envelope`), `read_compatible_admin_body` accepts
either body type, and the owned `Body` gains `store_all_limited` and
`Default`. `admin::operation::test_support::TestRequest` builds owned
requests for handler tests. Unused accessors carry a per-method dead-code
expectation that each migrating handler batch removes.

No handler behavior changes; the remaining migrations follow in batches.

* chore(admin): add the T1.12 admin handler codemod

The script rewrites an admin handler file from the s3s request and response
types to the owned boundary (imports, signatures, request field and helper
calls, body access, response construction, and test request literals) and
leaves a TODO comment at every site it cannot rewrite mechanically. It is
idempotent and is removed once T1.12 completes.

* refactor(admin): migrate replication and KMS handlers to owned requests (#8951)

Move the T1.12 batch 2 admin handlers (site_replication, kms_keys, event,
replication, diagnostics, on_demand_migration, audit, account, kms_rekey,
kms_key_lifecycle, object_data_cache, tls_debug, integrity) from the s3s
request/response types to the owned AdminRequest/AdminResponse boundary.
Handler logic, authorization order, error codes and response bytes are
unchanged; the request/response plumbing now goes through the envelope
accessors and helpers introduced by the owned boundary.

The shared response helpers (utils::json_response, utils::empty_response,
handlers::admin_json_response, target_descriptor::build_json_response,
profile::authorize_profile_request) and the site replication repair
executor still answer with the s3s types, so the migrated files carry
owned-boundary forms of those helpers and convert the repair response at
its call edge. The streaming console-log body and the two devnull drains
move from the s3s byte stream to the owned Body stream/frame API.

Drop the dead_code expectations for the owned accessors and envelope
helpers this batch starts using, and lower the s3s footprint baseline to
the measured 99 files.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* refactor(admin): add owned forms of the shared admin response helpers

Handler batches were copying `json_response`, `empty_response`,
`admin_json_response`, and `build_json_response` into each migrated file
because the shared helpers still return the s3s response. Provide owned
forms next to each legacy helper (`owned_json_response`,
`owned_empty_response`, `owned_admin_json_response`,
`owned_build_json_response`, `authorize_profile_envelope`) with parity
tests against the legacy shape, derive `Debug` for `AdminResponse` so
tests can use `expect_err`, and fix three codemod rules: strip owned
string conversions in test credentials, annotate non-literal URI parses,
and leave a rebound request parameter immutable.

* refactor(admin): use the shared owned response helpers in batch 2 (#8954)

Replace the per-file owned-boundary copies of json_response,
empty_response, admin_json_response, build_json_response and
authorize_profile_request in the batch 2 admin handlers with the shared
owned_* helpers and authorize_profile_envelope that the boundary branch
now provides, and restore expect_err in the handler tests now that
AdminResponse implements Debug; the `.err().expect(..)` stand-ins were
failing the base branch's clippy step (clippy::err_expect under
-D warnings). The site replication repair call edge
keeps its conversion because the repair executor still answers with the
s3s response type.

Drop the dead_code expectations on the shared owned helpers this batch
now uses in non-test code.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* fix(admin): preserve legacy body limit error responses

* refactor(admin): move policy, ILM and batch 3 handlers to owned requests (#8956)

Batch 3 of T1.12 (rustfs/backlog#2788, parent rustfs/backlog#2734) moves
14 admin handler files to AdminRequest/AdminResponse and the owned
Operation trait: ilm_transition, policies, idp_compat, config_admin,
plugins_instances, profile_admin, group, heal, object_zip_download,
kms_key_metadata, durability, inspect_archive, sts and
gateway_key_inventory. Handler logic, authorization order, error codes
and response bytes are unchanged; test request literals use
test_support::TestRequest.

The handlers use the shared owned helpers (owned_json_response,
owned_admin_json_response, owned_build_json_response,
authorize_profile_envelope). The dead_code expectations on the
AdminRequest accessors and RequestEnvelope::uri_mut they now exercise
are removed. The legacy target_descriptor::build_json_response keeps
only its equivalence test caller and carries the matching expectation
until it retires.

object_zip_download builds the s3s request the S3 gate still reads from
the owned request facts; heal authenticates through
authenticate_envelope_credentials; the trace, inspect-archive and zip
bodies use Body::from_stream. extract_plugin_instance_filters takes the
URI so extensions.rs keeps calling it, and the two idp_compat
delegations stay on the legacy trait until their delegates migrate.

The s3s footprint baseline drops from 99 to 90 files.

* test(admin): anchor the replication diff body contract on the call

The source-contract test looked for `let body = read_compatible_admin_body(`
on one line; rustfmt wrapped that binding after the handler moved to the
owned request boundary, so the marker was no longer found. Anchor on the
call itself so formatting cannot change what the contract checks.

* refactor(admin): migrate table catalog and batch 1 to owned requests (#8955)

* fix(admin): preserve catalog body limit response bytes

* refactor(admin): migrate identity and cluster handlers to owned requests (#8966)

* refactor(admin): migrate identity and cluster handlers to owned requests

Move the T1.12 batch 4 admin handlers (user, service_account, oidc, tier,
system, pools, quota, kms_dynamic, kms_backup, rebalance, module_switch,
bucket_meta, cluster_snapshot, realtime, bucket_recovery, profile) from the
s3s request/response types to the owned AdminRequest/AdminResponse
boundary. All 90 impl Operation blocks now take AdminRequest and return
AdminResponse; handler logic, authorization order, error codes and response
bytes are unchanged. The handlers use the shared owned helpers
(owned_json_response, owned_admin_json_response, authorize_profile_envelope,
the envelope-form runtime sources and audit constructors) instead of local
copies.

profile.rs drops the s3s forms of authorize_profile_request and
profile_not_implemented_response now that profile_admin.rs is on the
envelope gate too; profile_not_implemented_response answers with an
AdminResponse. The realtime metrics stream moves from the s3s ByteStream
adapter to Body::from_stream, and the rebalance idle-404 document still
renders through s3s::S3Error::to_http_response with its body converted at
the edge.

The idp_compat delegators ImportIamV2 and AddServiceAccountLdap move to the
owned trait as well: with their targets on both traits, the method-call
delegation became ambiguous between the owned impl and the router blanket
impl, so they now take the owned request directly.

KmsAdminAudit::from_request loses its last production caller with this batch
and is deleted; its comparison test now asserts the envelope facts directly.

Lower the s3s footprint baseline to the measured 68 files.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* refactor(admin): remove the legacy admin helpers batch 4 superseded

Batch 4 moved the last production callers of the s3s-typed admin helpers
to their owned forms, which left `admin_json_response`, `json_response`,
`authenticate_request`, `authorize_admin_request`, `object_store_from_req`
and `object_store_from_extensions` unused, along with the `Body`,
`S3Request` and `S3Response` re-exports of the admin `s3` facade. Remove
them. The owned forms keep their documentation; the behavior tests of the
removed `json_response` now exercise `owned_json_response`, and the
compat-prefix test asserts the owned response directly.

Two tests existed only to compare a removed legacy helper with its owned
twin (`owned_json_response_matches_the_legacy_shape`) or to cover the
removed legacy gate (`authorize_admin_request_without_credentials_is_rejected`);
they go with the helpers, and the owned equivalents stay. The rustfs
test-count row drops from 5957 to 5955 and the s3s footprint from 68 to 66
files.

* refactor(admin): drop the legacy target descriptor JSON response helper (#8968)

`target_descriptor::build_json_response` returned the s3s response and
had no production caller left once the event, audit and plugin instance
handlers moved to `owned_build_json_response`; it survived only for the
legacy/owned equivalence test. Remove it, give the owned helper its own
documentation, and make the test assert the owned response directly:
status, `application/json`, the echoed `x-request-id` when present and
only then, and the body bytes. The file no longer imports s3s, so the
footprint baseline drops from 66 to 65 files.

* test(ecstore): pin native history on cancellation under the cancellable queue

main moved the local write path onto the disk blocking queue, which skips an
operation once its waiter is dropped, so a borrowed write cancelled while it is
queued no longer creates its file. The integration-only native publication
test still expected the queued open to run. Assert the new execution model
instead (the open never runs) while keeping its safety claims: the Started
record is taken before queueing and the cancelled publication never regrants
native history.

* chore(gateway): raise test-count rows for tests merged from main

* test(s3select): use the owned JSON input types in merged planner tests

* fix(ecstore): scope the merged direct-writer test hook to test builds

* fix(ecstore): create parents before admitted truncate opens

Align the heal tests with removal-disabled preservation and local-only outcome
fields, and refresh the e2e-full selection for the synced tests.

* chore(gateway): update test-count baseline after main sync

Main #9241 removed the node pool-metadata readiness cache with 8 tests
and added 6, so the rustfs row drops from 6440 to 6438. The ecstore and
s3select rows rise for tests added by #9237, #9239 and #9241.

* feat(scripts): add junit_diff.py for dual-stack JUnit comparison (#9275)

* ci(gateway): make the test-count ratchet a floor, not exact (#9278)

ci(gateway): make the test-count ratchet a floor instead of an exact baseline

Tests below a row still fail; tests above it, a member without a row and ignores below the row now pass with a note. Ignores above a row still fail. Adds contract tests wired into make script-tests.

* test(guards): read file plus split directory in text guards

Text guards that read one file with include_str! or by path broke, or silently stopped seeing the code they guard, when that code moved to another file. They now read the file they read today, which stays required, plus the sibling directory a mechanical split will populate (storage/legacy_edge, app/bucket_usecase, app/multipart_usecase, app/lifecycle_transition_api_test). Counts are summed over the set and split markers must occur exactly once across it. Refs rustfs/backlog#2765.

* chore(gateway): record ignored microbenchmarks merged from main (#9290)

chore(gateway): record the ignored microbenchmarks merged from main

main added three throughput microbenchmarks that are ignored by design
(#9217): one each in rustfs-checksums, rustfs-rio and rustfs-utils. The
test-count ratchet treats a higher ignored count as a failure until the
row is raised, so raise those three rows to 1 and tighten the test rows
to the merged tree with --update.

* test(guards): avoid a typos false positive in a fixture

* ci(s3-tests): add reference parity lane against pinned 1.0.5 (#9288)

* build(codemods): add split_ecfs_legacy_edge codemod

Moves the impl S3 for FS method bodies out of rustfs/src/storage/ecfs.rs
into one file per migration batch under rustfs/src/storage/legacy_edge/
(rustfs/backlog#2765, preparatory task P1). The codemod refuses an input
it does not expect, such as a method without a batch assignment or an
already split tree. Its self-test runs from make script-tests.

* refactor(storage): move impl S3 for FS bodies into legacy_edge files

Output of scripts/codemods/split_ecfs_legacy_edge.py on the unsplit
ecfs.rs and ecfs_test.rs. Every trait method keeps its signature in
ecfs.rs and calls a FS::<method>_edge fn in the file of its batch; the
bodies, their single-batch helpers and their request-building tests move
unchanged. ListBuckets stays inline: the T2.5 skeleton owns it.

* test(guards): address review of the split-safe text guards

Make the super:: import rules depth-aware for split files, cover ecfs_test.rs moving to legacy_edge/tests, read production text only in the Rust count and contains guards, bound method slices by their own closing line, keep the split arrays safe on bash 3.2, and add an end-to-end split probe for the architecture, layer and logging guards. Refs rustfs/backlog#2765.

* fix(codemods): keep split imports that rustc credits elsewhere out

Two import decisions of split_ecfs_legacy_edge were wrong on the first CI
run: a fn-local use counted as a use of the module-level import of the
same name, and a batch file imported a trait that ecfs.rs still imports,
which rustc reports as unused because the ecfs glob already supplies it.
Both now have a regression case in the self-test.

* refactor(storage): regenerate the legacy_edge split

Output of the fixed codemod: ecfs_test.rs drops its now unused Method
import and object_meta.rs takes ObjectOperations from the ecfs glob.

* refactor(storage): regenerate ecfs_test.rs after merging main

The textual merge of the already split ecfs_test.rs kept persisted_lock
twice. Re-running split_ecfs_legacy_edge on the unsplit inputs (ecfs.rs
from the guards branch, ecfs_test.rs from main) yields the tree without
the duplicate; every other file regenerated byte-identical.

* fix(codemods): refuse an import no scanned code uses

The input compiles warning-free, so an import that neither the code left
in a module nor the moved code uses means the scan missed a use, most
likely a trait missing from TRAIT_METHODS. The codemod now fails instead
of dropping the import and leaving the error to the compiler.

* fix(guards): resolve attributed modules in test seam checks

---------

Co-authored-by: rustfs-cla-assistant[bot] <273121464+rustfs-cla-assistant[bot]@users.noreply.github.com>
2026-11-12 09:11:12 -05:00
GatewayJ 633c0d204a fix(s3-tables): reclaim dropped table commits and maintenance reports (#9351) 2026-11-12 09:10:58 -05:00
GatewayJ da6058e37b feat(storage): support per-instance internode data transports (#9257)
* feat(storage): support per-instance internode data transports

* perf(rpc): avoid staging and client cache overhead on TCP writes
2026-11-12 08:15:48 +00:00
Hauser Zhi 923739a00d chore(deps): update workspace dependencies (#9348)
Run cargo update --verbose and cargo upgrade --verbose twice from main.
Refresh dependency requirements and the workspace lockfile.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-12 07:29:50 +00:00
GatewayJ 06eca12064 fix(docker): preserve local pool arguments during expansion (#9352)
* fix(docker): preserve local pool arguments during expansion

* refactor(docker): use one path expansion for volume arguments
2026-11-12 18:02:16 +08:00
唐小鸭 7f90eb7ae8 fix(replication): seal SSE-KMS replicas under the same-named key (#9349)
* fix(replication): replicate SSE-KMS objects under key enforcement

With RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY=true the receiving site authorizes a replica PUT of an SSE-KMS object for kms:GenerateDataKey against the key it resolves to. The managed site-replicator account held no kms grant, so every SSE-KMS object stayed FAILED with AccessDenied.

Grant kms:GenerateDataKey on every key and treat a stored policy without it as outdated so the reconcile tick refreshes existing accounts. kms:Decrypt stays out: reads proxied to a peer run as this account.

* fix(replication): seal SSE-KMS replicas under the same-named key

A replica write carried only the aws:kms intent, so a site-replication
peer sealed the replica under its bucket default or site default key. An
object written under a key named in the request therefore ended up under
a different key on each site, and with key enforcement a caller scoped
to the named key could read it on one site and not on the other.

Site-replication peers are required to hold every KMS key under the same
name, so the replica write to a peer now names the source object's key.
Operator-configured bucket-replication targets still get the intent
only: their KMS has no reason to hold a key of that name.
2026-11-12 17:05:15 +08:00
Ramakrishna Chilaka 15636e97e2 perf(ecstore): parallelize shard hashing and coalesce local shard writes (#9346)
- Add EncodedBlock::shards_bytes to yield zero-copy Bytes slices without
  copying shards on the async request task.
- Add DiskFileWriter and CustomWriter::from_file_writer to avoid
  double-boxing FileWriter on PUT and UploadPart, and downcast local
  writers through SlowDiskWriter.
- Offload HighwayHash256S bitrot hashing and PartIntegrity::shard_digest
  SHA-256 computation into each disk's LazyLocalFileWriter blocking task
  and write [hash, shard] with EINTR-safe write_vectored.
- Reuse precomputed shard digests in IntegrityBuilder::push_with_precomputed
  and add a single-join_all zero-copy Spool::Memory fast path in
  PreparedIntegrity::write.
2026-11-12 15:09:07 +08:00
GatewayJ 78f02c196d fix(sts): validate root credentials and report parent access denials (#9295)
* fix(sts): validate root credentials and report parent access denials

* test(sts): use the existing root fixture at its minimum length

---------

Co-authored-by: Chris An <anzhengchao@gmail.com>
2026-11-11 21:07:40 -05:00
cxymds e6b2a576e3 feat(quota): report peer metadata reload acknowledgements (#9330)
Co-authored-by: Chris An <anzhengchao@gmail.com>
2026-11-11 21:07:11 -05:00
Ramakrishna Chilaka 7f3e8874a9 perf(ecstore): reduce syscalls in overwrite PUT, UploadPart, and multipart commit (#9333)
- Pass skip_parent instead of dst_volume_dir_owned when renaming the inline
  overwrite rollback backup in fused_attempt, and reuse open ancestor directory
  descriptors via fsync_commit_ancestor_directory_limited_std in fused multipart
  commit.
- Pop the pinned parent descriptor directly in
  rename_all_with_precondition_and_parent_sync_std instead of cloning it with
  fcntl(F_DUPFD_CLOEXEC), resolve open_existing_directory_below_base_std from
  publication_root in a single openat2 syscall on Linux, and probe existing
  multi-component parents in prepare_rename_with_retry.
- Avoid redundant attempt-root directory creation in
  prepare_multipart_commit_staging when hardlink_multipart_source_files creates
  the staging tree, walk owned backup and certificate directories relative to
  source while consuming their open descriptors across replacement and parent
  fsync barriers, and open existing retirement-anchor staging directories
  directly.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: RustFS <hello@rustfs.com>
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
Copilot-Session: 21d8304f-0f49-48eb-8c28-26f3689b081b
2026-11-11 21:05:29 -05:00
Chris An 5031c78fe1 refactor(app): split multipart_usecase.rs per operation (P3) (#9312)
* feat(server): add the RUSTFS_S3_STACK selector and dual-stack e2e lane (#8411)

* refactor(server): group the s3s-compat tower layers under legacy_compat (#8410)

Move the ten tower layers that patch s3s behaviour out of
rustfs/src/server/layer.rs into rustfs/src/server/legacy_compat/, one file
per layer with its tests. Each file header names the s3s behaviour it
patches, the gateway mechanism that replaces it (Replaced by:), and the task
that removes it (Removed by: T2.12, rustfs/backlog#2771). The layers keep
their code, their order in external_service_stack!, and their behaviour; no
layer is deleted.

Add server::http::tests::stack_census, which renders the ordered service
chain of the real external_service_stack! expansion and compares it with a
committed constant, so a reorder, insertion or removal is a visible diff.

Route the two s3s error types the moved layers use through a
storage_api::server::legacy_compat shim so the split adds no s3s-importing
file, and lower the s3s footprint baselines to the measured values (files 209
to 208, s3_error! lines 1586 to 1585). Keep the server runtime-source bypass
guard covering the moved code.

Refs rustfs/backlog#2787, rustfs/backlog#2734

* feat(s3-types): add the RustFS-owned S3 error type and s3_error! macro (#8406)

`rustfs-s3-types` now provides `S3ErrorCode` (generated by build.rs from
src/codes.txt, the grep-derived call surface, plus `Custom`), `S3Error`,
`S3Result`, `StdError` and the `s3_error` macro with the same call syntax
as s3s, so the T1.2 codemod is a `use` swap. Statuses are hand-written and
pinned to the s3s behaviour RustFS ships today by an oracle test that uses
s3s as a dev-dependency only. The `From` conversions for the legacy edge
live behind the off-by-default `compat-s3s` feature (the orphan rule
forbids them in rustfs/src/error.rs); `rustfs` enables it.

Refs rustfs/backlog#2742, rustfs/backlog#2734

* ci: add the per-crate test-count ratchet (#8404)

Quick Checks now fails any PR that lowers the number of test attributes
in a workspace crate or adds #[ignore], measured per crate against the
exact rows of .config/test-count-baseline.txt (rustfs/backlog#2734
section 1 rule 3). The counter widens the spec's literal
#[(tokio::)?test] to the tokio attributes that carry arguments and both
spellings of #[test_case(...)], and anchors at line start so
commented-out attributes do not count. The initial baseline covers 52
crates, 21077 tests and 164 ignored.

Refs rustfs/backlog#2738
Refs rustfs/backlog#2734

* test(ecstore): pin the persisted bucket-config XML bytes for 13 families (#8408)

Before any DTO or codec is replaced, pin the exact bytes that
deserialize followed by serialize produces for every persisted
bucket-configuration XML family in its three source shapes (AWS
documentation, MinIO-written, RustFS-written), and the bytes RustFS
writes today for a typed value. Byte equality only: a structural
comparison would let a codec replacement change the persisted encoding
unnoticed.

The generator script refuses to overwrite existing goldens without
--force, refuses on dirty codec paths, cannot report success from its
write mode, and records the generating commit in the fixtures README.

Refs rustfs/backlog#2744, rustfs/backlog#2734

* ci: raise the test-count baseline to the integration tree (#8426)

* ci: make the s3s footprint baseline exact and lower-only (#8403)

* ci: make the s3s footprint baseline exact and lower-only

The s3s footprint guard compared three counters against ceilings held in
the script itself, so a ceiling could sit above the tree and a drift was
fixable by raising it (rustfs/rustfs#5964 raised the s3_error! ceiling
to go green). The three values now live in
.config/s3s-footprint-baseline.txt as files, s3_error_lines and
ecstore_files, the script reads only that file, and a measurement that
differs from its baseline in either direction fails: growth prints
"s3s footprint grew: <measured> > <baseline>", shrinkage prints
"lower the baseline to <measured> in this PR". The counting method is
unchanged; the baseline is the current measurement of
gateway/integration (209 / 1585 / 36). The header records the end state:
once all three reach 0, T4.2 switches the script to failing on any
occurrence.

Refs rustfs/backlog#2739, rustfs/backlog#2734

* ci: add the s3s edge allowlist mode to the footprint guard (#8405)

`scripts/check_s3s_footprint.sh --mode allowlist` fails on every Rust
file that references s3s (`s3s::` or `s3s_sigv4`) and is not matched by
an entry of the new `.config/s3s-edge-allowlist.txt`; `--dry-run` lists
those files and exits 0. The allowlist carries the initial edge from the
spec, and the script rejects an entry that names or matches anything
under crates/ other than crates/e2e_test. The baseline mode and the
argument-less CI invocation are unchanged.

Refs rustfs/backlog#2741, rustfs/backlog#2734

* test(ecstore): reach the bucket-config codec through the test facade (#8428)

* refactor(s3-types): own the HTTP Body and drop s3s from three crates (#8431)

Add rustfs_s3_types::Body, a BoxBody<Bytes, StdError> wrapper with
empty() and From<Bytes | Vec<u8> | String | &'static str>, and the
legacy From conversions to and from s3s::Body behind compat-s3s.

The signer is now generic over the request body: it only reads and
writes the request head, so it needs no body type and its callers keep
theirs. s3-client sends its requests with the owned Body, madmin signs a
bodiless head, and targets drops its two S3Request/S3Response header
wrappers in favour of extract_params_header.

rustfs-signer, rustfs-madmin and rustfs-targets no longer depend on
s3s; the s3s footprint baseline drops from 209 to 202 files.

Refs rustfs/backlog#2747, rustfs/backlog#2734

* refactor(ecstore): move S3ErrorCode to rustfs-s3-types by codemod (#8432)

* test(ecstore): pin the InvalidRange responses of new_getobjectreader

Both error branches of new_getobjectreader build an ErrorResponse with S3ErrorCode::InvalidRange, and no test observed either code. Pin the code, its wire name and the location fields of each branch before the code type moves to rustfs-s3-types (rustfs/backlog#2743), so a changed code on either branch fails a test. The ecstore test-count row rises by the two tests.

* chore(scripts): add the s3s-to-rustfs-s3-types error codemod

Task T1.2 of rustfs/backlog#2734 (rustfs/backlog#2743) moves S3Error, S3ErrorCode, S3Result and s3_error! from s3s to rustfs-s3-types by script, one crate closure per PR. The codemod rewrites use trees (splitting mixed s3s imports) and qualified code paths, never comments or literals, adds the rustfs-s3-types dependency, drops s3s only from a crate that no longer names it, refuses every file of the s3s edge allowlist, takes --hold for a file whose type is fixed outside the closure, and prints a per-crate census. A second run changes nothing. Its self-test runs under make script-tests.

* refactor(ecstore): take S3ErrorCode from rustfs-s3-types

Output of scripts/codemods/s3_error_to_s3_types.py over crates/ecstore, crates/lifecycle, crates/scanner and crates/replication (rustfs/backlog#2743 batches 1 and 2). The ErrorResponse code field, the two InvalidRange constructions in new_getobjectreader and an unused tier import now use rustfs_s3_types::S3ErrorCode; the variant names and wire names are unchanged. lifecycle, scanner and replication reference none of the four error names. warm_backend.rs and warm_backend_s3.rs are held: they compare against rustfs_s3_client::ErrorResponse.code, which stays the s3s type until s3-client moves.

* ci(footprint): count only s3_error! lines still on s3s's macro

s3_error_lines counted every line holding the text `s3_error!`, which
also matches the RustFS-owned `rustfs_s3_types::s3_error!`. A codemod
that moves call sites onto rustfs-s3-types therefore could not lower it
(rustfs/backlog#2743, decision 3; rustfs/rustfs#8432, gotcha 2).

The counter now applies a textual, per-file and per-line rule that errs
on the side of counting (full rule in the script header):
- occurrences spelled `rustfs_s3_types::s3_error!` never count;
- a line holding `s3s::s3_error!` always counts;
- any other `s3_error!` line counts unless its file is cleared, that is,
  it imports the macro from rustfs_s3_types by name on a line starting
  with the use item (or defines it with `macro_rules! s3_error`) and does
  not also bring s3s's macro in (`use s3s::s3_error`, a `use s3s::{..}`
  tree naming it or a crate-root `*`, `use s3s::*`, or
  `#[macro_use] extern crate s3s`);
- fail closed: a file matching neither form (`use super::*`, a
  crate-local re-export, a renamed or glob rustfs_s3_types import, a
  commented-out import, a comment-only mention) counts.

Census of the 1585 lines on 96fc06776: 1478 in 64 files that import the
macro from s3s; 107 in 15 files matching neither form (105 through
`use super::*`, 1 qualified `s3s::s3_error!`, 1 comment line); none in
files that import it from rustfs_s3_types or define it. Every line stays
counted, so the baseline remains s3_error_lines=1585 (files=202 and
ecstore_files=36 unchanged). Rewriting rustfs/src/auth.rs's import to
rustfs_s3_types lowers the new counter to 1567; the old one stayed 1585.

`--self-test` (wired into `make script-tests`) copies the script into
22 fixture trees: 6 positive cases and 16 negative ones (lines that must
not count, forms that must not clear a file, and a missing baseline,
missing key or failing classification search that must fail). Each tree
has a control line that must count, and each run gets s3s text on stdin.
Seven cases were red against the old counter. 19 mutants each turn their
case red: old definition, flipped import test, dropped fail-closed
branch, dropped e2e_test exclusion, dropped s3s veto, dropped qualified
s3s rule, dropped rustfs-qualified strip, unanchored rustfs import,
renamed import accepted, nested glob taken as root glob, single-line
s3s trees only, dropped s3s glob forms, dropped macro_use form, dropped
definition clearing, ignored classification rg errors, error-line search
without its path, classification searches without their path, missing
baseline skipped, missing key skipped.

* test(e2e): compare two RUSTFS_S3_STACK servers over every S3 operation (#8435)

Add dual_stack_differential to e2e_test. It starts two rustfs processes
from the same binary, stack A on RUSTFS_S3_STACK=legacy and stack B on
RUSTFS_E2E_STACK_B (default legacy until rustfs/backlog#2734 T2.5),
sends both the same aws-sdk-s3 script covering the 73 methods of
impl S3 for FS (a normal request and a 4xx negative each), and compares
status, error code, an allowlist of response headers and a few typed
body fields. Error message text is never compared. Each run writes
<target>/dual-stack/<run>.json with compared_ops and the (op, field,
a, b) differences; any difference fails the test.

differential_detects_an_injected_difference starts B with
RUSTFS_REGION=eu-west-1 and requires GetBucketLocation to be reported.
The comparator is pure and unit-tested without a server.

Wire the module into the e2e-smoke filter and update the e2e-smoke and
e2e-full selection digests and the e2e_test test-count row.

Refs rustfs/backlog#2740

* feat(app): add the RustFS-owned RequestEnvelope and its s3s constructor (#8437)

* feat(app): add the RustFS-owned RequestEnvelope and its s3s constructor

* ci: classify the s3s edge adapters in storage/s3_compat as interface

* feat(s3-types): add Body::from_stream and move Swift off s3s::Body (#8438)

rustfs/backlog#2747 batch 3. `Body::from_stream` wraps a chunk stream of
unknown length over `StreamBody`/`BoxBody`: one data frame per chunk, the
first error ends the body, the size hint stays unknown, and dropping the
body drops the stream. Parity tests pin it to the s3s `StreamingBlob::wrap`
it replaces, at the `http_body` level and byte for byte through hyper.

The Swift handlers in rustfs-protocols build their responses as the owned
`Body`. `SwiftService` converts them into the wrapped S3 service's response
body type through `From<Body>`, so rustfs keeps serving `s3s::Body` at the
legacy edge unchanged. Request bodies stay Send-only (`UnsyncBoxBody`); the
two storage writers and the DLO/SLO segment streams get `Sync` from
`sync_wrapper::SyncStream`, which is what s3s's `http_body_unsync` mutex
provided before. The DTO and error uses of s3s in protocols are untouched
(T1.5 and T1.2).

* refactor(protocols): move error types to rustfs-s3-types by codemod (#8439)

T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and
the error items of protocols move off s3s by scripts/codemods, run with
--hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it
downcasts the backend error, whose runtime type is still s3s::S3Error.

- rustfs/src/protocols/client.rs: the two webdav session methods now
  return rustfs_s3_types::S3Result and convert the s3s error with From.
- codemod: a child module that calls s3_error! through `use super::*`
  gets its own `use rustfs_s3_types::s3_error;` once its parent imports
  that macro by name; otherwise it is listed as GLOB.
- s3-types: a table test pins code, status and the s3s error the legacy
  edge renders for every variant a moved call site names.
- s3_error_lines 1585 -> 1584 (a lifecycle comment reworded).

* test(connect): restore one-second drive request budgets

Apply the exact test-only repair already landed on main in 5972a09cc3. The filesystem probe and production CLI success test request one second and keep their existing one-second duration and parameter checks. All outcome, I/O, error, cleanup and resource-limit assertions remain.

Scope: rustfs/tests/connect_perf_drive.rs only. Patch-id: cb0a1dfc8f68a67727525adea7458e2cee7d09f8.

Validation environment: LANG=en_US.UTF-8, CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0, exclusive APFS-cloned CARGO_TARGET_DIR.

Quick Checks: every command from .github/actions/quick-checks/action.yml, with captured output and exit status below.

$ shellcheck --version && actionlint
ShellCheck - shell script analysis tool
version: 0.11.0
license: GNU General Public License, version 3
website: https://www.shellcheck.net
exit_code=0

$ cargo fmt --all --check
exit_code=0

$ ./scripts/check_unsafe_code_allowances.sh
exit_code=0

$ ./scripts/check_layer_dependencies.sh
Layer dependency guard passed (no new reverse dependencies/cycles).
exit_code=0

$ ./scripts/check_architecture_migration_rules.sh
Architecture migration rules passed.
exit_code=0

$ ./scripts/check_logging_guardrails.sh
✅ Logging guardrails check passed
exit_code=0

$ ./scripts/check_error_other_format_ratchet.sh
error(format!) ratchet passed (total call sites: 582).
exit_code=0

$ ./scripts/check_no_tokio_io_uring.sh
exit_code=0

$ ./scripts/check_extension_schema_boundaries.sh
Extension schema boundaries passed.
exit_code=0

$ ./scripts/check_body_cache_whitelist.sh
Body-cache whitelist guard passed.
exit_code=0

$ ./scripts/check_s3s_footprint.sh
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)
✅ s3s footprint baseline check passed
exit_code=0

$ ./scripts/check_test_count_ratchet.sh
test-count ratchet OK: 52 crates, 21222 tests, 164 ignored match .config/test-count-baseline.txt
exit_code=0

$ ./scripts/check_fips_wording.sh
FIPS wording guard passed (README.md and CHANGELOG.md contain no forbidden claims).
KMS wording guard passed (crates/kms claims no Vault KV2 Transit wrapping).
exit_code=0

$ ./scripts/check_embedded_secrets.sh
Embedded secret guard passed (no private key material or provider credential literal in the tree).
exit_code=0

$ make script-tests
Running script tests...
entrypoint credential policy tests passed
internode grpc A/B bench env tests passed
object batch benchmark enhanced tests passed
PASS package service script contract
hotpath warp A/B gate tests passed
hotpath warp ABBA tests passed
Scanner validation artifacts written to /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.1pFe6Kl1Yu/out
Scanner validation artifacts written to /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.1pFe6Kl1Yu/out-missing-pid
.............................
----------------------------------------------------------------------
Ran 29 tests in 181.286s

OK
{"decision":"verified","pending_gates":["G01","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G02"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-E"}
{"decision":"verified","pending_gates":["G01","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G02"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-E"}
{"decision":"verified","pending_gates":["G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G01"}
{"decision":"verified","pending_gates":["G01","G02","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G03"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-L"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G14"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G14"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G10"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P1"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P2"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P3"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G05"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G06"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-D"}
Status-and-outcome release descriptor verified: /private/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp8rwn1pe0/descriptor/release-bundle-status-outcome.json
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G05"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G06"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-D"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G07"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G08"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P4"}
exact 1MiB handoff ABBA tests passed
pinned paired ABBA bench tests passed
==> /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/bin/path_containment (-max_total_time=7, -seed=123456789, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/explicit-artifacts/path_containment/, corpus=/private/tmp/gw-orchestrator-2734/ci-drive/fuzz/corpus/path_containment)
-seed=123456789
target=path_containment
seed=123456789
max_total_time=7
git_revision=4d15b816236630b2c42f321ffc9ed3654edc94e3
git_dirty=true
runner_mode=prebuilt
==> /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/bin/path_containment (-max_total_time=1, -seed=2322617706, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/auto-artifacts/path_containment/, corpus=/private/tmp/gw-orchestrator-2734/ci-drive/fuzz/corpus/path_containment)
-seed=2322617706
==> cargo +nightly fuzz run path_containment (-max_total_time=9, -seed=987654321, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/cargo-artifacts/path_containment/)
+nightly
fuzz
run
path_containment
-seed=987654321
seed=987654321
runner_mode=cargo-fuzz
fuzz runner seed manifest tests passed
✅ scripts/python_bin.sh resolver checks passed
self-test ok: clean fixture with every exemption present passes
self-test ok: all 11 pattern families fire
self-test ok: moving a pinned source is reported
Embedded secret guard self-test passed.
self-test ok: p1-s3s-import: 'use s3s::s3_error;' counts every line, the rustfs_s3_types module's too (s3_error_lines=3)
self-test ok: p2-s3s-mixed-tree: a multi-line s3s use tree naming s3_error counts (s3_error_lines=3)
self-test ok: p3-s3s-glob: a crate-root glob of s3s, bare or inside a tree, counts (s3_error_lines=5)
self-test ok: p4-macro-use: '#[macro_use] extern crate s3s' counts (s3_error_lines=3)
self-test ok: p5-qualified-s3s: a qualified s3s::s3_error! line counts in a cleared file (s3_error_lines=2)
self-test ok: p6-exact-pass: a baseline equal to the tree passes (exit 0)
self-test ok: n1-rustfs-import: 'use rustfs_s3_types::s3_error;' alone counts nothing (s3_error_lines=1)
self-test ok: n2-rustfs-mixed-tree: a rustfs_s3_types tree plus an s3s tree without the macro counts nothing (s3_error_lines=1)
self-test ok: n3-defines: the file defining the macro counts nothing (s3_error_lines=1)
self-test ok: n4-qualified-rustfs: a qualified rustfs_s3_types::s3_error! line counts nothing (s3_error_lines=1)
self-test ok: n5-renamed-call: a call through a renamed macro is not text-matched (out of scope) (s3_error_lines=1)
self-test ok: n6-comment-string: a comment or string mentioning s3_error! in a cleared file counts nothing (s3_error_lines=1)
self-test ok: n7-e2e-excluded: crates/e2e_test is excluded (s3_error_lines=1)
self-test ok: n8-super-glob: an unrecognised import form ('use super::*;') counts (s3_error_lines=3)
self-test ok: n9-facade: a crate-local re-export counts (s3_error_lines=2)
self-test ok: n10-renamed-import: a renamed rustfs_s3_types import does not clear s3_error! lines (s3_error_lines=2)
self-test ok: n11-commented-import: a commented-out rustfs_s3_types import does not clear (s3_error_lines=2)
self-test ok: n12-rustfs-glob: a glob import of rustfs_s3_types does not clear (s3_error_lines=2)
self-test ok: n13-comment-only: a comment mentioning s3_error! in a file with no import counts (s3_error_lines=2)
self-test ok: f1-baseline-missing: a missing baseline file fails (exit 1)
self-test ok: f2-key-missing: a baseline without s3_error_lines fails (exit 1)
self-test ok: f3-rg-fails: a failing classification search fails the guard (exit 1)
✅ s3s footprint self-test passed
test_core_gate_rejects_missing_ignored_filtered_and_corrupt_inputs (__main__.SelfTests.test_core_gate_rejects_missing_ignored_filtered_and_corrupt_inputs) ... ok
test_e2e_requires_registration (__main__.SelfTests.test_e2e_requires_registration) ... ok
test_fuzz_runtime_uses_matrix_target (__main__.SelfTests.test_fuzz_runtime_uses_matrix_target) ... ok
test_ilm_lane_keeps_the_measured_cargo_build_budget (__main__.SelfTests.test_ilm_lane_keeps_the_measured_cargo_build_budget) ... ok
test_profile_listing_binds_platform_digest (__main__.SelfTests.test_profile_listing_binds_platform_digest) ... ok
test_profile_listing_enforces_selection (__main__.SelfTests.test_profile_listing_enforces_selection) ... ok
test_quick_checks_commands_propagate_failure (__main__.SelfTests.test_quick_checks_commands_propagate_failure) ... ok
test_quick_checks_rejects_caller_and_execution_bypasses (__main__.SelfTests.test_quick_checks_rejects_caller_and_execution_bypasses) ... ok
test_runner_readiness_fails_closed (__main__.SelfTests.test_runner_readiness_fails_closed) ... ok
test_s3_runner_rejects_unbounded_failure_locals (__main__.SelfTests.test_s3_runner_rejects_unbounded_failure_locals) ... ok
test_scanner_heal_begin_requires_embedded_source_provenance (__main__.SelfTests.test_scanner_heal_begin_requires_embedded_source_provenance) ... ok
test_scanner_heal_case_does_not_approve_pending_release (__main__.SelfTests.test_scanner_heal_case_does_not_approve_pending_release) ... ok
test_scanner_heal_case_lane_is_required_for_runner_profile (__main__.SelfTests.test_scanner_heal_case_lane_is_required_for_runner_profile) ... ok
test_scanner_heal_case_only_schema_cannot_approve_release (__main__.SelfTests.test_scanner_heal_case_only_schema_cannot_approve_release) ... ok
test_scanner_heal_crash_case_rejects_restart_oracle (__main__.SelfTests.test_scanner_heal_crash_case_rejects_restart_oracle) ... ok
test_scanner_heal_ec84_case_rejects_wrong_erasure_geometry (__main__.SelfTests.test_scanner_heal_ec84_case_rejects_wrong_erasure_geometry) ... ok
test_scanner_heal_finish_collects_oracles_from_registry (__main__.SelfTests.test_scanner_heal_finish_collects_oracles_from_registry) ... ok
test_scanner_heal_multipool_case_accepts_deferred_optional_outage_write (__main__.SelfTests.test_scanner_heal_multipool_case_accepts_deferred_optional_outage_write) ... ok
test_scanner_heal_pending_gate_cannot_map_to_implemented_lane (__main__.SelfTests.test_scanner_heal_pending_gate_cannot_map_to_implemented_lane) ... ok
test_scanner_heal_receipts_reject_reuse_and_missing_builds (__main__.SelfTests.test_scanner_heal_receipts_reject_reuse_and_missing_builds) ... ok
test_scanner_heal_rejects_boolean_fractional_and_out_of_geometry_integers (__main__.SelfTests.test_scanner_heal_rejects_boolean_fractional_and_out_of_geometry_integers) ... ok
test_scanner_heal_rejects_broken_execution_and_artifacts (__main__.SelfTests.test_scanner_heal_rejects_broken_execution_and_artifacts) ... ok
test_scanner_heal_rejects_copied_junit_and_wrong_suite_build (__main__.SelfTests.test_scanner_heal_rejects_copied_junit_and_wrong_suite_build) ... ok
test_scanner_heal_release_bundle_accepts_complete_measured_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_accepts_complete_measured_evidence) ... ok
test_scanner_heal_release_bundle_assembler_copies_measured_artifacts (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_copies_measured_artifacts) ... ok
test_scanner_heal_release_bundle_assembler_merges_measured_descriptors (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_merges_measured_descriptors) ... ok
test_scanner_heal_release_bundle_assembler_rejects_duplicate_descriptor_gate (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_duplicate_descriptor_gate) ... ok
test_scanner_heal_release_bundle_assembler_rejects_fixture_descriptor (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_fixture_descriptor) ... ok
test_scanner_heal_release_bundle_assembler_rejects_hash_mismatch (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_hash_mismatch) ... ok
test_scanner_heal_release_bundle_assembler_rejects_mixed_descriptor_revisions (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_mixed_descriptor_revisions) ... ok
test_scanner_heal_release_bundle_enforces_topology_duration_profile_and_versions (__main__.SelfTests.test_scanner_heal_release_bundle_enforces_topology_duration_profile_and_versions) ... ok
test_scanner_heal_release_bundle_fixture_writer_is_not_release_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_fixture_writer_is_not_release_evidence) ... ok
test_scanner_heal_release_bundle_gate_accepts_single_measured_gate (__main__.SelfTests.test_scanner_heal_release_bundle_gate_accepts_single_measured_gate) ... ok
test_scanner_heal_release_bundle_gate_rejects_missing_field (__main__.SelfTests.test_scanner_heal_release_bundle_gate_rejects_missing_field) ... ok
test_scanner_heal_release_bundle_gate_rejects_unknown_gate (__main__.SelfTests.test_scanner_heal_release_bundle_gate_rejects_unknown_gate) ... ok
test_scanner_heal_release_bundle_rejects_synthetic_or_missing_fields (__main__.SelfTests.test_scanner_heal_release_bundle_rejects_synthetic_or_missing_fields) ... ok
test_scanner_heal_release_bundle_requires_domain_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_requires_domain_evidence) ... ok
test_scanner_heal_release_bundle_requires_field_provenance (__main__.SelfTests.test_scanner_heal_release_bundle_requires_field_provenance) ... ok
test_scanner_heal_release_bundle_requires_json_artifact_provenance (__main__.SelfTests.test_scanner_heal_release_bundle_requires_json_artifact_provenance) ... ok
test_scanner_heal_release_bundle_requires_mixed_version_field_roles (__main__.SelfTests.test_scanner_heal_release_bundle_requires_mixed_version_field_roles) ... ok
test_scanner_heal_release_bundle_requires_same_gate_measurement_window (__main__.SelfTests.test_scanner_heal_release_bundle_requires_same_gate_measurement_window) ... ok
test_scanner_heal_release_consumes_multiple_registry_oracles (__main__.SelfTests.test_scanner_heal_release_consumes_multiple_registry_oracles) ... ok
test_scanner_heal_release_matrix_lanes_remain_pending_without_real_evidence (__main__.SelfTests.test_scanner_heal_release_matrix_lanes_remain_pending_without_real_evidence) ... ok
test_scanner_heal_release_requirements_cannot_be_cleared (__main__.SelfTests.test_scanner_heal_release_requirements_cannot_be_cleared) ... ok
test_scanner_heal_release_status_rejects_focused_case_run (__main__.SelfTests.test_scanner_heal_release_status_rejects_focused_case_run) ... ok
test_scanner_heal_release_status_rejects_synthetic_case (__main__.SelfTests.test_scanner_heal_release_status_rejects_synthetic_case) ... ok
test_scanner_heal_required_evidence_fields_cannot_be_removed (__main__.SelfTests.test_scanner_heal_required_evidence_fields_cannot_be_removed) ... ok
test_scheduled_alerts_require_completion_watchdog (__main__.SelfTests.test_scheduled_alerts_require_completion_watchdog) ... ok
test_update_profile_selection_changes_only_requested_platform (__main__.SelfTests.test_update_profile_selection_changes_only_requested_platform) ... ok
test_validate_includes_quick_checks (__main__.SelfTests.test_validate_includes_quick_checks) ... ok
test_vault_tests_require_cross_process_serialization (__main__.SelfTests.test_vault_tests_require_cross_process_serialization) ... ok
test_workflow_readiness_requires_dependency_probe (__main__.SelfTests.test_workflow_readiness_requires_dependency_probe) ... ok

----------------------------------------------------------------------
Ran 52 tests in 14.301s

OK
e2e-full selection OK: 1 tests, sha256=29e30e6ed2ed38d9e5b0a7cdd9536abf5c719f7021a02b578ae1381563a3c273
...................................
----------------------------------------------------------------------
Ran 35 tests in 1.161s

OK
...................
----------------------------------------------------------------------
Ran 19 tests in 17.397s

OK
..........
----------------------------------------------------------------------
Ran 10 tests in 0.183s

OK
test_actual_selector_bootstrap_uses_base_policy_and_fails_closed (__main__.SelfTests.test_actual_selector_bootstrap_uses_base_policy_and_fails_closed) ... ok
test_documentation_paths_do_not_hide_build_or_fixture_changes (__main__.SelfTests.test_documentation_paths_do_not_hide_build_or_fixture_changes) ... ok
test_e2e_scope_keeps_production_and_shared_configuration_full (__main__.SelfTests.test_e2e_scope_keeps_production_and_shared_configuration_full) ... ok
test_e2e_shortcut_rejects_new_direct_renamed_and_target_dependencies (__main__.SelfTests.test_e2e_shortcut_rejects_new_direct_renamed_and_target_dependencies) ... ok
test_e2e_shortcut_traverses_implicit_path_dependencies (__main__.SelfTests.test_e2e_shortcut_traverses_implicit_path_dependencies) ... ok
test_event_contract_requires_complete_candidate_and_optional_lanes (__main__.SelfTests.test_event_contract_requires_complete_candidate_and_optional_lanes) ... ok
test_every_wrong_result_missing_job_or_selection_fails_closed (__main__.SelfTests.test_every_wrong_result_missing_job_or_selection_fails_closed) ... ok
test_full_e2e_gate_preserves_workflow_branch_and_event_scope (__main__.SelfTests.test_full_e2e_gate_preserves_workflow_branch_and_event_scope) ... ok
test_git_range_includes_deleted_source_and_rename_origins (__main__.SelfTests.test_git_range_includes_deleted_source_and_rename_origins) ... ok
test_job_ids_and_display_names_cannot_hide_validation (__main__.SelfTests.test_job_ids_and_display_names_cannot_hide_validation) ... ok
test_report_only_stack_lane_mirrors_e2e_smoke_outside_the_aggregate (__main__.SelfTests.test_report_only_stack_lane_mirrors_e2e_smoke_outside_the_aggregate) ... ok
test_repository_wiring_and_missing_dependency_regression (__main__.SelfTests.test_repository_wiring_and_missing_dependency_regression) ... ok
test_verify_command_preserves_failures (__main__.SelfTests.test_verify_command_preserves_failures) ... ok

----------------------------------------------------------------------
Ran 13 tests in 3.280s

OK
security coverage self-test passed
test_api_failures_preserve_other_evidence_and_never_enter_grace (__main__.SelfTests.test_api_failures_preserve_other_evidence_and_never_enter_grace) ... ok
test_attempt_outcome_does_not_replace_recent_success (__main__.SelfTests.test_attempt_outcome_does_not_replace_recent_success) ... ok
test_cancelled_attempt_cannot_refresh_expired_success (__main__.SelfTests.test_cancelled_attempt_cannot_refresh_expired_success) ... ok
test_cli_requires_the_repository_default_branch (__main__.SelfTests.test_cli_requires_the_repository_default_branch) ... ok
test_config_rejects_duplicate_and_invalid_entries (__main__.SelfTests.test_config_rejects_duplicate_and_invalid_entries) ... ok
test_freshness_boundaries (__main__.SelfTests.test_freshness_boundaries) ... ok
test_grace_requires_two_successful_queries_with_no_history (__main__.SelfTests.test_grace_requires_two_successful_queries_with_no_history) ... ok
test_invalid_api_evidence_fails_closed (__main__.SelfTests.test_invalid_api_evidence_fails_closed) ... ok
test_report_retains_every_workflow (__main__.SelfTests.test_report_retains_every_workflow) ... ok
test_requests_filter_schedule_default_branch_and_success_on_server (__main__.SelfTests.test_requests_filter_schedule_default_branch_and_success_on_server) ... ok

----------------------------------------------------------------------
Ran 10 tests in 0.051s

OK
..............................
----------------------------------------------------------------------
Ran 30 tests in 22.354s

OK
..........
----------------------------------------------------------------------
Ran 10 tests in 2.969s

OK
.................................
----------------------------------------------------------------------
Ran 33 tests in 0.160s

OK
Online functional runner labels: pf-testing
.......
----------------------------------------------------------------------
Ran 7 tests in 0.002s

OK
.......
----------------------------------------------------------------------
Ran 7 tests in 0.001s

OK
....
----------------------------------------------------------------------
Ran 4 tests in 0.128s

OK
...
----------------------------------------------------------------------
Ran 3 tests in 0.547s

OK
PASS: cold stampede script self-test
exit_code=0

$ python3 ./scripts/test_object_generation_protocol_model.py
test_a_recovery_barrier_cannot_choose_an_empty_value (__main__.ObjectGenerationProtocolModel.test_a_recovery_barrier_cannot_choose_an_empty_value) ... ok
test_all_bounded_two_proposer_interleavings_choose_at_most_one_value (__main__.ObjectGenerationProtocolModel.test_all_bounded_two_proposer_interleavings_choose_at_most_one_value) ... ok
test_an_unavailable_decision_quorum_cannot_choose_a_candidate (__main__.ObjectGenerationProtocolModel.test_an_unavailable_decision_quorum_cannot_choose_a_candidate) ... ok
test_counter_allocation_requires_persistence_and_is_consumed_once (__main__.ObjectGenerationProtocolModel.test_counter_allocation_requires_persistence_and_is_consumed_once) ... ok
test_counter_restart_burns_persisted_allocations_before_or_after_issue (__main__.ObjectGenerationProtocolModel.test_counter_restart_burns_persisted_allocations_before_or_after_issue) ... ok
test_crash_after_promise_before_reply_does_not_restore_an_older_ballot (__main__.ObjectGenerationProtocolModel.test_crash_after_promise_before_reply_does_not_restore_an_older_ballot) ... ok
test_duplicate_or_unconfigured_voters_cannot_form_a_quorum (__main__.ObjectGenerationProtocolModel.test_duplicate_or_unconfigured_voters_cannot_form_a_quorum) ... ok
test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision (__main__.ObjectGenerationProtocolModel.test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision) ... ok
test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected (__main__.ObjectGenerationProtocolModel.test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected) ... ok
test_forgetting_tombstone_authority_allows_an_aba_create (__main__.ObjectGenerationProtocolModel.test_forgetting_tombstone_authority_allows_an_aba_create) ... ok
test_issuing_before_persistence_reuses_a_ballot_after_restart (__main__.ObjectGenerationProtocolModel.test_issuing_before_persistence_reuses_a_ballot_after_restart) ... ok
test_lost_ack_and_restart_preserve_the_chosen_operation (__main__.ObjectGenerationProtocolModel.test_lost_ack_and_restart_preserve_the_chosen_operation) ... ok
test_minority_accepted_work_is_adopted_and_finished (__main__.ObjectGenerationProtocolModel.test_minority_accepted_work_is_adopted_and_finished) ... ok
test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample (__main__.ObjectGenerationProtocolModel.test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample) ... ok
test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it (__main__.ObjectGenerationProtocolModel.test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it) ... ok
test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor (__main__.ObjectGenerationProtocolModel.test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor) ... ok
test_replacing_voters_with_empty_state_can_choose_a_conflicting_value (__main__.ObjectGenerationProtocolModel.test_replacing_voters_with_empty_state_can_choose_a_conflicting_value) ... ok
test_skipping_predecessor_recovery_chooses_a_broken_revision_chain (__main__.ObjectGenerationProtocolModel.test_skipping_predecessor_recovery_chooses_a_broken_revision_chain) ... ok
test_stale_publication_rollback_and_gc_cannot_change_the_winner (__main__.ObjectGenerationProtocolModel.test_stale_publication_rollback_and_gc_cannot_change_the_winner) ... ok
test_superseded_operation_identity_cannot_be_reused_for_different_input (__main__.ObjectGenerationProtocolModel.test_superseded_operation_identity_cannot_be_reused_for_different_input) ... ok
test_superseded_operation_retry_returns_its_original_successor (__main__.ObjectGenerationProtocolModel.test_superseded_operation_retry_returns_its_original_successor) ... ok
test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision (__main__.ObjectGenerationProtocolModel.test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision) ... ok
test_tombstone_rejects_a_delayed_never_present_create (__main__.ObjectGenerationProtocolModel.test_tombstone_rejects_a_delayed_never_present_create) ... ok
test_unlearned_predecessor_is_recovered_before_the_second_slot (__main__.ObjectGenerationProtocolModel.test_unlearned_predecessor_is_recovered_before_the_second_slot) ... ok

----------------------------------------------------------------------
Ran 24 tests in 0.958s

OK
model: states=51023, transitions=167520, terminal=732, chosen_candidates=2
negative control: double-choice trace=((0, 'prepare', 0), (0, 'prepare', 1), (0, 'prepare', 2), (0, 'accept', 0), (0, 'accept', 1), (0, 'accept', 2), (1, 'prepare', 0), (1, 'prepare', 1), (1, 'prepare', 2), (1, 'accept', 0), (1, 'accept', 1), (1, 'accept', 2))
multi-slot round schedules: 640
exit_code=0

$ python3 ./scripts/check_test_wiring.py
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired
exit_code=0

$ ./scripts/check_no_planning_docs.sh
exit_code=0

$ ./scripts/check_uring_lane_lib_only.sh
OK: no 'uring_'-matching test functions under crates/ecstore/tests; --lib is safe
exit_code=0

Initial complete gate
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
 Nextest run ID e07fc5b9-c6c8-4cdd-ae61-54839bf24437 with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.016s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.011s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.009s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.009s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  16.950s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   4.025s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.008s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.007s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.035s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  21.087s] 10 tests run: 10 passed, 0 skipped
exit_code=0

Restored complete gate
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
 Nextest run ID 233b7307-5361-4eb7-94d5-b384f17d2d6a with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.009s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.008s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.008s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.008s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  19.518s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   4.146s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.009s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.018s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.009s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.054s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  23.787s] 10 tests run: 10 passed, 0 skipped
exit_code=0

Mutation M1: success_measurement reports max(actual elapsed, requested budget + 1), forcing 1001 ms for the one-second request. The retained duration assertion at rustfs/tests/connect_perf_drive.rs:105 fails.
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast -E 'test(real_local_write_read_is_measured_and_scratch_is_removed)'
info: experimental features enabled: setup-scripts
   Compiling rustfs v1.0.1 (/private/tmp/gw-orchestrator-2734/ci-drive/rustfs)
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs") generated 1 warning
warning: `rustfs` (bin "rustfs-cli") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 7m 06s
────────────
 Nextest run ID 9f618777-7bf6-4cbc-8803-dd7f50f449ed with nextest profile: ci
    Starting 1 test across 1 binary (9 tests skipped)
        FAIL [   0.016s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
  stdout ───

    running 1 test
    test real_local_write_read_is_measured_and_scratch_is_removed ... FAILED

    failures:

    failures:
        real_local_write_read_is_measured_and_scratch_is_removed

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 0.01s

  stderr ───

    thread 'real_local_write_read_is_measured_and_scratch_is_removed' (106986846) panicked at rustfs/tests/connect_perf_drive.rs:105:5:
    assertion failed: (1..=1_000).contains(&data.duration_millis)
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

────────────
     Summary [   0.017s] 1 test run: 0 passed, 1 failed, 9 skipped
        FAIL [   0.016s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
error: test run failed
exit_code=100

Mutation M2: target_parameters reports requested duration + 1. The in-process assertion at rustfs/tests/connect_perf_drive.rs:111 and CLI JSON assertion at rustfs/tests/connect_perf_drive.rs:527 both reject 1001 against 1000.
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast -E 'test(real_local_write_read_is_measured_and_scratch_is_removed) | test(production_cli_measures_and_signs_exact_binary_provenance)'
info: experimental features enabled: setup-scripts
   Compiling rustfs v1.0.1 (/private/tmp/gw-orchestrator-2734/ci-drive/rustfs)
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs") generated 1 warning
warning: `rustfs` (bin "rustfs-cli") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 7m 20s
────────────
 Nextest run ID b73d46ab-c910-4e1e-8063-7fc7e332ec15 with nextest profile: ci
    Starting 2 tests across 1 binary (8 tests skipped)
        FAIL [  13.968s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
  stdout ───

    running 1 test
    test production_cli_measures_and_signs_exact_binary_provenance ... FAILED

    failures:

    failures:
        production_cli_measures_and_signs_exact_binary_provenance

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 13.96s

  stderr ───

    thread 'production_cli_measures_and_signs_exact_binary_provenance' (107015462) panicked at rustfs/tests/connect_perf_drive.rs:527:5:
    assertion `left == right` failed
      left: Number(1001)
     right: 1000
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

        FAIL [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
  stdout ───

    running 1 test
    test real_local_write_read_is_measured_and_scratch_is_removed ... FAILED

    failures:

    failures:
        real_local_write_read_is_measured_and_scratch_is_removed

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 0.01s

  stderr ───

    thread 'real_local_write_read_is_measured_and_scratch_is_removed' (107016667) panicked at rustfs/tests/connect_perf_drive.rs:111:5:
    assertion `left == right` failed
      left: 1001
     right: 1000
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

────────────
     Summary [  13.985s] 2 tests run: 0 passed, 2 failed, 8 skipped
        FAIL [  13.968s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        FAIL [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
error: test run failed
exit_code=100

Transient production mutations were restored byte-for-byte before the complete green rerun. Original/restored SHA256: 8d389e6d8836c041aab8c6a93e56addd65ba3f70b33264e1c74f639d0572a2da.

$ git diff --check

exit_code=0

$ git diff --cached --check

exit_code=0

Independent orchestrator acceptance
Tested code tree: e67c72fb01

Tested tree: 16fbbd2ea4
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

$ LANG=en_US.UTF-8 CARGO_BUILD_JOBS=1 CARGO_INCREMENTAL=0 cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
info: experimental features enabled: setup-scripts
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs-cli") generated 1 warning
warning: `rustfs` (bin "rustfs") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.69s
────────────
 Nextest run ID cd1edf4c-f1c0-456d-ad1a-76cc00ce05c9 with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.012s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.010s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.010s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.009s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  20.328s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   5.527s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.013s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.019s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.011s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.049s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  25.991s] 10 tests run: 10 passed, 0 skipped

exit_code=0

All 54 tracked Cargo.toml files parse without duplicate keys. git diff --check passed. The production source remains unchanged; the two deliberate implementation mutations above were restored before this complete green run.

* fix(test): compare legacy body wire parity as raw bytes

Lossy UTF-8 comparison concealed a changed binary response byte.
Compare the already captured Vec<u8> values directly.

Validation:
- Binary mutant: migrated payload 0x84 vs legacy 0x83; old comparator
  passed 1/1, raw comparator failed 0/1 (exit 100); restored green.
- Non-Swift body-drain mutant: untouched-body delegation test failed
  with empty vs payload (exit 100); restored handler without a diff.
- cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols
  --features rustfs-protocols/swift --no-fail-fast:
  543 tests run, 543 passed, 0 skipped after both mutants restored.
- cargo fmt --all --check and git diff --check passed.

Orchestrator acceptance (LANG=en_US.UTF-8):
cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
Summary [2.445s] 543 tests run: 543 passed, 0 skipped.

All shared quick checks on the final code tree:
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

Final independent acceptance after the drive-test CI repair
Parent: a99d8f4f89
Tested code tree: 93cbd96bf0

Tested tree: 075ae5161c2b62a96812bc1e3c904a7663b4f2f8
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

$ LANG=en_US.UTF-8 CARGO_BUILD_JOBS=1 CARGO_INCREMENTAL=0 cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
    Finished `test` profile [unoptimized + debuginfo] target(s) in 10m 08s
 Nextest run ID 5c7ff102-ec6d-479a-86e0-810859b28bbf with nextest profile: ci
    Starting 543 tests across 8 binaries
     Summary [   6.729s] 543 tests run: 543 passed, 0 skipped
exit_code=0

The only source change is the raw-byte wire comparator. Cargo manifests and the ratchet baselines remain unchanged from the independently validated parent. The invalid-byte and consumed-request mutations remain recorded above; their implementation changes were restored.

* fix: repair integration checks and IPC test readiness

Wait for the RPC/locks test sockets to pass the existing owner and mode
checks inside the original 5s readiness window and 10ms poll interval.

Use the existing admin S3 carrier for the new recovery and quota error
calls. Regenerate the test-count baseline for main's 118 existing tests
across seven crates. Pair the private pagination helper's raw-scan inputs
and remove the redundant Copy clone from the retirement event.

Keep the original assertions, responses, authorization, locks and storage
operations. The pagination body and raw-scan expressions at all five calls
are byte-identical; reinserting only the Copy clone restores the entire
base heal file. All ignored baseline values and s3s counts stay unchanged.

Validation base: c47432e353
Validation tree: f081084bc5
Patch SHA-256: 2dcf0028691bfb2772edfb38e262829b04f4151832dcce823c9dd9f7434b52de
Trace production-prefix SHA-256: fd994f218a21429d62866f4efb8eee1ee57b58f4a51865aa35d05a54ef1ccb45

All commands used LANG=en_US.UTF-8, CARGO_BUILD_JOBS=1,
CARGO_INCREMENTAL=0 and CARGO_NET_OFFLINE=true in one exclusive target.
Every gate retained the same tree and six source hashes before and after.
54 Cargo.toml files parse without duplicate keys.

cargo clippy -p rustfs --all-targets -- -D warnings
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 12m 50s
Exit: 0. Wall time: 770.855s.

The original 19 quick-check steps:
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
Wall time: 286.337s.

cargo nextest run --profile ci -p rustfs --no-fail-fast
     Summary [ 217.017s] 5382 tests run: 5382 passed (4 leaky), 40 skipped
Exit: 0. Wall time: 1324.414s.

cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
     Summary [   2.189s] 543 tests run: 543 passed, 0 skipped
Exit: 0. Wall time: 123.192s.

cargo nextest run --profile ci -p rustfs-ecstore list_objects_paginate --no-fail-fast
     Summary [   0.170s] 8 tests run: 8 passed, 5921 skipped
Exit: 0. Wall time: 261.324s.

cargo nextest run --profile ci -p rustfs-heal durable_mrf_anchor_retires_only_after_exact_bucket_generation_is_deleted --no-fail-fast
     Summary [   0.805s] 1 test run: 1 passed, 835 skipped
Exit: 0. Wall time: 199.801s.

Baseline regeneration:
./scripts/check_test_count_ratchet.sh --update
wrote .config/test-count-baseline.txt: 52 crates, 21340 tests, 164 ignored
Test-only increases: ecstore48, heal3, iam14, io-metrics2, obs3, policy14,
rustfs34. Existing ignored values and all other rows are byte-identical.
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)

Historical runtime red controls on f622 (same retained two assertions):
cargo nextest run --profile ci -p rustfs --lib -E 'test(=connect::diagnostics::trace_runtime::tests::local_top_rpc_reads_service_http_completions_and_signs_offline) | test(=connect::diagnostics::trace_runtime::tests::local_top_locks_reads_service_manager_and_signs_offline)' --no-fail-fast
Original path readiness with a temporary 500ms bind/seal gap:
     Summary [   0.863s] 2 tests run: 0 passed, 2 failed, 5113 skipped
Exit: 100.
Sealed readiness in that same gap:
     Summary [   5.621s] 2 tests run: 2 passed, 5113 skipped
Exit: 0.
Compiled M1: replace the helper predicate with path.exists():
     Summary [   0.870s] 2 tests run: 0 passed, 2 failed, 5113 skipped
Exit: 100.
Restore the sealed predicate byte-for-byte:
     Summary [   5.640s] 2 tests run: 2 passed, 5113 skipped
Exit: 0.
Both reds observe mode755 with owner_matches=true and fail at the original
RPC/locks unwraps with StateSecurity. M1 failure locations: trace_runtime.rs
2547:29 and 2713:29. No new assertions were introduced.

cargo nextest run --profile ci -p rustfs --lib -E 'test(connect::diagnostics::trace_runtime::tests::)' --no-fail-fast
     Summary [  20.748s] 28 tests run: 28 passed (1 leaky), 5087 skipped
Exit: 0. This security sweep is historical; the full final unit gate above
covers the actual c474 candidate.
The first disk-stopped compile ran no tests and is excluded from red credit.
All temporary seal instrumentation is removed from the final source.

Final unit cleanup markers: 4 leaky, 40 original skips (exit0). The ld
__eh_frame linker warning also appears verbatim in the original f622 run
and the earlier c474 run; there are no new source warnings.

* fix: remove redundant pending closure in disk test

Pass the capture-free pending future factory as a function item in the existing slow-operation test. This fixes the redundant_closure error at disk_store.rs:2914 that stopped strict Clippy before the feature tests could run.

The single-line test edit preserves the factory's result, warning and timeout behavior. Reversing the replacement reproduces the parent file byte for byte. All 220 assertion lines and the production prefix are unchanged; no tests, assertions, guards, profiles or baselines were added or relaxed.

Original Linux CI evidence (run 37888552268 at 53245be9f):
- RIO job113685483992: strict Clippy exit101, redundant closure at disk_store.rs:2914; feature tests skipped.
- Workspace Test and Lint job113685484058: same strict Clippy exit101; nextest and doc tests skipped.
These are lint failures, not runtime mutation kills or known flakes.

Native macOS arm64 validation on tree7a5bf47e7533b29479bc85207416102f1337c566, LANG=en_US.UTF-8, one build job, CARGO_INCREMENTAL=0 and offline dependencies:

shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
All 19 checks: exit0, 312.161s.

$ cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 20m 11s
exit0, 1213.210s total.

$ cargo nextest run --profile ci -p rustfs-ecstore --features rio-v2 slow_disk_operation_warns_once_while_pending_and_preserves_result --no-fail-fast
Finished `test` profile [unoptimized + debuginfo] target(s) in 4m 34s
PASS [0.026s] rustfs-ecstore disk::disk_store::tests::slow_disk_operation_warns_once_while_pending_and_preserves_result
Summary [   0.170s] 1 test run: 1 passed, 5943 skipped
exit0, 281.920s total; 5943 existing filter skips.

warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
This linker warning is retained; flags, profiles and allowances are unchanged.

54 tracked Cargo.toml files parsed without duplicate keys. git diff --check passed. No new assertions were introduced, so no new runtime mutation is claimed. Native compilation and the existing helper test do not substitute for actual Linux RIO execution; the new integration CI run still needs to recover before the RPC merge.

Preservation and output hashes:
parent file SHA256: 8e59ee745e3aaaa22848b943d3b10f6353b515956282cde1f3af2c394200c9ac
final file SHA256: b69ba3237061b161448ae8b62b615885d5770b98a5b4a2479bcfbd70f1bd1f17
unchanged production prefix SHA256: 7903a6e288252c420231242d7b58a25fa2a7140ea3dd430cbe7f0dde11369dcb
RIO Linux failure log SHA256:421addb9868fd88e4109a922311e54f91f0dfd70862f8ba7fcaba193482ed90b
Workspace Linux failure log SHA256:e0aea8d0f014c022252a9534968e083c98cfc1fd760da445ecd04ec3f522e26a
19 output SHA256:35a54f28d707956ed1edb29d492c4dfe8a9fe753d103b111b5a2f1b5c41c2b98
RIO Clippy output SHA256:d6046e3868f2b3992fe11cdd04e9e98eb50a87d1234d9a70dbfeb154ff61638d
slow-operation output SHA256:605b7b258d2fd09c6ef2ab2ffc90d433d0509480f1e45125d20c823759e83608

Refs rustfs/backlog#2734.

* test(ecstore): isolate tier-free recovery fixture startup

The fixture constructor can take the production recovery lock and wait for 60 seconds before this test installs its queue and seeds metadata. Disable its background recovery only while creating this fixture, then restore the environment before calling the existing production recovery entrypoint.

Only the existing test setup changes. Its production entrypoint, default lock parameters, original assertions, 30-second timeout, cancellation and cleanup remain intact. No tests or assertions were added. Production and store/init.rs are byte-identical to the parent.

Independent acceptance ran on merge tree 8ec358cb42ba9ebb3bcfeafcef048a3f0366355b; the runner header reports parent HEAD, while before/after index and source hashes confirm this candidate. Worker task branch d8055766b8 has the same tree.

Every command used LANG=en_US.UTF-8. Task compilation also used CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0, CARGO_NET_OFFLINE=true, an independent target, and empty RUSTC_WRAPPER/RUSTC_WORKSPACE_WRAPPER to keep compilers in its owned process group without changing repository configuration.

Command:
shellcheck --version && actionlint
Exit 0; observed output excerpt:
version: 0.11.0
Log SHA-256: fc9eec16b4191d058a295813ca9afbe9d2b37b59975da860170726be162a3924

Command:
cargo fmt --all --check
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_unsafe_code_allowances.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_layer_dependencies.sh
Exit 0; observed output excerpt:
Layer dependency guard passed (no new reverse dependencies/cycles).
Log SHA-256: 69c6cb1896c0586b28d1167380d3260d99d8017a4265c8df5a2348f289e93a12

Command:
./scripts/check_architecture_migration_rules.sh
Exit 0; observed output excerpt:
Architecture migration rules passed.
Log SHA-256: 28790e70d070607ed4de0704f292a65030be92ec224ef106d4b72fd8296f896e

Command:
./scripts/check_logging_guardrails.sh
Exit 0; observed output excerpt:
✅ Logging guardrails check passed
Log SHA-256: 7adf41abff8ed8f7617decf86498888ddde78937099dd9ce47f6d359dfc2d317

Command:
./scripts/check_error_other_format_ratchet.sh
Exit 0; observed output excerpt:
error(format!) ratchet passed (total call sites: 579).
Log SHA-256: 66172aef18f150ba0605af122e26ffcf91f6dd86f43e38b05e2e982ce8d783ee

Command:
./scripts/check_no_tokio_io_uring.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_extension_schema_boundaries.sh
Exit 0; observed output excerpt:
Extension schema boundaries passed.
Log SHA-256: f13d0acd0e2ce5d9815241b12b4dab7d01cc11b9b65bcf6dfb7812c878532a51

Command:
./scripts/check_body_cache_whitelist.sh
Exit 0; observed output excerpt:
Body-cache whitelist guard passed.
Log SHA-256: 6b828847b0383950ee6eef9cd846d0c322c945e47efd4c7d1ad881178810d6bb

Command:
./scripts/check_s3s_footprint.sh
Exit 0; observed output excerpt:
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)
✅ s3s footprint baseline check passed
Log SHA-256: 47f53d04ef645618563fe1f22abc64667f19078f1974feedcca46e4f2c016844

Command:
./scripts/check_test_count_ratchet.sh
Exit 0; observed output excerpt:
test-count ratchet OK: 52 crates, 21340 tests, 164 ignored match .config/test-count-baseline.txt
Log SHA-256: 618a5e330e933de0c6ff484546df401457024a88bb5986d11d588794e33dab60

Command:
./scripts/check_fips_wording.sh
Exit 0; observed output excerpt:
FIPS wording guard passed (README.md and CHANGELOG.md contain no forbidden claims).
KMS wording guard passed (crates/kms claims no Vault KV2 Transit wrapping).
Log SHA-256: 99221cc9979c480ec4f46f50d76c52b18c72fe499f12e15e4f4a22a1b8bfb140

Command:
./scripts/check_embedded_secrets.sh
Exit 0; observed output excerpt:
Embedded secret guard passed (no private key material or provider credential literal in the tree).
Log SHA-256: 437a3ceba32094d2901889965ad191e411aad32cdb18bf0b8659e818011c406b

Command:
make script-tests
Exit 0; observed output excerpt:
test_cli_requires_the_repository_default_branch (__main__.SelfTests.test_cli_requires_the_repository_default_branch) ... ok
test_config_rejects_duplicate_and_invalid_entries (__main__.SelfTests.test_config_rejects_duplicate_and_invalid_entries) ... ok
test_freshness_boundaries (__main__.SelfTests.test_freshness_boundaries) ... ok
test_grace_requires_two_successful_queries_with_no_history (__main__.SelfTests.test_grace_requires_two_successful_queries_with_no_history) ... ok
test_invalid_api_evidence_fails_closed (__main__.SelfTests.test_invalid_api_evidence_fails_closed) ... ok
test_report_retains_every_workflow (__main__.SelfTests.test_report_retains_every_workflow) ... ok
test_requests_filter_schedule_default_branch_and_success_on_server (__main__.SelfTests.test_requests_filter_schedule_default_branch_and_success_on_server) ... ok
Ran 10 tests in 0.042s
OK
Ran 30 tests in 20.627s
OK
Ran 10 tests in 2.268s
OK
Ran 33 tests in 0.127s
OK
Ran 7 tests in 0.002s
OK
Ran 7 tests in 0.001s
OK
Ran 4 tests in 0.108s
OK
Ran 3 tests in 0.564s
OK
PASS: cold stampede script self-test
Log SHA-256: 75fda2d069a373c63667b908baaf99e6591aab9099a739b0c4c22ceb6da8d68c

Command:
python3 ./scripts/test_object_generation_protocol_model.py
Exit 0; observed output excerpt:
test_an_unavailable_decision_quorum_cannot_choose_a_candidate (__main__.ObjectGenerationProtocolModel.test_an_unavailable_decision_quorum_cannot_choose_a_candidate) ... ok
test_counter_allocation_requires_persistence_and_is_consumed_once (__main__.ObjectGenerationProtocolModel.test_counter_allocation_requires_persistence_and_is_consumed_once) ... ok
test_counter_restart_burns_persisted_allocations_before_or_after_issue (__main__.ObjectGenerationProtocolModel.test_counter_restart_burns_persisted_allocations_before_or_after_issue) ... ok
test_crash_after_promise_before_reply_does_not_restore_an_older_ballot (__main__.ObjectGenerationProtocolModel.test_crash_after_promise_before_reply_does_not_restore_an_older_ballot) ... ok
test_duplicate_or_unconfigured_voters_cannot_form_a_quorum (__main__.ObjectGenerationProtocolModel.test_duplicate_or_unconfigured_voters_cannot_form_a_quorum) ... ok
test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision (__main__.ObjectGenerationProtocolModel.test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision) ... ok
test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected (__main__.ObjectGenerationProtocolModel.test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected) ... ok
test_forgetting_tombstone_authority_allows_an_aba_create (__main__.ObjectGenerationProtocolModel.test_forgetting_tombstone_authority_allows_an_aba_create) ... ok
test_issuing_before_persistence_reuses_a_ballot_after_restart (__main__.ObjectGenerationProtocolModel.test_issuing_before_persistence_reuses_a_ballot_after_restart) ... ok
test_lost_ack_and_restart_preserve_the_chosen_operation (__main__.ObjectGenerationProtocolModel.test_lost_ack_and_restart_preserve_the_chosen_operation) ... ok
test_minority_accepted_work_is_adopted_and_finished (__main__.ObjectGenerationProtocolModel.test_minority_accepted_work_is_adopted_and_finished) ... ok
test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample (__main__.ObjectGenerationProtocolModel.test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample) ... ok
test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it (__main__.ObjectGenerationProtocolModel.test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it) ... ok
test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor (__main__.ObjectGenerationProtocolModel.test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor) ... ok
test_replacing_voters_with_empty_state_can_choose_a_conflicting_value (__main__.ObjectGenerationProtocolModel.test_replacing_voters_with_empty_state_can_choose_a_conflicting_value) ... ok
test_skipping_predecessor_recovery_chooses_a_broken_revision_chain (__main__.ObjectGenerationProtocolModel.test_skipping_predecessor_recovery_chooses_a_broken_revision_chain) ... ok
test_stale_publication_rollback_and_gc_cannot_change_the_winner (__main__.ObjectGenerationProtocolModel.test_stale_publication_rollback_and_gc_cannot_change_the_winner) ... ok
test_superseded_operation_identity_cannot_be_reused_for_different_input (__main__.ObjectGenerationProtocolModel.test_superseded_operation_identity_cannot_be_reused_for_different_input) ... ok
test_superseded_operation_retry_returns_its_original_successor (__main__.ObjectGenerationProtocolModel.test_superseded_operation_retry_returns_its_original_successor) ... ok
test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision (__main__.ObjectGenerationProtocolModel.test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision) ... ok
test_tombstone_rejects_a_delayed_never_present_create (__main__.ObjectGenerationProtocolModel.test_tombstone_rejects_a_delayed_never_present_create) ... ok
test_unlearned_predecessor_is_recovered_before_the_second_slot (__main__.ObjectGenerationProtocolModel.test_unlearned_predecessor_is_recovered_before_the_second_slot) ... ok
Ran 24 tests in 0.929s
OK
Log SHA-256: 0b4be926efb0609d497ae58930340153f01957b606206cbbdb3e0d47eb29d3a2

Command:
python3 ./scripts/check_test_wiring.py
Exit 0; observed output excerpt:
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired
Log SHA-256: 21f676f458451525535ab0ef73d6a9c0d9774e16d76a4d19e396ad34936dd39c

Command:
./scripts/check_no_planning_docs.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_uring_lane_lib_only.sh
Exit 0; observed output excerpt:
OK: no 'uring_'-matching test functions under crates/ecstore/tests; --lib is safe
Log SHA-256: 43f3f851e27ec2ab9e34d6b05fefdede38af992699c473904bfe6f53a46d52e3

Command:
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast
Exit 0; wall 234.399s.
Observed output:
    Finished `test` profile [unoptimized + debuginfo] target(s) in 3m 49s
    Starting 1 test across 1 binary (5871 tests skipped)
  SETUP PASS [   0.006s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"'
        PASS [   0.440s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item
     Summary [   0.586s] 1 test run: 1 passed, 5871 skipped
Log SHA-256: 34e973f86e8dc2102c1a0e483e0ac802ce7df3d1b40a603932f62c0805c4de27

Command:
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over) | test(=store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence) | test(=store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable)' --no-fail-fast
Exit 0; wall 4.010s.
Observed output:
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.43s
    Starting 3 tests across 1 binary (5869 tests skipped)
  SETUP PASS [   0.005s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"'
        PASS [   0.068s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over
        PASS [   0.637s] rustfs-ecstore store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence
        PASS [   0.846s] rustfs-ecstore store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable
     Summary [   0.987s] 3 tests run: 3 passed, 5869 skipped
Log SHA-256: b5da2a927ff18ca9aa217762f7186a6d4371bd9e8feedd7b77940e10b64dffac

Command:
cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
Exit 0; wall 1084.508s.
Observed output:
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 18m 03s
Log SHA-256: 28f98be7970faaa71cd118da1da4e6e2e3d23b9dda7e6449eaa7a37e741260b5

Compiled mutation command (unchanged for each red and restored run):
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast

M1: reverse only this test-local constructor flag from false to true. The mutant compiled, then failed the original recovered-item queue assertion after 30.481s. Restoring the entire observed candidate passed in 0.460s. Red log 55bf3b1535529ac0bcfb8071f9b56f9f40631007a85c315367e7e93786d65293; restored log 204d5fbe4574a7917f5e740a795d864fe28a7e7fb5e33d27b0a9cf68e2f17f52.

M2: replace only the actual awaited enqueue with false, consuming its original inputs. The mutant compiled, then failed the same original queue assertion after 30.516s. Restoring the entire observed candidate passed in 0.457s. Red log 89186330413215e231e2b06bf013cc31097ea8f4a106d91421a36d6c8986a7fd; restored log e848b42d2302401afa5dbcd9e37bf76d2e56a96ab6de5a488513b997f53c5e9b.

All temporary observation and mutation code was removed before final acceptance. Historical baseline and diagnostic failures remain recorded; they are not mutation credit. The separate Object Lock intent-retry diagnostic still fails, and these native macOS checks do not establish Linux CI recovery.

* refactor(rpc): serve internode rpc bodies with the owned Body (#8441)

* test(rpc): pin the wire bytes of every internode rpc response body

rustfs/backlog#2747 batch 4, step 1 of 2, on the s3s bodies the rpc paths
serve today. Each response body builder in storage/rpc/http_service.rs is
served through hyper's HTTP/1.1 server and the raw bytes the peer reads are
compared with goldens captured from this commit: status, content-length or
chunked framing, chunk boundaries, the terminating chunk, HEAD, and the cut
connection a failure after data produces. The capability responses carry a
per-process proof, so their length header and payload are checked against
the body they encode instead.

The read_file_stream response builder moves into read_file_stream_response
so the test reaches the same code the handler runs; behaviour is unchanged.

* refactor(rpc): serve internode rpc bodies with the owned Body

rustfs/backlog#2747 batch 4, step 2 of 2. The internode rpc HTTP paths
build their responses with rustfs_s3_types::Body: Body::from_stream
replaces Body::from(StreamingBlob::wrap(..)) at the six stream sites, and
the in-memory and empty bodies keep their constructors. InternodeRpcService
converts its responses into the wrapped service's response body through
From<Body>, so the server keeps composing it with the s3s service as
before. The wire-byte goldens from the previous commit pass unchanged;
storage/rpc no longer names s3s.

* fix(rpc): simplify wire test response builder type

Name the boxed response builder used by the capability wire test so
all-target Clippy can check it without a type-complexity warning.

Validation (LANG=en_US.UTF-8, CARGO_INCREMENTAL=0):
- cargo clippy -p rustfs --all-targets -- -D warnings: exit 0.
- cargo nextest run --profile ci -p rustfs --lib -E 'test(/storage::rpc::/)': 279 tests run, 279 passed, 4842 skipped.
- All 19 quick-check commands, including script-tests, both Python scripts and cargo fmt --all --check: exit 0.
- Footprint baseline regenerated: files=194, s3_error_lines=1584, ecstore_files=36; test-count baseline regenerated: 52 crates, 21228 tests, 164 ignored.
- All 54 Cargo.toml files parse without duplicate keys.

No assertion was added or changed by this lint fix. R1-R8 are retained
in the existing PR verification record.

* fix(ecstore): resume fenced Object Lock creation intents

* fix(ecstore): bound creation retry futures on the stack

* fix(ecstore): require native proof for creation retries

* fix(ecstore): bound bucket creation stack usage (#8466)

* fix(ecstore): retain creation fences through rollback

Keep lifecycle, metadata-transaction, and namespace guards around rollback and metadata cleanup. Preserve the original publication error when a guard is lost. Build the timestamp retry fixture through the real interrupted producer so it checks a valid pending creation.

Validated source tree: 4e8e207999.
Two independent source reviews found no blocking regression.

Validation (LANG=en_US.UTF-8):
- bash -c 'shellcheck --version && actionlint' — exit 0
- cargo fmt --all --check — exit 0
- ./scripts/check_unsafe_code_allowances.sh — exit 0
- ./scripts/check_layer_dependencies.sh — exit 0
- ./scripts/check_architecture_migration_rules.sh — exit 0
- ./scripts/check_logging_guardrails.sh — exit 0
- ./scripts/check_error_other_format_ratchet.sh — exit 0
- ./scripts/check_no_tokio_io_uring.sh — exit 0
- ./scripts/check_extension_schema_boundaries.sh — exit 0
- ./scripts/check_body_cache_whitelist.sh — exit 0
- ./scripts/check_s3s_footprint.sh — exit 0
- ./scripts/check_test_count_ratchet.sh — exit 0
- ./scripts/check_fips_wording.sh — exit 0
- ./scripts/check_embedded_secrets.sh — exit 0
- make script-tests — exit 0
- python3 ./scripts/test_object_generation_protocol_model.py — exit 0
- python3 ./scripts/check_test_wiring.py — exit 0
- ./scripts/check_no_planning_docs.sh — exit 0
- ./scripts/check_uring_lane_lib_only.sh — exit 0
- cargo nextest run --profile ci -p rustfs-ecstore --lib --locked --no-fail-fast — 5894 passed, 0 skipped
- cargo check -p rustfs-ecstore --all-targets --locked — exit 0
- cargo nextest run --profile ci -p rustfs --lib --locked --no-fail-fast -E 'test(=admin::handlers::table_catalog::tests::cold_strong_registration_completes_while_other_bucket_backup_waits) | test(=admin::handlers::table_catalog::tests::commit_backend_borrowed_put_preserves_owner_authorization_and_precondition) | test(=admin::handlers::table_catalog::tests::disable_table_bucket_preserves_objects_and_can_be_reenabled) | test(=admin::handlers::table_catalog::tests::load_table_handler_negotiates_vended_credentials_without_breaking_metadata_only_callers) | test(=admin::handlers::table_catalog::tests::namespace_property_handler_updates_object_backed_catalog_and_maps_errors) | test(=app::object::get::tests::execute_get_object_resumes_from_relocated_pool_without_splicing_body) | test(=table_catalog::store::tests::catalog_writer_loss_prevents_recovery_storage_commit)' — 7 passed
- cargo nextest run --profile ci -p rustfs --lib --locked --no-fail-fast -E 'test(unsupported_remote_disk_)' — 2 passed

Additional fault-injection evidence remains incomplete. Timestamp M1 reached its original assertion; its restore run is being collected. M2–M5 and the three planned rollback faults have not run. An automated risk check interrupted the timestamp proof task; no assertions or validation settings were weakened.

* fix(ecstore): box nested bucket rollback state

Keep the guarded rollback future behind a heap allocation so bucket creation does not store its nested state inline. The rollback body, all three fence references and pending-proof revocation order are unchanged.

The same three original Linux tests overflowed on both e611 CI attempts. Local format, ecstore test compilation and all 19 quick-check steps pass; the original Linux feature lanes still need to validate this candidate.

* test(ecstore): isolate rollback guard loss fixtures

Run the three rollback metadata-cleanup cases in separate test processes so reused /proc/self/fd aliases from earlier cases cannot invalidate their creation witnesses. Keep the original fixture, guards, assertions and io_uring test selection, and verify that each child runs the intended case successfully.

Update the generated test-count baseline for the added macro arm.

Validation: cargo check -p rustfs-ecstore --tests; cargo fmt --all --check; all 19 quick-check commands, including make script-tests and the two Python checks. Linux io_uring acceptance and fault controls remain pending.

* refactor(replication): move the replication config onto gateway types (#8443)

* wip: T1.4 replication family

* refactor(replication): move the replication config onto gateway types

The replication configuration family leaves s3s: rustfs-replication no longer depends on s3s, and ECStore bucket metadata, the scanner and the rustfs app/admin layers hold rustfs_gateway_types::persistence::PersistedReplicationConfiguration. The persisted XML goes through the gateway persistence codec behind a new BucketConfigXml trait (the 39 byte goldens pass with unchanged fixtures); the legacy s3s edge converts once in storage/ecfs.rs through storage/s3_compat/replication.rs; the scanner data usage cache keeps its MessagePack bytes through a serde mirror pinned by bytes recorded from s3s 5761ddfe. rustfs-gateway-types 0.32.0 is pinned once (rev 772d3385). Refs rustfs/backlog#2745.

* test: use S3 serializer for list response tests

* fix(ci): restore integration test compilation

Gate the diagnostic test import with its feature, initialize the conditional
MetadataWriteStore fields explicitly, and decode the scanner replication fixture
through its existing cache boundary. Preserve the original fixture and assertions.

Validation was run on tree 544acb80c0 at 6003d4b8.
LANG=en_US.UTF-8; no source changes follow validation.

Quick checks: 19/19 passed.
- shellcheck --version && actionlint: exit 0
- cargo fmt --all --check: exit 0
- ./scripts/check_unsafe_code_allowances.sh: exit 0
- ./scripts/check_layer_dependencies.sh: exit 0
- ./scripts/check_architecture_migration_rules.sh: exit 0
- ./scripts/check_logging_guardrails.sh: exit 0
- ./scripts/check_error_other_format_ratchet.sh: exit 0
- ./scripts/check_no_tokio_io_uring.sh: exit 0
- ./scripts/check_extension_schema_boundaries.sh: exit 0
- ./scripts/check_body_cache_whitelist.sh: exit 0
- ./scripts/check_s3s_footprint.sh: exit 0
- ./scripts/check_test_count_ratchet.sh: exit 0
- ./scripts/check_fips_wording.sh: exit 0
- ./scripts/check_embedded_secrets.sh: exit 0
- make script-tests: exit 0
- python3 ./scripts/test_object_generation_protocol_model.py: exit 0
- python3 ./scripts/check_test_wiring.py: exit 0
- ./scripts/check_no_planning_docs.sh: exit 0
- ./scripts/check_uring_lane_lib_only.sh: exit 0

Darwin validation:
- cargo clippy --locked -p rustfs-ecstore -p rustfs-scanner --all-targets -- -D warnings: exit 0
- cargo nextest run --locked -p rustfs-ecstore -p rustfs-scanner --lib -E 'test(config::com::tests::save_config_preserves_retryable_system_volume_errors) | test(scanner_io::tests::replication_maintenance_refreshes_compacted_stats_across_cycles_and_retries)' --test-threads 1 --no-fail-fast: exit 0, 2/2 passed
- cargo clippy --locked -p rustfs-ecstore -p rustfs-scanner --all-targets --features rustfs-ecstore/diagnostic-read-version-caller,rustfs-scanner/diagnostic-read-version-caller -- -D warnings: exit 101
- cargo nextest run --locked -p rustfs-ecstore -p rustfs-scanner --lib --features rustfs-ecstore/diagnostic-read-version-caller,rustfs-scanner/diagnostic-read-version-caller -E 'test(config::com::tests::save_config_preserves_retryable_system_volume_errors) | test(scanner_io::tests::replication_maintenance_refreshes_compacted_stats_across_cycles_and_retries) | test(cluster::rpc::runtime_sources::tests::) | test(config::com::tests::diagnostic_config_write_retains_origin_and_api_for_results_and_cancellation)' --test-threads 1 --no-fail-fast: exit 0, 14/14 passed

Diagnostic strict Clippy remains red on four unchanged production lints:
runtime_sources.rs if_same_then_else and three redundant_locals reports in
set_disk/core/metadata_read.rs and set_disk/read.rs. Their source ranges and
feature wiring match the base; the base was not compiled separately.

Linux aarch64 validation (Rust/Clippy 1.99.0, UID 501, jobs=1):
- cargo clippy --all-targets -- -D warnings: exit 0; Finished dev profile in 68m 12s
- cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings: exit 0; Finished dev profile in 49m 45s
Source HEAD/tree and clean status matched before and after both commands.
No compiler or stack overrides were used. protoc 35.1 was installed from its
verified ARM64 release; flatc was absent and not invoked by the workspace build scripts.
The local environment differs from hosted CI.

* fix(ecstore): recover delete batches with missing versions

Treat an explicit non-null version as a no-op when it is absent from both recovery sources. Keep rejecting missing null versions, delete-marker mismatches and conflicting local generations.

Add returning-disk and partial-receipt regressions, extend journal boundary checks, and regenerate the ecstore test-count baseline. Production faults exercise the 24 added assertion sites and 7 semantic checks: 33 faults failed at the expected sites and passed after full restoration.

Validation (LANG=en_US.UTF-8, one build job):

cargo nextest run --locked -p rustfs-ecstore --lib -E 'test(delete_recovery) | test(quorum_delete_cleanup_coordinator_binds_single_and_batch) | test(crash_delete_retry_batch_reclaims_inline_and_reserved_external)' --test-threads 1 --no-fail-fast
Summary [  34.580s] 69 tests run: 69 passed, 6058 skipped; exit 0

cargo clippy --locked -p rustfs-ecstore --all-targets -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 5m 37s; exit 0

All 19 final Quick Checks completed with exit 0:

- shellcheck --version && actionlint
- cargo fmt --all --check
- ./scripts/check_unsafe_code_allowances.sh
- ./scripts/check_layer_dependencies.sh
- ./scripts/check_architecture_migration_rules.sh
- ./scripts/check_logging_guardrails.sh
- ./scripts/check_error_other_format_ratchet.sh
- ./scripts/check_no_tokio_io_uring.sh
- ./scripts/check_extension_schema_boundaries.sh
- ./scripts/check_body_cache_whitelist.sh
- ./scripts/check_s3s_footprint.sh
- ./scripts/check_test_count_ratchet.sh
- ./scripts/check_fips_wording.sh
- ./scripts/check_embedded_secrets.sh
- make script-tests
- python3 ./scripts/test_object_generation_protocol_model.py
- python3 ./scripts/check_test_wiring.py
- ./scripts/check_no_planning_docs.sh
- ./scripts/check_uring_lane_lib_only.sh

test-count ratchet OK: 52 crates, 22009 tests, 165 ignored match .config/test-count-baseline.txt
PASS: cold stampede script self-test
model: states=51023, transitions=167520, terminal=732, chosen_candidates=2
multi-slot round schedules: 640
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired

* fix(ecstore): retry tier commit intent lock contention

Retry transient namespace lock timeouts within the existing three-attempt
CAS budget. Reload and verify the original mutation identity after
contention, and preserve the final typed error when attempts are exhausted.
Cover save/read contention, exhausted retries, replacement identities and
CAS reconciliation without changing the existing timeout or lock policy.

Validation:
- cargo nextest run --locked -p rustfs-ecstore --features test-util --lib -E 'test(tier_mutation)' --test-threads 1 --no-fail-fast: 52 passed.
- cargo nextest run --locked -p rustfs-ecstore --features test-util --lib -E 'test(intent_advance_) | test(=services::tier::tier::tests::caller_cancellation_after_durable_prepare_does_not_hide_the_mutation)' --test-threads 1 --no-fail-fast: 5 passed.
- cargo clippy --locked -p rustfs-ecstore --all-targets -- -D warnings: passed.
- cargo clippy --locked -p rustfs-ecstore --all-targets --features test-util -- -D warnings: passed.
- 27 mutation pairs failed at the intended assertions and passed after restoring the production code.
- bash scripts/check_test_count_ratchet.sh --update, then bash scripts/check_test_count_ratchet.sh: 52 crates, 22013 tests, 165 ignored; ecstore increases from 6581 to 6585.

* refactor(lifecycle): replace s3s domain types with Gateway DTOs

Keep lifecycle rules, restore requests, Object Lock, and scanner cached tags in domain types. Convert at the existing HTTP boundary while preserving legacy XML, scanner cache bytes, date ordering, and tombstones.

Linux test validation before the final import-path-only adjustment:

cargo nextest run -p rustfs-ecstore bucket_config_goldens
39 passed; 6211 skipped by selector.

cargo nextest run --lib -p rustfs -E 'test(/test_zero_rule_lifecycle_tombstone_recognition/) | test(/test_local_lifecycle_staleness_axis_selection/) | test(/test_lifecycle_staleness_preserves_legacy_submillisecond_axis/) | test(/test_merge_incoming_lifecycle/) | test(/test_lifecycle_expiry_statement_matrix/) | test(/test_lifecycle_expiry_subset_xml/) | test(/put_bucket_lifecycle_validation_errors_keep_their_s3_code/) | test(/execute_put_bucket_lifecycle_configuration_rejects_missing_configuration/) | test(/execute_restore_object_rejects_missing_restore_request/) | test(/execute_restore_object_rejects_select_type/) | test(/execute_restore_object_maps_failures_to_typed_s3_errors/) | test(/execute_restore_object_returns_internal_error_when_store_uninitialized/) | test(/restore_request_conversion_preserves_legacy_output_values/) | test(/restore_request_conversion_preserves_presence_and_raw_enums/) | test(/restore_request_conversion_distinguishes_absent_and_empty_containers/)'
25 passed; 5318 skipped by selector.

cargo nextest run --profile ci -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-replication -p rustfs-scanner
7872 passed; 21 skipped under the existing CI profile.

Previously verified mutation controls: 39 production faults, 41 RED/GREEN pairs.

cargo clippy --locked -p rustfs -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-scanner --all-targets -- -D warnings
Passed in 40m 45s before the import-path adjustment, then in 1m 59s on
the final source with warnings denied. The final adjustment imports the
existing helper through the object domain facade; fmt and the architecture
check also passed.

Part of rustfs/backlog#2745. The remaining ecstore configuration families follow separately.

* refactor(admin): move error types to rustfs-s3-types (#8440)

* refactor(protocols): move error types to rustfs-s3-types by codemod

T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and
the error items of protocols move off s3s by scripts/codemods, run with
--hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it
downcasts the backend error, whose runtime type is still s3s::S3Error.

- rustfs/src/protocols/client.rs: the two webdav session methods now
  return rustfs_s3_types::S3Result and convert the s3s error with From.
- codemod: a child module that calls s3_error! through `use super::*`
  gets its own `use rustfs_s3_types::s3_error;` once its parent imports
  that macro by name; otherwise it is listed as GLOB.
- s3-types: a table test pins code, status and the s3s error the legacy
  edge renders for every variant a moved call site names.
- s3_error_lines 1585 -> 1584 (a lifecycle comment reworded).

* refactor(admin): move error types to rustfs-s3-types (WIP, no build)

T1.2 batch 4 (rustfs/backlog#2743), work in progress: rustfs check
still fails with 16 errors (listed in the PR body).

- codemod: --facade PATH treats a crate-local re-export of s3s's error
  items like s3s for the four names (9 new self-tests, 9 mutants red).
- codemod output for rustfs/src/admin and rustfs/src/site_replication
  (--facade crate::storage_api::site_replication::s3); 6 glob children
  get an explicit rustfs_s3_types::s3_error import.
- admin/router.rs: the Operation trait returns rustfs_s3_types::S3Result.
- error.rs: From<ApiError> builds the rustfs-s3-types error; the s3s
  conversion is derived from it.

* fix(admin): convert error types at the s3s boundary

Batch 4 of T1.2 left the admin and site_replication modules returning
rustfs_s3_types::S3Error while their callers and callees on the storage
and auth side still speak s3s::S3Error. Convert at each edge with the
existing From impls in rustfs-s3-types (compat_s3s.rs) instead of
changing any code, message or status:

- admin/auth.rs, admin/handlers/object_zip_download.rs,
  admin/handlers/table_catalog/mod.rs: map the s3s result returned by
  check_key_valid / authorize_request / authorize_internal_object_request
  into the admin error type with .map_err(Into::into).
- storage/ecfs.rs: the s3s-facing side maps the admin site_replication
  gate results the other way.
- admin/handlers/pools.rs: ApiError.code stays s3s::S3ErrorCode, so the
  rustfs code is converted with .into() when the struct is built, in
  production and in the two tests that build or compare it.
- admin/handlers/heal.rs, admin/runtime_sources.rs: the router returns
  s3s errors (S3Route impl); the expected rustfs code is converted with
  .into() before the comparison.
- admin/handlers/table_catalog/mod.rs: iceberg_rest_error takes
  &'static str so the Cow<'static, str> custom code borrows it; every
  caller already passes an ICEBERG_ERROR_* constant or a &'static str.
- admin/handlers/quota.rs: drop a now same-type .into() after s3_error!.
- storage_api.rs, site_replication/mod.rs: drop re-exports and an import
  that no longer have users (clippy -D warnings).

cargo check -p rustfs --all-targets and cargo clippy -p rustfs
--all-targets -- -D warnings are clean; the s3s footprint baseline is
unchanged at files=172, s3_error_lines=303, ecstore_files=28.

Refs rustfs/backlog#2743, rustfs/backlog#2734.

* refactor(ecstore): move bucket configs onto gateway persisted types (#8709)

* refactor(ecstore): move bucket configs onto gateway persisted types

Finish the ECStore batch of T1.4 (rustfs/backlog#2745). The remaining
eleven bucket configuration families (Website, CORS, Tagging, Versioning,
ObjectLock, Encryption, Notification, Logging, Accelerate, RequestPayment,
PublicAccessBlock) now read and write their stored XML through the
rustfs-gateway-types persistence and cors_tagging codecs instead of the
s3s codec, so the legacy s3s bucket-config codec and its
s3gate-metadata-xml compat marker are removed.

- ecstore: BucketConfigXml is implemented by the gateway codec for every
  family; metadata, metadata_sys, versioning, object lock, replication and
  lifecycle consumers take the persisted types. Region moves to
  rustfs_s3_types::Region, and the ETag conversion to s3s moves to the
  rustfs edge with its tests. Only warm_backend.rs and warm_backend_s3.rs
  still reference s3s, held for T1.5 (rustfs/backlog#2746).
- scanner: object lock cache, item actions and data usage use the
  persisted types.
- rustfs: the legacy S3 edge converts each family between s3s DTOs and the
  persisted types in storage/storage_api.rs by writing one codec's XML and
  reading it with the other; Tagging converts field by field.
- Bump the rustfs-gateway-types pin to 21283e87.

The 39 persisted-byte goldens are unchanged and pass, so a document the
old codec wrote reads back and re-encodes to the same bytes. s3s footprint:
files 181 -> 146, ecstore_files 28 -> 2, s3_error_lines unchanged.

* test(app): prepare the PutBucketEncryption config from the request

The uninitialized-store test passed an empty persisted configuration, which the use case now validates before it reaches the store check, so it answered MalformedXML instead of InternalError. Convert the request's configuration the way the S3 edge does.

* style(app): format the PutBucketEncryption test

* refactor(s3-client): move remote error codes and DTO enums off s3s (#8783)

Part of T1.5 (rustfs/backlog#2746), batch 1.

* refactor(s3select): run queries on gateway DTO request shapes (#8794)

* refactor(s3select): run queries on an engine request built from gateway DTO shapes

Part of T1.5 (rustfs/backlog#2746), batch 2.

* fix(s3select): keep the select request conversion in the app layer

The layer guard forbids app -> storage; the conversion moves from storage/s3_compat/select.rs to app/select_request.rs unchanged.

* fix(heal): retry transient metadata conflicts (#8812)

* fix(heal): retry transient metadata conflicts

* chore: update heal test count baseline

* test: reserve nextest capacity for progress probes (#8818)

* test(e2e): use a non-default region for the differential negative control

Since main's region-aware CreateBucket and GetBucketLocation (#8739, #8743), us-east-1 answers an empty LocationConstraint and 200 for an owned re-create, so the control no longer reported its own region. Also refresh the e2e-full linux selection from the CI listing (682 tests).

* refactor(protocols): move StorageBackend to gateway types (#8825)

* refactor(protocols): move storage backend to gateway types

* test: update e2e test count baseline

* fix: preserve copy source keys and unknown lengths

* test: update protocols test count baseline

* fix(protocols): preserve typed SFTP backend errors

* test(protocols): use gateway list input in policy tests

* fix(protocols): clear clippy failures after T1.5 merge (#8831)

fix(protocols): clear clippy failures after merge

* fix(ecstore): finalize restore metadata after local copy-back (#8836)

* fix(ecstore): finalize restore metadata after local copy-back

* test(ecstore): ratchet added restore finalization cases

* fix(ecstore): keep restore copyback compatible with older peers

* fix(ecstore): gate restore copyback on fleet compatibility

* fix(ecstore): retain transition identity during restore copyback

* fix(ecstore): route committed restore copyback locally

* fix restore copyback cleanup

* fix(ecstore): honor restore probe test fallback

* refactor(storage): migrate option errors to domain types (#8839)

* refactor(s3): migrate private multipart validation errors (#8844)

* refactor(auth): derive policy conditions from AuthzFacts (#8442)

* refactor(auth): extract authorization facts

* test(e2e): update smoke selection baseline

* test(e2e): update full selection baseline

* refactor(admin): migrate health and prefix usage to owned requests (#8887)

* refactor(admin): add owned health router boundary

* refactor(admin): migrate prefix usage to owned request

* refactor(admin): complete the owned admin request boundary (#8949)

* refactor(admin): complete the owned admin request boundary

Every type implementing the owned admin `Operation` now serves the legacy
router through one blanket impl that converts the s3s request and response
at that single edge, so handlers register with `AdminOperation(&Handler {})`
and need no adapter statics; `LegacyAdminAdapter` is removed.

`AdminRequest` gains the accessors handlers read today (`remote_addr`,
`client_info`, `request_context`, `req_info`, `req_info_mut`,
`install_req_info`, `envelope_mut`, `take_body`), the request helpers gain
envelope forms (`authenticate_envelope_credentials`,
`object_store_from_envelope`, the table-catalog `*_from_envelope` helpers,
`KmsAdminAudit::from_envelope`, `AccountAuditContext::from_envelope`,
`CallerIdentity::resolve_envelope`), `read_compatible_admin_body` accepts
either body type, and the owned `Body` gains `store_all_limited` and
`Default`. `admin::operation::test_support::TestRequest` builds owned
requests for handler tests. Unused accessors carry a per-method dead-code
expectation that each migrating handler batch removes.

No handler behavior changes; the remaining migrations follow in batches.

* chore(admin): add the T1.12 admin handler codemod

The script rewrites an admin handler file from the s3s request and response
types to the owned boundary (imports, signatures, request field and helper
calls, body access, response construction, and test request literals) and
leaves a TODO comment at every site it cannot rewrite mechanically. It is
idempotent and is removed once T1.12 completes.

* refactor(admin): migrate replication and KMS handlers to owned requests (#8951)

Move the T1.12 batch 2 admin handlers (site_replication, kms_keys, event,
replication, diagnostics, on_demand_migration, audit, account, kms_rekey,
kms_key_lifecycle, object_data_cache, tls_debug, integrity) from the s3s
request/response types to the owned AdminRequest/AdminResponse boundary.
Handler logic, authorization order, error codes and response bytes are
unchanged; the request/response plumbing now goes through the envelope
accessors and helpers introduced by the owned boundary.

The shared response helpers (utils::json_response, utils::empty_response,
handlers::admin_json_response, target_descriptor::build_json_response,
profile::authorize_profile_request) and the site replication repair
executor still answer with the s3s types, so the migrated files carry
owned-boundary forms of those helpers and convert the repair response at
its call edge. The streaming console-log body and the two devnull drains
move from the s3s byte stream to the owned Body stream/frame API.

Drop the dead_code expectations for the owned accessors and envelope
helpers this batch starts using, and lower the s3s footprint baseline to
the measured 99 files.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* refactor(admin): add owned forms of the shared admin response helpers

Handler batches were copying `json_response`, `empty_response`,
`admin_json_response`, and `build_json_response` into each migrated file
because the shared helpers still return the s3s response. Provide owned
forms next to each legacy helper (`owned_json_response`,
`owned_empty_response`, `owned_admin_json_response`,
`owned_build_json_response`, `authorize_profile_envelope`) with parity
tests against the legacy shape, derive `Debug` for `AdminResponse` so
tests can use `expect_err`, and fix three codemod rules: strip owned
string conversions in test credentials, annotate non-literal URI parses,
and leave a rebound request parameter immutable.

* refactor(admin): use the shared owned response helpers in batch 2 (#8954)

Replace the per-file owned-boundary copies of json_response,
empty_response, admin_json_response, build_json_response and
authorize_profile_request in the batch 2 admin handlers with the shared
owned_* helpers and authorize_profile_envelope that the boundary branch
now provides, and restore expect_err in the handler tests now that
AdminResponse implements Debug; the `.err().expect(..)` stand-ins were
failing the base branch's clippy step (clippy::err_expect under
-D warnings). The site replication repair call edge
keeps its conversion because the repair executor still answers with the
s3s response type.

Drop the dead_code expectations on the shared owned helpers this batch
now uses in non-test code.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* fix(admin): preserve legacy body limit error responses

* refactor(admin): move policy, ILM and batch 3 handlers to owned requests (#8956)

Batch 3 of T1.12 (rustfs/backlog#2788, parent rustfs/backlog#2734) moves
14 admin handler files to AdminRequest/AdminResponse and the owned
Operation trait: ilm_transition, policies, idp_compat, config_admin,
plugins_instances, profile_admin, group, heal, object_zip_download,
kms_key_metadata, durability, inspect_archive, sts and
gateway_key_inventory. Handler logic, authorization order, error codes
and response bytes are unchanged; test request literals use
test_support::TestRequest.

The handlers use the shared owned helpers (owned_json_response,
owned_admin_json_response, owned_build_json_response,
authorize_profile_envelope). The dead_code expectations on the
AdminRequest accessors and RequestEnvelope::uri_mut they now exercise
are removed. The legacy target_descriptor::build_json_response keeps
only its equivalence test caller and carries the matching expectation
until it retires.

object_zip_download builds the s3s request the S3 gate still reads from
the owned request facts; heal authenticates through
authenticate_envelope_credentials; the trace, inspect-archive and zip
bodies use Body::from_stream. extract_plugin_instance_filters takes the
URI so extensions.rs keeps calling it, and the two idp_compat
delegations stay on the legacy trait until their delegates migrate.

The s3s footprint baseline drops from 99 to 90 files.

* test(admin): anchor the replication diff body contract on the call

The source-contract test looked for `let body = read_compatible_admin_body(`
on one line; rustfmt wrapped that binding after the handler moved to the
owned request boundary, so the marker was no longer found. Anchor on the
call itself so formatting cannot change what the contract checks.

* refactor(admin): migrate table catalog and batch 1 to owned requests (#8955)

* fix(admin): preserve catalog body limit response bytes

* refactor(admin): migrate identity and cluster handlers to owned requests (#8966)

* refactor(admin): migrate identity and cluster handlers to owned requests

Move the T1.12 batch 4 admin handlers (user, service_account, oidc, tier,
system, pools, quota, kms_dynamic, kms_backup, rebalance, module_switch,
bucket_meta, cluster_snapshot, realtime, bucket_recovery, profile) from the
s3s request/response types to the owned AdminRequest/AdminResponse
boundary. All 90 impl Operation blocks now take AdminRequest and return
AdminResponse; handler logic, authorization order, error codes and response
bytes are unchanged. The handlers use the shared owned helpers
(owned_json_response, owned_admin_json_response, authorize_profile_envelope,
the envelope-form runtime sources and audit constructors) instead of local
copies.

profile.rs drops the s3s forms of authorize_profile_request and
profile_not_implemented_response now that profile_admin.rs is on the
envelope gate too; profile_not_implemented_response answers with an
AdminResponse. The realtime metrics stream moves from the s3s ByteStream
adapter to Body::from_stream, and the rebalance idle-404 document still
renders through s3s::S3Error::to_http_response with its body converted at
the edge.

The idp_compat delegators ImportIamV2 and AddServiceAccountLdap move to the
owned trait as well: with their targets on both traits, the method-call
delegation became ambiguous between the owned impl and the router blanket
impl, so they now take the owned request directly.

KmsAdminAudit::from_request loses its last production caller with this batch
and is deleted; its comparison test now asserts the envelope facts directly.

Lower the s3s footprint baseline to the measured 68 files.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* refactor(admin): remove the legacy admin helpers batch 4 superseded

Batch 4 moved the last production callers of the s3s-typed admin helpers
to their owned forms, which left `admin_json_response`, `json_response`,
`authenticate_request`, `authorize_admin_request`, `object_store_from_req`
and `object_store_from_extensions` unused, along with the `Body`,
`S3Request` and `S3Response` re-exports of the admin `s3` facade. Remove
them. The owned forms keep their documentation; the behavior tests of the
removed `json_response` now exercise `owned_json_response`, and the
compat-prefix test asserts the owned response directly.

Two tests existed only to compare a removed legacy helper with its owned
twin (`owned_json_response_matches_the_legacy_shape`) or to cover the
removed legacy gate (`authorize_admin_request_without_credentials_is_rejected`);
they go with the helpers, and the owned equivalents stay. The rustfs
test-count row drops from 5957 to 5955 and the s3s footprint from 68 to 66
files.

* refactor(admin): drop the legacy target descriptor JSON response helper (#8968)

`target_descriptor::build_json_response` returned the s3s response and
had no production caller left once the event, audit and plugin instance
handlers moved to `owned_build_json_response`; it survived only for the
legacy/owned equivalence test. Remove it, give the owned helper its own
documentation, and make the test assert the owned response directly:
status, `application/json`, the echoed `x-request-id` when present and
only then, and the body bytes. The file no longer imports s3s, so the
footprint baseline drops from 66 to 65 files.

* test(ecstore): pin native history on cancellation under the cancellable queue

main moved the local write path onto the disk blocking queue, which skips an
operation once its waiter is dropped, so a borrowed write cancelled while it is
queued no longer creates its file. The integration-only native publication
test still expected the queued open to run. Assert the new execution model
instead (the open never runs) while keeping its safety claims: the Started
record is taken before queueing and the cancelled publication never regrants
native history.

* chore(gateway): raise test-count rows for tests merged from main

* test(s3select): use the owned JSON input types in merged planner tests

* fix(ecstore): scope the merged direct-writer test hook to test builds

* fix(ecstore): create parents before admitted truncate opens

Align the heal tests with removal-disabled preservation and local-only outcome
fields, and refresh the e2e-full selection for the synced tests.

* chore(gateway): update test-count baseline after main sync

Main #9241 removed the node pool-metadata readiness cache with 8 tests
and added 6, so the rustfs row drops from 6440 to 6438. The ecstore and
s3select rows rise for tests added by #9237, #9239 and #9241.

* feat(scripts): add junit_diff.py for dual-stack JUnit comparison (#9275)

* ci(gateway): make the test-count ratchet a floor, not exact (#9278)

ci(gateway): make the test-count ratchet a floor instead of an exact baseline

Tests below a row still fail; tests above it, a member without a row and ignores below the row now pass with a note. Ignores above a row still fail. Adds contract tests wired into make script-tests.

* test(guards): read file plus split directory in text guards

Text guards that read one file with include_str! or by path broke, or silently stopped seeing the code they guard, when that code moved to another file. They now read the file they read today, which stays required, plus the sibling directory a mechanical split will populate (storage/legacy_edge, app/bucket_usecase, app/multipart_usecase, app/lifecycle_transition_api_test). Counts are summed over the set and split markers must occur exactly once across it. Refs rustfs/backlog#2765.

* chore(gateway): record ignored microbenchmarks merged from main (#9290)

chore(gateway): record the ignored microbenchmarks merged from main

main added three throughput microbenchmarks that are ignored by design
(#9217): one each in rustfs-checksums, rustfs-rio and rustfs-utils. The
test-count ratchet treats a higher ignored count as a failure until the
row is raised, so raise those three rows to 1 and tighten the test rows
to the merged tree with --update.

* test(guards): avoid a typos false positive in a fixture

* ci(s3-tests): add reference parity lane against pinned 1.0.5 (#9288)

* chore(app): add codemod that splits the multipart use case per operation

* refactor(app): split multipart_usecase.rs into one file per operation

Mechanical move produced by scripts/codemods/split_multipart_usecase.py:
no signature, behaviour or test-body change. The struct, its
constructors and the shared helpers stay in multipart_usecase.rs; each
operation, its helpers and its tests move to multipart_usecase/<op>.rs.
New files reach s3s only through crate::app::storage_api::s3, which
gains the dto and xml re-exports; the nextest filter that pins one test
follows it to its new module path.

* fix(app): keep relative paths valid in moved multipart code

Code that moves from multipart_usecase.rs into an operation file is one
module deeper, so its super:: paths gain a super:: (the codemod now does
this) and the regenerated split carries the result.

---------

Co-authored-by: rustfs-cla-assistant[bot] <273121464+rustfs-cla-assistant[bot]@users.noreply.github.com>
2026-11-11 18:59:17 -05:00
Chris An f97fcf36b2 refactor(app): split bucket_usecase into one file per config family (#9317)
* feat(server): add the RUSTFS_S3_STACK selector and dual-stack e2e lane (#8411)

* refactor(server): group the s3s-compat tower layers under legacy_compat (#8410)

Move the ten tower layers that patch s3s behaviour out of
rustfs/src/server/layer.rs into rustfs/src/server/legacy_compat/, one file
per layer with its tests. Each file header names the s3s behaviour it
patches, the gateway mechanism that replaces it (Replaced by:), and the task
that removes it (Removed by: T2.12, rustfs/backlog#2771). The layers keep
their code, their order in external_service_stack!, and their behaviour; no
layer is deleted.

Add server::http::tests::stack_census, which renders the ordered service
chain of the real external_service_stack! expansion and compares it with a
committed constant, so a reorder, insertion or removal is a visible diff.

Route the two s3s error types the moved layers use through a
storage_api::server::legacy_compat shim so the split adds no s3s-importing
file, and lower the s3s footprint baselines to the measured values (files 209
to 208, s3_error! lines 1586 to 1585). Keep the server runtime-source bypass
guard covering the moved code.

Refs rustfs/backlog#2787, rustfs/backlog#2734

* feat(s3-types): add the RustFS-owned S3 error type and s3_error! macro (#8406)

`rustfs-s3-types` now provides `S3ErrorCode` (generated by build.rs from
src/codes.txt, the grep-derived call surface, plus `Custom`), `S3Error`,
`S3Result`, `StdError` and the `s3_error` macro with the same call syntax
as s3s, so the T1.2 codemod is a `use` swap. Statuses are hand-written and
pinned to the s3s behaviour RustFS ships today by an oracle test that uses
s3s as a dev-dependency only. The `From` conversions for the legacy edge
live behind the off-by-default `compat-s3s` feature (the orphan rule
forbids them in rustfs/src/error.rs); `rustfs` enables it.

Refs rustfs/backlog#2742, rustfs/backlog#2734

* ci: add the per-crate test-count ratchet (#8404)

Quick Checks now fails any PR that lowers the number of test attributes
in a workspace crate or adds #[ignore], measured per crate against the
exact rows of .config/test-count-baseline.txt (rustfs/backlog#2734
section 1 rule 3). The counter widens the spec's literal
#[(tokio::)?test] to the tokio attributes that carry arguments and both
spellings of #[test_case(...)], and anchors at line start so
commented-out attributes do not count. The initial baseline covers 52
crates, 21077 tests and 164 ignored.

Refs rustfs/backlog#2738
Refs rustfs/backlog#2734

* test(ecstore): pin the persisted bucket-config XML bytes for 13 families (#8408)

Before any DTO or codec is replaced, pin the exact bytes that
deserialize followed by serialize produces for every persisted
bucket-configuration XML family in its three source shapes (AWS
documentation, MinIO-written, RustFS-written), and the bytes RustFS
writes today for a typed value. Byte equality only: a structural
comparison would let a codec replacement change the persisted encoding
unnoticed.

The generator script refuses to overwrite existing goldens without
--force, refuses on dirty codec paths, cannot report success from its
write mode, and records the generating commit in the fixtures README.

Refs rustfs/backlog#2744, rustfs/backlog#2734

* ci: raise the test-count baseline to the integration tree (#8426)

* ci: make the s3s footprint baseline exact and lower-only (#8403)

* ci: make the s3s footprint baseline exact and lower-only

The s3s footprint guard compared three counters against ceilings held in
the script itself, so a ceiling could sit above the tree and a drift was
fixable by raising it (rustfs/rustfs#5964 raised the s3_error! ceiling
to go green). The three values now live in
.config/s3s-footprint-baseline.txt as files, s3_error_lines and
ecstore_files, the script reads only that file, and a measurement that
differs from its baseline in either direction fails: growth prints
"s3s footprint grew: <measured> > <baseline>", shrinkage prints
"lower the baseline to <measured> in this PR". The counting method is
unchanged; the baseline is the current measurement of
gateway/integration (209 / 1585 / 36). The header records the end state:
once all three reach 0, T4.2 switches the script to failing on any
occurrence.

Refs rustfs/backlog#2739, rustfs/backlog#2734

* ci: add the s3s edge allowlist mode to the footprint guard (#8405)

`scripts/check_s3s_footprint.sh --mode allowlist` fails on every Rust
file that references s3s (`s3s::` or `s3s_sigv4`) and is not matched by
an entry of the new `.config/s3s-edge-allowlist.txt`; `--dry-run` lists
those files and exits 0. The allowlist carries the initial edge from the
spec, and the script rejects an entry that names or matches anything
under crates/ other than crates/e2e_test. The baseline mode and the
argument-less CI invocation are unchanged.

Refs rustfs/backlog#2741, rustfs/backlog#2734

* test(ecstore): reach the bucket-config codec through the test facade (#8428)

* refactor(s3-types): own the HTTP Body and drop s3s from three crates (#8431)

Add rustfs_s3_types::Body, a BoxBody<Bytes, StdError> wrapper with
empty() and From<Bytes | Vec<u8> | String | &'static str>, and the
legacy From conversions to and from s3s::Body behind compat-s3s.

The signer is now generic over the request body: it only reads and
writes the request head, so it needs no body type and its callers keep
theirs. s3-client sends its requests with the owned Body, madmin signs a
bodiless head, and targets drops its two S3Request/S3Response header
wrappers in favour of extract_params_header.

rustfs-signer, rustfs-madmin and rustfs-targets no longer depend on
s3s; the s3s footprint baseline drops from 209 to 202 files.

Refs rustfs/backlog#2747, rustfs/backlog#2734

* refactor(ecstore): move S3ErrorCode to rustfs-s3-types by codemod (#8432)

* test(ecstore): pin the InvalidRange responses of new_getobjectreader

Both error branches of new_getobjectreader build an ErrorResponse with S3ErrorCode::InvalidRange, and no test observed either code. Pin the code, its wire name and the location fields of each branch before the code type moves to rustfs-s3-types (rustfs/backlog#2743), so a changed code on either branch fails a test. The ecstore test-count row rises by the two tests.

* chore(scripts): add the s3s-to-rustfs-s3-types error codemod

Task T1.2 of rustfs/backlog#2734 (rustfs/backlog#2743) moves S3Error, S3ErrorCode, S3Result and s3_error! from s3s to rustfs-s3-types by script, one crate closure per PR. The codemod rewrites use trees (splitting mixed s3s imports) and qualified code paths, never comments or literals, adds the rustfs-s3-types dependency, drops s3s only from a crate that no longer names it, refuses every file of the s3s edge allowlist, takes --hold for a file whose type is fixed outside the closure, and prints a per-crate census. A second run changes nothing. Its self-test runs under make script-tests.

* refactor(ecstore): take S3ErrorCode from rustfs-s3-types

Output of scripts/codemods/s3_error_to_s3_types.py over crates/ecstore, crates/lifecycle, crates/scanner and crates/replication (rustfs/backlog#2743 batches 1 and 2). The ErrorResponse code field, the two InvalidRange constructions in new_getobjectreader and an unused tier import now use rustfs_s3_types::S3ErrorCode; the variant names and wire names are unchanged. lifecycle, scanner and replication reference none of the four error names. warm_backend.rs and warm_backend_s3.rs are held: they compare against rustfs_s3_client::ErrorResponse.code, which stays the s3s type until s3-client moves.

* ci(footprint): count only s3_error! lines still on s3s's macro

s3_error_lines counted every line holding the text `s3_error!`, which
also matches the RustFS-owned `rustfs_s3_types::s3_error!`. A codemod
that moves call sites onto rustfs-s3-types therefore could not lower it
(rustfs/backlog#2743, decision 3; rustfs/rustfs#8432, gotcha 2).

The counter now applies a textual, per-file and per-line rule that errs
on the side of counting (full rule in the script header):
- occurrences spelled `rustfs_s3_types::s3_error!` never count;
- a line holding `s3s::s3_error!` always counts;
- any other `s3_error!` line counts unless its file is cleared, that is,
  it imports the macro from rustfs_s3_types by name on a line starting
  with the use item (or defines it with `macro_rules! s3_error`) and does
  not also bring s3s's macro in (`use s3s::s3_error`, a `use s3s::{..}`
  tree naming it or a crate-root `*`, `use s3s::*`, or
  `#[macro_use] extern crate s3s`);
- fail closed: a file matching neither form (`use super::*`, a
  crate-local re-export, a renamed or glob rustfs_s3_types import, a
  commented-out import, a comment-only mention) counts.

Census of the 1585 lines on 96fc06776: 1478 in 64 files that import the
macro from s3s; 107 in 15 files matching neither form (105 through
`use super::*`, 1 qualified `s3s::s3_error!`, 1 comment line); none in
files that import it from rustfs_s3_types or define it. Every line stays
counted, so the baseline remains s3_error_lines=1585 (files=202 and
ecstore_files=36 unchanged). Rewriting rustfs/src/auth.rs's import to
rustfs_s3_types lowers the new counter to 1567; the old one stayed 1585.

`--self-test` (wired into `make script-tests`) copies the script into
22 fixture trees: 6 positive cases and 16 negative ones (lines that must
not count, forms that must not clear a file, and a missing baseline,
missing key or failing classification search that must fail). Each tree
has a control line that must count, and each run gets s3s text on stdin.
Seven cases were red against the old counter. 19 mutants each turn their
case red: old definition, flipped import test, dropped fail-closed
branch, dropped e2e_test exclusion, dropped s3s veto, dropped qualified
s3s rule, dropped rustfs-qualified strip, unanchored rustfs import,
renamed import accepted, nested glob taken as root glob, single-line
s3s trees only, dropped s3s glob forms, dropped macro_use form, dropped
definition clearing, ignored classification rg errors, error-line search
without its path, classification searches without their path, missing
baseline skipped, missing key skipped.

* test(e2e): compare two RUSTFS_S3_STACK servers over every S3 operation (#8435)

Add dual_stack_differential to e2e_test. It starts two rustfs processes
from the same binary, stack A on RUSTFS_S3_STACK=legacy and stack B on
RUSTFS_E2E_STACK_B (default legacy until rustfs/backlog#2734 T2.5),
sends both the same aws-sdk-s3 script covering the 73 methods of
impl S3 for FS (a normal request and a 4xx negative each), and compares
status, error code, an allowlist of response headers and a few typed
body fields. Error message text is never compared. Each run writes
<target>/dual-stack/<run>.json with compared_ops and the (op, field,
a, b) differences; any difference fails the test.

differential_detects_an_injected_difference starts B with
RUSTFS_REGION=eu-west-1 and requires GetBucketLocation to be reported.
The comparator is pure and unit-tested without a server.

Wire the module into the e2e-smoke filter and update the e2e-smoke and
e2e-full selection digests and the e2e_test test-count row.

Refs rustfs/backlog#2740

* feat(app): add the RustFS-owned RequestEnvelope and its s3s constructor (#8437)

* feat(app): add the RustFS-owned RequestEnvelope and its s3s constructor

* ci: classify the s3s edge adapters in storage/s3_compat as interface

* feat(s3-types): add Body::from_stream and move Swift off s3s::Body (#8438)

rustfs/backlog#2747 batch 3. `Body::from_stream` wraps a chunk stream of
unknown length over `StreamBody`/`BoxBody`: one data frame per chunk, the
first error ends the body, the size hint stays unknown, and dropping the
body drops the stream. Parity tests pin it to the s3s `StreamingBlob::wrap`
it replaces, at the `http_body` level and byte for byte through hyper.

The Swift handlers in rustfs-protocols build their responses as the owned
`Body`. `SwiftService` converts them into the wrapped S3 service's response
body type through `From<Body>`, so rustfs keeps serving `s3s::Body` at the
legacy edge unchanged. Request bodies stay Send-only (`UnsyncBoxBody`); the
two storage writers and the DLO/SLO segment streams get `Sync` from
`sync_wrapper::SyncStream`, which is what s3s's `http_body_unsync` mutex
provided before. The DTO and error uses of s3s in protocols are untouched
(T1.5 and T1.2).

* refactor(protocols): move error types to rustfs-s3-types by codemod (#8439)

T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and
the error items of protocols move off s3s by scripts/codemods, run with
--hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it
downcasts the backend error, whose runtime type is still s3s::S3Error.

- rustfs/src/protocols/client.rs: the two webdav session methods now
  return rustfs_s3_types::S3Result and convert the s3s error with From.
- codemod: a child module that calls s3_error! through `use super::*`
  gets its own `use rustfs_s3_types::s3_error;` once its parent imports
  that macro by name; otherwise it is listed as GLOB.
- s3-types: a table test pins code, status and the s3s error the legacy
  edge renders for every variant a moved call site names.
- s3_error_lines 1585 -> 1584 (a lifecycle comment reworded).

* test(connect): restore one-second drive request budgets

Apply the exact test-only repair already landed on main in 5972a09cc3. The filesystem probe and production CLI success test request one second and keep their existing one-second duration and parameter checks. All outcome, I/O, error, cleanup and resource-limit assertions remain.

Scope: rustfs/tests/connect_perf_drive.rs only. Patch-id: cb0a1dfc8f68a67727525adea7458e2cee7d09f8.

Validation environment: LANG=en_US.UTF-8, CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0, exclusive APFS-cloned CARGO_TARGET_DIR.

Quick Checks: every command from .github/actions/quick-checks/action.yml, with captured output and exit status below.

$ shellcheck --version && actionlint
ShellCheck - shell script analysis tool
version: 0.11.0
license: GNU General Public License, version 3
website: https://www.shellcheck.net
exit_code=0

$ cargo fmt --all --check
exit_code=0

$ ./scripts/check_unsafe_code_allowances.sh
exit_code=0

$ ./scripts/check_layer_dependencies.sh
Layer dependency guard passed (no new reverse dependencies/cycles).
exit_code=0

$ ./scripts/check_architecture_migration_rules.sh
Architecture migration rules passed.
exit_code=0

$ ./scripts/check_logging_guardrails.sh
✅ Logging guardrails check passed
exit_code=0

$ ./scripts/check_error_other_format_ratchet.sh
error(format!) ratchet passed (total call sites: 582).
exit_code=0

$ ./scripts/check_no_tokio_io_uring.sh
exit_code=0

$ ./scripts/check_extension_schema_boundaries.sh
Extension schema boundaries passed.
exit_code=0

$ ./scripts/check_body_cache_whitelist.sh
Body-cache whitelist guard passed.
exit_code=0

$ ./scripts/check_s3s_footprint.sh
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)
✅ s3s footprint baseline check passed
exit_code=0

$ ./scripts/check_test_count_ratchet.sh
test-count ratchet OK: 52 crates, 21222 tests, 164 ignored match .config/test-count-baseline.txt
exit_code=0

$ ./scripts/check_fips_wording.sh
FIPS wording guard passed (README.md and CHANGELOG.md contain no forbidden claims).
KMS wording guard passed (crates/kms claims no Vault KV2 Transit wrapping).
exit_code=0

$ ./scripts/check_embedded_secrets.sh
Embedded secret guard passed (no private key material or provider credential literal in the tree).
exit_code=0

$ make script-tests
Running script tests...
entrypoint credential policy tests passed
internode grpc A/B bench env tests passed
object batch benchmark enhanced tests passed
PASS package service script contract
hotpath warp A/B gate tests passed
hotpath warp ABBA tests passed
Scanner validation artifacts written to /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.1pFe6Kl1Yu/out
Scanner validation artifacts written to /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.1pFe6Kl1Yu/out-missing-pid
.............................
----------------------------------------------------------------------
Ran 29 tests in 181.286s

OK
{"decision":"verified","pending_gates":["G01","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G02"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-E"}
{"decision":"verified","pending_gates":["G01","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G02"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-E"}
{"decision":"verified","pending_gates":["G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G01"}
{"decision":"verified","pending_gates":["G01","G02","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G03"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-L"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G14"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G14"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G10"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P1"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P2"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P3"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G05"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G06"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-D"}
Status-and-outcome release descriptor verified: /private/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp8rwn1pe0/descriptor/release-bundle-status-outcome.json
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G05"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G06"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-D"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G07"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G08"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P4"}
exact 1MiB handoff ABBA tests passed
pinned paired ABBA bench tests passed
==> /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/bin/path_containment (-max_total_time=7, -seed=123456789, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/explicit-artifacts/path_containment/, corpus=/private/tmp/gw-orchestrator-2734/ci-drive/fuzz/corpus/path_containment)
-seed=123456789
target=path_containment
seed=123456789
max_total_time=7
git_revision=4d15b816236630b2c42f321ffc9ed3654edc94e3
git_dirty=true
runner_mode=prebuilt
==> /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/bin/path_containment (-max_total_time=1, -seed=2322617706, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/auto-artifacts/path_containment/, corpus=/private/tmp/gw-orchestrator-2734/ci-drive/fuzz/corpus/path_containment)
-seed=2322617706
==> cargo +nightly fuzz run path_containment (-max_total_time=9, -seed=987654321, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/cargo-artifacts/path_containment/)
+nightly
fuzz
run
path_containment
-seed=987654321
seed=987654321
runner_mode=cargo-fuzz
fuzz runner seed manifest tests passed
✅ scripts/python_bin.sh resolver checks passed
self-test ok: clean fixture with every exemption present passes
self-test ok: all 11 pattern families fire
self-test ok: moving a pinned source is reported
Embedded secret guard self-test passed.
self-test ok: p1-s3s-import: 'use s3s::s3_error;' counts every line, the rustfs_s3_types module's too (s3_error_lines=3)
self-test ok: p2-s3s-mixed-tree: a multi-line s3s use tree naming s3_error counts (s3_error_lines=3)
self-test ok: p3-s3s-glob: a crate-root glob of s3s, bare or inside a tree, counts (s3_error_lines=5)
self-test ok: p4-macro-use: '#[macro_use] extern crate s3s' counts (s3_error_lines=3)
self-test ok: p5-qualified-s3s: a qualified s3s::s3_error! line counts in a cleared file (s3_error_lines=2)
self-test ok: p6-exact-pass: a baseline equal to the tree passes (exit 0)
self-test ok: n1-rustfs-import: 'use rustfs_s3_types::s3_error;' alone counts nothing (s3_error_lines=1)
self-test ok: n2-rustfs-mixed-tree: a rustfs_s3_types tree plus an s3s tree without the macro counts nothing (s3_error_lines=1)
self-test ok: n3-defines: the file defining the macro counts nothing (s3_error_lines=1)
self-test ok: n4-qualified-rustfs: a qualified rustfs_s3_types::s3_error! line counts nothing (s3_error_lines=1)
self-test ok: n5-renamed-call: a call through a renamed macro is not text-matched (out of scope) (s3_error_lines=1)
self-test ok: n6-comment-string: a comment or string mentioning s3_error! in a cleared file counts nothing (s3_error_lines=1)
self-test ok: n7-e2e-excluded: crates/e2e_test is excluded (s3_error_lines=1)
self-test ok: n8-super-glob: an unrecognised import form ('use super::*;') counts (s3_error_lines=3)
self-test ok: n9-facade: a crate-local re-export counts (s3_error_lines=2)
self-test ok: n10-renamed-import: a renamed rustfs_s3_types import does not clear s3_error! lines (s3_error_lines=2)
self-test ok: n11-commented-import: a commented-out rustfs_s3_types import does not clear (s3_error_lines=2)
self-test ok: n12-rustfs-glob: a glob import of rustfs_s3_types does not clear (s3_error_lines=2)
self-test ok: n13-comment-only: a comment mentioning s3_error! in a file with no import counts (s3_error_lines=2)
self-test ok: f1-baseline-missing: a missing baseline file fails (exit 1)
self-test ok: f2-key-missing: a baseline without s3_error_lines fails (exit 1)
self-test ok: f3-rg-fails: a failing classification search fails the guard (exit 1)
✅ s3s footprint self-test passed
test_core_gate_rejects_missing_ignored_filtered_and_corrupt_inputs (__main__.SelfTests.test_core_gate_rejects_missing_ignored_filtered_and_corrupt_inputs) ... ok
test_e2e_requires_registration (__main__.SelfTests.test_e2e_requires_registration) ... ok
test_fuzz_runtime_uses_matrix_target (__main__.SelfTests.test_fuzz_runtime_uses_matrix_target) ... ok
test_ilm_lane_keeps_the_measured_cargo_build_budget (__main__.SelfTests.test_ilm_lane_keeps_the_measured_cargo_build_budget) ... ok
test_profile_listing_binds_platform_digest (__main__.SelfTests.test_profile_listing_binds_platform_digest) ... ok
test_profile_listing_enforces_selection (__main__.SelfTests.test_profile_listing_enforces_selection) ... ok
test_quick_checks_commands_propagate_failure (__main__.SelfTests.test_quick_checks_commands_propagate_failure) ... ok
test_quick_checks_rejects_caller_and_execution_bypasses (__main__.SelfTests.test_quick_checks_rejects_caller_and_execution_bypasses) ... ok
test_runner_readiness_fails_closed (__main__.SelfTests.test_runner_readiness_fails_closed) ... ok
test_s3_runner_rejects_unbounded_failure_locals (__main__.SelfTests.test_s3_runner_rejects_unbounded_failure_locals) ... ok
test_scanner_heal_begin_requires_embedded_source_provenance (__main__.SelfTests.test_scanner_heal_begin_requires_embedded_source_provenance) ... ok
test_scanner_heal_case_does_not_approve_pending_release (__main__.SelfTests.test_scanner_heal_case_does_not_approve_pending_release) ... ok
test_scanner_heal_case_lane_is_required_for_runner_profile (__main__.SelfTests.test_scanner_heal_case_lane_is_required_for_runner_profile) ... ok
test_scanner_heal_case_only_schema_cannot_approve_release (__main__.SelfTests.test_scanner_heal_case_only_schema_cannot_approve_release) ... ok
test_scanner_heal_crash_case_rejects_restart_oracle (__main__.SelfTests.test_scanner_heal_crash_case_rejects_restart_oracle) ... ok
test_scanner_heal_ec84_case_rejects_wrong_erasure_geometry (__main__.SelfTests.test_scanner_heal_ec84_case_rejects_wrong_erasure_geometry) ... ok
test_scanner_heal_finish_collects_oracles_from_registry (__main__.SelfTests.test_scanner_heal_finish_collects_oracles_from_registry) ... ok
test_scanner_heal_multipool_case_accepts_deferred_optional_outage_write (__main__.SelfTests.test_scanner_heal_multipool_case_accepts_deferred_optional_outage_write) ... ok
test_scanner_heal_pending_gate_cannot_map_to_implemented_lane (__main__.SelfTests.test_scanner_heal_pending_gate_cannot_map_to_implemented_lane) ... ok
test_scanner_heal_receipts_reject_reuse_and_missing_builds (__main__.SelfTests.test_scanner_heal_receipts_reject_reuse_and_missing_builds) ... ok
test_scanner_heal_rejects_boolean_fractional_and_out_of_geometry_integers (__main__.SelfTests.test_scanner_heal_rejects_boolean_fractional_and_out_of_geometry_integers) ... ok
test_scanner_heal_rejects_broken_execution_and_artifacts (__main__.SelfTests.test_scanner_heal_rejects_broken_execution_and_artifacts) ... ok
test_scanner_heal_rejects_copied_junit_and_wrong_suite_build (__main__.SelfTests.test_scanner_heal_rejects_copied_junit_and_wrong_suite_build) ... ok
test_scanner_heal_release_bundle_accepts_complete_measured_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_accepts_complete_measured_evidence) ... ok
test_scanner_heal_release_bundle_assembler_copies_measured_artifacts (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_copies_measured_artifacts) ... ok
test_scanner_heal_release_bundle_assembler_merges_measured_descriptors (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_merges_measured_descriptors) ... ok
test_scanner_heal_release_bundle_assembler_rejects_duplicate_descriptor_gate (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_duplicate_descriptor_gate) ... ok
test_scanner_heal_release_bundle_assembler_rejects_fixture_descriptor (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_fixture_descriptor) ... ok
test_scanner_heal_release_bundle_assembler_rejects_hash_mismatch (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_hash_mismatch) ... ok
test_scanner_heal_release_bundle_assembler_rejects_mixed_descriptor_revisions (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_mixed_descriptor_revisions) ... ok
test_scanner_heal_release_bundle_enforces_topology_duration_profile_and_versions (__main__.SelfTests.test_scanner_heal_release_bundle_enforces_topology_duration_profile_and_versions) ... ok
test_scanner_heal_release_bundle_fixture_writer_is_not_release_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_fixture_writer_is_not_release_evidence) ... ok
test_scanner_heal_release_bundle_gate_accepts_single_measured_gate (__main__.SelfTests.test_scanner_heal_release_bundle_gate_accepts_single_measured_gate) ... ok
test_scanner_heal_release_bundle_gate_rejects_missing_field (__main__.SelfTests.test_scanner_heal_release_bundle_gate_rejects_missing_field) ... ok
test_scanner_heal_release_bundle_gate_rejects_unknown_gate (__main__.SelfTests.test_scanner_heal_release_bundle_gate_rejects_unknown_gate) ... ok
test_scanner_heal_release_bundle_rejects_synthetic_or_missing_fields (__main__.SelfTests.test_scanner_heal_release_bundle_rejects_synthetic_or_missing_fields) ... ok
test_scanner_heal_release_bundle_requires_domain_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_requires_domain_evidence) ... ok
test_scanner_heal_release_bundle_requires_field_provenance (__main__.SelfTests.test_scanner_heal_release_bundle_requires_field_provenance) ... ok
test_scanner_heal_release_bundle_requires_json_artifact_provenance (__main__.SelfTests.test_scanner_heal_release_bundle_requires_json_artifact_provenance) ... ok
test_scanner_heal_release_bundle_requires_mixed_version_field_roles (__main__.SelfTests.test_scanner_heal_release_bundle_requires_mixed_version_field_roles) ... ok
test_scanner_heal_release_bundle_requires_same_gate_measurement_window (__main__.SelfTests.test_scanner_heal_release_bundle_requires_same_gate_measurement_window) ... ok
test_scanner_heal_release_consumes_multiple_registry_oracles (__main__.SelfTests.test_scanner_heal_release_consumes_multiple_registry_oracles) ... ok
test_scanner_heal_release_matrix_lanes_remain_pending_without_real_evidence (__main__.SelfTests.test_scanner_heal_release_matrix_lanes_remain_pending_without_real_evidence) ... ok
test_scanner_heal_release_requirements_cannot_be_cleared (__main__.SelfTests.test_scanner_heal_release_requirements_cannot_be_cleared) ... ok
test_scanner_heal_release_status_rejects_focused_case_run (__main__.SelfTests.test_scanner_heal_release_status_rejects_focused_case_run) ... ok
test_scanner_heal_release_status_rejects_synthetic_case (__main__.SelfTests.test_scanner_heal_release_status_rejects_synthetic_case) ... ok
test_scanner_heal_required_evidence_fields_cannot_be_removed (__main__.SelfTests.test_scanner_heal_required_evidence_fields_cannot_be_removed) ... ok
test_scheduled_alerts_require_completion_watchdog (__main__.SelfTests.test_scheduled_alerts_require_completion_watchdog) ... ok
test_update_profile_selection_changes_only_requested_platform (__main__.SelfTests.test_update_profile_selection_changes_only_requested_platform) ... ok
test_validate_includes_quick_checks (__main__.SelfTests.test_validate_includes_quick_checks) ... ok
test_vault_tests_require_cross_process_serialization (__main__.SelfTests.test_vault_tests_require_cross_process_serialization) ... ok
test_workflow_readiness_requires_dependency_probe (__main__.SelfTests.test_workflow_readiness_requires_dependency_probe) ... ok

----------------------------------------------------------------------
Ran 52 tests in 14.301s

OK
e2e-full selection OK: 1 tests, sha256=29e30e6ed2ed38d9e5b0a7cdd9536abf5c719f7021a02b578ae1381563a3c273
...................................
----------------------------------------------------------------------
Ran 35 tests in 1.161s

OK
...................
----------------------------------------------------------------------
Ran 19 tests in 17.397s

OK
..........
----------------------------------------------------------------------
Ran 10 tests in 0.183s

OK
test_actual_selector_bootstrap_uses_base_policy_and_fails_closed (__main__.SelfTests.test_actual_selector_bootstrap_uses_base_policy_and_fails_closed) ... ok
test_documentation_paths_do_not_hide_build_or_fixture_changes (__main__.SelfTests.test_documentation_paths_do_not_hide_build_or_fixture_changes) ... ok
test_e2e_scope_keeps_production_and_shared_configuration_full (__main__.SelfTests.test_e2e_scope_keeps_production_and_shared_configuration_full) ... ok
test_e2e_shortcut_rejects_new_direct_renamed_and_target_dependencies (__main__.SelfTests.test_e2e_shortcut_rejects_new_direct_renamed_and_target_dependencies) ... ok
test_e2e_shortcut_traverses_implicit_path_dependencies (__main__.SelfTests.test_e2e_shortcut_traverses_implicit_path_dependencies) ... ok
test_event_contract_requires_complete_candidate_and_optional_lanes (__main__.SelfTests.test_event_contract_requires_complete_candidate_and_optional_lanes) ... ok
test_every_wrong_result_missing_job_or_selection_fails_closed (__main__.SelfTests.test_every_wrong_result_missing_job_or_selection_fails_closed) ... ok
test_full_e2e_gate_preserves_workflow_branch_and_event_scope (__main__.SelfTests.test_full_e2e_gate_preserves_workflow_branch_and_event_scope) ... ok
test_git_range_includes_deleted_source_and_rename_origins (__main__.SelfTests.test_git_range_includes_deleted_source_and_rename_origins) ... ok
test_job_ids_and_display_names_cannot_hide_validation (__main__.SelfTests.test_job_ids_and_display_names_cannot_hide_validation) ... ok
test_report_only_stack_lane_mirrors_e2e_smoke_outside_the_aggregate (__main__.SelfTests.test_report_only_stack_lane_mirrors_e2e_smoke_outside_the_aggregate) ... ok
test_repository_wiring_and_missing_dependency_regression (__main__.SelfTests.test_repository_wiring_and_missing_dependency_regression) ... ok
test_verify_command_preserves_failures (__main__.SelfTests.test_verify_command_preserves_failures) ... ok

----------------------------------------------------------------------
Ran 13 tests in 3.280s

OK
security coverage self-test passed
test_api_failures_preserve_other_evidence_and_never_enter_grace (__main__.SelfTests.test_api_failures_preserve_other_evidence_and_never_enter_grace) ... ok
test_attempt_outcome_does_not_replace_recent_success (__main__.SelfTests.test_attempt_outcome_does_not_replace_recent_success) ... ok
test_cancelled_attempt_cannot_refresh_expired_success (__main__.SelfTests.test_cancelled_attempt_cannot_refresh_expired_success) ... ok
test_cli_requires_the_repository_default_branch (__main__.SelfTests.test_cli_requires_the_repository_default_branch) ... ok
test_config_rejects_duplicate_and_invalid_entries (__main__.SelfTests.test_config_rejects_duplicate_and_invalid_entries) ... ok
test_freshness_boundaries (__main__.SelfTests.test_freshness_boundaries) ... ok
test_grace_requires_two_successful_queries_with_no_history (__main__.SelfTests.test_grace_requires_two_successful_queries_with_no_history) ... ok
test_invalid_api_evidence_fails_closed (__main__.SelfTests.test_invalid_api_evidence_fails_closed) ... ok
test_report_retains_every_workflow (__main__.SelfTests.test_report_retains_every_workflow) ... ok
test_requests_filter_schedule_default_branch_and_success_on_server (__main__.SelfTests.test_requests_filter_schedule_default_branch_and_success_on_server) ... ok

----------------------------------------------------------------------
Ran 10 tests in 0.051s

OK
..............................
----------------------------------------------------------------------
Ran 30 tests in 22.354s

OK
..........
----------------------------------------------------------------------
Ran 10 tests in 2.969s

OK
.................................
----------------------------------------------------------------------
Ran 33 tests in 0.160s

OK
Online functional runner labels: pf-testing
.......
----------------------------------------------------------------------
Ran 7 tests in 0.002s

OK
.......
----------------------------------------------------------------------
Ran 7 tests in 0.001s

OK
....
----------------------------------------------------------------------
Ran 4 tests in 0.128s

OK
...
----------------------------------------------------------------------
Ran 3 tests in 0.547s

OK
PASS: cold stampede script self-test
exit_code=0

$ python3 ./scripts/test_object_generation_protocol_model.py
test_a_recovery_barrier_cannot_choose_an_empty_value (__main__.ObjectGenerationProtocolModel.test_a_recovery_barrier_cannot_choose_an_empty_value) ... ok
test_all_bounded_two_proposer_interleavings_choose_at_most_one_value (__main__.ObjectGenerationProtocolModel.test_all_bounded_two_proposer_interleavings_choose_at_most_one_value) ... ok
test_an_unavailable_decision_quorum_cannot_choose_a_candidate (__main__.ObjectGenerationProtocolModel.test_an_unavailable_decision_quorum_cannot_choose_a_candidate) ... ok
test_counter_allocation_requires_persistence_and_is_consumed_once (__main__.ObjectGenerationProtocolModel.test_counter_allocation_requires_persistence_and_is_consumed_once) ... ok
test_counter_restart_burns_persisted_allocations_before_or_after_issue (__main__.ObjectGenerationProtocolModel.test_counter_restart_burns_persisted_allocations_before_or_after_issue) ... ok
test_crash_after_promise_before_reply_does_not_restore_an_older_ballot (__main__.ObjectGenerationProtocolModel.test_crash_after_promise_before_reply_does_not_restore_an_older_ballot) ... ok
test_duplicate_or_unconfigured_voters_cannot_form_a_quorum (__main__.ObjectGenerationProtocolModel.test_duplicate_or_unconfigured_voters_cannot_form_a_quorum) ... ok
test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision (__main__.ObjectGenerationProtocolModel.test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision) ... ok
test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected (__main__.ObjectGenerationProtocolModel.test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected) ... ok
test_forgetting_tombstone_authority_allows_an_aba_create (__main__.ObjectGenerationProtocolModel.test_forgetting_tombstone_authority_allows_an_aba_create) ... ok
test_issuing_before_persistence_reuses_a_ballot_after_restart (__main__.ObjectGenerationProtocolModel.test_issuing_before_persistence_reuses_a_ballot_after_restart) ... ok
test_lost_ack_and_restart_preserve_the_chosen_operation (__main__.ObjectGenerationProtocolModel.test_lost_ack_and_restart_preserve_the_chosen_operation) ... ok
test_minority_accepted_work_is_adopted_and_finished (__main__.ObjectGenerationProtocolModel.test_minority_accepted_work_is_adopted_and_finished) ... ok
test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample (__main__.ObjectGenerationProtocolModel.test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample) ... ok
test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it (__main__.ObjectGenerationProtocolModel.test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it) ... ok
test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor (__main__.ObjectGenerationProtocolModel.test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor) ... ok
test_replacing_voters_with_empty_state_can_choose_a_conflicting_value (__main__.ObjectGenerationProtocolModel.test_replacing_voters_with_empty_state_can_choose_a_conflicting_value) ... ok
test_skipping_predecessor_recovery_chooses_a_broken_revision_chain (__main__.ObjectGenerationProtocolModel.test_skipping_predecessor_recovery_chooses_a_broken_revision_chain) ... ok
test_stale_publication_rollback_and_gc_cannot_change_the_winner (__main__.ObjectGenerationProtocolModel.test_stale_publication_rollback_and_gc_cannot_change_the_winner) ... ok
test_superseded_operation_identity_cannot_be_reused_for_different_input (__main__.ObjectGenerationProtocolModel.test_superseded_operation_identity_cannot_be_reused_for_different_input) ... ok
test_superseded_operation_retry_returns_its_original_successor (__main__.ObjectGenerationProtocolModel.test_superseded_operation_retry_returns_its_original_successor) ... ok
test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision (__main__.ObjectGenerationProtocolModel.test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision) ... ok
test_tombstone_rejects_a_delayed_never_present_create (__main__.ObjectGenerationProtocolModel.test_tombstone_rejects_a_delayed_never_present_create) ... ok
test_unlearned_predecessor_is_recovered_before_the_second_slot (__main__.ObjectGenerationProtocolModel.test_unlearned_predecessor_is_recovered_before_the_second_slot) ... ok

----------------------------------------------------------------------
Ran 24 tests in 0.958s

OK
model: states=51023, transitions=167520, terminal=732, chosen_candidates=2
negative control: double-choice trace=((0, 'prepare', 0), (0, 'prepare', 1), (0, 'prepare', 2), (0, 'accept', 0), (0, 'accept', 1), (0, 'accept', 2), (1, 'prepare', 0), (1, 'prepare', 1), (1, 'prepare', 2), (1, 'accept', 0), (1, 'accept', 1), (1, 'accept', 2))
multi-slot round schedules: 640
exit_code=0

$ python3 ./scripts/check_test_wiring.py
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired
exit_code=0

$ ./scripts/check_no_planning_docs.sh
exit_code=0

$ ./scripts/check_uring_lane_lib_only.sh
OK: no 'uring_'-matching test functions under crates/ecstore/tests; --lib is safe
exit_code=0

Initial complete gate
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
 Nextest run ID e07fc5b9-c6c8-4cdd-ae61-54839bf24437 with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.016s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.011s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.009s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.009s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  16.950s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   4.025s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.008s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.007s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.035s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  21.087s] 10 tests run: 10 passed, 0 skipped
exit_code=0

Restored complete gate
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
 Nextest run ID 233b7307-5361-4eb7-94d5-b384f17d2d6a with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.009s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.008s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.008s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.008s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  19.518s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   4.146s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.009s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.018s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.009s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.054s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  23.787s] 10 tests run: 10 passed, 0 skipped
exit_code=0

Mutation M1: success_measurement reports max(actual elapsed, requested budget + 1), forcing 1001 ms for the one-second request. The retained duration assertion at rustfs/tests/connect_perf_drive.rs:105 fails.
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast -E 'test(real_local_write_read_is_measured_and_scratch_is_removed)'
info: experimental features enabled: setup-scripts
   Compiling rustfs v1.0.1 (/private/tmp/gw-orchestrator-2734/ci-drive/rustfs)
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs") generated 1 warning
warning: `rustfs` (bin "rustfs-cli") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 7m 06s
────────────
 Nextest run ID 9f618777-7bf6-4cbc-8803-dd7f50f449ed with nextest profile: ci
    Starting 1 test across 1 binary (9 tests skipped)
        FAIL [   0.016s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
  stdout ───

    running 1 test
    test real_local_write_read_is_measured_and_scratch_is_removed ... FAILED

    failures:

    failures:
        real_local_write_read_is_measured_and_scratch_is_removed

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 0.01s

  stderr ───

    thread 'real_local_write_read_is_measured_and_scratch_is_removed' (106986846) panicked at rustfs/tests/connect_perf_drive.rs:105:5:
    assertion failed: (1..=1_000).contains(&data.duration_millis)
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

────────────
     Summary [   0.017s] 1 test run: 0 passed, 1 failed, 9 skipped
        FAIL [   0.016s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
error: test run failed
exit_code=100

Mutation M2: target_parameters reports requested duration + 1. The in-process assertion at rustfs/tests/connect_perf_drive.rs:111 and CLI JSON assertion at rustfs/tests/connect_perf_drive.rs:527 both reject 1001 against 1000.
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast -E 'test(real_local_write_read_is_measured_and_scratch_is_removed) | test(production_cli_measures_and_signs_exact_binary_provenance)'
info: experimental features enabled: setup-scripts
   Compiling rustfs v1.0.1 (/private/tmp/gw-orchestrator-2734/ci-drive/rustfs)
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs") generated 1 warning
warning: `rustfs` (bin "rustfs-cli") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 7m 20s
────────────
 Nextest run ID b73d46ab-c910-4e1e-8063-7fc7e332ec15 with nextest profile: ci
    Starting 2 tests across 1 binary (8 tests skipped)
        FAIL [  13.968s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
  stdout ───

    running 1 test
    test production_cli_measures_and_signs_exact_binary_provenance ... FAILED

    failures:

    failures:
        production_cli_measures_and_signs_exact_binary_provenance

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 13.96s

  stderr ───

    thread 'production_cli_measures_and_signs_exact_binary_provenance' (107015462) panicked at rustfs/tests/connect_perf_drive.rs:527:5:
    assertion `left == right` failed
      left: Number(1001)
     right: 1000
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

        FAIL [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
  stdout ───

    running 1 test
    test real_local_write_read_is_measured_and_scratch_is_removed ... FAILED

    failures:

    failures:
        real_local_write_read_is_measured_and_scratch_is_removed

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 0.01s

  stderr ───

    thread 'real_local_write_read_is_measured_and_scratch_is_removed' (107016667) panicked at rustfs/tests/connect_perf_drive.rs:111:5:
    assertion `left == right` failed
      left: 1001
     right: 1000
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

────────────
     Summary [  13.985s] 2 tests run: 0 passed, 2 failed, 8 skipped
        FAIL [  13.968s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        FAIL [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
error: test run failed
exit_code=100

Transient production mutations were restored byte-for-byte before the complete green rerun. Original/restored SHA256: 8d389e6d8836c041aab8c6a93e56addd65ba3f70b33264e1c74f639d0572a2da.

$ git diff --check

exit_code=0

$ git diff --cached --check

exit_code=0

Independent orchestrator acceptance
Tested code tree: e67c72fb01

Tested tree: 16fbbd2ea4
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

$ LANG=en_US.UTF-8 CARGO_BUILD_JOBS=1 CARGO_INCREMENTAL=0 cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
info: experimental features enabled: setup-scripts
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs-cli") generated 1 warning
warning: `rustfs` (bin "rustfs") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.69s
────────────
 Nextest run ID cd1edf4c-f1c0-456d-ad1a-76cc00ce05c9 with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.012s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.010s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.010s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.009s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  20.328s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   5.527s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.013s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.019s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.011s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.049s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  25.991s] 10 tests run: 10 passed, 0 skipped

exit_code=0

All 54 tracked Cargo.toml files parse without duplicate keys. git diff --check passed. The production source remains unchanged; the two deliberate implementation mutations above were restored before this complete green run.

* fix(test): compare legacy body wire parity as raw bytes

Lossy UTF-8 comparison concealed a changed binary response byte.
Compare the already captured Vec<u8> values directly.

Validation:
- Binary mutant: migrated payload 0x84 vs legacy 0x83; old comparator
  passed 1/1, raw comparator failed 0/1 (exit 100); restored green.
- Non-Swift body-drain mutant: untouched-body delegation test failed
  with empty vs payload (exit 100); restored handler without a diff.
- cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols
  --features rustfs-protocols/swift --no-fail-fast:
  543 tests run, 543 passed, 0 skipped after both mutants restored.
- cargo fmt --all --check and git diff --check passed.

Orchestrator acceptance (LANG=en_US.UTF-8):
cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
Summary [2.445s] 543 tests run: 543 passed, 0 skipped.

All shared quick checks on the final code tree:
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

Final independent acceptance after the drive-test CI repair
Parent: a99d8f4f89
Tested code tree: 93cbd96bf0

Tested tree: 075ae5161c2b62a96812bc1e3c904a7663b4f2f8
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

$ LANG=en_US.UTF-8 CARGO_BUILD_JOBS=1 CARGO_INCREMENTAL=0 cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
    Finished `test` profile [unoptimized + debuginfo] target(s) in 10m 08s
 Nextest run ID 5c7ff102-ec6d-479a-86e0-810859b28bbf with nextest profile: ci
    Starting 543 tests across 8 binaries
     Summary [   6.729s] 543 tests run: 543 passed, 0 skipped
exit_code=0

The only source change is the raw-byte wire comparator. Cargo manifests and the ratchet baselines remain unchanged from the independently validated parent. The invalid-byte and consumed-request mutations remain recorded above; their implementation changes were restored.

* fix: repair integration checks and IPC test readiness

Wait for the RPC/locks test sockets to pass the existing owner and mode
checks inside the original 5s readiness window and 10ms poll interval.

Use the existing admin S3 carrier for the new recovery and quota error
calls. Regenerate the test-count baseline for main's 118 existing tests
across seven crates. Pair the private pagination helper's raw-scan inputs
and remove the redundant Copy clone from the retirement event.

Keep the original assertions, responses, authorization, locks and storage
operations. The pagination body and raw-scan expressions at all five calls
are byte-identical; reinserting only the Copy clone restores the entire
base heal file. All ignored baseline values and s3s counts stay unchanged.

Validation base: c47432e353
Validation tree: f081084bc5
Patch SHA-256: 2dcf0028691bfb2772edfb38e262829b04f4151832dcce823c9dd9f7434b52de
Trace production-prefix SHA-256: fd994f218a21429d62866f4efb8eee1ee57b58f4a51865aa35d05a54ef1ccb45

All commands used LANG=en_US.UTF-8, CARGO_BUILD_JOBS=1,
CARGO_INCREMENTAL=0 and CARGO_NET_OFFLINE=true in one exclusive target.
Every gate retained the same tree and six source hashes before and after.
54 Cargo.toml files parse without duplicate keys.

cargo clippy -p rustfs --all-targets -- -D warnings
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 12m 50s
Exit: 0. Wall time: 770.855s.

The original 19 quick-check steps:
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
Wall time: 286.337s.

cargo nextest run --profile ci -p rustfs --no-fail-fast
     Summary [ 217.017s] 5382 tests run: 5382 passed (4 leaky), 40 skipped
Exit: 0. Wall time: 1324.414s.

cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
     Summary [   2.189s] 543 tests run: 543 passed, 0 skipped
Exit: 0. Wall time: 123.192s.

cargo nextest run --profile ci -p rustfs-ecstore list_objects_paginate --no-fail-fast
     Summary [   0.170s] 8 tests run: 8 passed, 5921 skipped
Exit: 0. Wall time: 261.324s.

cargo nextest run --profile ci -p rustfs-heal durable_mrf_anchor_retires_only_after_exact_bucket_generation_is_deleted --no-fail-fast
     Summary [   0.805s] 1 test run: 1 passed, 835 skipped
Exit: 0. Wall time: 199.801s.

Baseline regeneration:
./scripts/check_test_count_ratchet.sh --update
wrote .config/test-count-baseline.txt: 52 crates, 21340 tests, 164 ignored
Test-only increases: ecstore48, heal3, iam14, io-metrics2, obs3, policy14,
rustfs34. Existing ignored values and all other rows are byte-identical.
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)

Historical runtime red controls on f622 (same retained two assertions):
cargo nextest run --profile ci -p rustfs --lib -E 'test(=connect::diagnostics::trace_runtime::tests::local_top_rpc_reads_service_http_completions_and_signs_offline) | test(=connect::diagnostics::trace_runtime::tests::local_top_locks_reads_service_manager_and_signs_offline)' --no-fail-fast
Original path readiness with a temporary 500ms bind/seal gap:
     Summary [   0.863s] 2 tests run: 0 passed, 2 failed, 5113 skipped
Exit: 100.
Sealed readiness in that same gap:
     Summary [   5.621s] 2 tests run: 2 passed, 5113 skipped
Exit: 0.
Compiled M1: replace the helper predicate with path.exists():
     Summary [   0.870s] 2 tests run: 0 passed, 2 failed, 5113 skipped
Exit: 100.
Restore the sealed predicate byte-for-byte:
     Summary [   5.640s] 2 tests run: 2 passed, 5113 skipped
Exit: 0.
Both reds observe mode755 with owner_matches=true and fail at the original
RPC/locks unwraps with StateSecurity. M1 failure locations: trace_runtime.rs
2547:29 and 2713:29. No new assertions were introduced.

cargo nextest run --profile ci -p rustfs --lib -E 'test(connect::diagnostics::trace_runtime::tests::)' --no-fail-fast
     Summary [  20.748s] 28 tests run: 28 passed (1 leaky), 5087 skipped
Exit: 0. This security sweep is historical; the full final unit gate above
covers the actual c474 candidate.
The first disk-stopped compile ran no tests and is excluded from red credit.
All temporary seal instrumentation is removed from the final source.

Final unit cleanup markers: 4 leaky, 40 original skips (exit0). The ld
__eh_frame linker warning also appears verbatim in the original f622 run
and the earlier c474 run; there are no new source warnings.

* fix: remove redundant pending closure in disk test

Pass the capture-free pending future factory as a function item in the existing slow-operation test. This fixes the redundant_closure error at disk_store.rs:2914 that stopped strict Clippy before the feature tests could run.

The single-line test edit preserves the factory's result, warning and timeout behavior. Reversing the replacement reproduces the parent file byte for byte. All 220 assertion lines and the production prefix are unchanged; no tests, assertions, guards, profiles or baselines were added or relaxed.

Original Linux CI evidence (run 37888552268 at 53245be9f):
- RIO job113685483992: strict Clippy exit101, redundant closure at disk_store.rs:2914; feature tests skipped.
- Workspace Test and Lint job113685484058: same strict Clippy exit101; nextest and doc tests skipped.
These are lint failures, not runtime mutation kills or known flakes.

Native macOS arm64 validation on tree7a5bf47e7533b29479bc85207416102f1337c566, LANG=en_US.UTF-8, one build job, CARGO_INCREMENTAL=0 and offline dependencies:

shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
All 19 checks: exit0, 312.161s.

$ cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 20m 11s
exit0, 1213.210s total.

$ cargo nextest run --profile ci -p rustfs-ecstore --features rio-v2 slow_disk_operation_warns_once_while_pending_and_preserves_result --no-fail-fast
Finished `test` profile [unoptimized + debuginfo] target(s) in 4m 34s
PASS [0.026s] rustfs-ecstore disk::disk_store::tests::slow_disk_operation_warns_once_while_pending_and_preserves_result
Summary [   0.170s] 1 test run: 1 passed, 5943 skipped
exit0, 281.920s total; 5943 existing filter skips.

warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
This linker warning is retained; flags, profiles and allowances are unchanged.

54 tracked Cargo.toml files parsed without duplicate keys. git diff --check passed. No new assertions were introduced, so no new runtime mutation is claimed. Native compilation and the existing helper test do not substitute for actual Linux RIO execution; the new integration CI run still needs to recover before the RPC merge.

Preservation and output hashes:
parent file SHA256: 8e59ee745e3aaaa22848b943d3b10f6353b515956282cde1f3af2c394200c9ac
final file SHA256: b69ba3237061b161448ae8b62b615885d5770b98a5b4a2479bcfbd70f1bd1f17
unchanged production prefix SHA256: 7903a6e288252c420231242d7b58a25fa2a7140ea3dd430cbe7f0dde11369dcb
RIO Linux failure log SHA256:421addb9868fd88e4109a922311e54f91f0dfd70862f8ba7fcaba193482ed90b
Workspace Linux failure log SHA256:e0aea8d0f014c022252a9534968e083c98cfc1fd760da445ecd04ec3f522e26a
19 output SHA256:35a54f28d707956ed1edb29d492c4dfe8a9fe753d103b111b5a2f1b5c41c2b98
RIO Clippy output SHA256:d6046e3868f2b3992fe11cdd04e9e98eb50a87d1234d9a70dbfeb154ff61638d
slow-operation output SHA256:605b7b258d2fd09c6ef2ab2ffc90d433d0509480f1e45125d20c823759e83608

Refs rustfs/backlog#2734.

* test(ecstore): isolate tier-free recovery fixture startup

The fixture constructor can take the production recovery lock and wait for 60 seconds before this test installs its queue and seeds metadata. Disable its background recovery only while creating this fixture, then restore the environment before calling the existing production recovery entrypoint.

Only the existing test setup changes. Its production entrypoint, default lock parameters, original assertions, 30-second timeout, cancellation and cleanup remain intact. No tests or assertions were added. Production and store/init.rs are byte-identical to the parent.

Independent acceptance ran on merge tree 8ec358cb42ba9ebb3bcfeafcef048a3f0366355b; the runner header reports parent HEAD, while before/after index and source hashes confirm this candidate. Worker task branch d8055766b8 has the same tree.

Every command used LANG=en_US.UTF-8. Task compilation also used CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0, CARGO_NET_OFFLINE=true, an independent target, and empty RUSTC_WRAPPER/RUSTC_WORKSPACE_WRAPPER to keep compilers in its owned process group without changing repository configuration.

Command:
shellcheck --version && actionlint
Exit 0; observed output excerpt:
version: 0.11.0
Log SHA-256: fc9eec16b4191d058a295813ca9afbe9d2b37b59975da860170726be162a3924

Command:
cargo fmt --all --check
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_unsafe_code_allowances.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_layer_dependencies.sh
Exit 0; observed output excerpt:
Layer dependency guard passed (no new reverse dependencies/cycles).
Log SHA-256: 69c6cb1896c0586b28d1167380d3260d99d8017a4265c8df5a2348f289e93a12

Command:
./scripts/check_architecture_migration_rules.sh
Exit 0; observed output excerpt:
Architecture migration rules passed.
Log SHA-256: 28790e70d070607ed4de0704f292a65030be92ec224ef106d4b72fd8296f896e

Command:
./scripts/check_logging_guardrails.sh
Exit 0; observed output excerpt:
✅ Logging guardrails check passed
Log SHA-256: 7adf41abff8ed8f7617decf86498888ddde78937099dd9ce47f6d359dfc2d317

Command:
./scripts/check_error_other_format_ratchet.sh
Exit 0; observed output excerpt:
error(format!) ratchet passed (total call sites: 579).
Log SHA-256: 66172aef18f150ba0605af122e26ffcf91f6dd86f43e38b05e2e982ce8d783ee

Command:
./scripts/check_no_tokio_io_uring.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_extension_schema_boundaries.sh
Exit 0; observed output excerpt:
Extension schema boundaries passed.
Log SHA-256: f13d0acd0e2ce5d9815241b12b4dab7d01cc11b9b65bcf6dfb7812c878532a51

Command:
./scripts/check_body_cache_whitelist.sh
Exit 0; observed output excerpt:
Body-cache whitelist guard passed.
Log SHA-256: 6b828847b0383950ee6eef9cd846d0c322c945e47efd4c7d1ad881178810d6bb

Command:
./scripts/check_s3s_footprint.sh
Exit 0; observed output excerpt:
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)
✅ s3s footprint baseline check passed
Log SHA-256: 47f53d04ef645618563fe1f22abc64667f19078f1974feedcca46e4f2c016844

Command:
./scripts/check_test_count_ratchet.sh
Exit 0; observed output excerpt:
test-count ratchet OK: 52 crates, 21340 tests, 164 ignored match .config/test-count-baseline.txt
Log SHA-256: 618a5e330e933de0c6ff484546df401457024a88bb5986d11d588794e33dab60

Command:
./scripts/check_fips_wording.sh
Exit 0; observed output excerpt:
FIPS wording guard passed (README.md and CHANGELOG.md contain no forbidden claims).
KMS wording guard passed (crates/kms claims no Vault KV2 Transit wrapping).
Log SHA-256: 99221cc9979c480ec4f46f50d76c52b18c72fe499f12e15e4f4a22a1b8bfb140

Command:
./scripts/check_embedded_secrets.sh
Exit 0; observed output excerpt:
Embedded secret guard passed (no private key material or provider credential literal in the tree).
Log SHA-256: 437a3ceba32094d2901889965ad191e411aad32cdb18bf0b8659e818011c406b

Command:
make script-tests
Exit 0; observed output excerpt:
test_cli_requires_the_repository_default_branch (__main__.SelfTests.test_cli_requires_the_repository_default_branch) ... ok
test_config_rejects_duplicate_and_invalid_entries (__main__.SelfTests.test_config_rejects_duplicate_and_invalid_entries) ... ok
test_freshness_boundaries (__main__.SelfTests.test_freshness_boundaries) ... ok
test_grace_requires_two_successful_queries_with_no_history (__main__.SelfTests.test_grace_requires_two_successful_queries_with_no_history) ... ok
test_invalid_api_evidence_fails_closed (__main__.SelfTests.test_invalid_api_evidence_fails_closed) ... ok
test_report_retains_every_workflow (__main__.SelfTests.test_report_retains_every_workflow) ... ok
test_requests_filter_schedule_default_branch_and_success_on_server (__main__.SelfTests.test_requests_filter_schedule_default_branch_and_success_on_server) ... ok
Ran 10 tests in 0.042s
OK
Ran 30 tests in 20.627s
OK
Ran 10 tests in 2.268s
OK
Ran 33 tests in 0.127s
OK
Ran 7 tests in 0.002s
OK
Ran 7 tests in 0.001s
OK
Ran 4 tests in 0.108s
OK
Ran 3 tests in 0.564s
OK
PASS: cold stampede script self-test
Log SHA-256: 75fda2d069a373c63667b908baaf99e6591aab9099a739b0c4c22ceb6da8d68c

Command:
python3 ./scripts/test_object_generation_protocol_model.py
Exit 0; observed output excerpt:
test_an_unavailable_decision_quorum_cannot_choose_a_candidate (__main__.ObjectGenerationProtocolModel.test_an_unavailable_decision_quorum_cannot_choose_a_candidate) ... ok
test_counter_allocation_requires_persistence_and_is_consumed_once (__main__.ObjectGenerationProtocolModel.test_counter_allocation_requires_persistence_and_is_consumed_once) ... ok
test_counter_restart_burns_persisted_allocations_before_or_after_issue (__main__.ObjectGenerationProtocolModel.test_counter_restart_burns_persisted_allocations_before_or_after_issue) ... ok
test_crash_after_promise_before_reply_does_not_restore_an_older_ballot (__main__.ObjectGenerationProtocolModel.test_crash_after_promise_before_reply_does_not_restore_an_older_ballot) ... ok
test_duplicate_or_unconfigured_voters_cannot_form_a_quorum (__main__.ObjectGenerationProtocolModel.test_duplicate_or_unconfigured_voters_cannot_form_a_quorum) ... ok
test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision (__main__.ObjectGenerationProtocolModel.test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision) ... ok
test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected (__main__.ObjectGenerationProtocolModel.test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected) ... ok
test_forgetting_tombstone_authority_allows_an_aba_create (__main__.ObjectGenerationProtocolModel.test_forgetting_tombstone_authority_allows_an_aba_create) ... ok
test_issuing_before_persistence_reuses_a_ballot_after_restart (__main__.ObjectGenerationProtocolModel.test_issuing_before_persistence_reuses_a_ballot_after_restart) ... ok
test_lost_ack_and_restart_preserve_the_chosen_operation (__main__.ObjectGenerationProtocolModel.test_lost_ack_and_restart_preserve_the_chosen_operation) ... ok
test_minority_accepted_work_is_adopted_and_finished (__main__.ObjectGenerationProtocolModel.test_minority_accepted_work_is_adopted_and_finished) ... ok
test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample (__main__.ObjectGenerationProtocolModel.test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample) ... ok
test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it (__main__.ObjectGenerationProtocolModel.test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it) ... ok
test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor (__main__.ObjectGenerationProtocolModel.test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor) ... ok
test_replacing_voters_with_empty_state_can_choose_a_conflicting_value (__main__.ObjectGenerationProtocolModel.test_replacing_voters_with_empty_state_can_choose_a_conflicting_value) ... ok
test_skipping_predecessor_recovery_chooses_a_broken_revision_chain (__main__.ObjectGenerationProtocolModel.test_skipping_predecessor_recovery_chooses_a_broken_revision_chain) ... ok
test_stale_publication_rollback_and_gc_cannot_change_the_winner (__main__.ObjectGenerationProtocolModel.test_stale_publication_rollback_and_gc_cannot_change_the_winner) ... ok
test_superseded_operation_identity_cannot_be_reused_for_different_input (__main__.ObjectGenerationProtocolModel.test_superseded_operation_identity_cannot_be_reused_for_different_input) ... ok
test_superseded_operation_retry_returns_its_original_successor (__main__.ObjectGenerationProtocolModel.test_superseded_operation_retry_returns_its_original_successor) ... ok
test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision (__main__.ObjectGenerationProtocolModel.test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision) ... ok
test_tombstone_rejects_a_delayed_never_present_create (__main__.ObjectGenerationProtocolModel.test_tombstone_rejects_a_delayed_never_present_create) ... ok
test_unlearned_predecessor_is_recovered_before_the_second_slot (__main__.ObjectGenerationProtocolModel.test_unlearned_predecessor_is_recovered_before_the_second_slot) ... ok
Ran 24 tests in 0.929s
OK
Log SHA-256: 0b4be926efb0609d497ae58930340153f01957b606206cbbdb3e0d47eb29d3a2

Command:
python3 ./scripts/check_test_wiring.py
Exit 0; observed output excerpt:
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired
Log SHA-256: 21f676f458451525535ab0ef73d6a9c0d9774e16d76a4d19e396ad34936dd39c

Command:
./scripts/check_no_planning_docs.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_uring_lane_lib_only.sh
Exit 0; observed output excerpt:
OK: no 'uring_'-matching test functions under crates/ecstore/tests; --lib is safe
Log SHA-256: 43f3f851e27ec2ab9e34d6b05fefdede38af992699c473904bfe6f53a46d52e3

Command:
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast
Exit 0; wall 234.399s.
Observed output:
    Finished `test` profile [unoptimized + debuginfo] target(s) in 3m 49s
    Starting 1 test across 1 binary (5871 tests skipped)
  SETUP PASS [   0.006s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"'
        PASS [   0.440s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item
     Summary [   0.586s] 1 test run: 1 passed, 5871 skipped
Log SHA-256: 34e973f86e8dc2102c1a0e483e0ac802ce7df3d1b40a603932f62c0805c4de27

Command:
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over) | test(=store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence) | test(=store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable)' --no-fail-fast
Exit 0; wall 4.010s.
Observed output:
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.43s
    Starting 3 tests across 1 binary (5869 tests skipped)
  SETUP PASS [   0.005s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"'
        PASS [   0.068s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over
        PASS [   0.637s] rustfs-ecstore store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence
        PASS [   0.846s] rustfs-ecstore store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable
     Summary [   0.987s] 3 tests run: 3 passed, 5869 skipped
Log SHA-256: b5da2a927ff18ca9aa217762f7186a6d4371bd9e8feedd7b77940e10b64dffac

Command:
cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
Exit 0; wall 1084.508s.
Observed output:
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 18m 03s
Log SHA-256: 28f98be7970faaa71cd118da1da4e6e2e3d23b9dda7e6449eaa7a37e741260b5

Compiled mutation command (unchanged for each red and restored run):
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast

M1: reverse only this test-local constructor flag from false to true. The mutant compiled, then failed the original recovered-item queue assertion after 30.481s. Restoring the entire observed candidate passed in 0.460s. Red log 55bf3b1535529ac0bcfb8071f9b56f9f40631007a85c315367e7e93786d65293; restored log 204d5fbe4574a7917f5e740a795d864fe28a7e7fb5e33d27b0a9cf68e2f17f52.

M2: replace only the actual awaited enqueue with false, consuming its original inputs. The mutant compiled, then failed the same original queue assertion after 30.516s. Restoring the entire observed candidate passed in 0.457s. Red log 89186330413215e231e2b06bf013cc31097ea8f4a106d91421a36d6c8986a7fd; restored log e848b42d2302401afa5dbcd9e37bf76d2e56a96ab6de5a488513b997f53c5e9b.

All temporary observation and mutation code was removed before final acceptance. Historical baseline and diagnostic failures remain recorded; they are not mutation credit. The separate Object Lock intent-retry diagnostic still fails, and these native macOS checks do not establish Linux CI recovery.

* refactor(rpc): serve internode rpc bodies with the owned Body (#8441)

* test(rpc): pin the wire bytes of every internode rpc response body

rustfs/backlog#2747 batch 4, step 1 of 2, on the s3s bodies the rpc paths
serve today. Each response body builder in storage/rpc/http_service.rs is
served through hyper's HTTP/1.1 server and the raw bytes the peer reads are
compared with goldens captured from this commit: status, content-length or
chunked framing, chunk boundaries, the terminating chunk, HEAD, and the cut
connection a failure after data produces. The capability responses carry a
per-process proof, so their length header and payload are checked against
the body they encode instead.

The read_file_stream response builder moves into read_file_stream_response
so the test reaches the same code the handler runs; behaviour is unchanged.

* refactor(rpc): serve internode rpc bodies with the owned Body

rustfs/backlog#2747 batch 4, step 2 of 2. The internode rpc HTTP paths
build their responses with rustfs_s3_types::Body: Body::from_stream
replaces Body::from(StreamingBlob::wrap(..)) at the six stream sites, and
the in-memory and empty bodies keep their constructors. InternodeRpcService
converts its responses into the wrapped service's response body through
From<Body>, so the server keeps composing it with the s3s service as
before. The wire-byte goldens from the previous commit pass unchanged;
storage/rpc no longer names s3s.

* fix(rpc): simplify wire test response builder type

Name the boxed response builder used by the capability wire test so
all-target Clippy can check it without a type-complexity warning.

Validation (LANG=en_US.UTF-8, CARGO_INCREMENTAL=0):
- cargo clippy -p rustfs --all-targets -- -D warnings: exit 0.
- cargo nextest run --profile ci -p rustfs --lib -E 'test(/storage::rpc::/)': 279 tests run, 279 passed, 4842 skipped.
- All 19 quick-check commands, including script-tests, both Python scripts and cargo fmt --all --check: exit 0.
- Footprint baseline regenerated: files=194, s3_error_lines=1584, ecstore_files=36; test-count baseline regenerated: 52 crates, 21228 tests, 164 ignored.
- All 54 Cargo.toml files parse without duplicate keys.

No assertion was added or changed by this lint fix. R1-R8 are retained
in the existing PR verification record.

* fix(ecstore): resume fenced Object Lock creation intents

* fix(ecstore): bound creation retry futures on the stack

* fix(ecstore): require native proof for creation retries

* fix(ecstore): bound bucket creation stack usage (#8466)

* fix(ecstore): retain creation fences through rollback

Keep lifecycle, metadata-transaction, and namespace guards around rollback and metadata cleanup. Preserve the original publication error when a guard is lost. Build the timestamp retry fixture through the real interrupted producer so it checks a valid pending creation.

Validated source tree: 4e8e207999.
Two independent source reviews found no blocking regression.

Validation (LANG=en_US.UTF-8):
- bash -c 'shellcheck --version && actionlint' — exit 0
- cargo fmt --all --check — exit 0
- ./scripts/check_unsafe_code_allowances.sh — exit 0
- ./scripts/check_layer_dependencies.sh — exit 0
- ./scripts/check_architecture_migration_rules.sh — exit 0
- ./scripts/check_logging_guardrails.sh — exit 0
- ./scripts/check_error_other_format_ratchet.sh — exit 0
- ./scripts/check_no_tokio_io_uring.sh — exit 0
- ./scripts/check_extension_schema_boundaries.sh — exit 0
- ./scripts/check_body_cache_whitelist.sh — exit 0
- ./scripts/check_s3s_footprint.sh — exit 0
- ./scripts/check_test_count_ratchet.sh — exit 0
- ./scripts/check_fips_wording.sh — exit 0
- ./scripts/check_embedded_secrets.sh — exit 0
- make script-tests — exit 0
- python3 ./scripts/test_object_generation_protocol_model.py — exit 0
- python3 ./scripts/check_test_wiring.py — exit 0
- ./scripts/check_no_planning_docs.sh — exit 0
- ./scripts/check_uring_lane_lib_only.sh — exit 0
- cargo nextest run --profile ci -p rustfs-ecstore --lib --locked --no-fail-fast — 5894 passed, 0 skipped
- cargo check -p rustfs-ecstore --all-targets --locked — exit 0
- cargo nextest run --profile ci -p rustfs --lib --locked --no-fail-fast -E 'test(=admin::handlers::table_catalog::tests::cold_strong_registration_completes_while_other_bucket_backup_waits) | test(=admin::handlers::table_catalog::tests::commit_backend_borrowed_put_preserves_owner_authorization_and_precondition) | test(=admin::handlers::table_catalog::tests::disable_table_bucket_preserves_objects_and_can_be_reenabled) | test(=admin::handlers::table_catalog::tests::load_table_handler_negotiates_vended_credentials_without_breaking_metadata_only_callers) | test(=admin::handlers::table_catalog::tests::namespace_property_handler_updates_object_backed_catalog_and_maps_errors) | test(=app::object::get::tests::execute_get_object_resumes_from_relocated_pool_without_splicing_body) | test(=table_catalog::store::tests::catalog_writer_loss_prevents_recovery_storage_commit)' — 7 passed
- cargo nextest run --profile ci -p rustfs --lib --locked --no-fail-fast -E 'test(unsupported_remote_disk_)' — 2 passed

Additional fault-injection evidence remains incomplete. Timestamp M1 reached its original assertion; its restore run is being collected. M2–M5 and the three planned rollback faults have not run. An automated risk check interrupted the timestamp proof task; no assertions or validation settings were weakened.

* fix(ecstore): box nested bucket rollback state

Keep the guarded rollback future behind a heap allocation so bucket creation does not store its nested state inline. The rollback body, all three fence references and pending-proof revocation order are unchanged.

The same three original Linux tests overflowed on both e611 CI attempts. Local format, ecstore test compilation and all 19 quick-check steps pass; the original Linux feature lanes still need to validate this candidate.

* test(ecstore): isolate rollback guard loss fixtures

Run the three rollback metadata-cleanup cases in separate test processes so reused /proc/self/fd aliases from earlier cases cannot invalidate their creation witnesses. Keep the original fixture, guards, assertions and io_uring test selection, and verify that each child runs the intended case successfully.

Update the generated test-count baseline for the added macro arm.

Validation: cargo check -p rustfs-ecstore --tests; cargo fmt --all --check; all 19 quick-check commands, including make script-tests and the two Python checks. Linux io_uring acceptance and fault controls remain pending.

* refactor(replication): move the replication config onto gateway types (#8443)

* wip: T1.4 replication family

* refactor(replication): move the replication config onto gateway types

The replication configuration family leaves s3s: rustfs-replication no longer depends on s3s, and ECStore bucket metadata, the scanner and the rustfs app/admin layers hold rustfs_gateway_types::persistence::PersistedReplicationConfiguration. The persisted XML goes through the gateway persistence codec behind a new BucketConfigXml trait (the 39 byte goldens pass with unchanged fixtures); the legacy s3s edge converts once in storage/ecfs.rs through storage/s3_compat/replication.rs; the scanner data usage cache keeps its MessagePack bytes through a serde mirror pinned by bytes recorded from s3s 5761ddfe. rustfs-gateway-types 0.32.0 is pinned once (rev 772d3385). Refs rustfs/backlog#2745.

* test: use S3 serializer for list response tests

* fix(ci): restore integration test compilation

Gate the diagnostic test import with its feature, initialize the conditional
MetadataWriteStore fields explicitly, and decode the scanner replication fixture
through its existing cache boundary. Preserve the original fixture and assertions.

Validation was run on tree 544acb80c0 at 6003d4b8.
LANG=en_US.UTF-8; no source changes follow validation.

Quick checks: 19/19 passed.
- shellcheck --version && actionlint: exit 0
- cargo fmt --all --check: exit 0
- ./scripts/check_unsafe_code_allowances.sh: exit 0
- ./scripts/check_layer_dependencies.sh: exit 0
- ./scripts/check_architecture_migration_rules.sh: exit 0
- ./scripts/check_logging_guardrails.sh: exit 0
- ./scripts/check_error_other_format_ratchet.sh: exit 0
- ./scripts/check_no_tokio_io_uring.sh: exit 0
- ./scripts/check_extension_schema_boundaries.sh: exit 0
- ./scripts/check_body_cache_whitelist.sh: exit 0
- ./scripts/check_s3s_footprint.sh: exit 0
- ./scripts/check_test_count_ratchet.sh: exit 0
- ./scripts/check_fips_wording.sh: exit 0
- ./scripts/check_embedded_secrets.sh: exit 0
- make script-tests: exit 0
- python3 ./scripts/test_object_generation_protocol_model.py: exit 0
- python3 ./scripts/check_test_wiring.py: exit 0
- ./scripts/check_no_planning_docs.sh: exit 0
- ./scripts/check_uring_lane_lib_only.sh: exit 0

Darwin validation:
- cargo clippy --locked -p rustfs-ecstore -p rustfs-scanner --all-targets -- -D warnings: exit 0
- cargo nextest run --locked -p rustfs-ecstore -p rustfs-scanner --lib -E 'test(config::com::tests::save_config_preserves_retryable_system_volume_errors) | test(scanner_io::tests::replication_maintenance_refreshes_compacted_stats_across_cycles_and_retries)' --test-threads 1 --no-fail-fast: exit 0, 2/2 passed
- cargo clippy --locked -p rustfs-ecstore -p rustfs-scanner --all-targets --features rustfs-ecstore/diagnostic-read-version-caller,rustfs-scanner/diagnostic-read-version-caller -- -D warnings: exit 101
- cargo nextest run --locked -p rustfs-ecstore -p rustfs-scanner --lib --features rustfs-ecstore/diagnostic-read-version-caller,rustfs-scanner/diagnostic-read-version-caller -E 'test(config::com::tests::save_config_preserves_retryable_system_volume_errors) | test(scanner_io::tests::replication_maintenance_refreshes_compacted_stats_across_cycles_and_retries) | test(cluster::rpc::runtime_sources::tests::) | test(config::com::tests::diagnostic_config_write_retains_origin_and_api_for_results_and_cancellation)' --test-threads 1 --no-fail-fast: exit 0, 14/14 passed

Diagnostic strict Clippy remains red on four unchanged production lints:
runtime_sources.rs if_same_then_else and three redundant_locals reports in
set_disk/core/metadata_read.rs and set_disk/read.rs. Their source ranges and
feature wiring match the base; the base was not compiled separately.

Linux aarch64 validation (Rust/Clippy 1.99.0, UID 501, jobs=1):
- cargo clippy --all-targets -- -D warnings: exit 0; Finished dev profile in 68m 12s
- cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings: exit 0; Finished dev profile in 49m 45s
Source HEAD/tree and clean status matched before and after both commands.
No compiler or stack overrides were used. protoc 35.1 was installed from its
verified ARM64 release; flatc was absent and not invoked by the workspace build scripts.
The local environment differs from hosted CI.

* fix(ecstore): recover delete batches with missing versions

Treat an explicit non-null version as a no-op when it is absent from both recovery sources. Keep rejecting missing null versions, delete-marker mismatches and conflicting local generations.

Add returning-disk and partial-receipt regressions, extend journal boundary checks, and regenerate the ecstore test-count baseline. Production faults exercise the 24 added assertion sites and 7 semantic checks: 33 faults failed at the expected sites and passed after full restoration.

Validation (LANG=en_US.UTF-8, one build job):

cargo nextest run --locked -p rustfs-ecstore --lib -E 'test(delete_recovery) | test(quorum_delete_cleanup_coordinator_binds_single_and_batch) | test(crash_delete_retry_batch_reclaims_inline_and_reserved_external)' --test-threads 1 --no-fail-fast
Summary [  34.580s] 69 tests run: 69 passed, 6058 skipped; exit 0

cargo clippy --locked -p rustfs-ecstore --all-targets -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 5m 37s; exit 0

All 19 final Quick Checks completed with exit 0:

- shellcheck --version && actionlint
- cargo fmt --all --check
- ./scripts/check_unsafe_code_allowances.sh
- ./scripts/check_layer_dependencies.sh
- ./scripts/check_architecture_migration_rules.sh
- ./scripts/check_logging_guardrails.sh
- ./scripts/check_error_other_format_ratchet.sh
- ./scripts/check_no_tokio_io_uring.sh
- ./scripts/check_extension_schema_boundaries.sh
- ./scripts/check_body_cache_whitelist.sh
- ./scripts/check_s3s_footprint.sh
- ./scripts/check_test_count_ratchet.sh
- ./scripts/check_fips_wording.sh
- ./scripts/check_embedded_secrets.sh
- make script-tests
- python3 ./scripts/test_object_generation_protocol_model.py
- python3 ./scripts/check_test_wiring.py
- ./scripts/check_no_planning_docs.sh
- ./scripts/check_uring_lane_lib_only.sh

test-count ratchet OK: 52 crates, 22009 tests, 165 ignored match .config/test-count-baseline.txt
PASS: cold stampede script self-test
model: states=51023, transitions=167520, terminal=732, chosen_candidates=2
multi-slot round schedules: 640
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired

* fix(ecstore): retry tier commit intent lock contention

Retry transient namespace lock timeouts within the existing three-attempt
CAS budget. Reload and verify the original mutation identity after
contention, and preserve the final typed error when attempts are exhausted.
Cover save/read contention, exhausted retries, replacement identities and
CAS reconciliation without changing the existing timeout or lock policy.

Validation:
- cargo nextest run --locked -p rustfs-ecstore --features test-util --lib -E 'test(tier_mutation)' --test-threads 1 --no-fail-fast: 52 passed.
- cargo nextest run --locked -p rustfs-ecstore --features test-util --lib -E 'test(intent_advance_) | test(=services::tier::tier::tests::caller_cancellation_after_durable_prepare_does_not_hide_the_mutation)' --test-threads 1 --no-fail-fast: 5 passed.
- cargo clippy --locked -p rustfs-ecstore --all-targets -- -D warnings: passed.
- cargo clippy --locked -p rustfs-ecstore --all-targets --features test-util -- -D warnings: passed.
- 27 mutation pairs failed at the intended assertions and passed after restoring the production code.
- bash scripts/check_test_count_ratchet.sh --update, then bash scripts/check_test_count_ratchet.sh: 52 crates, 22013 tests, 165 ignored; ecstore increases from 6581 to 6585.

* refactor(lifecycle): replace s3s domain types with Gateway DTOs

Keep lifecycle rules, restore requests, Object Lock, and scanner cached tags in domain types. Convert at the existing HTTP boundary while preserving legacy XML, scanner cache bytes, date ordering, and tombstones.

Linux test validation before the final import-path-only adjustment:

cargo nextest run -p rustfs-ecstore bucket_config_goldens
39 passed; 6211 skipped by selector.

cargo nextest run --lib -p rustfs -E 'test(/test_zero_rule_lifecycle_tombstone_recognition/) | test(/test_local_lifecycle_staleness_axis_selection/) | test(/test_lifecycle_staleness_preserves_legacy_submillisecond_axis/) | test(/test_merge_incoming_lifecycle/) | test(/test_lifecycle_expiry_statement_matrix/) | test(/test_lifecycle_expiry_subset_xml/) | test(/put_bucket_lifecycle_validation_errors_keep_their_s3_code/) | test(/execute_put_bucket_lifecycle_configuration_rejects_missing_configuration/) | test(/execute_restore_object_rejects_missing_restore_request/) | test(/execute_restore_object_rejects_select_type/) | test(/execute_restore_object_maps_failures_to_typed_s3_errors/) | test(/execute_restore_object_returns_internal_error_when_store_uninitialized/) | test(/restore_request_conversion_preserves_legacy_output_values/) | test(/restore_request_conversion_preserves_presence_and_raw_enums/) | test(/restore_request_conversion_distinguishes_absent_and_empty_containers/)'
25 passed; 5318 skipped by selector.

cargo nextest run --profile ci -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-replication -p rustfs-scanner
7872 passed; 21 skipped under the existing CI profile.

Previously verified mutation controls: 39 production faults, 41 RED/GREEN pairs.

cargo clippy --locked -p rustfs -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-scanner --all-targets -- -D warnings
Passed in 40m 45s before the import-path adjustment, then in 1m 59s on
the final source with warnings denied. The final adjustment imports the
existing helper through the object domain facade; fmt and the architecture
check also passed.

Part of rustfs/backlog#2745. The remaining ecstore configuration families follow separately.

* refactor(admin): move error types to rustfs-s3-types (#8440)

* refactor(protocols): move error types to rustfs-s3-types by codemod

T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and
the error items of protocols move off s3s by scripts/codemods, run with
--hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it
downcasts the backend error, whose runtime type is still s3s::S3Error.

- rustfs/src/protocols/client.rs: the two webdav session methods now
  return rustfs_s3_types::S3Result and convert the s3s error with From.
- codemod: a child module that calls s3_error! through `use super::*`
  gets its own `use rustfs_s3_types::s3_error;` once its parent imports
  that macro by name; otherwise it is listed as GLOB.
- s3-types: a table test pins code, status and the s3s error the legacy
  edge renders for every variant a moved call site names.
- s3_error_lines 1585 -> 1584 (a lifecycle comment reworded).

* refactor(admin): move error types to rustfs-s3-types (WIP, no build)

T1.2 batch 4 (rustfs/backlog#2743), work in progress: rustfs check
still fails with 16 errors (listed in the PR body).

- codemod: --facade PATH treats a crate-local re-export of s3s's error
  items like s3s for the four names (9 new self-tests, 9 mutants red).
- codemod output for rustfs/src/admin and rustfs/src/site_replication
  (--facade crate::storage_api::site_replication::s3); 6 glob children
  get an explicit rustfs_s3_types::s3_error import.
- admin/router.rs: the Operation trait returns rustfs_s3_types::S3Result.
- error.rs: From<ApiError> builds the rustfs-s3-types error; the s3s
  conversion is derived from it.

* fix(admin): convert error types at the s3s boundary

Batch 4 of T1.2 left the admin and site_replication modules returning
rustfs_s3_types::S3Error while their callers and callees on the storage
and auth side still speak s3s::S3Error. Convert at each edge with the
existing From impls in rustfs-s3-types (compat_s3s.rs) instead of
changing any code, message or status:

- admin/auth.rs, admin/handlers/object_zip_download.rs,
  admin/handlers/table_catalog/mod.rs: map the s3s result returned by
  check_key_valid / authorize_request / authorize_internal_object_request
  into the admin error type with .map_err(Into::into).
- storage/ecfs.rs: the s3s-facing side maps the admin site_replication
  gate results the other way.
- admin/handlers/pools.rs: ApiError.code stays s3s::S3ErrorCode, so the
  rustfs code is converted with .into() when the struct is built, in
  production and in the two tests that build or compare it.
- admin/handlers/heal.rs, admin/runtime_sources.rs: the router returns
  s3s errors (S3Route impl); the expected rustfs code is converted with
  .into() before the comparison.
- admin/handlers/table_catalog/mod.rs: iceberg_rest_error takes
  &'static str so the Cow<'static, str> custom code borrows it; every
  caller already passes an ICEBERG_ERROR_* constant or a &'static str.
- admin/handlers/quota.rs: drop a now same-type .into() after s3_error!.
- storage_api.rs, site_replication/mod.rs: drop re-exports and an import
  that no longer have users (clippy -D warnings).

cargo check -p rustfs --all-targets and cargo clippy -p rustfs
--all-targets -- -D warnings are clean; the s3s footprint baseline is
unchanged at files=172, s3_error_lines=303, ecstore_files=28.

Refs rustfs/backlog#2743, rustfs/backlog#2734.

* refactor(ecstore): move bucket configs onto gateway persisted types (#8709)

* refactor(ecstore): move bucket configs onto gateway persisted types

Finish the ECStore batch of T1.4 (rustfs/backlog#2745). The remaining
eleven bucket configuration families (Website, CORS, Tagging, Versioning,
ObjectLock, Encryption, Notification, Logging, Accelerate, RequestPayment,
PublicAccessBlock) now read and write their stored XML through the
rustfs-gateway-types persistence and cors_tagging codecs instead of the
s3s codec, so the legacy s3s bucket-config codec and its
s3gate-metadata-xml compat marker are removed.

- ecstore: BucketConfigXml is implemented by the gateway codec for every
  family; metadata, metadata_sys, versioning, object lock, replication and
  lifecycle consumers take the persisted types. Region moves to
  rustfs_s3_types::Region, and the ETag conversion to s3s moves to the
  rustfs edge with its tests. Only warm_backend.rs and warm_backend_s3.rs
  still reference s3s, held for T1.5 (rustfs/backlog#2746).
- scanner: object lock cache, item actions and data usage use the
  persisted types.
- rustfs: the legacy S3 edge converts each family between s3s DTOs and the
  persisted types in storage/storage_api.rs by writing one codec's XML and
  reading it with the other; Tagging converts field by field.
- Bump the rustfs-gateway-types pin to 21283e87.

The 39 persisted-byte goldens are unchanged and pass, so a document the
old codec wrote reads back and re-encodes to the same bytes. s3s footprint:
files 181 -> 146, ecstore_files 28 -> 2, s3_error_lines unchanged.

* test(app): prepare the PutBucketEncryption config from the request

The uninitialized-store test passed an empty persisted configuration, which the use case now validates before it reaches the store check, so it answered MalformedXML instead of InternalError. Convert the request's configuration the way the S3 edge does.

* style(app): format the PutBucketEncryption test

* refactor(s3-client): move remote error codes and DTO enums off s3s (#8783)

Part of T1.5 (rustfs/backlog#2746), batch 1.

* refactor(s3select): run queries on gateway DTO request shapes (#8794)

* refactor(s3select): run queries on an engine request built from gateway DTO shapes

Part of T1.5 (rustfs/backlog#2746), batch 2.

* fix(s3select): keep the select request conversion in the app layer

The layer guard forbids app -> storage; the conversion moves from storage/s3_compat/select.rs to app/select_request.rs unchanged.

* fix(heal): retry transient metadata conflicts (#8812)

* fix(heal): retry transient metadata conflicts

* chore: update heal test count baseline

* test: reserve nextest capacity for progress probes (#8818)

* test(e2e): use a non-default region for the differential negative control

Since main's region-aware CreateBucket and GetBucketLocation (#8739, #8743), us-east-1 answers an empty LocationConstraint and 200 for an owned re-create, so the control no longer reported its own region. Also refresh the e2e-full linux selection from the CI listing (682 tests).

* refactor(protocols): move StorageBackend to gateway types (#8825)

* refactor(protocols): move storage backend to gateway types

* test: update e2e test count baseline

* fix: preserve copy source keys and unknown lengths

* test: update protocols test count baseline

* fix(protocols): preserve typed SFTP backend errors

* test(protocols): use gateway list input in policy tests

* fix(protocols): clear clippy failures after T1.5 merge (#8831)

fix(protocols): clear clippy failures after merge

* fix(ecstore): finalize restore metadata after local copy-back (#8836)

* fix(ecstore): finalize restore metadata after local copy-back

* test(ecstore): ratchet added restore finalization cases

* fix(ecstore): keep restore copyback compatible with older peers

* fix(ecstore): gate restore copyback on fleet compatibility

* fix(ecstore): retain transition identity during restore copyback

* fix(ecstore): route committed restore copyback locally

* fix restore copyback cleanup

* fix(ecstore): honor restore probe test fallback

* refactor(storage): migrate option errors to domain types (#8839)

* refactor(s3): migrate private multipart validation errors (#8844)

* refactor(auth): derive policy conditions from AuthzFacts (#8442)

* refactor(auth): extract authorization facts

* test(e2e): update smoke selection baseline

* test(e2e): update full selection baseline

* refactor(admin): migrate health and prefix usage to owned requests (#8887)

* refactor(admin): add owned health router boundary

* refactor(admin): migrate prefix usage to owned request

* refactor(admin): complete the owned admin request boundary (#8949)

* refactor(admin): complete the owned admin request boundary

Every type implementing the owned admin `Operation` now serves the legacy
router through one blanket impl that converts the s3s request and response
at that single edge, so handlers register with `AdminOperation(&Handler {})`
and need no adapter statics; `LegacyAdminAdapter` is removed.

`AdminRequest` gains the accessors handlers read today (`remote_addr`,
`client_info`, `request_context`, `req_info`, `req_info_mut`,
`install_req_info`, `envelope_mut`, `take_body`), the request helpers gain
envelope forms (`authenticate_envelope_credentials`,
`object_store_from_envelope`, the table-catalog `*_from_envelope` helpers,
`KmsAdminAudit::from_envelope`, `AccountAuditContext::from_envelope`,
`CallerIdentity::resolve_envelope`), `read_compatible_admin_body` accepts
either body type, and the owned `Body` gains `store_all_limited` and
`Default`. `admin::operation::test_support::TestRequest` builds owned
requests for handler tests. Unused accessors carry a per-method dead-code
expectation that each migrating handler batch removes.

No handler behavior changes; the remaining migrations follow in batches.

* chore(admin): add the T1.12 admin handler codemod

The script rewrites an admin handler file from the s3s request and response
types to the owned boundary (imports, signatures, request field and helper
calls, body access, response construction, and test request literals) and
leaves a TODO comment at every site it cannot rewrite mechanically. It is
idempotent and is removed once T1.12 completes.

* refactor(admin): migrate replication and KMS handlers to owned requests (#8951)

Move the T1.12 batch 2 admin handlers (site_replication, kms_keys, event,
replication, diagnostics, on_demand_migration, audit, account, kms_rekey,
kms_key_lifecycle, object_data_cache, tls_debug, integrity) from the s3s
request/response types to the owned AdminRequest/AdminResponse boundary.
Handler logic, authorization order, error codes and response bytes are
unchanged; the request/response plumbing now goes through the envelope
accessors and helpers introduced by the owned boundary.

The shared response helpers (utils::json_response, utils::empty_response,
handlers::admin_json_response, target_descriptor::build_json_response,
profile::authorize_profile_request) and the site replication repair
executor still answer with the s3s types, so the migrated files carry
owned-boundary forms of those helpers and convert the repair response at
its call edge. The streaming console-log body and the two devnull drains
move from the s3s byte stream to the owned Body stream/frame API.

Drop the dead_code expectations for the owned accessors and envelope
helpers this batch starts using, and lower the s3s footprint baseline to
the measured 99 files.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* refactor(admin): add owned forms of the shared admin response helpers

Handler batches were copying `json_response`, `empty_response`,
`admin_json_response`, and `build_json_response` into each migrated file
because the shared helpers still return the s3s response. Provide owned
forms next to each legacy helper (`owned_json_response`,
`owned_empty_response`, `owned_admin_json_response`,
`owned_build_json_response`, `authorize_profile_envelope`) with parity
tests against the legacy shape, derive `Debug` for `AdminResponse` so
tests can use `expect_err`, and fix three codemod rules: strip owned
string conversions in test credentials, annotate non-literal URI parses,
and leave a rebound request parameter immutable.

* refactor(admin): use the shared owned response helpers in batch 2 (#8954)

Replace the per-file owned-boundary copies of json_response,
empty_response, admin_json_response, build_json_response and
authorize_profile_request in the batch 2 admin handlers with the shared
owned_* helpers and authorize_profile_envelope that the boundary branch
now provides, and restore expect_err in the handler tests now that
AdminResponse implements Debug; the `.err().expect(..)` stand-ins were
failing the base branch's clippy step (clippy::err_expect under
-D warnings). The site replication repair call edge
keeps its conversion because the repair executor still answers with the
s3s response type.

Drop the dead_code expectations on the shared owned helpers this batch
now uses in non-test code.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* fix(admin): preserve legacy body limit error responses

* refactor(admin): move policy, ILM and batch 3 handlers to owned requests (#8956)

Batch 3 of T1.12 (rustfs/backlog#2788, parent rustfs/backlog#2734) moves
14 admin handler files to AdminRequest/AdminResponse and the owned
Operation trait: ilm_transition, policies, idp_compat, config_admin,
plugins_instances, profile_admin, group, heal, object_zip_download,
kms_key_metadata, durability, inspect_archive, sts and
gateway_key_inventory. Handler logic, authorization order, error codes
and response bytes are unchanged; test request literals use
test_support::TestRequest.

The handlers use the shared owned helpers (owned_json_response,
owned_admin_json_response, owned_build_json_response,
authorize_profile_envelope). The dead_code expectations on the
AdminRequest accessors and RequestEnvelope::uri_mut they now exercise
are removed. The legacy target_descriptor::build_json_response keeps
only its equivalence test caller and carries the matching expectation
until it retires.

object_zip_download builds the s3s request the S3 gate still reads from
the owned request facts; heal authenticates through
authenticate_envelope_credentials; the trace, inspect-archive and zip
bodies use Body::from_stream. extract_plugin_instance_filters takes the
URI so extensions.rs keeps calling it, and the two idp_compat
delegations stay on the legacy trait until their delegates migrate.

The s3s footprint baseline drops from 99 to 90 files.

* test(admin): anchor the replication diff body contract on the call

The source-contract test looked for `let body = read_compatible_admin_body(`
on one line; rustfmt wrapped that binding after the handler moved to the
owned request boundary, so the marker was no longer found. Anchor on the
call itself so formatting cannot change what the contract checks.

* refactor(admin): migrate table catalog and batch 1 to owned requests (#8955)

* fix(admin): preserve catalog body limit response bytes

* refactor(admin): migrate identity and cluster handlers to owned requests (#8966)

* refactor(admin): migrate identity and cluster handlers to owned requests

Move the T1.12 batch 4 admin handlers (user, service_account, oidc, tier,
system, pools, quota, kms_dynamic, kms_backup, rebalance, module_switch,
bucket_meta, cluster_snapshot, realtime, bucket_recovery, profile) from the
s3s request/response types to the owned AdminRequest/AdminResponse
boundary. All 90 impl Operation blocks now take AdminRequest and return
AdminResponse; handler logic, authorization order, error codes and response
bytes are unchanged. The handlers use the shared owned helpers
(owned_json_response, owned_admin_json_response, authorize_profile_envelope,
the envelope-form runtime sources and audit constructors) instead of local
copies.

profile.rs drops the s3s forms of authorize_profile_request and
profile_not_implemented_response now that profile_admin.rs is on the
envelope gate too; profile_not_implemented_response answers with an
AdminResponse. The realtime metrics stream moves from the s3s ByteStream
adapter to Body::from_stream, and the rebalance idle-404 document still
renders through s3s::S3Error::to_http_response with its body converted at
the edge.

The idp_compat delegators ImportIamV2 and AddServiceAccountLdap move to the
owned trait as well: with their targets on both traits, the method-call
delegation became ambiguous between the owned impl and the router blanket
impl, so they now take the owned request directly.

KmsAdminAudit::from_request loses its last production caller with this batch
and is deleted; its comparison test now asserts the envelope facts directly.

Lower the s3s footprint baseline to the measured 68 files.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* refactor(admin): remove the legacy admin helpers batch 4 superseded

Batch 4 moved the last production callers of the s3s-typed admin helpers
to their owned forms, which left `admin_json_response`, `json_response`,
`authenticate_request`, `authorize_admin_request`, `object_store_from_req`
and `object_store_from_extensions` unused, along with the `Body`,
`S3Request` and `S3Response` re-exports of the admin `s3` facade. Remove
them. The owned forms keep their documentation; the behavior tests of the
removed `json_response` now exercise `owned_json_response`, and the
compat-prefix test asserts the owned response directly.

Two tests existed only to compare a removed legacy helper with its owned
twin (`owned_json_response_matches_the_legacy_shape`) or to cover the
removed legacy gate (`authorize_admin_request_without_credentials_is_rejected`);
they go with the helpers, and the owned equivalents stay. The rustfs
test-count row drops from 5957 to 5955 and the s3s footprint from 68 to 66
files.

* refactor(admin): drop the legacy target descriptor JSON response helper (#8968)

`target_descriptor::build_json_response` returned the s3s response and
had no production caller left once the event, audit and plugin instance
handlers moved to `owned_build_json_response`; it survived only for the
legacy/owned equivalence test. Remove it, give the owned helper its own
documentation, and make the test assert the owned response directly:
status, `application/json`, the echoed `x-request-id` when present and
only then, and the body bytes. The file no longer imports s3s, so the
footprint baseline drops from 66 to 65 files.

* test(ecstore): pin native history on cancellation under the cancellable queue

main moved the local write path onto the disk blocking queue, which skips an
operation once its waiter is dropped, so a borrowed write cancelled while it is
queued no longer creates its file. The integration-only native publication
test still expected the queued open to run. Assert the new execution model
instead (the open never runs) while keeping its safety claims: the Started
record is taken before queueing and the cancelled publication never regrants
native history.

* chore(gateway): raise test-count rows for tests merged from main

* test(s3select): use the owned JSON input types in merged planner tests

* fix(ecstore): scope the merged direct-writer test hook to test builds

* fix(ecstore): create parents before admitted truncate opens

Align the heal tests with removal-disabled preservation and local-only outcome
fields, and refresh the e2e-full selection for the synced tests.

* chore(gateway): update test-count baseline after main sync

Main #9241 removed the node pool-metadata readiness cache with 8 tests
and added 6, so the rustfs row drops from 6440 to 6438. The ecstore and
s3select rows rise for tests added by #9237, #9239 and #9241.

* feat(scripts): add junit_diff.py for dual-stack JUnit comparison (#9275)

* ci(gateway): make the test-count ratchet a floor, not exact (#9278)

ci(gateway): make the test-count ratchet a floor instead of an exact baseline

Tests below a row still fail; tests above it, a member without a row and ignores below the row now pass with a note. Ignores above a row still fail. Adds contract tests wired into make script-tests.

* test(guards): read file plus split directory in text guards

Text guards that read one file with include_str! or by path broke, or silently stopped seeing the code they guard, when that code moved to another file. They now read the file they read today, which stays required, plus the sibling directory a mechanical split will populate (storage/legacy_edge, app/bucket_usecase, app/multipart_usecase, app/lifecycle_transition_api_test). Counts are summed over the set and split markers must occur exactly once across it. Refs rustfs/backlog#2765.

* chore(gateway): record ignored microbenchmarks merged from main (#9290)

chore(gateway): record the ignored microbenchmarks merged from main

main added three throughput microbenchmarks that are ignored by design
(#9217): one each in rustfs-checksums, rustfs-rio and rustfs-utils. The
test-count ratchet treats a higher ignored count as a failure until the
row is raised, so raise those three rows to 1 and tighten the test rows
to the merged tree with --update.

* test(guards): avoid a typos false positive in a fixture

* ci(s3-tests): add reference parity lane against pinned 1.0.5 (#9288)

* chore(scripts): add codemod that splits bucket_usecase by family

Moves each configuration family of rustfs/src/app/bucket_usecase.rs and its tests into its own file under bucket_usecase/ through explicit name-to-file tables, and stops on any item without an assignment. Preparatory refactor P2 of the gateway migration.

* refactor(app): split bucket_usecase into one file per config family

Pure move produced by scripts/codemods/split_bucket_usecase.py: the struct, admission control and shared helpers stay in bucket_usecase.rs; the operations of each configuration family, their helpers and their tests move to bucket_usecase/<family>.rs, request fixtures to test_support.rs and the source-text guards to guards.rs. Empty stubs reserve the files the next batches add. No signature, body or test changes.

* docs: point the metadata listing reference at bucket_usecase/listing.rs

---------

Co-authored-by: rustfs-cla-assistant[bot] <273121464+rustfs-cla-assistant[bot]@users.noreply.github.com>
2026-11-11 18:59:00 -05:00
Chris An 83ce509ff3 ci: run the gateway smoke lane only where the gateway stack is touched (#9344)
* ci: run the gateway smoke lane only where the gateway stack is touched

The report-only `e2e-smoke (gateway, report-only until T2.10)` lane ran on
every pull request and failed on every one of them: the gateway S3 stack
does not serve the whole e2e-smoke suite before T2.10. A pull request that
touches none of the lane's inputs can neither cause nor fix that failure,
yet it was shown a red check and occupied an sm-standard-2 runner.

Select CI scope now publishes one more output, `gateway_smoke`, computed by
`scripts/ci_gate.py select` from the change set: true when a path in
GATEWAY_SMOKE_PATHS changed or a `rustfs-gateway*` pin in Cargo.toml or
Cargo.lock differs, and on every event that is not a pull request. The lane
is skipped only on a pull request whose selection is the literal `false`; a
missing or refused selection runs it. Job-level continue-on-error stays, and
no step may carry it, so a failed suite is never rendered green.

`ci_gate.py --check-workflow` pins the condition by truth table: the lane
must run wherever End-to-End Tests runs except on unselected pull requests,
must need classify-changes, and the output must be published.

* ci: keep the e2e suite's own files out of the gateway smoke selection

Feature pull requests add e2e tests and touch the smoke selection all the
time, and until T2.10 the gateway lane cannot pass on any of them, so
selecting on `crates/e2e_test/*`, `.config/nextest.toml` and
`.config/e2e-smoke-selection.txt` kept producing the red check this change
exists to remove. What the suite contains carries no signal while the stack
cannot serve it. The gateway code paths, the pin comparison, the workflow,
the test-runtime action and the two scripts stay.

An e2e-only pull request being skipped is now a pinned self-test case. At
T2.10 the selection is deleted, not extended.
2026-11-11 18:52:15 -05:00
唐小鸭 e2bd880fa4 fix(kms): make AWS-backed SSE objects readable and key policies hold (#9334)
* fix(kms): read back SSE objects written through the AWS backend

The AWS backend returned the GenerateDataKey blob bare, so the read path took the unrecognized bytes for a locally wrapped key and asked for RUSTFS_SSE_S3_MASTER_KEY without ever calling AWS Decrypt. Every SSE-KMS and SSE-S3 object written through this backend was unreadable, and UploadPart failed the same way.

The backend now wraps the blob in the data-key envelope the other backends use. A stored key that is neither an envelope nor a local-master-key wrap is offered to the KMS when its backend reports that released versions persisted it bare, so objects written by 1.0.4 and 1.0.5 read back without being rewritten.

* fix(admin): percent-decode the key id in the KMS describe path

GET /rustfs/admin/v3/kms/keys/{key_id} read the path segment verbatim, so an AWS key ARN, which has to be escaped to fit in one segment, was looked up, authorized and audited under its escaped spelling and answered 404.

* fix(policy): match an AWS key ARN as its key id in KMS statements

A key-scoped statement was matched against the key identifier exactly as the request spelled it. A Deny on key/<id> therefore did not cover a request naming the same key by its AWS ARN, on the SSE-KMS write, on the read of an object stored under the ARN spelling, and on the per-key admin endpoints.

A key ARN is now reduced to its key id before matching. A request naming a key by alias, which only the backend can resolve, fails closed against key-scoped statements. The deletion impact report compares key references the same way.

Adds the AWS KMS backend guide in English and Chinese.

* fix(kms): read rio-v2 objects that carry a bare AWS data key

A rio-v2 build stores a sealed object key beside the data key, so an AWS-backed object written before the envelope carries the same shape as a MinIO object and was sent to the MinIO decoder, failing before AWS Decrypt was tried.

MinIO's sealing is authenticated, so opening the data key under the local master key is a proof: the bare blob reaches the KMS only once that has failed, or when no local master key exists. Without a backend that reads bare ciphertext the MinIO route is unchanged.

* docs(kms): describe pairing an AWS KMS site with a Vault site

Site replication synchronizes key names and never keys, and AWS assigns key ids, so in a mixed pair the Vault site has to name its keys after the AWS key ids.
2026-11-11 18:09:43 -05:00
唐小鸭 acf0d3ff72 fix(replication): replicate SSE-KMS objects under key-policy enforcement (#9340)
fix(replication): replicate SSE-KMS objects under key enforcement

With RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY=true the receiving site authorizes a replica PUT of an SSE-KMS object for kms:GenerateDataKey against the key it resolves to. The managed site-replicator account held no kms grant, so every SSE-KMS object stayed FAILED with AccessDenied.

Grant kms:GenerateDataKey on every key and treat a stored policy without it as outdated so the reconcile tick refreshes existing accounts. kms:Decrypt stays out: reads proxied to a peer run as this account.
2026-11-11 18:09:25 -05:00
Chris An 013a4ae3b3 refactor(admin): move the admin route table out of the s3s router (#9320)
* feat(server): add the RUSTFS_S3_STACK selector and dual-stack e2e lane (#8411)

* refactor(server): group the s3s-compat tower layers under legacy_compat (#8410)

Move the ten tower layers that patch s3s behaviour out of
rustfs/src/server/layer.rs into rustfs/src/server/legacy_compat/, one file
per layer with its tests. Each file header names the s3s behaviour it
patches, the gateway mechanism that replaces it (Replaced by:), and the task
that removes it (Removed by: T2.12, rustfs/backlog#2771). The layers keep
their code, their order in external_service_stack!, and their behaviour; no
layer is deleted.

Add server::http::tests::stack_census, which renders the ordered service
chain of the real external_service_stack! expansion and compares it with a
committed constant, so a reorder, insertion or removal is a visible diff.

Route the two s3s error types the moved layers use through a
storage_api::server::legacy_compat shim so the split adds no s3s-importing
file, and lower the s3s footprint baselines to the measured values (files 209
to 208, s3_error! lines 1586 to 1585). Keep the server runtime-source bypass
guard covering the moved code.

Refs rustfs/backlog#2787, rustfs/backlog#2734

* feat(s3-types): add the RustFS-owned S3 error type and s3_error! macro (#8406)

`rustfs-s3-types` now provides `S3ErrorCode` (generated by build.rs from
src/codes.txt, the grep-derived call surface, plus `Custom`), `S3Error`,
`S3Result`, `StdError` and the `s3_error` macro with the same call syntax
as s3s, so the T1.2 codemod is a `use` swap. Statuses are hand-written and
pinned to the s3s behaviour RustFS ships today by an oracle test that uses
s3s as a dev-dependency only. The `From` conversions for the legacy edge
live behind the off-by-default `compat-s3s` feature (the orphan rule
forbids them in rustfs/src/error.rs); `rustfs` enables it.

Refs rustfs/backlog#2742, rustfs/backlog#2734

* ci: add the per-crate test-count ratchet (#8404)

Quick Checks now fails any PR that lowers the number of test attributes
in a workspace crate or adds #[ignore], measured per crate against the
exact rows of .config/test-count-baseline.txt (rustfs/backlog#2734
section 1 rule 3). The counter widens the spec's literal
#[(tokio::)?test] to the tokio attributes that carry arguments and both
spellings of #[test_case(...)], and anchors at line start so
commented-out attributes do not count. The initial baseline covers 52
crates, 21077 tests and 164 ignored.

Refs rustfs/backlog#2738
Refs rustfs/backlog#2734

* test(ecstore): pin the persisted bucket-config XML bytes for 13 families (#8408)

Before any DTO or codec is replaced, pin the exact bytes that
deserialize followed by serialize produces for every persisted
bucket-configuration XML family in its three source shapes (AWS
documentation, MinIO-written, RustFS-written), and the bytes RustFS
writes today for a typed value. Byte equality only: a structural
comparison would let a codec replacement change the persisted encoding
unnoticed.

The generator script refuses to overwrite existing goldens without
--force, refuses on dirty codec paths, cannot report success from its
write mode, and records the generating commit in the fixtures README.

Refs rustfs/backlog#2744, rustfs/backlog#2734

* ci: raise the test-count baseline to the integration tree (#8426)

* ci: make the s3s footprint baseline exact and lower-only (#8403)

* ci: make the s3s footprint baseline exact and lower-only

The s3s footprint guard compared three counters against ceilings held in
the script itself, so a ceiling could sit above the tree and a drift was
fixable by raising it (rustfs/rustfs#5964 raised the s3_error! ceiling
to go green). The three values now live in
.config/s3s-footprint-baseline.txt as files, s3_error_lines and
ecstore_files, the script reads only that file, and a measurement that
differs from its baseline in either direction fails: growth prints
"s3s footprint grew: <measured> > <baseline>", shrinkage prints
"lower the baseline to <measured> in this PR". The counting method is
unchanged; the baseline is the current measurement of
gateway/integration (209 / 1585 / 36). The header records the end state:
once all three reach 0, T4.2 switches the script to failing on any
occurrence.

Refs rustfs/backlog#2739, rustfs/backlog#2734

* ci: add the s3s edge allowlist mode to the footprint guard (#8405)

`scripts/check_s3s_footprint.sh --mode allowlist` fails on every Rust
file that references s3s (`s3s::` or `s3s_sigv4`) and is not matched by
an entry of the new `.config/s3s-edge-allowlist.txt`; `--dry-run` lists
those files and exits 0. The allowlist carries the initial edge from the
spec, and the script rejects an entry that names or matches anything
under crates/ other than crates/e2e_test. The baseline mode and the
argument-less CI invocation are unchanged.

Refs rustfs/backlog#2741, rustfs/backlog#2734

* test(ecstore): reach the bucket-config codec through the test facade (#8428)

* refactor(s3-types): own the HTTP Body and drop s3s from three crates (#8431)

Add rustfs_s3_types::Body, a BoxBody<Bytes, StdError> wrapper with
empty() and From<Bytes | Vec<u8> | String | &'static str>, and the
legacy From conversions to and from s3s::Body behind compat-s3s.

The signer is now generic over the request body: it only reads and
writes the request head, so it needs no body type and its callers keep
theirs. s3-client sends its requests with the owned Body, madmin signs a
bodiless head, and targets drops its two S3Request/S3Response header
wrappers in favour of extract_params_header.

rustfs-signer, rustfs-madmin and rustfs-targets no longer depend on
s3s; the s3s footprint baseline drops from 209 to 202 files.

Refs rustfs/backlog#2747, rustfs/backlog#2734

* refactor(ecstore): move S3ErrorCode to rustfs-s3-types by codemod (#8432)

* test(ecstore): pin the InvalidRange responses of new_getobjectreader

Both error branches of new_getobjectreader build an ErrorResponse with S3ErrorCode::InvalidRange, and no test observed either code. Pin the code, its wire name and the location fields of each branch before the code type moves to rustfs-s3-types (rustfs/backlog#2743), so a changed code on either branch fails a test. The ecstore test-count row rises by the two tests.

* chore(scripts): add the s3s-to-rustfs-s3-types error codemod

Task T1.2 of rustfs/backlog#2734 (rustfs/backlog#2743) moves S3Error, S3ErrorCode, S3Result and s3_error! from s3s to rustfs-s3-types by script, one crate closure per PR. The codemod rewrites use trees (splitting mixed s3s imports) and qualified code paths, never comments or literals, adds the rustfs-s3-types dependency, drops s3s only from a crate that no longer names it, refuses every file of the s3s edge allowlist, takes --hold for a file whose type is fixed outside the closure, and prints a per-crate census. A second run changes nothing. Its self-test runs under make script-tests.

* refactor(ecstore): take S3ErrorCode from rustfs-s3-types

Output of scripts/codemods/s3_error_to_s3_types.py over crates/ecstore, crates/lifecycle, crates/scanner and crates/replication (rustfs/backlog#2743 batches 1 and 2). The ErrorResponse code field, the two InvalidRange constructions in new_getobjectreader and an unused tier import now use rustfs_s3_types::S3ErrorCode; the variant names and wire names are unchanged. lifecycle, scanner and replication reference none of the four error names. warm_backend.rs and warm_backend_s3.rs are held: they compare against rustfs_s3_client::ErrorResponse.code, which stays the s3s type until s3-client moves.

* ci(footprint): count only s3_error! lines still on s3s's macro

s3_error_lines counted every line holding the text `s3_error!`, which
also matches the RustFS-owned `rustfs_s3_types::s3_error!`. A codemod
that moves call sites onto rustfs-s3-types therefore could not lower it
(rustfs/backlog#2743, decision 3; rustfs/rustfs#8432, gotcha 2).

The counter now applies a textual, per-file and per-line rule that errs
on the side of counting (full rule in the script header):
- occurrences spelled `rustfs_s3_types::s3_error!` never count;
- a line holding `s3s::s3_error!` always counts;
- any other `s3_error!` line counts unless its file is cleared, that is,
  it imports the macro from rustfs_s3_types by name on a line starting
  with the use item (or defines it with `macro_rules! s3_error`) and does
  not also bring s3s's macro in (`use s3s::s3_error`, a `use s3s::{..}`
  tree naming it or a crate-root `*`, `use s3s::*`, or
  `#[macro_use] extern crate s3s`);
- fail closed: a file matching neither form (`use super::*`, a
  crate-local re-export, a renamed or glob rustfs_s3_types import, a
  commented-out import, a comment-only mention) counts.

Census of the 1585 lines on 96fc06776: 1478 in 64 files that import the
macro from s3s; 107 in 15 files matching neither form (105 through
`use super::*`, 1 qualified `s3s::s3_error!`, 1 comment line); none in
files that import it from rustfs_s3_types or define it. Every line stays
counted, so the baseline remains s3_error_lines=1585 (files=202 and
ecstore_files=36 unchanged). Rewriting rustfs/src/auth.rs's import to
rustfs_s3_types lowers the new counter to 1567; the old one stayed 1585.

`--self-test` (wired into `make script-tests`) copies the script into
22 fixture trees: 6 positive cases and 16 negative ones (lines that must
not count, forms that must not clear a file, and a missing baseline,
missing key or failing classification search that must fail). Each tree
has a control line that must count, and each run gets s3s text on stdin.
Seven cases were red against the old counter. 19 mutants each turn their
case red: old definition, flipped import test, dropped fail-closed
branch, dropped e2e_test exclusion, dropped s3s veto, dropped qualified
s3s rule, dropped rustfs-qualified strip, unanchored rustfs import,
renamed import accepted, nested glob taken as root glob, single-line
s3s trees only, dropped s3s glob forms, dropped macro_use form, dropped
definition clearing, ignored classification rg errors, error-line search
without its path, classification searches without their path, missing
baseline skipped, missing key skipped.

* test(e2e): compare two RUSTFS_S3_STACK servers over every S3 operation (#8435)

Add dual_stack_differential to e2e_test. It starts two rustfs processes
from the same binary, stack A on RUSTFS_S3_STACK=legacy and stack B on
RUSTFS_E2E_STACK_B (default legacy until rustfs/backlog#2734 T2.5),
sends both the same aws-sdk-s3 script covering the 73 methods of
impl S3 for FS (a normal request and a 4xx negative each), and compares
status, error code, an allowlist of response headers and a few typed
body fields. Error message text is never compared. Each run writes
<target>/dual-stack/<run>.json with compared_ops and the (op, field,
a, b) differences; any difference fails the test.

differential_detects_an_injected_difference starts B with
RUSTFS_REGION=eu-west-1 and requires GetBucketLocation to be reported.
The comparator is pure and unit-tested without a server.

Wire the module into the e2e-smoke filter and update the e2e-smoke and
e2e-full selection digests and the e2e_test test-count row.

Refs rustfs/backlog#2740

* feat(app): add the RustFS-owned RequestEnvelope and its s3s constructor (#8437)

* feat(app): add the RustFS-owned RequestEnvelope and its s3s constructor

* ci: classify the s3s edge adapters in storage/s3_compat as interface

* feat(s3-types): add Body::from_stream and move Swift off s3s::Body (#8438)

rustfs/backlog#2747 batch 3. `Body::from_stream` wraps a chunk stream of
unknown length over `StreamBody`/`BoxBody`: one data frame per chunk, the
first error ends the body, the size hint stays unknown, and dropping the
body drops the stream. Parity tests pin it to the s3s `StreamingBlob::wrap`
it replaces, at the `http_body` level and byte for byte through hyper.

The Swift handlers in rustfs-protocols build their responses as the owned
`Body`. `SwiftService` converts them into the wrapped S3 service's response
body type through `From<Body>`, so rustfs keeps serving `s3s::Body` at the
legacy edge unchanged. Request bodies stay Send-only (`UnsyncBoxBody`); the
two storage writers and the DLO/SLO segment streams get `Sync` from
`sync_wrapper::SyncStream`, which is what s3s's `http_body_unsync` mutex
provided before. The DTO and error uses of s3s in protocols are untouched
(T1.5 and T1.2).

* refactor(protocols): move error types to rustfs-s3-types by codemod (#8439)

T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and
the error items of protocols move off s3s by scripts/codemods, run with
--hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it
downcasts the backend error, whose runtime type is still s3s::S3Error.

- rustfs/src/protocols/client.rs: the two webdav session methods now
  return rustfs_s3_types::S3Result and convert the s3s error with From.
- codemod: a child module that calls s3_error! through `use super::*`
  gets its own `use rustfs_s3_types::s3_error;` once its parent imports
  that macro by name; otherwise it is listed as GLOB.
- s3-types: a table test pins code, status and the s3s error the legacy
  edge renders for every variant a moved call site names.
- s3_error_lines 1585 -> 1584 (a lifecycle comment reworded).

* test(connect): restore one-second drive request budgets

Apply the exact test-only repair already landed on main in 5972a09cc3. The filesystem probe and production CLI success test request one second and keep their existing one-second duration and parameter checks. All outcome, I/O, error, cleanup and resource-limit assertions remain.

Scope: rustfs/tests/connect_perf_drive.rs only. Patch-id: cb0a1dfc8f68a67727525adea7458e2cee7d09f8.

Validation environment: LANG=en_US.UTF-8, CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0, exclusive APFS-cloned CARGO_TARGET_DIR.

Quick Checks: every command from .github/actions/quick-checks/action.yml, with captured output and exit status below.

$ shellcheck --version && actionlint
ShellCheck - shell script analysis tool
version: 0.11.0
license: GNU General Public License, version 3
website: https://www.shellcheck.net
exit_code=0

$ cargo fmt --all --check
exit_code=0

$ ./scripts/check_unsafe_code_allowances.sh
exit_code=0

$ ./scripts/check_layer_dependencies.sh
Layer dependency guard passed (no new reverse dependencies/cycles).
exit_code=0

$ ./scripts/check_architecture_migration_rules.sh
Architecture migration rules passed.
exit_code=0

$ ./scripts/check_logging_guardrails.sh
✅ Logging guardrails check passed
exit_code=0

$ ./scripts/check_error_other_format_ratchet.sh
error(format!) ratchet passed (total call sites: 582).
exit_code=0

$ ./scripts/check_no_tokio_io_uring.sh
exit_code=0

$ ./scripts/check_extension_schema_boundaries.sh
Extension schema boundaries passed.
exit_code=0

$ ./scripts/check_body_cache_whitelist.sh
Body-cache whitelist guard passed.
exit_code=0

$ ./scripts/check_s3s_footprint.sh
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)
✅ s3s footprint baseline check passed
exit_code=0

$ ./scripts/check_test_count_ratchet.sh
test-count ratchet OK: 52 crates, 21222 tests, 164 ignored match .config/test-count-baseline.txt
exit_code=0

$ ./scripts/check_fips_wording.sh
FIPS wording guard passed (README.md and CHANGELOG.md contain no forbidden claims).
KMS wording guard passed (crates/kms claims no Vault KV2 Transit wrapping).
exit_code=0

$ ./scripts/check_embedded_secrets.sh
Embedded secret guard passed (no private key material or provider credential literal in the tree).
exit_code=0

$ make script-tests
Running script tests...
entrypoint credential policy tests passed
internode grpc A/B bench env tests passed
object batch benchmark enhanced tests passed
PASS package service script contract
hotpath warp A/B gate tests passed
hotpath warp ABBA tests passed
Scanner validation artifacts written to /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.1pFe6Kl1Yu/out
Scanner validation artifacts written to /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.1pFe6Kl1Yu/out-missing-pid
.............................
----------------------------------------------------------------------
Ran 29 tests in 181.286s

OK
{"decision":"verified","pending_gates":["G01","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G02"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-E"}
{"decision":"verified","pending_gates":["G01","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G02"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-E"}
{"decision":"verified","pending_gates":["G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G01"}
{"decision":"verified","pending_gates":["G01","G02","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G03"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-L"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G14"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G14"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G10"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P1"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P2"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P3"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G05"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G06"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-D"}
Status-and-outcome release descriptor verified: /private/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp8rwn1pe0/descriptor/release-bundle-status-outcome.json
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G05"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G06"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-D"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G07"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G08"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P4"}
exact 1MiB handoff ABBA tests passed
pinned paired ABBA bench tests passed
==> /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/bin/path_containment (-max_total_time=7, -seed=123456789, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/explicit-artifacts/path_containment/, corpus=/private/tmp/gw-orchestrator-2734/ci-drive/fuzz/corpus/path_containment)
-seed=123456789
target=path_containment
seed=123456789
max_total_time=7
git_revision=4d15b816236630b2c42f321ffc9ed3654edc94e3
git_dirty=true
runner_mode=prebuilt
==> /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/bin/path_containment (-max_total_time=1, -seed=2322617706, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/auto-artifacts/path_containment/, corpus=/private/tmp/gw-orchestrator-2734/ci-drive/fuzz/corpus/path_containment)
-seed=2322617706
==> cargo +nightly fuzz run path_containment (-max_total_time=9, -seed=987654321, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/cargo-artifacts/path_containment/)
+nightly
fuzz
run
path_containment
-seed=987654321
seed=987654321
runner_mode=cargo-fuzz
fuzz runner seed manifest tests passed
✅ scripts/python_bin.sh resolver checks passed
self-test ok: clean fixture with every exemption present passes
self-test ok: all 11 pattern families fire
self-test ok: moving a pinned source is reported
Embedded secret guard self-test passed.
self-test ok: p1-s3s-import: 'use s3s::s3_error;' counts every line, the rustfs_s3_types module's too (s3_error_lines=3)
self-test ok: p2-s3s-mixed-tree: a multi-line s3s use tree naming s3_error counts (s3_error_lines=3)
self-test ok: p3-s3s-glob: a crate-root glob of s3s, bare or inside a tree, counts (s3_error_lines=5)
self-test ok: p4-macro-use: '#[macro_use] extern crate s3s' counts (s3_error_lines=3)
self-test ok: p5-qualified-s3s: a qualified s3s::s3_error! line counts in a cleared file (s3_error_lines=2)
self-test ok: p6-exact-pass: a baseline equal to the tree passes (exit 0)
self-test ok: n1-rustfs-import: 'use rustfs_s3_types::s3_error;' alone counts nothing (s3_error_lines=1)
self-test ok: n2-rustfs-mixed-tree: a rustfs_s3_types tree plus an s3s tree without the macro counts nothing (s3_error_lines=1)
self-test ok: n3-defines: the file defining the macro counts nothing (s3_error_lines=1)
self-test ok: n4-qualified-rustfs: a qualified rustfs_s3_types::s3_error! line counts nothing (s3_error_lines=1)
self-test ok: n5-renamed-call: a call through a renamed macro is not text-matched (out of scope) (s3_error_lines=1)
self-test ok: n6-comment-string: a comment or string mentioning s3_error! in a cleared file counts nothing (s3_error_lines=1)
self-test ok: n7-e2e-excluded: crates/e2e_test is excluded (s3_error_lines=1)
self-test ok: n8-super-glob: an unrecognised import form ('use super::*;') counts (s3_error_lines=3)
self-test ok: n9-facade: a crate-local re-export counts (s3_error_lines=2)
self-test ok: n10-renamed-import: a renamed rustfs_s3_types import does not clear s3_error! lines (s3_error_lines=2)
self-test ok: n11-commented-import: a commented-out rustfs_s3_types import does not clear (s3_error_lines=2)
self-test ok: n12-rustfs-glob: a glob import of rustfs_s3_types does not clear (s3_error_lines=2)
self-test ok: n13-comment-only: a comment mentioning s3_error! in a file with no import counts (s3_error_lines=2)
self-test ok: f1-baseline-missing: a missing baseline file fails (exit 1)
self-test ok: f2-key-missing: a baseline without s3_error_lines fails (exit 1)
self-test ok: f3-rg-fails: a failing classification search fails the guard (exit 1)
✅ s3s footprint self-test passed
test_core_gate_rejects_missing_ignored_filtered_and_corrupt_inputs (__main__.SelfTests.test_core_gate_rejects_missing_ignored_filtered_and_corrupt_inputs) ... ok
test_e2e_requires_registration (__main__.SelfTests.test_e2e_requires_registration) ... ok
test_fuzz_runtime_uses_matrix_target (__main__.SelfTests.test_fuzz_runtime_uses_matrix_target) ... ok
test_ilm_lane_keeps_the_measured_cargo_build_budget (__main__.SelfTests.test_ilm_lane_keeps_the_measured_cargo_build_budget) ... ok
test_profile_listing_binds_platform_digest (__main__.SelfTests.test_profile_listing_binds_platform_digest) ... ok
test_profile_listing_enforces_selection (__main__.SelfTests.test_profile_listing_enforces_selection) ... ok
test_quick_checks_commands_propagate_failure (__main__.SelfTests.test_quick_checks_commands_propagate_failure) ... ok
test_quick_checks_rejects_caller_and_execution_bypasses (__main__.SelfTests.test_quick_checks_rejects_caller_and_execution_bypasses) ... ok
test_runner_readiness_fails_closed (__main__.SelfTests.test_runner_readiness_fails_closed) ... ok
test_s3_runner_rejects_unbounded_failure_locals (__main__.SelfTests.test_s3_runner_rejects_unbounded_failure_locals) ... ok
test_scanner_heal_begin_requires_embedded_source_provenance (__main__.SelfTests.test_scanner_heal_begin_requires_embedded_source_provenance) ... ok
test_scanner_heal_case_does_not_approve_pending_release (__main__.SelfTests.test_scanner_heal_case_does_not_approve_pending_release) ... ok
test_scanner_heal_case_lane_is_required_for_runner_profile (__main__.SelfTests.test_scanner_heal_case_lane_is_required_for_runner_profile) ... ok
test_scanner_heal_case_only_schema_cannot_approve_release (__main__.SelfTests.test_scanner_heal_case_only_schema_cannot_approve_release) ... ok
test_scanner_heal_crash_case_rejects_restart_oracle (__main__.SelfTests.test_scanner_heal_crash_case_rejects_restart_oracle) ... ok
test_scanner_heal_ec84_case_rejects_wrong_erasure_geometry (__main__.SelfTests.test_scanner_heal_ec84_case_rejects_wrong_erasure_geometry) ... ok
test_scanner_heal_finish_collects_oracles_from_registry (__main__.SelfTests.test_scanner_heal_finish_collects_oracles_from_registry) ... ok
test_scanner_heal_multipool_case_accepts_deferred_optional_outage_write (__main__.SelfTests.test_scanner_heal_multipool_case_accepts_deferred_optional_outage_write) ... ok
test_scanner_heal_pending_gate_cannot_map_to_implemented_lane (__main__.SelfTests.test_scanner_heal_pending_gate_cannot_map_to_implemented_lane) ... ok
test_scanner_heal_receipts_reject_reuse_and_missing_builds (__main__.SelfTests.test_scanner_heal_receipts_reject_reuse_and_missing_builds) ... ok
test_scanner_heal_rejects_boolean_fractional_and_out_of_geometry_integers (__main__.SelfTests.test_scanner_heal_rejects_boolean_fractional_and_out_of_geometry_integers) ... ok
test_scanner_heal_rejects_broken_execution_and_artifacts (__main__.SelfTests.test_scanner_heal_rejects_broken_execution_and_artifacts) ... ok
test_scanner_heal_rejects_copied_junit_and_wrong_suite_build (__main__.SelfTests.test_scanner_heal_rejects_copied_junit_and_wrong_suite_build) ... ok
test_scanner_heal_release_bundle_accepts_complete_measured_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_accepts_complete_measured_evidence) ... ok
test_scanner_heal_release_bundle_assembler_copies_measured_artifacts (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_copies_measured_artifacts) ... ok
test_scanner_heal_release_bundle_assembler_merges_measured_descriptors (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_merges_measured_descriptors) ... ok
test_scanner_heal_release_bundle_assembler_rejects_duplicate_descriptor_gate (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_duplicate_descriptor_gate) ... ok
test_scanner_heal_release_bundle_assembler_rejects_fixture_descriptor (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_fixture_descriptor) ... ok
test_scanner_heal_release_bundle_assembler_rejects_hash_mismatch (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_hash_mismatch) ... ok
test_scanner_heal_release_bundle_assembler_rejects_mixed_descriptor_revisions (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_mixed_descriptor_revisions) ... ok
test_scanner_heal_release_bundle_enforces_topology_duration_profile_and_versions (__main__.SelfTests.test_scanner_heal_release_bundle_enforces_topology_duration_profile_and_versions) ... ok
test_scanner_heal_release_bundle_fixture_writer_is_not_release_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_fixture_writer_is_not_release_evidence) ... ok
test_scanner_heal_release_bundle_gate_accepts_single_measured_gate (__main__.SelfTests.test_scanner_heal_release_bundle_gate_accepts_single_measured_gate) ... ok
test_scanner_heal_release_bundle_gate_rejects_missing_field (__main__.SelfTests.test_scanner_heal_release_bundle_gate_rejects_missing_field) ... ok
test_scanner_heal_release_bundle_gate_rejects_unknown_gate (__main__.SelfTests.test_scanner_heal_release_bundle_gate_rejects_unknown_gate) ... ok
test_scanner_heal_release_bundle_rejects_synthetic_or_missing_fields (__main__.SelfTests.test_scanner_heal_release_bundle_rejects_synthetic_or_missing_fields) ... ok
test_scanner_heal_release_bundle_requires_domain_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_requires_domain_evidence) ... ok
test_scanner_heal_release_bundle_requires_field_provenance (__main__.SelfTests.test_scanner_heal_release_bundle_requires_field_provenance) ... ok
test_scanner_heal_release_bundle_requires_json_artifact_provenance (__main__.SelfTests.test_scanner_heal_release_bundle_requires_json_artifact_provenance) ... ok
test_scanner_heal_release_bundle_requires_mixed_version_field_roles (__main__.SelfTests.test_scanner_heal_release_bundle_requires_mixed_version_field_roles) ... ok
test_scanner_heal_release_bundle_requires_same_gate_measurement_window (__main__.SelfTests.test_scanner_heal_release_bundle_requires_same_gate_measurement_window) ... ok
test_scanner_heal_release_consumes_multiple_registry_oracles (__main__.SelfTests.test_scanner_heal_release_consumes_multiple_registry_oracles) ... ok
test_scanner_heal_release_matrix_lanes_remain_pending_without_real_evidence (__main__.SelfTests.test_scanner_heal_release_matrix_lanes_remain_pending_without_real_evidence) ... ok
test_scanner_heal_release_requirements_cannot_be_cleared (__main__.SelfTests.test_scanner_heal_release_requirements_cannot_be_cleared) ... ok
test_scanner_heal_release_status_rejects_focused_case_run (__main__.SelfTests.test_scanner_heal_release_status_rejects_focused_case_run) ... ok
test_scanner_heal_release_status_rejects_synthetic_case (__main__.SelfTests.test_scanner_heal_release_status_rejects_synthetic_case) ... ok
test_scanner_heal_required_evidence_fields_cannot_be_removed (__main__.SelfTests.test_scanner_heal_required_evidence_fields_cannot_be_removed) ... ok
test_scheduled_alerts_require_completion_watchdog (__main__.SelfTests.test_scheduled_alerts_require_completion_watchdog) ... ok
test_update_profile_selection_changes_only_requested_platform (__main__.SelfTests.test_update_profile_selection_changes_only_requested_platform) ... ok
test_validate_includes_quick_checks (__main__.SelfTests.test_validate_includes_quick_checks) ... ok
test_vault_tests_require_cross_process_serialization (__main__.SelfTests.test_vault_tests_require_cross_process_serialization) ... ok
test_workflow_readiness_requires_dependency_probe (__main__.SelfTests.test_workflow_readiness_requires_dependency_probe) ... ok

----------------------------------------------------------------------
Ran 52 tests in 14.301s

OK
e2e-full selection OK: 1 tests, sha256=29e30e6ed2ed38d9e5b0a7cdd9536abf5c719f7021a02b578ae1381563a3c273
...................................
----------------------------------------------------------------------
Ran 35 tests in 1.161s

OK
...................
----------------------------------------------------------------------
Ran 19 tests in 17.397s

OK
..........
----------------------------------------------------------------------
Ran 10 tests in 0.183s

OK
test_actual_selector_bootstrap_uses_base_policy_and_fails_closed (__main__.SelfTests.test_actual_selector_bootstrap_uses_base_policy_and_fails_closed) ... ok
test_documentation_paths_do_not_hide_build_or_fixture_changes (__main__.SelfTests.test_documentation_paths_do_not_hide_build_or_fixture_changes) ... ok
test_e2e_scope_keeps_production_and_shared_configuration_full (__main__.SelfTests.test_e2e_scope_keeps_production_and_shared_configuration_full) ... ok
test_e2e_shortcut_rejects_new_direct_renamed_and_target_dependencies (__main__.SelfTests.test_e2e_shortcut_rejects_new_direct_renamed_and_target_dependencies) ... ok
test_e2e_shortcut_traverses_implicit_path_dependencies (__main__.SelfTests.test_e2e_shortcut_traverses_implicit_path_dependencies) ... ok
test_event_contract_requires_complete_candidate_and_optional_lanes (__main__.SelfTests.test_event_contract_requires_complete_candidate_and_optional_lanes) ... ok
test_every_wrong_result_missing_job_or_selection_fails_closed (__main__.SelfTests.test_every_wrong_result_missing_job_or_selection_fails_closed) ... ok
test_full_e2e_gate_preserves_workflow_branch_and_event_scope (__main__.SelfTests.test_full_e2e_gate_preserves_workflow_branch_and_event_scope) ... ok
test_git_range_includes_deleted_source_and_rename_origins (__main__.SelfTests.test_git_range_includes_deleted_source_and_rename_origins) ... ok
test_job_ids_and_display_names_cannot_hide_validation (__main__.SelfTests.test_job_ids_and_display_names_cannot_hide_validation) ... ok
test_report_only_stack_lane_mirrors_e2e_smoke_outside_the_aggregate (__main__.SelfTests.test_report_only_stack_lane_mirrors_e2e_smoke_outside_the_aggregate) ... ok
test_repository_wiring_and_missing_dependency_regression (__main__.SelfTests.test_repository_wiring_and_missing_dependency_regression) ... ok
test_verify_command_preserves_failures (__main__.SelfTests.test_verify_command_preserves_failures) ... ok

----------------------------------------------------------------------
Ran 13 tests in 3.280s

OK
security coverage self-test passed
test_api_failures_preserve_other_evidence_and_never_enter_grace (__main__.SelfTests.test_api_failures_preserve_other_evidence_and_never_enter_grace) ... ok
test_attempt_outcome_does_not_replace_recent_success (__main__.SelfTests.test_attempt_outcome_does_not_replace_recent_success) ... ok
test_cancelled_attempt_cannot_refresh_expired_success (__main__.SelfTests.test_cancelled_attempt_cannot_refresh_expired_success) ... ok
test_cli_requires_the_repository_default_branch (__main__.SelfTests.test_cli_requires_the_repository_default_branch) ... ok
test_config_rejects_duplicate_and_invalid_entries (__main__.SelfTests.test_config_rejects_duplicate_and_invalid_entries) ... ok
test_freshness_boundaries (__main__.SelfTests.test_freshness_boundaries) ... ok
test_grace_requires_two_successful_queries_with_no_history (__main__.SelfTests.test_grace_requires_two_successful_queries_with_no_history) ... ok
test_invalid_api_evidence_fails_closed (__main__.SelfTests.test_invalid_api_evidence_fails_closed) ... ok
test_report_retains_every_workflow (__main__.SelfTests.test_report_retains_every_workflow) ... ok
test_requests_filter_schedule_default_branch_and_success_on_server (__main__.SelfTests.test_requests_filter_schedule_default_branch_and_success_on_server) ... ok

----------------------------------------------------------------------
Ran 10 tests in 0.051s

OK
..............................
----------------------------------------------------------------------
Ran 30 tests in 22.354s

OK
..........
----------------------------------------------------------------------
Ran 10 tests in 2.969s

OK
.................................
----------------------------------------------------------------------
Ran 33 tests in 0.160s

OK
Online functional runner labels: pf-testing
.......
----------------------------------------------------------------------
Ran 7 tests in 0.002s

OK
.......
----------------------------------------------------------------------
Ran 7 tests in 0.001s

OK
....
----------------------------------------------------------------------
Ran 4 tests in 0.128s

OK
...
----------------------------------------------------------------------
Ran 3 tests in 0.547s

OK
PASS: cold stampede script self-test
exit_code=0

$ python3 ./scripts/test_object_generation_protocol_model.py
test_a_recovery_barrier_cannot_choose_an_empty_value (__main__.ObjectGenerationProtocolModel.test_a_recovery_barrier_cannot_choose_an_empty_value) ... ok
test_all_bounded_two_proposer_interleavings_choose_at_most_one_value (__main__.ObjectGenerationProtocolModel.test_all_bounded_two_proposer_interleavings_choose_at_most_one_value) ... ok
test_an_unavailable_decision_quorum_cannot_choose_a_candidate (__main__.ObjectGenerationProtocolModel.test_an_unavailable_decision_quorum_cannot_choose_a_candidate) ... ok
test_counter_allocation_requires_persistence_and_is_consumed_once (__main__.ObjectGenerationProtocolModel.test_counter_allocation_requires_persistence_and_is_consumed_once) ... ok
test_counter_restart_burns_persisted_allocations_before_or_after_issue (__main__.ObjectGenerationProtocolModel.test_counter_restart_burns_persisted_allocations_before_or_after_issue) ... ok
test_crash_after_promise_before_reply_does_not_restore_an_older_ballot (__main__.ObjectGenerationProtocolModel.test_crash_after_promise_before_reply_does_not_restore_an_older_ballot) ... ok
test_duplicate_or_unconfigured_voters_cannot_form_a_quorum (__main__.ObjectGenerationProtocolModel.test_duplicate_or_unconfigured_voters_cannot_form_a_quorum) ... ok
test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision (__main__.ObjectGenerationProtocolModel.test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision) ... ok
test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected (__main__.ObjectGenerationProtocolModel.test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected) ... ok
test_forgetting_tombstone_authority_allows_an_aba_create (__main__.ObjectGenerationProtocolModel.test_forgetting_tombstone_authority_allows_an_aba_create) ... ok
test_issuing_before_persistence_reuses_a_ballot_after_restart (__main__.ObjectGenerationProtocolModel.test_issuing_before_persistence_reuses_a_ballot_after_restart) ... ok
test_lost_ack_and_restart_preserve_the_chosen_operation (__main__.ObjectGenerationProtocolModel.test_lost_ack_and_restart_preserve_the_chosen_operation) ... ok
test_minority_accepted_work_is_adopted_and_finished (__main__.ObjectGenerationProtocolModel.test_minority_accepted_work_is_adopted_and_finished) ... ok
test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample (__main__.ObjectGenerationProtocolModel.test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample) ... ok
test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it (__main__.ObjectGenerationProtocolModel.test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it) ... ok
test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor (__main__.ObjectGenerationProtocolModel.test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor) ... ok
test_replacing_voters_with_empty_state_can_choose_a_conflicting_value (__main__.ObjectGenerationProtocolModel.test_replacing_voters_with_empty_state_can_choose_a_conflicting_value) ... ok
test_skipping_predecessor_recovery_chooses_a_broken_revision_chain (__main__.ObjectGenerationProtocolModel.test_skipping_predecessor_recovery_chooses_a_broken_revision_chain) ... ok
test_stale_publication_rollback_and_gc_cannot_change_the_winner (__main__.ObjectGenerationProtocolModel.test_stale_publication_rollback_and_gc_cannot_change_the_winner) ... ok
test_superseded_operation_identity_cannot_be_reused_for_different_input (__main__.ObjectGenerationProtocolModel.test_superseded_operation_identity_cannot_be_reused_for_different_input) ... ok
test_superseded_operation_retry_returns_its_original_successor (__main__.ObjectGenerationProtocolModel.test_superseded_operation_retry_returns_its_original_successor) ... ok
test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision (__main__.ObjectGenerationProtocolModel.test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision) ... ok
test_tombstone_rejects_a_delayed_never_present_create (__main__.ObjectGenerationProtocolModel.test_tombstone_rejects_a_delayed_never_present_create) ... ok
test_unlearned_predecessor_is_recovered_before_the_second_slot (__main__.ObjectGenerationProtocolModel.test_unlearned_predecessor_is_recovered_before_the_second_slot) ... ok

----------------------------------------------------------------------
Ran 24 tests in 0.958s

OK
model: states=51023, transitions=167520, terminal=732, chosen_candidates=2
negative control: double-choice trace=((0, 'prepare', 0), (0, 'prepare', 1), (0, 'prepare', 2), (0, 'accept', 0), (0, 'accept', 1), (0, 'accept', 2), (1, 'prepare', 0), (1, 'prepare', 1), (1, 'prepare', 2), (1, 'accept', 0), (1, 'accept', 1), (1, 'accept', 2))
multi-slot round schedules: 640
exit_code=0

$ python3 ./scripts/check_test_wiring.py
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired
exit_code=0

$ ./scripts/check_no_planning_docs.sh
exit_code=0

$ ./scripts/check_uring_lane_lib_only.sh
OK: no 'uring_'-matching test functions under crates/ecstore/tests; --lib is safe
exit_code=0

Initial complete gate
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
 Nextest run ID e07fc5b9-c6c8-4cdd-ae61-54839bf24437 with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.016s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.011s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.009s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.009s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  16.950s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   4.025s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.008s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.007s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.035s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  21.087s] 10 tests run: 10 passed, 0 skipped
exit_code=0

Restored complete gate
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
 Nextest run ID 233b7307-5361-4eb7-94d5-b384f17d2d6a with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.009s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.008s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.008s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.008s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  19.518s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   4.146s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.009s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.018s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.009s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.054s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  23.787s] 10 tests run: 10 passed, 0 skipped
exit_code=0

Mutation M1: success_measurement reports max(actual elapsed, requested budget + 1), forcing 1001 ms for the one-second request. The retained duration assertion at rustfs/tests/connect_perf_drive.rs:105 fails.
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast -E 'test(real_local_write_read_is_measured_and_scratch_is_removed)'
info: experimental features enabled: setup-scripts
   Compiling rustfs v1.0.1 (/private/tmp/gw-orchestrator-2734/ci-drive/rustfs)
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs") generated 1 warning
warning: `rustfs` (bin "rustfs-cli") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 7m 06s
────────────
 Nextest run ID 9f618777-7bf6-4cbc-8803-dd7f50f449ed with nextest profile: ci
    Starting 1 test across 1 binary (9 tests skipped)
        FAIL [   0.016s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
  stdout ───

    running 1 test
    test real_local_write_read_is_measured_and_scratch_is_removed ... FAILED

    failures:

    failures:
        real_local_write_read_is_measured_and_scratch_is_removed

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 0.01s

  stderr ───

    thread 'real_local_write_read_is_measured_and_scratch_is_removed' (106986846) panicked at rustfs/tests/connect_perf_drive.rs:105:5:
    assertion failed: (1..=1_000).contains(&data.duration_millis)
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

────────────
     Summary [   0.017s] 1 test run: 0 passed, 1 failed, 9 skipped
        FAIL [   0.016s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
error: test run failed
exit_code=100

Mutation M2: target_parameters reports requested duration + 1. The in-process assertion at rustfs/tests/connect_perf_drive.rs:111 and CLI JSON assertion at rustfs/tests/connect_perf_drive.rs:527 both reject 1001 against 1000.
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast -E 'test(real_local_write_read_is_measured_and_scratch_is_removed) | test(production_cli_measures_and_signs_exact_binary_provenance)'
info: experimental features enabled: setup-scripts
   Compiling rustfs v1.0.1 (/private/tmp/gw-orchestrator-2734/ci-drive/rustfs)
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs") generated 1 warning
warning: `rustfs` (bin "rustfs-cli") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 7m 20s
────────────
 Nextest run ID b73d46ab-c910-4e1e-8063-7fc7e332ec15 with nextest profile: ci
    Starting 2 tests across 1 binary (8 tests skipped)
        FAIL [  13.968s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
  stdout ───

    running 1 test
    test production_cli_measures_and_signs_exact_binary_provenance ... FAILED

    failures:

    failures:
        production_cli_measures_and_signs_exact_binary_provenance

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 13.96s

  stderr ───

    thread 'production_cli_measures_and_signs_exact_binary_provenance' (107015462) panicked at rustfs/tests/connect_perf_drive.rs:527:5:
    assertion `left == right` failed
      left: Number(1001)
     right: 1000
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

        FAIL [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
  stdout ───

    running 1 test
    test real_local_write_read_is_measured_and_scratch_is_removed ... FAILED

    failures:

    failures:
        real_local_write_read_is_measured_and_scratch_is_removed

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 0.01s

  stderr ───

    thread 'real_local_write_read_is_measured_and_scratch_is_removed' (107016667) panicked at rustfs/tests/connect_perf_drive.rs:111:5:
    assertion `left == right` failed
      left: 1001
     right: 1000
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

────────────
     Summary [  13.985s] 2 tests run: 0 passed, 2 failed, 8 skipped
        FAIL [  13.968s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        FAIL [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
error: test run failed
exit_code=100

Transient production mutations were restored byte-for-byte before the complete green rerun. Original/restored SHA256: 8d389e6d8836c041aab8c6a93e56addd65ba3f70b33264e1c74f639d0572a2da.

$ git diff --check

exit_code=0

$ git diff --cached --check

exit_code=0

Independent orchestrator acceptance
Tested code tree: e67c72fb01

Tested tree: 16fbbd2ea4
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

$ LANG=en_US.UTF-8 CARGO_BUILD_JOBS=1 CARGO_INCREMENTAL=0 cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
info: experimental features enabled: setup-scripts
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs-cli") generated 1 warning
warning: `rustfs` (bin "rustfs") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.69s
────────────
 Nextest run ID cd1edf4c-f1c0-456d-ad1a-76cc00ce05c9 with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.012s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.010s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.010s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.009s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  20.328s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   5.527s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.013s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.019s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.011s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.049s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  25.991s] 10 tests run: 10 passed, 0 skipped

exit_code=0

All 54 tracked Cargo.toml files parse without duplicate keys. git diff --check passed. The production source remains unchanged; the two deliberate implementation mutations above were restored before this complete green run.

* fix(test): compare legacy body wire parity as raw bytes

Lossy UTF-8 comparison concealed a changed binary response byte.
Compare the already captured Vec<u8> values directly.

Validation:
- Binary mutant: migrated payload 0x84 vs legacy 0x83; old comparator
  passed 1/1, raw comparator failed 0/1 (exit 100); restored green.
- Non-Swift body-drain mutant: untouched-body delegation test failed
  with empty vs payload (exit 100); restored handler without a diff.
- cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols
  --features rustfs-protocols/swift --no-fail-fast:
  543 tests run, 543 passed, 0 skipped after both mutants restored.
- cargo fmt --all --check and git diff --check passed.

Orchestrator acceptance (LANG=en_US.UTF-8):
cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
Summary [2.445s] 543 tests run: 543 passed, 0 skipped.

All shared quick checks on the final code tree:
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

Final independent acceptance after the drive-test CI repair
Parent: a99d8f4f89
Tested code tree: 93cbd96bf0

Tested tree: 075ae5161c2b62a96812bc1e3c904a7663b4f2f8
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

$ LANG=en_US.UTF-8 CARGO_BUILD_JOBS=1 CARGO_INCREMENTAL=0 cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
    Finished `test` profile [unoptimized + debuginfo] target(s) in 10m 08s
 Nextest run ID 5c7ff102-ec6d-479a-86e0-810859b28bbf with nextest profile: ci
    Starting 543 tests across 8 binaries
     Summary [   6.729s] 543 tests run: 543 passed, 0 skipped
exit_code=0

The only source change is the raw-byte wire comparator. Cargo manifests and the ratchet baselines remain unchanged from the independently validated parent. The invalid-byte and consumed-request mutations remain recorded above; their implementation changes were restored.

* fix: repair integration checks and IPC test readiness

Wait for the RPC/locks test sockets to pass the existing owner and mode
checks inside the original 5s readiness window and 10ms poll interval.

Use the existing admin S3 carrier for the new recovery and quota error
calls. Regenerate the test-count baseline for main's 118 existing tests
across seven crates. Pair the private pagination helper's raw-scan inputs
and remove the redundant Copy clone from the retirement event.

Keep the original assertions, responses, authorization, locks and storage
operations. The pagination body and raw-scan expressions at all five calls
are byte-identical; reinserting only the Copy clone restores the entire
base heal file. All ignored baseline values and s3s counts stay unchanged.

Validation base: c47432e353
Validation tree: f081084bc5
Patch SHA-256: 2dcf0028691bfb2772edfb38e262829b04f4151832dcce823c9dd9f7434b52de
Trace production-prefix SHA-256: fd994f218a21429d62866f4efb8eee1ee57b58f4a51865aa35d05a54ef1ccb45

All commands used LANG=en_US.UTF-8, CARGO_BUILD_JOBS=1,
CARGO_INCREMENTAL=0 and CARGO_NET_OFFLINE=true in one exclusive target.
Every gate retained the same tree and six source hashes before and after.
54 Cargo.toml files parse without duplicate keys.

cargo clippy -p rustfs --all-targets -- -D warnings
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 12m 50s
Exit: 0. Wall time: 770.855s.

The original 19 quick-check steps:
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
Wall time: 286.337s.

cargo nextest run --profile ci -p rustfs --no-fail-fast
     Summary [ 217.017s] 5382 tests run: 5382 passed (4 leaky), 40 skipped
Exit: 0. Wall time: 1324.414s.

cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
     Summary [   2.189s] 543 tests run: 543 passed, 0 skipped
Exit: 0. Wall time: 123.192s.

cargo nextest run --profile ci -p rustfs-ecstore list_objects_paginate --no-fail-fast
     Summary [   0.170s] 8 tests run: 8 passed, 5921 skipped
Exit: 0. Wall time: 261.324s.

cargo nextest run --profile ci -p rustfs-heal durable_mrf_anchor_retires_only_after_exact_bucket_generation_is_deleted --no-fail-fast
     Summary [   0.805s] 1 test run: 1 passed, 835 skipped
Exit: 0. Wall time: 199.801s.

Baseline regeneration:
./scripts/check_test_count_ratchet.sh --update
wrote .config/test-count-baseline.txt: 52 crates, 21340 tests, 164 ignored
Test-only increases: ecstore48, heal3, iam14, io-metrics2, obs3, policy14,
rustfs34. Existing ignored values and all other rows are byte-identical.
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)

Historical runtime red controls on f622 (same retained two assertions):
cargo nextest run --profile ci -p rustfs --lib -E 'test(=connect::diagnostics::trace_runtime::tests::local_top_rpc_reads_service_http_completions_and_signs_offline) | test(=connect::diagnostics::trace_runtime::tests::local_top_locks_reads_service_manager_and_signs_offline)' --no-fail-fast
Original path readiness with a temporary 500ms bind/seal gap:
     Summary [   0.863s] 2 tests run: 0 passed, 2 failed, 5113 skipped
Exit: 100.
Sealed readiness in that same gap:
     Summary [   5.621s] 2 tests run: 2 passed, 5113 skipped
Exit: 0.
Compiled M1: replace the helper predicate with path.exists():
     Summary [   0.870s] 2 tests run: 0 passed, 2 failed, 5113 skipped
Exit: 100.
Restore the sealed predicate byte-for-byte:
     Summary [   5.640s] 2 tests run: 2 passed, 5113 skipped
Exit: 0.
Both reds observe mode755 with owner_matches=true and fail at the original
RPC/locks unwraps with StateSecurity. M1 failure locations: trace_runtime.rs
2547:29 and 2713:29. No new assertions were introduced.

cargo nextest run --profile ci -p rustfs --lib -E 'test(connect::diagnostics::trace_runtime::tests::)' --no-fail-fast
     Summary [  20.748s] 28 tests run: 28 passed (1 leaky), 5087 skipped
Exit: 0. This security sweep is historical; the full final unit gate above
covers the actual c474 candidate.
The first disk-stopped compile ran no tests and is excluded from red credit.
All temporary seal instrumentation is removed from the final source.

Final unit cleanup markers: 4 leaky, 40 original skips (exit0). The ld
__eh_frame linker warning also appears verbatim in the original f622 run
and the earlier c474 run; there are no new source warnings.

* fix: remove redundant pending closure in disk test

Pass the capture-free pending future factory as a function item in the existing slow-operation test. This fixes the redundant_closure error at disk_store.rs:2914 that stopped strict Clippy before the feature tests could run.

The single-line test edit preserves the factory's result, warning and timeout behavior. Reversing the replacement reproduces the parent file byte for byte. All 220 assertion lines and the production prefix are unchanged; no tests, assertions, guards, profiles or baselines were added or relaxed.

Original Linux CI evidence (run 37888552268 at 53245be9f):
- RIO job113685483992: strict Clippy exit101, redundant closure at disk_store.rs:2914; feature tests skipped.
- Workspace Test and Lint job113685484058: same strict Clippy exit101; nextest and doc tests skipped.
These are lint failures, not runtime mutation kills or known flakes.

Native macOS arm64 validation on tree7a5bf47e7533b29479bc85207416102f1337c566, LANG=en_US.UTF-8, one build job, CARGO_INCREMENTAL=0 and offline dependencies:

shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
All 19 checks: exit0, 312.161s.

$ cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 20m 11s
exit0, 1213.210s total.

$ cargo nextest run --profile ci -p rustfs-ecstore --features rio-v2 slow_disk_operation_warns_once_while_pending_and_preserves_result --no-fail-fast
Finished `test` profile [unoptimized + debuginfo] target(s) in 4m 34s
PASS [0.026s] rustfs-ecstore disk::disk_store::tests::slow_disk_operation_warns_once_while_pending_and_preserves_result
Summary [   0.170s] 1 test run: 1 passed, 5943 skipped
exit0, 281.920s total; 5943 existing filter skips.

warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
This linker warning is retained; flags, profiles and allowances are unchanged.

54 tracked Cargo.toml files parsed without duplicate keys. git diff --check passed. No new assertions were introduced, so no new runtime mutation is claimed. Native compilation and the existing helper test do not substitute for actual Linux RIO execution; the new integration CI run still needs to recover before the RPC merge.

Preservation and output hashes:
parent file SHA256: 8e59ee745e3aaaa22848b943d3b10f6353b515956282cde1f3af2c394200c9ac
final file SHA256: b69ba3237061b161448ae8b62b615885d5770b98a5b4a2479bcfbd70f1bd1f17
unchanged production prefix SHA256: 7903a6e288252c420231242d7b58a25fa2a7140ea3dd430cbe7f0dde11369dcb
RIO Linux failure log SHA256:421addb9868fd88e4109a922311e54f91f0dfd70862f8ba7fcaba193482ed90b
Workspace Linux failure log SHA256:e0aea8d0f014c022252a9534968e083c98cfc1fd760da445ecd04ec3f522e26a
19 output SHA256:35a54f28d707956ed1edb29d492c4dfe8a9fe753d103b111b5a2f1b5c41c2b98
RIO Clippy output SHA256:d6046e3868f2b3992fe11cdd04e9e98eb50a87d1234d9a70dbfeb154ff61638d
slow-operation output SHA256:605b7b258d2fd09c6ef2ab2ffc90d433d0509480f1e45125d20c823759e83608

Refs rustfs/backlog#2734.

* test(ecstore): isolate tier-free recovery fixture startup

The fixture constructor can take the production recovery lock and wait for 60 seconds before this test installs its queue and seeds metadata. Disable its background recovery only while creating this fixture, then restore the environment before calling the existing production recovery entrypoint.

Only the existing test setup changes. Its production entrypoint, default lock parameters, original assertions, 30-second timeout, cancellation and cleanup remain intact. No tests or assertions were added. Production and store/init.rs are byte-identical to the parent.

Independent acceptance ran on merge tree 8ec358cb42ba9ebb3bcfeafcef048a3f0366355b; the runner header reports parent HEAD, while before/after index and source hashes confirm this candidate. Worker task branch d8055766b8 has the same tree.

Every command used LANG=en_US.UTF-8. Task compilation also used CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0, CARGO_NET_OFFLINE=true, an independent target, and empty RUSTC_WRAPPER/RUSTC_WORKSPACE_WRAPPER to keep compilers in its owned process group without changing repository configuration.

Command:
shellcheck --version && actionlint
Exit 0; observed output excerpt:
version: 0.11.0
Log SHA-256: fc9eec16b4191d058a295813ca9afbe9d2b37b59975da860170726be162a3924

Command:
cargo fmt --all --check
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_unsafe_code_allowances.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_layer_dependencies.sh
Exit 0; observed output excerpt:
Layer dependency guard passed (no new reverse dependencies/cycles).
Log SHA-256: 69c6cb1896c0586b28d1167380d3260d99d8017a4265c8df5a2348f289e93a12

Command:
./scripts/check_architecture_migration_rules.sh
Exit 0; observed output excerpt:
Architecture migration rules passed.
Log SHA-256: 28790e70d070607ed4de0704f292a65030be92ec224ef106d4b72fd8296f896e

Command:
./scripts/check_logging_guardrails.sh
Exit 0; observed output excerpt:
✅ Logging guardrails check passed
Log SHA-256: 7adf41abff8ed8f7617decf86498888ddde78937099dd9ce47f6d359dfc2d317

Command:
./scripts/check_error_other_format_ratchet.sh
Exit 0; observed output excerpt:
error(format!) ratchet passed (total call sites: 579).
Log SHA-256: 66172aef18f150ba0605af122e26ffcf91f6dd86f43e38b05e2e982ce8d783ee

Command:
./scripts/check_no_tokio_io_uring.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_extension_schema_boundaries.sh
Exit 0; observed output excerpt:
Extension schema boundaries passed.
Log SHA-256: f13d0acd0e2ce5d9815241b12b4dab7d01cc11b9b65bcf6dfb7812c878532a51

Command:
./scripts/check_body_cache_whitelist.sh
Exit 0; observed output excerpt:
Body-cache whitelist guard passed.
Log SHA-256: 6b828847b0383950ee6eef9cd846d0c322c945e47efd4c7d1ad881178810d6bb

Command:
./scripts/check_s3s_footprint.sh
Exit 0; observed output excerpt:
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)
✅ s3s footprint baseline check passed
Log SHA-256: 47f53d04ef645618563fe1f22abc64667f19078f1974feedcca46e4f2c016844

Command:
./scripts/check_test_count_ratchet.sh
Exit 0; observed output excerpt:
test-count ratchet OK: 52 crates, 21340 tests, 164 ignored match .config/test-count-baseline.txt
Log SHA-256: 618a5e330e933de0c6ff484546df401457024a88bb5986d11d588794e33dab60

Command:
./scripts/check_fips_wording.sh
Exit 0; observed output excerpt:
FIPS wording guard passed (README.md and CHANGELOG.md contain no forbidden claims).
KMS wording guard passed (crates/kms claims no Vault KV2 Transit wrapping).
Log SHA-256: 99221cc9979c480ec4f46f50d76c52b18c72fe499f12e15e4f4a22a1b8bfb140

Command:
./scripts/check_embedded_secrets.sh
Exit 0; observed output excerpt:
Embedded secret guard passed (no private key material or provider credential literal in the tree).
Log SHA-256: 437a3ceba32094d2901889965ad191e411aad32cdb18bf0b8659e818011c406b

Command:
make script-tests
Exit 0; observed output excerpt:
test_cli_requires_the_repository_default_branch (__main__.SelfTests.test_cli_requires_the_repository_default_branch) ... ok
test_config_rejects_duplicate_and_invalid_entries (__main__.SelfTests.test_config_rejects_duplicate_and_invalid_entries) ... ok
test_freshness_boundaries (__main__.SelfTests.test_freshness_boundaries) ... ok
test_grace_requires_two_successful_queries_with_no_history (__main__.SelfTests.test_grace_requires_two_successful_queries_with_no_history) ... ok
test_invalid_api_evidence_fails_closed (__main__.SelfTests.test_invalid_api_evidence_fails_closed) ... ok
test_report_retains_every_workflow (__main__.SelfTests.test_report_retains_every_workflow) ... ok
test_requests_filter_schedule_default_branch_and_success_on_server (__main__.SelfTests.test_requests_filter_schedule_default_branch_and_success_on_server) ... ok
Ran 10 tests in 0.042s
OK
Ran 30 tests in 20.627s
OK
Ran 10 tests in 2.268s
OK
Ran 33 tests in 0.127s
OK
Ran 7 tests in 0.002s
OK
Ran 7 tests in 0.001s
OK
Ran 4 tests in 0.108s
OK
Ran 3 tests in 0.564s
OK
PASS: cold stampede script self-test
Log SHA-256: 75fda2d069a373c63667b908baaf99e6591aab9099a739b0c4c22ceb6da8d68c

Command:
python3 ./scripts/test_object_generation_protocol_model.py
Exit 0; observed output excerpt:
test_an_unavailable_decision_quorum_cannot_choose_a_candidate (__main__.ObjectGenerationProtocolModel.test_an_unavailable_decision_quorum_cannot_choose_a_candidate) ... ok
test_counter_allocation_requires_persistence_and_is_consumed_once (__main__.ObjectGenerationProtocolModel.test_counter_allocation_requires_persistence_and_is_consumed_once) ... ok
test_counter_restart_burns_persisted_allocations_before_or_after_issue (__main__.ObjectGenerationProtocolModel.test_counter_restart_burns_persisted_allocations_before_or_after_issue) ... ok
test_crash_after_promise_before_reply_does_not_restore_an_older_ballot (__main__.ObjectGenerationProtocolModel.test_crash_after_promise_before_reply_does_not_restore_an_older_ballot) ... ok
test_duplicate_or_unconfigured_voters_cannot_form_a_quorum (__main__.ObjectGenerationProtocolModel.test_duplicate_or_unconfigured_voters_cannot_form_a_quorum) ... ok
test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision (__main__.ObjectGenerationProtocolModel.test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision) ... ok
test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected (__main__.ObjectGenerationProtocolModel.test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected) ... ok
test_forgetting_tombstone_authority_allows_an_aba_create (__main__.ObjectGenerationProtocolModel.test_forgetting_tombstone_authority_allows_an_aba_create) ... ok
test_issuing_before_persistence_reuses_a_ballot_after_restart (__main__.ObjectGenerationProtocolModel.test_issuing_before_persistence_reuses_a_ballot_after_restart) ... ok
test_lost_ack_and_restart_preserve_the_chosen_operation (__main__.ObjectGenerationProtocolModel.test_lost_ack_and_restart_preserve_the_chosen_operation) ... ok
test_minority_accepted_work_is_adopted_and_finished (__main__.ObjectGenerationProtocolModel.test_minority_accepted_work_is_adopted_and_finished) ... ok
test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample (__main__.ObjectGenerationProtocolModel.test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample) ... ok
test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it (__main__.ObjectGenerationProtocolModel.test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it) ... ok
test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor (__main__.ObjectGenerationProtocolModel.test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor) ... ok
test_replacing_voters_with_empty_state_can_choose_a_conflicting_value (__main__.ObjectGenerationProtocolModel.test_replacing_voters_with_empty_state_can_choose_a_conflicting_value) ... ok
test_skipping_predecessor_recovery_chooses_a_broken_revision_chain (__main__.ObjectGenerationProtocolModel.test_skipping_predecessor_recovery_chooses_a_broken_revision_chain) ... ok
test_stale_publication_rollback_and_gc_cannot_change_the_winner (__main__.ObjectGenerationProtocolModel.test_stale_publication_rollback_and_gc_cannot_change_the_winner) ... ok
test_superseded_operation_identity_cannot_be_reused_for_different_input (__main__.ObjectGenerationProtocolModel.test_superseded_operation_identity_cannot_be_reused_for_different_input) ... ok
test_superseded_operation_retry_returns_its_original_successor (__main__.ObjectGenerationProtocolModel.test_superseded_operation_retry_returns_its_original_successor) ... ok
test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision (__main__.ObjectGenerationProtocolModel.test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision) ... ok
test_tombstone_rejects_a_delayed_never_present_create (__main__.ObjectGenerationProtocolModel.test_tombstone_rejects_a_delayed_never_present_create) ... ok
test_unlearned_predecessor_is_recovered_before_the_second_slot (__main__.ObjectGenerationProtocolModel.test_unlearned_predecessor_is_recovered_before_the_second_slot) ... ok
Ran 24 tests in 0.929s
OK
Log SHA-256: 0b4be926efb0609d497ae58930340153f01957b606206cbbdb3e0d47eb29d3a2

Command:
python3 ./scripts/check_test_wiring.py
Exit 0; observed output excerpt:
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired
Log SHA-256: 21f676f458451525535ab0ef73d6a9c0d9774e16d76a4d19e396ad34936dd39c

Command:
./scripts/check_no_planning_docs.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_uring_lane_lib_only.sh
Exit 0; observed output excerpt:
OK: no 'uring_'-matching test functions under crates/ecstore/tests; --lib is safe
Log SHA-256: 43f3f851e27ec2ab9e34d6b05fefdede38af992699c473904bfe6f53a46d52e3

Command:
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast
Exit 0; wall 234.399s.
Observed output:
    Finished `test` profile [unoptimized + debuginfo] target(s) in 3m 49s
    Starting 1 test across 1 binary (5871 tests skipped)
  SETUP PASS [   0.006s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"'
        PASS [   0.440s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item
     Summary [   0.586s] 1 test run: 1 passed, 5871 skipped
Log SHA-256: 34e973f86e8dc2102c1a0e483e0ac802ce7df3d1b40a603932f62c0805c4de27

Command:
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over) | test(=store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence) | test(=store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable)' --no-fail-fast
Exit 0; wall 4.010s.
Observed output:
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.43s
    Starting 3 tests across 1 binary (5869 tests skipped)
  SETUP PASS [   0.005s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"'
        PASS [   0.068s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over
        PASS [   0.637s] rustfs-ecstore store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence
        PASS [   0.846s] rustfs-ecstore store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable
     Summary [   0.987s] 3 tests run: 3 passed, 5869 skipped
Log SHA-256: b5da2a927ff18ca9aa217762f7186a6d4371bd9e8feedd7b77940e10b64dffac

Command:
cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
Exit 0; wall 1084.508s.
Observed output:
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 18m 03s
Log SHA-256: 28f98be7970faaa71cd118da1da4e6e2e3d23b9dda7e6449eaa7a37e741260b5

Compiled mutation command (unchanged for each red and restored run):
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast

M1: reverse only this test-local constructor flag from false to true. The mutant compiled, then failed the original recovered-item queue assertion after 30.481s. Restoring the entire observed candidate passed in 0.460s. Red log 55bf3b1535529ac0bcfb8071f9b56f9f40631007a85c315367e7e93786d65293; restored log 204d5fbe4574a7917f5e740a795d864fe28a7e7fb5e33d27b0a9cf68e2f17f52.

M2: replace only the actual awaited enqueue with false, consuming its original inputs. The mutant compiled, then failed the same original queue assertion after 30.516s. Restoring the entire observed candidate passed in 0.457s. Red log 89186330413215e231e2b06bf013cc31097ea8f4a106d91421a36d6c8986a7fd; restored log e848b42d2302401afa5dbcd9e37bf76d2e56a96ab6de5a488513b997f53c5e9b.

All temporary observation and mutation code was removed before final acceptance. Historical baseline and diagnostic failures remain recorded; they are not mutation credit. The separate Object Lock intent-retry diagnostic still fails, and these native macOS checks do not establish Linux CI recovery.

* refactor(rpc): serve internode rpc bodies with the owned Body (#8441)

* test(rpc): pin the wire bytes of every internode rpc response body

rustfs/backlog#2747 batch 4, step 1 of 2, on the s3s bodies the rpc paths
serve today. Each response body builder in storage/rpc/http_service.rs is
served through hyper's HTTP/1.1 server and the raw bytes the peer reads are
compared with goldens captured from this commit: status, content-length or
chunked framing, chunk boundaries, the terminating chunk, HEAD, and the cut
connection a failure after data produces. The capability responses carry a
per-process proof, so their length header and payload are checked against
the body they encode instead.

The read_file_stream response builder moves into read_file_stream_response
so the test reaches the same code the handler runs; behaviour is unchanged.

* refactor(rpc): serve internode rpc bodies with the owned Body

rustfs/backlog#2747 batch 4, step 2 of 2. The internode rpc HTTP paths
build their responses with rustfs_s3_types::Body: Body::from_stream
replaces Body::from(StreamingBlob::wrap(..)) at the six stream sites, and
the in-memory and empty bodies keep their constructors. InternodeRpcService
converts its responses into the wrapped service's response body through
From<Body>, so the server keeps composing it with the s3s service as
before. The wire-byte goldens from the previous commit pass unchanged;
storage/rpc no longer names s3s.

* fix(rpc): simplify wire test response builder type

Name the boxed response builder used by the capability wire test so
all-target Clippy can check it without a type-complexity warning.

Validation (LANG=en_US.UTF-8, CARGO_INCREMENTAL=0):
- cargo clippy -p rustfs --all-targets -- -D warnings: exit 0.
- cargo nextest run --profile ci -p rustfs --lib -E 'test(/storage::rpc::/)': 279 tests run, 279 passed, 4842 skipped.
- All 19 quick-check commands, including script-tests, both Python scripts and cargo fmt --all --check: exit 0.
- Footprint baseline regenerated: files=194, s3_error_lines=1584, ecstore_files=36; test-count baseline regenerated: 52 crates, 21228 tests, 164 ignored.
- All 54 Cargo.toml files parse without duplicate keys.

No assertion was added or changed by this lint fix. R1-R8 are retained
in the existing PR verification record.

* fix(ecstore): resume fenced Object Lock creation intents

* fix(ecstore): bound creation retry futures on the stack

* fix(ecstore): require native proof for creation retries

* fix(ecstore): bound bucket creation stack usage (#8466)

* fix(ecstore): retain creation fences through rollback

Keep lifecycle, metadata-transaction, and namespace guards around rollback and metadata cleanup. Preserve the original publication error when a guard is lost. Build the timestamp retry fixture through the real interrupted producer so it checks a valid pending creation.

Validated source tree: 4e8e207999.
Two independent source reviews found no blocking regression.

Validation (LANG=en_US.UTF-8):
- bash -c 'shellcheck --version && actionlint' — exit 0
- cargo fmt --all --check — exit 0
- ./scripts/check_unsafe_code_allowances.sh — exit 0
- ./scripts/check_layer_dependencies.sh — exit 0
- ./scripts/check_architecture_migration_rules.sh — exit 0
- ./scripts/check_logging_guardrails.sh — exit 0
- ./scripts/check_error_other_format_ratchet.sh — exit 0
- ./scripts/check_no_tokio_io_uring.sh — exit 0
- ./scripts/check_extension_schema_boundaries.sh — exit 0
- ./scripts/check_body_cache_whitelist.sh — exit 0
- ./scripts/check_s3s_footprint.sh — exit 0
- ./scripts/check_test_count_ratchet.sh — exit 0
- ./scripts/check_fips_wording.sh — exit 0
- ./scripts/check_embedded_secrets.sh — exit 0
- make script-tests — exit 0
- python3 ./scripts/test_object_generation_protocol_model.py — exit 0
- python3 ./scripts/check_test_wiring.py — exit 0
- ./scripts/check_no_planning_docs.sh — exit 0
- ./scripts/check_uring_lane_lib_only.sh — exit 0
- cargo nextest run --profile ci -p rustfs-ecstore --lib --locked --no-fail-fast — 5894 passed, 0 skipped
- cargo check -p rustfs-ecstore --all-targets --locked — exit 0
- cargo nextest run --profile ci -p rustfs --lib --locked --no-fail-fast -E 'test(=admin::handlers::table_catalog::tests::cold_strong_registration_completes_while_other_bucket_backup_waits) | test(=admin::handlers::table_catalog::tests::commit_backend_borrowed_put_preserves_owner_authorization_and_precondition) | test(=admin::handlers::table_catalog::tests::disable_table_bucket_preserves_objects_and_can_be_reenabled) | test(=admin::handlers::table_catalog::tests::load_table_handler_negotiates_vended_credentials_without_breaking_metadata_only_callers) | test(=admin::handlers::table_catalog::tests::namespace_property_handler_updates_object_backed_catalog_and_maps_errors) | test(=app::object::get::tests::execute_get_object_resumes_from_relocated_pool_without_splicing_body) | test(=table_catalog::store::tests::catalog_writer_loss_prevents_recovery_storage_commit)' — 7 passed
- cargo nextest run --profile ci -p rustfs --lib --locked --no-fail-fast -E 'test(unsupported_remote_disk_)' — 2 passed

Additional fault-injection evidence remains incomplete. Timestamp M1 reached its original assertion; its restore run is being collected. M2–M5 and the three planned rollback faults have not run. An automated risk check interrupted the timestamp proof task; no assertions or validation settings were weakened.

* fix(ecstore): box nested bucket rollback state

Keep the guarded rollback future behind a heap allocation so bucket creation does not store its nested state inline. The rollback body, all three fence references and pending-proof revocation order are unchanged.

The same three original Linux tests overflowed on both e611 CI attempts. Local format, ecstore test compilation and all 19 quick-check steps pass; the original Linux feature lanes still need to validate this candidate.

* test(ecstore): isolate rollback guard loss fixtures

Run the three rollback metadata-cleanup cases in separate test processes so reused /proc/self/fd aliases from earlier cases cannot invalidate their creation witnesses. Keep the original fixture, guards, assertions and io_uring test selection, and verify that each child runs the intended case successfully.

Update the generated test-count baseline for the added macro arm.

Validation: cargo check -p rustfs-ecstore --tests; cargo fmt --all --check; all 19 quick-check commands, including make script-tests and the two Python checks. Linux io_uring acceptance and fault controls remain pending.

* refactor(replication): move the replication config onto gateway types (#8443)

* wip: T1.4 replication family

* refactor(replication): move the replication config onto gateway types

The replication configuration family leaves s3s: rustfs-replication no longer depends on s3s, and ECStore bucket metadata, the scanner and the rustfs app/admin layers hold rustfs_gateway_types::persistence::PersistedReplicationConfiguration. The persisted XML goes through the gateway persistence codec behind a new BucketConfigXml trait (the 39 byte goldens pass with unchanged fixtures); the legacy s3s edge converts once in storage/ecfs.rs through storage/s3_compat/replication.rs; the scanner data usage cache keeps its MessagePack bytes through a serde mirror pinned by bytes recorded from s3s 5761ddfe. rustfs-gateway-types 0.32.0 is pinned once (rev 772d3385). Refs rustfs/backlog#2745.

* test: use S3 serializer for list response tests

* fix(ci): restore integration test compilation

Gate the diagnostic test import with its feature, initialize the conditional
MetadataWriteStore fields explicitly, and decode the scanner replication fixture
through its existing cache boundary. Preserve the original fixture and assertions.

Validation was run on tree 544acb80c0 at 6003d4b8.
LANG=en_US.UTF-8; no source changes follow validation.

Quick checks: 19/19 passed.
- shellcheck --version && actionlint: exit 0
- cargo fmt --all --check: exit 0
- ./scripts/check_unsafe_code_allowances.sh: exit 0
- ./scripts/check_layer_dependencies.sh: exit 0
- ./scripts/check_architecture_migration_rules.sh: exit 0
- ./scripts/check_logging_guardrails.sh: exit 0
- ./scripts/check_error_other_format_ratchet.sh: exit 0
- ./scripts/check_no_tokio_io_uring.sh: exit 0
- ./scripts/check_extension_schema_boundaries.sh: exit 0
- ./scripts/check_body_cache_whitelist.sh: exit 0
- ./scripts/check_s3s_footprint.sh: exit 0
- ./scripts/check_test_count_ratchet.sh: exit 0
- ./scripts/check_fips_wording.sh: exit 0
- ./scripts/check_embedded_secrets.sh: exit 0
- make script-tests: exit 0
- python3 ./scripts/test_object_generation_protocol_model.py: exit 0
- python3 ./scripts/check_test_wiring.py: exit 0
- ./scripts/check_no_planning_docs.sh: exit 0
- ./scripts/check_uring_lane_lib_only.sh: exit 0

Darwin validation:
- cargo clippy --locked -p rustfs-ecstore -p rustfs-scanner --all-targets -- -D warnings: exit 0
- cargo nextest run --locked -p rustfs-ecstore -p rustfs-scanner --lib -E 'test(config::com::tests::save_config_preserves_retryable_system_volume_errors) | test(scanner_io::tests::replication_maintenance_refreshes_compacted_stats_across_cycles_and_retries)' --test-threads 1 --no-fail-fast: exit 0, 2/2 passed
- cargo clippy --locked -p rustfs-ecstore -p rustfs-scanner --all-targets --features rustfs-ecstore/diagnostic-read-version-caller,rustfs-scanner/diagnostic-read-version-caller -- -D warnings: exit 101
- cargo nextest run --locked -p rustfs-ecstore -p rustfs-scanner --lib --features rustfs-ecstore/diagnostic-read-version-caller,rustfs-scanner/diagnostic-read-version-caller -E 'test(config::com::tests::save_config_preserves_retryable_system_volume_errors) | test(scanner_io::tests::replication_maintenance_refreshes_compacted_stats_across_cycles_and_retries) | test(cluster::rpc::runtime_sources::tests::) | test(config::com::tests::diagnostic_config_write_retains_origin_and_api_for_results_and_cancellation)' --test-threads 1 --no-fail-fast: exit 0, 14/14 passed

Diagnostic strict Clippy remains red on four unchanged production lints:
runtime_sources.rs if_same_then_else and three redundant_locals reports in
set_disk/core/metadata_read.rs and set_disk/read.rs. Their source ranges and
feature wiring match the base; the base was not compiled separately.

Linux aarch64 validation (Rust/Clippy 1.99.0, UID 501, jobs=1):
- cargo clippy --all-targets -- -D warnings: exit 0; Finished dev profile in 68m 12s
- cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings: exit 0; Finished dev profile in 49m 45s
Source HEAD/tree and clean status matched before and after both commands.
No compiler or stack overrides were used. protoc 35.1 was installed from its
verified ARM64 release; flatc was absent and not invoked by the workspace build scripts.
The local environment differs from hosted CI.

* fix(ecstore): recover delete batches with missing versions

Treat an explicit non-null version as a no-op when it is absent from both recovery sources. Keep rejecting missing null versions, delete-marker mismatches and conflicting local generations.

Add returning-disk and partial-receipt regressions, extend journal boundary checks, and regenerate the ecstore test-count baseline. Production faults exercise the 24 added assertion sites and 7 semantic checks: 33 faults failed at the expected sites and passed after full restoration.

Validation (LANG=en_US.UTF-8, one build job):

cargo nextest run --locked -p rustfs-ecstore --lib -E 'test(delete_recovery) | test(quorum_delete_cleanup_coordinator_binds_single_and_batch) | test(crash_delete_retry_batch_reclaims_inline_and_reserved_external)' --test-threads 1 --no-fail-fast
Summary [  34.580s] 69 tests run: 69 passed, 6058 skipped; exit 0

cargo clippy --locked -p rustfs-ecstore --all-targets -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 5m 37s; exit 0

All 19 final Quick Checks completed with exit 0:

- shellcheck --version && actionlint
- cargo fmt --all --check
- ./scripts/check_unsafe_code_allowances.sh
- ./scripts/check_layer_dependencies.sh
- ./scripts/check_architecture_migration_rules.sh
- ./scripts/check_logging_guardrails.sh
- ./scripts/check_error_other_format_ratchet.sh
- ./scripts/check_no_tokio_io_uring.sh
- ./scripts/check_extension_schema_boundaries.sh
- ./scripts/check_body_cache_whitelist.sh
- ./scripts/check_s3s_footprint.sh
- ./scripts/check_test_count_ratchet.sh
- ./scripts/check_fips_wording.sh
- ./scripts/check_embedded_secrets.sh
- make script-tests
- python3 ./scripts/test_object_generation_protocol_model.py
- python3 ./scripts/check_test_wiring.py
- ./scripts/check_no_planning_docs.sh
- ./scripts/check_uring_lane_lib_only.sh

test-count ratchet OK: 52 crates, 22009 tests, 165 ignored match .config/test-count-baseline.txt
PASS: cold stampede script self-test
model: states=51023, transitions=167520, terminal=732, chosen_candidates=2
multi-slot round schedules: 640
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired

* fix(ecstore): retry tier commit intent lock contention

Retry transient namespace lock timeouts within the existing three-attempt
CAS budget. Reload and verify the original mutation identity after
contention, and preserve the final typed error when attempts are exhausted.
Cover save/read contention, exhausted retries, replacement identities and
CAS reconciliation without changing the existing timeout or lock policy.

Validation:
- cargo nextest run --locked -p rustfs-ecstore --features test-util --lib -E 'test(tier_mutation)' --test-threads 1 --no-fail-fast: 52 passed.
- cargo nextest run --locked -p rustfs-ecstore --features test-util --lib -E 'test(intent_advance_) | test(=services::tier::tier::tests::caller_cancellation_after_durable_prepare_does_not_hide_the_mutation)' --test-threads 1 --no-fail-fast: 5 passed.
- cargo clippy --locked -p rustfs-ecstore --all-targets -- -D warnings: passed.
- cargo clippy --locked -p rustfs-ecstore --all-targets --features test-util -- -D warnings: passed.
- 27 mutation pairs failed at the intended assertions and passed after restoring the production code.
- bash scripts/check_test_count_ratchet.sh --update, then bash scripts/check_test_count_ratchet.sh: 52 crates, 22013 tests, 165 ignored; ecstore increases from 6581 to 6585.

* refactor(lifecycle): replace s3s domain types with Gateway DTOs

Keep lifecycle rules, restore requests, Object Lock, and scanner cached tags in domain types. Convert at the existing HTTP boundary while preserving legacy XML, scanner cache bytes, date ordering, and tombstones.

Linux test validation before the final import-path-only adjustment:

cargo nextest run -p rustfs-ecstore bucket_config_goldens
39 passed; 6211 skipped by selector.

cargo nextest run --lib -p rustfs -E 'test(/test_zero_rule_lifecycle_tombstone_recognition/) | test(/test_local_lifecycle_staleness_axis_selection/) | test(/test_lifecycle_staleness_preserves_legacy_submillisecond_axis/) | test(/test_merge_incoming_lifecycle/) | test(/test_lifecycle_expiry_statement_matrix/) | test(/test_lifecycle_expiry_subset_xml/) | test(/put_bucket_lifecycle_validation_errors_keep_their_s3_code/) | test(/execute_put_bucket_lifecycle_configuration_rejects_missing_configuration/) | test(/execute_restore_object_rejects_missing_restore_request/) | test(/execute_restore_object_rejects_select_type/) | test(/execute_restore_object_maps_failures_to_typed_s3_errors/) | test(/execute_restore_object_returns_internal_error_when_store_uninitialized/) | test(/restore_request_conversion_preserves_legacy_output_values/) | test(/restore_request_conversion_preserves_presence_and_raw_enums/) | test(/restore_request_conversion_distinguishes_absent_and_empty_containers/)'
25 passed; 5318 skipped by selector.

cargo nextest run --profile ci -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-replication -p rustfs-scanner
7872 passed; 21 skipped under the existing CI profile.

Previously verified mutation controls: 39 production faults, 41 RED/GREEN pairs.

cargo clippy --locked -p rustfs -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-scanner --all-targets -- -D warnings
Passed in 40m 45s before the import-path adjustment, then in 1m 59s on
the final source with warnings denied. The final adjustment imports the
existing helper through the object domain facade; fmt and the architecture
check also passed.

Part of rustfs/backlog#2745. The remaining ecstore configuration families follow separately.

* refactor(admin): move error types to rustfs-s3-types (#8440)

* refactor(protocols): move error types to rustfs-s3-types by codemod

T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and
the error items of protocols move off s3s by scripts/codemods, run with
--hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it
downcasts the backend error, whose runtime type is still s3s::S3Error.

- rustfs/src/protocols/client.rs: the two webdav session methods now
  return rustfs_s3_types::S3Result and convert the s3s error with From.
- codemod: a child module that calls s3_error! through `use super::*`
  gets its own `use rustfs_s3_types::s3_error;` once its parent imports
  that macro by name; otherwise it is listed as GLOB.
- s3-types: a table test pins code, status and the s3s error the legacy
  edge renders for every variant a moved call site names.
- s3_error_lines 1585 -> 1584 (a lifecycle comment reworded).

* refactor(admin): move error types to rustfs-s3-types (WIP, no build)

T1.2 batch 4 (rustfs/backlog#2743), work in progress: rustfs check
still fails with 16 errors (listed in the PR body).

- codemod: --facade PATH treats a crate-local re-export of s3s's error
  items like s3s for the four names (9 new self-tests, 9 mutants red).
- codemod output for rustfs/src/admin and rustfs/src/site_replication
  (--facade crate::storage_api::site_replication::s3); 6 glob children
  get an explicit rustfs_s3_types::s3_error import.
- admin/router.rs: the Operation trait returns rustfs_s3_types::S3Result.
- error.rs: From<ApiError> builds the rustfs-s3-types error; the s3s
  conversion is derived from it.

* fix(admin): convert error types at the s3s boundary

Batch 4 of T1.2 left the admin and site_replication modules returning
rustfs_s3_types::S3Error while their callers and callees on the storage
and auth side still speak s3s::S3Error. Convert at each edge with the
existing From impls in rustfs-s3-types (compat_s3s.rs) instead of
changing any code, message or status:

- admin/auth.rs, admin/handlers/object_zip_download.rs,
  admin/handlers/table_catalog/mod.rs: map the s3s result returned by
  check_key_valid / authorize_request / authorize_internal_object_request
  into the admin error type with .map_err(Into::into).
- storage/ecfs.rs: the s3s-facing side maps the admin site_replication
  gate results the other way.
- admin/handlers/pools.rs: ApiError.code stays s3s::S3ErrorCode, so the
  rustfs code is converted with .into() when the struct is built, in
  production and in the two tests that build or compare it.
- admin/handlers/heal.rs, admin/runtime_sources.rs: the router returns
  s3s errors (S3Route impl); the expected rustfs code is converted with
  .into() before the comparison.
- admin/handlers/table_catalog/mod.rs: iceberg_rest_error takes
  &'static str so the Cow<'static, str> custom code borrows it; every
  caller already passes an ICEBERG_ERROR_* constant or a &'static str.
- admin/handlers/quota.rs: drop a now same-type .into() after s3_error!.
- storage_api.rs, site_replication/mod.rs: drop re-exports and an import
  that no longer have users (clippy -D warnings).

cargo check -p rustfs --all-targets and cargo clippy -p rustfs
--all-targets -- -D warnings are clean; the s3s footprint baseline is
unchanged at files=172, s3_error_lines=303, ecstore_files=28.

Refs rustfs/backlog#2743, rustfs/backlog#2734.

* refactor(ecstore): move bucket configs onto gateway persisted types (#8709)

* refactor(ecstore): move bucket configs onto gateway persisted types

Finish the ECStore batch of T1.4 (rustfs/backlog#2745). The remaining
eleven bucket configuration families (Website, CORS, Tagging, Versioning,
ObjectLock, Encryption, Notification, Logging, Accelerate, RequestPayment,
PublicAccessBlock) now read and write their stored XML through the
rustfs-gateway-types persistence and cors_tagging codecs instead of the
s3s codec, so the legacy s3s bucket-config codec and its
s3gate-metadata-xml compat marker are removed.

- ecstore: BucketConfigXml is implemented by the gateway codec for every
  family; metadata, metadata_sys, versioning, object lock, replication and
  lifecycle consumers take the persisted types. Region moves to
  rustfs_s3_types::Region, and the ETag conversion to s3s moves to the
  rustfs edge with its tests. Only warm_backend.rs and warm_backend_s3.rs
  still reference s3s, held for T1.5 (rustfs/backlog#2746).
- scanner: object lock cache, item actions and data usage use the
  persisted types.
- rustfs: the legacy S3 edge converts each family between s3s DTOs and the
  persisted types in storage/storage_api.rs by writing one codec's XML and
  reading it with the other; Tagging converts field by field.
- Bump the rustfs-gateway-types pin to 21283e87.

The 39 persisted-byte goldens are unchanged and pass, so a document the
old codec wrote reads back and re-encodes to the same bytes. s3s footprint:
files 181 -> 146, ecstore_files 28 -> 2, s3_error_lines unchanged.

* test(app): prepare the PutBucketEncryption config from the request

The uninitialized-store test passed an empty persisted configuration, which the use case now validates before it reaches the store check, so it answered MalformedXML instead of InternalError. Convert the request's configuration the way the S3 edge does.

* style(app): format the PutBucketEncryption test

* refactor(s3-client): move remote error codes and DTO enums off s3s (#8783)

Part of T1.5 (rustfs/backlog#2746), batch 1.

* refactor(s3select): run queries on gateway DTO request shapes (#8794)

* refactor(s3select): run queries on an engine request built from gateway DTO shapes

Part of T1.5 (rustfs/backlog#2746), batch 2.

* fix(s3select): keep the select request conversion in the app layer

The layer guard forbids app -> storage; the conversion moves from storage/s3_compat/select.rs to app/select_request.rs unchanged.

* fix(heal): retry transient metadata conflicts (#8812)

* fix(heal): retry transient metadata conflicts

* chore: update heal test count baseline

* test: reserve nextest capacity for progress probes (#8818)

* test(e2e): use a non-default region for the differential negative control

Since main's region-aware CreateBucket and GetBucketLocation (#8739, #8743), us-east-1 answers an empty LocationConstraint and 200 for an owned re-create, so the control no longer reported its own region. Also refresh the e2e-full linux selection from the CI listing (682 tests).

* refactor(protocols): move StorageBackend to gateway types (#8825)

* refactor(protocols): move storage backend to gateway types

* test: update e2e test count baseline

* fix: preserve copy source keys and unknown lengths

* test: update protocols test count baseline

* fix(protocols): preserve typed SFTP backend errors

* test(protocols): use gateway list input in policy tests

* fix(protocols): clear clippy failures after T1.5 merge (#8831)

fix(protocols): clear clippy failures after merge

* fix(ecstore): finalize restore metadata after local copy-back (#8836)

* fix(ecstore): finalize restore metadata after local copy-back

* test(ecstore): ratchet added restore finalization cases

* fix(ecstore): keep restore copyback compatible with older peers

* fix(ecstore): gate restore copyback on fleet compatibility

* fix(ecstore): retain transition identity during restore copyback

* fix(ecstore): route committed restore copyback locally

* fix restore copyback cleanup

* fix(ecstore): honor restore probe test fallback

* refactor(storage): migrate option errors to domain types (#8839)

* refactor(s3): migrate private multipart validation errors (#8844)

* refactor(auth): derive policy conditions from AuthzFacts (#8442)

* refactor(auth): extract authorization facts

* test(e2e): update smoke selection baseline

* test(e2e): update full selection baseline

* refactor(admin): migrate health and prefix usage to owned requests (#8887)

* refactor(admin): add owned health router boundary

* refactor(admin): migrate prefix usage to owned request

* refactor(admin): complete the owned admin request boundary (#8949)

* refactor(admin): complete the owned admin request boundary

Every type implementing the owned admin `Operation` now serves the legacy
router through one blanket impl that converts the s3s request and response
at that single edge, so handlers register with `AdminOperation(&Handler {})`
and need no adapter statics; `LegacyAdminAdapter` is removed.

`AdminRequest` gains the accessors handlers read today (`remote_addr`,
`client_info`, `request_context`, `req_info`, `req_info_mut`,
`install_req_info`, `envelope_mut`, `take_body`), the request helpers gain
envelope forms (`authenticate_envelope_credentials`,
`object_store_from_envelope`, the table-catalog `*_from_envelope` helpers,
`KmsAdminAudit::from_envelope`, `AccountAuditContext::from_envelope`,
`CallerIdentity::resolve_envelope`), `read_compatible_admin_body` accepts
either body type, and the owned `Body` gains `store_all_limited` and
`Default`. `admin::operation::test_support::TestRequest` builds owned
requests for handler tests. Unused accessors carry a per-method dead-code
expectation that each migrating handler batch removes.

No handler behavior changes; the remaining migrations follow in batches.

* chore(admin): add the T1.12 admin handler codemod

The script rewrites an admin handler file from the s3s request and response
types to the owned boundary (imports, signatures, request field and helper
calls, body access, response construction, and test request literals) and
leaves a TODO comment at every site it cannot rewrite mechanically. It is
idempotent and is removed once T1.12 completes.

* refactor(admin): migrate replication and KMS handlers to owned requests (#8951)

Move the T1.12 batch 2 admin handlers (site_replication, kms_keys, event,
replication, diagnostics, on_demand_migration, audit, account, kms_rekey,
kms_key_lifecycle, object_data_cache, tls_debug, integrity) from the s3s
request/response types to the owned AdminRequest/AdminResponse boundary.
Handler logic, authorization order, error codes and response bytes are
unchanged; the request/response plumbing now goes through the envelope
accessors and helpers introduced by the owned boundary.

The shared response helpers (utils::json_response, utils::empty_response,
handlers::admin_json_response, target_descriptor::build_json_response,
profile::authorize_profile_request) and the site replication repair
executor still answer with the s3s types, so the migrated files carry
owned-boundary forms of those helpers and convert the repair response at
its call edge. The streaming console-log body and the two devnull drains
move from the s3s byte stream to the owned Body stream/frame API.

Drop the dead_code expectations for the owned accessors and envelope
helpers this batch starts using, and lower the s3s footprint baseline to
the measured 99 files.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* refactor(admin): add owned forms of the shared admin response helpers

Handler batches were copying `json_response`, `empty_response`,
`admin_json_response`, and `build_json_response` into each migrated file
because the shared helpers still return the s3s response. Provide owned
forms next to each legacy helper (`owned_json_response`,
`owned_empty_response`, `owned_admin_json_response`,
`owned_build_json_response`, `authorize_profile_envelope`) with parity
tests against the legacy shape, derive `Debug` for `AdminResponse` so
tests can use `expect_err`, and fix three codemod rules: strip owned
string conversions in test credentials, annotate non-literal URI parses,
and leave a rebound request parameter immutable.

* refactor(admin): use the shared owned response helpers in batch 2 (#8954)

Replace the per-file owned-boundary copies of json_response,
empty_response, admin_json_response, build_json_response and
authorize_profile_request in the batch 2 admin handlers with the shared
owned_* helpers and authorize_profile_envelope that the boundary branch
now provides, and restore expect_err in the handler tests now that
AdminResponse implements Debug; the `.err().expect(..)` stand-ins were
failing the base branch's clippy step (clippy::err_expect under
-D warnings). The site replication repair call edge
keeps its conversion because the repair executor still answers with the
s3s response type.

Drop the dead_code expectations on the shared owned helpers this batch
now uses in non-test code.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* fix(admin): preserve legacy body limit error responses

* refactor(admin): move policy, ILM and batch 3 handlers to owned requests (#8956)

Batch 3 of T1.12 (rustfs/backlog#2788, parent rustfs/backlog#2734) moves
14 admin handler files to AdminRequest/AdminResponse and the owned
Operation trait: ilm_transition, policies, idp_compat, config_admin,
plugins_instances, profile_admin, group, heal, object_zip_download,
kms_key_metadata, durability, inspect_archive, sts and
gateway_key_inventory. Handler logic, authorization order, error codes
and response bytes are unchanged; test request literals use
test_support::TestRequest.

The handlers use the shared owned helpers (owned_json_response,
owned_admin_json_response, owned_build_json_response,
authorize_profile_envelope). The dead_code expectations on the
AdminRequest accessors and RequestEnvelope::uri_mut they now exercise
are removed. The legacy target_descriptor::build_json_response keeps
only its equivalence test caller and carries the matching expectation
until it retires.

object_zip_download builds the s3s request the S3 gate still reads from
the owned request facts; heal authenticates through
authenticate_envelope_credentials; the trace, inspect-archive and zip
bodies use Body::from_stream. extract_plugin_instance_filters takes the
URI so extensions.rs keeps calling it, and the two idp_compat
delegations stay on the legacy trait until their delegates migrate.

The s3s footprint baseline drops from 99 to 90 files.

* test(admin): anchor the replication diff body contract on the call

The source-contract test looked for `let body = read_compatible_admin_body(`
on one line; rustfmt wrapped that binding after the handler moved to the
owned request boundary, so the marker was no longer found. Anchor on the
call itself so formatting cannot change what the contract checks.

* refactor(admin): migrate table catalog and batch 1 to owned requests (#8955)

* fix(admin): preserve catalog body limit response bytes

* refactor(admin): migrate identity and cluster handlers to owned requests (#8966)

* refactor(admin): migrate identity and cluster handlers to owned requests

Move the T1.12 batch 4 admin handlers (user, service_account, oidc, tier,
system, pools, quota, kms_dynamic, kms_backup, rebalance, module_switch,
bucket_meta, cluster_snapshot, realtime, bucket_recovery, profile) from the
s3s request/response types to the owned AdminRequest/AdminResponse
boundary. All 90 impl Operation blocks now take AdminRequest and return
AdminResponse; handler logic, authorization order, error codes and response
bytes are unchanged. The handlers use the shared owned helpers
(owned_json_response, owned_admin_json_response, authorize_profile_envelope,
the envelope-form runtime sources and audit constructors) instead of local
copies.

profile.rs drops the s3s forms of authorize_profile_request and
profile_not_implemented_response now that profile_admin.rs is on the
envelope gate too; profile_not_implemented_response answers with an
AdminResponse. The realtime metrics stream moves from the s3s ByteStream
adapter to Body::from_stream, and the rebalance idle-404 document still
renders through s3s::S3Error::to_http_response with its body converted at
the edge.

The idp_compat delegators ImportIamV2 and AddServiceAccountLdap move to the
owned trait as well: with their targets on both traits, the method-call
delegation became ambiguous between the owned impl and the router blanket
impl, so they now take the owned request directly.

KmsAdminAudit::from_request loses its last production caller with this batch
and is deleted; its comparison test now asserts the envelope facts directly.

Lower the s3s footprint baseline to the measured 68 files.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* refactor(admin): remove the legacy admin helpers batch 4 superseded

Batch 4 moved the last production callers of the s3s-typed admin helpers
to their owned forms, which left `admin_json_response`, `json_response`,
`authenticate_request`, `authorize_admin_request`, `object_store_from_req`
and `object_store_from_extensions` unused, along with the `Body`,
`S3Request` and `S3Response` re-exports of the admin `s3` facade. Remove
them. The owned forms keep their documentation; the behavior tests of the
removed `json_response` now exercise `owned_json_response`, and the
compat-prefix test asserts the owned response directly.

Two tests existed only to compare a removed legacy helper with its owned
twin (`owned_json_response_matches_the_legacy_shape`) or to cover the
removed legacy gate (`authorize_admin_request_without_credentials_is_rejected`);
they go with the helpers, and the owned equivalents stay. The rustfs
test-count row drops from 5957 to 5955 and the s3s footprint from 68 to 66
files.

* refactor(admin): drop the legacy target descriptor JSON response helper (#8968)

`target_descriptor::build_json_response` returned the s3s response and
had no production caller left once the event, audit and plugin instance
handlers moved to `owned_build_json_response`; it survived only for the
legacy/owned equivalence test. Remove it, give the owned helper its own
documentation, and make the test assert the owned response directly:
status, `application/json`, the echoed `x-request-id` when present and
only then, and the body bytes. The file no longer imports s3s, so the
footprint baseline drops from 66 to 65 files.

* test(ecstore): pin native history on cancellation under the cancellable queue

main moved the local write path onto the disk blocking queue, which skips an
operation once its waiter is dropped, so a borrowed write cancelled while it is
queued no longer creates its file. The integration-only native publication
test still expected the queued open to run. Assert the new execution model
instead (the open never runs) while keeping its safety claims: the Started
record is taken before queueing and the cancelled publication never regrants
native history.

* chore(gateway): raise test-count rows for tests merged from main

* test(s3select): use the owned JSON input types in merged planner tests

* fix(ecstore): scope the merged direct-writer test hook to test builds

* fix(ecstore): create parents before admitted truncate opens

Align the heal tests with removal-disabled preservation and local-only outcome
fields, and refresh the e2e-full selection for the synced tests.

* chore(gateway): update test-count baseline after main sync

Main #9241 removed the node pool-metadata readiness cache with 8 tests
and added 6, so the rustfs row drops from 6440 to 6438. The ecstore and
s3select rows rise for tests added by #9237, #9239 and #9241.

* feat(scripts): add junit_diff.py for dual-stack JUnit comparison (#9275)

* refactor(auth): share authorization core with owned entry points

Extract the policy decision behind authorize_request into a core that both the legacy s3s request and the RustFS-owned RequestEnvelope feed, and move the zip-create and table-catalog admin callers to the owned entry points (authorize_owned_request, authorize_owned_internal_object_request). Request-body policy inputs added on main since the candidate base (POST form headers, copy source, legal hold and tagging bodies) become named envelope fields so both edges reach the same decision. Refs rustfs/backlog#2749.

* ci(gateway): make the test-count ratchet a floor, not exact (#9278)

ci(gateway): make the test-count ratchet a floor instead of an exact baseline

Tests below a row still fail; tests above it, a member without a row and ignores below the row now pass with a note. Ignores above a row still fail. Adds contract tests wired into make script-tests.

* refactor(admin): move remaining admin handlers to owned requests

Migrate the ten table-catalog commit and publication handlers to the
owned admin Operation trait, redeem object zip download tokens through
authorize_owned_request, and remove the s3s-typed router Operation
trait. S3Router now converts the s3s request into an AdminRequest and
the AdminResponse back at its edge, so no admin operation takes an s3s
type (rustfs/backlog#2788).

* chore(gateway): record ignored microbenchmarks merged from main (#9290)

chore(gateway): record the ignored microbenchmarks merged from main

main added three throughput microbenchmarks that are ignored by design
(#9217): one each in rustfs-checksums, rustfs-rio and rustfs-utils. The
test-count ratchet treats a higher ignored count as a failure until the
row is raised, so raise those three rows to 1 and tighten the test rows
to the merged tree with --update.

* fix(auth): keep owned authorization in the app layer and fail closed

Move the RequestEnvelope impl of AuthorizationRequest and the owned entry points into the app layer so storage depends only on the request-facts trait (layer guard). The owned entry points now hoist signed x-amz-meta-* query parameters like the legacy edge and refuse actions whose decision needs a request-body input the envelope does not carry. Revert unrelated TODO rewording and redact PostObjectPolicyHeaders debug output.

* test(auth): move owned parity tests to the app layer

Owned-versus-legacy parity, hoist and fail-closed tests now live in app/owned_authorization_test.rs beside the envelope's facts impl, so storage names no app type in test code either. Private storage steps are reached through a test-only parity_test_support module. Also fix the S3Error conversion CI reported in prepare_owned_authorization.

* test(admin): convert the tagged table catalog request before installing identity

* ci(s3-tests): add reference parity lane against pinned 1.0.5 (#9288)

* fix(admin): keep owned router entry points crate-visible

* test(auth): read storage-private parity facts through test accessors

The parity tests moved to the app layer can no longer read ObjectTagConditions and TableDataPlanePublicationGuards fields directly; parity_test_support exposes read-only accessors instead. Drop an unused import.

* test(auth): owned and legacy entry points reach the same decisions

Decision-level parity over IAM allow and explicit deny, owner, anonymous on private and public buckets, bucket-policy explicit deny, aws:SourceIp from the peer and from a verified client, and one allow and one deny per body-derived input (legal hold, tagging, retention mode and retain-until, resolved copy source, POST form fields).

* fix(admin): drop the owned trait bound from S3Router's inherent impls

* test(auth): satisfy clippy on the parity test facade

* test(admin): cover zip download token redemption authorization

* test(auth): report every decision mismatch in the parity matrix

* refactor(admin): move the admin route table out of the s3s router

* test(auth): cover multipart staging in the owned/legacy decision matrix

* fix(admin): authorize warehouse locations with the owned commit request

* fix(admin): return the owned table catalog authorization result unchanged

* fix(table-catalog): restore owned request lines the main merge reverted

The merge onto main replaced two lines with main's text: the warehouse location check again wrapped an already owned request in legacy_request_view, and an error conversion was re-added. Restore the branch's version of the file.

* fix(auth): make an absent caller secret unrepresentable as a key

RequestCredentials::secret_key now returns Option<&str>: None when the edge kept the secret, and an empty handed-over secret is refused at construction. Secret comparisons go through secret_key_matches, which never matches an absent secret, and readers that need the value fail closed with an internal error.

* fix(auth): keep the test-only credentials rebuild out of the lib build

---------

Co-authored-by: rustfs-cla-assistant[bot] <273121464+rustfs-cla-assistant[bot]@users.noreply.github.com>
2026-11-11 18:08:28 -05:00
Chris An 6e82e08f8e test(guards): let the split probe check already-split sources (#9343)
test(guards): let the split probe check sources that are already split

The end-to-end split probe simulated every split itself and stopped on a tree where bucket_usecase.rs is already split. It now decides per source (ecfs.rs, bucket_usecase.rs, multipart_usecase.rs) whether the tree is unsplit or split: unsplit sources are simulated as before, split sources get the same planted violations at the top of real split files plus a silence check over every other real split file, and a source in neither state fails with its name. Refs rustfs/backlog#2765.
2026-11-11 17:39:16 -05:00
Chris An e98b9132ca test(guards): read file plus split directory in text guards (#9292)
* feat(server): add the RUSTFS_S3_STACK selector and dual-stack e2e lane (#8411)

* refactor(server): group the s3s-compat tower layers under legacy_compat (#8410)

Move the ten tower layers that patch s3s behaviour out of
rustfs/src/server/layer.rs into rustfs/src/server/legacy_compat/, one file
per layer with its tests. Each file header names the s3s behaviour it
patches, the gateway mechanism that replaces it (Replaced by:), and the task
that removes it (Removed by: T2.12, rustfs/backlog#2771). The layers keep
their code, their order in external_service_stack!, and their behaviour; no
layer is deleted.

Add server::http::tests::stack_census, which renders the ordered service
chain of the real external_service_stack! expansion and compares it with a
committed constant, so a reorder, insertion or removal is a visible diff.

Route the two s3s error types the moved layers use through a
storage_api::server::legacy_compat shim so the split adds no s3s-importing
file, and lower the s3s footprint baselines to the measured values (files 209
to 208, s3_error! lines 1586 to 1585). Keep the server runtime-source bypass
guard covering the moved code.

Refs rustfs/backlog#2787, rustfs/backlog#2734

* feat(s3-types): add the RustFS-owned S3 error type and s3_error! macro (#8406)

`rustfs-s3-types` now provides `S3ErrorCode` (generated by build.rs from
src/codes.txt, the grep-derived call surface, plus `Custom`), `S3Error`,
`S3Result`, `StdError` and the `s3_error` macro with the same call syntax
as s3s, so the T1.2 codemod is a `use` swap. Statuses are hand-written and
pinned to the s3s behaviour RustFS ships today by an oracle test that uses
s3s as a dev-dependency only. The `From` conversions for the legacy edge
live behind the off-by-default `compat-s3s` feature (the orphan rule
forbids them in rustfs/src/error.rs); `rustfs` enables it.

Refs rustfs/backlog#2742, rustfs/backlog#2734

* ci: add the per-crate test-count ratchet (#8404)

Quick Checks now fails any PR that lowers the number of test attributes
in a workspace crate or adds #[ignore], measured per crate against the
exact rows of .config/test-count-baseline.txt (rustfs/backlog#2734
section 1 rule 3). The counter widens the spec's literal
#[(tokio::)?test] to the tokio attributes that carry arguments and both
spellings of #[test_case(...)], and anchors at line start so
commented-out attributes do not count. The initial baseline covers 52
crates, 21077 tests and 164 ignored.

Refs rustfs/backlog#2738
Refs rustfs/backlog#2734

* test(ecstore): pin the persisted bucket-config XML bytes for 13 families (#8408)

Before any DTO or codec is replaced, pin the exact bytes that
deserialize followed by serialize produces for every persisted
bucket-configuration XML family in its three source shapes (AWS
documentation, MinIO-written, RustFS-written), and the bytes RustFS
writes today for a typed value. Byte equality only: a structural
comparison would let a codec replacement change the persisted encoding
unnoticed.

The generator script refuses to overwrite existing goldens without
--force, refuses on dirty codec paths, cannot report success from its
write mode, and records the generating commit in the fixtures README.

Refs rustfs/backlog#2744, rustfs/backlog#2734

* ci: raise the test-count baseline to the integration tree (#8426)

* ci: make the s3s footprint baseline exact and lower-only (#8403)

* ci: make the s3s footprint baseline exact and lower-only

The s3s footprint guard compared three counters against ceilings held in
the script itself, so a ceiling could sit above the tree and a drift was
fixable by raising it (rustfs/rustfs#5964 raised the s3_error! ceiling
to go green). The three values now live in
.config/s3s-footprint-baseline.txt as files, s3_error_lines and
ecstore_files, the script reads only that file, and a measurement that
differs from its baseline in either direction fails: growth prints
"s3s footprint grew: <measured> > <baseline>", shrinkage prints
"lower the baseline to <measured> in this PR". The counting method is
unchanged; the baseline is the current measurement of
gateway/integration (209 / 1585 / 36). The header records the end state:
once all three reach 0, T4.2 switches the script to failing on any
occurrence.

Refs rustfs/backlog#2739, rustfs/backlog#2734

* ci: add the s3s edge allowlist mode to the footprint guard (#8405)

`scripts/check_s3s_footprint.sh --mode allowlist` fails on every Rust
file that references s3s (`s3s::` or `s3s_sigv4`) and is not matched by
an entry of the new `.config/s3s-edge-allowlist.txt`; `--dry-run` lists
those files and exits 0. The allowlist carries the initial edge from the
spec, and the script rejects an entry that names or matches anything
under crates/ other than crates/e2e_test. The baseline mode and the
argument-less CI invocation are unchanged.

Refs rustfs/backlog#2741, rustfs/backlog#2734

* test(ecstore): reach the bucket-config codec through the test facade (#8428)

* refactor(s3-types): own the HTTP Body and drop s3s from three crates (#8431)

Add rustfs_s3_types::Body, a BoxBody<Bytes, StdError> wrapper with
empty() and From<Bytes | Vec<u8> | String | &'static str>, and the
legacy From conversions to and from s3s::Body behind compat-s3s.

The signer is now generic over the request body: it only reads and
writes the request head, so it needs no body type and its callers keep
theirs. s3-client sends its requests with the owned Body, madmin signs a
bodiless head, and targets drops its two S3Request/S3Response header
wrappers in favour of extract_params_header.

rustfs-signer, rustfs-madmin and rustfs-targets no longer depend on
s3s; the s3s footprint baseline drops from 209 to 202 files.

Refs rustfs/backlog#2747, rustfs/backlog#2734

* refactor(ecstore): move S3ErrorCode to rustfs-s3-types by codemod (#8432)

* test(ecstore): pin the InvalidRange responses of new_getobjectreader

Both error branches of new_getobjectreader build an ErrorResponse with S3ErrorCode::InvalidRange, and no test observed either code. Pin the code, its wire name and the location fields of each branch before the code type moves to rustfs-s3-types (rustfs/backlog#2743), so a changed code on either branch fails a test. The ecstore test-count row rises by the two tests.

* chore(scripts): add the s3s-to-rustfs-s3-types error codemod

Task T1.2 of rustfs/backlog#2734 (rustfs/backlog#2743) moves S3Error, S3ErrorCode, S3Result and s3_error! from s3s to rustfs-s3-types by script, one crate closure per PR. The codemod rewrites use trees (splitting mixed s3s imports) and qualified code paths, never comments or literals, adds the rustfs-s3-types dependency, drops s3s only from a crate that no longer names it, refuses every file of the s3s edge allowlist, takes --hold for a file whose type is fixed outside the closure, and prints a per-crate census. A second run changes nothing. Its self-test runs under make script-tests.

* refactor(ecstore): take S3ErrorCode from rustfs-s3-types

Output of scripts/codemods/s3_error_to_s3_types.py over crates/ecstore, crates/lifecycle, crates/scanner and crates/replication (rustfs/backlog#2743 batches 1 and 2). The ErrorResponse code field, the two InvalidRange constructions in new_getobjectreader and an unused tier import now use rustfs_s3_types::S3ErrorCode; the variant names and wire names are unchanged. lifecycle, scanner and replication reference none of the four error names. warm_backend.rs and warm_backend_s3.rs are held: they compare against rustfs_s3_client::ErrorResponse.code, which stays the s3s type until s3-client moves.

* ci(footprint): count only s3_error! lines still on s3s's macro

s3_error_lines counted every line holding the text `s3_error!`, which
also matches the RustFS-owned `rustfs_s3_types::s3_error!`. A codemod
that moves call sites onto rustfs-s3-types therefore could not lower it
(rustfs/backlog#2743, decision 3; rustfs/rustfs#8432, gotcha 2).

The counter now applies a textual, per-file and per-line rule that errs
on the side of counting (full rule in the script header):
- occurrences spelled `rustfs_s3_types::s3_error!` never count;
- a line holding `s3s::s3_error!` always counts;
- any other `s3_error!` line counts unless its file is cleared, that is,
  it imports the macro from rustfs_s3_types by name on a line starting
  with the use item (or defines it with `macro_rules! s3_error`) and does
  not also bring s3s's macro in (`use s3s::s3_error`, a `use s3s::{..}`
  tree naming it or a crate-root `*`, `use s3s::*`, or
  `#[macro_use] extern crate s3s`);
- fail closed: a file matching neither form (`use super::*`, a
  crate-local re-export, a renamed or glob rustfs_s3_types import, a
  commented-out import, a comment-only mention) counts.

Census of the 1585 lines on 96fc06776: 1478 in 64 files that import the
macro from s3s; 107 in 15 files matching neither form (105 through
`use super::*`, 1 qualified `s3s::s3_error!`, 1 comment line); none in
files that import it from rustfs_s3_types or define it. Every line stays
counted, so the baseline remains s3_error_lines=1585 (files=202 and
ecstore_files=36 unchanged). Rewriting rustfs/src/auth.rs's import to
rustfs_s3_types lowers the new counter to 1567; the old one stayed 1585.

`--self-test` (wired into `make script-tests`) copies the script into
22 fixture trees: 6 positive cases and 16 negative ones (lines that must
not count, forms that must not clear a file, and a missing baseline,
missing key or failing classification search that must fail). Each tree
has a control line that must count, and each run gets s3s text on stdin.
Seven cases were red against the old counter. 19 mutants each turn their
case red: old definition, flipped import test, dropped fail-closed
branch, dropped e2e_test exclusion, dropped s3s veto, dropped qualified
s3s rule, dropped rustfs-qualified strip, unanchored rustfs import,
renamed import accepted, nested glob taken as root glob, single-line
s3s trees only, dropped s3s glob forms, dropped macro_use form, dropped
definition clearing, ignored classification rg errors, error-line search
without its path, classification searches without their path, missing
baseline skipped, missing key skipped.

* test(e2e): compare two RUSTFS_S3_STACK servers over every S3 operation (#8435)

Add dual_stack_differential to e2e_test. It starts two rustfs processes
from the same binary, stack A on RUSTFS_S3_STACK=legacy and stack B on
RUSTFS_E2E_STACK_B (default legacy until rustfs/backlog#2734 T2.5),
sends both the same aws-sdk-s3 script covering the 73 methods of
impl S3 for FS (a normal request and a 4xx negative each), and compares
status, error code, an allowlist of response headers and a few typed
body fields. Error message text is never compared. Each run writes
<target>/dual-stack/<run>.json with compared_ops and the (op, field,
a, b) differences; any difference fails the test.

differential_detects_an_injected_difference starts B with
RUSTFS_REGION=eu-west-1 and requires GetBucketLocation to be reported.
The comparator is pure and unit-tested without a server.

Wire the module into the e2e-smoke filter and update the e2e-smoke and
e2e-full selection digests and the e2e_test test-count row.

Refs rustfs/backlog#2740

* feat(app): add the RustFS-owned RequestEnvelope and its s3s constructor (#8437)

* feat(app): add the RustFS-owned RequestEnvelope and its s3s constructor

* ci: classify the s3s edge adapters in storage/s3_compat as interface

* feat(s3-types): add Body::from_stream and move Swift off s3s::Body (#8438)

rustfs/backlog#2747 batch 3. `Body::from_stream` wraps a chunk stream of
unknown length over `StreamBody`/`BoxBody`: one data frame per chunk, the
first error ends the body, the size hint stays unknown, and dropping the
body drops the stream. Parity tests pin it to the s3s `StreamingBlob::wrap`
it replaces, at the `http_body` level and byte for byte through hyper.

The Swift handlers in rustfs-protocols build their responses as the owned
`Body`. `SwiftService` converts them into the wrapped S3 service's response
body type through `From<Body>`, so rustfs keeps serving `s3s::Body` at the
legacy edge unchanged. Request bodies stay Send-only (`UnsyncBoxBody`); the
two storage writers and the DLO/SLO segment streams get `Sync` from
`sync_wrapper::SyncStream`, which is what s3s's `http_body_unsync` mutex
provided before. The DTO and error uses of s3s in protocols are untouched
(T1.5 and T1.2).

* refactor(protocols): move error types to rustfs-s3-types by codemod (#8439)

T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and
the error items of protocols move off s3s by scripts/codemods, run with
--hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it
downcasts the backend error, whose runtime type is still s3s::S3Error.

- rustfs/src/protocols/client.rs: the two webdav session methods now
  return rustfs_s3_types::S3Result and convert the s3s error with From.
- codemod: a child module that calls s3_error! through `use super::*`
  gets its own `use rustfs_s3_types::s3_error;` once its parent imports
  that macro by name; otherwise it is listed as GLOB.
- s3-types: a table test pins code, status and the s3s error the legacy
  edge renders for every variant a moved call site names.
- s3_error_lines 1585 -> 1584 (a lifecycle comment reworded).

* test(connect): restore one-second drive request budgets

Apply the exact test-only repair already landed on main in 5972a09cc3. The filesystem probe and production CLI success test request one second and keep their existing one-second duration and parameter checks. All outcome, I/O, error, cleanup and resource-limit assertions remain.

Scope: rustfs/tests/connect_perf_drive.rs only. Patch-id: cb0a1dfc8f68a67727525adea7458e2cee7d09f8.

Validation environment: LANG=en_US.UTF-8, CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0, exclusive APFS-cloned CARGO_TARGET_DIR.

Quick Checks: every command from .github/actions/quick-checks/action.yml, with captured output and exit status below.

$ shellcheck --version && actionlint
ShellCheck - shell script analysis tool
version: 0.11.0
license: GNU General Public License, version 3
website: https://www.shellcheck.net
exit_code=0

$ cargo fmt --all --check
exit_code=0

$ ./scripts/check_unsafe_code_allowances.sh
exit_code=0

$ ./scripts/check_layer_dependencies.sh
Layer dependency guard passed (no new reverse dependencies/cycles).
exit_code=0

$ ./scripts/check_architecture_migration_rules.sh
Architecture migration rules passed.
exit_code=0

$ ./scripts/check_logging_guardrails.sh
✅ Logging guardrails check passed
exit_code=0

$ ./scripts/check_error_other_format_ratchet.sh
error(format!) ratchet passed (total call sites: 582).
exit_code=0

$ ./scripts/check_no_tokio_io_uring.sh
exit_code=0

$ ./scripts/check_extension_schema_boundaries.sh
Extension schema boundaries passed.
exit_code=0

$ ./scripts/check_body_cache_whitelist.sh
Body-cache whitelist guard passed.
exit_code=0

$ ./scripts/check_s3s_footprint.sh
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)
✅ s3s footprint baseline check passed
exit_code=0

$ ./scripts/check_test_count_ratchet.sh
test-count ratchet OK: 52 crates, 21222 tests, 164 ignored match .config/test-count-baseline.txt
exit_code=0

$ ./scripts/check_fips_wording.sh
FIPS wording guard passed (README.md and CHANGELOG.md contain no forbidden claims).
KMS wording guard passed (crates/kms claims no Vault KV2 Transit wrapping).
exit_code=0

$ ./scripts/check_embedded_secrets.sh
Embedded secret guard passed (no private key material or provider credential literal in the tree).
exit_code=0

$ make script-tests
Running script tests...
entrypoint credential policy tests passed
internode grpc A/B bench env tests passed
object batch benchmark enhanced tests passed
PASS package service script contract
hotpath warp A/B gate tests passed
hotpath warp ABBA tests passed
Scanner validation artifacts written to /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.1pFe6Kl1Yu/out
Scanner validation artifacts written to /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.1pFe6Kl1Yu/out-missing-pid
.............................
----------------------------------------------------------------------
Ran 29 tests in 181.286s

OK
{"decision":"verified","pending_gates":["G01","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G02"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-E"}
{"decision":"verified","pending_gates":["G01","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G02"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-E"}
{"decision":"verified","pending_gates":["G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G01"}
{"decision":"verified","pending_gates":["G01","G02","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G03"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-L"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G14"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G14"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G10"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P1"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P2"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P3"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G05"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G06"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-D"}
Status-and-outcome release descriptor verified: /private/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp8rwn1pe0/descriptor/release-bundle-status-outcome.json
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G05"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G06"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-D"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G07"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G08"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P4"}
exact 1MiB handoff ABBA tests passed
pinned paired ABBA bench tests passed
==> /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/bin/path_containment (-max_total_time=7, -seed=123456789, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/explicit-artifacts/path_containment/, corpus=/private/tmp/gw-orchestrator-2734/ci-drive/fuzz/corpus/path_containment)
-seed=123456789
target=path_containment
seed=123456789
max_total_time=7
git_revision=4d15b816236630b2c42f321ffc9ed3654edc94e3
git_dirty=true
runner_mode=prebuilt
==> /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/bin/path_containment (-max_total_time=1, -seed=2322617706, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/auto-artifacts/path_containment/, corpus=/private/tmp/gw-orchestrator-2734/ci-drive/fuzz/corpus/path_containment)
-seed=2322617706
==> cargo +nightly fuzz run path_containment (-max_total_time=9, -seed=987654321, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/cargo-artifacts/path_containment/)
+nightly
fuzz
run
path_containment
-seed=987654321
seed=987654321
runner_mode=cargo-fuzz
fuzz runner seed manifest tests passed
✅ scripts/python_bin.sh resolver checks passed
self-test ok: clean fixture with every exemption present passes
self-test ok: all 11 pattern families fire
self-test ok: moving a pinned source is reported
Embedded secret guard self-test passed.
self-test ok: p1-s3s-import: 'use s3s::s3_error;' counts every line, the rustfs_s3_types module's too (s3_error_lines=3)
self-test ok: p2-s3s-mixed-tree: a multi-line s3s use tree naming s3_error counts (s3_error_lines=3)
self-test ok: p3-s3s-glob: a crate-root glob of s3s, bare or inside a tree, counts (s3_error_lines=5)
self-test ok: p4-macro-use: '#[macro_use] extern crate s3s' counts (s3_error_lines=3)
self-test ok: p5-qualified-s3s: a qualified s3s::s3_error! line counts in a cleared file (s3_error_lines=2)
self-test ok: p6-exact-pass: a baseline equal to the tree passes (exit 0)
self-test ok: n1-rustfs-import: 'use rustfs_s3_types::s3_error;' alone counts nothing (s3_error_lines=1)
self-test ok: n2-rustfs-mixed-tree: a rustfs_s3_types tree plus an s3s tree without the macro counts nothing (s3_error_lines=1)
self-test ok: n3-defines: the file defining the macro counts nothing (s3_error_lines=1)
self-test ok: n4-qualified-rustfs: a qualified rustfs_s3_types::s3_error! line counts nothing (s3_error_lines=1)
self-test ok: n5-renamed-call: a call through a renamed macro is not text-matched (out of scope) (s3_error_lines=1)
self-test ok: n6-comment-string: a comment or string mentioning s3_error! in a cleared file counts nothing (s3_error_lines=1)
self-test ok: n7-e2e-excluded: crates/e2e_test is excluded (s3_error_lines=1)
self-test ok: n8-super-glob: an unrecognised import form ('use super::*;') counts (s3_error_lines=3)
self-test ok: n9-facade: a crate-local re-export counts (s3_error_lines=2)
self-test ok: n10-renamed-import: a renamed rustfs_s3_types import does not clear s3_error! lines (s3_error_lines=2)
self-test ok: n11-commented-import: a commented-out rustfs_s3_types import does not clear (s3_error_lines=2)
self-test ok: n12-rustfs-glob: a glob import of rustfs_s3_types does not clear (s3_error_lines=2)
self-test ok: n13-comment-only: a comment mentioning s3_error! in a file with no import counts (s3_error_lines=2)
self-test ok: f1-baseline-missing: a missing baseline file fails (exit 1)
self-test ok: f2-key-missing: a baseline without s3_error_lines fails (exit 1)
self-test ok: f3-rg-fails: a failing classification search fails the guard (exit 1)
✅ s3s footprint self-test passed
test_core_gate_rejects_missing_ignored_filtered_and_corrupt_inputs (__main__.SelfTests.test_core_gate_rejects_missing_ignored_filtered_and_corrupt_inputs) ... ok
test_e2e_requires_registration (__main__.SelfTests.test_e2e_requires_registration) ... ok
test_fuzz_runtime_uses_matrix_target (__main__.SelfTests.test_fuzz_runtime_uses_matrix_target) ... ok
test_ilm_lane_keeps_the_measured_cargo_build_budget (__main__.SelfTests.test_ilm_lane_keeps_the_measured_cargo_build_budget) ... ok
test_profile_listing_binds_platform_digest (__main__.SelfTests.test_profile_listing_binds_platform_digest) ... ok
test_profile_listing_enforces_selection (__main__.SelfTests.test_profile_listing_enforces_selection) ... ok
test_quick_checks_commands_propagate_failure (__main__.SelfTests.test_quick_checks_commands_propagate_failure) ... ok
test_quick_checks_rejects_caller_and_execution_bypasses (__main__.SelfTests.test_quick_checks_rejects_caller_and_execution_bypasses) ... ok
test_runner_readiness_fails_closed (__main__.SelfTests.test_runner_readiness_fails_closed) ... ok
test_s3_runner_rejects_unbounded_failure_locals (__main__.SelfTests.test_s3_runner_rejects_unbounded_failure_locals) ... ok
test_scanner_heal_begin_requires_embedded_source_provenance (__main__.SelfTests.test_scanner_heal_begin_requires_embedded_source_provenance) ... ok
test_scanner_heal_case_does_not_approve_pending_release (__main__.SelfTests.test_scanner_heal_case_does_not_approve_pending_release) ... ok
test_scanner_heal_case_lane_is_required_for_runner_profile (__main__.SelfTests.test_scanner_heal_case_lane_is_required_for_runner_profile) ... ok
test_scanner_heal_case_only_schema_cannot_approve_release (__main__.SelfTests.test_scanner_heal_case_only_schema_cannot_approve_release) ... ok
test_scanner_heal_crash_case_rejects_restart_oracle (__main__.SelfTests.test_scanner_heal_crash_case_rejects_restart_oracle) ... ok
test_scanner_heal_ec84_case_rejects_wrong_erasure_geometry (__main__.SelfTests.test_scanner_heal_ec84_case_rejects_wrong_erasure_geometry) ... ok
test_scanner_heal_finish_collects_oracles_from_registry (__main__.SelfTests.test_scanner_heal_finish_collects_oracles_from_registry) ... ok
test_scanner_heal_multipool_case_accepts_deferred_optional_outage_write (__main__.SelfTests.test_scanner_heal_multipool_case_accepts_deferred_optional_outage_write) ... ok
test_scanner_heal_pending_gate_cannot_map_to_implemented_lane (__main__.SelfTests.test_scanner_heal_pending_gate_cannot_map_to_implemented_lane) ... ok
test_scanner_heal_receipts_reject_reuse_and_missing_builds (__main__.SelfTests.test_scanner_heal_receipts_reject_reuse_and_missing_builds) ... ok
test_scanner_heal_rejects_boolean_fractional_and_out_of_geometry_integers (__main__.SelfTests.test_scanner_heal_rejects_boolean_fractional_and_out_of_geometry_integers) ... ok
test_scanner_heal_rejects_broken_execution_and_artifacts (__main__.SelfTests.test_scanner_heal_rejects_broken_execution_and_artifacts) ... ok
test_scanner_heal_rejects_copied_junit_and_wrong_suite_build (__main__.SelfTests.test_scanner_heal_rejects_copied_junit_and_wrong_suite_build) ... ok
test_scanner_heal_release_bundle_accepts_complete_measured_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_accepts_complete_measured_evidence) ... ok
test_scanner_heal_release_bundle_assembler_copies_measured_artifacts (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_copies_measured_artifacts) ... ok
test_scanner_heal_release_bundle_assembler_merges_measured_descriptors (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_merges_measured_descriptors) ... ok
test_scanner_heal_release_bundle_assembler_rejects_duplicate_descriptor_gate (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_duplicate_descriptor_gate) ... ok
test_scanner_heal_release_bundle_assembler_rejects_fixture_descriptor (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_fixture_descriptor) ... ok
test_scanner_heal_release_bundle_assembler_rejects_hash_mismatch (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_hash_mismatch) ... ok
test_scanner_heal_release_bundle_assembler_rejects_mixed_descriptor_revisions (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_mixed_descriptor_revisions) ... ok
test_scanner_heal_release_bundle_enforces_topology_duration_profile_and_versions (__main__.SelfTests.test_scanner_heal_release_bundle_enforces_topology_duration_profile_and_versions) ... ok
test_scanner_heal_release_bundle_fixture_writer_is_not_release_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_fixture_writer_is_not_release_evidence) ... ok
test_scanner_heal_release_bundle_gate_accepts_single_measured_gate (__main__.SelfTests.test_scanner_heal_release_bundle_gate_accepts_single_measured_gate) ... ok
test_scanner_heal_release_bundle_gate_rejects_missing_field (__main__.SelfTests.test_scanner_heal_release_bundle_gate_rejects_missing_field) ... ok
test_scanner_heal_release_bundle_gate_rejects_unknown_gate (__main__.SelfTests.test_scanner_heal_release_bundle_gate_rejects_unknown_gate) ... ok
test_scanner_heal_release_bundle_rejects_synthetic_or_missing_fields (__main__.SelfTests.test_scanner_heal_release_bundle_rejects_synthetic_or_missing_fields) ... ok
test_scanner_heal_release_bundle_requires_domain_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_requires_domain_evidence) ... ok
test_scanner_heal_release_bundle_requires_field_provenance (__main__.SelfTests.test_scanner_heal_release_bundle_requires_field_provenance) ... ok
test_scanner_heal_release_bundle_requires_json_artifact_provenance (__main__.SelfTests.test_scanner_heal_release_bundle_requires_json_artifact_provenance) ... ok
test_scanner_heal_release_bundle_requires_mixed_version_field_roles (__main__.SelfTests.test_scanner_heal_release_bundle_requires_mixed_version_field_roles) ... ok
test_scanner_heal_release_bundle_requires_same_gate_measurement_window (__main__.SelfTests.test_scanner_heal_release_bundle_requires_same_gate_measurement_window) ... ok
test_scanner_heal_release_consumes_multiple_registry_oracles (__main__.SelfTests.test_scanner_heal_release_consumes_multiple_registry_oracles) ... ok
test_scanner_heal_release_matrix_lanes_remain_pending_without_real_evidence (__main__.SelfTests.test_scanner_heal_release_matrix_lanes_remain_pending_without_real_evidence) ... ok
test_scanner_heal_release_requirements_cannot_be_cleared (__main__.SelfTests.test_scanner_heal_release_requirements_cannot_be_cleared) ... ok
test_scanner_heal_release_status_rejects_focused_case_run (__main__.SelfTests.test_scanner_heal_release_status_rejects_focused_case_run) ... ok
test_scanner_heal_release_status_rejects_synthetic_case (__main__.SelfTests.test_scanner_heal_release_status_rejects_synthetic_case) ... ok
test_scanner_heal_required_evidence_fields_cannot_be_removed (__main__.SelfTests.test_scanner_heal_required_evidence_fields_cannot_be_removed) ... ok
test_scheduled_alerts_require_completion_watchdog (__main__.SelfTests.test_scheduled_alerts_require_completion_watchdog) ... ok
test_update_profile_selection_changes_only_requested_platform (__main__.SelfTests.test_update_profile_selection_changes_only_requested_platform) ... ok
test_validate_includes_quick_checks (__main__.SelfTests.test_validate_includes_quick_checks) ... ok
test_vault_tests_require_cross_process_serialization (__main__.SelfTests.test_vault_tests_require_cross_process_serialization) ... ok
test_workflow_readiness_requires_dependency_probe (__main__.SelfTests.test_workflow_readiness_requires_dependency_probe) ... ok

----------------------------------------------------------------------
Ran 52 tests in 14.301s

OK
e2e-full selection OK: 1 tests, sha256=29e30e6ed2ed38d9e5b0a7cdd9536abf5c719f7021a02b578ae1381563a3c273
...................................
----------------------------------------------------------------------
Ran 35 tests in 1.161s

OK
...................
----------------------------------------------------------------------
Ran 19 tests in 17.397s

OK
..........
----------------------------------------------------------------------
Ran 10 tests in 0.183s

OK
test_actual_selector_bootstrap_uses_base_policy_and_fails_closed (__main__.SelfTests.test_actual_selector_bootstrap_uses_base_policy_and_fails_closed) ... ok
test_documentation_paths_do_not_hide_build_or_fixture_changes (__main__.SelfTests.test_documentation_paths_do_not_hide_build_or_fixture_changes) ... ok
test_e2e_scope_keeps_production_and_shared_configuration_full (__main__.SelfTests.test_e2e_scope_keeps_production_and_shared_configuration_full) ... ok
test_e2e_shortcut_rejects_new_direct_renamed_and_target_dependencies (__main__.SelfTests.test_e2e_shortcut_rejects_new_direct_renamed_and_target_dependencies) ... ok
test_e2e_shortcut_traverses_implicit_path_dependencies (__main__.SelfTests.test_e2e_shortcut_traverses_implicit_path_dependencies) ... ok
test_event_contract_requires_complete_candidate_and_optional_lanes (__main__.SelfTests.test_event_contract_requires_complete_candidate_and_optional_lanes) ... ok
test_every_wrong_result_missing_job_or_selection_fails_closed (__main__.SelfTests.test_every_wrong_result_missing_job_or_selection_fails_closed) ... ok
test_full_e2e_gate_preserves_workflow_branch_and_event_scope (__main__.SelfTests.test_full_e2e_gate_preserves_workflow_branch_and_event_scope) ... ok
test_git_range_includes_deleted_source_and_rename_origins (__main__.SelfTests.test_git_range_includes_deleted_source_and_rename_origins) ... ok
test_job_ids_and_display_names_cannot_hide_validation (__main__.SelfTests.test_job_ids_and_display_names_cannot_hide_validation) ... ok
test_report_only_stack_lane_mirrors_e2e_smoke_outside_the_aggregate (__main__.SelfTests.test_report_only_stack_lane_mirrors_e2e_smoke_outside_the_aggregate) ... ok
test_repository_wiring_and_missing_dependency_regression (__main__.SelfTests.test_repository_wiring_and_missing_dependency_regression) ... ok
test_verify_command_preserves_failures (__main__.SelfTests.test_verify_command_preserves_failures) ... ok

----------------------------------------------------------------------
Ran 13 tests in 3.280s

OK
security coverage self-test passed
test_api_failures_preserve_other_evidence_and_never_enter_grace (__main__.SelfTests.test_api_failures_preserve_other_evidence_and_never_enter_grace) ... ok
test_attempt_outcome_does_not_replace_recent_success (__main__.SelfTests.test_attempt_outcome_does_not_replace_recent_success) ... ok
test_cancelled_attempt_cannot_refresh_expired_success (__main__.SelfTests.test_cancelled_attempt_cannot_refresh_expired_success) ... ok
test_cli_requires_the_repository_default_branch (__main__.SelfTests.test_cli_requires_the_repository_default_branch) ... ok
test_config_rejects_duplicate_and_invalid_entries (__main__.SelfTests.test_config_rejects_duplicate_and_invalid_entries) ... ok
test_freshness_boundaries (__main__.SelfTests.test_freshness_boundaries) ... ok
test_grace_requires_two_successful_queries_with_no_history (__main__.SelfTests.test_grace_requires_two_successful_queries_with_no_history) ... ok
test_invalid_api_evidence_fails_closed (__main__.SelfTests.test_invalid_api_evidence_fails_closed) ... ok
test_report_retains_every_workflow (__main__.SelfTests.test_report_retains_every_workflow) ... ok
test_requests_filter_schedule_default_branch_and_success_on_server (__main__.SelfTests.test_requests_filter_schedule_default_branch_and_success_on_server) ... ok

----------------------------------------------------------------------
Ran 10 tests in 0.051s

OK
..............................
----------------------------------------------------------------------
Ran 30 tests in 22.354s

OK
..........
----------------------------------------------------------------------
Ran 10 tests in 2.969s

OK
.................................
----------------------------------------------------------------------
Ran 33 tests in 0.160s

OK
Online functional runner labels: pf-testing
.......
----------------------------------------------------------------------
Ran 7 tests in 0.002s

OK
.......
----------------------------------------------------------------------
Ran 7 tests in 0.001s

OK
....
----------------------------------------------------------------------
Ran 4 tests in 0.128s

OK
...
----------------------------------------------------------------------
Ran 3 tests in 0.547s

OK
PASS: cold stampede script self-test
exit_code=0

$ python3 ./scripts/test_object_generation_protocol_model.py
test_a_recovery_barrier_cannot_choose_an_empty_value (__main__.ObjectGenerationProtocolModel.test_a_recovery_barrier_cannot_choose_an_empty_value) ... ok
test_all_bounded_two_proposer_interleavings_choose_at_most_one_value (__main__.ObjectGenerationProtocolModel.test_all_bounded_two_proposer_interleavings_choose_at_most_one_value) ... ok
test_an_unavailable_decision_quorum_cannot_choose_a_candidate (__main__.ObjectGenerationProtocolModel.test_an_unavailable_decision_quorum_cannot_choose_a_candidate) ... ok
test_counter_allocation_requires_persistence_and_is_consumed_once (__main__.ObjectGenerationProtocolModel.test_counter_allocation_requires_persistence_and_is_consumed_once) ... ok
test_counter_restart_burns_persisted_allocations_before_or_after_issue (__main__.ObjectGenerationProtocolModel.test_counter_restart_burns_persisted_allocations_before_or_after_issue) ... ok
test_crash_after_promise_before_reply_does_not_restore_an_older_ballot (__main__.ObjectGenerationProtocolModel.test_crash_after_promise_before_reply_does_not_restore_an_older_ballot) ... ok
test_duplicate_or_unconfigured_voters_cannot_form_a_quorum (__main__.ObjectGenerationProtocolModel.test_duplicate_or_unconfigured_voters_cannot_form_a_quorum) ... ok
test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision (__main__.ObjectGenerationProtocolModel.test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision) ... ok
test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected (__main__.ObjectGenerationProtocolModel.test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected) ... ok
test_forgetting_tombstone_authority_allows_an_aba_create (__main__.ObjectGenerationProtocolModel.test_forgetting_tombstone_authority_allows_an_aba_create) ... ok
test_issuing_before_persistence_reuses_a_ballot_after_restart (__main__.ObjectGenerationProtocolModel.test_issuing_before_persistence_reuses_a_ballot_after_restart) ... ok
test_lost_ack_and_restart_preserve_the_chosen_operation (__main__.ObjectGenerationProtocolModel.test_lost_ack_and_restart_preserve_the_chosen_operation) ... ok
test_minority_accepted_work_is_adopted_and_finished (__main__.ObjectGenerationProtocolModel.test_minority_accepted_work_is_adopted_and_finished) ... ok
test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample (__main__.ObjectGenerationProtocolModel.test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample) ... ok
test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it (__main__.ObjectGenerationProtocolModel.test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it) ... ok
test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor (__main__.ObjectGenerationProtocolModel.test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor) ... ok
test_replacing_voters_with_empty_state_can_choose_a_conflicting_value (__main__.ObjectGenerationProtocolModel.test_replacing_voters_with_empty_state_can_choose_a_conflicting_value) ... ok
test_skipping_predecessor_recovery_chooses_a_broken_revision_chain (__main__.ObjectGenerationProtocolModel.test_skipping_predecessor_recovery_chooses_a_broken_revision_chain) ... ok
test_stale_publication_rollback_and_gc_cannot_change_the_winner (__main__.ObjectGenerationProtocolModel.test_stale_publication_rollback_and_gc_cannot_change_the_winner) ... ok
test_superseded_operation_identity_cannot_be_reused_for_different_input (__main__.ObjectGenerationProtocolModel.test_superseded_operation_identity_cannot_be_reused_for_different_input) ... ok
test_superseded_operation_retry_returns_its_original_successor (__main__.ObjectGenerationProtocolModel.test_superseded_operation_retry_returns_its_original_successor) ... ok
test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision (__main__.ObjectGenerationProtocolModel.test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision) ... ok
test_tombstone_rejects_a_delayed_never_present_create (__main__.ObjectGenerationProtocolModel.test_tombstone_rejects_a_delayed_never_present_create) ... ok
test_unlearned_predecessor_is_recovered_before_the_second_slot (__main__.ObjectGenerationProtocolModel.test_unlearned_predecessor_is_recovered_before_the_second_slot) ... ok

----------------------------------------------------------------------
Ran 24 tests in 0.958s

OK
model: states=51023, transitions=167520, terminal=732, chosen_candidates=2
negative control: double-choice trace=((0, 'prepare', 0), (0, 'prepare', 1), (0, 'prepare', 2), (0, 'accept', 0), (0, 'accept', 1), (0, 'accept', 2), (1, 'prepare', 0), (1, 'prepare', 1), (1, 'prepare', 2), (1, 'accept', 0), (1, 'accept', 1), (1, 'accept', 2))
multi-slot round schedules: 640
exit_code=0

$ python3 ./scripts/check_test_wiring.py
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired
exit_code=0

$ ./scripts/check_no_planning_docs.sh
exit_code=0

$ ./scripts/check_uring_lane_lib_only.sh
OK: no 'uring_'-matching test functions under crates/ecstore/tests; --lib is safe
exit_code=0

Initial complete gate
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
 Nextest run ID e07fc5b9-c6c8-4cdd-ae61-54839bf24437 with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.016s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.011s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.009s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.009s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  16.950s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   4.025s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.008s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.007s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.035s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  21.087s] 10 tests run: 10 passed, 0 skipped
exit_code=0

Restored complete gate
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
 Nextest run ID 233b7307-5361-4eb7-94d5-b384f17d2d6a with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.009s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.008s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.008s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.008s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  19.518s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   4.146s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.009s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.018s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.009s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.054s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  23.787s] 10 tests run: 10 passed, 0 skipped
exit_code=0

Mutation M1: success_measurement reports max(actual elapsed, requested budget + 1), forcing 1001 ms for the one-second request. The retained duration assertion at rustfs/tests/connect_perf_drive.rs:105 fails.
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast -E 'test(real_local_write_read_is_measured_and_scratch_is_removed)'
info: experimental features enabled: setup-scripts
   Compiling rustfs v1.0.1 (/private/tmp/gw-orchestrator-2734/ci-drive/rustfs)
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs") generated 1 warning
warning: `rustfs` (bin "rustfs-cli") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 7m 06s
────────────
 Nextest run ID 9f618777-7bf6-4cbc-8803-dd7f50f449ed with nextest profile: ci
    Starting 1 test across 1 binary (9 tests skipped)
        FAIL [   0.016s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
  stdout ───

    running 1 test
    test real_local_write_read_is_measured_and_scratch_is_removed ... FAILED

    failures:

    failures:
        real_local_write_read_is_measured_and_scratch_is_removed

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 0.01s

  stderr ───

    thread 'real_local_write_read_is_measured_and_scratch_is_removed' (106986846) panicked at rustfs/tests/connect_perf_drive.rs:105:5:
    assertion failed: (1..=1_000).contains(&data.duration_millis)
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

────────────
     Summary [   0.017s] 1 test run: 0 passed, 1 failed, 9 skipped
        FAIL [   0.016s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
error: test run failed
exit_code=100

Mutation M2: target_parameters reports requested duration + 1. The in-process assertion at rustfs/tests/connect_perf_drive.rs:111 and CLI JSON assertion at rustfs/tests/connect_perf_drive.rs:527 both reject 1001 against 1000.
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast -E 'test(real_local_write_read_is_measured_and_scratch_is_removed) | test(production_cli_measures_and_signs_exact_binary_provenance)'
info: experimental features enabled: setup-scripts
   Compiling rustfs v1.0.1 (/private/tmp/gw-orchestrator-2734/ci-drive/rustfs)
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs") generated 1 warning
warning: `rustfs` (bin "rustfs-cli") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 7m 20s
────────────
 Nextest run ID b73d46ab-c910-4e1e-8063-7fc7e332ec15 with nextest profile: ci
    Starting 2 tests across 1 binary (8 tests skipped)
        FAIL [  13.968s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
  stdout ───

    running 1 test
    test production_cli_measures_and_signs_exact_binary_provenance ... FAILED

    failures:

    failures:
        production_cli_measures_and_signs_exact_binary_provenance

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 13.96s

  stderr ───

    thread 'production_cli_measures_and_signs_exact_binary_provenance' (107015462) panicked at rustfs/tests/connect_perf_drive.rs:527:5:
    assertion `left == right` failed
      left: Number(1001)
     right: 1000
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

        FAIL [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
  stdout ───

    running 1 test
    test real_local_write_read_is_measured_and_scratch_is_removed ... FAILED

    failures:

    failures:
        real_local_write_read_is_measured_and_scratch_is_removed

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 0.01s

  stderr ───

    thread 'real_local_write_read_is_measured_and_scratch_is_removed' (107016667) panicked at rustfs/tests/connect_perf_drive.rs:111:5:
    assertion `left == right` failed
      left: 1001
     right: 1000
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

────────────
     Summary [  13.985s] 2 tests run: 0 passed, 2 failed, 8 skipped
        FAIL [  13.968s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        FAIL [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
error: test run failed
exit_code=100

Transient production mutations were restored byte-for-byte before the complete green rerun. Original/restored SHA256: 8d389e6d8836c041aab8c6a93e56addd65ba3f70b33264e1c74f639d0572a2da.

$ git diff --check

exit_code=0

$ git diff --cached --check

exit_code=0

Independent orchestrator acceptance
Tested code tree: e67c72fb01

Tested tree: 16fbbd2ea4
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

$ LANG=en_US.UTF-8 CARGO_BUILD_JOBS=1 CARGO_INCREMENTAL=0 cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
info: experimental features enabled: setup-scripts
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs-cli") generated 1 warning
warning: `rustfs` (bin "rustfs") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.69s
────────────
 Nextest run ID cd1edf4c-f1c0-456d-ad1a-76cc00ce05c9 with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.012s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.010s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.010s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.009s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  20.328s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   5.527s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.013s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.019s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.011s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.049s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  25.991s] 10 tests run: 10 passed, 0 skipped

exit_code=0

All 54 tracked Cargo.toml files parse without duplicate keys. git diff --check passed. The production source remains unchanged; the two deliberate implementation mutations above were restored before this complete green run.

* fix(test): compare legacy body wire parity as raw bytes

Lossy UTF-8 comparison concealed a changed binary response byte.
Compare the already captured Vec<u8> values directly.

Validation:
- Binary mutant: migrated payload 0x84 vs legacy 0x83; old comparator
  passed 1/1, raw comparator failed 0/1 (exit 100); restored green.
- Non-Swift body-drain mutant: untouched-body delegation test failed
  with empty vs payload (exit 100); restored handler without a diff.
- cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols
  --features rustfs-protocols/swift --no-fail-fast:
  543 tests run, 543 passed, 0 skipped after both mutants restored.
- cargo fmt --all --check and git diff --check passed.

Orchestrator acceptance (LANG=en_US.UTF-8):
cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
Summary [2.445s] 543 tests run: 543 passed, 0 skipped.

All shared quick checks on the final code tree:
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

Final independent acceptance after the drive-test CI repair
Parent: a99d8f4f89
Tested code tree: 93cbd96bf0

Tested tree: 075ae5161c2b62a96812bc1e3c904a7663b4f2f8
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

$ LANG=en_US.UTF-8 CARGO_BUILD_JOBS=1 CARGO_INCREMENTAL=0 cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
    Finished `test` profile [unoptimized + debuginfo] target(s) in 10m 08s
 Nextest run ID 5c7ff102-ec6d-479a-86e0-810859b28bbf with nextest profile: ci
    Starting 543 tests across 8 binaries
     Summary [   6.729s] 543 tests run: 543 passed, 0 skipped
exit_code=0

The only source change is the raw-byte wire comparator. Cargo manifests and the ratchet baselines remain unchanged from the independently validated parent. The invalid-byte and consumed-request mutations remain recorded above; their implementation changes were restored.

* fix: repair integration checks and IPC test readiness

Wait for the RPC/locks test sockets to pass the existing owner and mode
checks inside the original 5s readiness window and 10ms poll interval.

Use the existing admin S3 carrier for the new recovery and quota error
calls. Regenerate the test-count baseline for main's 118 existing tests
across seven crates. Pair the private pagination helper's raw-scan inputs
and remove the redundant Copy clone from the retirement event.

Keep the original assertions, responses, authorization, locks and storage
operations. The pagination body and raw-scan expressions at all five calls
are byte-identical; reinserting only the Copy clone restores the entire
base heal file. All ignored baseline values and s3s counts stay unchanged.

Validation base: c47432e353
Validation tree: f081084bc5
Patch SHA-256: 2dcf0028691bfb2772edfb38e262829b04f4151832dcce823c9dd9f7434b52de
Trace production-prefix SHA-256: fd994f218a21429d62866f4efb8eee1ee57b58f4a51865aa35d05a54ef1ccb45

All commands used LANG=en_US.UTF-8, CARGO_BUILD_JOBS=1,
CARGO_INCREMENTAL=0 and CARGO_NET_OFFLINE=true in one exclusive target.
Every gate retained the same tree and six source hashes before and after.
54 Cargo.toml files parse without duplicate keys.

cargo clippy -p rustfs --all-targets -- -D warnings
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 12m 50s
Exit: 0. Wall time: 770.855s.

The original 19 quick-check steps:
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
Wall time: 286.337s.

cargo nextest run --profile ci -p rustfs --no-fail-fast
     Summary [ 217.017s] 5382 tests run: 5382 passed (4 leaky), 40 skipped
Exit: 0. Wall time: 1324.414s.

cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
     Summary [   2.189s] 543 tests run: 543 passed, 0 skipped
Exit: 0. Wall time: 123.192s.

cargo nextest run --profile ci -p rustfs-ecstore list_objects_paginate --no-fail-fast
     Summary [   0.170s] 8 tests run: 8 passed, 5921 skipped
Exit: 0. Wall time: 261.324s.

cargo nextest run --profile ci -p rustfs-heal durable_mrf_anchor_retires_only_after_exact_bucket_generation_is_deleted --no-fail-fast
     Summary [   0.805s] 1 test run: 1 passed, 835 skipped
Exit: 0. Wall time: 199.801s.

Baseline regeneration:
./scripts/check_test_count_ratchet.sh --update
wrote .config/test-count-baseline.txt: 52 crates, 21340 tests, 164 ignored
Test-only increases: ecstore48, heal3, iam14, io-metrics2, obs3, policy14,
rustfs34. Existing ignored values and all other rows are byte-identical.
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)

Historical runtime red controls on f622 (same retained two assertions):
cargo nextest run --profile ci -p rustfs --lib -E 'test(=connect::diagnostics::trace_runtime::tests::local_top_rpc_reads_service_http_completions_and_signs_offline) | test(=connect::diagnostics::trace_runtime::tests::local_top_locks_reads_service_manager_and_signs_offline)' --no-fail-fast
Original path readiness with a temporary 500ms bind/seal gap:
     Summary [   0.863s] 2 tests run: 0 passed, 2 failed, 5113 skipped
Exit: 100.
Sealed readiness in that same gap:
     Summary [   5.621s] 2 tests run: 2 passed, 5113 skipped
Exit: 0.
Compiled M1: replace the helper predicate with path.exists():
     Summary [   0.870s] 2 tests run: 0 passed, 2 failed, 5113 skipped
Exit: 100.
Restore the sealed predicate byte-for-byte:
     Summary [   5.640s] 2 tests run: 2 passed, 5113 skipped
Exit: 0.
Both reds observe mode755 with owner_matches=true and fail at the original
RPC/locks unwraps with StateSecurity. M1 failure locations: trace_runtime.rs
2547:29 and 2713:29. No new assertions were introduced.

cargo nextest run --profile ci -p rustfs --lib -E 'test(connect::diagnostics::trace_runtime::tests::)' --no-fail-fast
     Summary [  20.748s] 28 tests run: 28 passed (1 leaky), 5087 skipped
Exit: 0. This security sweep is historical; the full final unit gate above
covers the actual c474 candidate.
The first disk-stopped compile ran no tests and is excluded from red credit.
All temporary seal instrumentation is removed from the final source.

Final unit cleanup markers: 4 leaky, 40 original skips (exit0). The ld
__eh_frame linker warning also appears verbatim in the original f622 run
and the earlier c474 run; there are no new source warnings.

* fix: remove redundant pending closure in disk test

Pass the capture-free pending future factory as a function item in the existing slow-operation test. This fixes the redundant_closure error at disk_store.rs:2914 that stopped strict Clippy before the feature tests could run.

The single-line test edit preserves the factory's result, warning and timeout behavior. Reversing the replacement reproduces the parent file byte for byte. All 220 assertion lines and the production prefix are unchanged; no tests, assertions, guards, profiles or baselines were added or relaxed.

Original Linux CI evidence (run 37888552268 at 53245be9f):
- RIO job113685483992: strict Clippy exit101, redundant closure at disk_store.rs:2914; feature tests skipped.
- Workspace Test and Lint job113685484058: same strict Clippy exit101; nextest and doc tests skipped.
These are lint failures, not runtime mutation kills or known flakes.

Native macOS arm64 validation on tree7a5bf47e7533b29479bc85207416102f1337c566, LANG=en_US.UTF-8, one build job, CARGO_INCREMENTAL=0 and offline dependencies:

shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
All 19 checks: exit0, 312.161s.

$ cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 20m 11s
exit0, 1213.210s total.

$ cargo nextest run --profile ci -p rustfs-ecstore --features rio-v2 slow_disk_operation_warns_once_while_pending_and_preserves_result --no-fail-fast
Finished `test` profile [unoptimized + debuginfo] target(s) in 4m 34s
PASS [0.026s] rustfs-ecstore disk::disk_store::tests::slow_disk_operation_warns_once_while_pending_and_preserves_result
Summary [   0.170s] 1 test run: 1 passed, 5943 skipped
exit0, 281.920s total; 5943 existing filter skips.

warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
This linker warning is retained; flags, profiles and allowances are unchanged.

54 tracked Cargo.toml files parsed without duplicate keys. git diff --check passed. No new assertions were introduced, so no new runtime mutation is claimed. Native compilation and the existing helper test do not substitute for actual Linux RIO execution; the new integration CI run still needs to recover before the RPC merge.

Preservation and output hashes:
parent file SHA256: 8e59ee745e3aaaa22848b943d3b10f6353b515956282cde1f3af2c394200c9ac
final file SHA256: b69ba3237061b161448ae8b62b615885d5770b98a5b4a2479bcfbd70f1bd1f17
unchanged production prefix SHA256: 7903a6e288252c420231242d7b58a25fa2a7140ea3dd430cbe7f0dde11369dcb
RIO Linux failure log SHA256:421addb9868fd88e4109a922311e54f91f0dfd70862f8ba7fcaba193482ed90b
Workspace Linux failure log SHA256:e0aea8d0f014c022252a9534968e083c98cfc1fd760da445ecd04ec3f522e26a
19 output SHA256:35a54f28d707956ed1edb29d492c4dfe8a9fe753d103b111b5a2f1b5c41c2b98
RIO Clippy output SHA256:d6046e3868f2b3992fe11cdd04e9e98eb50a87d1234d9a70dbfeb154ff61638d
slow-operation output SHA256:605b7b258d2fd09c6ef2ab2ffc90d433d0509480f1e45125d20c823759e83608

Refs rustfs/backlog#2734.

* test(ecstore): isolate tier-free recovery fixture startup

The fixture constructor can take the production recovery lock and wait for 60 seconds before this test installs its queue and seeds metadata. Disable its background recovery only while creating this fixture, then restore the environment before calling the existing production recovery entrypoint.

Only the existing test setup changes. Its production entrypoint, default lock parameters, original assertions, 30-second timeout, cancellation and cleanup remain intact. No tests or assertions were added. Production and store/init.rs are byte-identical to the parent.

Independent acceptance ran on merge tree 8ec358cb42ba9ebb3bcfeafcef048a3f0366355b; the runner header reports parent HEAD, while before/after index and source hashes confirm this candidate. Worker task branch d8055766b8 has the same tree.

Every command used LANG=en_US.UTF-8. Task compilation also used CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0, CARGO_NET_OFFLINE=true, an independent target, and empty RUSTC_WRAPPER/RUSTC_WORKSPACE_WRAPPER to keep compilers in its owned process group without changing repository configuration.

Command:
shellcheck --version && actionlint
Exit 0; observed output excerpt:
version: 0.11.0
Log SHA-256: fc9eec16b4191d058a295813ca9afbe9d2b37b59975da860170726be162a3924

Command:
cargo fmt --all --check
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_unsafe_code_allowances.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_layer_dependencies.sh
Exit 0; observed output excerpt:
Layer dependency guard passed (no new reverse dependencies/cycles).
Log SHA-256: 69c6cb1896c0586b28d1167380d3260d99d8017a4265c8df5a2348f289e93a12

Command:
./scripts/check_architecture_migration_rules.sh
Exit 0; observed output excerpt:
Architecture migration rules passed.
Log SHA-256: 28790e70d070607ed4de0704f292a65030be92ec224ef106d4b72fd8296f896e

Command:
./scripts/check_logging_guardrails.sh
Exit 0; observed output excerpt:
✅ Logging guardrails check passed
Log SHA-256: 7adf41abff8ed8f7617decf86498888ddde78937099dd9ce47f6d359dfc2d317

Command:
./scripts/check_error_other_format_ratchet.sh
Exit 0; observed output excerpt:
error(format!) ratchet passed (total call sites: 579).
Log SHA-256: 66172aef18f150ba0605af122e26ffcf91f6dd86f43e38b05e2e982ce8d783ee

Command:
./scripts/check_no_tokio_io_uring.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_extension_schema_boundaries.sh
Exit 0; observed output excerpt:
Extension schema boundaries passed.
Log SHA-256: f13d0acd0e2ce5d9815241b12b4dab7d01cc11b9b65bcf6dfb7812c878532a51

Command:
./scripts/check_body_cache_whitelist.sh
Exit 0; observed output excerpt:
Body-cache whitelist guard passed.
Log SHA-256: 6b828847b0383950ee6eef9cd846d0c322c945e47efd4c7d1ad881178810d6bb

Command:
./scripts/check_s3s_footprint.sh
Exit 0; observed output excerpt:
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)
✅ s3s footprint baseline check passed
Log SHA-256: 47f53d04ef645618563fe1f22abc64667f19078f1974feedcca46e4f2c016844

Command:
./scripts/check_test_count_ratchet.sh
Exit 0; observed output excerpt:
test-count ratchet OK: 52 crates, 21340 tests, 164 ignored match .config/test-count-baseline.txt
Log SHA-256: 618a5e330e933de0c6ff484546df401457024a88bb5986d11d588794e33dab60

Command:
./scripts/check_fips_wording.sh
Exit 0; observed output excerpt:
FIPS wording guard passed (README.md and CHANGELOG.md contain no forbidden claims).
KMS wording guard passed (crates/kms claims no Vault KV2 Transit wrapping).
Log SHA-256: 99221cc9979c480ec4f46f50d76c52b18c72fe499f12e15e4f4a22a1b8bfb140

Command:
./scripts/check_embedded_secrets.sh
Exit 0; observed output excerpt:
Embedded secret guard passed (no private key material or provider credential literal in the tree).
Log SHA-256: 437a3ceba32094d2901889965ad191e411aad32cdb18bf0b8659e818011c406b

Command:
make script-tests
Exit 0; observed output excerpt:
test_cli_requires_the_repository_default_branch (__main__.SelfTests.test_cli_requires_the_repository_default_branch) ... ok
test_config_rejects_duplicate_and_invalid_entries (__main__.SelfTests.test_config_rejects_duplicate_and_invalid_entries) ... ok
test_freshness_boundaries (__main__.SelfTests.test_freshness_boundaries) ... ok
test_grace_requires_two_successful_queries_with_no_history (__main__.SelfTests.test_grace_requires_two_successful_queries_with_no_history) ... ok
test_invalid_api_evidence_fails_closed (__main__.SelfTests.test_invalid_api_evidence_fails_closed) ... ok
test_report_retains_every_workflow (__main__.SelfTests.test_report_retains_every_workflow) ... ok
test_requests_filter_schedule_default_branch_and_success_on_server (__main__.SelfTests.test_requests_filter_schedule_default_branch_and_success_on_server) ... ok
Ran 10 tests in 0.042s
OK
Ran 30 tests in 20.627s
OK
Ran 10 tests in 2.268s
OK
Ran 33 tests in 0.127s
OK
Ran 7 tests in 0.002s
OK
Ran 7 tests in 0.001s
OK
Ran 4 tests in 0.108s
OK
Ran 3 tests in 0.564s
OK
PASS: cold stampede script self-test
Log SHA-256: 75fda2d069a373c63667b908baaf99e6591aab9099a739b0c4c22ceb6da8d68c

Command:
python3 ./scripts/test_object_generation_protocol_model.py
Exit 0; observed output excerpt:
test_an_unavailable_decision_quorum_cannot_choose_a_candidate (__main__.ObjectGenerationProtocolModel.test_an_unavailable_decision_quorum_cannot_choose_a_candidate) ... ok
test_counter_allocation_requires_persistence_and_is_consumed_once (__main__.ObjectGenerationProtocolModel.test_counter_allocation_requires_persistence_and_is_consumed_once) ... ok
test_counter_restart_burns_persisted_allocations_before_or_after_issue (__main__.ObjectGenerationProtocolModel.test_counter_restart_burns_persisted_allocations_before_or_after_issue) ... ok
test_crash_after_promise_before_reply_does_not_restore_an_older_ballot (__main__.ObjectGenerationProtocolModel.test_crash_after_promise_before_reply_does_not_restore_an_older_ballot) ... ok
test_duplicate_or_unconfigured_voters_cannot_form_a_quorum (__main__.ObjectGenerationProtocolModel.test_duplicate_or_unconfigured_voters_cannot_form_a_quorum) ... ok
test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision (__main__.ObjectGenerationProtocolModel.test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision) ... ok
test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected (__main__.ObjectGenerationProtocolModel.test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected) ... ok
test_forgetting_tombstone_authority_allows_an_aba_create (__main__.ObjectGenerationProtocolModel.test_forgetting_tombstone_authority_allows_an_aba_create) ... ok
test_issuing_before_persistence_reuses_a_ballot_after_restart (__main__.ObjectGenerationProtocolModel.test_issuing_before_persistence_reuses_a_ballot_after_restart) ... ok
test_lost_ack_and_restart_preserve_the_chosen_operation (__main__.ObjectGenerationProtocolModel.test_lost_ack_and_restart_preserve_the_chosen_operation) ... ok
test_minority_accepted_work_is_adopted_and_finished (__main__.ObjectGenerationProtocolModel.test_minority_accepted_work_is_adopted_and_finished) ... ok
test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample (__main__.ObjectGenerationProtocolModel.test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample) ... ok
test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it (__main__.ObjectGenerationProtocolModel.test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it) ... ok
test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor (__main__.ObjectGenerationProtocolModel.test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor) ... ok
test_replacing_voters_with_empty_state_can_choose_a_conflicting_value (__main__.ObjectGenerationProtocolModel.test_replacing_voters_with_empty_state_can_choose_a_conflicting_value) ... ok
test_skipping_predecessor_recovery_chooses_a_broken_revision_chain (__main__.ObjectGenerationProtocolModel.test_skipping_predecessor_recovery_chooses_a_broken_revision_chain) ... ok
test_stale_publication_rollback_and_gc_cannot_change_the_winner (__main__.ObjectGenerationProtocolModel.test_stale_publication_rollback_and_gc_cannot_change_the_winner) ... ok
test_superseded_operation_identity_cannot_be_reused_for_different_input (__main__.ObjectGenerationProtocolModel.test_superseded_operation_identity_cannot_be_reused_for_different_input) ... ok
test_superseded_operation_retry_returns_its_original_successor (__main__.ObjectGenerationProtocolModel.test_superseded_operation_retry_returns_its_original_successor) ... ok
test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision (__main__.ObjectGenerationProtocolModel.test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision) ... ok
test_tombstone_rejects_a_delayed_never_present_create (__main__.ObjectGenerationProtocolModel.test_tombstone_rejects_a_delayed_never_present_create) ... ok
test_unlearned_predecessor_is_recovered_before_the_second_slot (__main__.ObjectGenerationProtocolModel.test_unlearned_predecessor_is_recovered_before_the_second_slot) ... ok
Ran 24 tests in 0.929s
OK
Log SHA-256: 0b4be926efb0609d497ae58930340153f01957b606206cbbdb3e0d47eb29d3a2

Command:
python3 ./scripts/check_test_wiring.py
Exit 0; observed output excerpt:
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired
Log SHA-256: 21f676f458451525535ab0ef73d6a9c0d9774e16d76a4d19e396ad34936dd39c

Command:
./scripts/check_no_planning_docs.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_uring_lane_lib_only.sh
Exit 0; observed output excerpt:
OK: no 'uring_'-matching test functions under crates/ecstore/tests; --lib is safe
Log SHA-256: 43f3f851e27ec2ab9e34d6b05fefdede38af992699c473904bfe6f53a46d52e3

Command:
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast
Exit 0; wall 234.399s.
Observed output:
    Finished `test` profile [unoptimized + debuginfo] target(s) in 3m 49s
    Starting 1 test across 1 binary (5871 tests skipped)
  SETUP PASS [   0.006s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"'
        PASS [   0.440s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item
     Summary [   0.586s] 1 test run: 1 passed, 5871 skipped
Log SHA-256: 34e973f86e8dc2102c1a0e483e0ac802ce7df3d1b40a603932f62c0805c4de27

Command:
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over) | test(=store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence) | test(=store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable)' --no-fail-fast
Exit 0; wall 4.010s.
Observed output:
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.43s
    Starting 3 tests across 1 binary (5869 tests skipped)
  SETUP PASS [   0.005s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"'
        PASS [   0.068s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over
        PASS [   0.637s] rustfs-ecstore store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence
        PASS [   0.846s] rustfs-ecstore store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable
     Summary [   0.987s] 3 tests run: 3 passed, 5869 skipped
Log SHA-256: b5da2a927ff18ca9aa217762f7186a6d4371bd9e8feedd7b77940e10b64dffac

Command:
cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
Exit 0; wall 1084.508s.
Observed output:
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 18m 03s
Log SHA-256: 28f98be7970faaa71cd118da1da4e6e2e3d23b9dda7e6449eaa7a37e741260b5

Compiled mutation command (unchanged for each red and restored run):
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast

M1: reverse only this test-local constructor flag from false to true. The mutant compiled, then failed the original recovered-item queue assertion after 30.481s. Restoring the entire observed candidate passed in 0.460s. Red log 55bf3b1535529ac0bcfb8071f9b56f9f40631007a85c315367e7e93786d65293; restored log 204d5fbe4574a7917f5e740a795d864fe28a7e7fb5e33d27b0a9cf68e2f17f52.

M2: replace only the actual awaited enqueue with false, consuming its original inputs. The mutant compiled, then failed the same original queue assertion after 30.516s. Restoring the entire observed candidate passed in 0.457s. Red log 89186330413215e231e2b06bf013cc31097ea8f4a106d91421a36d6c8986a7fd; restored log e848b42d2302401afa5dbcd9e37bf76d2e56a96ab6de5a488513b997f53c5e9b.

All temporary observation and mutation code was removed before final acceptance. Historical baseline and diagnostic failures remain recorded; they are not mutation credit. The separate Object Lock intent-retry diagnostic still fails, and these native macOS checks do not establish Linux CI recovery.

* refactor(rpc): serve internode rpc bodies with the owned Body (#8441)

* test(rpc): pin the wire bytes of every internode rpc response body

rustfs/backlog#2747 batch 4, step 1 of 2, on the s3s bodies the rpc paths
serve today. Each response body builder in storage/rpc/http_service.rs is
served through hyper's HTTP/1.1 server and the raw bytes the peer reads are
compared with goldens captured from this commit: status, content-length or
chunked framing, chunk boundaries, the terminating chunk, HEAD, and the cut
connection a failure after data produces. The capability responses carry a
per-process proof, so their length header and payload are checked against
the body they encode instead.

The read_file_stream response builder moves into read_file_stream_response
so the test reaches the same code the handler runs; behaviour is unchanged.

* refactor(rpc): serve internode rpc bodies with the owned Body

rustfs/backlog#2747 batch 4, step 2 of 2. The internode rpc HTTP paths
build their responses with rustfs_s3_types::Body: Body::from_stream
replaces Body::from(StreamingBlob::wrap(..)) at the six stream sites, and
the in-memory and empty bodies keep their constructors. InternodeRpcService
converts its responses into the wrapped service's response body through
From<Body>, so the server keeps composing it with the s3s service as
before. The wire-byte goldens from the previous commit pass unchanged;
storage/rpc no longer names s3s.

* fix(rpc): simplify wire test response builder type

Name the boxed response builder used by the capability wire test so
all-target Clippy can check it without a type-complexity warning.

Validation (LANG=en_US.UTF-8, CARGO_INCREMENTAL=0):
- cargo clippy -p rustfs --all-targets -- -D warnings: exit 0.
- cargo nextest run --profile ci -p rustfs --lib -E 'test(/storage::rpc::/)': 279 tests run, 279 passed, 4842 skipped.
- All 19 quick-check commands, including script-tests, both Python scripts and cargo fmt --all --check: exit 0.
- Footprint baseline regenerated: files=194, s3_error_lines=1584, ecstore_files=36; test-count baseline regenerated: 52 crates, 21228 tests, 164 ignored.
- All 54 Cargo.toml files parse without duplicate keys.

No assertion was added or changed by this lint fix. R1-R8 are retained
in the existing PR verification record.

* fix(ecstore): resume fenced Object Lock creation intents

* fix(ecstore): bound creation retry futures on the stack

* fix(ecstore): require native proof for creation retries

* fix(ecstore): bound bucket creation stack usage (#8466)

* fix(ecstore): retain creation fences through rollback

Keep lifecycle, metadata-transaction, and namespace guards around rollback and metadata cleanup. Preserve the original publication error when a guard is lost. Build the timestamp retry fixture through the real interrupted producer so it checks a valid pending creation.

Validated source tree: 4e8e207999.
Two independent source reviews found no blocking regression.

Validation (LANG=en_US.UTF-8):
- bash -c 'shellcheck --version && actionlint' — exit 0
- cargo fmt --all --check — exit 0
- ./scripts/check_unsafe_code_allowances.sh — exit 0
- ./scripts/check_layer_dependencies.sh — exit 0
- ./scripts/check_architecture_migration_rules.sh — exit 0
- ./scripts/check_logging_guardrails.sh — exit 0
- ./scripts/check_error_other_format_ratchet.sh — exit 0
- ./scripts/check_no_tokio_io_uring.sh — exit 0
- ./scripts/check_extension_schema_boundaries.sh — exit 0
- ./scripts/check_body_cache_whitelist.sh — exit 0
- ./scripts/check_s3s_footprint.sh — exit 0
- ./scripts/check_test_count_ratchet.sh — exit 0
- ./scripts/check_fips_wording.sh — exit 0
- ./scripts/check_embedded_secrets.sh — exit 0
- make script-tests — exit 0
- python3 ./scripts/test_object_generation_protocol_model.py — exit 0
- python3 ./scripts/check_test_wiring.py — exit 0
- ./scripts/check_no_planning_docs.sh — exit 0
- ./scripts/check_uring_lane_lib_only.sh — exit 0
- cargo nextest run --profile ci -p rustfs-ecstore --lib --locked --no-fail-fast — 5894 passed, 0 skipped
- cargo check -p rustfs-ecstore --all-targets --locked — exit 0
- cargo nextest run --profile ci -p rustfs --lib --locked --no-fail-fast -E 'test(=admin::handlers::table_catalog::tests::cold_strong_registration_completes_while_other_bucket_backup_waits) | test(=admin::handlers::table_catalog::tests::commit_backend_borrowed_put_preserves_owner_authorization_and_precondition) | test(=admin::handlers::table_catalog::tests::disable_table_bucket_preserves_objects_and_can_be_reenabled) | test(=admin::handlers::table_catalog::tests::load_table_handler_negotiates_vended_credentials_without_breaking_metadata_only_callers) | test(=admin::handlers::table_catalog::tests::namespace_property_handler_updates_object_backed_catalog_and_maps_errors) | test(=app::object::get::tests::execute_get_object_resumes_from_relocated_pool_without_splicing_body) | test(=table_catalog::store::tests::catalog_writer_loss_prevents_recovery_storage_commit)' — 7 passed
- cargo nextest run --profile ci -p rustfs --lib --locked --no-fail-fast -E 'test(unsupported_remote_disk_)' — 2 passed

Additional fault-injection evidence remains incomplete. Timestamp M1 reached its original assertion; its restore run is being collected. M2–M5 and the three planned rollback faults have not run. An automated risk check interrupted the timestamp proof task; no assertions or validation settings were weakened.

* fix(ecstore): box nested bucket rollback state

Keep the guarded rollback future behind a heap allocation so bucket creation does not store its nested state inline. The rollback body, all three fence references and pending-proof revocation order are unchanged.

The same three original Linux tests overflowed on both e611 CI attempts. Local format, ecstore test compilation and all 19 quick-check steps pass; the original Linux feature lanes still need to validate this candidate.

* test(ecstore): isolate rollback guard loss fixtures

Run the three rollback metadata-cleanup cases in separate test processes so reused /proc/self/fd aliases from earlier cases cannot invalidate their creation witnesses. Keep the original fixture, guards, assertions and io_uring test selection, and verify that each child runs the intended case successfully.

Update the generated test-count baseline for the added macro arm.

Validation: cargo check -p rustfs-ecstore --tests; cargo fmt --all --check; all 19 quick-check commands, including make script-tests and the two Python checks. Linux io_uring acceptance and fault controls remain pending.

* refactor(replication): move the replication config onto gateway types (#8443)

* wip: T1.4 replication family

* refactor(replication): move the replication config onto gateway types

The replication configuration family leaves s3s: rustfs-replication no longer depends on s3s, and ECStore bucket metadata, the scanner and the rustfs app/admin layers hold rustfs_gateway_types::persistence::PersistedReplicationConfiguration. The persisted XML goes through the gateway persistence codec behind a new BucketConfigXml trait (the 39 byte goldens pass with unchanged fixtures); the legacy s3s edge converts once in storage/ecfs.rs through storage/s3_compat/replication.rs; the scanner data usage cache keeps its MessagePack bytes through a serde mirror pinned by bytes recorded from s3s 5761ddfe. rustfs-gateway-types 0.32.0 is pinned once (rev 772d3385). Refs rustfs/backlog#2745.

* test: use S3 serializer for list response tests

* fix(ci): restore integration test compilation

Gate the diagnostic test import with its feature, initialize the conditional
MetadataWriteStore fields explicitly, and decode the scanner replication fixture
through its existing cache boundary. Preserve the original fixture and assertions.

Validation was run on tree 544acb80c0 at 6003d4b8.
LANG=en_US.UTF-8; no source changes follow validation.

Quick checks: 19/19 passed.
- shellcheck --version && actionlint: exit 0
- cargo fmt --all --check: exit 0
- ./scripts/check_unsafe_code_allowances.sh: exit 0
- ./scripts/check_layer_dependencies.sh: exit 0
- ./scripts/check_architecture_migration_rules.sh: exit 0
- ./scripts/check_logging_guardrails.sh: exit 0
- ./scripts/check_error_other_format_ratchet.sh: exit 0
- ./scripts/check_no_tokio_io_uring.sh: exit 0
- ./scripts/check_extension_schema_boundaries.sh: exit 0
- ./scripts/check_body_cache_whitelist.sh: exit 0
- ./scripts/check_s3s_footprint.sh: exit 0
- ./scripts/check_test_count_ratchet.sh: exit 0
- ./scripts/check_fips_wording.sh: exit 0
- ./scripts/check_embedded_secrets.sh: exit 0
- make script-tests: exit 0
- python3 ./scripts/test_object_generation_protocol_model.py: exit 0
- python3 ./scripts/check_test_wiring.py: exit 0
- ./scripts/check_no_planning_docs.sh: exit 0
- ./scripts/check_uring_lane_lib_only.sh: exit 0

Darwin validation:
- cargo clippy --locked -p rustfs-ecstore -p rustfs-scanner --all-targets -- -D warnings: exit 0
- cargo nextest run --locked -p rustfs-ecstore -p rustfs-scanner --lib -E 'test(config::com::tests::save_config_preserves_retryable_system_volume_errors) | test(scanner_io::tests::replication_maintenance_refreshes_compacted_stats_across_cycles_and_retries)' --test-threads 1 --no-fail-fast: exit 0, 2/2 passed
- cargo clippy --locked -p rustfs-ecstore -p rustfs-scanner --all-targets --features rustfs-ecstore/diagnostic-read-version-caller,rustfs-scanner/diagnostic-read-version-caller -- -D warnings: exit 101
- cargo nextest run --locked -p rustfs-ecstore -p rustfs-scanner --lib --features rustfs-ecstore/diagnostic-read-version-caller,rustfs-scanner/diagnostic-read-version-caller -E 'test(config::com::tests::save_config_preserves_retryable_system_volume_errors) | test(scanner_io::tests::replication_maintenance_refreshes_compacted_stats_across_cycles_and_retries) | test(cluster::rpc::runtime_sources::tests::) | test(config::com::tests::diagnostic_config_write_retains_origin_and_api_for_results_and_cancellation)' --test-threads 1 --no-fail-fast: exit 0, 14/14 passed

Diagnostic strict Clippy remains red on four unchanged production lints:
runtime_sources.rs if_same_then_else and three redundant_locals reports in
set_disk/core/metadata_read.rs and set_disk/read.rs. Their source ranges and
feature wiring match the base; the base was not compiled separately.

Linux aarch64 validation (Rust/Clippy 1.99.0, UID 501, jobs=1):
- cargo clippy --all-targets -- -D warnings: exit 0; Finished dev profile in 68m 12s
- cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings: exit 0; Finished dev profile in 49m 45s
Source HEAD/tree and clean status matched before and after both commands.
No compiler or stack overrides were used. protoc 35.1 was installed from its
verified ARM64 release; flatc was absent and not invoked by the workspace build scripts.
The local environment differs from hosted CI.

* fix(ecstore): recover delete batches with missing versions

Treat an explicit non-null version as a no-op when it is absent from both recovery sources. Keep rejecting missing null versions, delete-marker mismatches and conflicting local generations.

Add returning-disk and partial-receipt regressions, extend journal boundary checks, and regenerate the ecstore test-count baseline. Production faults exercise the 24 added assertion sites and 7 semantic checks: 33 faults failed at the expected sites and passed after full restoration.

Validation (LANG=en_US.UTF-8, one build job):

cargo nextest run --locked -p rustfs-ecstore --lib -E 'test(delete_recovery) | test(quorum_delete_cleanup_coordinator_binds_single_and_batch) | test(crash_delete_retry_batch_reclaims_inline_and_reserved_external)' --test-threads 1 --no-fail-fast
Summary [  34.580s] 69 tests run: 69 passed, 6058 skipped; exit 0

cargo clippy --locked -p rustfs-ecstore --all-targets -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 5m 37s; exit 0

All 19 final Quick Checks completed with exit 0:

- shellcheck --version && actionlint
- cargo fmt --all --check
- ./scripts/check_unsafe_code_allowances.sh
- ./scripts/check_layer_dependencies.sh
- ./scripts/check_architecture_migration_rules.sh
- ./scripts/check_logging_guardrails.sh
- ./scripts/check_error_other_format_ratchet.sh
- ./scripts/check_no_tokio_io_uring.sh
- ./scripts/check_extension_schema_boundaries.sh
- ./scripts/check_body_cache_whitelist.sh
- ./scripts/check_s3s_footprint.sh
- ./scripts/check_test_count_ratchet.sh
- ./scripts/check_fips_wording.sh
- ./scripts/check_embedded_secrets.sh
- make script-tests
- python3 ./scripts/test_object_generation_protocol_model.py
- python3 ./scripts/check_test_wiring.py
- ./scripts/check_no_planning_docs.sh
- ./scripts/check_uring_lane_lib_only.sh

test-count ratchet OK: 52 crates, 22009 tests, 165 ignored match .config/test-count-baseline.txt
PASS: cold stampede script self-test
model: states=51023, transitions=167520, terminal=732, chosen_candidates=2
multi-slot round schedules: 640
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired

* fix(ecstore): retry tier commit intent lock contention

Retry transient namespace lock timeouts within the existing three-attempt
CAS budget. Reload and verify the original mutation identity after
contention, and preserve the final typed error when attempts are exhausted.
Cover save/read contention, exhausted retries, replacement identities and
CAS reconciliation without changing the existing timeout or lock policy.

Validation:
- cargo nextest run --locked -p rustfs-ecstore --features test-util --lib -E 'test(tier_mutation)' --test-threads 1 --no-fail-fast: 52 passed.
- cargo nextest run --locked -p rustfs-ecstore --features test-util --lib -E 'test(intent_advance_) | test(=services::tier::tier::tests::caller_cancellation_after_durable_prepare_does_not_hide_the_mutation)' --test-threads 1 --no-fail-fast: 5 passed.
- cargo clippy --locked -p rustfs-ecstore --all-targets -- -D warnings: passed.
- cargo clippy --locked -p rustfs-ecstore --all-targets --features test-util -- -D warnings: passed.
- 27 mutation pairs failed at the intended assertions and passed after restoring the production code.
- bash scripts/check_test_count_ratchet.sh --update, then bash scripts/check_test_count_ratchet.sh: 52 crates, 22013 tests, 165 ignored; ecstore increases from 6581 to 6585.

* refactor(lifecycle): replace s3s domain types with Gateway DTOs

Keep lifecycle rules, restore requests, Object Lock, and scanner cached tags in domain types. Convert at the existing HTTP boundary while preserving legacy XML, scanner cache bytes, date ordering, and tombstones.

Linux test validation before the final import-path-only adjustment:

cargo nextest run -p rustfs-ecstore bucket_config_goldens
39 passed; 6211 skipped by selector.

cargo nextest run --lib -p rustfs -E 'test(/test_zero_rule_lifecycle_tombstone_recognition/) | test(/test_local_lifecycle_staleness_axis_selection/) | test(/test_lifecycle_staleness_preserves_legacy_submillisecond_axis/) | test(/test_merge_incoming_lifecycle/) | test(/test_lifecycle_expiry_statement_matrix/) | test(/test_lifecycle_expiry_subset_xml/) | test(/put_bucket_lifecycle_validation_errors_keep_their_s3_code/) | test(/execute_put_bucket_lifecycle_configuration_rejects_missing_configuration/) | test(/execute_restore_object_rejects_missing_restore_request/) | test(/execute_restore_object_rejects_select_type/) | test(/execute_restore_object_maps_failures_to_typed_s3_errors/) | test(/execute_restore_object_returns_internal_error_when_store_uninitialized/) | test(/restore_request_conversion_preserves_legacy_output_values/) | test(/restore_request_conversion_preserves_presence_and_raw_enums/) | test(/restore_request_conversion_distinguishes_absent_and_empty_containers/)'
25 passed; 5318 skipped by selector.

cargo nextest run --profile ci -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-replication -p rustfs-scanner
7872 passed; 21 skipped under the existing CI profile.

Previously verified mutation controls: 39 production faults, 41 RED/GREEN pairs.

cargo clippy --locked -p rustfs -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-scanner --all-targets -- -D warnings
Passed in 40m 45s before the import-path adjustment, then in 1m 59s on
the final source with warnings denied. The final adjustment imports the
existing helper through the object domain facade; fmt and the architecture
check also passed.

Part of rustfs/backlog#2745. The remaining ecstore configuration families follow separately.

* refactor(admin): move error types to rustfs-s3-types (#8440)

* refactor(protocols): move error types to rustfs-s3-types by codemod

T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and
the error items of protocols move off s3s by scripts/codemods, run with
--hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it
downcasts the backend error, whose runtime type is still s3s::S3Error.

- rustfs/src/protocols/client.rs: the two webdav session methods now
  return rustfs_s3_types::S3Result and convert the s3s error with From.
- codemod: a child module that calls s3_error! through `use super::*`
  gets its own `use rustfs_s3_types::s3_error;` once its parent imports
  that macro by name; otherwise it is listed as GLOB.
- s3-types: a table test pins code, status and the s3s error the legacy
  edge renders for every variant a moved call site names.
- s3_error_lines 1585 -> 1584 (a lifecycle comment reworded).

* refactor(admin): move error types to rustfs-s3-types (WIP, no build)

T1.2 batch 4 (rustfs/backlog#2743), work in progress: rustfs check
still fails with 16 errors (listed in the PR body).

- codemod: --facade PATH treats a crate-local re-export of s3s's error
  items like s3s for the four names (9 new self-tests, 9 mutants red).
- codemod output for rustfs/src/admin and rustfs/src/site_replication
  (--facade crate::storage_api::site_replication::s3); 6 glob children
  get an explicit rustfs_s3_types::s3_error import.
- admin/router.rs: the Operation trait returns rustfs_s3_types::S3Result.
- error.rs: From<ApiError> builds the rustfs-s3-types error; the s3s
  conversion is derived from it.

* fix(admin): convert error types at the s3s boundary

Batch 4 of T1.2 left the admin and site_replication modules returning
rustfs_s3_types::S3Error while their callers and callees on the storage
and auth side still speak s3s::S3Error. Convert at each edge with the
existing From impls in rustfs-s3-types (compat_s3s.rs) instead of
changing any code, message or status:

- admin/auth.rs, admin/handlers/object_zip_download.rs,
  admin/handlers/table_catalog/mod.rs: map the s3s result returned by
  check_key_valid / authorize_request / authorize_internal_object_request
  into the admin error type with .map_err(Into::into).
- storage/ecfs.rs: the s3s-facing side maps the admin site_replication
  gate results the other way.
- admin/handlers/pools.rs: ApiError.code stays s3s::S3ErrorCode, so the
  rustfs code is converted with .into() when the struct is built, in
  production and in the two tests that build or compare it.
- admin/handlers/heal.rs, admin/runtime_sources.rs: the router returns
  s3s errors (S3Route impl); the expected rustfs code is converted with
  .into() before the comparison.
- admin/handlers/table_catalog/mod.rs: iceberg_rest_error takes
  &'static str so the Cow<'static, str> custom code borrows it; every
  caller already passes an ICEBERG_ERROR_* constant or a &'static str.
- admin/handlers/quota.rs: drop a now same-type .into() after s3_error!.
- storage_api.rs, site_replication/mod.rs: drop re-exports and an import
  that no longer have users (clippy -D warnings).

cargo check -p rustfs --all-targets and cargo clippy -p rustfs
--all-targets -- -D warnings are clean; the s3s footprint baseline is
unchanged at files=172, s3_error_lines=303, ecstore_files=28.

Refs rustfs/backlog#2743, rustfs/backlog#2734.

* refactor(ecstore): move bucket configs onto gateway persisted types (#8709)

* refactor(ecstore): move bucket configs onto gateway persisted types

Finish the ECStore batch of T1.4 (rustfs/backlog#2745). The remaining
eleven bucket configuration families (Website, CORS, Tagging, Versioning,
ObjectLock, Encryption, Notification, Logging, Accelerate, RequestPayment,
PublicAccessBlock) now read and write their stored XML through the
rustfs-gateway-types persistence and cors_tagging codecs instead of the
s3s codec, so the legacy s3s bucket-config codec and its
s3gate-metadata-xml compat marker are removed.

- ecstore: BucketConfigXml is implemented by the gateway codec for every
  family; metadata, metadata_sys, versioning, object lock, replication and
  lifecycle consumers take the persisted types. Region moves to
  rustfs_s3_types::Region, and the ETag conversion to s3s moves to the
  rustfs edge with its tests. Only warm_backend.rs and warm_backend_s3.rs
  still reference s3s, held for T1.5 (rustfs/backlog#2746).
- scanner: object lock cache, item actions and data usage use the
  persisted types.
- rustfs: the legacy S3 edge converts each family between s3s DTOs and the
  persisted types in storage/storage_api.rs by writing one codec's XML and
  reading it with the other; Tagging converts field by field.
- Bump the rustfs-gateway-types pin to 21283e87.

The 39 persisted-byte goldens are unchanged and pass, so a document the
old codec wrote reads back and re-encodes to the same bytes. s3s footprint:
files 181 -> 146, ecstore_files 28 -> 2, s3_error_lines unchanged.

* test(app): prepare the PutBucketEncryption config from the request

The uninitialized-store test passed an empty persisted configuration, which the use case now validates before it reaches the store check, so it answered MalformedXML instead of InternalError. Convert the request's configuration the way the S3 edge does.

* style(app): format the PutBucketEncryption test

* refactor(s3-client): move remote error codes and DTO enums off s3s (#8783)

Part of T1.5 (rustfs/backlog#2746), batch 1.

* refactor(s3select): run queries on gateway DTO request shapes (#8794)

* refactor(s3select): run queries on an engine request built from gateway DTO shapes

Part of T1.5 (rustfs/backlog#2746), batch 2.

* fix(s3select): keep the select request conversion in the app layer

The layer guard forbids app -> storage; the conversion moves from storage/s3_compat/select.rs to app/select_request.rs unchanged.

* fix(heal): retry transient metadata conflicts (#8812)

* fix(heal): retry transient metadata conflicts

* chore: update heal test count baseline

* test: reserve nextest capacity for progress probes (#8818)

* test(e2e): use a non-default region for the differential negative control

Since main's region-aware CreateBucket and GetBucketLocation (#8739, #8743), us-east-1 answers an empty LocationConstraint and 200 for an owned re-create, so the control no longer reported its own region. Also refresh the e2e-full linux selection from the CI listing (682 tests).

* refactor(protocols): move StorageBackend to gateway types (#8825)

* refactor(protocols): move storage backend to gateway types

* test: update e2e test count baseline

* fix: preserve copy source keys and unknown lengths

* test: update protocols test count baseline

* fix(protocols): preserve typed SFTP backend errors

* test(protocols): use gateway list input in policy tests

* fix(protocols): clear clippy failures after T1.5 merge (#8831)

fix(protocols): clear clippy failures after merge

* fix(ecstore): finalize restore metadata after local copy-back (#8836)

* fix(ecstore): finalize restore metadata after local copy-back

* test(ecstore): ratchet added restore finalization cases

* fix(ecstore): keep restore copyback compatible with older peers

* fix(ecstore): gate restore copyback on fleet compatibility

* fix(ecstore): retain transition identity during restore copyback

* fix(ecstore): route committed restore copyback locally

* fix restore copyback cleanup

* fix(ecstore): honor restore probe test fallback

* refactor(storage): migrate option errors to domain types (#8839)

* refactor(s3): migrate private multipart validation errors (#8844)

* refactor(auth): derive policy conditions from AuthzFacts (#8442)

* refactor(auth): extract authorization facts

* test(e2e): update smoke selection baseline

* test(e2e): update full selection baseline

* refactor(admin): migrate health and prefix usage to owned requests (#8887)

* refactor(admin): add owned health router boundary

* refactor(admin): migrate prefix usage to owned request

* refactor(admin): complete the owned admin request boundary (#8949)

* refactor(admin): complete the owned admin request boundary

Every type implementing the owned admin `Operation` now serves the legacy
router through one blanket impl that converts the s3s request and response
at that single edge, so handlers register with `AdminOperation(&Handler {})`
and need no adapter statics; `LegacyAdminAdapter` is removed.

`AdminRequest` gains the accessors handlers read today (`remote_addr`,
`client_info`, `request_context`, `req_info`, `req_info_mut`,
`install_req_info`, `envelope_mut`, `take_body`), the request helpers gain
envelope forms (`authenticate_envelope_credentials`,
`object_store_from_envelope`, the table-catalog `*_from_envelope` helpers,
`KmsAdminAudit::from_envelope`, `AccountAuditContext::from_envelope`,
`CallerIdentity::resolve_envelope`), `read_compatible_admin_body` accepts
either body type, and the owned `Body` gains `store_all_limited` and
`Default`. `admin::operation::test_support::TestRequest` builds owned
requests for handler tests. Unused accessors carry a per-method dead-code
expectation that each migrating handler batch removes.

No handler behavior changes; the remaining migrations follow in batches.

* chore(admin): add the T1.12 admin handler codemod

The script rewrites an admin handler file from the s3s request and response
types to the owned boundary (imports, signatures, request field and helper
calls, body access, response construction, and test request literals) and
leaves a TODO comment at every site it cannot rewrite mechanically. It is
idempotent and is removed once T1.12 completes.

* refactor(admin): migrate replication and KMS handlers to owned requests (#8951)

Move the T1.12 batch 2 admin handlers (site_replication, kms_keys, event,
replication, diagnostics, on_demand_migration, audit, account, kms_rekey,
kms_key_lifecycle, object_data_cache, tls_debug, integrity) from the s3s
request/response types to the owned AdminRequest/AdminResponse boundary.
Handler logic, authorization order, error codes and response bytes are
unchanged; the request/response plumbing now goes through the envelope
accessors and helpers introduced by the owned boundary.

The shared response helpers (utils::json_response, utils::empty_response,
handlers::admin_json_response, target_descriptor::build_json_response,
profile::authorize_profile_request) and the site replication repair
executor still answer with the s3s types, so the migrated files carry
owned-boundary forms of those helpers and convert the repair response at
its call edge. The streaming console-log body and the two devnull drains
move from the s3s byte stream to the owned Body stream/frame API.

Drop the dead_code expectations for the owned accessors and envelope
helpers this batch starts using, and lower the s3s footprint baseline to
the measured 99 files.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* refactor(admin): add owned forms of the shared admin response helpers

Handler batches were copying `json_response`, `empty_response`,
`admin_json_response`, and `build_json_response` into each migrated file
because the shared helpers still return the s3s response. Provide owned
forms next to each legacy helper (`owned_json_response`,
`owned_empty_response`, `owned_admin_json_response`,
`owned_build_json_response`, `authorize_profile_envelope`) with parity
tests against the legacy shape, derive `Debug` for `AdminResponse` so
tests can use `expect_err`, and fix three codemod rules: strip owned
string conversions in test credentials, annotate non-literal URI parses,
and leave a rebound request parameter immutable.

* refactor(admin): use the shared owned response helpers in batch 2 (#8954)

Replace the per-file owned-boundary copies of json_response,
empty_response, admin_json_response, build_json_response and
authorize_profile_request in the batch 2 admin handlers with the shared
owned_* helpers and authorize_profile_envelope that the boundary branch
now provides, and restore expect_err in the handler tests now that
AdminResponse implements Debug; the `.err().expect(..)` stand-ins were
failing the base branch's clippy step (clippy::err_expect under
-D warnings). The site replication repair call edge
keeps its conversion because the repair executor still answers with the
s3s response type.

Drop the dead_code expectations on the shared owned helpers this batch
now uses in non-test code.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* fix(admin): preserve legacy body limit error responses

* refactor(admin): move policy, ILM and batch 3 handlers to owned requests (#8956)

Batch 3 of T1.12 (rustfs/backlog#2788, parent rustfs/backlog#2734) moves
14 admin handler files to AdminRequest/AdminResponse and the owned
Operation trait: ilm_transition, policies, idp_compat, config_admin,
plugins_instances, profile_admin, group, heal, object_zip_download,
kms_key_metadata, durability, inspect_archive, sts and
gateway_key_inventory. Handler logic, authorization order, error codes
and response bytes are unchanged; test request literals use
test_support::TestRequest.

The handlers use the shared owned helpers (owned_json_response,
owned_admin_json_response, owned_build_json_response,
authorize_profile_envelope). The dead_code expectations on the
AdminRequest accessors and RequestEnvelope::uri_mut they now exercise
are removed. The legacy target_descriptor::build_json_response keeps
only its equivalence test caller and carries the matching expectation
until it retires.

object_zip_download builds the s3s request the S3 gate still reads from
the owned request facts; heal authenticates through
authenticate_envelope_credentials; the trace, inspect-archive and zip
bodies use Body::from_stream. extract_plugin_instance_filters takes the
URI so extensions.rs keeps calling it, and the two idp_compat
delegations stay on the legacy trait until their delegates migrate.

The s3s footprint baseline drops from 99 to 90 files.

* test(admin): anchor the replication diff body contract on the call

The source-contract test looked for `let body = read_compatible_admin_body(`
on one line; rustfmt wrapped that binding after the handler moved to the
owned request boundary, so the marker was no longer found. Anchor on the
call itself so formatting cannot change what the contract checks.

* refactor(admin): migrate table catalog and batch 1 to owned requests (#8955)

* fix(admin): preserve catalog body limit response bytes

* refactor(admin): migrate identity and cluster handlers to owned requests (#8966)

* refactor(admin): migrate identity and cluster handlers to owned requests

Move the T1.12 batch 4 admin handlers (user, service_account, oidc, tier,
system, pools, quota, kms_dynamic, kms_backup, rebalance, module_switch,
bucket_meta, cluster_snapshot, realtime, bucket_recovery, profile) from the
s3s request/response types to the owned AdminRequest/AdminResponse
boundary. All 90 impl Operation blocks now take AdminRequest and return
AdminResponse; handler logic, authorization order, error codes and response
bytes are unchanged. The handlers use the shared owned helpers
(owned_json_response, owned_admin_json_response, authorize_profile_envelope,
the envelope-form runtime sources and audit constructors) instead of local
copies.

profile.rs drops the s3s forms of authorize_profile_request and
profile_not_implemented_response now that profile_admin.rs is on the
envelope gate too; profile_not_implemented_response answers with an
AdminResponse. The realtime metrics stream moves from the s3s ByteStream
adapter to Body::from_stream, and the rebalance idle-404 document still
renders through s3s::S3Error::to_http_response with its body converted at
the edge.

The idp_compat delegators ImportIamV2 and AddServiceAccountLdap move to the
owned trait as well: with their targets on both traits, the method-call
delegation became ambiguous between the owned impl and the router blanket
impl, so they now take the owned request directly.

KmsAdminAudit::from_request loses its last production caller with this batch
and is deleted; its comparison test now asserts the envelope facts directly.

Lower the s3s footprint baseline to the measured 68 files.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* refactor(admin): remove the legacy admin helpers batch 4 superseded

Batch 4 moved the last production callers of the s3s-typed admin helpers
to their owned forms, which left `admin_json_response`, `json_response`,
`authenticate_request`, `authorize_admin_request`, `object_store_from_req`
and `object_store_from_extensions` unused, along with the `Body`,
`S3Request` and `S3Response` re-exports of the admin `s3` facade. Remove
them. The owned forms keep their documentation; the behavior tests of the
removed `json_response` now exercise `owned_json_response`, and the
compat-prefix test asserts the owned response directly.

Two tests existed only to compare a removed legacy helper with its owned
twin (`owned_json_response_matches_the_legacy_shape`) or to cover the
removed legacy gate (`authorize_admin_request_without_credentials_is_rejected`);
they go with the helpers, and the owned equivalents stay. The rustfs
test-count row drops from 5957 to 5955 and the s3s footprint from 68 to 66
files.

* refactor(admin): drop the legacy target descriptor JSON response helper (#8968)

`target_descriptor::build_json_response` returned the s3s response and
had no production caller left once the event, audit and plugin instance
handlers moved to `owned_build_json_response`; it survived only for the
legacy/owned equivalence test. Remove it, give the owned helper its own
documentation, and make the test assert the owned response directly:
status, `application/json`, the echoed `x-request-id` when present and
only then, and the body bytes. The file no longer imports s3s, so the
footprint baseline drops from 66 to 65 files.

* test(ecstore): pin native history on cancellation under the cancellable queue

main moved the local write path onto the disk blocking queue, which skips an
operation once its waiter is dropped, so a borrowed write cancelled while it is
queued no longer creates its file. The integration-only native publication
test still expected the queued open to run. Assert the new execution model
instead (the open never runs) while keeping its safety claims: the Started
record is taken before queueing and the cancelled publication never regrants
native history.

* chore(gateway): raise test-count rows for tests merged from main

* test(s3select): use the owned JSON input types in merged planner tests

* fix(ecstore): scope the merged direct-writer test hook to test builds

* fix(ecstore): create parents before admitted truncate opens

Align the heal tests with removal-disabled preservation and local-only outcome
fields, and refresh the e2e-full selection for the synced tests.

* chore(gateway): update test-count baseline after main sync

Main #9241 removed the node pool-metadata readiness cache with 8 tests
and added 6, so the rustfs row drops from 6440 to 6438. The ecstore and
s3select rows rise for tests added by #9237, #9239 and #9241.

* feat(scripts): add junit_diff.py for dual-stack JUnit comparison (#9275)

* ci(gateway): make the test-count ratchet a floor, not exact (#9278)

ci(gateway): make the test-count ratchet a floor instead of an exact baseline

Tests below a row still fail; tests above it, a member without a row and ignores below the row now pass with a note. Ignores above a row still fail. Adds contract tests wired into make script-tests.

* test(guards): read file plus split directory in text guards

Text guards that read one file with include_str! or by path broke, or silently stopped seeing the code they guard, when that code moved to another file. They now read the file they read today, which stays required, plus the sibling directory a mechanical split will populate (storage/legacy_edge, app/bucket_usecase, app/multipart_usecase, app/lifecycle_transition_api_test). Counts are summed over the set and split markers must occur exactly once across it. Refs rustfs/backlog#2765.

* chore(gateway): record ignored microbenchmarks merged from main (#9290)

chore(gateway): record the ignored microbenchmarks merged from main

main added three throughput microbenchmarks that are ignored by design
(#9217): one each in rustfs-checksums, rustfs-rio and rustfs-utils. The
test-count ratchet treats a higher ignored count as a failure until the
row is raised, so raise those three rows to 1 and tighten the test rows
to the merged tree with --update.

* test(guards): avoid a typos false positive in a fixture

* ci(s3-tests): add reference parity lane against pinned 1.0.5 (#9288)

* test(guards): address review of the split-safe text guards

Make the super:: import rules depth-aware for split files, cover ecfs_test.rs moving to legacy_edge/tests, read production text only in the Rust count and contains guards, bound method slices by their own closing line, keep the split arrays safe on bash 3.2, and add an end-to-end split probe for the architecture, layer and logging guards. Refs rustfs/backlog#2765.

* test(guards): plant the lifecycle split probe in its own file

P4 created lifecycle_transition_api_test/put.rs; the end-to-end split probe now plants its violations in split_probe.rs instead of overwriting it.

---------

Co-authored-by: rustfs-cla-assistant[bot] <273121464+rustfs-cla-assistant[bot]@users.noreply.github.com>
2026-11-11 17:09:38 -05:00
Chris An 0e69351973 refactor(admin): move remaining admin handlers to owned requests (#9287)
* feat(server): add the RUSTFS_S3_STACK selector and dual-stack e2e lane (#8411)

* refactor(server): group the s3s-compat tower layers under legacy_compat (#8410)

Move the ten tower layers that patch s3s behaviour out of
rustfs/src/server/layer.rs into rustfs/src/server/legacy_compat/, one file
per layer with its tests. Each file header names the s3s behaviour it
patches, the gateway mechanism that replaces it (Replaced by:), and the task
that removes it (Removed by: T2.12, rustfs/backlog#2771). The layers keep
their code, their order in external_service_stack!, and their behaviour; no
layer is deleted.

Add server::http::tests::stack_census, which renders the ordered service
chain of the real external_service_stack! expansion and compares it with a
committed constant, so a reorder, insertion or removal is a visible diff.

Route the two s3s error types the moved layers use through a
storage_api::server::legacy_compat shim so the split adds no s3s-importing
file, and lower the s3s footprint baselines to the measured values (files 209
to 208, s3_error! lines 1586 to 1585). Keep the server runtime-source bypass
guard covering the moved code.

Refs rustfs/backlog#2787, rustfs/backlog#2734

* feat(s3-types): add the RustFS-owned S3 error type and s3_error! macro (#8406)

`rustfs-s3-types` now provides `S3ErrorCode` (generated by build.rs from
src/codes.txt, the grep-derived call surface, plus `Custom`), `S3Error`,
`S3Result`, `StdError` and the `s3_error` macro with the same call syntax
as s3s, so the T1.2 codemod is a `use` swap. Statuses are hand-written and
pinned to the s3s behaviour RustFS ships today by an oracle test that uses
s3s as a dev-dependency only. The `From` conversions for the legacy edge
live behind the off-by-default `compat-s3s` feature (the orphan rule
forbids them in rustfs/src/error.rs); `rustfs` enables it.

Refs rustfs/backlog#2742, rustfs/backlog#2734

* ci: add the per-crate test-count ratchet (#8404)

Quick Checks now fails any PR that lowers the number of test attributes
in a workspace crate or adds #[ignore], measured per crate against the
exact rows of .config/test-count-baseline.txt (rustfs/backlog#2734
section 1 rule 3). The counter widens the spec's literal
#[(tokio::)?test] to the tokio attributes that carry arguments and both
spellings of #[test_case(...)], and anchors at line start so
commented-out attributes do not count. The initial baseline covers 52
crates, 21077 tests and 164 ignored.

Refs rustfs/backlog#2738
Refs rustfs/backlog#2734

* test(ecstore): pin the persisted bucket-config XML bytes for 13 families (#8408)

Before any DTO or codec is replaced, pin the exact bytes that
deserialize followed by serialize produces for every persisted
bucket-configuration XML family in its three source shapes (AWS
documentation, MinIO-written, RustFS-written), and the bytes RustFS
writes today for a typed value. Byte equality only: a structural
comparison would let a codec replacement change the persisted encoding
unnoticed.

The generator script refuses to overwrite existing goldens without
--force, refuses on dirty codec paths, cannot report success from its
write mode, and records the generating commit in the fixtures README.

Refs rustfs/backlog#2744, rustfs/backlog#2734

* ci: raise the test-count baseline to the integration tree (#8426)

* ci: make the s3s footprint baseline exact and lower-only (#8403)

* ci: make the s3s footprint baseline exact and lower-only

The s3s footprint guard compared three counters against ceilings held in
the script itself, so a ceiling could sit above the tree and a drift was
fixable by raising it (rustfs/rustfs#5964 raised the s3_error! ceiling
to go green). The three values now live in
.config/s3s-footprint-baseline.txt as files, s3_error_lines and
ecstore_files, the script reads only that file, and a measurement that
differs from its baseline in either direction fails: growth prints
"s3s footprint grew: <measured> > <baseline>", shrinkage prints
"lower the baseline to <measured> in this PR". The counting method is
unchanged; the baseline is the current measurement of
gateway/integration (209 / 1585 / 36). The header records the end state:
once all three reach 0, T4.2 switches the script to failing on any
occurrence.

Refs rustfs/backlog#2739, rustfs/backlog#2734

* ci: add the s3s edge allowlist mode to the footprint guard (#8405)

`scripts/check_s3s_footprint.sh --mode allowlist` fails on every Rust
file that references s3s (`s3s::` or `s3s_sigv4`) and is not matched by
an entry of the new `.config/s3s-edge-allowlist.txt`; `--dry-run` lists
those files and exits 0. The allowlist carries the initial edge from the
spec, and the script rejects an entry that names or matches anything
under crates/ other than crates/e2e_test. The baseline mode and the
argument-less CI invocation are unchanged.

Refs rustfs/backlog#2741, rustfs/backlog#2734

* test(ecstore): reach the bucket-config codec through the test facade (#8428)

* refactor(s3-types): own the HTTP Body and drop s3s from three crates (#8431)

Add rustfs_s3_types::Body, a BoxBody<Bytes, StdError> wrapper with
empty() and From<Bytes | Vec<u8> | String | &'static str>, and the
legacy From conversions to and from s3s::Body behind compat-s3s.

The signer is now generic over the request body: it only reads and
writes the request head, so it needs no body type and its callers keep
theirs. s3-client sends its requests with the owned Body, madmin signs a
bodiless head, and targets drops its two S3Request/S3Response header
wrappers in favour of extract_params_header.

rustfs-signer, rustfs-madmin and rustfs-targets no longer depend on
s3s; the s3s footprint baseline drops from 209 to 202 files.

Refs rustfs/backlog#2747, rustfs/backlog#2734

* refactor(ecstore): move S3ErrorCode to rustfs-s3-types by codemod (#8432)

* test(ecstore): pin the InvalidRange responses of new_getobjectreader

Both error branches of new_getobjectreader build an ErrorResponse with S3ErrorCode::InvalidRange, and no test observed either code. Pin the code, its wire name and the location fields of each branch before the code type moves to rustfs-s3-types (rustfs/backlog#2743), so a changed code on either branch fails a test. The ecstore test-count row rises by the two tests.

* chore(scripts): add the s3s-to-rustfs-s3-types error codemod

Task T1.2 of rustfs/backlog#2734 (rustfs/backlog#2743) moves S3Error, S3ErrorCode, S3Result and s3_error! from s3s to rustfs-s3-types by script, one crate closure per PR. The codemod rewrites use trees (splitting mixed s3s imports) and qualified code paths, never comments or literals, adds the rustfs-s3-types dependency, drops s3s only from a crate that no longer names it, refuses every file of the s3s edge allowlist, takes --hold for a file whose type is fixed outside the closure, and prints a per-crate census. A second run changes nothing. Its self-test runs under make script-tests.

* refactor(ecstore): take S3ErrorCode from rustfs-s3-types

Output of scripts/codemods/s3_error_to_s3_types.py over crates/ecstore, crates/lifecycle, crates/scanner and crates/replication (rustfs/backlog#2743 batches 1 and 2). The ErrorResponse code field, the two InvalidRange constructions in new_getobjectreader and an unused tier import now use rustfs_s3_types::S3ErrorCode; the variant names and wire names are unchanged. lifecycle, scanner and replication reference none of the four error names. warm_backend.rs and warm_backend_s3.rs are held: they compare against rustfs_s3_client::ErrorResponse.code, which stays the s3s type until s3-client moves.

* ci(footprint): count only s3_error! lines still on s3s's macro

s3_error_lines counted every line holding the text `s3_error!`, which
also matches the RustFS-owned `rustfs_s3_types::s3_error!`. A codemod
that moves call sites onto rustfs-s3-types therefore could not lower it
(rustfs/backlog#2743, decision 3; rustfs/rustfs#8432, gotcha 2).

The counter now applies a textual, per-file and per-line rule that errs
on the side of counting (full rule in the script header):
- occurrences spelled `rustfs_s3_types::s3_error!` never count;
- a line holding `s3s::s3_error!` always counts;
- any other `s3_error!` line counts unless its file is cleared, that is,
  it imports the macro from rustfs_s3_types by name on a line starting
  with the use item (or defines it with `macro_rules! s3_error`) and does
  not also bring s3s's macro in (`use s3s::s3_error`, a `use s3s::{..}`
  tree naming it or a crate-root `*`, `use s3s::*`, or
  `#[macro_use] extern crate s3s`);
- fail closed: a file matching neither form (`use super::*`, a
  crate-local re-export, a renamed or glob rustfs_s3_types import, a
  commented-out import, a comment-only mention) counts.

Census of the 1585 lines on 96fc06776: 1478 in 64 files that import the
macro from s3s; 107 in 15 files matching neither form (105 through
`use super::*`, 1 qualified `s3s::s3_error!`, 1 comment line); none in
files that import it from rustfs_s3_types or define it. Every line stays
counted, so the baseline remains s3_error_lines=1585 (files=202 and
ecstore_files=36 unchanged). Rewriting rustfs/src/auth.rs's import to
rustfs_s3_types lowers the new counter to 1567; the old one stayed 1585.

`--self-test` (wired into `make script-tests`) copies the script into
22 fixture trees: 6 positive cases and 16 negative ones (lines that must
not count, forms that must not clear a file, and a missing baseline,
missing key or failing classification search that must fail). Each tree
has a control line that must count, and each run gets s3s text on stdin.
Seven cases were red against the old counter. 19 mutants each turn their
case red: old definition, flipped import test, dropped fail-closed
branch, dropped e2e_test exclusion, dropped s3s veto, dropped qualified
s3s rule, dropped rustfs-qualified strip, unanchored rustfs import,
renamed import accepted, nested glob taken as root glob, single-line
s3s trees only, dropped s3s glob forms, dropped macro_use form, dropped
definition clearing, ignored classification rg errors, error-line search
without its path, classification searches without their path, missing
baseline skipped, missing key skipped.

* test(e2e): compare two RUSTFS_S3_STACK servers over every S3 operation (#8435)

Add dual_stack_differential to e2e_test. It starts two rustfs processes
from the same binary, stack A on RUSTFS_S3_STACK=legacy and stack B on
RUSTFS_E2E_STACK_B (default legacy until rustfs/backlog#2734 T2.5),
sends both the same aws-sdk-s3 script covering the 73 methods of
impl S3 for FS (a normal request and a 4xx negative each), and compares
status, error code, an allowlist of response headers and a few typed
body fields. Error message text is never compared. Each run writes
<target>/dual-stack/<run>.json with compared_ops and the (op, field,
a, b) differences; any difference fails the test.

differential_detects_an_injected_difference starts B with
RUSTFS_REGION=eu-west-1 and requires GetBucketLocation to be reported.
The comparator is pure and unit-tested without a server.

Wire the module into the e2e-smoke filter and update the e2e-smoke and
e2e-full selection digests and the e2e_test test-count row.

Refs rustfs/backlog#2740

* feat(app): add the RustFS-owned RequestEnvelope and its s3s constructor (#8437)

* feat(app): add the RustFS-owned RequestEnvelope and its s3s constructor

* ci: classify the s3s edge adapters in storage/s3_compat as interface

* feat(s3-types): add Body::from_stream and move Swift off s3s::Body (#8438)

rustfs/backlog#2747 batch 3. `Body::from_stream` wraps a chunk stream of
unknown length over `StreamBody`/`BoxBody`: one data frame per chunk, the
first error ends the body, the size hint stays unknown, and dropping the
body drops the stream. Parity tests pin it to the s3s `StreamingBlob::wrap`
it replaces, at the `http_body` level and byte for byte through hyper.

The Swift handlers in rustfs-protocols build their responses as the owned
`Body`. `SwiftService` converts them into the wrapped S3 service's response
body type through `From<Body>`, so rustfs keeps serving `s3s::Body` at the
legacy edge unchanged. Request bodies stay Send-only (`UnsyncBoxBody`); the
two storage writers and the DLO/SLO segment streams get `Sync` from
`sync_wrapper::SyncStream`, which is what s3s's `http_body_unsync` mutex
provided before. The DTO and error uses of s3s in protocols are untouched
(T1.5 and T1.2).

* refactor(protocols): move error types to rustfs-s3-types by codemod (#8439)

T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and
the error items of protocols move off s3s by scripts/codemods, run with
--hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it
downcasts the backend error, whose runtime type is still s3s::S3Error.

- rustfs/src/protocols/client.rs: the two webdav session methods now
  return rustfs_s3_types::S3Result and convert the s3s error with From.
- codemod: a child module that calls s3_error! through `use super::*`
  gets its own `use rustfs_s3_types::s3_error;` once its parent imports
  that macro by name; otherwise it is listed as GLOB.
- s3-types: a table test pins code, status and the s3s error the legacy
  edge renders for every variant a moved call site names.
- s3_error_lines 1585 -> 1584 (a lifecycle comment reworded).

* test(connect): restore one-second drive request budgets

Apply the exact test-only repair already landed on main in 5972a09cc3. The filesystem probe and production CLI success test request one second and keep their existing one-second duration and parameter checks. All outcome, I/O, error, cleanup and resource-limit assertions remain.

Scope: rustfs/tests/connect_perf_drive.rs only. Patch-id: cb0a1dfc8f68a67727525adea7458e2cee7d09f8.

Validation environment: LANG=en_US.UTF-8, CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0, exclusive APFS-cloned CARGO_TARGET_DIR.

Quick Checks: every command from .github/actions/quick-checks/action.yml, with captured output and exit status below.

$ shellcheck --version && actionlint
ShellCheck - shell script analysis tool
version: 0.11.0
license: GNU General Public License, version 3
website: https://www.shellcheck.net
exit_code=0

$ cargo fmt --all --check
exit_code=0

$ ./scripts/check_unsafe_code_allowances.sh
exit_code=0

$ ./scripts/check_layer_dependencies.sh
Layer dependency guard passed (no new reverse dependencies/cycles).
exit_code=0

$ ./scripts/check_architecture_migration_rules.sh
Architecture migration rules passed.
exit_code=0

$ ./scripts/check_logging_guardrails.sh
✅ Logging guardrails check passed
exit_code=0

$ ./scripts/check_error_other_format_ratchet.sh
error(format!) ratchet passed (total call sites: 582).
exit_code=0

$ ./scripts/check_no_tokio_io_uring.sh
exit_code=0

$ ./scripts/check_extension_schema_boundaries.sh
Extension schema boundaries passed.
exit_code=0

$ ./scripts/check_body_cache_whitelist.sh
Body-cache whitelist guard passed.
exit_code=0

$ ./scripts/check_s3s_footprint.sh
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)
✅ s3s footprint baseline check passed
exit_code=0

$ ./scripts/check_test_count_ratchet.sh
test-count ratchet OK: 52 crates, 21222 tests, 164 ignored match .config/test-count-baseline.txt
exit_code=0

$ ./scripts/check_fips_wording.sh
FIPS wording guard passed (README.md and CHANGELOG.md contain no forbidden claims).
KMS wording guard passed (crates/kms claims no Vault KV2 Transit wrapping).
exit_code=0

$ ./scripts/check_embedded_secrets.sh
Embedded secret guard passed (no private key material or provider credential literal in the tree).
exit_code=0

$ make script-tests
Running script tests...
entrypoint credential policy tests passed
internode grpc A/B bench env tests passed
object batch benchmark enhanced tests passed
PASS package service script contract
hotpath warp A/B gate tests passed
hotpath warp ABBA tests passed
Scanner validation artifacts written to /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.1pFe6Kl1Yu/out
Scanner validation artifacts written to /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.1pFe6Kl1Yu/out-missing-pid
.............................
----------------------------------------------------------------------
Ran 29 tests in 181.286s

OK
{"decision":"verified","pending_gates":["G01","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G02"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-E"}
{"decision":"verified","pending_gates":["G01","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G02"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-E"}
{"decision":"verified","pending_gates":["G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G01"}
{"decision":"verified","pending_gates":["G01","G02","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G03"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-L"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G14"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G14"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G10"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P1"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P2"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P3"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G05"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G06"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-D"}
Status-and-outcome release descriptor verified: /private/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp8rwn1pe0/descriptor/release-bundle-status-outcome.json
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G05"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G06"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-D"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G07"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G08"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P4"}
exact 1MiB handoff ABBA tests passed
pinned paired ABBA bench tests passed
==> /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/bin/path_containment (-max_total_time=7, -seed=123456789, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/explicit-artifacts/path_containment/, corpus=/private/tmp/gw-orchestrator-2734/ci-drive/fuzz/corpus/path_containment)
-seed=123456789
target=path_containment
seed=123456789
max_total_time=7
git_revision=4d15b816236630b2c42f321ffc9ed3654edc94e3
git_dirty=true
runner_mode=prebuilt
==> /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/bin/path_containment (-max_total_time=1, -seed=2322617706, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/auto-artifacts/path_containment/, corpus=/private/tmp/gw-orchestrator-2734/ci-drive/fuzz/corpus/path_containment)
-seed=2322617706
==> cargo +nightly fuzz run path_containment (-max_total_time=9, -seed=987654321, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/cargo-artifacts/path_containment/)
+nightly
fuzz
run
path_containment
-seed=987654321
seed=987654321
runner_mode=cargo-fuzz
fuzz runner seed manifest tests passed
✅ scripts/python_bin.sh resolver checks passed
self-test ok: clean fixture with every exemption present passes
self-test ok: all 11 pattern families fire
self-test ok: moving a pinned source is reported
Embedded secret guard self-test passed.
self-test ok: p1-s3s-import: 'use s3s::s3_error;' counts every line, the rustfs_s3_types module's too (s3_error_lines=3)
self-test ok: p2-s3s-mixed-tree: a multi-line s3s use tree naming s3_error counts (s3_error_lines=3)
self-test ok: p3-s3s-glob: a crate-root glob of s3s, bare or inside a tree, counts (s3_error_lines=5)
self-test ok: p4-macro-use: '#[macro_use] extern crate s3s' counts (s3_error_lines=3)
self-test ok: p5-qualified-s3s: a qualified s3s::s3_error! line counts in a cleared file (s3_error_lines=2)
self-test ok: p6-exact-pass: a baseline equal to the tree passes (exit 0)
self-test ok: n1-rustfs-import: 'use rustfs_s3_types::s3_error;' alone counts nothing (s3_error_lines=1)
self-test ok: n2-rustfs-mixed-tree: a rustfs_s3_types tree plus an s3s tree without the macro counts nothing (s3_error_lines=1)
self-test ok: n3-defines: the file defining the macro counts nothing (s3_error_lines=1)
self-test ok: n4-qualified-rustfs: a qualified rustfs_s3_types::s3_error! line counts nothing (s3_error_lines=1)
self-test ok: n5-renamed-call: a call through a renamed macro is not text-matched (out of scope) (s3_error_lines=1)
self-test ok: n6-comment-string: a comment or string mentioning s3_error! in a cleared file counts nothing (s3_error_lines=1)
self-test ok: n7-e2e-excluded: crates/e2e_test is excluded (s3_error_lines=1)
self-test ok: n8-super-glob: an unrecognised import form ('use super::*;') counts (s3_error_lines=3)
self-test ok: n9-facade: a crate-local re-export counts (s3_error_lines=2)
self-test ok: n10-renamed-import: a renamed rustfs_s3_types import does not clear s3_error! lines (s3_error_lines=2)
self-test ok: n11-commented-import: a commented-out rustfs_s3_types import does not clear (s3_error_lines=2)
self-test ok: n12-rustfs-glob: a glob import of rustfs_s3_types does not clear (s3_error_lines=2)
self-test ok: n13-comment-only: a comment mentioning s3_error! in a file with no import counts (s3_error_lines=2)
self-test ok: f1-baseline-missing: a missing baseline file fails (exit 1)
self-test ok: f2-key-missing: a baseline without s3_error_lines fails (exit 1)
self-test ok: f3-rg-fails: a failing classification search fails the guard (exit 1)
✅ s3s footprint self-test passed
test_core_gate_rejects_missing_ignored_filtered_and_corrupt_inputs (__main__.SelfTests.test_core_gate_rejects_missing_ignored_filtered_and_corrupt_inputs) ... ok
test_e2e_requires_registration (__main__.SelfTests.test_e2e_requires_registration) ... ok
test_fuzz_runtime_uses_matrix_target (__main__.SelfTests.test_fuzz_runtime_uses_matrix_target) ... ok
test_ilm_lane_keeps_the_measured_cargo_build_budget (__main__.SelfTests.test_ilm_lane_keeps_the_measured_cargo_build_budget) ... ok
test_profile_listing_binds_platform_digest (__main__.SelfTests.test_profile_listing_binds_platform_digest) ... ok
test_profile_listing_enforces_selection (__main__.SelfTests.test_profile_listing_enforces_selection) ... ok
test_quick_checks_commands_propagate_failure (__main__.SelfTests.test_quick_checks_commands_propagate_failure) ... ok
test_quick_checks_rejects_caller_and_execution_bypasses (__main__.SelfTests.test_quick_checks_rejects_caller_and_execution_bypasses) ... ok
test_runner_readiness_fails_closed (__main__.SelfTests.test_runner_readiness_fails_closed) ... ok
test_s3_runner_rejects_unbounded_failure_locals (__main__.SelfTests.test_s3_runner_rejects_unbounded_failure_locals) ... ok
test_scanner_heal_begin_requires_embedded_source_provenance (__main__.SelfTests.test_scanner_heal_begin_requires_embedded_source_provenance) ... ok
test_scanner_heal_case_does_not_approve_pending_release (__main__.SelfTests.test_scanner_heal_case_does_not_approve_pending_release) ... ok
test_scanner_heal_case_lane_is_required_for_runner_profile (__main__.SelfTests.test_scanner_heal_case_lane_is_required_for_runner_profile) ... ok
test_scanner_heal_case_only_schema_cannot_approve_release (__main__.SelfTests.test_scanner_heal_case_only_schema_cannot_approve_release) ... ok
test_scanner_heal_crash_case_rejects_restart_oracle (__main__.SelfTests.test_scanner_heal_crash_case_rejects_restart_oracle) ... ok
test_scanner_heal_ec84_case_rejects_wrong_erasure_geometry (__main__.SelfTests.test_scanner_heal_ec84_case_rejects_wrong_erasure_geometry) ... ok
test_scanner_heal_finish_collects_oracles_from_registry (__main__.SelfTests.test_scanner_heal_finish_collects_oracles_from_registry) ... ok
test_scanner_heal_multipool_case_accepts_deferred_optional_outage_write (__main__.SelfTests.test_scanner_heal_multipool_case_accepts_deferred_optional_outage_write) ... ok
test_scanner_heal_pending_gate_cannot_map_to_implemented_lane (__main__.SelfTests.test_scanner_heal_pending_gate_cannot_map_to_implemented_lane) ... ok
test_scanner_heal_receipts_reject_reuse_and_missing_builds (__main__.SelfTests.test_scanner_heal_receipts_reject_reuse_and_missing_builds) ... ok
test_scanner_heal_rejects_boolean_fractional_and_out_of_geometry_integers (__main__.SelfTests.test_scanner_heal_rejects_boolean_fractional_and_out_of_geometry_integers) ... ok
test_scanner_heal_rejects_broken_execution_and_artifacts (__main__.SelfTests.test_scanner_heal_rejects_broken_execution_and_artifacts) ... ok
test_scanner_heal_rejects_copied_junit_and_wrong_suite_build (__main__.SelfTests.test_scanner_heal_rejects_copied_junit_and_wrong_suite_build) ... ok
test_scanner_heal_release_bundle_accepts_complete_measured_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_accepts_complete_measured_evidence) ... ok
test_scanner_heal_release_bundle_assembler_copies_measured_artifacts (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_copies_measured_artifacts) ... ok
test_scanner_heal_release_bundle_assembler_merges_measured_descriptors (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_merges_measured_descriptors) ... ok
test_scanner_heal_release_bundle_assembler_rejects_duplicate_descriptor_gate (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_duplicate_descriptor_gate) ... ok
test_scanner_heal_release_bundle_assembler_rejects_fixture_descriptor (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_fixture_descriptor) ... ok
test_scanner_heal_release_bundle_assembler_rejects_hash_mismatch (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_hash_mismatch) ... ok
test_scanner_heal_release_bundle_assembler_rejects_mixed_descriptor_revisions (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_mixed_descriptor_revisions) ... ok
test_scanner_heal_release_bundle_enforces_topology_duration_profile_and_versions (__main__.SelfTests.test_scanner_heal_release_bundle_enforces_topology_duration_profile_and_versions) ... ok
test_scanner_heal_release_bundle_fixture_writer_is_not_release_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_fixture_writer_is_not_release_evidence) ... ok
test_scanner_heal_release_bundle_gate_accepts_single_measured_gate (__main__.SelfTests.test_scanner_heal_release_bundle_gate_accepts_single_measured_gate) ... ok
test_scanner_heal_release_bundle_gate_rejects_missing_field (__main__.SelfTests.test_scanner_heal_release_bundle_gate_rejects_missing_field) ... ok
test_scanner_heal_release_bundle_gate_rejects_unknown_gate (__main__.SelfTests.test_scanner_heal_release_bundle_gate_rejects_unknown_gate) ... ok
test_scanner_heal_release_bundle_rejects_synthetic_or_missing_fields (__main__.SelfTests.test_scanner_heal_release_bundle_rejects_synthetic_or_missing_fields) ... ok
test_scanner_heal_release_bundle_requires_domain_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_requires_domain_evidence) ... ok
test_scanner_heal_release_bundle_requires_field_provenance (__main__.SelfTests.test_scanner_heal_release_bundle_requires_field_provenance) ... ok
test_scanner_heal_release_bundle_requires_json_artifact_provenance (__main__.SelfTests.test_scanner_heal_release_bundle_requires_json_artifact_provenance) ... ok
test_scanner_heal_release_bundle_requires_mixed_version_field_roles (__main__.SelfTests.test_scanner_heal_release_bundle_requires_mixed_version_field_roles) ... ok
test_scanner_heal_release_bundle_requires_same_gate_measurement_window (__main__.SelfTests.test_scanner_heal_release_bundle_requires_same_gate_measurement_window) ... ok
test_scanner_heal_release_consumes_multiple_registry_oracles (__main__.SelfTests.test_scanner_heal_release_consumes_multiple_registry_oracles) ... ok
test_scanner_heal_release_matrix_lanes_remain_pending_without_real_evidence (__main__.SelfTests.test_scanner_heal_release_matrix_lanes_remain_pending_without_real_evidence) ... ok
test_scanner_heal_release_requirements_cannot_be_cleared (__main__.SelfTests.test_scanner_heal_release_requirements_cannot_be_cleared) ... ok
test_scanner_heal_release_status_rejects_focused_case_run (__main__.SelfTests.test_scanner_heal_release_status_rejects_focused_case_run) ... ok
test_scanner_heal_release_status_rejects_synthetic_case (__main__.SelfTests.test_scanner_heal_release_status_rejects_synthetic_case) ... ok
test_scanner_heal_required_evidence_fields_cannot_be_removed (__main__.SelfTests.test_scanner_heal_required_evidence_fields_cannot_be_removed) ... ok
test_scheduled_alerts_require_completion_watchdog (__main__.SelfTests.test_scheduled_alerts_require_completion_watchdog) ... ok
test_update_profile_selection_changes_only_requested_platform (__main__.SelfTests.test_update_profile_selection_changes_only_requested_platform) ... ok
test_validate_includes_quick_checks (__main__.SelfTests.test_validate_includes_quick_checks) ... ok
test_vault_tests_require_cross_process_serialization (__main__.SelfTests.test_vault_tests_require_cross_process_serialization) ... ok
test_workflow_readiness_requires_dependency_probe (__main__.SelfTests.test_workflow_readiness_requires_dependency_probe) ... ok

----------------------------------------------------------------------
Ran 52 tests in 14.301s

OK
e2e-full selection OK: 1 tests, sha256=29e30e6ed2ed38d9e5b0a7cdd9536abf5c719f7021a02b578ae1381563a3c273
...................................
----------------------------------------------------------------------
Ran 35 tests in 1.161s

OK
...................
----------------------------------------------------------------------
Ran 19 tests in 17.397s

OK
..........
----------------------------------------------------------------------
Ran 10 tests in 0.183s

OK
test_actual_selector_bootstrap_uses_base_policy_and_fails_closed (__main__.SelfTests.test_actual_selector_bootstrap_uses_base_policy_and_fails_closed) ... ok
test_documentation_paths_do_not_hide_build_or_fixture_changes (__main__.SelfTests.test_documentation_paths_do_not_hide_build_or_fixture_changes) ... ok
test_e2e_scope_keeps_production_and_shared_configuration_full (__main__.SelfTests.test_e2e_scope_keeps_production_and_shared_configuration_full) ... ok
test_e2e_shortcut_rejects_new_direct_renamed_and_target_dependencies (__main__.SelfTests.test_e2e_shortcut_rejects_new_direct_renamed_and_target_dependencies) ... ok
test_e2e_shortcut_traverses_implicit_path_dependencies (__main__.SelfTests.test_e2e_shortcut_traverses_implicit_path_dependencies) ... ok
test_event_contract_requires_complete_candidate_and_optional_lanes (__main__.SelfTests.test_event_contract_requires_complete_candidate_and_optional_lanes) ... ok
test_every_wrong_result_missing_job_or_selection_fails_closed (__main__.SelfTests.test_every_wrong_result_missing_job_or_selection_fails_closed) ... ok
test_full_e2e_gate_preserves_workflow_branch_and_event_scope (__main__.SelfTests.test_full_e2e_gate_preserves_workflow_branch_and_event_scope) ... ok
test_git_range_includes_deleted_source_and_rename_origins (__main__.SelfTests.test_git_range_includes_deleted_source_and_rename_origins) ... ok
test_job_ids_and_display_names_cannot_hide_validation (__main__.SelfTests.test_job_ids_and_display_names_cannot_hide_validation) ... ok
test_report_only_stack_lane_mirrors_e2e_smoke_outside_the_aggregate (__main__.SelfTests.test_report_only_stack_lane_mirrors_e2e_smoke_outside_the_aggregate) ... ok
test_repository_wiring_and_missing_dependency_regression (__main__.SelfTests.test_repository_wiring_and_missing_dependency_regression) ... ok
test_verify_command_preserves_failures (__main__.SelfTests.test_verify_command_preserves_failures) ... ok

----------------------------------------------------------------------
Ran 13 tests in 3.280s

OK
security coverage self-test passed
test_api_failures_preserve_other_evidence_and_never_enter_grace (__main__.SelfTests.test_api_failures_preserve_other_evidence_and_never_enter_grace) ... ok
test_attempt_outcome_does_not_replace_recent_success (__main__.SelfTests.test_attempt_outcome_does_not_replace_recent_success) ... ok
test_cancelled_attempt_cannot_refresh_expired_success (__main__.SelfTests.test_cancelled_attempt_cannot_refresh_expired_success) ... ok
test_cli_requires_the_repository_default_branch (__main__.SelfTests.test_cli_requires_the_repository_default_branch) ... ok
test_config_rejects_duplicate_and_invalid_entries (__main__.SelfTests.test_config_rejects_duplicate_and_invalid_entries) ... ok
test_freshness_boundaries (__main__.SelfTests.test_freshness_boundaries) ... ok
test_grace_requires_two_successful_queries_with_no_history (__main__.SelfTests.test_grace_requires_two_successful_queries_with_no_history) ... ok
test_invalid_api_evidence_fails_closed (__main__.SelfTests.test_invalid_api_evidence_fails_closed) ... ok
test_report_retains_every_workflow (__main__.SelfTests.test_report_retains_every_workflow) ... ok
test_requests_filter_schedule_default_branch_and_success_on_server (__main__.SelfTests.test_requests_filter_schedule_default_branch_and_success_on_server) ... ok

----------------------------------------------------------------------
Ran 10 tests in 0.051s

OK
..............................
----------------------------------------------------------------------
Ran 30 tests in 22.354s

OK
..........
----------------------------------------------------------------------
Ran 10 tests in 2.969s

OK
.................................
----------------------------------------------------------------------
Ran 33 tests in 0.160s

OK
Online functional runner labels: pf-testing
.......
----------------------------------------------------------------------
Ran 7 tests in 0.002s

OK
.......
----------------------------------------------------------------------
Ran 7 tests in 0.001s

OK
....
----------------------------------------------------------------------
Ran 4 tests in 0.128s

OK
...
----------------------------------------------------------------------
Ran 3 tests in 0.547s

OK
PASS: cold stampede script self-test
exit_code=0

$ python3 ./scripts/test_object_generation_protocol_model.py
test_a_recovery_barrier_cannot_choose_an_empty_value (__main__.ObjectGenerationProtocolModel.test_a_recovery_barrier_cannot_choose_an_empty_value) ... ok
test_all_bounded_two_proposer_interleavings_choose_at_most_one_value (__main__.ObjectGenerationProtocolModel.test_all_bounded_two_proposer_interleavings_choose_at_most_one_value) ... ok
test_an_unavailable_decision_quorum_cannot_choose_a_candidate (__main__.ObjectGenerationProtocolModel.test_an_unavailable_decision_quorum_cannot_choose_a_candidate) ... ok
test_counter_allocation_requires_persistence_and_is_consumed_once (__main__.ObjectGenerationProtocolModel.test_counter_allocation_requires_persistence_and_is_consumed_once) ... ok
test_counter_restart_burns_persisted_allocations_before_or_after_issue (__main__.ObjectGenerationProtocolModel.test_counter_restart_burns_persisted_allocations_before_or_after_issue) ... ok
test_crash_after_promise_before_reply_does_not_restore_an_older_ballot (__main__.ObjectGenerationProtocolModel.test_crash_after_promise_before_reply_does_not_restore_an_older_ballot) ... ok
test_duplicate_or_unconfigured_voters_cannot_form_a_quorum (__main__.ObjectGenerationProtocolModel.test_duplicate_or_unconfigured_voters_cannot_form_a_quorum) ... ok
test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision (__main__.ObjectGenerationProtocolModel.test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision) ... ok
test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected (__main__.ObjectGenerationProtocolModel.test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected) ... ok
test_forgetting_tombstone_authority_allows_an_aba_create (__main__.ObjectGenerationProtocolModel.test_forgetting_tombstone_authority_allows_an_aba_create) ... ok
test_issuing_before_persistence_reuses_a_ballot_after_restart (__main__.ObjectGenerationProtocolModel.test_issuing_before_persistence_reuses_a_ballot_after_restart) ... ok
test_lost_ack_and_restart_preserve_the_chosen_operation (__main__.ObjectGenerationProtocolModel.test_lost_ack_and_restart_preserve_the_chosen_operation) ... ok
test_minority_accepted_work_is_adopted_and_finished (__main__.ObjectGenerationProtocolModel.test_minority_accepted_work_is_adopted_and_finished) ... ok
test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample (__main__.ObjectGenerationProtocolModel.test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample) ... ok
test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it (__main__.ObjectGenerationProtocolModel.test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it) ... ok
test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor (__main__.ObjectGenerationProtocolModel.test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor) ... ok
test_replacing_voters_with_empty_state_can_choose_a_conflicting_value (__main__.ObjectGenerationProtocolModel.test_replacing_voters_with_empty_state_can_choose_a_conflicting_value) ... ok
test_skipping_predecessor_recovery_chooses_a_broken_revision_chain (__main__.ObjectGenerationProtocolModel.test_skipping_predecessor_recovery_chooses_a_broken_revision_chain) ... ok
test_stale_publication_rollback_and_gc_cannot_change_the_winner (__main__.ObjectGenerationProtocolModel.test_stale_publication_rollback_and_gc_cannot_change_the_winner) ... ok
test_superseded_operation_identity_cannot_be_reused_for_different_input (__main__.ObjectGenerationProtocolModel.test_superseded_operation_identity_cannot_be_reused_for_different_input) ... ok
test_superseded_operation_retry_returns_its_original_successor (__main__.ObjectGenerationProtocolModel.test_superseded_operation_retry_returns_its_original_successor) ... ok
test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision (__main__.ObjectGenerationProtocolModel.test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision) ... ok
test_tombstone_rejects_a_delayed_never_present_create (__main__.ObjectGenerationProtocolModel.test_tombstone_rejects_a_delayed_never_present_create) ... ok
test_unlearned_predecessor_is_recovered_before_the_second_slot (__main__.ObjectGenerationProtocolModel.test_unlearned_predecessor_is_recovered_before_the_second_slot) ... ok

----------------------------------------------------------------------
Ran 24 tests in 0.958s

OK
model: states=51023, transitions=167520, terminal=732, chosen_candidates=2
negative control: double-choice trace=((0, 'prepare', 0), (0, 'prepare', 1), (0, 'prepare', 2), (0, 'accept', 0), (0, 'accept', 1), (0, 'accept', 2), (1, 'prepare', 0), (1, 'prepare', 1), (1, 'prepare', 2), (1, 'accept', 0), (1, 'accept', 1), (1, 'accept', 2))
multi-slot round schedules: 640
exit_code=0

$ python3 ./scripts/check_test_wiring.py
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired
exit_code=0

$ ./scripts/check_no_planning_docs.sh
exit_code=0

$ ./scripts/check_uring_lane_lib_only.sh
OK: no 'uring_'-matching test functions under crates/ecstore/tests; --lib is safe
exit_code=0

Initial complete gate
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
 Nextest run ID e07fc5b9-c6c8-4cdd-ae61-54839bf24437 with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.016s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.011s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.009s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.009s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  16.950s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   4.025s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.008s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.007s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.035s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  21.087s] 10 tests run: 10 passed, 0 skipped
exit_code=0

Restored complete gate
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
 Nextest run ID 233b7307-5361-4eb7-94d5-b384f17d2d6a with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.009s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.008s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.008s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.008s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  19.518s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   4.146s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.009s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.018s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.009s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.054s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  23.787s] 10 tests run: 10 passed, 0 skipped
exit_code=0

Mutation M1: success_measurement reports max(actual elapsed, requested budget + 1), forcing 1001 ms for the one-second request. The retained duration assertion at rustfs/tests/connect_perf_drive.rs:105 fails.
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast -E 'test(real_local_write_read_is_measured_and_scratch_is_removed)'
info: experimental features enabled: setup-scripts
   Compiling rustfs v1.0.1 (/private/tmp/gw-orchestrator-2734/ci-drive/rustfs)
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs") generated 1 warning
warning: `rustfs` (bin "rustfs-cli") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 7m 06s
────────────
 Nextest run ID 9f618777-7bf6-4cbc-8803-dd7f50f449ed with nextest profile: ci
    Starting 1 test across 1 binary (9 tests skipped)
        FAIL [   0.016s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
  stdout ───

    running 1 test
    test real_local_write_read_is_measured_and_scratch_is_removed ... FAILED

    failures:

    failures:
        real_local_write_read_is_measured_and_scratch_is_removed

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 0.01s

  stderr ───

    thread 'real_local_write_read_is_measured_and_scratch_is_removed' (106986846) panicked at rustfs/tests/connect_perf_drive.rs:105:5:
    assertion failed: (1..=1_000).contains(&data.duration_millis)
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

────────────
     Summary [   0.017s] 1 test run: 0 passed, 1 failed, 9 skipped
        FAIL [   0.016s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
error: test run failed
exit_code=100

Mutation M2: target_parameters reports requested duration + 1. The in-process assertion at rustfs/tests/connect_perf_drive.rs:111 and CLI JSON assertion at rustfs/tests/connect_perf_drive.rs:527 both reject 1001 against 1000.
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast -E 'test(real_local_write_read_is_measured_and_scratch_is_removed) | test(production_cli_measures_and_signs_exact_binary_provenance)'
info: experimental features enabled: setup-scripts
   Compiling rustfs v1.0.1 (/private/tmp/gw-orchestrator-2734/ci-drive/rustfs)
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs") generated 1 warning
warning: `rustfs` (bin "rustfs-cli") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 7m 20s
────────────
 Nextest run ID b73d46ab-c910-4e1e-8063-7fc7e332ec15 with nextest profile: ci
    Starting 2 tests across 1 binary (8 tests skipped)
        FAIL [  13.968s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
  stdout ───

    running 1 test
    test production_cli_measures_and_signs_exact_binary_provenance ... FAILED

    failures:

    failures:
        production_cli_measures_and_signs_exact_binary_provenance

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 13.96s

  stderr ───

    thread 'production_cli_measures_and_signs_exact_binary_provenance' (107015462) panicked at rustfs/tests/connect_perf_drive.rs:527:5:
    assertion `left == right` failed
      left: Number(1001)
     right: 1000
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

        FAIL [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
  stdout ───

    running 1 test
    test real_local_write_read_is_measured_and_scratch_is_removed ... FAILED

    failures:

    failures:
        real_local_write_read_is_measured_and_scratch_is_removed

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 0.01s

  stderr ───

    thread 'real_local_write_read_is_measured_and_scratch_is_removed' (107016667) panicked at rustfs/tests/connect_perf_drive.rs:111:5:
    assertion `left == right` failed
      left: 1001
     right: 1000
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

────────────
     Summary [  13.985s] 2 tests run: 0 passed, 2 failed, 8 skipped
        FAIL [  13.968s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        FAIL [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
error: test run failed
exit_code=100

Transient production mutations were restored byte-for-byte before the complete green rerun. Original/restored SHA256: 8d389e6d8836c041aab8c6a93e56addd65ba3f70b33264e1c74f639d0572a2da.

$ git diff --check

exit_code=0

$ git diff --cached --check

exit_code=0

Independent orchestrator acceptance
Tested code tree: e67c72fb01

Tested tree: 16fbbd2ea4
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

$ LANG=en_US.UTF-8 CARGO_BUILD_JOBS=1 CARGO_INCREMENTAL=0 cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
info: experimental features enabled: setup-scripts
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs-cli") generated 1 warning
warning: `rustfs` (bin "rustfs") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.69s
────────────
 Nextest run ID cd1edf4c-f1c0-456d-ad1a-76cc00ce05c9 with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.012s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.010s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.010s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.009s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  20.328s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   5.527s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.013s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.019s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.011s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.049s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  25.991s] 10 tests run: 10 passed, 0 skipped

exit_code=0

All 54 tracked Cargo.toml files parse without duplicate keys. git diff --check passed. The production source remains unchanged; the two deliberate implementation mutations above were restored before this complete green run.

* fix(test): compare legacy body wire parity as raw bytes

Lossy UTF-8 comparison concealed a changed binary response byte.
Compare the already captured Vec<u8> values directly.

Validation:
- Binary mutant: migrated payload 0x84 vs legacy 0x83; old comparator
  passed 1/1, raw comparator failed 0/1 (exit 100); restored green.
- Non-Swift body-drain mutant: untouched-body delegation test failed
  with empty vs payload (exit 100); restored handler without a diff.
- cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols
  --features rustfs-protocols/swift --no-fail-fast:
  543 tests run, 543 passed, 0 skipped after both mutants restored.
- cargo fmt --all --check and git diff --check passed.

Orchestrator acceptance (LANG=en_US.UTF-8):
cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
Summary [2.445s] 543 tests run: 543 passed, 0 skipped.

All shared quick checks on the final code tree:
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

Final independent acceptance after the drive-test CI repair
Parent: a99d8f4f89
Tested code tree: 93cbd96bf0

Tested tree: 075ae5161c2b62a96812bc1e3c904a7663b4f2f8
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

$ LANG=en_US.UTF-8 CARGO_BUILD_JOBS=1 CARGO_INCREMENTAL=0 cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
    Finished `test` profile [unoptimized + debuginfo] target(s) in 10m 08s
 Nextest run ID 5c7ff102-ec6d-479a-86e0-810859b28bbf with nextest profile: ci
    Starting 543 tests across 8 binaries
     Summary [   6.729s] 543 tests run: 543 passed, 0 skipped
exit_code=0

The only source change is the raw-byte wire comparator. Cargo manifests and the ratchet baselines remain unchanged from the independently validated parent. The invalid-byte and consumed-request mutations remain recorded above; their implementation changes were restored.

* fix: repair integration checks and IPC test readiness

Wait for the RPC/locks test sockets to pass the existing owner and mode
checks inside the original 5s readiness window and 10ms poll interval.

Use the existing admin S3 carrier for the new recovery and quota error
calls. Regenerate the test-count baseline for main's 118 existing tests
across seven crates. Pair the private pagination helper's raw-scan inputs
and remove the redundant Copy clone from the retirement event.

Keep the original assertions, responses, authorization, locks and storage
operations. The pagination body and raw-scan expressions at all five calls
are byte-identical; reinserting only the Copy clone restores the entire
base heal file. All ignored baseline values and s3s counts stay unchanged.

Validation base: c47432e353
Validation tree: f081084bc5
Patch SHA-256: 2dcf0028691bfb2772edfb38e262829b04f4151832dcce823c9dd9f7434b52de
Trace production-prefix SHA-256: fd994f218a21429d62866f4efb8eee1ee57b58f4a51865aa35d05a54ef1ccb45

All commands used LANG=en_US.UTF-8, CARGO_BUILD_JOBS=1,
CARGO_INCREMENTAL=0 and CARGO_NET_OFFLINE=true in one exclusive target.
Every gate retained the same tree and six source hashes before and after.
54 Cargo.toml files parse without duplicate keys.

cargo clippy -p rustfs --all-targets -- -D warnings
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 12m 50s
Exit: 0. Wall time: 770.855s.

The original 19 quick-check steps:
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
Wall time: 286.337s.

cargo nextest run --profile ci -p rustfs --no-fail-fast
     Summary [ 217.017s] 5382 tests run: 5382 passed (4 leaky), 40 skipped
Exit: 0. Wall time: 1324.414s.

cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
     Summary [   2.189s] 543 tests run: 543 passed, 0 skipped
Exit: 0. Wall time: 123.192s.

cargo nextest run --profile ci -p rustfs-ecstore list_objects_paginate --no-fail-fast
     Summary [   0.170s] 8 tests run: 8 passed, 5921 skipped
Exit: 0. Wall time: 261.324s.

cargo nextest run --profile ci -p rustfs-heal durable_mrf_anchor_retires_only_after_exact_bucket_generation_is_deleted --no-fail-fast
     Summary [   0.805s] 1 test run: 1 passed, 835 skipped
Exit: 0. Wall time: 199.801s.

Baseline regeneration:
./scripts/check_test_count_ratchet.sh --update
wrote .config/test-count-baseline.txt: 52 crates, 21340 tests, 164 ignored
Test-only increases: ecstore48, heal3, iam14, io-metrics2, obs3, policy14,
rustfs34. Existing ignored values and all other rows are byte-identical.
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)

Historical runtime red controls on f622 (same retained two assertions):
cargo nextest run --profile ci -p rustfs --lib -E 'test(=connect::diagnostics::trace_runtime::tests::local_top_rpc_reads_service_http_completions_and_signs_offline) | test(=connect::diagnostics::trace_runtime::tests::local_top_locks_reads_service_manager_and_signs_offline)' --no-fail-fast
Original path readiness with a temporary 500ms bind/seal gap:
     Summary [   0.863s] 2 tests run: 0 passed, 2 failed, 5113 skipped
Exit: 100.
Sealed readiness in that same gap:
     Summary [   5.621s] 2 tests run: 2 passed, 5113 skipped
Exit: 0.
Compiled M1: replace the helper predicate with path.exists():
     Summary [   0.870s] 2 tests run: 0 passed, 2 failed, 5113 skipped
Exit: 100.
Restore the sealed predicate byte-for-byte:
     Summary [   5.640s] 2 tests run: 2 passed, 5113 skipped
Exit: 0.
Both reds observe mode755 with owner_matches=true and fail at the original
RPC/locks unwraps with StateSecurity. M1 failure locations: trace_runtime.rs
2547:29 and 2713:29. No new assertions were introduced.

cargo nextest run --profile ci -p rustfs --lib -E 'test(connect::diagnostics::trace_runtime::tests::)' --no-fail-fast
     Summary [  20.748s] 28 tests run: 28 passed (1 leaky), 5087 skipped
Exit: 0. This security sweep is historical; the full final unit gate above
covers the actual c474 candidate.
The first disk-stopped compile ran no tests and is excluded from red credit.
All temporary seal instrumentation is removed from the final source.

Final unit cleanup markers: 4 leaky, 40 original skips (exit0). The ld
__eh_frame linker warning also appears verbatim in the original f622 run
and the earlier c474 run; there are no new source warnings.

* fix: remove redundant pending closure in disk test

Pass the capture-free pending future factory as a function item in the existing slow-operation test. This fixes the redundant_closure error at disk_store.rs:2914 that stopped strict Clippy before the feature tests could run.

The single-line test edit preserves the factory's result, warning and timeout behavior. Reversing the replacement reproduces the parent file byte for byte. All 220 assertion lines and the production prefix are unchanged; no tests, assertions, guards, profiles or baselines were added or relaxed.

Original Linux CI evidence (run 37888552268 at 53245be9f):
- RIO job113685483992: strict Clippy exit101, redundant closure at disk_store.rs:2914; feature tests skipped.
- Workspace Test and Lint job113685484058: same strict Clippy exit101; nextest and doc tests skipped.
These are lint failures, not runtime mutation kills or known flakes.

Native macOS arm64 validation on tree7a5bf47e7533b29479bc85207416102f1337c566, LANG=en_US.UTF-8, one build job, CARGO_INCREMENTAL=0 and offline dependencies:

shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
All 19 checks: exit0, 312.161s.

$ cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 20m 11s
exit0, 1213.210s total.

$ cargo nextest run --profile ci -p rustfs-ecstore --features rio-v2 slow_disk_operation_warns_once_while_pending_and_preserves_result --no-fail-fast
Finished `test` profile [unoptimized + debuginfo] target(s) in 4m 34s
PASS [0.026s] rustfs-ecstore disk::disk_store::tests::slow_disk_operation_warns_once_while_pending_and_preserves_result
Summary [   0.170s] 1 test run: 1 passed, 5943 skipped
exit0, 281.920s total; 5943 existing filter skips.

warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
This linker warning is retained; flags, profiles and allowances are unchanged.

54 tracked Cargo.toml files parsed without duplicate keys. git diff --check passed. No new assertions were introduced, so no new runtime mutation is claimed. Native compilation and the existing helper test do not substitute for actual Linux RIO execution; the new integration CI run still needs to recover before the RPC merge.

Preservation and output hashes:
parent file SHA256: 8e59ee745e3aaaa22848b943d3b10f6353b515956282cde1f3af2c394200c9ac
final file SHA256: b69ba3237061b161448ae8b62b615885d5770b98a5b4a2479bcfbd70f1bd1f17
unchanged production prefix SHA256: 7903a6e288252c420231242d7b58a25fa2a7140ea3dd430cbe7f0dde11369dcb
RIO Linux failure log SHA256:421addb9868fd88e4109a922311e54f91f0dfd70862f8ba7fcaba193482ed90b
Workspace Linux failure log SHA256:e0aea8d0f014c022252a9534968e083c98cfc1fd760da445ecd04ec3f522e26a
19 output SHA256:35a54f28d707956ed1edb29d492c4dfe8a9fe753d103b111b5a2f1b5c41c2b98
RIO Clippy output SHA256:d6046e3868f2b3992fe11cdd04e9e98eb50a87d1234d9a70dbfeb154ff61638d
slow-operation output SHA256:605b7b258d2fd09c6ef2ab2ffc90d433d0509480f1e45125d20c823759e83608

Refs rustfs/backlog#2734.

* test(ecstore): isolate tier-free recovery fixture startup

The fixture constructor can take the production recovery lock and wait for 60 seconds before this test installs its queue and seeds metadata. Disable its background recovery only while creating this fixture, then restore the environment before calling the existing production recovery entrypoint.

Only the existing test setup changes. Its production entrypoint, default lock parameters, original assertions, 30-second timeout, cancellation and cleanup remain intact. No tests or assertions were added. Production and store/init.rs are byte-identical to the parent.

Independent acceptance ran on merge tree 8ec358cb42ba9ebb3bcfeafcef048a3f0366355b; the runner header reports parent HEAD, while before/after index and source hashes confirm this candidate. Worker task branch d8055766b8 has the same tree.

Every command used LANG=en_US.UTF-8. Task compilation also used CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0, CARGO_NET_OFFLINE=true, an independent target, and empty RUSTC_WRAPPER/RUSTC_WORKSPACE_WRAPPER to keep compilers in its owned process group without changing repository configuration.

Command:
shellcheck --version && actionlint
Exit 0; observed output excerpt:
version: 0.11.0
Log SHA-256: fc9eec16b4191d058a295813ca9afbe9d2b37b59975da860170726be162a3924

Command:
cargo fmt --all --check
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_unsafe_code_allowances.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_layer_dependencies.sh
Exit 0; observed output excerpt:
Layer dependency guard passed (no new reverse dependencies/cycles).
Log SHA-256: 69c6cb1896c0586b28d1167380d3260d99d8017a4265c8df5a2348f289e93a12

Command:
./scripts/check_architecture_migration_rules.sh
Exit 0; observed output excerpt:
Architecture migration rules passed.
Log SHA-256: 28790e70d070607ed4de0704f292a65030be92ec224ef106d4b72fd8296f896e

Command:
./scripts/check_logging_guardrails.sh
Exit 0; observed output excerpt:
✅ Logging guardrails check passed
Log SHA-256: 7adf41abff8ed8f7617decf86498888ddde78937099dd9ce47f6d359dfc2d317

Command:
./scripts/check_error_other_format_ratchet.sh
Exit 0; observed output excerpt:
error(format!) ratchet passed (total call sites: 579).
Log SHA-256: 66172aef18f150ba0605af122e26ffcf91f6dd86f43e38b05e2e982ce8d783ee

Command:
./scripts/check_no_tokio_io_uring.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_extension_schema_boundaries.sh
Exit 0; observed output excerpt:
Extension schema boundaries passed.
Log SHA-256: f13d0acd0e2ce5d9815241b12b4dab7d01cc11b9b65bcf6dfb7812c878532a51

Command:
./scripts/check_body_cache_whitelist.sh
Exit 0; observed output excerpt:
Body-cache whitelist guard passed.
Log SHA-256: 6b828847b0383950ee6eef9cd846d0c322c945e47efd4c7d1ad881178810d6bb

Command:
./scripts/check_s3s_footprint.sh
Exit 0; observed output excerpt:
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)
✅ s3s footprint baseline check passed
Log SHA-256: 47f53d04ef645618563fe1f22abc64667f19078f1974feedcca46e4f2c016844

Command:
./scripts/check_test_count_ratchet.sh
Exit 0; observed output excerpt:
test-count ratchet OK: 52 crates, 21340 tests, 164 ignored match .config/test-count-baseline.txt
Log SHA-256: 618a5e330e933de0c6ff484546df401457024a88bb5986d11d588794e33dab60

Command:
./scripts/check_fips_wording.sh
Exit 0; observed output excerpt:
FIPS wording guard passed (README.md and CHANGELOG.md contain no forbidden claims).
KMS wording guard passed (crates/kms claims no Vault KV2 Transit wrapping).
Log SHA-256: 99221cc9979c480ec4f46f50d76c52b18c72fe499f12e15e4f4a22a1b8bfb140

Command:
./scripts/check_embedded_secrets.sh
Exit 0; observed output excerpt:
Embedded secret guard passed (no private key material or provider credential literal in the tree).
Log SHA-256: 437a3ceba32094d2901889965ad191e411aad32cdb18bf0b8659e818011c406b

Command:
make script-tests
Exit 0; observed output excerpt:
test_cli_requires_the_repository_default_branch (__main__.SelfTests.test_cli_requires_the_repository_default_branch) ... ok
test_config_rejects_duplicate_and_invalid_entries (__main__.SelfTests.test_config_rejects_duplicate_and_invalid_entries) ... ok
test_freshness_boundaries (__main__.SelfTests.test_freshness_boundaries) ... ok
test_grace_requires_two_successful_queries_with_no_history (__main__.SelfTests.test_grace_requires_two_successful_queries_with_no_history) ... ok
test_invalid_api_evidence_fails_closed (__main__.SelfTests.test_invalid_api_evidence_fails_closed) ... ok
test_report_retains_every_workflow (__main__.SelfTests.test_report_retains_every_workflow) ... ok
test_requests_filter_schedule_default_branch_and_success_on_server (__main__.SelfTests.test_requests_filter_schedule_default_branch_and_success_on_server) ... ok
Ran 10 tests in 0.042s
OK
Ran 30 tests in 20.627s
OK
Ran 10 tests in 2.268s
OK
Ran 33 tests in 0.127s
OK
Ran 7 tests in 0.002s
OK
Ran 7 tests in 0.001s
OK
Ran 4 tests in 0.108s
OK
Ran 3 tests in 0.564s
OK
PASS: cold stampede script self-test
Log SHA-256: 75fda2d069a373c63667b908baaf99e6591aab9099a739b0c4c22ceb6da8d68c

Command:
python3 ./scripts/test_object_generation_protocol_model.py
Exit 0; observed output excerpt:
test_an_unavailable_decision_quorum_cannot_choose_a_candidate (__main__.ObjectGenerationProtocolModel.test_an_unavailable_decision_quorum_cannot_choose_a_candidate) ... ok
test_counter_allocation_requires_persistence_and_is_consumed_once (__main__.ObjectGenerationProtocolModel.test_counter_allocation_requires_persistence_and_is_consumed_once) ... ok
test_counter_restart_burns_persisted_allocations_before_or_after_issue (__main__.ObjectGenerationProtocolModel.test_counter_restart_burns_persisted_allocations_before_or_after_issue) ... ok
test_crash_after_promise_before_reply_does_not_restore_an_older_ballot (__main__.ObjectGenerationProtocolModel.test_crash_after_promise_before_reply_does_not_restore_an_older_ballot) ... ok
test_duplicate_or_unconfigured_voters_cannot_form_a_quorum (__main__.ObjectGenerationProtocolModel.test_duplicate_or_unconfigured_voters_cannot_form_a_quorum) ... ok
test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision (__main__.ObjectGenerationProtocolModel.test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision) ... ok
test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected (__main__.ObjectGenerationProtocolModel.test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected) ... ok
test_forgetting_tombstone_authority_allows_an_aba_create (__main__.ObjectGenerationProtocolModel.test_forgetting_tombstone_authority_allows_an_aba_create) ... ok
test_issuing_before_persistence_reuses_a_ballot_after_restart (__main__.ObjectGenerationProtocolModel.test_issuing_before_persistence_reuses_a_ballot_after_restart) ... ok
test_lost_ack_and_restart_preserve_the_chosen_operation (__main__.ObjectGenerationProtocolModel.test_lost_ack_and_restart_preserve_the_chosen_operation) ... ok
test_minority_accepted_work_is_adopted_and_finished (__main__.ObjectGenerationProtocolModel.test_minority_accepted_work_is_adopted_and_finished) ... ok
test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample (__main__.ObjectGenerationProtocolModel.test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample) ... ok
test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it (__main__.ObjectGenerationProtocolModel.test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it) ... ok
test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor (__main__.ObjectGenerationProtocolModel.test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor) ... ok
test_replacing_voters_with_empty_state_can_choose_a_conflicting_value (__main__.ObjectGenerationProtocolModel.test_replacing_voters_with_empty_state_can_choose_a_conflicting_value) ... ok
test_skipping_predecessor_recovery_chooses_a_broken_revision_chain (__main__.ObjectGenerationProtocolModel.test_skipping_predecessor_recovery_chooses_a_broken_revision_chain) ... ok
test_stale_publication_rollback_and_gc_cannot_change_the_winner (__main__.ObjectGenerationProtocolModel.test_stale_publication_rollback_and_gc_cannot_change_the_winner) ... ok
test_superseded_operation_identity_cannot_be_reused_for_different_input (__main__.ObjectGenerationProtocolModel.test_superseded_operation_identity_cannot_be_reused_for_different_input) ... ok
test_superseded_operation_retry_returns_its_original_successor (__main__.ObjectGenerationProtocolModel.test_superseded_operation_retry_returns_its_original_successor) ... ok
test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision (__main__.ObjectGenerationProtocolModel.test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision) ... ok
test_tombstone_rejects_a_delayed_never_present_create (__main__.ObjectGenerationProtocolModel.test_tombstone_rejects_a_delayed_never_present_create) ... ok
test_unlearned_predecessor_is_recovered_before_the_second_slot (__main__.ObjectGenerationProtocolModel.test_unlearned_predecessor_is_recovered_before_the_second_slot) ... ok
Ran 24 tests in 0.929s
OK
Log SHA-256: 0b4be926efb0609d497ae58930340153f01957b606206cbbdb3e0d47eb29d3a2

Command:
python3 ./scripts/check_test_wiring.py
Exit 0; observed output excerpt:
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired
Log SHA-256: 21f676f458451525535ab0ef73d6a9c0d9774e16d76a4d19e396ad34936dd39c

Command:
./scripts/check_no_planning_docs.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_uring_lane_lib_only.sh
Exit 0; observed output excerpt:
OK: no 'uring_'-matching test functions under crates/ecstore/tests; --lib is safe
Log SHA-256: 43f3f851e27ec2ab9e34d6b05fefdede38af992699c473904bfe6f53a46d52e3

Command:
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast
Exit 0; wall 234.399s.
Observed output:
    Finished `test` profile [unoptimized + debuginfo] target(s) in 3m 49s
    Starting 1 test across 1 binary (5871 tests skipped)
  SETUP PASS [   0.006s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"'
        PASS [   0.440s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item
     Summary [   0.586s] 1 test run: 1 passed, 5871 skipped
Log SHA-256: 34e973f86e8dc2102c1a0e483e0ac802ce7df3d1b40a603932f62c0805c4de27

Command:
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over) | test(=store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence) | test(=store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable)' --no-fail-fast
Exit 0; wall 4.010s.
Observed output:
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.43s
    Starting 3 tests across 1 binary (5869 tests skipped)
  SETUP PASS [   0.005s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"'
        PASS [   0.068s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over
        PASS [   0.637s] rustfs-ecstore store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence
        PASS [   0.846s] rustfs-ecstore store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable
     Summary [   0.987s] 3 tests run: 3 passed, 5869 skipped
Log SHA-256: b5da2a927ff18ca9aa217762f7186a6d4371bd9e8feedd7b77940e10b64dffac

Command:
cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
Exit 0; wall 1084.508s.
Observed output:
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 18m 03s
Log SHA-256: 28f98be7970faaa71cd118da1da4e6e2e3d23b9dda7e6449eaa7a37e741260b5

Compiled mutation command (unchanged for each red and restored run):
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast

M1: reverse only this test-local constructor flag from false to true. The mutant compiled, then failed the original recovered-item queue assertion after 30.481s. Restoring the entire observed candidate passed in 0.460s. Red log 55bf3b1535529ac0bcfb8071f9b56f9f40631007a85c315367e7e93786d65293; restored log 204d5fbe4574a7917f5e740a795d864fe28a7e7fb5e33d27b0a9cf68e2f17f52.

M2: replace only the actual awaited enqueue with false, consuming its original inputs. The mutant compiled, then failed the same original queue assertion after 30.516s. Restoring the entire observed candidate passed in 0.457s. Red log 89186330413215e231e2b06bf013cc31097ea8f4a106d91421a36d6c8986a7fd; restored log e848b42d2302401afa5dbcd9e37bf76d2e56a96ab6de5a488513b997f53c5e9b.

All temporary observation and mutation code was removed before final acceptance. Historical baseline and diagnostic failures remain recorded; they are not mutation credit. The separate Object Lock intent-retry diagnostic still fails, and these native macOS checks do not establish Linux CI recovery.

* refactor(rpc): serve internode rpc bodies with the owned Body (#8441)

* test(rpc): pin the wire bytes of every internode rpc response body

rustfs/backlog#2747 batch 4, step 1 of 2, on the s3s bodies the rpc paths
serve today. Each response body builder in storage/rpc/http_service.rs is
served through hyper's HTTP/1.1 server and the raw bytes the peer reads are
compared with goldens captured from this commit: status, content-length or
chunked framing, chunk boundaries, the terminating chunk, HEAD, and the cut
connection a failure after data produces. The capability responses carry a
per-process proof, so their length header and payload are checked against
the body they encode instead.

The read_file_stream response builder moves into read_file_stream_response
so the test reaches the same code the handler runs; behaviour is unchanged.

* refactor(rpc): serve internode rpc bodies with the owned Body

rustfs/backlog#2747 batch 4, step 2 of 2. The internode rpc HTTP paths
build their responses with rustfs_s3_types::Body: Body::from_stream
replaces Body::from(StreamingBlob::wrap(..)) at the six stream sites, and
the in-memory and empty bodies keep their constructors. InternodeRpcService
converts its responses into the wrapped service's response body through
From<Body>, so the server keeps composing it with the s3s service as
before. The wire-byte goldens from the previous commit pass unchanged;
storage/rpc no longer names s3s.

* fix(rpc): simplify wire test response builder type

Name the boxed response builder used by the capability wire test so
all-target Clippy can check it without a type-complexity warning.

Validation (LANG=en_US.UTF-8, CARGO_INCREMENTAL=0):
- cargo clippy -p rustfs --all-targets -- -D warnings: exit 0.
- cargo nextest run --profile ci -p rustfs --lib -E 'test(/storage::rpc::/)': 279 tests run, 279 passed, 4842 skipped.
- All 19 quick-check commands, including script-tests, both Python scripts and cargo fmt --all --check: exit 0.
- Footprint baseline regenerated: files=194, s3_error_lines=1584, ecstore_files=36; test-count baseline regenerated: 52 crates, 21228 tests, 164 ignored.
- All 54 Cargo.toml files parse without duplicate keys.

No assertion was added or changed by this lint fix. R1-R8 are retained
in the existing PR verification record.

* fix(ecstore): resume fenced Object Lock creation intents

* fix(ecstore): bound creation retry futures on the stack

* fix(ecstore): require native proof for creation retries

* fix(ecstore): bound bucket creation stack usage (#8466)

* fix(ecstore): retain creation fences through rollback

Keep lifecycle, metadata-transaction, and namespace guards around rollback and metadata cleanup. Preserve the original publication error when a guard is lost. Build the timestamp retry fixture through the real interrupted producer so it checks a valid pending creation.

Validated source tree: 4e8e207999.
Two independent source reviews found no blocking regression.

Validation (LANG=en_US.UTF-8):
- bash -c 'shellcheck --version && actionlint' — exit 0
- cargo fmt --all --check — exit 0
- ./scripts/check_unsafe_code_allowances.sh — exit 0
- ./scripts/check_layer_dependencies.sh — exit 0
- ./scripts/check_architecture_migration_rules.sh — exit 0
- ./scripts/check_logging_guardrails.sh — exit 0
- ./scripts/check_error_other_format_ratchet.sh — exit 0
- ./scripts/check_no_tokio_io_uring.sh — exit 0
- ./scripts/check_extension_schema_boundaries.sh — exit 0
- ./scripts/check_body_cache_whitelist.sh — exit 0
- ./scripts/check_s3s_footprint.sh — exit 0
- ./scripts/check_test_count_ratchet.sh — exit 0
- ./scripts/check_fips_wording.sh — exit 0
- ./scripts/check_embedded_secrets.sh — exit 0
- make script-tests — exit 0
- python3 ./scripts/test_object_generation_protocol_model.py — exit 0
- python3 ./scripts/check_test_wiring.py — exit 0
- ./scripts/check_no_planning_docs.sh — exit 0
- ./scripts/check_uring_lane_lib_only.sh — exit 0
- cargo nextest run --profile ci -p rustfs-ecstore --lib --locked --no-fail-fast — 5894 passed, 0 skipped
- cargo check -p rustfs-ecstore --all-targets --locked — exit 0
- cargo nextest run --profile ci -p rustfs --lib --locked --no-fail-fast -E 'test(=admin::handlers::table_catalog::tests::cold_strong_registration_completes_while_other_bucket_backup_waits) | test(=admin::handlers::table_catalog::tests::commit_backend_borrowed_put_preserves_owner_authorization_and_precondition) | test(=admin::handlers::table_catalog::tests::disable_table_bucket_preserves_objects_and_can_be_reenabled) | test(=admin::handlers::table_catalog::tests::load_table_handler_negotiates_vended_credentials_without_breaking_metadata_only_callers) | test(=admin::handlers::table_catalog::tests::namespace_property_handler_updates_object_backed_catalog_and_maps_errors) | test(=app::object::get::tests::execute_get_object_resumes_from_relocated_pool_without_splicing_body) | test(=table_catalog::store::tests::catalog_writer_loss_prevents_recovery_storage_commit)' — 7 passed
- cargo nextest run --profile ci -p rustfs --lib --locked --no-fail-fast -E 'test(unsupported_remote_disk_)' — 2 passed

Additional fault-injection evidence remains incomplete. Timestamp M1 reached its original assertion; its restore run is being collected. M2–M5 and the three planned rollback faults have not run. An automated risk check interrupted the timestamp proof task; no assertions or validation settings were weakened.

* fix(ecstore): box nested bucket rollback state

Keep the guarded rollback future behind a heap allocation so bucket creation does not store its nested state inline. The rollback body, all three fence references and pending-proof revocation order are unchanged.

The same three original Linux tests overflowed on both e611 CI attempts. Local format, ecstore test compilation and all 19 quick-check steps pass; the original Linux feature lanes still need to validate this candidate.

* test(ecstore): isolate rollback guard loss fixtures

Run the three rollback metadata-cleanup cases in separate test processes so reused /proc/self/fd aliases from earlier cases cannot invalidate their creation witnesses. Keep the original fixture, guards, assertions and io_uring test selection, and verify that each child runs the intended case successfully.

Update the generated test-count baseline for the added macro arm.

Validation: cargo check -p rustfs-ecstore --tests; cargo fmt --all --check; all 19 quick-check commands, including make script-tests and the two Python checks. Linux io_uring acceptance and fault controls remain pending.

* refactor(replication): move the replication config onto gateway types (#8443)

* wip: T1.4 replication family

* refactor(replication): move the replication config onto gateway types

The replication configuration family leaves s3s: rustfs-replication no longer depends on s3s, and ECStore bucket metadata, the scanner and the rustfs app/admin layers hold rustfs_gateway_types::persistence::PersistedReplicationConfiguration. The persisted XML goes through the gateway persistence codec behind a new BucketConfigXml trait (the 39 byte goldens pass with unchanged fixtures); the legacy s3s edge converts once in storage/ecfs.rs through storage/s3_compat/replication.rs; the scanner data usage cache keeps its MessagePack bytes through a serde mirror pinned by bytes recorded from s3s 5761ddfe. rustfs-gateway-types 0.32.0 is pinned once (rev 772d3385). Refs rustfs/backlog#2745.

* test: use S3 serializer for list response tests

* fix(ci): restore integration test compilation

Gate the diagnostic test import with its feature, initialize the conditional
MetadataWriteStore fields explicitly, and decode the scanner replication fixture
through its existing cache boundary. Preserve the original fixture and assertions.

Validation was run on tree 544acb80c0 at 6003d4b8.
LANG=en_US.UTF-8; no source changes follow validation.

Quick checks: 19/19 passed.
- shellcheck --version && actionlint: exit 0
- cargo fmt --all --check: exit 0
- ./scripts/check_unsafe_code_allowances.sh: exit 0
- ./scripts/check_layer_dependencies.sh: exit 0
- ./scripts/check_architecture_migration_rules.sh: exit 0
- ./scripts/check_logging_guardrails.sh: exit 0
- ./scripts/check_error_other_format_ratchet.sh: exit 0
- ./scripts/check_no_tokio_io_uring.sh: exit 0
- ./scripts/check_extension_schema_boundaries.sh: exit 0
- ./scripts/check_body_cache_whitelist.sh: exit 0
- ./scripts/check_s3s_footprint.sh: exit 0
- ./scripts/check_test_count_ratchet.sh: exit 0
- ./scripts/check_fips_wording.sh: exit 0
- ./scripts/check_embedded_secrets.sh: exit 0
- make script-tests: exit 0
- python3 ./scripts/test_object_generation_protocol_model.py: exit 0
- python3 ./scripts/check_test_wiring.py: exit 0
- ./scripts/check_no_planning_docs.sh: exit 0
- ./scripts/check_uring_lane_lib_only.sh: exit 0

Darwin validation:
- cargo clippy --locked -p rustfs-ecstore -p rustfs-scanner --all-targets -- -D warnings: exit 0
- cargo nextest run --locked -p rustfs-ecstore -p rustfs-scanner --lib -E 'test(config::com::tests::save_config_preserves_retryable_system_volume_errors) | test(scanner_io::tests::replication_maintenance_refreshes_compacted_stats_across_cycles_and_retries)' --test-threads 1 --no-fail-fast: exit 0, 2/2 passed
- cargo clippy --locked -p rustfs-ecstore -p rustfs-scanner --all-targets --features rustfs-ecstore/diagnostic-read-version-caller,rustfs-scanner/diagnostic-read-version-caller -- -D warnings: exit 101
- cargo nextest run --locked -p rustfs-ecstore -p rustfs-scanner --lib --features rustfs-ecstore/diagnostic-read-version-caller,rustfs-scanner/diagnostic-read-version-caller -E 'test(config::com::tests::save_config_preserves_retryable_system_volume_errors) | test(scanner_io::tests::replication_maintenance_refreshes_compacted_stats_across_cycles_and_retries) | test(cluster::rpc::runtime_sources::tests::) | test(config::com::tests::diagnostic_config_write_retains_origin_and_api_for_results_and_cancellation)' --test-threads 1 --no-fail-fast: exit 0, 14/14 passed

Diagnostic strict Clippy remains red on four unchanged production lints:
runtime_sources.rs if_same_then_else and three redundant_locals reports in
set_disk/core/metadata_read.rs and set_disk/read.rs. Their source ranges and
feature wiring match the base; the base was not compiled separately.

Linux aarch64 validation (Rust/Clippy 1.99.0, UID 501, jobs=1):
- cargo clippy --all-targets -- -D warnings: exit 0; Finished dev profile in 68m 12s
- cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings: exit 0; Finished dev profile in 49m 45s
Source HEAD/tree and clean status matched before and after both commands.
No compiler or stack overrides were used. protoc 35.1 was installed from its
verified ARM64 release; flatc was absent and not invoked by the workspace build scripts.
The local environment differs from hosted CI.

* fix(ecstore): recover delete batches with missing versions

Treat an explicit non-null version as a no-op when it is absent from both recovery sources. Keep rejecting missing null versions, delete-marker mismatches and conflicting local generations.

Add returning-disk and partial-receipt regressions, extend journal boundary checks, and regenerate the ecstore test-count baseline. Production faults exercise the 24 added assertion sites and 7 semantic checks: 33 faults failed at the expected sites and passed after full restoration.

Validation (LANG=en_US.UTF-8, one build job):

cargo nextest run --locked -p rustfs-ecstore --lib -E 'test(delete_recovery) | test(quorum_delete_cleanup_coordinator_binds_single_and_batch) | test(crash_delete_retry_batch_reclaims_inline_and_reserved_external)' --test-threads 1 --no-fail-fast
Summary [  34.580s] 69 tests run: 69 passed, 6058 skipped; exit 0

cargo clippy --locked -p rustfs-ecstore --all-targets -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 5m 37s; exit 0

All 19 final Quick Checks completed with exit 0:

- shellcheck --version && actionlint
- cargo fmt --all --check
- ./scripts/check_unsafe_code_allowances.sh
- ./scripts/check_layer_dependencies.sh
- ./scripts/check_architecture_migration_rules.sh
- ./scripts/check_logging_guardrails.sh
- ./scripts/check_error_other_format_ratchet.sh
- ./scripts/check_no_tokio_io_uring.sh
- ./scripts/check_extension_schema_boundaries.sh
- ./scripts/check_body_cache_whitelist.sh
- ./scripts/check_s3s_footprint.sh
- ./scripts/check_test_count_ratchet.sh
- ./scripts/check_fips_wording.sh
- ./scripts/check_embedded_secrets.sh
- make script-tests
- python3 ./scripts/test_object_generation_protocol_model.py
- python3 ./scripts/check_test_wiring.py
- ./scripts/check_no_planning_docs.sh
- ./scripts/check_uring_lane_lib_only.sh

test-count ratchet OK: 52 crates, 22009 tests, 165 ignored match .config/test-count-baseline.txt
PASS: cold stampede script self-test
model: states=51023, transitions=167520, terminal=732, chosen_candidates=2
multi-slot round schedules: 640
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired

* fix(ecstore): retry tier commit intent lock contention

Retry transient namespace lock timeouts within the existing three-attempt
CAS budget. Reload and verify the original mutation identity after
contention, and preserve the final typed error when attempts are exhausted.
Cover save/read contention, exhausted retries, replacement identities and
CAS reconciliation without changing the existing timeout or lock policy.

Validation:
- cargo nextest run --locked -p rustfs-ecstore --features test-util --lib -E 'test(tier_mutation)' --test-threads 1 --no-fail-fast: 52 passed.
- cargo nextest run --locked -p rustfs-ecstore --features test-util --lib -E 'test(intent_advance_) | test(=services::tier::tier::tests::caller_cancellation_after_durable_prepare_does_not_hide_the_mutation)' --test-threads 1 --no-fail-fast: 5 passed.
- cargo clippy --locked -p rustfs-ecstore --all-targets -- -D warnings: passed.
- cargo clippy --locked -p rustfs-ecstore --all-targets --features test-util -- -D warnings: passed.
- 27 mutation pairs failed at the intended assertions and passed after restoring the production code.
- bash scripts/check_test_count_ratchet.sh --update, then bash scripts/check_test_count_ratchet.sh: 52 crates, 22013 tests, 165 ignored; ecstore increases from 6581 to 6585.

* refactor(lifecycle): replace s3s domain types with Gateway DTOs

Keep lifecycle rules, restore requests, Object Lock, and scanner cached tags in domain types. Convert at the existing HTTP boundary while preserving legacy XML, scanner cache bytes, date ordering, and tombstones.

Linux test validation before the final import-path-only adjustment:

cargo nextest run -p rustfs-ecstore bucket_config_goldens
39 passed; 6211 skipped by selector.

cargo nextest run --lib -p rustfs -E 'test(/test_zero_rule_lifecycle_tombstone_recognition/) | test(/test_local_lifecycle_staleness_axis_selection/) | test(/test_lifecycle_staleness_preserves_legacy_submillisecond_axis/) | test(/test_merge_incoming_lifecycle/) | test(/test_lifecycle_expiry_statement_matrix/) | test(/test_lifecycle_expiry_subset_xml/) | test(/put_bucket_lifecycle_validation_errors_keep_their_s3_code/) | test(/execute_put_bucket_lifecycle_configuration_rejects_missing_configuration/) | test(/execute_restore_object_rejects_missing_restore_request/) | test(/execute_restore_object_rejects_select_type/) | test(/execute_restore_object_maps_failures_to_typed_s3_errors/) | test(/execute_restore_object_returns_internal_error_when_store_uninitialized/) | test(/restore_request_conversion_preserves_legacy_output_values/) | test(/restore_request_conversion_preserves_presence_and_raw_enums/) | test(/restore_request_conversion_distinguishes_absent_and_empty_containers/)'
25 passed; 5318 skipped by selector.

cargo nextest run --profile ci -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-replication -p rustfs-scanner
7872 passed; 21 skipped under the existing CI profile.

Previously verified mutation controls: 39 production faults, 41 RED/GREEN pairs.

cargo clippy --locked -p rustfs -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-scanner --all-targets -- -D warnings
Passed in 40m 45s before the import-path adjustment, then in 1m 59s on
the final source with warnings denied. The final adjustment imports the
existing helper through the object domain facade; fmt and the architecture
check also passed.

Part of rustfs/backlog#2745. The remaining ecstore configuration families follow separately.

* refactor(admin): move error types to rustfs-s3-types (#8440)

* refactor(protocols): move error types to rustfs-s3-types by codemod

T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and
the error items of protocols move off s3s by scripts/codemods, run with
--hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it
downcasts the backend error, whose runtime type is still s3s::S3Error.

- rustfs/src/protocols/client.rs: the two webdav session methods now
  return rustfs_s3_types::S3Result and convert the s3s error with From.
- codemod: a child module that calls s3_error! through `use super::*`
  gets its own `use rustfs_s3_types::s3_error;` once its parent imports
  that macro by name; otherwise it is listed as GLOB.
- s3-types: a table test pins code, status and the s3s error the legacy
  edge renders for every variant a moved call site names.
- s3_error_lines 1585 -> 1584 (a lifecycle comment reworded).

* refactor(admin): move error types to rustfs-s3-types (WIP, no build)

T1.2 batch 4 (rustfs/backlog#2743), work in progress: rustfs check
still fails with 16 errors (listed in the PR body).

- codemod: --facade PATH treats a crate-local re-export of s3s's error
  items like s3s for the four names (9 new self-tests, 9 mutants red).
- codemod output for rustfs/src/admin and rustfs/src/site_replication
  (--facade crate::storage_api::site_replication::s3); 6 glob children
  get an explicit rustfs_s3_types::s3_error import.
- admin/router.rs: the Operation trait returns rustfs_s3_types::S3Result.
- error.rs: From<ApiError> builds the rustfs-s3-types error; the s3s
  conversion is derived from it.

* fix(admin): convert error types at the s3s boundary

Batch 4 of T1.2 left the admin and site_replication modules returning
rustfs_s3_types::S3Error while their callers and callees on the storage
and auth side still speak s3s::S3Error. Convert at each edge with the
existing From impls in rustfs-s3-types (compat_s3s.rs) instead of
changing any code, message or status:

- admin/auth.rs, admin/handlers/object_zip_download.rs,
  admin/handlers/table_catalog/mod.rs: map the s3s result returned by
  check_key_valid / authorize_request / authorize_internal_object_request
  into the admin error type with .map_err(Into::into).
- storage/ecfs.rs: the s3s-facing side maps the admin site_replication
  gate results the other way.
- admin/handlers/pools.rs: ApiError.code stays s3s::S3ErrorCode, so the
  rustfs code is converted with .into() when the struct is built, in
  production and in the two tests that build or compare it.
- admin/handlers/heal.rs, admin/runtime_sources.rs: the router returns
  s3s errors (S3Route impl); the expected rustfs code is converted with
  .into() before the comparison.
- admin/handlers/table_catalog/mod.rs: iceberg_rest_error takes
  &'static str so the Cow<'static, str> custom code borrows it; every
  caller already passes an ICEBERG_ERROR_* constant or a &'static str.
- admin/handlers/quota.rs: drop a now same-type .into() after s3_error!.
- storage_api.rs, site_replication/mod.rs: drop re-exports and an import
  that no longer have users (clippy -D warnings).

cargo check -p rustfs --all-targets and cargo clippy -p rustfs
--all-targets -- -D warnings are clean; the s3s footprint baseline is
unchanged at files=172, s3_error_lines=303, ecstore_files=28.

Refs rustfs/backlog#2743, rustfs/backlog#2734.

* refactor(ecstore): move bucket configs onto gateway persisted types (#8709)

* refactor(ecstore): move bucket configs onto gateway persisted types

Finish the ECStore batch of T1.4 (rustfs/backlog#2745). The remaining
eleven bucket configuration families (Website, CORS, Tagging, Versioning,
ObjectLock, Encryption, Notification, Logging, Accelerate, RequestPayment,
PublicAccessBlock) now read and write their stored XML through the
rustfs-gateway-types persistence and cors_tagging codecs instead of the
s3s codec, so the legacy s3s bucket-config codec and its
s3gate-metadata-xml compat marker are removed.

- ecstore: BucketConfigXml is implemented by the gateway codec for every
  family; metadata, metadata_sys, versioning, object lock, replication and
  lifecycle consumers take the persisted types. Region moves to
  rustfs_s3_types::Region, and the ETag conversion to s3s moves to the
  rustfs edge with its tests. Only warm_backend.rs and warm_backend_s3.rs
  still reference s3s, held for T1.5 (rustfs/backlog#2746).
- scanner: object lock cache, item actions and data usage use the
  persisted types.
- rustfs: the legacy S3 edge converts each family between s3s DTOs and the
  persisted types in storage/storage_api.rs by writing one codec's XML and
  reading it with the other; Tagging converts field by field.
- Bump the rustfs-gateway-types pin to 21283e87.

The 39 persisted-byte goldens are unchanged and pass, so a document the
old codec wrote reads back and re-encodes to the same bytes. s3s footprint:
files 181 -> 146, ecstore_files 28 -> 2, s3_error_lines unchanged.

* test(app): prepare the PutBucketEncryption config from the request

The uninitialized-store test passed an empty persisted configuration, which the use case now validates before it reaches the store check, so it answered MalformedXML instead of InternalError. Convert the request's configuration the way the S3 edge does.

* style(app): format the PutBucketEncryption test

* refactor(s3-client): move remote error codes and DTO enums off s3s (#8783)

Part of T1.5 (rustfs/backlog#2746), batch 1.

* refactor(s3select): run queries on gateway DTO request shapes (#8794)

* refactor(s3select): run queries on an engine request built from gateway DTO shapes

Part of T1.5 (rustfs/backlog#2746), batch 2.

* fix(s3select): keep the select request conversion in the app layer

The layer guard forbids app -> storage; the conversion moves from storage/s3_compat/select.rs to app/select_request.rs unchanged.

* fix(heal): retry transient metadata conflicts (#8812)

* fix(heal): retry transient metadata conflicts

* chore: update heal test count baseline

* test: reserve nextest capacity for progress probes (#8818)

* test(e2e): use a non-default region for the differential negative control

Since main's region-aware CreateBucket and GetBucketLocation (#8739, #8743), us-east-1 answers an empty LocationConstraint and 200 for an owned re-create, so the control no longer reported its own region. Also refresh the e2e-full linux selection from the CI listing (682 tests).

* refactor(protocols): move StorageBackend to gateway types (#8825)

* refactor(protocols): move storage backend to gateway types

* test: update e2e test count baseline

* fix: preserve copy source keys and unknown lengths

* test: update protocols test count baseline

* fix(protocols): preserve typed SFTP backend errors

* test(protocols): use gateway list input in policy tests

* fix(protocols): clear clippy failures after T1.5 merge (#8831)

fix(protocols): clear clippy failures after merge

* fix(ecstore): finalize restore metadata after local copy-back (#8836)

* fix(ecstore): finalize restore metadata after local copy-back

* test(ecstore): ratchet added restore finalization cases

* fix(ecstore): keep restore copyback compatible with older peers

* fix(ecstore): gate restore copyback on fleet compatibility

* fix(ecstore): retain transition identity during restore copyback

* fix(ecstore): route committed restore copyback locally

* fix restore copyback cleanup

* fix(ecstore): honor restore probe test fallback

* refactor(storage): migrate option errors to domain types (#8839)

* refactor(s3): migrate private multipart validation errors (#8844)

* refactor(auth): derive policy conditions from AuthzFacts (#8442)

* refactor(auth): extract authorization facts

* test(e2e): update smoke selection baseline

* test(e2e): update full selection baseline

* refactor(admin): migrate health and prefix usage to owned requests (#8887)

* refactor(admin): add owned health router boundary

* refactor(admin): migrate prefix usage to owned request

* refactor(admin): complete the owned admin request boundary (#8949)

* refactor(admin): complete the owned admin request boundary

Every type implementing the owned admin `Operation` now serves the legacy
router through one blanket impl that converts the s3s request and response
at that single edge, so handlers register with `AdminOperation(&Handler {})`
and need no adapter statics; `LegacyAdminAdapter` is removed.

`AdminRequest` gains the accessors handlers read today (`remote_addr`,
`client_info`, `request_context`, `req_info`, `req_info_mut`,
`install_req_info`, `envelope_mut`, `take_body`), the request helpers gain
envelope forms (`authenticate_envelope_credentials`,
`object_store_from_envelope`, the table-catalog `*_from_envelope` helpers,
`KmsAdminAudit::from_envelope`, `AccountAuditContext::from_envelope`,
`CallerIdentity::resolve_envelope`), `read_compatible_admin_body` accepts
either body type, and the owned `Body` gains `store_all_limited` and
`Default`. `admin::operation::test_support::TestRequest` builds owned
requests for handler tests. Unused accessors carry a per-method dead-code
expectation that each migrating handler batch removes.

No handler behavior changes; the remaining migrations follow in batches.

* chore(admin): add the T1.12 admin handler codemod

The script rewrites an admin handler file from the s3s request and response
types to the owned boundary (imports, signatures, request field and helper
calls, body access, response construction, and test request literals) and
leaves a TODO comment at every site it cannot rewrite mechanically. It is
idempotent and is removed once T1.12 completes.

* refactor(admin): migrate replication and KMS handlers to owned requests (#8951)

Move the T1.12 batch 2 admin handlers (site_replication, kms_keys, event,
replication, diagnostics, on_demand_migration, audit, account, kms_rekey,
kms_key_lifecycle, object_data_cache, tls_debug, integrity) from the s3s
request/response types to the owned AdminRequest/AdminResponse boundary.
Handler logic, authorization order, error codes and response bytes are
unchanged; the request/response plumbing now goes through the envelope
accessors and helpers introduced by the owned boundary.

The shared response helpers (utils::json_response, utils::empty_response,
handlers::admin_json_response, target_descriptor::build_json_response,
profile::authorize_profile_request) and the site replication repair
executor still answer with the s3s types, so the migrated files carry
owned-boundary forms of those helpers and convert the repair response at
its call edge. The streaming console-log body and the two devnull drains
move from the s3s byte stream to the owned Body stream/frame API.

Drop the dead_code expectations for the owned accessors and envelope
helpers this batch starts using, and lower the s3s footprint baseline to
the measured 99 files.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* refactor(admin): add owned forms of the shared admin response helpers

Handler batches were copying `json_response`, `empty_response`,
`admin_json_response`, and `build_json_response` into each migrated file
because the shared helpers still return the s3s response. Provide owned
forms next to each legacy helper (`owned_json_response`,
`owned_empty_response`, `owned_admin_json_response`,
`owned_build_json_response`, `authorize_profile_envelope`) with parity
tests against the legacy shape, derive `Debug` for `AdminResponse` so
tests can use `expect_err`, and fix three codemod rules: strip owned
string conversions in test credentials, annotate non-literal URI parses,
and leave a rebound request parameter immutable.

* refactor(admin): use the shared owned response helpers in batch 2 (#8954)

Replace the per-file owned-boundary copies of json_response,
empty_response, admin_json_response, build_json_response and
authorize_profile_request in the batch 2 admin handlers with the shared
owned_* helpers and authorize_profile_envelope that the boundary branch
now provides, and restore expect_err in the handler tests now that
AdminResponse implements Debug; the `.err().expect(..)` stand-ins were
failing the base branch's clippy step (clippy::err_expect under
-D warnings). The site replication repair call edge
keeps its conversion because the repair executor still answers with the
s3s response type.

Drop the dead_code expectations on the shared owned helpers this batch
now uses in non-test code.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* fix(admin): preserve legacy body limit error responses

* refactor(admin): move policy, ILM and batch 3 handlers to owned requests (#8956)

Batch 3 of T1.12 (rustfs/backlog#2788, parent rustfs/backlog#2734) moves
14 admin handler files to AdminRequest/AdminResponse and the owned
Operation trait: ilm_transition, policies, idp_compat, config_admin,
plugins_instances, profile_admin, group, heal, object_zip_download,
kms_key_metadata, durability, inspect_archive, sts and
gateway_key_inventory. Handler logic, authorization order, error codes
and response bytes are unchanged; test request literals use
test_support::TestRequest.

The handlers use the shared owned helpers (owned_json_response,
owned_admin_json_response, owned_build_json_response,
authorize_profile_envelope). The dead_code expectations on the
AdminRequest accessors and RequestEnvelope::uri_mut they now exercise
are removed. The legacy target_descriptor::build_json_response keeps
only its equivalence test caller and carries the matching expectation
until it retires.

object_zip_download builds the s3s request the S3 gate still reads from
the owned request facts; heal authenticates through
authenticate_envelope_credentials; the trace, inspect-archive and zip
bodies use Body::from_stream. extract_plugin_instance_filters takes the
URI so extensions.rs keeps calling it, and the two idp_compat
delegations stay on the legacy trait until their delegates migrate.

The s3s footprint baseline drops from 99 to 90 files.

* test(admin): anchor the replication diff body contract on the call

The source-contract test looked for `let body = read_compatible_admin_body(`
on one line; rustfmt wrapped that binding after the handler moved to the
owned request boundary, so the marker was no longer found. Anchor on the
call itself so formatting cannot change what the contract checks.

* refactor(admin): migrate table catalog and batch 1 to owned requests (#8955)

* fix(admin): preserve catalog body limit response bytes

* refactor(admin): migrate identity and cluster handlers to owned requests (#8966)

* refactor(admin): migrate identity and cluster handlers to owned requests

Move the T1.12 batch 4 admin handlers (user, service_account, oidc, tier,
system, pools, quota, kms_dynamic, kms_backup, rebalance, module_switch,
bucket_meta, cluster_snapshot, realtime, bucket_recovery, profile) from the
s3s request/response types to the owned AdminRequest/AdminResponse
boundary. All 90 impl Operation blocks now take AdminRequest and return
AdminResponse; handler logic, authorization order, error codes and response
bytes are unchanged. The handlers use the shared owned helpers
(owned_json_response, owned_admin_json_response, authorize_profile_envelope,
the envelope-form runtime sources and audit constructors) instead of local
copies.

profile.rs drops the s3s forms of authorize_profile_request and
profile_not_implemented_response now that profile_admin.rs is on the
envelope gate too; profile_not_implemented_response answers with an
AdminResponse. The realtime metrics stream moves from the s3s ByteStream
adapter to Body::from_stream, and the rebalance idle-404 document still
renders through s3s::S3Error::to_http_response with its body converted at
the edge.

The idp_compat delegators ImportIamV2 and AddServiceAccountLdap move to the
owned trait as well: with their targets on both traits, the method-call
delegation became ambiguous between the owned impl and the router blanket
impl, so they now take the owned request directly.

KmsAdminAudit::from_request loses its last production caller with this batch
and is deleted; its comparison test now asserts the envelope facts directly.

Lower the s3s footprint baseline to the measured 68 files.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* refactor(admin): remove the legacy admin helpers batch 4 superseded

Batch 4 moved the last production callers of the s3s-typed admin helpers
to their owned forms, which left `admin_json_response`, `json_response`,
`authenticate_request`, `authorize_admin_request`, `object_store_from_req`
and `object_store_from_extensions` unused, along with the `Body`,
`S3Request` and `S3Response` re-exports of the admin `s3` facade. Remove
them. The owned forms keep their documentation; the behavior tests of the
removed `json_response` now exercise `owned_json_response`, and the
compat-prefix test asserts the owned response directly.

Two tests existed only to compare a removed legacy helper with its owned
twin (`owned_json_response_matches_the_legacy_shape`) or to cover the
removed legacy gate (`authorize_admin_request_without_credentials_is_rejected`);
they go with the helpers, and the owned equivalents stay. The rustfs
test-count row drops from 5957 to 5955 and the s3s footprint from 68 to 66
files.

* refactor(admin): drop the legacy target descriptor JSON response helper (#8968)

`target_descriptor::build_json_response` returned the s3s response and
had no production caller left once the event, audit and plugin instance
handlers moved to `owned_build_json_response`; it survived only for the
legacy/owned equivalence test. Remove it, give the owned helper its own
documentation, and make the test assert the owned response directly:
status, `application/json`, the echoed `x-request-id` when present and
only then, and the body bytes. The file no longer imports s3s, so the
footprint baseline drops from 66 to 65 files.

* test(ecstore): pin native history on cancellation under the cancellable queue

main moved the local write path onto the disk blocking queue, which skips an
operation once its waiter is dropped, so a borrowed write cancelled while it is
queued no longer creates its file. The integration-only native publication
test still expected the queued open to run. Assert the new execution model
instead (the open never runs) while keeping its safety claims: the Started
record is taken before queueing and the cancelled publication never regrants
native history.

* chore(gateway): raise test-count rows for tests merged from main

* test(s3select): use the owned JSON input types in merged planner tests

* fix(ecstore): scope the merged direct-writer test hook to test builds

* fix(ecstore): create parents before admitted truncate opens

Align the heal tests with removal-disabled preservation and local-only outcome
fields, and refresh the e2e-full selection for the synced tests.

* chore(gateway): update test-count baseline after main sync

Main #9241 removed the node pool-metadata readiness cache with 8 tests
and added 6, so the rustfs row drops from 6440 to 6438. The ecstore and
s3select rows rise for tests added by #9237, #9239 and #9241.

* feat(scripts): add junit_diff.py for dual-stack JUnit comparison (#9275)

* refactor(auth): share authorization core with owned entry points

Extract the policy decision behind authorize_request into a core that both the legacy s3s request and the RustFS-owned RequestEnvelope feed, and move the zip-create and table-catalog admin callers to the owned entry points (authorize_owned_request, authorize_owned_internal_object_request). Request-body policy inputs added on main since the candidate base (POST form headers, copy source, legal hold and tagging bodies) become named envelope fields so both edges reach the same decision. Refs rustfs/backlog#2749.

* ci(gateway): make the test-count ratchet a floor, not exact (#9278)

ci(gateway): make the test-count ratchet a floor instead of an exact baseline

Tests below a row still fail; tests above it, a member without a row and ignores below the row now pass with a note. Ignores above a row still fail. Adds contract tests wired into make script-tests.

* refactor(admin): move remaining admin handlers to owned requests

Migrate the ten table-catalog commit and publication handlers to the
owned admin Operation trait, redeem object zip download tokens through
authorize_owned_request, and remove the s3s-typed router Operation
trait. S3Router now converts the s3s request into an AdminRequest and
the AdminResponse back at its edge, so no admin operation takes an s3s
type (rustfs/backlog#2788).

* chore(gateway): record ignored microbenchmarks merged from main (#9290)

chore(gateway): record the ignored microbenchmarks merged from main

main added three throughput microbenchmarks that are ignored by design
(#9217): one each in rustfs-checksums, rustfs-rio and rustfs-utils. The
test-count ratchet treats a higher ignored count as a failure until the
row is raised, so raise those three rows to 1 and tighten the test rows
to the merged tree with --update.

* fix(auth): keep owned authorization in the app layer and fail closed

Move the RequestEnvelope impl of AuthorizationRequest and the owned entry points into the app layer so storage depends only on the request-facts trait (layer guard). The owned entry points now hoist signed x-amz-meta-* query parameters like the legacy edge and refuse actions whose decision needs a request-body input the envelope does not carry. Revert unrelated TODO rewording and redact PostObjectPolicyHeaders debug output.

* test(auth): move owned parity tests to the app layer

Owned-versus-legacy parity, hoist and fail-closed tests now live in app/owned_authorization_test.rs beside the envelope's facts impl, so storage names no app type in test code either. Private storage steps are reached through a test-only parity_test_support module. Also fix the S3Error conversion CI reported in prepare_owned_authorization.

* test(admin): convert the tagged table catalog request before installing identity

* ci(s3-tests): add reference parity lane against pinned 1.0.5 (#9288)

* fix(admin): keep owned router entry points crate-visible

* test(auth): read storage-private parity facts through test accessors

The parity tests moved to the app layer can no longer read ObjectTagConditions and TableDataPlanePublicationGuards fields directly; parity_test_support exposes read-only accessors instead. Drop an unused import.

* test(auth): owned and legacy entry points reach the same decisions

Decision-level parity over IAM allow and explicit deny, owner, anonymous on private and public buckets, bucket-policy explicit deny, aws:SourceIp from the peer and from a verified client, and one allow and one deny per body-derived input (legal hold, tagging, retention mode and retain-until, resolved copy source, POST form fields).

* fix(admin): drop the owned trait bound from S3Router's inherent impls

* test(auth): satisfy clippy on the parity test facade

* test(admin): cover zip download token redemption authorization

* test(auth): report every decision mismatch in the parity matrix

* test(auth): cover multipart staging in the owned/legacy decision matrix

* fix(admin): authorize warehouse locations with the owned commit request

* fix(admin): return the owned table catalog authorization result unchanged

* fix(table-catalog): restore owned request lines the main merge reverted

The merge onto main replaced two lines with main's text: the warehouse location check again wrapped an already owned request in legacy_request_view, and an error conversion was re-added. Restore the branch's version of the file.

* fix(auth): make an absent caller secret unrepresentable as a key

RequestCredentials::secret_key now returns Option<&str>: None when the edge kept the secret, and an empty handed-over secret is refused at construction. Secret comparisons go through secret_key_matches, which never matches an absent secret, and readers that need the value fail closed with an internal error.

* fix(auth): keep the test-only credentials rebuild out of the lib build

* fix(app): drop the PutObjectInput re-export main now provides

---------

Co-authored-by: rustfs-cla-assistant[bot] <273121464+rustfs-cla-assistant[bot]@users.noreply.github.com>
2026-11-11 16:01:44 -05:00
Chris An 09811acf91 ci(parity): run dual-stack differential against pinned release (#9315)
* feat(server): add the RUSTFS_S3_STACK selector and dual-stack e2e lane (#8411)

* refactor(server): group the s3s-compat tower layers under legacy_compat (#8410)

Move the ten tower layers that patch s3s behaviour out of
rustfs/src/server/layer.rs into rustfs/src/server/legacy_compat/, one file
per layer with its tests. Each file header names the s3s behaviour it
patches, the gateway mechanism that replaces it (Replaced by:), and the task
that removes it (Removed by: T2.12, rustfs/backlog#2771). The layers keep
their code, their order in external_service_stack!, and their behaviour; no
layer is deleted.

Add server::http::tests::stack_census, which renders the ordered service
chain of the real external_service_stack! expansion and compares it with a
committed constant, so a reorder, insertion or removal is a visible diff.

Route the two s3s error types the moved layers use through a
storage_api::server::legacy_compat shim so the split adds no s3s-importing
file, and lower the s3s footprint baselines to the measured values (files 209
to 208, s3_error! lines 1586 to 1585). Keep the server runtime-source bypass
guard covering the moved code.

Refs rustfs/backlog#2787, rustfs/backlog#2734

* feat(s3-types): add the RustFS-owned S3 error type and s3_error! macro (#8406)

`rustfs-s3-types` now provides `S3ErrorCode` (generated by build.rs from
src/codes.txt, the grep-derived call surface, plus `Custom`), `S3Error`,
`S3Result`, `StdError` and the `s3_error` macro with the same call syntax
as s3s, so the T1.2 codemod is a `use` swap. Statuses are hand-written and
pinned to the s3s behaviour RustFS ships today by an oracle test that uses
s3s as a dev-dependency only. The `From` conversions for the legacy edge
live behind the off-by-default `compat-s3s` feature (the orphan rule
forbids them in rustfs/src/error.rs); `rustfs` enables it.

Refs rustfs/backlog#2742, rustfs/backlog#2734

* ci: add the per-crate test-count ratchet (#8404)

Quick Checks now fails any PR that lowers the number of test attributes
in a workspace crate or adds #[ignore], measured per crate against the
exact rows of .config/test-count-baseline.txt (rustfs/backlog#2734
section 1 rule 3). The counter widens the spec's literal
#[(tokio::)?test] to the tokio attributes that carry arguments and both
spellings of #[test_case(...)], and anchors at line start so
commented-out attributes do not count. The initial baseline covers 52
crates, 21077 tests and 164 ignored.

Refs rustfs/backlog#2738
Refs rustfs/backlog#2734

* test(ecstore): pin the persisted bucket-config XML bytes for 13 families (#8408)

Before any DTO or codec is replaced, pin the exact bytes that
deserialize followed by serialize produces for every persisted
bucket-configuration XML family in its three source shapes (AWS
documentation, MinIO-written, RustFS-written), and the bytes RustFS
writes today for a typed value. Byte equality only: a structural
comparison would let a codec replacement change the persisted encoding
unnoticed.

The generator script refuses to overwrite existing goldens without
--force, refuses on dirty codec paths, cannot report success from its
write mode, and records the generating commit in the fixtures README.

Refs rustfs/backlog#2744, rustfs/backlog#2734

* ci: raise the test-count baseline to the integration tree (#8426)

* ci: make the s3s footprint baseline exact and lower-only (#8403)

* ci: make the s3s footprint baseline exact and lower-only

The s3s footprint guard compared three counters against ceilings held in
the script itself, so a ceiling could sit above the tree and a drift was
fixable by raising it (rustfs/rustfs#5964 raised the s3_error! ceiling
to go green). The three values now live in
.config/s3s-footprint-baseline.txt as files, s3_error_lines and
ecstore_files, the script reads only that file, and a measurement that
differs from its baseline in either direction fails: growth prints
"s3s footprint grew: <measured> > <baseline>", shrinkage prints
"lower the baseline to <measured> in this PR". The counting method is
unchanged; the baseline is the current measurement of
gateway/integration (209 / 1585 / 36). The header records the end state:
once all three reach 0, T4.2 switches the script to failing on any
occurrence.

Refs rustfs/backlog#2739, rustfs/backlog#2734

* ci: add the s3s edge allowlist mode to the footprint guard (#8405)

`scripts/check_s3s_footprint.sh --mode allowlist` fails on every Rust
file that references s3s (`s3s::` or `s3s_sigv4`) and is not matched by
an entry of the new `.config/s3s-edge-allowlist.txt`; `--dry-run` lists
those files and exits 0. The allowlist carries the initial edge from the
spec, and the script rejects an entry that names or matches anything
under crates/ other than crates/e2e_test. The baseline mode and the
argument-less CI invocation are unchanged.

Refs rustfs/backlog#2741, rustfs/backlog#2734

* test(ecstore): reach the bucket-config codec through the test facade (#8428)

* refactor(s3-types): own the HTTP Body and drop s3s from three crates (#8431)

Add rustfs_s3_types::Body, a BoxBody<Bytes, StdError> wrapper with
empty() and From<Bytes | Vec<u8> | String | &'static str>, and the
legacy From conversions to and from s3s::Body behind compat-s3s.

The signer is now generic over the request body: it only reads and
writes the request head, so it needs no body type and its callers keep
theirs. s3-client sends its requests with the owned Body, madmin signs a
bodiless head, and targets drops its two S3Request/S3Response header
wrappers in favour of extract_params_header.

rustfs-signer, rustfs-madmin and rustfs-targets no longer depend on
s3s; the s3s footprint baseline drops from 209 to 202 files.

Refs rustfs/backlog#2747, rustfs/backlog#2734

* refactor(ecstore): move S3ErrorCode to rustfs-s3-types by codemod (#8432)

* test(ecstore): pin the InvalidRange responses of new_getobjectreader

Both error branches of new_getobjectreader build an ErrorResponse with S3ErrorCode::InvalidRange, and no test observed either code. Pin the code, its wire name and the location fields of each branch before the code type moves to rustfs-s3-types (rustfs/backlog#2743), so a changed code on either branch fails a test. The ecstore test-count row rises by the two tests.

* chore(scripts): add the s3s-to-rustfs-s3-types error codemod

Task T1.2 of rustfs/backlog#2734 (rustfs/backlog#2743) moves S3Error, S3ErrorCode, S3Result and s3_error! from s3s to rustfs-s3-types by script, one crate closure per PR. The codemod rewrites use trees (splitting mixed s3s imports) and qualified code paths, never comments or literals, adds the rustfs-s3-types dependency, drops s3s only from a crate that no longer names it, refuses every file of the s3s edge allowlist, takes --hold for a file whose type is fixed outside the closure, and prints a per-crate census. A second run changes nothing. Its self-test runs under make script-tests.

* refactor(ecstore): take S3ErrorCode from rustfs-s3-types

Output of scripts/codemods/s3_error_to_s3_types.py over crates/ecstore, crates/lifecycle, crates/scanner and crates/replication (rustfs/backlog#2743 batches 1 and 2). The ErrorResponse code field, the two InvalidRange constructions in new_getobjectreader and an unused tier import now use rustfs_s3_types::S3ErrorCode; the variant names and wire names are unchanged. lifecycle, scanner and replication reference none of the four error names. warm_backend.rs and warm_backend_s3.rs are held: they compare against rustfs_s3_client::ErrorResponse.code, which stays the s3s type until s3-client moves.

* ci(footprint): count only s3_error! lines still on s3s's macro

s3_error_lines counted every line holding the text `s3_error!`, which
also matches the RustFS-owned `rustfs_s3_types::s3_error!`. A codemod
that moves call sites onto rustfs-s3-types therefore could not lower it
(rustfs/backlog#2743, decision 3; rustfs/rustfs#8432, gotcha 2).

The counter now applies a textual, per-file and per-line rule that errs
on the side of counting (full rule in the script header):
- occurrences spelled `rustfs_s3_types::s3_error!` never count;
- a line holding `s3s::s3_error!` always counts;
- any other `s3_error!` line counts unless its file is cleared, that is,
  it imports the macro from rustfs_s3_types by name on a line starting
  with the use item (or defines it with `macro_rules! s3_error`) and does
  not also bring s3s's macro in (`use s3s::s3_error`, a `use s3s::{..}`
  tree naming it or a crate-root `*`, `use s3s::*`, or
  `#[macro_use] extern crate s3s`);
- fail closed: a file matching neither form (`use super::*`, a
  crate-local re-export, a renamed or glob rustfs_s3_types import, a
  commented-out import, a comment-only mention) counts.

Census of the 1585 lines on 96fc06776: 1478 in 64 files that import the
macro from s3s; 107 in 15 files matching neither form (105 through
`use super::*`, 1 qualified `s3s::s3_error!`, 1 comment line); none in
files that import it from rustfs_s3_types or define it. Every line stays
counted, so the baseline remains s3_error_lines=1585 (files=202 and
ecstore_files=36 unchanged). Rewriting rustfs/src/auth.rs's import to
rustfs_s3_types lowers the new counter to 1567; the old one stayed 1585.

`--self-test` (wired into `make script-tests`) copies the script into
22 fixture trees: 6 positive cases and 16 negative ones (lines that must
not count, forms that must not clear a file, and a missing baseline,
missing key or failing classification search that must fail). Each tree
has a control line that must count, and each run gets s3s text on stdin.
Seven cases were red against the old counter. 19 mutants each turn their
case red: old definition, flipped import test, dropped fail-closed
branch, dropped e2e_test exclusion, dropped s3s veto, dropped qualified
s3s rule, dropped rustfs-qualified strip, unanchored rustfs import,
renamed import accepted, nested glob taken as root glob, single-line
s3s trees only, dropped s3s glob forms, dropped macro_use form, dropped
definition clearing, ignored classification rg errors, error-line search
without its path, classification searches without their path, missing
baseline skipped, missing key skipped.

* test(e2e): compare two RUSTFS_S3_STACK servers over every S3 operation (#8435)

Add dual_stack_differential to e2e_test. It starts two rustfs processes
from the same binary, stack A on RUSTFS_S3_STACK=legacy and stack B on
RUSTFS_E2E_STACK_B (default legacy until rustfs/backlog#2734 T2.5),
sends both the same aws-sdk-s3 script covering the 73 methods of
impl S3 for FS (a normal request and a 4xx negative each), and compares
status, error code, an allowlist of response headers and a few typed
body fields. Error message text is never compared. Each run writes
<target>/dual-stack/<run>.json with compared_ops and the (op, field,
a, b) differences; any difference fails the test.

differential_detects_an_injected_difference starts B with
RUSTFS_REGION=eu-west-1 and requires GetBucketLocation to be reported.
The comparator is pure and unit-tested without a server.

Wire the module into the e2e-smoke filter and update the e2e-smoke and
e2e-full selection digests and the e2e_test test-count row.

Refs rustfs/backlog#2740

* feat(app): add the RustFS-owned RequestEnvelope and its s3s constructor (#8437)

* feat(app): add the RustFS-owned RequestEnvelope and its s3s constructor

* ci: classify the s3s edge adapters in storage/s3_compat as interface

* feat(s3-types): add Body::from_stream and move Swift off s3s::Body (#8438)

rustfs/backlog#2747 batch 3. `Body::from_stream` wraps a chunk stream of
unknown length over `StreamBody`/`BoxBody`: one data frame per chunk, the
first error ends the body, the size hint stays unknown, and dropping the
body drops the stream. Parity tests pin it to the s3s `StreamingBlob::wrap`
it replaces, at the `http_body` level and byte for byte through hyper.

The Swift handlers in rustfs-protocols build their responses as the owned
`Body`. `SwiftService` converts them into the wrapped S3 service's response
body type through `From<Body>`, so rustfs keeps serving `s3s::Body` at the
legacy edge unchanged. Request bodies stay Send-only (`UnsyncBoxBody`); the
two storage writers and the DLO/SLO segment streams get `Sync` from
`sync_wrapper::SyncStream`, which is what s3s's `http_body_unsync` mutex
provided before. The DTO and error uses of s3s in protocols are untouched
(T1.5 and T1.2).

* refactor(protocols): move error types to rustfs-s3-types by codemod (#8439)

T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and
the error items of protocols move off s3s by scripts/codemods, run with
--hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it
downcasts the backend error, whose runtime type is still s3s::S3Error.

- rustfs/src/protocols/client.rs: the two webdav session methods now
  return rustfs_s3_types::S3Result and convert the s3s error with From.
- codemod: a child module that calls s3_error! through `use super::*`
  gets its own `use rustfs_s3_types::s3_error;` once its parent imports
  that macro by name; otherwise it is listed as GLOB.
- s3-types: a table test pins code, status and the s3s error the legacy
  edge renders for every variant a moved call site names.
- s3_error_lines 1585 -> 1584 (a lifecycle comment reworded).

* test(connect): restore one-second drive request budgets

Apply the exact test-only repair already landed on main in 5972a09cc3. The filesystem probe and production CLI success test request one second and keep their existing one-second duration and parameter checks. All outcome, I/O, error, cleanup and resource-limit assertions remain.

Scope: rustfs/tests/connect_perf_drive.rs only. Patch-id: cb0a1dfc8f68a67727525adea7458e2cee7d09f8.

Validation environment: LANG=en_US.UTF-8, CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0, exclusive APFS-cloned CARGO_TARGET_DIR.

Quick Checks: every command from .github/actions/quick-checks/action.yml, with captured output and exit status below.

$ shellcheck --version && actionlint
ShellCheck - shell script analysis tool
version: 0.11.0
license: GNU General Public License, version 3
website: https://www.shellcheck.net
exit_code=0

$ cargo fmt --all --check
exit_code=0

$ ./scripts/check_unsafe_code_allowances.sh
exit_code=0

$ ./scripts/check_layer_dependencies.sh
Layer dependency guard passed (no new reverse dependencies/cycles).
exit_code=0

$ ./scripts/check_architecture_migration_rules.sh
Architecture migration rules passed.
exit_code=0

$ ./scripts/check_logging_guardrails.sh
✅ Logging guardrails check passed
exit_code=0

$ ./scripts/check_error_other_format_ratchet.sh
error(format!) ratchet passed (total call sites: 582).
exit_code=0

$ ./scripts/check_no_tokio_io_uring.sh
exit_code=0

$ ./scripts/check_extension_schema_boundaries.sh
Extension schema boundaries passed.
exit_code=0

$ ./scripts/check_body_cache_whitelist.sh
Body-cache whitelist guard passed.
exit_code=0

$ ./scripts/check_s3s_footprint.sh
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)
✅ s3s footprint baseline check passed
exit_code=0

$ ./scripts/check_test_count_ratchet.sh
test-count ratchet OK: 52 crates, 21222 tests, 164 ignored match .config/test-count-baseline.txt
exit_code=0

$ ./scripts/check_fips_wording.sh
FIPS wording guard passed (README.md and CHANGELOG.md contain no forbidden claims).
KMS wording guard passed (crates/kms claims no Vault KV2 Transit wrapping).
exit_code=0

$ ./scripts/check_embedded_secrets.sh
Embedded secret guard passed (no private key material or provider credential literal in the tree).
exit_code=0

$ make script-tests
Running script tests...
entrypoint credential policy tests passed
internode grpc A/B bench env tests passed
object batch benchmark enhanced tests passed
PASS package service script contract
hotpath warp A/B gate tests passed
hotpath warp ABBA tests passed
Scanner validation artifacts written to /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.1pFe6Kl1Yu/out
Scanner validation artifacts written to /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.1pFe6Kl1Yu/out-missing-pid
.............................
----------------------------------------------------------------------
Ran 29 tests in 181.286s

OK
{"decision":"verified","pending_gates":["G01","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G02"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-E"}
{"decision":"verified","pending_gates":["G01","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G02"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-E"}
{"decision":"verified","pending_gates":["G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G01"}
{"decision":"verified","pending_gates":["G01","G02","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G03"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-L"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G14"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G14"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G10"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P1"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P2"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P3"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G05"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G06"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-D"}
Status-and-outcome release descriptor verified: /private/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp8rwn1pe0/descriptor/release-bundle-status-outcome.json
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G05"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G06"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-D"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G07"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G08"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P4"}
exact 1MiB handoff ABBA tests passed
pinned paired ABBA bench tests passed
==> /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/bin/path_containment (-max_total_time=7, -seed=123456789, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/explicit-artifacts/path_containment/, corpus=/private/tmp/gw-orchestrator-2734/ci-drive/fuzz/corpus/path_containment)
-seed=123456789
target=path_containment
seed=123456789
max_total_time=7
git_revision=4d15b816236630b2c42f321ffc9ed3654edc94e3
git_dirty=true
runner_mode=prebuilt
==> /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/bin/path_containment (-max_total_time=1, -seed=2322617706, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/auto-artifacts/path_containment/, corpus=/private/tmp/gw-orchestrator-2734/ci-drive/fuzz/corpus/path_containment)
-seed=2322617706
==> cargo +nightly fuzz run path_containment (-max_total_time=9, -seed=987654321, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/cargo-artifacts/path_containment/)
+nightly
fuzz
run
path_containment
-seed=987654321
seed=987654321
runner_mode=cargo-fuzz
fuzz runner seed manifest tests passed
✅ scripts/python_bin.sh resolver checks passed
self-test ok: clean fixture with every exemption present passes
self-test ok: all 11 pattern families fire
self-test ok: moving a pinned source is reported
Embedded secret guard self-test passed.
self-test ok: p1-s3s-import: 'use s3s::s3_error;' counts every line, the rustfs_s3_types module's too (s3_error_lines=3)
self-test ok: p2-s3s-mixed-tree: a multi-line s3s use tree naming s3_error counts (s3_error_lines=3)
self-test ok: p3-s3s-glob: a crate-root glob of s3s, bare or inside a tree, counts (s3_error_lines=5)
self-test ok: p4-macro-use: '#[macro_use] extern crate s3s' counts (s3_error_lines=3)
self-test ok: p5-qualified-s3s: a qualified s3s::s3_error! line counts in a cleared file (s3_error_lines=2)
self-test ok: p6-exact-pass: a baseline equal to the tree passes (exit 0)
self-test ok: n1-rustfs-import: 'use rustfs_s3_types::s3_error;' alone counts nothing (s3_error_lines=1)
self-test ok: n2-rustfs-mixed-tree: a rustfs_s3_types tree plus an s3s tree without the macro counts nothing (s3_error_lines=1)
self-test ok: n3-defines: the file defining the macro counts nothing (s3_error_lines=1)
self-test ok: n4-qualified-rustfs: a qualified rustfs_s3_types::s3_error! line counts nothing (s3_error_lines=1)
self-test ok: n5-renamed-call: a call through a renamed macro is not text-matched (out of scope) (s3_error_lines=1)
self-test ok: n6-comment-string: a comment or string mentioning s3_error! in a cleared file counts nothing (s3_error_lines=1)
self-test ok: n7-e2e-excluded: crates/e2e_test is excluded (s3_error_lines=1)
self-test ok: n8-super-glob: an unrecognised import form ('use super::*;') counts (s3_error_lines=3)
self-test ok: n9-facade: a crate-local re-export counts (s3_error_lines=2)
self-test ok: n10-renamed-import: a renamed rustfs_s3_types import does not clear s3_error! lines (s3_error_lines=2)
self-test ok: n11-commented-import: a commented-out rustfs_s3_types import does not clear (s3_error_lines=2)
self-test ok: n12-rustfs-glob: a glob import of rustfs_s3_types does not clear (s3_error_lines=2)
self-test ok: n13-comment-only: a comment mentioning s3_error! in a file with no import counts (s3_error_lines=2)
self-test ok: f1-baseline-missing: a missing baseline file fails (exit 1)
self-test ok: f2-key-missing: a baseline without s3_error_lines fails (exit 1)
self-test ok: f3-rg-fails: a failing classification search fails the guard (exit 1)
✅ s3s footprint self-test passed
test_core_gate_rejects_missing_ignored_filtered_and_corrupt_inputs (__main__.SelfTests.test_core_gate_rejects_missing_ignored_filtered_and_corrupt_inputs) ... ok
test_e2e_requires_registration (__main__.SelfTests.test_e2e_requires_registration) ... ok
test_fuzz_runtime_uses_matrix_target (__main__.SelfTests.test_fuzz_runtime_uses_matrix_target) ... ok
test_ilm_lane_keeps_the_measured_cargo_build_budget (__main__.SelfTests.test_ilm_lane_keeps_the_measured_cargo_build_budget) ... ok
test_profile_listing_binds_platform_digest (__main__.SelfTests.test_profile_listing_binds_platform_digest) ... ok
test_profile_listing_enforces_selection (__main__.SelfTests.test_profile_listing_enforces_selection) ... ok
test_quick_checks_commands_propagate_failure (__main__.SelfTests.test_quick_checks_commands_propagate_failure) ... ok
test_quick_checks_rejects_caller_and_execution_bypasses (__main__.SelfTests.test_quick_checks_rejects_caller_and_execution_bypasses) ... ok
test_runner_readiness_fails_closed (__main__.SelfTests.test_runner_readiness_fails_closed) ... ok
test_s3_runner_rejects_unbounded_failure_locals (__main__.SelfTests.test_s3_runner_rejects_unbounded_failure_locals) ... ok
test_scanner_heal_begin_requires_embedded_source_provenance (__main__.SelfTests.test_scanner_heal_begin_requires_embedded_source_provenance) ... ok
test_scanner_heal_case_does_not_approve_pending_release (__main__.SelfTests.test_scanner_heal_case_does_not_approve_pending_release) ... ok
test_scanner_heal_case_lane_is_required_for_runner_profile (__main__.SelfTests.test_scanner_heal_case_lane_is_required_for_runner_profile) ... ok
test_scanner_heal_case_only_schema_cannot_approve_release (__main__.SelfTests.test_scanner_heal_case_only_schema_cannot_approve_release) ... ok
test_scanner_heal_crash_case_rejects_restart_oracle (__main__.SelfTests.test_scanner_heal_crash_case_rejects_restart_oracle) ... ok
test_scanner_heal_ec84_case_rejects_wrong_erasure_geometry (__main__.SelfTests.test_scanner_heal_ec84_case_rejects_wrong_erasure_geometry) ... ok
test_scanner_heal_finish_collects_oracles_from_registry (__main__.SelfTests.test_scanner_heal_finish_collects_oracles_from_registry) ... ok
test_scanner_heal_multipool_case_accepts_deferred_optional_outage_write (__main__.SelfTests.test_scanner_heal_multipool_case_accepts_deferred_optional_outage_write) ... ok
test_scanner_heal_pending_gate_cannot_map_to_implemented_lane (__main__.SelfTests.test_scanner_heal_pending_gate_cannot_map_to_implemented_lane) ... ok
test_scanner_heal_receipts_reject_reuse_and_missing_builds (__main__.SelfTests.test_scanner_heal_receipts_reject_reuse_and_missing_builds) ... ok
test_scanner_heal_rejects_boolean_fractional_and_out_of_geometry_integers (__main__.SelfTests.test_scanner_heal_rejects_boolean_fractional_and_out_of_geometry_integers) ... ok
test_scanner_heal_rejects_broken_execution_and_artifacts (__main__.SelfTests.test_scanner_heal_rejects_broken_execution_and_artifacts) ... ok
test_scanner_heal_rejects_copied_junit_and_wrong_suite_build (__main__.SelfTests.test_scanner_heal_rejects_copied_junit_and_wrong_suite_build) ... ok
test_scanner_heal_release_bundle_accepts_complete_measured_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_accepts_complete_measured_evidence) ... ok
test_scanner_heal_release_bundle_assembler_copies_measured_artifacts (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_copies_measured_artifacts) ... ok
test_scanner_heal_release_bundle_assembler_merges_measured_descriptors (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_merges_measured_descriptors) ... ok
test_scanner_heal_release_bundle_assembler_rejects_duplicate_descriptor_gate (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_duplicate_descriptor_gate) ... ok
test_scanner_heal_release_bundle_assembler_rejects_fixture_descriptor (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_fixture_descriptor) ... ok
test_scanner_heal_release_bundle_assembler_rejects_hash_mismatch (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_hash_mismatch) ... ok
test_scanner_heal_release_bundle_assembler_rejects_mixed_descriptor_revisions (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_mixed_descriptor_revisions) ... ok
test_scanner_heal_release_bundle_enforces_topology_duration_profile_and_versions (__main__.SelfTests.test_scanner_heal_release_bundle_enforces_topology_duration_profile_and_versions) ... ok
test_scanner_heal_release_bundle_fixture_writer_is_not_release_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_fixture_writer_is_not_release_evidence) ... ok
test_scanner_heal_release_bundle_gate_accepts_single_measured_gate (__main__.SelfTests.test_scanner_heal_release_bundle_gate_accepts_single_measured_gate) ... ok
test_scanner_heal_release_bundle_gate_rejects_missing_field (__main__.SelfTests.test_scanner_heal_release_bundle_gate_rejects_missing_field) ... ok
test_scanner_heal_release_bundle_gate_rejects_unknown_gate (__main__.SelfTests.test_scanner_heal_release_bundle_gate_rejects_unknown_gate) ... ok
test_scanner_heal_release_bundle_rejects_synthetic_or_missing_fields (__main__.SelfTests.test_scanner_heal_release_bundle_rejects_synthetic_or_missing_fields) ... ok
test_scanner_heal_release_bundle_requires_domain_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_requires_domain_evidence) ... ok
test_scanner_heal_release_bundle_requires_field_provenance (__main__.SelfTests.test_scanner_heal_release_bundle_requires_field_provenance) ... ok
test_scanner_heal_release_bundle_requires_json_artifact_provenance (__main__.SelfTests.test_scanner_heal_release_bundle_requires_json_artifact_provenance) ... ok
test_scanner_heal_release_bundle_requires_mixed_version_field_roles (__main__.SelfTests.test_scanner_heal_release_bundle_requires_mixed_version_field_roles) ... ok
test_scanner_heal_release_bundle_requires_same_gate_measurement_window (__main__.SelfTests.test_scanner_heal_release_bundle_requires_same_gate_measurement_window) ... ok
test_scanner_heal_release_consumes_multiple_registry_oracles (__main__.SelfTests.test_scanner_heal_release_consumes_multiple_registry_oracles) ... ok
test_scanner_heal_release_matrix_lanes_remain_pending_without_real_evidence (__main__.SelfTests.test_scanner_heal_release_matrix_lanes_remain_pending_without_real_evidence) ... ok
test_scanner_heal_release_requirements_cannot_be_cleared (__main__.SelfTests.test_scanner_heal_release_requirements_cannot_be_cleared) ... ok
test_scanner_heal_release_status_rejects_focused_case_run (__main__.SelfTests.test_scanner_heal_release_status_rejects_focused_case_run) ... ok
test_scanner_heal_release_status_rejects_synthetic_case (__main__.SelfTests.test_scanner_heal_release_status_rejects_synthetic_case) ... ok
test_scanner_heal_required_evidence_fields_cannot_be_removed (__main__.SelfTests.test_scanner_heal_required_evidence_fields_cannot_be_removed) ... ok
test_scheduled_alerts_require_completion_watchdog (__main__.SelfTests.test_scheduled_alerts_require_completion_watchdog) ... ok
test_update_profile_selection_changes_only_requested_platform (__main__.SelfTests.test_update_profile_selection_changes_only_requested_platform) ... ok
test_validate_includes_quick_checks (__main__.SelfTests.test_validate_includes_quick_checks) ... ok
test_vault_tests_require_cross_process_serialization (__main__.SelfTests.test_vault_tests_require_cross_process_serialization) ... ok
test_workflow_readiness_requires_dependency_probe (__main__.SelfTests.test_workflow_readiness_requires_dependency_probe) ... ok

----------------------------------------------------------------------
Ran 52 tests in 14.301s

OK
e2e-full selection OK: 1 tests, sha256=29e30e6ed2ed38d9e5b0a7cdd9536abf5c719f7021a02b578ae1381563a3c273
...................................
----------------------------------------------------------------------
Ran 35 tests in 1.161s

OK
...................
----------------------------------------------------------------------
Ran 19 tests in 17.397s

OK
..........
----------------------------------------------------------------------
Ran 10 tests in 0.183s

OK
test_actual_selector_bootstrap_uses_base_policy_and_fails_closed (__main__.SelfTests.test_actual_selector_bootstrap_uses_base_policy_and_fails_closed) ... ok
test_documentation_paths_do_not_hide_build_or_fixture_changes (__main__.SelfTests.test_documentation_paths_do_not_hide_build_or_fixture_changes) ... ok
test_e2e_scope_keeps_production_and_shared_configuration_full (__main__.SelfTests.test_e2e_scope_keeps_production_and_shared_configuration_full) ... ok
test_e2e_shortcut_rejects_new_direct_renamed_and_target_dependencies (__main__.SelfTests.test_e2e_shortcut_rejects_new_direct_renamed_and_target_dependencies) ... ok
test_e2e_shortcut_traverses_implicit_path_dependencies (__main__.SelfTests.test_e2e_shortcut_traverses_implicit_path_dependencies) ... ok
test_event_contract_requires_complete_candidate_and_optional_lanes (__main__.SelfTests.test_event_contract_requires_complete_candidate_and_optional_lanes) ... ok
test_every_wrong_result_missing_job_or_selection_fails_closed (__main__.SelfTests.test_every_wrong_result_missing_job_or_selection_fails_closed) ... ok
test_full_e2e_gate_preserves_workflow_branch_and_event_scope (__main__.SelfTests.test_full_e2e_gate_preserves_workflow_branch_and_event_scope) ... ok
test_git_range_includes_deleted_source_and_rename_origins (__main__.SelfTests.test_git_range_includes_deleted_source_and_rename_origins) ... ok
test_job_ids_and_display_names_cannot_hide_validation (__main__.SelfTests.test_job_ids_and_display_names_cannot_hide_validation) ... ok
test_report_only_stack_lane_mirrors_e2e_smoke_outside_the_aggregate (__main__.SelfTests.test_report_only_stack_lane_mirrors_e2e_smoke_outside_the_aggregate) ... ok
test_repository_wiring_and_missing_dependency_regression (__main__.SelfTests.test_repository_wiring_and_missing_dependency_regression) ... ok
test_verify_command_preserves_failures (__main__.SelfTests.test_verify_command_preserves_failures) ... ok

----------------------------------------------------------------------
Ran 13 tests in 3.280s

OK
security coverage self-test passed
test_api_failures_preserve_other_evidence_and_never_enter_grace (__main__.SelfTests.test_api_failures_preserve_other_evidence_and_never_enter_grace) ... ok
test_attempt_outcome_does_not_replace_recent_success (__main__.SelfTests.test_attempt_outcome_does_not_replace_recent_success) ... ok
test_cancelled_attempt_cannot_refresh_expired_success (__main__.SelfTests.test_cancelled_attempt_cannot_refresh_expired_success) ... ok
test_cli_requires_the_repository_default_branch (__main__.SelfTests.test_cli_requires_the_repository_default_branch) ... ok
test_config_rejects_duplicate_and_invalid_entries (__main__.SelfTests.test_config_rejects_duplicate_and_invalid_entries) ... ok
test_freshness_boundaries (__main__.SelfTests.test_freshness_boundaries) ... ok
test_grace_requires_two_successful_queries_with_no_history (__main__.SelfTests.test_grace_requires_two_successful_queries_with_no_history) ... ok
test_invalid_api_evidence_fails_closed (__main__.SelfTests.test_invalid_api_evidence_fails_closed) ... ok
test_report_retains_every_workflow (__main__.SelfTests.test_report_retains_every_workflow) ... ok
test_requests_filter_schedule_default_branch_and_success_on_server (__main__.SelfTests.test_requests_filter_schedule_default_branch_and_success_on_server) ... ok

----------------------------------------------------------------------
Ran 10 tests in 0.051s

OK
..............................
----------------------------------------------------------------------
Ran 30 tests in 22.354s

OK
..........
----------------------------------------------------------------------
Ran 10 tests in 2.969s

OK
.................................
----------------------------------------------------------------------
Ran 33 tests in 0.160s

OK
Online functional runner labels: pf-testing
.......
----------------------------------------------------------------------
Ran 7 tests in 0.002s

OK
.......
----------------------------------------------------------------------
Ran 7 tests in 0.001s

OK
....
----------------------------------------------------------------------
Ran 4 tests in 0.128s

OK
...
----------------------------------------------------------------------
Ran 3 tests in 0.547s

OK
PASS: cold stampede script self-test
exit_code=0

$ python3 ./scripts/test_object_generation_protocol_model.py
test_a_recovery_barrier_cannot_choose_an_empty_value (__main__.ObjectGenerationProtocolModel.test_a_recovery_barrier_cannot_choose_an_empty_value) ... ok
test_all_bounded_two_proposer_interleavings_choose_at_most_one_value (__main__.ObjectGenerationProtocolModel.test_all_bounded_two_proposer_interleavings_choose_at_most_one_value) ... ok
test_an_unavailable_decision_quorum_cannot_choose_a_candidate (__main__.ObjectGenerationProtocolModel.test_an_unavailable_decision_quorum_cannot_choose_a_candidate) ... ok
test_counter_allocation_requires_persistence_and_is_consumed_once (__main__.ObjectGenerationProtocolModel.test_counter_allocation_requires_persistence_and_is_consumed_once) ... ok
test_counter_restart_burns_persisted_allocations_before_or_after_issue (__main__.ObjectGenerationProtocolModel.test_counter_restart_burns_persisted_allocations_before_or_after_issue) ... ok
test_crash_after_promise_before_reply_does_not_restore_an_older_ballot (__main__.ObjectGenerationProtocolModel.test_crash_after_promise_before_reply_does_not_restore_an_older_ballot) ... ok
test_duplicate_or_unconfigured_voters_cannot_form_a_quorum (__main__.ObjectGenerationProtocolModel.test_duplicate_or_unconfigured_voters_cannot_form_a_quorum) ... ok
test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision (__main__.ObjectGenerationProtocolModel.test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision) ... ok
test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected (__main__.ObjectGenerationProtocolModel.test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected) ... ok
test_forgetting_tombstone_authority_allows_an_aba_create (__main__.ObjectGenerationProtocolModel.test_forgetting_tombstone_authority_allows_an_aba_create) ... ok
test_issuing_before_persistence_reuses_a_ballot_after_restart (__main__.ObjectGenerationProtocolModel.test_issuing_before_persistence_reuses_a_ballot_after_restart) ... ok
test_lost_ack_and_restart_preserve_the_chosen_operation (__main__.ObjectGenerationProtocolModel.test_lost_ack_and_restart_preserve_the_chosen_operation) ... ok
test_minority_accepted_work_is_adopted_and_finished (__main__.ObjectGenerationProtocolModel.test_minority_accepted_work_is_adopted_and_finished) ... ok
test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample (__main__.ObjectGenerationProtocolModel.test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample) ... ok
test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it (__main__.ObjectGenerationProtocolModel.test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it) ... ok
test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor (__main__.ObjectGenerationProtocolModel.test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor) ... ok
test_replacing_voters_with_empty_state_can_choose_a_conflicting_value (__main__.ObjectGenerationProtocolModel.test_replacing_voters_with_empty_state_can_choose_a_conflicting_value) ... ok
test_skipping_predecessor_recovery_chooses_a_broken_revision_chain (__main__.ObjectGenerationProtocolModel.test_skipping_predecessor_recovery_chooses_a_broken_revision_chain) ... ok
test_stale_publication_rollback_and_gc_cannot_change_the_winner (__main__.ObjectGenerationProtocolModel.test_stale_publication_rollback_and_gc_cannot_change_the_winner) ... ok
test_superseded_operation_identity_cannot_be_reused_for_different_input (__main__.ObjectGenerationProtocolModel.test_superseded_operation_identity_cannot_be_reused_for_different_input) ... ok
test_superseded_operation_retry_returns_its_original_successor (__main__.ObjectGenerationProtocolModel.test_superseded_operation_retry_returns_its_original_successor) ... ok
test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision (__main__.ObjectGenerationProtocolModel.test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision) ... ok
test_tombstone_rejects_a_delayed_never_present_create (__main__.ObjectGenerationProtocolModel.test_tombstone_rejects_a_delayed_never_present_create) ... ok
test_unlearned_predecessor_is_recovered_before_the_second_slot (__main__.ObjectGenerationProtocolModel.test_unlearned_predecessor_is_recovered_before_the_second_slot) ... ok

----------------------------------------------------------------------
Ran 24 tests in 0.958s

OK
model: states=51023, transitions=167520, terminal=732, chosen_candidates=2
negative control: double-choice trace=((0, 'prepare', 0), (0, 'prepare', 1), (0, 'prepare', 2), (0, 'accept', 0), (0, 'accept', 1), (0, 'accept', 2), (1, 'prepare', 0), (1, 'prepare', 1), (1, 'prepare', 2), (1, 'accept', 0), (1, 'accept', 1), (1, 'accept', 2))
multi-slot round schedules: 640
exit_code=0

$ python3 ./scripts/check_test_wiring.py
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired
exit_code=0

$ ./scripts/check_no_planning_docs.sh
exit_code=0

$ ./scripts/check_uring_lane_lib_only.sh
OK: no 'uring_'-matching test functions under crates/ecstore/tests; --lib is safe
exit_code=0

Initial complete gate
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
 Nextest run ID e07fc5b9-c6c8-4cdd-ae61-54839bf24437 with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.016s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.011s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.009s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.009s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  16.950s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   4.025s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.008s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.007s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.035s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  21.087s] 10 tests run: 10 passed, 0 skipped
exit_code=0

Restored complete gate
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
 Nextest run ID 233b7307-5361-4eb7-94d5-b384f17d2d6a with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.009s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.008s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.008s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.008s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  19.518s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   4.146s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.009s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.018s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.009s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.054s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  23.787s] 10 tests run: 10 passed, 0 skipped
exit_code=0

Mutation M1: success_measurement reports max(actual elapsed, requested budget + 1), forcing 1001 ms for the one-second request. The retained duration assertion at rustfs/tests/connect_perf_drive.rs:105 fails.
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast -E 'test(real_local_write_read_is_measured_and_scratch_is_removed)'
info: experimental features enabled: setup-scripts
   Compiling rustfs v1.0.1 (/private/tmp/gw-orchestrator-2734/ci-drive/rustfs)
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs") generated 1 warning
warning: `rustfs` (bin "rustfs-cli") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 7m 06s
────────────
 Nextest run ID 9f618777-7bf6-4cbc-8803-dd7f50f449ed with nextest profile: ci
    Starting 1 test across 1 binary (9 tests skipped)
        FAIL [   0.016s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
  stdout ───

    running 1 test
    test real_local_write_read_is_measured_and_scratch_is_removed ... FAILED

    failures:

    failures:
        real_local_write_read_is_measured_and_scratch_is_removed

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 0.01s

  stderr ───

    thread 'real_local_write_read_is_measured_and_scratch_is_removed' (106986846) panicked at rustfs/tests/connect_perf_drive.rs:105:5:
    assertion failed: (1..=1_000).contains(&data.duration_millis)
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

────────────
     Summary [   0.017s] 1 test run: 0 passed, 1 failed, 9 skipped
        FAIL [   0.016s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
error: test run failed
exit_code=100

Mutation M2: target_parameters reports requested duration + 1. The in-process assertion at rustfs/tests/connect_perf_drive.rs:111 and CLI JSON assertion at rustfs/tests/connect_perf_drive.rs:527 both reject 1001 against 1000.
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast -E 'test(real_local_write_read_is_measured_and_scratch_is_removed) | test(production_cli_measures_and_signs_exact_binary_provenance)'
info: experimental features enabled: setup-scripts
   Compiling rustfs v1.0.1 (/private/tmp/gw-orchestrator-2734/ci-drive/rustfs)
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs") generated 1 warning
warning: `rustfs` (bin "rustfs-cli") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 7m 20s
────────────
 Nextest run ID b73d46ab-c910-4e1e-8063-7fc7e332ec15 with nextest profile: ci
    Starting 2 tests across 1 binary (8 tests skipped)
        FAIL [  13.968s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
  stdout ───

    running 1 test
    test production_cli_measures_and_signs_exact_binary_provenance ... FAILED

    failures:

    failures:
        production_cli_measures_and_signs_exact_binary_provenance

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 13.96s

  stderr ───

    thread 'production_cli_measures_and_signs_exact_binary_provenance' (107015462) panicked at rustfs/tests/connect_perf_drive.rs:527:5:
    assertion `left == right` failed
      left: Number(1001)
     right: 1000
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

        FAIL [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
  stdout ───

    running 1 test
    test real_local_write_read_is_measured_and_scratch_is_removed ... FAILED

    failures:

    failures:
        real_local_write_read_is_measured_and_scratch_is_removed

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 0.01s

  stderr ───

    thread 'real_local_write_read_is_measured_and_scratch_is_removed' (107016667) panicked at rustfs/tests/connect_perf_drive.rs:111:5:
    assertion `left == right` failed
      left: 1001
     right: 1000
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

────────────
     Summary [  13.985s] 2 tests run: 0 passed, 2 failed, 8 skipped
        FAIL [  13.968s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        FAIL [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
error: test run failed
exit_code=100

Transient production mutations were restored byte-for-byte before the complete green rerun. Original/restored SHA256: 8d389e6d8836c041aab8c6a93e56addd65ba3f70b33264e1c74f639d0572a2da.

$ git diff --check

exit_code=0

$ git diff --cached --check

exit_code=0

Independent orchestrator acceptance
Tested code tree: e67c72fb01

Tested tree: 16fbbd2ea4
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

$ LANG=en_US.UTF-8 CARGO_BUILD_JOBS=1 CARGO_INCREMENTAL=0 cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
info: experimental features enabled: setup-scripts
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs-cli") generated 1 warning
warning: `rustfs` (bin "rustfs") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.69s
────────────
 Nextest run ID cd1edf4c-f1c0-456d-ad1a-76cc00ce05c9 with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.012s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.010s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.010s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.009s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  20.328s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   5.527s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.013s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.019s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.011s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.049s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  25.991s] 10 tests run: 10 passed, 0 skipped

exit_code=0

All 54 tracked Cargo.toml files parse without duplicate keys. git diff --check passed. The production source remains unchanged; the two deliberate implementation mutations above were restored before this complete green run.

* fix(test): compare legacy body wire parity as raw bytes

Lossy UTF-8 comparison concealed a changed binary response byte.
Compare the already captured Vec<u8> values directly.

Validation:
- Binary mutant: migrated payload 0x84 vs legacy 0x83; old comparator
  passed 1/1, raw comparator failed 0/1 (exit 100); restored green.
- Non-Swift body-drain mutant: untouched-body delegation test failed
  with empty vs payload (exit 100); restored handler without a diff.
- cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols
  --features rustfs-protocols/swift --no-fail-fast:
  543 tests run, 543 passed, 0 skipped after both mutants restored.
- cargo fmt --all --check and git diff --check passed.

Orchestrator acceptance (LANG=en_US.UTF-8):
cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
Summary [2.445s] 543 tests run: 543 passed, 0 skipped.

All shared quick checks on the final code tree:
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

Final independent acceptance after the drive-test CI repair
Parent: a99d8f4f89
Tested code tree: 93cbd96bf0

Tested tree: 075ae5161c2b62a96812bc1e3c904a7663b4f2f8
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

$ LANG=en_US.UTF-8 CARGO_BUILD_JOBS=1 CARGO_INCREMENTAL=0 cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
    Finished `test` profile [unoptimized + debuginfo] target(s) in 10m 08s
 Nextest run ID 5c7ff102-ec6d-479a-86e0-810859b28bbf with nextest profile: ci
    Starting 543 tests across 8 binaries
     Summary [   6.729s] 543 tests run: 543 passed, 0 skipped
exit_code=0

The only source change is the raw-byte wire comparator. Cargo manifests and the ratchet baselines remain unchanged from the independently validated parent. The invalid-byte and consumed-request mutations remain recorded above; their implementation changes were restored.

* fix: repair integration checks and IPC test readiness

Wait for the RPC/locks test sockets to pass the existing owner and mode
checks inside the original 5s readiness window and 10ms poll interval.

Use the existing admin S3 carrier for the new recovery and quota error
calls. Regenerate the test-count baseline for main's 118 existing tests
across seven crates. Pair the private pagination helper's raw-scan inputs
and remove the redundant Copy clone from the retirement event.

Keep the original assertions, responses, authorization, locks and storage
operations. The pagination body and raw-scan expressions at all five calls
are byte-identical; reinserting only the Copy clone restores the entire
base heal file. All ignored baseline values and s3s counts stay unchanged.

Validation base: c47432e353
Validation tree: f081084bc5
Patch SHA-256: 2dcf0028691bfb2772edfb38e262829b04f4151832dcce823c9dd9f7434b52de
Trace production-prefix SHA-256: fd994f218a21429d62866f4efb8eee1ee57b58f4a51865aa35d05a54ef1ccb45

All commands used LANG=en_US.UTF-8, CARGO_BUILD_JOBS=1,
CARGO_INCREMENTAL=0 and CARGO_NET_OFFLINE=true in one exclusive target.
Every gate retained the same tree and six source hashes before and after.
54 Cargo.toml files parse without duplicate keys.

cargo clippy -p rustfs --all-targets -- -D warnings
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 12m 50s
Exit: 0. Wall time: 770.855s.

The original 19 quick-check steps:
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
Wall time: 286.337s.

cargo nextest run --profile ci -p rustfs --no-fail-fast
     Summary [ 217.017s] 5382 tests run: 5382 passed (4 leaky), 40 skipped
Exit: 0. Wall time: 1324.414s.

cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
     Summary [   2.189s] 543 tests run: 543 passed, 0 skipped
Exit: 0. Wall time: 123.192s.

cargo nextest run --profile ci -p rustfs-ecstore list_objects_paginate --no-fail-fast
     Summary [   0.170s] 8 tests run: 8 passed, 5921 skipped
Exit: 0. Wall time: 261.324s.

cargo nextest run --profile ci -p rustfs-heal durable_mrf_anchor_retires_only_after_exact_bucket_generation_is_deleted --no-fail-fast
     Summary [   0.805s] 1 test run: 1 passed, 835 skipped
Exit: 0. Wall time: 199.801s.

Baseline regeneration:
./scripts/check_test_count_ratchet.sh --update
wrote .config/test-count-baseline.txt: 52 crates, 21340 tests, 164 ignored
Test-only increases: ecstore48, heal3, iam14, io-metrics2, obs3, policy14,
rustfs34. Existing ignored values and all other rows are byte-identical.
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)

Historical runtime red controls on f622 (same retained two assertions):
cargo nextest run --profile ci -p rustfs --lib -E 'test(=connect::diagnostics::trace_runtime::tests::local_top_rpc_reads_service_http_completions_and_signs_offline) | test(=connect::diagnostics::trace_runtime::tests::local_top_locks_reads_service_manager_and_signs_offline)' --no-fail-fast
Original path readiness with a temporary 500ms bind/seal gap:
     Summary [   0.863s] 2 tests run: 0 passed, 2 failed, 5113 skipped
Exit: 100.
Sealed readiness in that same gap:
     Summary [   5.621s] 2 tests run: 2 passed, 5113 skipped
Exit: 0.
Compiled M1: replace the helper predicate with path.exists():
     Summary [   0.870s] 2 tests run: 0 passed, 2 failed, 5113 skipped
Exit: 100.
Restore the sealed predicate byte-for-byte:
     Summary [   5.640s] 2 tests run: 2 passed, 5113 skipped
Exit: 0.
Both reds observe mode755 with owner_matches=true and fail at the original
RPC/locks unwraps with StateSecurity. M1 failure locations: trace_runtime.rs
2547:29 and 2713:29. No new assertions were introduced.

cargo nextest run --profile ci -p rustfs --lib -E 'test(connect::diagnostics::trace_runtime::tests::)' --no-fail-fast
     Summary [  20.748s] 28 tests run: 28 passed (1 leaky), 5087 skipped
Exit: 0. This security sweep is historical; the full final unit gate above
covers the actual c474 candidate.
The first disk-stopped compile ran no tests and is excluded from red credit.
All temporary seal instrumentation is removed from the final source.

Final unit cleanup markers: 4 leaky, 40 original skips (exit0). The ld
__eh_frame linker warning also appears verbatim in the original f622 run
and the earlier c474 run; there are no new source warnings.

* fix: remove redundant pending closure in disk test

Pass the capture-free pending future factory as a function item in the existing slow-operation test. This fixes the redundant_closure error at disk_store.rs:2914 that stopped strict Clippy before the feature tests could run.

The single-line test edit preserves the factory's result, warning and timeout behavior. Reversing the replacement reproduces the parent file byte for byte. All 220 assertion lines and the production prefix are unchanged; no tests, assertions, guards, profiles or baselines were added or relaxed.

Original Linux CI evidence (run 37888552268 at 53245be9f):
- RIO job113685483992: strict Clippy exit101, redundant closure at disk_store.rs:2914; feature tests skipped.
- Workspace Test and Lint job113685484058: same strict Clippy exit101; nextest and doc tests skipped.
These are lint failures, not runtime mutation kills or known flakes.

Native macOS arm64 validation on tree7a5bf47e7533b29479bc85207416102f1337c566, LANG=en_US.UTF-8, one build job, CARGO_INCREMENTAL=0 and offline dependencies:

shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
All 19 checks: exit0, 312.161s.

$ cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 20m 11s
exit0, 1213.210s total.

$ cargo nextest run --profile ci -p rustfs-ecstore --features rio-v2 slow_disk_operation_warns_once_while_pending_and_preserves_result --no-fail-fast
Finished `test` profile [unoptimized + debuginfo] target(s) in 4m 34s
PASS [0.026s] rustfs-ecstore disk::disk_store::tests::slow_disk_operation_warns_once_while_pending_and_preserves_result
Summary [   0.170s] 1 test run: 1 passed, 5943 skipped
exit0, 281.920s total; 5943 existing filter skips.

warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
This linker warning is retained; flags, profiles and allowances are unchanged.

54 tracked Cargo.toml files parsed without duplicate keys. git diff --check passed. No new assertions were introduced, so no new runtime mutation is claimed. Native compilation and the existing helper test do not substitute for actual Linux RIO execution; the new integration CI run still needs to recover before the RPC merge.

Preservation and output hashes:
parent file SHA256: 8e59ee745e3aaaa22848b943d3b10f6353b515956282cde1f3af2c394200c9ac
final file SHA256: b69ba3237061b161448ae8b62b615885d5770b98a5b4a2479bcfbd70f1bd1f17
unchanged production prefix SHA256: 7903a6e288252c420231242d7b58a25fa2a7140ea3dd430cbe7f0dde11369dcb
RIO Linux failure log SHA256:421addb9868fd88e4109a922311e54f91f0dfd70862f8ba7fcaba193482ed90b
Workspace Linux failure log SHA256:e0aea8d0f014c022252a9534968e083c98cfc1fd760da445ecd04ec3f522e26a
19 output SHA256:35a54f28d707956ed1edb29d492c4dfe8a9fe753d103b111b5a2f1b5c41c2b98
RIO Clippy output SHA256:d6046e3868f2b3992fe11cdd04e9e98eb50a87d1234d9a70dbfeb154ff61638d
slow-operation output SHA256:605b7b258d2fd09c6ef2ab2ffc90d433d0509480f1e45125d20c823759e83608

Refs rustfs/backlog#2734.

* test(ecstore): isolate tier-free recovery fixture startup

The fixture constructor can take the production recovery lock and wait for 60 seconds before this test installs its queue and seeds metadata. Disable its background recovery only while creating this fixture, then restore the environment before calling the existing production recovery entrypoint.

Only the existing test setup changes. Its production entrypoint, default lock parameters, original assertions, 30-second timeout, cancellation and cleanup remain intact. No tests or assertions were added. Production and store/init.rs are byte-identical to the parent.

Independent acceptance ran on merge tree 8ec358cb42ba9ebb3bcfeafcef048a3f0366355b; the runner header reports parent HEAD, while before/after index and source hashes confirm this candidate. Worker task branch d8055766b8 has the same tree.

Every command used LANG=en_US.UTF-8. Task compilation also used CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0, CARGO_NET_OFFLINE=true, an independent target, and empty RUSTC_WRAPPER/RUSTC_WORKSPACE_WRAPPER to keep compilers in its owned process group without changing repository configuration.

Command:
shellcheck --version && actionlint
Exit 0; observed output excerpt:
version: 0.11.0
Log SHA-256: fc9eec16b4191d058a295813ca9afbe9d2b37b59975da860170726be162a3924

Command:
cargo fmt --all --check
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_unsafe_code_allowances.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_layer_dependencies.sh
Exit 0; observed output excerpt:
Layer dependency guard passed (no new reverse dependencies/cycles).
Log SHA-256: 69c6cb1896c0586b28d1167380d3260d99d8017a4265c8df5a2348f289e93a12

Command:
./scripts/check_architecture_migration_rules.sh
Exit 0; observed output excerpt:
Architecture migration rules passed.
Log SHA-256: 28790e70d070607ed4de0704f292a65030be92ec224ef106d4b72fd8296f896e

Command:
./scripts/check_logging_guardrails.sh
Exit 0; observed output excerpt:
✅ Logging guardrails check passed
Log SHA-256: 7adf41abff8ed8f7617decf86498888ddde78937099dd9ce47f6d359dfc2d317

Command:
./scripts/check_error_other_format_ratchet.sh
Exit 0; observed output excerpt:
error(format!) ratchet passed (total call sites: 579).
Log SHA-256: 66172aef18f150ba0605af122e26ffcf91f6dd86f43e38b05e2e982ce8d783ee

Command:
./scripts/check_no_tokio_io_uring.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_extension_schema_boundaries.sh
Exit 0; observed output excerpt:
Extension schema boundaries passed.
Log SHA-256: f13d0acd0e2ce5d9815241b12b4dab7d01cc11b9b65bcf6dfb7812c878532a51

Command:
./scripts/check_body_cache_whitelist.sh
Exit 0; observed output excerpt:
Body-cache whitelist guard passed.
Log SHA-256: 6b828847b0383950ee6eef9cd846d0c322c945e47efd4c7d1ad881178810d6bb

Command:
./scripts/check_s3s_footprint.sh
Exit 0; observed output excerpt:
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)
✅ s3s footprint baseline check passed
Log SHA-256: 47f53d04ef645618563fe1f22abc64667f19078f1974feedcca46e4f2c016844

Command:
./scripts/check_test_count_ratchet.sh
Exit 0; observed output excerpt:
test-count ratchet OK: 52 crates, 21340 tests, 164 ignored match .config/test-count-baseline.txt
Log SHA-256: 618a5e330e933de0c6ff484546df401457024a88bb5986d11d588794e33dab60

Command:
./scripts/check_fips_wording.sh
Exit 0; observed output excerpt:
FIPS wording guard passed (README.md and CHANGELOG.md contain no forbidden claims).
KMS wording guard passed (crates/kms claims no Vault KV2 Transit wrapping).
Log SHA-256: 99221cc9979c480ec4f46f50d76c52b18c72fe499f12e15e4f4a22a1b8bfb140

Command:
./scripts/check_embedded_secrets.sh
Exit 0; observed output excerpt:
Embedded secret guard passed (no private key material or provider credential literal in the tree).
Log SHA-256: 437a3ceba32094d2901889965ad191e411aad32cdb18bf0b8659e818011c406b

Command:
make script-tests
Exit 0; observed output excerpt:
test_cli_requires_the_repository_default_branch (__main__.SelfTests.test_cli_requires_the_repository_default_branch) ... ok
test_config_rejects_duplicate_and_invalid_entries (__main__.SelfTests.test_config_rejects_duplicate_and_invalid_entries) ... ok
test_freshness_boundaries (__main__.SelfTests.test_freshness_boundaries) ... ok
test_grace_requires_two_successful_queries_with_no_history (__main__.SelfTests.test_grace_requires_two_successful_queries_with_no_history) ... ok
test_invalid_api_evidence_fails_closed (__main__.SelfTests.test_invalid_api_evidence_fails_closed) ... ok
test_report_retains_every_workflow (__main__.SelfTests.test_report_retains_every_workflow) ... ok
test_requests_filter_schedule_default_branch_and_success_on_server (__main__.SelfTests.test_requests_filter_schedule_default_branch_and_success_on_server) ... ok
Ran 10 tests in 0.042s
OK
Ran 30 tests in 20.627s
OK
Ran 10 tests in 2.268s
OK
Ran 33 tests in 0.127s
OK
Ran 7 tests in 0.002s
OK
Ran 7 tests in 0.001s
OK
Ran 4 tests in 0.108s
OK
Ran 3 tests in 0.564s
OK
PASS: cold stampede script self-test
Log SHA-256: 75fda2d069a373c63667b908baaf99e6591aab9099a739b0c4c22ceb6da8d68c

Command:
python3 ./scripts/test_object_generation_protocol_model.py
Exit 0; observed output excerpt:
test_an_unavailable_decision_quorum_cannot_choose_a_candidate (__main__.ObjectGenerationProtocolModel.test_an_unavailable_decision_quorum_cannot_choose_a_candidate) ... ok
test_counter_allocation_requires_persistence_and_is_consumed_once (__main__.ObjectGenerationProtocolModel.test_counter_allocation_requires_persistence_and_is_consumed_once) ... ok
test_counter_restart_burns_persisted_allocations_before_or_after_issue (__main__.ObjectGenerationProtocolModel.test_counter_restart_burns_persisted_allocations_before_or_after_issue) ... ok
test_crash_after_promise_before_reply_does_not_restore_an_older_ballot (__main__.ObjectGenerationProtocolModel.test_crash_after_promise_before_reply_does_not_restore_an_older_ballot) ... ok
test_duplicate_or_unconfigured_voters_cannot_form_a_quorum (__main__.ObjectGenerationProtocolModel.test_duplicate_or_unconfigured_voters_cannot_form_a_quorum) ... ok
test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision (__main__.ObjectGenerationProtocolModel.test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision) ... ok
test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected (__main__.ObjectGenerationProtocolModel.test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected) ... ok
test_forgetting_tombstone_authority_allows_an_aba_create (__main__.ObjectGenerationProtocolModel.test_forgetting_tombstone_authority_allows_an_aba_create) ... ok
test_issuing_before_persistence_reuses_a_ballot_after_restart (__main__.ObjectGenerationProtocolModel.test_issuing_before_persistence_reuses_a_ballot_after_restart) ... ok
test_lost_ack_and_restart_preserve_the_chosen_operation (__main__.ObjectGenerationProtocolModel.test_lost_ack_and_restart_preserve_the_chosen_operation) ... ok
test_minority_accepted_work_is_adopted_and_finished (__main__.ObjectGenerationProtocolModel.test_minority_accepted_work_is_adopted_and_finished) ... ok
test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample (__main__.ObjectGenerationProtocolModel.test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample) ... ok
test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it (__main__.ObjectGenerationProtocolModel.test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it) ... ok
test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor (__main__.ObjectGenerationProtocolModel.test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor) ... ok
test_replacing_voters_with_empty_state_can_choose_a_conflicting_value (__main__.ObjectGenerationProtocolModel.test_replacing_voters_with_empty_state_can_choose_a_conflicting_value) ... ok
test_skipping_predecessor_recovery_chooses_a_broken_revision_chain (__main__.ObjectGenerationProtocolModel.test_skipping_predecessor_recovery_chooses_a_broken_revision_chain) ... ok
test_stale_publication_rollback_and_gc_cannot_change_the_winner (__main__.ObjectGenerationProtocolModel.test_stale_publication_rollback_and_gc_cannot_change_the_winner) ... ok
test_superseded_operation_identity_cannot_be_reused_for_different_input (__main__.ObjectGenerationProtocolModel.test_superseded_operation_identity_cannot_be_reused_for_different_input) ... ok
test_superseded_operation_retry_returns_its_original_successor (__main__.ObjectGenerationProtocolModel.test_superseded_operation_retry_returns_its_original_successor) ... ok
test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision (__main__.ObjectGenerationProtocolModel.test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision) ... ok
test_tombstone_rejects_a_delayed_never_present_create (__main__.ObjectGenerationProtocolModel.test_tombstone_rejects_a_delayed_never_present_create) ... ok
test_unlearned_predecessor_is_recovered_before_the_second_slot (__main__.ObjectGenerationProtocolModel.test_unlearned_predecessor_is_recovered_before_the_second_slot) ... ok
Ran 24 tests in 0.929s
OK
Log SHA-256: 0b4be926efb0609d497ae58930340153f01957b606206cbbdb3e0d47eb29d3a2

Command:
python3 ./scripts/check_test_wiring.py
Exit 0; observed output excerpt:
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired
Log SHA-256: 21f676f458451525535ab0ef73d6a9c0d9774e16d76a4d19e396ad34936dd39c

Command:
./scripts/check_no_planning_docs.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_uring_lane_lib_only.sh
Exit 0; observed output excerpt:
OK: no 'uring_'-matching test functions under crates/ecstore/tests; --lib is safe
Log SHA-256: 43f3f851e27ec2ab9e34d6b05fefdede38af992699c473904bfe6f53a46d52e3

Command:
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast
Exit 0; wall 234.399s.
Observed output:
    Finished `test` profile [unoptimized + debuginfo] target(s) in 3m 49s
    Starting 1 test across 1 binary (5871 tests skipped)
  SETUP PASS [   0.006s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"'
        PASS [   0.440s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item
     Summary [   0.586s] 1 test run: 1 passed, 5871 skipped
Log SHA-256: 34e973f86e8dc2102c1a0e483e0ac802ce7df3d1b40a603932f62c0805c4de27

Command:
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over) | test(=store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence) | test(=store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable)' --no-fail-fast
Exit 0; wall 4.010s.
Observed output:
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.43s
    Starting 3 tests across 1 binary (5869 tests skipped)
  SETUP PASS [   0.005s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"'
        PASS [   0.068s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over
        PASS [   0.637s] rustfs-ecstore store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence
        PASS [   0.846s] rustfs-ecstore store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable
     Summary [   0.987s] 3 tests run: 3 passed, 5869 skipped
Log SHA-256: b5da2a927ff18ca9aa217762f7186a6d4371bd9e8feedd7b77940e10b64dffac

Command:
cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
Exit 0; wall 1084.508s.
Observed output:
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 18m 03s
Log SHA-256: 28f98be7970faaa71cd118da1da4e6e2e3d23b9dda7e6449eaa7a37e741260b5

Compiled mutation command (unchanged for each red and restored run):
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast

M1: reverse only this test-local constructor flag from false to true. The mutant compiled, then failed the original recovered-item queue assertion after 30.481s. Restoring the entire observed candidate passed in 0.460s. Red log 55bf3b1535529ac0bcfb8071f9b56f9f40631007a85c315367e7e93786d65293; restored log 204d5fbe4574a7917f5e740a795d864fe28a7e7fb5e33d27b0a9cf68e2f17f52.

M2: replace only the actual awaited enqueue with false, consuming its original inputs. The mutant compiled, then failed the same original queue assertion after 30.516s. Restoring the entire observed candidate passed in 0.457s. Red log 89186330413215e231e2b06bf013cc31097ea8f4a106d91421a36d6c8986a7fd; restored log e848b42d2302401afa5dbcd9e37bf76d2e56a96ab6de5a488513b997f53c5e9b.

All temporary observation and mutation code was removed before final acceptance. Historical baseline and diagnostic failures remain recorded; they are not mutation credit. The separate Object Lock intent-retry diagnostic still fails, and these native macOS checks do not establish Linux CI recovery.

* refactor(rpc): serve internode rpc bodies with the owned Body (#8441)

* test(rpc): pin the wire bytes of every internode rpc response body

rustfs/backlog#2747 batch 4, step 1 of 2, on the s3s bodies the rpc paths
serve today. Each response body builder in storage/rpc/http_service.rs is
served through hyper's HTTP/1.1 server and the raw bytes the peer reads are
compared with goldens captured from this commit: status, content-length or
chunked framing, chunk boundaries, the terminating chunk, HEAD, and the cut
connection a failure after data produces. The capability responses carry a
per-process proof, so their length header and payload are checked against
the body they encode instead.

The read_file_stream response builder moves into read_file_stream_response
so the test reaches the same code the handler runs; behaviour is unchanged.

* refactor(rpc): serve internode rpc bodies with the owned Body

rustfs/backlog#2747 batch 4, step 2 of 2. The internode rpc HTTP paths
build their responses with rustfs_s3_types::Body: Body::from_stream
replaces Body::from(StreamingBlob::wrap(..)) at the six stream sites, and
the in-memory and empty bodies keep their constructors. InternodeRpcService
converts its responses into the wrapped service's response body through
From<Body>, so the server keeps composing it with the s3s service as
before. The wire-byte goldens from the previous commit pass unchanged;
storage/rpc no longer names s3s.

* fix(rpc): simplify wire test response builder type

Name the boxed response builder used by the capability wire test so
all-target Clippy can check it without a type-complexity warning.

Validation (LANG=en_US.UTF-8, CARGO_INCREMENTAL=0):
- cargo clippy -p rustfs --all-targets -- -D warnings: exit 0.
- cargo nextest run --profile ci -p rustfs --lib -E 'test(/storage::rpc::/)': 279 tests run, 279 passed, 4842 skipped.
- All 19 quick-check commands, including script-tests, both Python scripts and cargo fmt --all --check: exit 0.
- Footprint baseline regenerated: files=194, s3_error_lines=1584, ecstore_files=36; test-count baseline regenerated: 52 crates, 21228 tests, 164 ignored.
- All 54 Cargo.toml files parse without duplicate keys.

No assertion was added or changed by this lint fix. R1-R8 are retained
in the existing PR verification record.

* fix(ecstore): resume fenced Object Lock creation intents

* fix(ecstore): bound creation retry futures on the stack

* fix(ecstore): require native proof for creation retries

* fix(ecstore): bound bucket creation stack usage (#8466)

* fix(ecstore): retain creation fences through rollback

Keep lifecycle, metadata-transaction, and namespace guards around rollback and metadata cleanup. Preserve the original publication error when a guard is lost. Build the timestamp retry fixture through the real interrupted producer so it checks a valid pending creation.

Validated source tree: 4e8e207999.
Two independent source reviews found no blocking regression.

Validation (LANG=en_US.UTF-8):
- bash -c 'shellcheck --version && actionlint' — exit 0
- cargo fmt --all --check — exit 0
- ./scripts/check_unsafe_code_allowances.sh — exit 0
- ./scripts/check_layer_dependencies.sh — exit 0
- ./scripts/check_architecture_migration_rules.sh — exit 0
- ./scripts/check_logging_guardrails.sh — exit 0
- ./scripts/check_error_other_format_ratchet.sh — exit 0
- ./scripts/check_no_tokio_io_uring.sh — exit 0
- ./scripts/check_extension_schema_boundaries.sh — exit 0
- ./scripts/check_body_cache_whitelist.sh — exit 0
- ./scripts/check_s3s_footprint.sh — exit 0
- ./scripts/check_test_count_ratchet.sh — exit 0
- ./scripts/check_fips_wording.sh — exit 0
- ./scripts/check_embedded_secrets.sh — exit 0
- make script-tests — exit 0
- python3 ./scripts/test_object_generation_protocol_model.py — exit 0
- python3 ./scripts/check_test_wiring.py — exit 0
- ./scripts/check_no_planning_docs.sh — exit 0
- ./scripts/check_uring_lane_lib_only.sh — exit 0
- cargo nextest run --profile ci -p rustfs-ecstore --lib --locked --no-fail-fast — 5894 passed, 0 skipped
- cargo check -p rustfs-ecstore --all-targets --locked — exit 0
- cargo nextest run --profile ci -p rustfs --lib --locked --no-fail-fast -E 'test(=admin::handlers::table_catalog::tests::cold_strong_registration_completes_while_other_bucket_backup_waits) | test(=admin::handlers::table_catalog::tests::commit_backend_borrowed_put_preserves_owner_authorization_and_precondition) | test(=admin::handlers::table_catalog::tests::disable_table_bucket_preserves_objects_and_can_be_reenabled) | test(=admin::handlers::table_catalog::tests::load_table_handler_negotiates_vended_credentials_without_breaking_metadata_only_callers) | test(=admin::handlers::table_catalog::tests::namespace_property_handler_updates_object_backed_catalog_and_maps_errors) | test(=app::object::get::tests::execute_get_object_resumes_from_relocated_pool_without_splicing_body) | test(=table_catalog::store::tests::catalog_writer_loss_prevents_recovery_storage_commit)' — 7 passed
- cargo nextest run --profile ci -p rustfs --lib --locked --no-fail-fast -E 'test(unsupported_remote_disk_)' — 2 passed

Additional fault-injection evidence remains incomplete. Timestamp M1 reached its original assertion; its restore run is being collected. M2–M5 and the three planned rollback faults have not run. An automated risk check interrupted the timestamp proof task; no assertions or validation settings were weakened.

* fix(ecstore): box nested bucket rollback state

Keep the guarded rollback future behind a heap allocation so bucket creation does not store its nested state inline. The rollback body, all three fence references and pending-proof revocation order are unchanged.

The same three original Linux tests overflowed on both e611 CI attempts. Local format, ecstore test compilation and all 19 quick-check steps pass; the original Linux feature lanes still need to validate this candidate.

* test(ecstore): isolate rollback guard loss fixtures

Run the three rollback metadata-cleanup cases in separate test processes so reused /proc/self/fd aliases from earlier cases cannot invalidate their creation witnesses. Keep the original fixture, guards, assertions and io_uring test selection, and verify that each child runs the intended case successfully.

Update the generated test-count baseline for the added macro arm.

Validation: cargo check -p rustfs-ecstore --tests; cargo fmt --all --check; all 19 quick-check commands, including make script-tests and the two Python checks. Linux io_uring acceptance and fault controls remain pending.

* refactor(replication): move the replication config onto gateway types (#8443)

* wip: T1.4 replication family

* refactor(replication): move the replication config onto gateway types

The replication configuration family leaves s3s: rustfs-replication no longer depends on s3s, and ECStore bucket metadata, the scanner and the rustfs app/admin layers hold rustfs_gateway_types::persistence::PersistedReplicationConfiguration. The persisted XML goes through the gateway persistence codec behind a new BucketConfigXml trait (the 39 byte goldens pass with unchanged fixtures); the legacy s3s edge converts once in storage/ecfs.rs through storage/s3_compat/replication.rs; the scanner data usage cache keeps its MessagePack bytes through a serde mirror pinned by bytes recorded from s3s 5761ddfe. rustfs-gateway-types 0.32.0 is pinned once (rev 772d3385). Refs rustfs/backlog#2745.

* test: use S3 serializer for list response tests

* fix(ci): restore integration test compilation

Gate the diagnostic test import with its feature, initialize the conditional
MetadataWriteStore fields explicitly, and decode the scanner replication fixture
through its existing cache boundary. Preserve the original fixture and assertions.

Validation was run on tree 544acb80c0 at 6003d4b8.
LANG=en_US.UTF-8; no source changes follow validation.

Quick checks: 19/19 passed.
- shellcheck --version && actionlint: exit 0
- cargo fmt --all --check: exit 0
- ./scripts/check_unsafe_code_allowances.sh: exit 0
- ./scripts/check_layer_dependencies.sh: exit 0
- ./scripts/check_architecture_migration_rules.sh: exit 0
- ./scripts/check_logging_guardrails.sh: exit 0
- ./scripts/check_error_other_format_ratchet.sh: exit 0
- ./scripts/check_no_tokio_io_uring.sh: exit 0
- ./scripts/check_extension_schema_boundaries.sh: exit 0
- ./scripts/check_body_cache_whitelist.sh: exit 0
- ./scripts/check_s3s_footprint.sh: exit 0
- ./scripts/check_test_count_ratchet.sh: exit 0
- ./scripts/check_fips_wording.sh: exit 0
- ./scripts/check_embedded_secrets.sh: exit 0
- make script-tests: exit 0
- python3 ./scripts/test_object_generation_protocol_model.py: exit 0
- python3 ./scripts/check_test_wiring.py: exit 0
- ./scripts/check_no_planning_docs.sh: exit 0
- ./scripts/check_uring_lane_lib_only.sh: exit 0

Darwin validation:
- cargo clippy --locked -p rustfs-ecstore -p rustfs-scanner --all-targets -- -D warnings: exit 0
- cargo nextest run --locked -p rustfs-ecstore -p rustfs-scanner --lib -E 'test(config::com::tests::save_config_preserves_retryable_system_volume_errors) | test(scanner_io::tests::replication_maintenance_refreshes_compacted_stats_across_cycles_and_retries)' --test-threads 1 --no-fail-fast: exit 0, 2/2 passed
- cargo clippy --locked -p rustfs-ecstore -p rustfs-scanner --all-targets --features rustfs-ecstore/diagnostic-read-version-caller,rustfs-scanner/diagnostic-read-version-caller -- -D warnings: exit 101
- cargo nextest run --locked -p rustfs-ecstore -p rustfs-scanner --lib --features rustfs-ecstore/diagnostic-read-version-caller,rustfs-scanner/diagnostic-read-version-caller -E 'test(config::com::tests::save_config_preserves_retryable_system_volume_errors) | test(scanner_io::tests::replication_maintenance_refreshes_compacted_stats_across_cycles_and_retries) | test(cluster::rpc::runtime_sources::tests::) | test(config::com::tests::diagnostic_config_write_retains_origin_and_api_for_results_and_cancellation)' --test-threads 1 --no-fail-fast: exit 0, 14/14 passed

Diagnostic strict Clippy remains red on four unchanged production lints:
runtime_sources.rs if_same_then_else and three redundant_locals reports in
set_disk/core/metadata_read.rs and set_disk/read.rs. Their source ranges and
feature wiring match the base; the base was not compiled separately.

Linux aarch64 validation (Rust/Clippy 1.99.0, UID 501, jobs=1):
- cargo clippy --all-targets -- -D warnings: exit 0; Finished dev profile in 68m 12s
- cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings: exit 0; Finished dev profile in 49m 45s
Source HEAD/tree and clean status matched before and after both commands.
No compiler or stack overrides were used. protoc 35.1 was installed from its
verified ARM64 release; flatc was absent and not invoked by the workspace build scripts.
The local environment differs from hosted CI.

* fix(ecstore): recover delete batches with missing versions

Treat an explicit non-null version as a no-op when it is absent from both recovery sources. Keep rejecting missing null versions, delete-marker mismatches and conflicting local generations.

Add returning-disk and partial-receipt regressions, extend journal boundary checks, and regenerate the ecstore test-count baseline. Production faults exercise the 24 added assertion sites and 7 semantic checks: 33 faults failed at the expected sites and passed after full restoration.

Validation (LANG=en_US.UTF-8, one build job):

cargo nextest run --locked -p rustfs-ecstore --lib -E 'test(delete_recovery) | test(quorum_delete_cleanup_coordinator_binds_single_and_batch) | test(crash_delete_retry_batch_reclaims_inline_and_reserved_external)' --test-threads 1 --no-fail-fast
Summary [  34.580s] 69 tests run: 69 passed, 6058 skipped; exit 0

cargo clippy --locked -p rustfs-ecstore --all-targets -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 5m 37s; exit 0

All 19 final Quick Checks completed with exit 0:

- shellcheck --version && actionlint
- cargo fmt --all --check
- ./scripts/check_unsafe_code_allowances.sh
- ./scripts/check_layer_dependencies.sh
- ./scripts/check_architecture_migration_rules.sh
- ./scripts/check_logging_guardrails.sh
- ./scripts/check_error_other_format_ratchet.sh
- ./scripts/check_no_tokio_io_uring.sh
- ./scripts/check_extension_schema_boundaries.sh
- ./scripts/check_body_cache_whitelist.sh
- ./scripts/check_s3s_footprint.sh
- ./scripts/check_test_count_ratchet.sh
- ./scripts/check_fips_wording.sh
- ./scripts/check_embedded_secrets.sh
- make script-tests
- python3 ./scripts/test_object_generation_protocol_model.py
- python3 ./scripts/check_test_wiring.py
- ./scripts/check_no_planning_docs.sh
- ./scripts/check_uring_lane_lib_only.sh

test-count ratchet OK: 52 crates, 22009 tests, 165 ignored match .config/test-count-baseline.txt
PASS: cold stampede script self-test
model: states=51023, transitions=167520, terminal=732, chosen_candidates=2
multi-slot round schedules: 640
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired

* fix(ecstore): retry tier commit intent lock contention

Retry transient namespace lock timeouts within the existing three-attempt
CAS budget. Reload and verify the original mutation identity after
contention, and preserve the final typed error when attempts are exhausted.
Cover save/read contention, exhausted retries, replacement identities and
CAS reconciliation without changing the existing timeout or lock policy.

Validation:
- cargo nextest run --locked -p rustfs-ecstore --features test-util --lib -E 'test(tier_mutation)' --test-threads 1 --no-fail-fast: 52 passed.
- cargo nextest run --locked -p rustfs-ecstore --features test-util --lib -E 'test(intent_advance_) | test(=services::tier::tier::tests::caller_cancellation_after_durable_prepare_does_not_hide_the_mutation)' --test-threads 1 --no-fail-fast: 5 passed.
- cargo clippy --locked -p rustfs-ecstore --all-targets -- -D warnings: passed.
- cargo clippy --locked -p rustfs-ecstore --all-targets --features test-util -- -D warnings: passed.
- 27 mutation pairs failed at the intended assertions and passed after restoring the production code.
- bash scripts/check_test_count_ratchet.sh --update, then bash scripts/check_test_count_ratchet.sh: 52 crates, 22013 tests, 165 ignored; ecstore increases from 6581 to 6585.

* refactor(lifecycle): replace s3s domain types with Gateway DTOs

Keep lifecycle rules, restore requests, Object Lock, and scanner cached tags in domain types. Convert at the existing HTTP boundary while preserving legacy XML, scanner cache bytes, date ordering, and tombstones.

Linux test validation before the final import-path-only adjustment:

cargo nextest run -p rustfs-ecstore bucket_config_goldens
39 passed; 6211 skipped by selector.

cargo nextest run --lib -p rustfs -E 'test(/test_zero_rule_lifecycle_tombstone_recognition/) | test(/test_local_lifecycle_staleness_axis_selection/) | test(/test_lifecycle_staleness_preserves_legacy_submillisecond_axis/) | test(/test_merge_incoming_lifecycle/) | test(/test_lifecycle_expiry_statement_matrix/) | test(/test_lifecycle_expiry_subset_xml/) | test(/put_bucket_lifecycle_validation_errors_keep_their_s3_code/) | test(/execute_put_bucket_lifecycle_configuration_rejects_missing_configuration/) | test(/execute_restore_object_rejects_missing_restore_request/) | test(/execute_restore_object_rejects_select_type/) | test(/execute_restore_object_maps_failures_to_typed_s3_errors/) | test(/execute_restore_object_returns_internal_error_when_store_uninitialized/) | test(/restore_request_conversion_preserves_legacy_output_values/) | test(/restore_request_conversion_preserves_presence_and_raw_enums/) | test(/restore_request_conversion_distinguishes_absent_and_empty_containers/)'
25 passed; 5318 skipped by selector.

cargo nextest run --profile ci -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-replication -p rustfs-scanner
7872 passed; 21 skipped under the existing CI profile.

Previously verified mutation controls: 39 production faults, 41 RED/GREEN pairs.

cargo clippy --locked -p rustfs -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-scanner --all-targets -- -D warnings
Passed in 40m 45s before the import-path adjustment, then in 1m 59s on
the final source with warnings denied. The final adjustment imports the
existing helper through the object domain facade; fmt and the architecture
check also passed.

Part of rustfs/backlog#2745. The remaining ecstore configuration families follow separately.

* refactor(admin): move error types to rustfs-s3-types (#8440)

* refactor(protocols): move error types to rustfs-s3-types by codemod

T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and
the error items of protocols move off s3s by scripts/codemods, run with
--hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it
downcasts the backend error, whose runtime type is still s3s::S3Error.

- rustfs/src/protocols/client.rs: the two webdav session methods now
  return rustfs_s3_types::S3Result and convert the s3s error with From.
- codemod: a child module that calls s3_error! through `use super::*`
  gets its own `use rustfs_s3_types::s3_error;` once its parent imports
  that macro by name; otherwise it is listed as GLOB.
- s3-types: a table test pins code, status and the s3s error the legacy
  edge renders for every variant a moved call site names.
- s3_error_lines 1585 -> 1584 (a lifecycle comment reworded).

* refactor(admin): move error types to rustfs-s3-types (WIP, no build)

T1.2 batch 4 (rustfs/backlog#2743), work in progress: rustfs check
still fails with 16 errors (listed in the PR body).

- codemod: --facade PATH treats a crate-local re-export of s3s's error
  items like s3s for the four names (9 new self-tests, 9 mutants red).
- codemod output for rustfs/src/admin and rustfs/src/site_replication
  (--facade crate::storage_api::site_replication::s3); 6 glob children
  get an explicit rustfs_s3_types::s3_error import.
- admin/router.rs: the Operation trait returns rustfs_s3_types::S3Result.
- error.rs: From<ApiError> builds the rustfs-s3-types error; the s3s
  conversion is derived from it.

* fix(admin): convert error types at the s3s boundary

Batch 4 of T1.2 left the admin and site_replication modules returning
rustfs_s3_types::S3Error while their callers and callees on the storage
and auth side still speak s3s::S3Error. Convert at each edge with the
existing From impls in rustfs-s3-types (compat_s3s.rs) instead of
changing any code, message or status:

- admin/auth.rs, admin/handlers/object_zip_download.rs,
  admin/handlers/table_catalog/mod.rs: map the s3s result returned by
  check_key_valid / authorize_request / authorize_internal_object_request
  into the admin error type with .map_err(Into::into).
- storage/ecfs.rs: the s3s-facing side maps the admin site_replication
  gate results the other way.
- admin/handlers/pools.rs: ApiError.code stays s3s::S3ErrorCode, so the
  rustfs code is converted with .into() when the struct is built, in
  production and in the two tests that build or compare it.
- admin/handlers/heal.rs, admin/runtime_sources.rs: the router returns
  s3s errors (S3Route impl); the expected rustfs code is converted with
  .into() before the comparison.
- admin/handlers/table_catalog/mod.rs: iceberg_rest_error takes
  &'static str so the Cow<'static, str> custom code borrows it; every
  caller already passes an ICEBERG_ERROR_* constant or a &'static str.
- admin/handlers/quota.rs: drop a now same-type .into() after s3_error!.
- storage_api.rs, site_replication/mod.rs: drop re-exports and an import
  that no longer have users (clippy -D warnings).

cargo check -p rustfs --all-targets and cargo clippy -p rustfs
--all-targets -- -D warnings are clean; the s3s footprint baseline is
unchanged at files=172, s3_error_lines=303, ecstore_files=28.

Refs rustfs/backlog#2743, rustfs/backlog#2734.

* refactor(ecstore): move bucket configs onto gateway persisted types (#8709)

* refactor(ecstore): move bucket configs onto gateway persisted types

Finish the ECStore batch of T1.4 (rustfs/backlog#2745). The remaining
eleven bucket configuration families (Website, CORS, Tagging, Versioning,
ObjectLock, Encryption, Notification, Logging, Accelerate, RequestPayment,
PublicAccessBlock) now read and write their stored XML through the
rustfs-gateway-types persistence and cors_tagging codecs instead of the
s3s codec, so the legacy s3s bucket-config codec and its
s3gate-metadata-xml compat marker are removed.

- ecstore: BucketConfigXml is implemented by the gateway codec for every
  family; metadata, metadata_sys, versioning, object lock, replication and
  lifecycle consumers take the persisted types. Region moves to
  rustfs_s3_types::Region, and the ETag conversion to s3s moves to the
  rustfs edge with its tests. Only warm_backend.rs and warm_backend_s3.rs
  still reference s3s, held for T1.5 (rustfs/backlog#2746).
- scanner: object lock cache, item actions and data usage use the
  persisted types.
- rustfs: the legacy S3 edge converts each family between s3s DTOs and the
  persisted types in storage/storage_api.rs by writing one codec's XML and
  reading it with the other; Tagging converts field by field.
- Bump the rustfs-gateway-types pin to 21283e87.

The 39 persisted-byte goldens are unchanged and pass, so a document the
old codec wrote reads back and re-encodes to the same bytes. s3s footprint:
files 181 -> 146, ecstore_files 28 -> 2, s3_error_lines unchanged.

* test(app): prepare the PutBucketEncryption config from the request

The uninitialized-store test passed an empty persisted configuration, which the use case now validates before it reaches the store check, so it answered MalformedXML instead of InternalError. Convert the request's configuration the way the S3 edge does.

* style(app): format the PutBucketEncryption test

* refactor(s3-client): move remote error codes and DTO enums off s3s (#8783)

Part of T1.5 (rustfs/backlog#2746), batch 1.

* refactor(s3select): run queries on gateway DTO request shapes (#8794)

* refactor(s3select): run queries on an engine request built from gateway DTO shapes

Part of T1.5 (rustfs/backlog#2746), batch 2.

* fix(s3select): keep the select request conversion in the app layer

The layer guard forbids app -> storage; the conversion moves from storage/s3_compat/select.rs to app/select_request.rs unchanged.

* fix(heal): retry transient metadata conflicts (#8812)

* fix(heal): retry transient metadata conflicts

* chore: update heal test count baseline

* test: reserve nextest capacity for progress probes (#8818)

* test(e2e): use a non-default region for the differential negative control

Since main's region-aware CreateBucket and GetBucketLocation (#8739, #8743), us-east-1 answers an empty LocationConstraint and 200 for an owned re-create, so the control no longer reported its own region. Also refresh the e2e-full linux selection from the CI listing (682 tests).

* refactor(protocols): move StorageBackend to gateway types (#8825)

* refactor(protocols): move storage backend to gateway types

* test: update e2e test count baseline

* fix: preserve copy source keys and unknown lengths

* test: update protocols test count baseline

* fix(protocols): preserve typed SFTP backend errors

* test(protocols): use gateway list input in policy tests

* fix(protocols): clear clippy failures after T1.5 merge (#8831)

fix(protocols): clear clippy failures after merge

* fix(ecstore): finalize restore metadata after local copy-back (#8836)

* fix(ecstore): finalize restore metadata after local copy-back

* test(ecstore): ratchet added restore finalization cases

* fix(ecstore): keep restore copyback compatible with older peers

* fix(ecstore): gate restore copyback on fleet compatibility

* fix(ecstore): retain transition identity during restore copyback

* fix(ecstore): route committed restore copyback locally

* fix restore copyback cleanup

* fix(ecstore): honor restore probe test fallback

* refactor(storage): migrate option errors to domain types (#8839)

* refactor(s3): migrate private multipart validation errors (#8844)

* refactor(auth): derive policy conditions from AuthzFacts (#8442)

* refactor(auth): extract authorization facts

* test(e2e): update smoke selection baseline

* test(e2e): update full selection baseline

* refactor(admin): migrate health and prefix usage to owned requests (#8887)

* refactor(admin): add owned health router boundary

* refactor(admin): migrate prefix usage to owned request

* refactor(admin): complete the owned admin request boundary (#8949)

* refactor(admin): complete the owned admin request boundary

Every type implementing the owned admin `Operation` now serves the legacy
router through one blanket impl that converts the s3s request and response
at that single edge, so handlers register with `AdminOperation(&Handler {})`
and need no adapter statics; `LegacyAdminAdapter` is removed.

`AdminRequest` gains the accessors handlers read today (`remote_addr`,
`client_info`, `request_context`, `req_info`, `req_info_mut`,
`install_req_info`, `envelope_mut`, `take_body`), the request helpers gain
envelope forms (`authenticate_envelope_credentials`,
`object_store_from_envelope`, the table-catalog `*_from_envelope` helpers,
`KmsAdminAudit::from_envelope`, `AccountAuditContext::from_envelope`,
`CallerIdentity::resolve_envelope`), `read_compatible_admin_body` accepts
either body type, and the owned `Body` gains `store_all_limited` and
`Default`. `admin::operation::test_support::TestRequest` builds owned
requests for handler tests. Unused accessors carry a per-method dead-code
expectation that each migrating handler batch removes.

No handler behavior changes; the remaining migrations follow in batches.

* chore(admin): add the T1.12 admin handler codemod

The script rewrites an admin handler file from the s3s request and response
types to the owned boundary (imports, signatures, request field and helper
calls, body access, response construction, and test request literals) and
leaves a TODO comment at every site it cannot rewrite mechanically. It is
idempotent and is removed once T1.12 completes.

* refactor(admin): migrate replication and KMS handlers to owned requests (#8951)

Move the T1.12 batch 2 admin handlers (site_replication, kms_keys, event,
replication, diagnostics, on_demand_migration, audit, account, kms_rekey,
kms_key_lifecycle, object_data_cache, tls_debug, integrity) from the s3s
request/response types to the owned AdminRequest/AdminResponse boundary.
Handler logic, authorization order, error codes and response bytes are
unchanged; the request/response plumbing now goes through the envelope
accessors and helpers introduced by the owned boundary.

The shared response helpers (utils::json_response, utils::empty_response,
handlers::admin_json_response, target_descriptor::build_json_response,
profile::authorize_profile_request) and the site replication repair
executor still answer with the s3s types, so the migrated files carry
owned-boundary forms of those helpers and convert the repair response at
its call edge. The streaming console-log body and the two devnull drains
move from the s3s byte stream to the owned Body stream/frame API.

Drop the dead_code expectations for the owned accessors and envelope
helpers this batch starts using, and lower the s3s footprint baseline to
the measured 99 files.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* refactor(admin): add owned forms of the shared admin response helpers

Handler batches were copying `json_response`, `empty_response`,
`admin_json_response`, and `build_json_response` into each migrated file
because the shared helpers still return the s3s response. Provide owned
forms next to each legacy helper (`owned_json_response`,
`owned_empty_response`, `owned_admin_json_response`,
`owned_build_json_response`, `authorize_profile_envelope`) with parity
tests against the legacy shape, derive `Debug` for `AdminResponse` so
tests can use `expect_err`, and fix three codemod rules: strip owned
string conversions in test credentials, annotate non-literal URI parses,
and leave a rebound request parameter immutable.

* refactor(admin): use the shared owned response helpers in batch 2 (#8954)

Replace the per-file owned-boundary copies of json_response,
empty_response, admin_json_response, build_json_response and
authorize_profile_request in the batch 2 admin handlers with the shared
owned_* helpers and authorize_profile_envelope that the boundary branch
now provides, and restore expect_err in the handler tests now that
AdminResponse implements Debug; the `.err().expect(..)` stand-ins were
failing the base branch's clippy step (clippy::err_expect under
-D warnings). The site replication repair call edge
keeps its conversion because the repair executor still answers with the
s3s response type.

Drop the dead_code expectations on the shared owned helpers this batch
now uses in non-test code.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* fix(admin): preserve legacy body limit error responses

* refactor(admin): move policy, ILM and batch 3 handlers to owned requests (#8956)

Batch 3 of T1.12 (rustfs/backlog#2788, parent rustfs/backlog#2734) moves
14 admin handler files to AdminRequest/AdminResponse and the owned
Operation trait: ilm_transition, policies, idp_compat, config_admin,
plugins_instances, profile_admin, group, heal, object_zip_download,
kms_key_metadata, durability, inspect_archive, sts and
gateway_key_inventory. Handler logic, authorization order, error codes
and response bytes are unchanged; test request literals use
test_support::TestRequest.

The handlers use the shared owned helpers (owned_json_response,
owned_admin_json_response, owned_build_json_response,
authorize_profile_envelope). The dead_code expectations on the
AdminRequest accessors and RequestEnvelope::uri_mut they now exercise
are removed. The legacy target_descriptor::build_json_response keeps
only its equivalence test caller and carries the matching expectation
until it retires.

object_zip_download builds the s3s request the S3 gate still reads from
the owned request facts; heal authenticates through
authenticate_envelope_credentials; the trace, inspect-archive and zip
bodies use Body::from_stream. extract_plugin_instance_filters takes the
URI so extensions.rs keeps calling it, and the two idp_compat
delegations stay on the legacy trait until their delegates migrate.

The s3s footprint baseline drops from 99 to 90 files.

* test(admin): anchor the replication diff body contract on the call

The source-contract test looked for `let body = read_compatible_admin_body(`
on one line; rustfmt wrapped that binding after the handler moved to the
owned request boundary, so the marker was no longer found. Anchor on the
call itself so formatting cannot change what the contract checks.

* refactor(admin): migrate table catalog and batch 1 to owned requests (#8955)

* fix(admin): preserve catalog body limit response bytes

* refactor(admin): migrate identity and cluster handlers to owned requests (#8966)

* refactor(admin): migrate identity and cluster handlers to owned requests

Move the T1.12 batch 4 admin handlers (user, service_account, oidc, tier,
system, pools, quota, kms_dynamic, kms_backup, rebalance, module_switch,
bucket_meta, cluster_snapshot, realtime, bucket_recovery, profile) from the
s3s request/response types to the owned AdminRequest/AdminResponse
boundary. All 90 impl Operation blocks now take AdminRequest and return
AdminResponse; handler logic, authorization order, error codes and response
bytes are unchanged. The handlers use the shared owned helpers
(owned_json_response, owned_admin_json_response, authorize_profile_envelope,
the envelope-form runtime sources and audit constructors) instead of local
copies.

profile.rs drops the s3s forms of authorize_profile_request and
profile_not_implemented_response now that profile_admin.rs is on the
envelope gate too; profile_not_implemented_response answers with an
AdminResponse. The realtime metrics stream moves from the s3s ByteStream
adapter to Body::from_stream, and the rebalance idle-404 document still
renders through s3s::S3Error::to_http_response with its body converted at
the edge.

The idp_compat delegators ImportIamV2 and AddServiceAccountLdap move to the
owned trait as well: with their targets on both traits, the method-call
delegation became ambiguous between the owned impl and the router blanket
impl, so they now take the owned request directly.

KmsAdminAudit::from_request loses its last production caller with this batch
and is deleted; its comparison test now asserts the envelope facts directly.

Lower the s3s footprint baseline to the measured 68 files.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* refactor(admin): remove the legacy admin helpers batch 4 superseded

Batch 4 moved the last production callers of the s3s-typed admin helpers
to their owned forms, which left `admin_json_response`, `json_response`,
`authenticate_request`, `authorize_admin_request`, `object_store_from_req`
and `object_store_from_extensions` unused, along with the `Body`,
`S3Request` and `S3Response` re-exports of the admin `s3` facade. Remove
them. The owned forms keep their documentation; the behavior tests of the
removed `json_response` now exercise `owned_json_response`, and the
compat-prefix test asserts the owned response directly.

Two tests existed only to compare a removed legacy helper with its owned
twin (`owned_json_response_matches_the_legacy_shape`) or to cover the
removed legacy gate (`authorize_admin_request_without_credentials_is_rejected`);
they go with the helpers, and the owned equivalents stay. The rustfs
test-count row drops from 5957 to 5955 and the s3s footprint from 68 to 66
files.

* refactor(admin): drop the legacy target descriptor JSON response helper (#8968)

`target_descriptor::build_json_response` returned the s3s response and
had no production caller left once the event, audit and plugin instance
handlers moved to `owned_build_json_response`; it survived only for the
legacy/owned equivalence test. Remove it, give the owned helper its own
documentation, and make the test assert the owned response directly:
status, `application/json`, the echoed `x-request-id` when present and
only then, and the body bytes. The file no longer imports s3s, so the
footprint baseline drops from 66 to 65 files.

* test(ecstore): pin native history on cancellation under the cancellable queue

main moved the local write path onto the disk blocking queue, which skips an
operation once its waiter is dropped, so a borrowed write cancelled while it is
queued no longer creates its file. The integration-only native publication
test still expected the queued open to run. Assert the new execution model
instead (the open never runs) while keeping its safety claims: the Started
record is taken before queueing and the cancelled publication never regrants
native history.

* chore(gateway): raise test-count rows for tests merged from main

* test(s3select): use the owned JSON input types in merged planner tests

* fix(ecstore): scope the merged direct-writer test hook to test builds

* fix(ecstore): create parents before admitted truncate opens

Align the heal tests with removal-disabled preservation and local-only outcome
fields, and refresh the e2e-full selection for the synced tests.

* chore(gateway): update test-count baseline after main sync

Main #9241 removed the node pool-metadata readiness cache with 8 tests
and added 6, so the rustfs row drops from 6440 to 6438. The ecstore and
s3select rows rise for tests added by #9237, #9239 and #9241.

* feat(scripts): add junit_diff.py for dual-stack JUnit comparison (#9275)

* test(e2e): allow a pinned release binary as differential reference side

RUSTFS_E2E_REFERENCE_BINARY makes side A of the dual-stack differential a released rustfs binary, started without RUSTFS_S3_STACK. A missing, non-file or non-executable path fails the run instead of falling back to the same-binary mode. The report records which side is the reference and each side's --version output.

* ci(gateway): make the test-count ratchet a floor, not exact (#9278)

ci(gateway): make the test-count ratchet a floor instead of an exact baseline

Tests below a row still fail; tests above it, a member without a row and ignores below the row now pass with a note. Ignores above a row still fail. Adds contract tests wired into make script-tests.

* chore(gateway): record ignored microbenchmarks merged from main (#9290)

chore(gateway): record the ignored microbenchmarks merged from main

main added three throughput microbenchmarks that are ignored by design
(#9217): one each in rustfs-checksums, rustfs-rio and rustfs-utils. The
test-count ratchet treats a higher ignored count as a failure until the
row is raised, so raise those three rows to 1 and tighten the test rows
to the merged tree with --update.

* test(e2e): update smoke selection for reference-binary unit tests

* ci(s3-tests): add reference parity lane against pinned 1.0.5 (#9288)

* ci(parity): run the dual-stack differential against the pinned reference

Adds a differential job to reference-parity.yml: side A is the pinned release, side B the candidate artifact built for the s3-tests comparison. The job prints reference_side, version_a and version_b from the report, fails when side A is not the reference or both versions are equal, and lists every difference in the job summary.

* ci(parity): print the first version line in the differential summary

* chore: restore main's Cargo and baseline files after merging main

---------

Co-authored-by: rustfs-cla-assistant[bot] <273121464+rustfs-cla-assistant[bot]@users.noreply.github.com>
2026-11-11 15:15:12 -05:00
Chris An 0872f0be9b ci(e2e): exclude reference unit tests from e2e-full selection (#9342)
ci(e2e): keep reference-selection unit tests out of e2e-full

The eight dual_stack_differential::reference tests added in #9285 entered the e2e-full selection and changed its digest. They are hermetic and already run in e2e-smoke on every pull request, so exclude them from e2e-full; the selection digest returns to the value recorded in .config/e2e-full-selection.txt.
2026-11-11 14:13:10 -05:00
Chris An 773a5c34a5 fix(admin): read the caller secret only where a handler uses it (#9339)
AddUser reads the current secret only for a self-rotation, orphan-bucket recovery only to seal a /minio/admin compat response, and a missing secret key answers AccessDenied instead of InternalError in AddUser, bucket recovery and ChangeOwnPassword.
2026-11-11 14:12:54 -05:00
唐小鸭 056ddce8f1 fix(kms): require a scheduled deletion before destroying a key (#9337)
On the Vault KV2 and Vault Transit backends, an immediate deletion of a
key that was not already pending deletion only marked it PendingDeletion
and still reported success, so objects under the key stayed readable
after the caller was told the key was gone.

Key destruction is now two explicit steps on the Local, Vault KV2 and
Vault Transit backends: a deletion is scheduled first, and
`force_immediate` only destroys a key that is already PendingDeletion.
A request against an Enabled or Disabled key is refused with an invalid
state error and changes nothing. Vault Transit's "encryption key not
found" answer for a destroyed key now maps to KeyNotFound instead of a
backend error.

Document the feature in docs/operations/kms-key-destruction.md with a
Chinese translation.
2026-11-11 14:07:20 -05:00
Chris An 153d6b880d build(deps): re-pin rustfs/gateway to 7915d069 (#9283)
* feat(server): add the RUSTFS_S3_STACK selector and dual-stack e2e lane (#8411)

* refactor(server): group the s3s-compat tower layers under legacy_compat (#8410)

Move the ten tower layers that patch s3s behaviour out of
rustfs/src/server/layer.rs into rustfs/src/server/legacy_compat/, one file
per layer with its tests. Each file header names the s3s behaviour it
patches, the gateway mechanism that replaces it (Replaced by:), and the task
that removes it (Removed by: T2.12, rustfs/backlog#2771). The layers keep
their code, their order in external_service_stack!, and their behaviour; no
layer is deleted.

Add server::http::tests::stack_census, which renders the ordered service
chain of the real external_service_stack! expansion and compares it with a
committed constant, so a reorder, insertion or removal is a visible diff.

Route the two s3s error types the moved layers use through a
storage_api::server::legacy_compat shim so the split adds no s3s-importing
file, and lower the s3s footprint baselines to the measured values (files 209
to 208, s3_error! lines 1586 to 1585). Keep the server runtime-source bypass
guard covering the moved code.

Refs rustfs/backlog#2787, rustfs/backlog#2734

* feat(s3-types): add the RustFS-owned S3 error type and s3_error! macro (#8406)

`rustfs-s3-types` now provides `S3ErrorCode` (generated by build.rs from
src/codes.txt, the grep-derived call surface, plus `Custom`), `S3Error`,
`S3Result`, `StdError` and the `s3_error` macro with the same call syntax
as s3s, so the T1.2 codemod is a `use` swap. Statuses are hand-written and
pinned to the s3s behaviour RustFS ships today by an oracle test that uses
s3s as a dev-dependency only. The `From` conversions for the legacy edge
live behind the off-by-default `compat-s3s` feature (the orphan rule
forbids them in rustfs/src/error.rs); `rustfs` enables it.

Refs rustfs/backlog#2742, rustfs/backlog#2734

* ci: add the per-crate test-count ratchet (#8404)

Quick Checks now fails any PR that lowers the number of test attributes
in a workspace crate or adds #[ignore], measured per crate against the
exact rows of .config/test-count-baseline.txt (rustfs/backlog#2734
section 1 rule 3). The counter widens the spec's literal
#[(tokio::)?test] to the tokio attributes that carry arguments and both
spellings of #[test_case(...)], and anchors at line start so
commented-out attributes do not count. The initial baseline covers 52
crates, 21077 tests and 164 ignored.

Refs rustfs/backlog#2738
Refs rustfs/backlog#2734

* test(ecstore): pin the persisted bucket-config XML bytes for 13 families (#8408)

Before any DTO or codec is replaced, pin the exact bytes that
deserialize followed by serialize produces for every persisted
bucket-configuration XML family in its three source shapes (AWS
documentation, MinIO-written, RustFS-written), and the bytes RustFS
writes today for a typed value. Byte equality only: a structural
comparison would let a codec replacement change the persisted encoding
unnoticed.

The generator script refuses to overwrite existing goldens without
--force, refuses on dirty codec paths, cannot report success from its
write mode, and records the generating commit in the fixtures README.

Refs rustfs/backlog#2744, rustfs/backlog#2734

* ci: raise the test-count baseline to the integration tree (#8426)

* ci: make the s3s footprint baseline exact and lower-only (#8403)

* ci: make the s3s footprint baseline exact and lower-only

The s3s footprint guard compared three counters against ceilings held in
the script itself, so a ceiling could sit above the tree and a drift was
fixable by raising it (rustfs/rustfs#5964 raised the s3_error! ceiling
to go green). The three values now live in
.config/s3s-footprint-baseline.txt as files, s3_error_lines and
ecstore_files, the script reads only that file, and a measurement that
differs from its baseline in either direction fails: growth prints
"s3s footprint grew: <measured> > <baseline>", shrinkage prints
"lower the baseline to <measured> in this PR". The counting method is
unchanged; the baseline is the current measurement of
gateway/integration (209 / 1585 / 36). The header records the end state:
once all three reach 0, T4.2 switches the script to failing on any
occurrence.

Refs rustfs/backlog#2739, rustfs/backlog#2734

* ci: add the s3s edge allowlist mode to the footprint guard (#8405)

`scripts/check_s3s_footprint.sh --mode allowlist` fails on every Rust
file that references s3s (`s3s::` or `s3s_sigv4`) and is not matched by
an entry of the new `.config/s3s-edge-allowlist.txt`; `--dry-run` lists
those files and exits 0. The allowlist carries the initial edge from the
spec, and the script rejects an entry that names or matches anything
under crates/ other than crates/e2e_test. The baseline mode and the
argument-less CI invocation are unchanged.

Refs rustfs/backlog#2741, rustfs/backlog#2734

* test(ecstore): reach the bucket-config codec through the test facade (#8428)

* refactor(s3-types): own the HTTP Body and drop s3s from three crates (#8431)

Add rustfs_s3_types::Body, a BoxBody<Bytes, StdError> wrapper with
empty() and From<Bytes | Vec<u8> | String | &'static str>, and the
legacy From conversions to and from s3s::Body behind compat-s3s.

The signer is now generic over the request body: it only reads and
writes the request head, so it needs no body type and its callers keep
theirs. s3-client sends its requests with the owned Body, madmin signs a
bodiless head, and targets drops its two S3Request/S3Response header
wrappers in favour of extract_params_header.

rustfs-signer, rustfs-madmin and rustfs-targets no longer depend on
s3s; the s3s footprint baseline drops from 209 to 202 files.

Refs rustfs/backlog#2747, rustfs/backlog#2734

* refactor(ecstore): move S3ErrorCode to rustfs-s3-types by codemod (#8432)

* test(ecstore): pin the InvalidRange responses of new_getobjectreader

Both error branches of new_getobjectreader build an ErrorResponse with S3ErrorCode::InvalidRange, and no test observed either code. Pin the code, its wire name and the location fields of each branch before the code type moves to rustfs-s3-types (rustfs/backlog#2743), so a changed code on either branch fails a test. The ecstore test-count row rises by the two tests.

* chore(scripts): add the s3s-to-rustfs-s3-types error codemod

Task T1.2 of rustfs/backlog#2734 (rustfs/backlog#2743) moves S3Error, S3ErrorCode, S3Result and s3_error! from s3s to rustfs-s3-types by script, one crate closure per PR. The codemod rewrites use trees (splitting mixed s3s imports) and qualified code paths, never comments or literals, adds the rustfs-s3-types dependency, drops s3s only from a crate that no longer names it, refuses every file of the s3s edge allowlist, takes --hold for a file whose type is fixed outside the closure, and prints a per-crate census. A second run changes nothing. Its self-test runs under make script-tests.

* refactor(ecstore): take S3ErrorCode from rustfs-s3-types

Output of scripts/codemods/s3_error_to_s3_types.py over crates/ecstore, crates/lifecycle, crates/scanner and crates/replication (rustfs/backlog#2743 batches 1 and 2). The ErrorResponse code field, the two InvalidRange constructions in new_getobjectreader and an unused tier import now use rustfs_s3_types::S3ErrorCode; the variant names and wire names are unchanged. lifecycle, scanner and replication reference none of the four error names. warm_backend.rs and warm_backend_s3.rs are held: they compare against rustfs_s3_client::ErrorResponse.code, which stays the s3s type until s3-client moves.

* ci(footprint): count only s3_error! lines still on s3s's macro

s3_error_lines counted every line holding the text `s3_error!`, which
also matches the RustFS-owned `rustfs_s3_types::s3_error!`. A codemod
that moves call sites onto rustfs-s3-types therefore could not lower it
(rustfs/backlog#2743, decision 3; rustfs/rustfs#8432, gotcha 2).

The counter now applies a textual, per-file and per-line rule that errs
on the side of counting (full rule in the script header):
- occurrences spelled `rustfs_s3_types::s3_error!` never count;
- a line holding `s3s::s3_error!` always counts;
- any other `s3_error!` line counts unless its file is cleared, that is,
  it imports the macro from rustfs_s3_types by name on a line starting
  with the use item (or defines it with `macro_rules! s3_error`) and does
  not also bring s3s's macro in (`use s3s::s3_error`, a `use s3s::{..}`
  tree naming it or a crate-root `*`, `use s3s::*`, or
  `#[macro_use] extern crate s3s`);
- fail closed: a file matching neither form (`use super::*`, a
  crate-local re-export, a renamed or glob rustfs_s3_types import, a
  commented-out import, a comment-only mention) counts.

Census of the 1585 lines on 96fc06776: 1478 in 64 files that import the
macro from s3s; 107 in 15 files matching neither form (105 through
`use super::*`, 1 qualified `s3s::s3_error!`, 1 comment line); none in
files that import it from rustfs_s3_types or define it. Every line stays
counted, so the baseline remains s3_error_lines=1585 (files=202 and
ecstore_files=36 unchanged). Rewriting rustfs/src/auth.rs's import to
rustfs_s3_types lowers the new counter to 1567; the old one stayed 1585.

`--self-test` (wired into `make script-tests`) copies the script into
22 fixture trees: 6 positive cases and 16 negative ones (lines that must
not count, forms that must not clear a file, and a missing baseline,
missing key or failing classification search that must fail). Each tree
has a control line that must count, and each run gets s3s text on stdin.
Seven cases were red against the old counter. 19 mutants each turn their
case red: old definition, flipped import test, dropped fail-closed
branch, dropped e2e_test exclusion, dropped s3s veto, dropped qualified
s3s rule, dropped rustfs-qualified strip, unanchored rustfs import,
renamed import accepted, nested glob taken as root glob, single-line
s3s trees only, dropped s3s glob forms, dropped macro_use form, dropped
definition clearing, ignored classification rg errors, error-line search
without its path, classification searches without their path, missing
baseline skipped, missing key skipped.

* test(e2e): compare two RUSTFS_S3_STACK servers over every S3 operation (#8435)

Add dual_stack_differential to e2e_test. It starts two rustfs processes
from the same binary, stack A on RUSTFS_S3_STACK=legacy and stack B on
RUSTFS_E2E_STACK_B (default legacy until rustfs/backlog#2734 T2.5),
sends both the same aws-sdk-s3 script covering the 73 methods of
impl S3 for FS (a normal request and a 4xx negative each), and compares
status, error code, an allowlist of response headers and a few typed
body fields. Error message text is never compared. Each run writes
<target>/dual-stack/<run>.json with compared_ops and the (op, field,
a, b) differences; any difference fails the test.

differential_detects_an_injected_difference starts B with
RUSTFS_REGION=eu-west-1 and requires GetBucketLocation to be reported.
The comparator is pure and unit-tested without a server.

Wire the module into the e2e-smoke filter and update the e2e-smoke and
e2e-full selection digests and the e2e_test test-count row.

Refs rustfs/backlog#2740

* feat(app): add the RustFS-owned RequestEnvelope and its s3s constructor (#8437)

* feat(app): add the RustFS-owned RequestEnvelope and its s3s constructor

* ci: classify the s3s edge adapters in storage/s3_compat as interface

* feat(s3-types): add Body::from_stream and move Swift off s3s::Body (#8438)

rustfs/backlog#2747 batch 3. `Body::from_stream` wraps a chunk stream of
unknown length over `StreamBody`/`BoxBody`: one data frame per chunk, the
first error ends the body, the size hint stays unknown, and dropping the
body drops the stream. Parity tests pin it to the s3s `StreamingBlob::wrap`
it replaces, at the `http_body` level and byte for byte through hyper.

The Swift handlers in rustfs-protocols build their responses as the owned
`Body`. `SwiftService` converts them into the wrapped S3 service's response
body type through `From<Body>`, so rustfs keeps serving `s3s::Body` at the
legacy edge unchanged. Request bodies stay Send-only (`UnsyncBoxBody`); the
two storage writers and the DLO/SLO segment streams get `Sync` from
`sync_wrapper::SyncStream`, which is what s3s's `http_body_unsync` mutex
provided before. The DTO and error uses of s3s in protocols are untouched
(T1.5 and T1.2).

* refactor(protocols): move error types to rustfs-s3-types by codemod (#8439)

T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and
the error items of protocols move off s3s by scripts/codemods, run with
--hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it
downcasts the backend error, whose runtime type is still s3s::S3Error.

- rustfs/src/protocols/client.rs: the two webdav session methods now
  return rustfs_s3_types::S3Result and convert the s3s error with From.
- codemod: a child module that calls s3_error! through `use super::*`
  gets its own `use rustfs_s3_types::s3_error;` once its parent imports
  that macro by name; otherwise it is listed as GLOB.
- s3-types: a table test pins code, status and the s3s error the legacy
  edge renders for every variant a moved call site names.
- s3_error_lines 1585 -> 1584 (a lifecycle comment reworded).

* test(connect): restore one-second drive request budgets

Apply the exact test-only repair already landed on main in 5972a09cc3. The filesystem probe and production CLI success test request one second and keep their existing one-second duration and parameter checks. All outcome, I/O, error, cleanup and resource-limit assertions remain.

Scope: rustfs/tests/connect_perf_drive.rs only. Patch-id: cb0a1dfc8f68a67727525adea7458e2cee7d09f8.

Validation environment: LANG=en_US.UTF-8, CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0, exclusive APFS-cloned CARGO_TARGET_DIR.

Quick Checks: every command from .github/actions/quick-checks/action.yml, with captured output and exit status below.

$ shellcheck --version && actionlint
ShellCheck - shell script analysis tool
version: 0.11.0
license: GNU General Public License, version 3
website: https://www.shellcheck.net
exit_code=0

$ cargo fmt --all --check
exit_code=0

$ ./scripts/check_unsafe_code_allowances.sh
exit_code=0

$ ./scripts/check_layer_dependencies.sh
Layer dependency guard passed (no new reverse dependencies/cycles).
exit_code=0

$ ./scripts/check_architecture_migration_rules.sh
Architecture migration rules passed.
exit_code=0

$ ./scripts/check_logging_guardrails.sh
✅ Logging guardrails check passed
exit_code=0

$ ./scripts/check_error_other_format_ratchet.sh
error(format!) ratchet passed (total call sites: 582).
exit_code=0

$ ./scripts/check_no_tokio_io_uring.sh
exit_code=0

$ ./scripts/check_extension_schema_boundaries.sh
Extension schema boundaries passed.
exit_code=0

$ ./scripts/check_body_cache_whitelist.sh
Body-cache whitelist guard passed.
exit_code=0

$ ./scripts/check_s3s_footprint.sh
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)
✅ s3s footprint baseline check passed
exit_code=0

$ ./scripts/check_test_count_ratchet.sh
test-count ratchet OK: 52 crates, 21222 tests, 164 ignored match .config/test-count-baseline.txt
exit_code=0

$ ./scripts/check_fips_wording.sh
FIPS wording guard passed (README.md and CHANGELOG.md contain no forbidden claims).
KMS wording guard passed (crates/kms claims no Vault KV2 Transit wrapping).
exit_code=0

$ ./scripts/check_embedded_secrets.sh
Embedded secret guard passed (no private key material or provider credential literal in the tree).
exit_code=0

$ make script-tests
Running script tests...
entrypoint credential policy tests passed
internode grpc A/B bench env tests passed
object batch benchmark enhanced tests passed
PASS package service script contract
hotpath warp A/B gate tests passed
hotpath warp ABBA tests passed
Scanner validation artifacts written to /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.1pFe6Kl1Yu/out
Scanner validation artifacts written to /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.1pFe6Kl1Yu/out-missing-pid
.............................
----------------------------------------------------------------------
Ran 29 tests in 181.286s

OK
{"decision":"verified","pending_gates":["G01","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G02"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-E"}
{"decision":"verified","pending_gates":["G01","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G02"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-E"}
{"decision":"verified","pending_gates":["G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G01"}
{"decision":"verified","pending_gates":["G01","G02","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G03"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-L"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G14"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G14"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G10"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P1"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P2"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P3"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G05"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G06"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-D"}
Status-and-outcome release descriptor verified: /private/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp8rwn1pe0/descriptor/release-bundle-status-outcome.json
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G05"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G06"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-D"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G07"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G08"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P4"}
exact 1MiB handoff ABBA tests passed
pinned paired ABBA bench tests passed
==> /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/bin/path_containment (-max_total_time=7, -seed=123456789, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/explicit-artifacts/path_containment/, corpus=/private/tmp/gw-orchestrator-2734/ci-drive/fuzz/corpus/path_containment)
-seed=123456789
target=path_containment
seed=123456789
max_total_time=7
git_revision=4d15b816236630b2c42f321ffc9ed3654edc94e3
git_dirty=true
runner_mode=prebuilt
==> /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/bin/path_containment (-max_total_time=1, -seed=2322617706, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/auto-artifacts/path_containment/, corpus=/private/tmp/gw-orchestrator-2734/ci-drive/fuzz/corpus/path_containment)
-seed=2322617706
==> cargo +nightly fuzz run path_containment (-max_total_time=9, -seed=987654321, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/cargo-artifacts/path_containment/)
+nightly
fuzz
run
path_containment
-seed=987654321
seed=987654321
runner_mode=cargo-fuzz
fuzz runner seed manifest tests passed
✅ scripts/python_bin.sh resolver checks passed
self-test ok: clean fixture with every exemption present passes
self-test ok: all 11 pattern families fire
self-test ok: moving a pinned source is reported
Embedded secret guard self-test passed.
self-test ok: p1-s3s-import: 'use s3s::s3_error;' counts every line, the rustfs_s3_types module's too (s3_error_lines=3)
self-test ok: p2-s3s-mixed-tree: a multi-line s3s use tree naming s3_error counts (s3_error_lines=3)
self-test ok: p3-s3s-glob: a crate-root glob of s3s, bare or inside a tree, counts (s3_error_lines=5)
self-test ok: p4-macro-use: '#[macro_use] extern crate s3s' counts (s3_error_lines=3)
self-test ok: p5-qualified-s3s: a qualified s3s::s3_error! line counts in a cleared file (s3_error_lines=2)
self-test ok: p6-exact-pass: a baseline equal to the tree passes (exit 0)
self-test ok: n1-rustfs-import: 'use rustfs_s3_types::s3_error;' alone counts nothing (s3_error_lines=1)
self-test ok: n2-rustfs-mixed-tree: a rustfs_s3_types tree plus an s3s tree without the macro counts nothing (s3_error_lines=1)
self-test ok: n3-defines: the file defining the macro counts nothing (s3_error_lines=1)
self-test ok: n4-qualified-rustfs: a qualified rustfs_s3_types::s3_error! line counts nothing (s3_error_lines=1)
self-test ok: n5-renamed-call: a call through a renamed macro is not text-matched (out of scope) (s3_error_lines=1)
self-test ok: n6-comment-string: a comment or string mentioning s3_error! in a cleared file counts nothing (s3_error_lines=1)
self-test ok: n7-e2e-excluded: crates/e2e_test is excluded (s3_error_lines=1)
self-test ok: n8-super-glob: an unrecognised import form ('use super::*;') counts (s3_error_lines=3)
self-test ok: n9-facade: a crate-local re-export counts (s3_error_lines=2)
self-test ok: n10-renamed-import: a renamed rustfs_s3_types import does not clear s3_error! lines (s3_error_lines=2)
self-test ok: n11-commented-import: a commented-out rustfs_s3_types import does not clear (s3_error_lines=2)
self-test ok: n12-rustfs-glob: a glob import of rustfs_s3_types does not clear (s3_error_lines=2)
self-test ok: n13-comment-only: a comment mentioning s3_error! in a file with no import counts (s3_error_lines=2)
self-test ok: f1-baseline-missing: a missing baseline file fails (exit 1)
self-test ok: f2-key-missing: a baseline without s3_error_lines fails (exit 1)
self-test ok: f3-rg-fails: a failing classification search fails the guard (exit 1)
✅ s3s footprint self-test passed
test_core_gate_rejects_missing_ignored_filtered_and_corrupt_inputs (__main__.SelfTests.test_core_gate_rejects_missing_ignored_filtered_and_corrupt_inputs) ... ok
test_e2e_requires_registration (__main__.SelfTests.test_e2e_requires_registration) ... ok
test_fuzz_runtime_uses_matrix_target (__main__.SelfTests.test_fuzz_runtime_uses_matrix_target) ... ok
test_ilm_lane_keeps_the_measured_cargo_build_budget (__main__.SelfTests.test_ilm_lane_keeps_the_measured_cargo_build_budget) ... ok
test_profile_listing_binds_platform_digest (__main__.SelfTests.test_profile_listing_binds_platform_digest) ... ok
test_profile_listing_enforces_selection (__main__.SelfTests.test_profile_listing_enforces_selection) ... ok
test_quick_checks_commands_propagate_failure (__main__.SelfTests.test_quick_checks_commands_propagate_failure) ... ok
test_quick_checks_rejects_caller_and_execution_bypasses (__main__.SelfTests.test_quick_checks_rejects_caller_and_execution_bypasses) ... ok
test_runner_readiness_fails_closed (__main__.SelfTests.test_runner_readiness_fails_closed) ... ok
test_s3_runner_rejects_unbounded_failure_locals (__main__.SelfTests.test_s3_runner_rejects_unbounded_failure_locals) ... ok
test_scanner_heal_begin_requires_embedded_source_provenance (__main__.SelfTests.test_scanner_heal_begin_requires_embedded_source_provenance) ... ok
test_scanner_heal_case_does_not_approve_pending_release (__main__.SelfTests.test_scanner_heal_case_does_not_approve_pending_release) ... ok
test_scanner_heal_case_lane_is_required_for_runner_profile (__main__.SelfTests.test_scanner_heal_case_lane_is_required_for_runner_profile) ... ok
test_scanner_heal_case_only_schema_cannot_approve_release (__main__.SelfTests.test_scanner_heal_case_only_schema_cannot_approve_release) ... ok
test_scanner_heal_crash_case_rejects_restart_oracle (__main__.SelfTests.test_scanner_heal_crash_case_rejects_restart_oracle) ... ok
test_scanner_heal_ec84_case_rejects_wrong_erasure_geometry (__main__.SelfTests.test_scanner_heal_ec84_case_rejects_wrong_erasure_geometry) ... ok
test_scanner_heal_finish_collects_oracles_from_registry (__main__.SelfTests.test_scanner_heal_finish_collects_oracles_from_registry) ... ok
test_scanner_heal_multipool_case_accepts_deferred_optional_outage_write (__main__.SelfTests.test_scanner_heal_multipool_case_accepts_deferred_optional_outage_write) ... ok
test_scanner_heal_pending_gate_cannot_map_to_implemented_lane (__main__.SelfTests.test_scanner_heal_pending_gate_cannot_map_to_implemented_lane) ... ok
test_scanner_heal_receipts_reject_reuse_and_missing_builds (__main__.SelfTests.test_scanner_heal_receipts_reject_reuse_and_missing_builds) ... ok
test_scanner_heal_rejects_boolean_fractional_and_out_of_geometry_integers (__main__.SelfTests.test_scanner_heal_rejects_boolean_fractional_and_out_of_geometry_integers) ... ok
test_scanner_heal_rejects_broken_execution_and_artifacts (__main__.SelfTests.test_scanner_heal_rejects_broken_execution_and_artifacts) ... ok
test_scanner_heal_rejects_copied_junit_and_wrong_suite_build (__main__.SelfTests.test_scanner_heal_rejects_copied_junit_and_wrong_suite_build) ... ok
test_scanner_heal_release_bundle_accepts_complete_measured_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_accepts_complete_measured_evidence) ... ok
test_scanner_heal_release_bundle_assembler_copies_measured_artifacts (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_copies_measured_artifacts) ... ok
test_scanner_heal_release_bundle_assembler_merges_measured_descriptors (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_merges_measured_descriptors) ... ok
test_scanner_heal_release_bundle_assembler_rejects_duplicate_descriptor_gate (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_duplicate_descriptor_gate) ... ok
test_scanner_heal_release_bundle_assembler_rejects_fixture_descriptor (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_fixture_descriptor) ... ok
test_scanner_heal_release_bundle_assembler_rejects_hash_mismatch (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_hash_mismatch) ... ok
test_scanner_heal_release_bundle_assembler_rejects_mixed_descriptor_revisions (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_mixed_descriptor_revisions) ... ok
test_scanner_heal_release_bundle_enforces_topology_duration_profile_and_versions (__main__.SelfTests.test_scanner_heal_release_bundle_enforces_topology_duration_profile_and_versions) ... ok
test_scanner_heal_release_bundle_fixture_writer_is_not_release_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_fixture_writer_is_not_release_evidence) ... ok
test_scanner_heal_release_bundle_gate_accepts_single_measured_gate (__main__.SelfTests.test_scanner_heal_release_bundle_gate_accepts_single_measured_gate) ... ok
test_scanner_heal_release_bundle_gate_rejects_missing_field (__main__.SelfTests.test_scanner_heal_release_bundle_gate_rejects_missing_field) ... ok
test_scanner_heal_release_bundle_gate_rejects_unknown_gate (__main__.SelfTests.test_scanner_heal_release_bundle_gate_rejects_unknown_gate) ... ok
test_scanner_heal_release_bundle_rejects_synthetic_or_missing_fields (__main__.SelfTests.test_scanner_heal_release_bundle_rejects_synthetic_or_missing_fields) ... ok
test_scanner_heal_release_bundle_requires_domain_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_requires_domain_evidence) ... ok
test_scanner_heal_release_bundle_requires_field_provenance (__main__.SelfTests.test_scanner_heal_release_bundle_requires_field_provenance) ... ok
test_scanner_heal_release_bundle_requires_json_artifact_provenance (__main__.SelfTests.test_scanner_heal_release_bundle_requires_json_artifact_provenance) ... ok
test_scanner_heal_release_bundle_requires_mixed_version_field_roles (__main__.SelfTests.test_scanner_heal_release_bundle_requires_mixed_version_field_roles) ... ok
test_scanner_heal_release_bundle_requires_same_gate_measurement_window (__main__.SelfTests.test_scanner_heal_release_bundle_requires_same_gate_measurement_window) ... ok
test_scanner_heal_release_consumes_multiple_registry_oracles (__main__.SelfTests.test_scanner_heal_release_consumes_multiple_registry_oracles) ... ok
test_scanner_heal_release_matrix_lanes_remain_pending_without_real_evidence (__main__.SelfTests.test_scanner_heal_release_matrix_lanes_remain_pending_without_real_evidence) ... ok
test_scanner_heal_release_requirements_cannot_be_cleared (__main__.SelfTests.test_scanner_heal_release_requirements_cannot_be_cleared) ... ok
test_scanner_heal_release_status_rejects_focused_case_run (__main__.SelfTests.test_scanner_heal_release_status_rejects_focused_case_run) ... ok
test_scanner_heal_release_status_rejects_synthetic_case (__main__.SelfTests.test_scanner_heal_release_status_rejects_synthetic_case) ... ok
test_scanner_heal_required_evidence_fields_cannot_be_removed (__main__.SelfTests.test_scanner_heal_required_evidence_fields_cannot_be_removed) ... ok
test_scheduled_alerts_require_completion_watchdog (__main__.SelfTests.test_scheduled_alerts_require_completion_watchdog) ... ok
test_update_profile_selection_changes_only_requested_platform (__main__.SelfTests.test_update_profile_selection_changes_only_requested_platform) ... ok
test_validate_includes_quick_checks (__main__.SelfTests.test_validate_includes_quick_checks) ... ok
test_vault_tests_require_cross_process_serialization (__main__.SelfTests.test_vault_tests_require_cross_process_serialization) ... ok
test_workflow_readiness_requires_dependency_probe (__main__.SelfTests.test_workflow_readiness_requires_dependency_probe) ... ok

----------------------------------------------------------------------
Ran 52 tests in 14.301s

OK
e2e-full selection OK: 1 tests, sha256=29e30e6ed2ed38d9e5b0a7cdd9536abf5c719f7021a02b578ae1381563a3c273
...................................
----------------------------------------------------------------------
Ran 35 tests in 1.161s

OK
...................
----------------------------------------------------------------------
Ran 19 tests in 17.397s

OK
..........
----------------------------------------------------------------------
Ran 10 tests in 0.183s

OK
test_actual_selector_bootstrap_uses_base_policy_and_fails_closed (__main__.SelfTests.test_actual_selector_bootstrap_uses_base_policy_and_fails_closed) ... ok
test_documentation_paths_do_not_hide_build_or_fixture_changes (__main__.SelfTests.test_documentation_paths_do_not_hide_build_or_fixture_changes) ... ok
test_e2e_scope_keeps_production_and_shared_configuration_full (__main__.SelfTests.test_e2e_scope_keeps_production_and_shared_configuration_full) ... ok
test_e2e_shortcut_rejects_new_direct_renamed_and_target_dependencies (__main__.SelfTests.test_e2e_shortcut_rejects_new_direct_renamed_and_target_dependencies) ... ok
test_e2e_shortcut_traverses_implicit_path_dependencies (__main__.SelfTests.test_e2e_shortcut_traverses_implicit_path_dependencies) ... ok
test_event_contract_requires_complete_candidate_and_optional_lanes (__main__.SelfTests.test_event_contract_requires_complete_candidate_and_optional_lanes) ... ok
test_every_wrong_result_missing_job_or_selection_fails_closed (__main__.SelfTests.test_every_wrong_result_missing_job_or_selection_fails_closed) ... ok
test_full_e2e_gate_preserves_workflow_branch_and_event_scope (__main__.SelfTests.test_full_e2e_gate_preserves_workflow_branch_and_event_scope) ... ok
test_git_range_includes_deleted_source_and_rename_origins (__main__.SelfTests.test_git_range_includes_deleted_source_and_rename_origins) ... ok
test_job_ids_and_display_names_cannot_hide_validation (__main__.SelfTests.test_job_ids_and_display_names_cannot_hide_validation) ... ok
test_report_only_stack_lane_mirrors_e2e_smoke_outside_the_aggregate (__main__.SelfTests.test_report_only_stack_lane_mirrors_e2e_smoke_outside_the_aggregate) ... ok
test_repository_wiring_and_missing_dependency_regression (__main__.SelfTests.test_repository_wiring_and_missing_dependency_regression) ... ok
test_verify_command_preserves_failures (__main__.SelfTests.test_verify_command_preserves_failures) ... ok

----------------------------------------------------------------------
Ran 13 tests in 3.280s

OK
security coverage self-test passed
test_api_failures_preserve_other_evidence_and_never_enter_grace (__main__.SelfTests.test_api_failures_preserve_other_evidence_and_never_enter_grace) ... ok
test_attempt_outcome_does_not_replace_recent_success (__main__.SelfTests.test_attempt_outcome_does_not_replace_recent_success) ... ok
test_cancelled_attempt_cannot_refresh_expired_success (__main__.SelfTests.test_cancelled_attempt_cannot_refresh_expired_success) ... ok
test_cli_requires_the_repository_default_branch (__main__.SelfTests.test_cli_requires_the_repository_default_branch) ... ok
test_config_rejects_duplicate_and_invalid_entries (__main__.SelfTests.test_config_rejects_duplicate_and_invalid_entries) ... ok
test_freshness_boundaries (__main__.SelfTests.test_freshness_boundaries) ... ok
test_grace_requires_two_successful_queries_with_no_history (__main__.SelfTests.test_grace_requires_two_successful_queries_with_no_history) ... ok
test_invalid_api_evidence_fails_closed (__main__.SelfTests.test_invalid_api_evidence_fails_closed) ... ok
test_report_retains_every_workflow (__main__.SelfTests.test_report_retains_every_workflow) ... ok
test_requests_filter_schedule_default_branch_and_success_on_server (__main__.SelfTests.test_requests_filter_schedule_default_branch_and_success_on_server) ... ok

----------------------------------------------------------------------
Ran 10 tests in 0.051s

OK
..............................
----------------------------------------------------------------------
Ran 30 tests in 22.354s

OK
..........
----------------------------------------------------------------------
Ran 10 tests in 2.969s

OK
.................................
----------------------------------------------------------------------
Ran 33 tests in 0.160s

OK
Online functional runner labels: pf-testing
.......
----------------------------------------------------------------------
Ran 7 tests in 0.002s

OK
.......
----------------------------------------------------------------------
Ran 7 tests in 0.001s

OK
....
----------------------------------------------------------------------
Ran 4 tests in 0.128s

OK
...
----------------------------------------------------------------------
Ran 3 tests in 0.547s

OK
PASS: cold stampede script self-test
exit_code=0

$ python3 ./scripts/test_object_generation_protocol_model.py
test_a_recovery_barrier_cannot_choose_an_empty_value (__main__.ObjectGenerationProtocolModel.test_a_recovery_barrier_cannot_choose_an_empty_value) ... ok
test_all_bounded_two_proposer_interleavings_choose_at_most_one_value (__main__.ObjectGenerationProtocolModel.test_all_bounded_two_proposer_interleavings_choose_at_most_one_value) ... ok
test_an_unavailable_decision_quorum_cannot_choose_a_candidate (__main__.ObjectGenerationProtocolModel.test_an_unavailable_decision_quorum_cannot_choose_a_candidate) ... ok
test_counter_allocation_requires_persistence_and_is_consumed_once (__main__.ObjectGenerationProtocolModel.test_counter_allocation_requires_persistence_and_is_consumed_once) ... ok
test_counter_restart_burns_persisted_allocations_before_or_after_issue (__main__.ObjectGenerationProtocolModel.test_counter_restart_burns_persisted_allocations_before_or_after_issue) ... ok
test_crash_after_promise_before_reply_does_not_restore_an_older_ballot (__main__.ObjectGenerationProtocolModel.test_crash_after_promise_before_reply_does_not_restore_an_older_ballot) ... ok
test_duplicate_or_unconfigured_voters_cannot_form_a_quorum (__main__.ObjectGenerationProtocolModel.test_duplicate_or_unconfigured_voters_cannot_form_a_quorum) ... ok
test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision (__main__.ObjectGenerationProtocolModel.test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision) ... ok
test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected (__main__.ObjectGenerationProtocolModel.test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected) ... ok
test_forgetting_tombstone_authority_allows_an_aba_create (__main__.ObjectGenerationProtocolModel.test_forgetting_tombstone_authority_allows_an_aba_create) ... ok
test_issuing_before_persistence_reuses_a_ballot_after_restart (__main__.ObjectGenerationProtocolModel.test_issuing_before_persistence_reuses_a_ballot_after_restart) ... ok
test_lost_ack_and_restart_preserve_the_chosen_operation (__main__.ObjectGenerationProtocolModel.test_lost_ack_and_restart_preserve_the_chosen_operation) ... ok
test_minority_accepted_work_is_adopted_and_finished (__main__.ObjectGenerationProtocolModel.test_minority_accepted_work_is_adopted_and_finished) ... ok
test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample (__main__.ObjectGenerationProtocolModel.test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample) ... ok
test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it (__main__.ObjectGenerationProtocolModel.test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it) ... ok
test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor (__main__.ObjectGenerationProtocolModel.test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor) ... ok
test_replacing_voters_with_empty_state_can_choose_a_conflicting_value (__main__.ObjectGenerationProtocolModel.test_replacing_voters_with_empty_state_can_choose_a_conflicting_value) ... ok
test_skipping_predecessor_recovery_chooses_a_broken_revision_chain (__main__.ObjectGenerationProtocolModel.test_skipping_predecessor_recovery_chooses_a_broken_revision_chain) ... ok
test_stale_publication_rollback_and_gc_cannot_change_the_winner (__main__.ObjectGenerationProtocolModel.test_stale_publication_rollback_and_gc_cannot_change_the_winner) ... ok
test_superseded_operation_identity_cannot_be_reused_for_different_input (__main__.ObjectGenerationProtocolModel.test_superseded_operation_identity_cannot_be_reused_for_different_input) ... ok
test_superseded_operation_retry_returns_its_original_successor (__main__.ObjectGenerationProtocolModel.test_superseded_operation_retry_returns_its_original_successor) ... ok
test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision (__main__.ObjectGenerationProtocolModel.test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision) ... ok
test_tombstone_rejects_a_delayed_never_present_create (__main__.ObjectGenerationProtocolModel.test_tombstone_rejects_a_delayed_never_present_create) ... ok
test_unlearned_predecessor_is_recovered_before_the_second_slot (__main__.ObjectGenerationProtocolModel.test_unlearned_predecessor_is_recovered_before_the_second_slot) ... ok

----------------------------------------------------------------------
Ran 24 tests in 0.958s

OK
model: states=51023, transitions=167520, terminal=732, chosen_candidates=2
negative control: double-choice trace=((0, 'prepare', 0), (0, 'prepare', 1), (0, 'prepare', 2), (0, 'accept', 0), (0, 'accept', 1), (0, 'accept', 2), (1, 'prepare', 0), (1, 'prepare', 1), (1, 'prepare', 2), (1, 'accept', 0), (1, 'accept', 1), (1, 'accept', 2))
multi-slot round schedules: 640
exit_code=0

$ python3 ./scripts/check_test_wiring.py
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired
exit_code=0

$ ./scripts/check_no_planning_docs.sh
exit_code=0

$ ./scripts/check_uring_lane_lib_only.sh
OK: no 'uring_'-matching test functions under crates/ecstore/tests; --lib is safe
exit_code=0

Initial complete gate
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
 Nextest run ID e07fc5b9-c6c8-4cdd-ae61-54839bf24437 with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.016s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.011s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.009s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.009s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  16.950s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   4.025s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.008s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.007s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.035s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  21.087s] 10 tests run: 10 passed, 0 skipped
exit_code=0

Restored complete gate
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
 Nextest run ID 233b7307-5361-4eb7-94d5-b384f17d2d6a with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.009s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.008s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.008s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.008s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  19.518s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   4.146s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.009s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.018s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.009s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.054s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  23.787s] 10 tests run: 10 passed, 0 skipped
exit_code=0

Mutation M1: success_measurement reports max(actual elapsed, requested budget + 1), forcing 1001 ms for the one-second request. The retained duration assertion at rustfs/tests/connect_perf_drive.rs:105 fails.
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast -E 'test(real_local_write_read_is_measured_and_scratch_is_removed)'
info: experimental features enabled: setup-scripts
   Compiling rustfs v1.0.1 (/private/tmp/gw-orchestrator-2734/ci-drive/rustfs)
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs") generated 1 warning
warning: `rustfs` (bin "rustfs-cli") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 7m 06s
────────────
 Nextest run ID 9f618777-7bf6-4cbc-8803-dd7f50f449ed with nextest profile: ci
    Starting 1 test across 1 binary (9 tests skipped)
        FAIL [   0.016s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
  stdout ───

    running 1 test
    test real_local_write_read_is_measured_and_scratch_is_removed ... FAILED

    failures:

    failures:
        real_local_write_read_is_measured_and_scratch_is_removed

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 0.01s

  stderr ───

    thread 'real_local_write_read_is_measured_and_scratch_is_removed' (106986846) panicked at rustfs/tests/connect_perf_drive.rs:105:5:
    assertion failed: (1..=1_000).contains(&data.duration_millis)
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

────────────
     Summary [   0.017s] 1 test run: 0 passed, 1 failed, 9 skipped
        FAIL [   0.016s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
error: test run failed
exit_code=100

Mutation M2: target_parameters reports requested duration + 1. The in-process assertion at rustfs/tests/connect_perf_drive.rs:111 and CLI JSON assertion at rustfs/tests/connect_perf_drive.rs:527 both reject 1001 against 1000.
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast -E 'test(real_local_write_read_is_measured_and_scratch_is_removed) | test(production_cli_measures_and_signs_exact_binary_provenance)'
info: experimental features enabled: setup-scripts
   Compiling rustfs v1.0.1 (/private/tmp/gw-orchestrator-2734/ci-drive/rustfs)
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs") generated 1 warning
warning: `rustfs` (bin "rustfs-cli") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 7m 20s
────────────
 Nextest run ID b73d46ab-c910-4e1e-8063-7fc7e332ec15 with nextest profile: ci
    Starting 2 tests across 1 binary (8 tests skipped)
        FAIL [  13.968s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
  stdout ───

    running 1 test
    test production_cli_measures_and_signs_exact_binary_provenance ... FAILED

    failures:

    failures:
        production_cli_measures_and_signs_exact_binary_provenance

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 13.96s

  stderr ───

    thread 'production_cli_measures_and_signs_exact_binary_provenance' (107015462) panicked at rustfs/tests/connect_perf_drive.rs:527:5:
    assertion `left == right` failed
      left: Number(1001)
     right: 1000
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

        FAIL [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
  stdout ───

    running 1 test
    test real_local_write_read_is_measured_and_scratch_is_removed ... FAILED

    failures:

    failures:
        real_local_write_read_is_measured_and_scratch_is_removed

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 0.01s

  stderr ───

    thread 'real_local_write_read_is_measured_and_scratch_is_removed' (107016667) panicked at rustfs/tests/connect_perf_drive.rs:111:5:
    assertion `left == right` failed
      left: 1001
     right: 1000
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

────────────
     Summary [  13.985s] 2 tests run: 0 passed, 2 failed, 8 skipped
        FAIL [  13.968s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        FAIL [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
error: test run failed
exit_code=100

Transient production mutations were restored byte-for-byte before the complete green rerun. Original/restored SHA256: 8d389e6d8836c041aab8c6a93e56addd65ba3f70b33264e1c74f639d0572a2da.

$ git diff --check

exit_code=0

$ git diff --cached --check

exit_code=0

Independent orchestrator acceptance
Tested code tree: e67c72fb01

Tested tree: 16fbbd2ea4
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

$ LANG=en_US.UTF-8 CARGO_BUILD_JOBS=1 CARGO_INCREMENTAL=0 cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
info: experimental features enabled: setup-scripts
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs-cli") generated 1 warning
warning: `rustfs` (bin "rustfs") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.69s
────────────
 Nextest run ID cd1edf4c-f1c0-456d-ad1a-76cc00ce05c9 with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.012s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.010s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.010s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.009s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  20.328s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   5.527s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.013s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.019s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.011s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.049s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  25.991s] 10 tests run: 10 passed, 0 skipped

exit_code=0

All 54 tracked Cargo.toml files parse without duplicate keys. git diff --check passed. The production source remains unchanged; the two deliberate implementation mutations above were restored before this complete green run.

* fix(test): compare legacy body wire parity as raw bytes

Lossy UTF-8 comparison concealed a changed binary response byte.
Compare the already captured Vec<u8> values directly.

Validation:
- Binary mutant: migrated payload 0x84 vs legacy 0x83; old comparator
  passed 1/1, raw comparator failed 0/1 (exit 100); restored green.
- Non-Swift body-drain mutant: untouched-body delegation test failed
  with empty vs payload (exit 100); restored handler without a diff.
- cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols
  --features rustfs-protocols/swift --no-fail-fast:
  543 tests run, 543 passed, 0 skipped after both mutants restored.
- cargo fmt --all --check and git diff --check passed.

Orchestrator acceptance (LANG=en_US.UTF-8):
cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
Summary [2.445s] 543 tests run: 543 passed, 0 skipped.

All shared quick checks on the final code tree:
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

Final independent acceptance after the drive-test CI repair
Parent: a99d8f4f89
Tested code tree: 93cbd96bf0

Tested tree: 075ae5161c2b62a96812bc1e3c904a7663b4f2f8
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

$ LANG=en_US.UTF-8 CARGO_BUILD_JOBS=1 CARGO_INCREMENTAL=0 cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
    Finished `test` profile [unoptimized + debuginfo] target(s) in 10m 08s
 Nextest run ID 5c7ff102-ec6d-479a-86e0-810859b28bbf with nextest profile: ci
    Starting 543 tests across 8 binaries
     Summary [   6.729s] 543 tests run: 543 passed, 0 skipped
exit_code=0

The only source change is the raw-byte wire comparator. Cargo manifests and the ratchet baselines remain unchanged from the independently validated parent. The invalid-byte and consumed-request mutations remain recorded above; their implementation changes were restored.

* fix: repair integration checks and IPC test readiness

Wait for the RPC/locks test sockets to pass the existing owner and mode
checks inside the original 5s readiness window and 10ms poll interval.

Use the existing admin S3 carrier for the new recovery and quota error
calls. Regenerate the test-count baseline for main's 118 existing tests
across seven crates. Pair the private pagination helper's raw-scan inputs
and remove the redundant Copy clone from the retirement event.

Keep the original assertions, responses, authorization, locks and storage
operations. The pagination body and raw-scan expressions at all five calls
are byte-identical; reinserting only the Copy clone restores the entire
base heal file. All ignored baseline values and s3s counts stay unchanged.

Validation base: c47432e353
Validation tree: f081084bc5
Patch SHA-256: 2dcf0028691bfb2772edfb38e262829b04f4151832dcce823c9dd9f7434b52de
Trace production-prefix SHA-256: fd994f218a21429d62866f4efb8eee1ee57b58f4a51865aa35d05a54ef1ccb45

All commands used LANG=en_US.UTF-8, CARGO_BUILD_JOBS=1,
CARGO_INCREMENTAL=0 and CARGO_NET_OFFLINE=true in one exclusive target.
Every gate retained the same tree and six source hashes before and after.
54 Cargo.toml files parse without duplicate keys.

cargo clippy -p rustfs --all-targets -- -D warnings
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 12m 50s
Exit: 0. Wall time: 770.855s.

The original 19 quick-check steps:
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
Wall time: 286.337s.

cargo nextest run --profile ci -p rustfs --no-fail-fast
     Summary [ 217.017s] 5382 tests run: 5382 passed (4 leaky), 40 skipped
Exit: 0. Wall time: 1324.414s.

cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
     Summary [   2.189s] 543 tests run: 543 passed, 0 skipped
Exit: 0. Wall time: 123.192s.

cargo nextest run --profile ci -p rustfs-ecstore list_objects_paginate --no-fail-fast
     Summary [   0.170s] 8 tests run: 8 passed, 5921 skipped
Exit: 0. Wall time: 261.324s.

cargo nextest run --profile ci -p rustfs-heal durable_mrf_anchor_retires_only_after_exact_bucket_generation_is_deleted --no-fail-fast
     Summary [   0.805s] 1 test run: 1 passed, 835 skipped
Exit: 0. Wall time: 199.801s.

Baseline regeneration:
./scripts/check_test_count_ratchet.sh --update
wrote .config/test-count-baseline.txt: 52 crates, 21340 tests, 164 ignored
Test-only increases: ecstore48, heal3, iam14, io-metrics2, obs3, policy14,
rustfs34. Existing ignored values and all other rows are byte-identical.
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)

Historical runtime red controls on f622 (same retained two assertions):
cargo nextest run --profile ci -p rustfs --lib -E 'test(=connect::diagnostics::trace_runtime::tests::local_top_rpc_reads_service_http_completions_and_signs_offline) | test(=connect::diagnostics::trace_runtime::tests::local_top_locks_reads_service_manager_and_signs_offline)' --no-fail-fast
Original path readiness with a temporary 500ms bind/seal gap:
     Summary [   0.863s] 2 tests run: 0 passed, 2 failed, 5113 skipped
Exit: 100.
Sealed readiness in that same gap:
     Summary [   5.621s] 2 tests run: 2 passed, 5113 skipped
Exit: 0.
Compiled M1: replace the helper predicate with path.exists():
     Summary [   0.870s] 2 tests run: 0 passed, 2 failed, 5113 skipped
Exit: 100.
Restore the sealed predicate byte-for-byte:
     Summary [   5.640s] 2 tests run: 2 passed, 5113 skipped
Exit: 0.
Both reds observe mode755 with owner_matches=true and fail at the original
RPC/locks unwraps with StateSecurity. M1 failure locations: trace_runtime.rs
2547:29 and 2713:29. No new assertions were introduced.

cargo nextest run --profile ci -p rustfs --lib -E 'test(connect::diagnostics::trace_runtime::tests::)' --no-fail-fast
     Summary [  20.748s] 28 tests run: 28 passed (1 leaky), 5087 skipped
Exit: 0. This security sweep is historical; the full final unit gate above
covers the actual c474 candidate.
The first disk-stopped compile ran no tests and is excluded from red credit.
All temporary seal instrumentation is removed from the final source.

Final unit cleanup markers: 4 leaky, 40 original skips (exit0). The ld
__eh_frame linker warning also appears verbatim in the original f622 run
and the earlier c474 run; there are no new source warnings.

* fix: remove redundant pending closure in disk test

Pass the capture-free pending future factory as a function item in the existing slow-operation test. This fixes the redundant_closure error at disk_store.rs:2914 that stopped strict Clippy before the feature tests could run.

The single-line test edit preserves the factory's result, warning and timeout behavior. Reversing the replacement reproduces the parent file byte for byte. All 220 assertion lines and the production prefix are unchanged; no tests, assertions, guards, profiles or baselines were added or relaxed.

Original Linux CI evidence (run 37888552268 at 53245be9f):
- RIO job113685483992: strict Clippy exit101, redundant closure at disk_store.rs:2914; feature tests skipped.
- Workspace Test and Lint job113685484058: same strict Clippy exit101; nextest and doc tests skipped.
These are lint failures, not runtime mutation kills or known flakes.

Native macOS arm64 validation on tree7a5bf47e7533b29479bc85207416102f1337c566, LANG=en_US.UTF-8, one build job, CARGO_INCREMENTAL=0 and offline dependencies:

shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
All 19 checks: exit0, 312.161s.

$ cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 20m 11s
exit0, 1213.210s total.

$ cargo nextest run --profile ci -p rustfs-ecstore --features rio-v2 slow_disk_operation_warns_once_while_pending_and_preserves_result --no-fail-fast
Finished `test` profile [unoptimized + debuginfo] target(s) in 4m 34s
PASS [0.026s] rustfs-ecstore disk::disk_store::tests::slow_disk_operation_warns_once_while_pending_and_preserves_result
Summary [   0.170s] 1 test run: 1 passed, 5943 skipped
exit0, 281.920s total; 5943 existing filter skips.

warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
This linker warning is retained; flags, profiles and allowances are unchanged.

54 tracked Cargo.toml files parsed without duplicate keys. git diff --check passed. No new assertions were introduced, so no new runtime mutation is claimed. Native compilation and the existing helper test do not substitute for actual Linux RIO execution; the new integration CI run still needs to recover before the RPC merge.

Preservation and output hashes:
parent file SHA256: 8e59ee745e3aaaa22848b943d3b10f6353b515956282cde1f3af2c394200c9ac
final file SHA256: b69ba3237061b161448ae8b62b615885d5770b98a5b4a2479bcfbd70f1bd1f17
unchanged production prefix SHA256: 7903a6e288252c420231242d7b58a25fa2a7140ea3dd430cbe7f0dde11369dcb
RIO Linux failure log SHA256:421addb9868fd88e4109a922311e54f91f0dfd70862f8ba7fcaba193482ed90b
Workspace Linux failure log SHA256:e0aea8d0f014c022252a9534968e083c98cfc1fd760da445ecd04ec3f522e26a
19 output SHA256:35a54f28d707956ed1edb29d492c4dfe8a9fe753d103b111b5a2f1b5c41c2b98
RIO Clippy output SHA256:d6046e3868f2b3992fe11cdd04e9e98eb50a87d1234d9a70dbfeb154ff61638d
slow-operation output SHA256:605b7b258d2fd09c6ef2ab2ffc90d433d0509480f1e45125d20c823759e83608

Refs rustfs/backlog#2734.

* test(ecstore): isolate tier-free recovery fixture startup

The fixture constructor can take the production recovery lock and wait for 60 seconds before this test installs its queue and seeds metadata. Disable its background recovery only while creating this fixture, then restore the environment before calling the existing production recovery entrypoint.

Only the existing test setup changes. Its production entrypoint, default lock parameters, original assertions, 30-second timeout, cancellation and cleanup remain intact. No tests or assertions were added. Production and store/init.rs are byte-identical to the parent.

Independent acceptance ran on merge tree 8ec358cb42ba9ebb3bcfeafcef048a3f0366355b; the runner header reports parent HEAD, while before/after index and source hashes confirm this candidate. Worker task branch d8055766b8 has the same tree.

Every command used LANG=en_US.UTF-8. Task compilation also used CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0, CARGO_NET_OFFLINE=true, an independent target, and empty RUSTC_WRAPPER/RUSTC_WORKSPACE_WRAPPER to keep compilers in its owned process group without changing repository configuration.

Command:
shellcheck --version && actionlint
Exit 0; observed output excerpt:
version: 0.11.0
Log SHA-256: fc9eec16b4191d058a295813ca9afbe9d2b37b59975da860170726be162a3924

Command:
cargo fmt --all --check
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_unsafe_code_allowances.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_layer_dependencies.sh
Exit 0; observed output excerpt:
Layer dependency guard passed (no new reverse dependencies/cycles).
Log SHA-256: 69c6cb1896c0586b28d1167380d3260d99d8017a4265c8df5a2348f289e93a12

Command:
./scripts/check_architecture_migration_rules.sh
Exit 0; observed output excerpt:
Architecture migration rules passed.
Log SHA-256: 28790e70d070607ed4de0704f292a65030be92ec224ef106d4b72fd8296f896e

Command:
./scripts/check_logging_guardrails.sh
Exit 0; observed output excerpt:
✅ Logging guardrails check passed
Log SHA-256: 7adf41abff8ed8f7617decf86498888ddde78937099dd9ce47f6d359dfc2d317

Command:
./scripts/check_error_other_format_ratchet.sh
Exit 0; observed output excerpt:
error(format!) ratchet passed (total call sites: 579).
Log SHA-256: 66172aef18f150ba0605af122e26ffcf91f6dd86f43e38b05e2e982ce8d783ee

Command:
./scripts/check_no_tokio_io_uring.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_extension_schema_boundaries.sh
Exit 0; observed output excerpt:
Extension schema boundaries passed.
Log SHA-256: f13d0acd0e2ce5d9815241b12b4dab7d01cc11b9b65bcf6dfb7812c878532a51

Command:
./scripts/check_body_cache_whitelist.sh
Exit 0; observed output excerpt:
Body-cache whitelist guard passed.
Log SHA-256: 6b828847b0383950ee6eef9cd846d0c322c945e47efd4c7d1ad881178810d6bb

Command:
./scripts/check_s3s_footprint.sh
Exit 0; observed output excerpt:
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)
✅ s3s footprint baseline check passed
Log SHA-256: 47f53d04ef645618563fe1f22abc64667f19078f1974feedcca46e4f2c016844

Command:
./scripts/check_test_count_ratchet.sh
Exit 0; observed output excerpt:
test-count ratchet OK: 52 crates, 21340 tests, 164 ignored match .config/test-count-baseline.txt
Log SHA-256: 618a5e330e933de0c6ff484546df401457024a88bb5986d11d588794e33dab60

Command:
./scripts/check_fips_wording.sh
Exit 0; observed output excerpt:
FIPS wording guard passed (README.md and CHANGELOG.md contain no forbidden claims).
KMS wording guard passed (crates/kms claims no Vault KV2 Transit wrapping).
Log SHA-256: 99221cc9979c480ec4f46f50d76c52b18c72fe499f12e15e4f4a22a1b8bfb140

Command:
./scripts/check_embedded_secrets.sh
Exit 0; observed output excerpt:
Embedded secret guard passed (no private key material or provider credential literal in the tree).
Log SHA-256: 437a3ceba32094d2901889965ad191e411aad32cdb18bf0b8659e818011c406b

Command:
make script-tests
Exit 0; observed output excerpt:
test_cli_requires_the_repository_default_branch (__main__.SelfTests.test_cli_requires_the_repository_default_branch) ... ok
test_config_rejects_duplicate_and_invalid_entries (__main__.SelfTests.test_config_rejects_duplicate_and_invalid_entries) ... ok
test_freshness_boundaries (__main__.SelfTests.test_freshness_boundaries) ... ok
test_grace_requires_two_successful_queries_with_no_history (__main__.SelfTests.test_grace_requires_two_successful_queries_with_no_history) ... ok
test_invalid_api_evidence_fails_closed (__main__.SelfTests.test_invalid_api_evidence_fails_closed) ... ok
test_report_retains_every_workflow (__main__.SelfTests.test_report_retains_every_workflow) ... ok
test_requests_filter_schedule_default_branch_and_success_on_server (__main__.SelfTests.test_requests_filter_schedule_default_branch_and_success_on_server) ... ok
Ran 10 tests in 0.042s
OK
Ran 30 tests in 20.627s
OK
Ran 10 tests in 2.268s
OK
Ran 33 tests in 0.127s
OK
Ran 7 tests in 0.002s
OK
Ran 7 tests in 0.001s
OK
Ran 4 tests in 0.108s
OK
Ran 3 tests in 0.564s
OK
PASS: cold stampede script self-test
Log SHA-256: 75fda2d069a373c63667b908baaf99e6591aab9099a739b0c4c22ceb6da8d68c

Command:
python3 ./scripts/test_object_generation_protocol_model.py
Exit 0; observed output excerpt:
test_an_unavailable_decision_quorum_cannot_choose_a_candidate (__main__.ObjectGenerationProtocolModel.test_an_unavailable_decision_quorum_cannot_choose_a_candidate) ... ok
test_counter_allocation_requires_persistence_and_is_consumed_once (__main__.ObjectGenerationProtocolModel.test_counter_allocation_requires_persistence_and_is_consumed_once) ... ok
test_counter_restart_burns_persisted_allocations_before_or_after_issue (__main__.ObjectGenerationProtocolModel.test_counter_restart_burns_persisted_allocations_before_or_after_issue) ... ok
test_crash_after_promise_before_reply_does_not_restore_an_older_ballot (__main__.ObjectGenerationProtocolModel.test_crash_after_promise_before_reply_does_not_restore_an_older_ballot) ... ok
test_duplicate_or_unconfigured_voters_cannot_form_a_quorum (__main__.ObjectGenerationProtocolModel.test_duplicate_or_unconfigured_voters_cannot_form_a_quorum) ... ok
test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision (__main__.ObjectGenerationProtocolModel.test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision) ... ok
test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected (__main__.ObjectGenerationProtocolModel.test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected) ... ok
test_forgetting_tombstone_authority_allows_an_aba_create (__main__.ObjectGenerationProtocolModel.test_forgetting_tombstone_authority_allows_an_aba_create) ... ok
test_issuing_before_persistence_reuses_a_ballot_after_restart (__main__.ObjectGenerationProtocolModel.test_issuing_before_persistence_reuses_a_ballot_after_restart) ... ok
test_lost_ack_and_restart_preserve_the_chosen_operation (__main__.ObjectGenerationProtocolModel.test_lost_ack_and_restart_preserve_the_chosen_operation) ... ok
test_minority_accepted_work_is_adopted_and_finished (__main__.ObjectGenerationProtocolModel.test_minority_accepted_work_is_adopted_and_finished) ... ok
test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample (__main__.ObjectGenerationProtocolModel.test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample) ... ok
test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it (__main__.ObjectGenerationProtocolModel.test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it) ... ok
test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor (__main__.ObjectGenerationProtocolModel.test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor) ... ok
test_replacing_voters_with_empty_state_can_choose_a_conflicting_value (__main__.ObjectGenerationProtocolModel.test_replacing_voters_with_empty_state_can_choose_a_conflicting_value) ... ok
test_skipping_predecessor_recovery_chooses_a_broken_revision_chain (__main__.ObjectGenerationProtocolModel.test_skipping_predecessor_recovery_chooses_a_broken_revision_chain) ... ok
test_stale_publication_rollback_and_gc_cannot_change_the_winner (__main__.ObjectGenerationProtocolModel.test_stale_publication_rollback_and_gc_cannot_change_the_winner) ... ok
test_superseded_operation_identity_cannot_be_reused_for_different_input (__main__.ObjectGenerationProtocolModel.test_superseded_operation_identity_cannot_be_reused_for_different_input) ... ok
test_superseded_operation_retry_returns_its_original_successor (__main__.ObjectGenerationProtocolModel.test_superseded_operation_retry_returns_its_original_successor) ... ok
test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision (__main__.ObjectGenerationProtocolModel.test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision) ... ok
test_tombstone_rejects_a_delayed_never_present_create (__main__.ObjectGenerationProtocolModel.test_tombstone_rejects_a_delayed_never_present_create) ... ok
test_unlearned_predecessor_is_recovered_before_the_second_slot (__main__.ObjectGenerationProtocolModel.test_unlearned_predecessor_is_recovered_before_the_second_slot) ... ok
Ran 24 tests in 0.929s
OK
Log SHA-256: 0b4be926efb0609d497ae58930340153f01957b606206cbbdb3e0d47eb29d3a2

Command:
python3 ./scripts/check_test_wiring.py
Exit 0; observed output excerpt:
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired
Log SHA-256: 21f676f458451525535ab0ef73d6a9c0d9774e16d76a4d19e396ad34936dd39c

Command:
./scripts/check_no_planning_docs.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_uring_lane_lib_only.sh
Exit 0; observed output excerpt:
OK: no 'uring_'-matching test functions under crates/ecstore/tests; --lib is safe
Log SHA-256: 43f3f851e27ec2ab9e34d6b05fefdede38af992699c473904bfe6f53a46d52e3

Command:
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast
Exit 0; wall 234.399s.
Observed output:
    Finished `test` profile [unoptimized + debuginfo] target(s) in 3m 49s
    Starting 1 test across 1 binary (5871 tests skipped)
  SETUP PASS [   0.006s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"'
        PASS [   0.440s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item
     Summary [   0.586s] 1 test run: 1 passed, 5871 skipped
Log SHA-256: 34e973f86e8dc2102c1a0e483e0ac802ce7df3d1b40a603932f62c0805c4de27

Command:
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over) | test(=store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence) | test(=store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable)' --no-fail-fast
Exit 0; wall 4.010s.
Observed output:
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.43s
    Starting 3 tests across 1 binary (5869 tests skipped)
  SETUP PASS [   0.005s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"'
        PASS [   0.068s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over
        PASS [   0.637s] rustfs-ecstore store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence
        PASS [   0.846s] rustfs-ecstore store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable
     Summary [   0.987s] 3 tests run: 3 passed, 5869 skipped
Log SHA-256: b5da2a927ff18ca9aa217762f7186a6d4371bd9e8feedd7b77940e10b64dffac

Command:
cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
Exit 0; wall 1084.508s.
Observed output:
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 18m 03s
Log SHA-256: 28f98be7970faaa71cd118da1da4e6e2e3d23b9dda7e6449eaa7a37e741260b5

Compiled mutation command (unchanged for each red and restored run):
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast

M1: reverse only this test-local constructor flag from false to true. The mutant compiled, then failed the original recovered-item queue assertion after 30.481s. Restoring the entire observed candidate passed in 0.460s. Red log 55bf3b1535529ac0bcfb8071f9b56f9f40631007a85c315367e7e93786d65293; restored log 204d5fbe4574a7917f5e740a795d864fe28a7e7fb5e33d27b0a9cf68e2f17f52.

M2: replace only the actual awaited enqueue with false, consuming its original inputs. The mutant compiled, then failed the same original queue assertion after 30.516s. Restoring the entire observed candidate passed in 0.457s. Red log 89186330413215e231e2b06bf013cc31097ea8f4a106d91421a36d6c8986a7fd; restored log e848b42d2302401afa5dbcd9e37bf76d2e56a96ab6de5a488513b997f53c5e9b.

All temporary observation and mutation code was removed before final acceptance. Historical baseline and diagnostic failures remain recorded; they are not mutation credit. The separate Object Lock intent-retry diagnostic still fails, and these native macOS checks do not establish Linux CI recovery.

* refactor(rpc): serve internode rpc bodies with the owned Body (#8441)

* test(rpc): pin the wire bytes of every internode rpc response body

rustfs/backlog#2747 batch 4, step 1 of 2, on the s3s bodies the rpc paths
serve today. Each response body builder in storage/rpc/http_service.rs is
served through hyper's HTTP/1.1 server and the raw bytes the peer reads are
compared with goldens captured from this commit: status, content-length or
chunked framing, chunk boundaries, the terminating chunk, HEAD, and the cut
connection a failure after data produces. The capability responses carry a
per-process proof, so their length header and payload are checked against
the body they encode instead.

The read_file_stream response builder moves into read_file_stream_response
so the test reaches the same code the handler runs; behaviour is unchanged.

* refactor(rpc): serve internode rpc bodies with the owned Body

rustfs/backlog#2747 batch 4, step 2 of 2. The internode rpc HTTP paths
build their responses with rustfs_s3_types::Body: Body::from_stream
replaces Body::from(StreamingBlob::wrap(..)) at the six stream sites, and
the in-memory and empty bodies keep their constructors. InternodeRpcService
converts its responses into the wrapped service's response body through
From<Body>, so the server keeps composing it with the s3s service as
before. The wire-byte goldens from the previous commit pass unchanged;
storage/rpc no longer names s3s.

* fix(rpc): simplify wire test response builder type

Name the boxed response builder used by the capability wire test so
all-target Clippy can check it without a type-complexity warning.

Validation (LANG=en_US.UTF-8, CARGO_INCREMENTAL=0):
- cargo clippy -p rustfs --all-targets -- -D warnings: exit 0.
- cargo nextest run --profile ci -p rustfs --lib -E 'test(/storage::rpc::/)': 279 tests run, 279 passed, 4842 skipped.
- All 19 quick-check commands, including script-tests, both Python scripts and cargo fmt --all --check: exit 0.
- Footprint baseline regenerated: files=194, s3_error_lines=1584, ecstore_files=36; test-count baseline regenerated: 52 crates, 21228 tests, 164 ignored.
- All 54 Cargo.toml files parse without duplicate keys.

No assertion was added or changed by this lint fix. R1-R8 are retained
in the existing PR verification record.

* fix(ecstore): resume fenced Object Lock creation intents

* fix(ecstore): bound creation retry futures on the stack

* fix(ecstore): require native proof for creation retries

* fix(ecstore): bound bucket creation stack usage (#8466)

* fix(ecstore): retain creation fences through rollback

Keep lifecycle, metadata-transaction, and namespace guards around rollback and metadata cleanup. Preserve the original publication error when a guard is lost. Build the timestamp retry fixture through the real interrupted producer so it checks a valid pending creation.

Validated source tree: 4e8e207999.
Two independent source reviews found no blocking regression.

Validation (LANG=en_US.UTF-8):
- bash -c 'shellcheck --version && actionlint' — exit 0
- cargo fmt --all --check — exit 0
- ./scripts/check_unsafe_code_allowances.sh — exit 0
- ./scripts/check_layer_dependencies.sh — exit 0
- ./scripts/check_architecture_migration_rules.sh — exit 0
- ./scripts/check_logging_guardrails.sh — exit 0
- ./scripts/check_error_other_format_ratchet.sh — exit 0
- ./scripts/check_no_tokio_io_uring.sh — exit 0
- ./scripts/check_extension_schema_boundaries.sh — exit 0
- ./scripts/check_body_cache_whitelist.sh — exit 0
- ./scripts/check_s3s_footprint.sh — exit 0
- ./scripts/check_test_count_ratchet.sh — exit 0
- ./scripts/check_fips_wording.sh — exit 0
- ./scripts/check_embedded_secrets.sh — exit 0
- make script-tests — exit 0
- python3 ./scripts/test_object_generation_protocol_model.py — exit 0
- python3 ./scripts/check_test_wiring.py — exit 0
- ./scripts/check_no_planning_docs.sh — exit 0
- ./scripts/check_uring_lane_lib_only.sh — exit 0
- cargo nextest run --profile ci -p rustfs-ecstore --lib --locked --no-fail-fast — 5894 passed, 0 skipped
- cargo check -p rustfs-ecstore --all-targets --locked — exit 0
- cargo nextest run --profile ci -p rustfs --lib --locked --no-fail-fast -E 'test(=admin::handlers::table_catalog::tests::cold_strong_registration_completes_while_other_bucket_backup_waits) | test(=admin::handlers::table_catalog::tests::commit_backend_borrowed_put_preserves_owner_authorization_and_precondition) | test(=admin::handlers::table_catalog::tests::disable_table_bucket_preserves_objects_and_can_be_reenabled) | test(=admin::handlers::table_catalog::tests::load_table_handler_negotiates_vended_credentials_without_breaking_metadata_only_callers) | test(=admin::handlers::table_catalog::tests::namespace_property_handler_updates_object_backed_catalog_and_maps_errors) | test(=app::object::get::tests::execute_get_object_resumes_from_relocated_pool_without_splicing_body) | test(=table_catalog::store::tests::catalog_writer_loss_prevents_recovery_storage_commit)' — 7 passed
- cargo nextest run --profile ci -p rustfs --lib --locked --no-fail-fast -E 'test(unsupported_remote_disk_)' — 2 passed

Additional fault-injection evidence remains incomplete. Timestamp M1 reached its original assertion; its restore run is being collected. M2–M5 and the three planned rollback faults have not run. An automated risk check interrupted the timestamp proof task; no assertions or validation settings were weakened.

* fix(ecstore): box nested bucket rollback state

Keep the guarded rollback future behind a heap allocation so bucket creation does not store its nested state inline. The rollback body, all three fence references and pending-proof revocation order are unchanged.

The same three original Linux tests overflowed on both e611 CI attempts. Local format, ecstore test compilation and all 19 quick-check steps pass; the original Linux feature lanes still need to validate this candidate.

* test(ecstore): isolate rollback guard loss fixtures

Run the three rollback metadata-cleanup cases in separate test processes so reused /proc/self/fd aliases from earlier cases cannot invalidate their creation witnesses. Keep the original fixture, guards, assertions and io_uring test selection, and verify that each child runs the intended case successfully.

Update the generated test-count baseline for the added macro arm.

Validation: cargo check -p rustfs-ecstore --tests; cargo fmt --all --check; all 19 quick-check commands, including make script-tests and the two Python checks. Linux io_uring acceptance and fault controls remain pending.

* refactor(replication): move the replication config onto gateway types (#8443)

* wip: T1.4 replication family

* refactor(replication): move the replication config onto gateway types

The replication configuration family leaves s3s: rustfs-replication no longer depends on s3s, and ECStore bucket metadata, the scanner and the rustfs app/admin layers hold rustfs_gateway_types::persistence::PersistedReplicationConfiguration. The persisted XML goes through the gateway persistence codec behind a new BucketConfigXml trait (the 39 byte goldens pass with unchanged fixtures); the legacy s3s edge converts once in storage/ecfs.rs through storage/s3_compat/replication.rs; the scanner data usage cache keeps its MessagePack bytes through a serde mirror pinned by bytes recorded from s3s 5761ddfe. rustfs-gateway-types 0.32.0 is pinned once (rev 772d3385). Refs rustfs/backlog#2745.

* test: use S3 serializer for list response tests

* fix(ci): restore integration test compilation

Gate the diagnostic test import with its feature, initialize the conditional
MetadataWriteStore fields explicitly, and decode the scanner replication fixture
through its existing cache boundary. Preserve the original fixture and assertions.

Validation was run on tree 544acb80c0 at 6003d4b8.
LANG=en_US.UTF-8; no source changes follow validation.

Quick checks: 19/19 passed.
- shellcheck --version && actionlint: exit 0
- cargo fmt --all --check: exit 0
- ./scripts/check_unsafe_code_allowances.sh: exit 0
- ./scripts/check_layer_dependencies.sh: exit 0
- ./scripts/check_architecture_migration_rules.sh: exit 0
- ./scripts/check_logging_guardrails.sh: exit 0
- ./scripts/check_error_other_format_ratchet.sh: exit 0
- ./scripts/check_no_tokio_io_uring.sh: exit 0
- ./scripts/check_extension_schema_boundaries.sh: exit 0
- ./scripts/check_body_cache_whitelist.sh: exit 0
- ./scripts/check_s3s_footprint.sh: exit 0
- ./scripts/check_test_count_ratchet.sh: exit 0
- ./scripts/check_fips_wording.sh: exit 0
- ./scripts/check_embedded_secrets.sh: exit 0
- make script-tests: exit 0
- python3 ./scripts/test_object_generation_protocol_model.py: exit 0
- python3 ./scripts/check_test_wiring.py: exit 0
- ./scripts/check_no_planning_docs.sh: exit 0
- ./scripts/check_uring_lane_lib_only.sh: exit 0

Darwin validation:
- cargo clippy --locked -p rustfs-ecstore -p rustfs-scanner --all-targets -- -D warnings: exit 0
- cargo nextest run --locked -p rustfs-ecstore -p rustfs-scanner --lib -E 'test(config::com::tests::save_config_preserves_retryable_system_volume_errors) | test(scanner_io::tests::replication_maintenance_refreshes_compacted_stats_across_cycles_and_retries)' --test-threads 1 --no-fail-fast: exit 0, 2/2 passed
- cargo clippy --locked -p rustfs-ecstore -p rustfs-scanner --all-targets --features rustfs-ecstore/diagnostic-read-version-caller,rustfs-scanner/diagnostic-read-version-caller -- -D warnings: exit 101
- cargo nextest run --locked -p rustfs-ecstore -p rustfs-scanner --lib --features rustfs-ecstore/diagnostic-read-version-caller,rustfs-scanner/diagnostic-read-version-caller -E 'test(config::com::tests::save_config_preserves_retryable_system_volume_errors) | test(scanner_io::tests::replication_maintenance_refreshes_compacted_stats_across_cycles_and_retries) | test(cluster::rpc::runtime_sources::tests::) | test(config::com::tests::diagnostic_config_write_retains_origin_and_api_for_results_and_cancellation)' --test-threads 1 --no-fail-fast: exit 0, 14/14 passed

Diagnostic strict Clippy remains red on four unchanged production lints:
runtime_sources.rs if_same_then_else and three redundant_locals reports in
set_disk/core/metadata_read.rs and set_disk/read.rs. Their source ranges and
feature wiring match the base; the base was not compiled separately.

Linux aarch64 validation (Rust/Clippy 1.99.0, UID 501, jobs=1):
- cargo clippy --all-targets -- -D warnings: exit 0; Finished dev profile in 68m 12s
- cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings: exit 0; Finished dev profile in 49m 45s
Source HEAD/tree and clean status matched before and after both commands.
No compiler or stack overrides were used. protoc 35.1 was installed from its
verified ARM64 release; flatc was absent and not invoked by the workspace build scripts.
The local environment differs from hosted CI.

* fix(ecstore): recover delete batches with missing versions

Treat an explicit non-null version as a no-op when it is absent from both recovery sources. Keep rejecting missing null versions, delete-marker mismatches and conflicting local generations.

Add returning-disk and partial-receipt regressions, extend journal boundary checks, and regenerate the ecstore test-count baseline. Production faults exercise the 24 added assertion sites and 7 semantic checks: 33 faults failed at the expected sites and passed after full restoration.

Validation (LANG=en_US.UTF-8, one build job):

cargo nextest run --locked -p rustfs-ecstore --lib -E 'test(delete_recovery) | test(quorum_delete_cleanup_coordinator_binds_single_and_batch) | test(crash_delete_retry_batch_reclaims_inline_and_reserved_external)' --test-threads 1 --no-fail-fast
Summary [  34.580s] 69 tests run: 69 passed, 6058 skipped; exit 0

cargo clippy --locked -p rustfs-ecstore --all-targets -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 5m 37s; exit 0

All 19 final Quick Checks completed with exit 0:

- shellcheck --version && actionlint
- cargo fmt --all --check
- ./scripts/check_unsafe_code_allowances.sh
- ./scripts/check_layer_dependencies.sh
- ./scripts/check_architecture_migration_rules.sh
- ./scripts/check_logging_guardrails.sh
- ./scripts/check_error_other_format_ratchet.sh
- ./scripts/check_no_tokio_io_uring.sh
- ./scripts/check_extension_schema_boundaries.sh
- ./scripts/check_body_cache_whitelist.sh
- ./scripts/check_s3s_footprint.sh
- ./scripts/check_test_count_ratchet.sh
- ./scripts/check_fips_wording.sh
- ./scripts/check_embedded_secrets.sh
- make script-tests
- python3 ./scripts/test_object_generation_protocol_model.py
- python3 ./scripts/check_test_wiring.py
- ./scripts/check_no_planning_docs.sh
- ./scripts/check_uring_lane_lib_only.sh

test-count ratchet OK: 52 crates, 22009 tests, 165 ignored match .config/test-count-baseline.txt
PASS: cold stampede script self-test
model: states=51023, transitions=167520, terminal=732, chosen_candidates=2
multi-slot round schedules: 640
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired

* fix(ecstore): retry tier commit intent lock contention

Retry transient namespace lock timeouts within the existing three-attempt
CAS budget. Reload and verify the original mutation identity after
contention, and preserve the final typed error when attempts are exhausted.
Cover save/read contention, exhausted retries, replacement identities and
CAS reconciliation without changing the existing timeout or lock policy.

Validation:
- cargo nextest run --locked -p rustfs-ecstore --features test-util --lib -E 'test(tier_mutation)' --test-threads 1 --no-fail-fast: 52 passed.
- cargo nextest run --locked -p rustfs-ecstore --features test-util --lib -E 'test(intent_advance_) | test(=services::tier::tier::tests::caller_cancellation_after_durable_prepare_does_not_hide_the_mutation)' --test-threads 1 --no-fail-fast: 5 passed.
- cargo clippy --locked -p rustfs-ecstore --all-targets -- -D warnings: passed.
- cargo clippy --locked -p rustfs-ecstore --all-targets --features test-util -- -D warnings: passed.
- 27 mutation pairs failed at the intended assertions and passed after restoring the production code.
- bash scripts/check_test_count_ratchet.sh --update, then bash scripts/check_test_count_ratchet.sh: 52 crates, 22013 tests, 165 ignored; ecstore increases from 6581 to 6585.

* refactor(lifecycle): replace s3s domain types with Gateway DTOs

Keep lifecycle rules, restore requests, Object Lock, and scanner cached tags in domain types. Convert at the existing HTTP boundary while preserving legacy XML, scanner cache bytes, date ordering, and tombstones.

Linux test validation before the final import-path-only adjustment:

cargo nextest run -p rustfs-ecstore bucket_config_goldens
39 passed; 6211 skipped by selector.

cargo nextest run --lib -p rustfs -E 'test(/test_zero_rule_lifecycle_tombstone_recognition/) | test(/test_local_lifecycle_staleness_axis_selection/) | test(/test_lifecycle_staleness_preserves_legacy_submillisecond_axis/) | test(/test_merge_incoming_lifecycle/) | test(/test_lifecycle_expiry_statement_matrix/) | test(/test_lifecycle_expiry_subset_xml/) | test(/put_bucket_lifecycle_validation_errors_keep_their_s3_code/) | test(/execute_put_bucket_lifecycle_configuration_rejects_missing_configuration/) | test(/execute_restore_object_rejects_missing_restore_request/) | test(/execute_restore_object_rejects_select_type/) | test(/execute_restore_object_maps_failures_to_typed_s3_errors/) | test(/execute_restore_object_returns_internal_error_when_store_uninitialized/) | test(/restore_request_conversion_preserves_legacy_output_values/) | test(/restore_request_conversion_preserves_presence_and_raw_enums/) | test(/restore_request_conversion_distinguishes_absent_and_empty_containers/)'
25 passed; 5318 skipped by selector.

cargo nextest run --profile ci -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-replication -p rustfs-scanner
7872 passed; 21 skipped under the existing CI profile.

Previously verified mutation controls: 39 production faults, 41 RED/GREEN pairs.

cargo clippy --locked -p rustfs -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-scanner --all-targets -- -D warnings
Passed in 40m 45s before the import-path adjustment, then in 1m 59s on
the final source with warnings denied. The final adjustment imports the
existing helper through the object domain facade; fmt and the architecture
check also passed.

Part of rustfs/backlog#2745. The remaining ecstore configuration families follow separately.

* refactor(admin): move error types to rustfs-s3-types (#8440)

* refactor(protocols): move error types to rustfs-s3-types by codemod

T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and
the error items of protocols move off s3s by scripts/codemods, run with
--hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it
downcasts the backend error, whose runtime type is still s3s::S3Error.

- rustfs/src/protocols/client.rs: the two webdav session methods now
  return rustfs_s3_types::S3Result and convert the s3s error with From.
- codemod: a child module that calls s3_error! through `use super::*`
  gets its own `use rustfs_s3_types::s3_error;` once its parent imports
  that macro by name; otherwise it is listed as GLOB.
- s3-types: a table test pins code, status and the s3s error the legacy
  edge renders for every variant a moved call site names.
- s3_error_lines 1585 -> 1584 (a lifecycle comment reworded).

* refactor(admin): move error types to rustfs-s3-types (WIP, no build)

T1.2 batch 4 (rustfs/backlog#2743), work in progress: rustfs check
still fails with 16 errors (listed in the PR body).

- codemod: --facade PATH treats a crate-local re-export of s3s's error
  items like s3s for the four names (9 new self-tests, 9 mutants red).
- codemod output for rustfs/src/admin and rustfs/src/site_replication
  (--facade crate::storage_api::site_replication::s3); 6 glob children
  get an explicit rustfs_s3_types::s3_error import.
- admin/router.rs: the Operation trait returns rustfs_s3_types::S3Result.
- error.rs: From<ApiError> builds the rustfs-s3-types error; the s3s
  conversion is derived from it.

* fix(admin): convert error types at the s3s boundary

Batch 4 of T1.2 left the admin and site_replication modules returning
rustfs_s3_types::S3Error while their callers and callees on the storage
and auth side still speak s3s::S3Error. Convert at each edge with the
existing From impls in rustfs-s3-types (compat_s3s.rs) instead of
changing any code, message or status:

- admin/auth.rs, admin/handlers/object_zip_download.rs,
  admin/handlers/table_catalog/mod.rs: map the s3s result returned by
  check_key_valid / authorize_request / authorize_internal_object_request
  into the admin error type with .map_err(Into::into).
- storage/ecfs.rs: the s3s-facing side maps the admin site_replication
  gate results the other way.
- admin/handlers/pools.rs: ApiError.code stays s3s::S3ErrorCode, so the
  rustfs code is converted with .into() when the struct is built, in
  production and in the two tests that build or compare it.
- admin/handlers/heal.rs, admin/runtime_sources.rs: the router returns
  s3s errors (S3Route impl); the expected rustfs code is converted with
  .into() before the comparison.
- admin/handlers/table_catalog/mod.rs: iceberg_rest_error takes
  &'static str so the Cow<'static, str> custom code borrows it; every
  caller already passes an ICEBERG_ERROR_* constant or a &'static str.
- admin/handlers/quota.rs: drop a now same-type .into() after s3_error!.
- storage_api.rs, site_replication/mod.rs: drop re-exports and an import
  that no longer have users (clippy -D warnings).

cargo check -p rustfs --all-targets and cargo clippy -p rustfs
--all-targets -- -D warnings are clean; the s3s footprint baseline is
unchanged at files=172, s3_error_lines=303, ecstore_files=28.

Refs rustfs/backlog#2743, rustfs/backlog#2734.

* refactor(ecstore): move bucket configs onto gateway persisted types (#8709)

* refactor(ecstore): move bucket configs onto gateway persisted types

Finish the ECStore batch of T1.4 (rustfs/backlog#2745). The remaining
eleven bucket configuration families (Website, CORS, Tagging, Versioning,
ObjectLock, Encryption, Notification, Logging, Accelerate, RequestPayment,
PublicAccessBlock) now read and write their stored XML through the
rustfs-gateway-types persistence and cors_tagging codecs instead of the
s3s codec, so the legacy s3s bucket-config codec and its
s3gate-metadata-xml compat marker are removed.

- ecstore: BucketConfigXml is implemented by the gateway codec for every
  family; metadata, metadata_sys, versioning, object lock, replication and
  lifecycle consumers take the persisted types. Region moves to
  rustfs_s3_types::Region, and the ETag conversion to s3s moves to the
  rustfs edge with its tests. Only warm_backend.rs and warm_backend_s3.rs
  still reference s3s, held for T1.5 (rustfs/backlog#2746).
- scanner: object lock cache, item actions and data usage use the
  persisted types.
- rustfs: the legacy S3 edge converts each family between s3s DTOs and the
  persisted types in storage/storage_api.rs by writing one codec's XML and
  reading it with the other; Tagging converts field by field.
- Bump the rustfs-gateway-types pin to 21283e87.

The 39 persisted-byte goldens are unchanged and pass, so a document the
old codec wrote reads back and re-encodes to the same bytes. s3s footprint:
files 181 -> 146, ecstore_files 28 -> 2, s3_error_lines unchanged.

* test(app): prepare the PutBucketEncryption config from the request

The uninitialized-store test passed an empty persisted configuration, which the use case now validates before it reaches the store check, so it answered MalformedXML instead of InternalError. Convert the request's configuration the way the S3 edge does.

* style(app): format the PutBucketEncryption test

* refactor(s3-client): move remote error codes and DTO enums off s3s (#8783)

Part of T1.5 (rustfs/backlog#2746), batch 1.

* refactor(s3select): run queries on gateway DTO request shapes (#8794)

* refactor(s3select): run queries on an engine request built from gateway DTO shapes

Part of T1.5 (rustfs/backlog#2746), batch 2.

* fix(s3select): keep the select request conversion in the app layer

The layer guard forbids app -> storage; the conversion moves from storage/s3_compat/select.rs to app/select_request.rs unchanged.

* fix(heal): retry transient metadata conflicts (#8812)

* fix(heal): retry transient metadata conflicts

* chore: update heal test count baseline

* test: reserve nextest capacity for progress probes (#8818)

* test(e2e): use a non-default region for the differential negative control

Since main's region-aware CreateBucket and GetBucketLocation (#8739, #8743), us-east-1 answers an empty LocationConstraint and 200 for an owned re-create, so the control no longer reported its own region. Also refresh the e2e-full linux selection from the CI listing (682 tests).

* refactor(protocols): move StorageBackend to gateway types (#8825)

* refactor(protocols): move storage backend to gateway types

* test: update e2e test count baseline

* fix: preserve copy source keys and unknown lengths

* test: update protocols test count baseline

* fix(protocols): preserve typed SFTP backend errors

* test(protocols): use gateway list input in policy tests

* fix(protocols): clear clippy failures after T1.5 merge (#8831)

fix(protocols): clear clippy failures after merge

* fix(ecstore): finalize restore metadata after local copy-back (#8836)

* fix(ecstore): finalize restore metadata after local copy-back

* test(ecstore): ratchet added restore finalization cases

* fix(ecstore): keep restore copyback compatible with older peers

* fix(ecstore): gate restore copyback on fleet compatibility

* fix(ecstore): retain transition identity during restore copyback

* fix(ecstore): route committed restore copyback locally

* fix restore copyback cleanup

* fix(ecstore): honor restore probe test fallback

* refactor(storage): migrate option errors to domain types (#8839)

* refactor(s3): migrate private multipart validation errors (#8844)

* refactor(auth): derive policy conditions from AuthzFacts (#8442)

* refactor(auth): extract authorization facts

* test(e2e): update smoke selection baseline

* test(e2e): update full selection baseline

* refactor(admin): migrate health and prefix usage to owned requests (#8887)

* refactor(admin): add owned health router boundary

* refactor(admin): migrate prefix usage to owned request

* refactor(admin): complete the owned admin request boundary (#8949)

* refactor(admin): complete the owned admin request boundary

Every type implementing the owned admin `Operation` now serves the legacy
router through one blanket impl that converts the s3s request and response
at that single edge, so handlers register with `AdminOperation(&Handler {})`
and need no adapter statics; `LegacyAdminAdapter` is removed.

`AdminRequest` gains the accessors handlers read today (`remote_addr`,
`client_info`, `request_context`, `req_info`, `req_info_mut`,
`install_req_info`, `envelope_mut`, `take_body`), the request helpers gain
envelope forms (`authenticate_envelope_credentials`,
`object_store_from_envelope`, the table-catalog `*_from_envelope` helpers,
`KmsAdminAudit::from_envelope`, `AccountAuditContext::from_envelope`,
`CallerIdentity::resolve_envelope`), `read_compatible_admin_body` accepts
either body type, and the owned `Body` gains `store_all_limited` and
`Default`. `admin::operation::test_support::TestRequest` builds owned
requests for handler tests. Unused accessors carry a per-method dead-code
expectation that each migrating handler batch removes.

No handler behavior changes; the remaining migrations follow in batches.

* chore(admin): add the T1.12 admin handler codemod

The script rewrites an admin handler file from the s3s request and response
types to the owned boundary (imports, signatures, request field and helper
calls, body access, response construction, and test request literals) and
leaves a TODO comment at every site it cannot rewrite mechanically. It is
idempotent and is removed once T1.12 completes.

* refactor(admin): migrate replication and KMS handlers to owned requests (#8951)

Move the T1.12 batch 2 admin handlers (site_replication, kms_keys, event,
replication, diagnostics, on_demand_migration, audit, account, kms_rekey,
kms_key_lifecycle, object_data_cache, tls_debug, integrity) from the s3s
request/response types to the owned AdminRequest/AdminResponse boundary.
Handler logic, authorization order, error codes and response bytes are
unchanged; the request/response plumbing now goes through the envelope
accessors and helpers introduced by the owned boundary.

The shared response helpers (utils::json_response, utils::empty_response,
handlers::admin_json_response, target_descriptor::build_json_response,
profile::authorize_profile_request) and the site replication repair
executor still answer with the s3s types, so the migrated files carry
owned-boundary forms of those helpers and convert the repair response at
its call edge. The streaming console-log body and the two devnull drains
move from the s3s byte stream to the owned Body stream/frame API.

Drop the dead_code expectations for the owned accessors and envelope
helpers this batch starts using, and lower the s3s footprint baseline to
the measured 99 files.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* refactor(admin): add owned forms of the shared admin response helpers

Handler batches were copying `json_response`, `empty_response`,
`admin_json_response`, and `build_json_response` into each migrated file
because the shared helpers still return the s3s response. Provide owned
forms next to each legacy helper (`owned_json_response`,
`owned_empty_response`, `owned_admin_json_response`,
`owned_build_json_response`, `authorize_profile_envelope`) with parity
tests against the legacy shape, derive `Debug` for `AdminResponse` so
tests can use `expect_err`, and fix three codemod rules: strip owned
string conversions in test credentials, annotate non-literal URI parses,
and leave a rebound request parameter immutable.

* refactor(admin): use the shared owned response helpers in batch 2 (#8954)

Replace the per-file owned-boundary copies of json_response,
empty_response, admin_json_response, build_json_response and
authorize_profile_request in the batch 2 admin handlers with the shared
owned_* helpers and authorize_profile_envelope that the boundary branch
now provides, and restore expect_err in the handler tests now that
AdminResponse implements Debug; the `.err().expect(..)` stand-ins were
failing the base branch's clippy step (clippy::err_expect under
-D warnings). The site replication repair call edge
keeps its conversion because the repair executor still answers with the
s3s response type.

Drop the dead_code expectations on the shared owned helpers this batch
now uses in non-test code.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* fix(admin): preserve legacy body limit error responses

* refactor(admin): move policy, ILM and batch 3 handlers to owned requests (#8956)

Batch 3 of T1.12 (rustfs/backlog#2788, parent rustfs/backlog#2734) moves
14 admin handler files to AdminRequest/AdminResponse and the owned
Operation trait: ilm_transition, policies, idp_compat, config_admin,
plugins_instances, profile_admin, group, heal, object_zip_download,
kms_key_metadata, durability, inspect_archive, sts and
gateway_key_inventory. Handler logic, authorization order, error codes
and response bytes are unchanged; test request literals use
test_support::TestRequest.

The handlers use the shared owned helpers (owned_json_response,
owned_admin_json_response, owned_build_json_response,
authorize_profile_envelope). The dead_code expectations on the
AdminRequest accessors and RequestEnvelope::uri_mut they now exercise
are removed. The legacy target_descriptor::build_json_response keeps
only its equivalence test caller and carries the matching expectation
until it retires.

object_zip_download builds the s3s request the S3 gate still reads from
the owned request facts; heal authenticates through
authenticate_envelope_credentials; the trace, inspect-archive and zip
bodies use Body::from_stream. extract_plugin_instance_filters takes the
URI so extensions.rs keeps calling it, and the two idp_compat
delegations stay on the legacy trait until their delegates migrate.

The s3s footprint baseline drops from 99 to 90 files.

* test(admin): anchor the replication diff body contract on the call

The source-contract test looked for `let body = read_compatible_admin_body(`
on one line; rustfmt wrapped that binding after the handler moved to the
owned request boundary, so the marker was no longer found. Anchor on the
call itself so formatting cannot change what the contract checks.

* refactor(admin): migrate table catalog and batch 1 to owned requests (#8955)

* fix(admin): preserve catalog body limit response bytes

* refactor(admin): migrate identity and cluster handlers to owned requests (#8966)

* refactor(admin): migrate identity and cluster handlers to owned requests

Move the T1.12 batch 4 admin handlers (user, service_account, oidc, tier,
system, pools, quota, kms_dynamic, kms_backup, rebalance, module_switch,
bucket_meta, cluster_snapshot, realtime, bucket_recovery, profile) from the
s3s request/response types to the owned AdminRequest/AdminResponse
boundary. All 90 impl Operation blocks now take AdminRequest and return
AdminResponse; handler logic, authorization order, error codes and response
bytes are unchanged. The handlers use the shared owned helpers
(owned_json_response, owned_admin_json_response, authorize_profile_envelope,
the envelope-form runtime sources and audit constructors) instead of local
copies.

profile.rs drops the s3s forms of authorize_profile_request and
profile_not_implemented_response now that profile_admin.rs is on the
envelope gate too; profile_not_implemented_response answers with an
AdminResponse. The realtime metrics stream moves from the s3s ByteStream
adapter to Body::from_stream, and the rebalance idle-404 document still
renders through s3s::S3Error::to_http_response with its body converted at
the edge.

The idp_compat delegators ImportIamV2 and AddServiceAccountLdap move to the
owned trait as well: with their targets on both traits, the method-call
delegation became ambiguous between the owned impl and the router blanket
impl, so they now take the owned request directly.

KmsAdminAudit::from_request loses its last production caller with this batch
and is deleted; its comparison test now asserts the envelope facts directly.

Lower the s3s footprint baseline to the measured 68 files.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* refactor(admin): remove the legacy admin helpers batch 4 superseded

Batch 4 moved the last production callers of the s3s-typed admin helpers
to their owned forms, which left `admin_json_response`, `json_response`,
`authenticate_request`, `authorize_admin_request`, `object_store_from_req`
and `object_store_from_extensions` unused, along with the `Body`,
`S3Request` and `S3Response` re-exports of the admin `s3` facade. Remove
them. The owned forms keep their documentation; the behavior tests of the
removed `json_response` now exercise `owned_json_response`, and the
compat-prefix test asserts the owned response directly.

Two tests existed only to compare a removed legacy helper with its owned
twin (`owned_json_response_matches_the_legacy_shape`) or to cover the
removed legacy gate (`authorize_admin_request_without_credentials_is_rejected`);
they go with the helpers, and the owned equivalents stay. The rustfs
test-count row drops from 5957 to 5955 and the s3s footprint from 68 to 66
files.

* refactor(admin): drop the legacy target descriptor JSON response helper (#8968)

`target_descriptor::build_json_response` returned the s3s response and
had no production caller left once the event, audit and plugin instance
handlers moved to `owned_build_json_response`; it survived only for the
legacy/owned equivalence test. Remove it, give the owned helper its own
documentation, and make the test assert the owned response directly:
status, `application/json`, the echoed `x-request-id` when present and
only then, and the body bytes. The file no longer imports s3s, so the
footprint baseline drops from 66 to 65 files.

* test(ecstore): pin native history on cancellation under the cancellable queue

main moved the local write path onto the disk blocking queue, which skips an
operation once its waiter is dropped, so a borrowed write cancelled while it is
queued no longer creates its file. The integration-only native publication
test still expected the queued open to run. Assert the new execution model
instead (the open never runs) while keeping its safety claims: the Started
record is taken before queueing and the cancelled publication never regrants
native history.

* chore(gateway): raise test-count rows for tests merged from main

* test(s3select): use the owned JSON input types in merged planner tests

* fix(ecstore): scope the merged direct-writer test hook to test builds

* fix(ecstore): create parents before admitted truncate opens

Align the heal tests with removal-disabled preservation and local-only outcome
fields, and refresh the e2e-full selection for the synced tests.

* chore(gateway): update test-count baseline after main sync

Main #9241 removed the node pool-metadata readiness cache with 8 tests
and added 6, so the rustfs row drops from 6440 to 6438. The ecstore and
s3select rows rise for tests added by #9237, #9239 and #9241.

* feat(scripts): add junit_diff.py for dual-stack JUnit comparison (#9275)

* build(deps): re-pin rustfs/gateway to 371d91b2

* fix(protocols): pass copy sources as CopySourceValue

* ci(gateway): make the test-count ratchet a floor, not exact (#9278)

ci(gateway): make the test-count ratchet a floor instead of an exact baseline

Tests below a row still fail; tests above it, a member without a row and ignores below the row now pass with a note. Ignores above a row still fail. Adds contract tests wired into make script-tests.

* test(protocols): raise rustfs test-count baseline for copy source tests

* fix(protocols): keep refusing listings without object ETags

* chore(gateway): record ignored microbenchmarks merged from main (#9290)

chore(gateway): record the ignored microbenchmarks merged from main

main added three throughput microbenchmarks that are ignored by design
(#9217): one each in rustfs-checksums, rustfs-rio and rustfs-utils. The
test-count ratchet treats a higher ignored count as a failure until the
row is raised, so raise those three rows to 1 and tighten the test rows
to the merged tree with --update.

* ci(s3-tests): add reference parity lane against pinned 1.0.5 (#9288)

* build(deps): re-pin rustfs/gateway to f9aa6045

* build(deps): re-pin rustfs/gateway to aa1d1a29

---------

Co-authored-by: rustfs-cla-assistant[bot] <273121464+rustfs-cla-assistant[bot]@users.noreply.github.com>
2026-11-11 12:16:15 -05:00
唐小鸭 20c046af6c fix(targets): resolve default queue dirs per node (#9262)
The built-in notify and audit queue directories /opt/rustfs/events and
/opt/rustfs/audit are persisted cluster-wide as literal defaults. On
native Windows they have no drive prefix, so every target that kept the
default failed validation as a non-absolute path.

Treat those two literals as logical defaults and resolve them on each
node when a queue directory is validated or opened. A new
RUSTFS_TARGET_STORE_DIR relocates both default queues on any platform;
without it, Windows uses %ProgramData%\RustFS. Explicit operator paths,
the persisted configuration, and Linux/macOS behavior are unchanged, and
pending entries are not migrated when the root changes.

Expose the variable through the Helm chart as
config.rustfs.target_store_dir and document the resolution rules.
2026-11-11 11:01:00 -05:00
cxymds 1f823d4ee8 fix(lock): preserve batch ownership and bound peer waits (#9266)
* chore(observability): classify bulk delete failure stages

* fix(lock): retain cleanup ownership for batch acquisition

* fix(lock): bound batch waits and retry confirmed rejections
2026-11-11 10:54:38 -05:00
Chris An 0622419f65 fix(obs): bound the log directory with default settings (#9327)
* fix(obs): bound the log directory with default settings

With the defaults the active log file had no size limit (size rolling was off), and the cleaner counted only rolled files older than the minimum file age against the byte budget, on a 30-minute interval. A log storm could therefore fill the disk for up to two hours without the budget applying.

The active file now rolls at 256 MiB when the single-file limit is unset; an explicit 0 still turns size rolling off, and the cleaner's per-file trim keeps its previous default. The appender wakes the cleaner after every roll, and once the rolled files exceed the byte budget the cleaner processes the oldest ones regardless of age. The README no longer calls the budget a hard cap on the directory and states the actual bound.

* fix(obs): trim only what the log byte budget needs

Lifting the minimum file age for an over-budget pass merged fresh files into the regular selection, so the keep_files ceiling and the per-file limit applied to them too: one byte over the budget could process every fresh file but keep_files of them. The override is now a byte trim on the files that would remain, oldest first, until the budget holds.

An unset single-file limit now rolls at a quarter of a small byte budget instead of 256 MiB, so a freshly rolled file is not larger than the whole budget. Both logging setups open the appender through one function that returns the roll signal the cleanup task listens on. The README states that a log directory must not be shared by processes whose file names match the same pattern.
2026-11-11 10:47:23 -05:00
GatewayJ ac8d81319b fix(s3-tables): reclaim dropped table maintenance configs (#9331) 2026-11-11 10:43:57 -05:00
Chris An 1b3c477f05 fix(ecstore): bound bitrot and remote disk failure logs (#9328)
* fix(ecstore): bound bitrot and remote disk failure logs

A failing drive logs one error per shard block of every read it serves, and an unreachable peer logs one warning per in-flight operation. Neither had a rate limit, so a degraded drive or peer could produce log volume at the speed of the failing I/O.

The three bitrot read failures (short shard, hash mismatch, read error) and the two remote disk failures (timeout, network-like error) now log at most once per five seconds and report how many events were suppressed. The remote disk throttles belong to each disk handle, so one noisy peer cannot hide another peer's first warning. Error results, health marking and the per-operation counters are unchanged.

* fix(ecstore): bound reader retirement warnings in the decoder

A slow shard reader is retired once per stripe of every read it lags on, and a parity reader that cannot be aligned is retired with a warning each time. Both lines name only a shard index. They now log at most once per five seconds with the number suppressed; the retirement itself and the error recorded for the shard are unchanged.
2026-11-11 10:42:51 -05:00
唐小鸭 ea286defc6 fix(windows): release rebuild source pins and align test fixtures (#9263)
* fix(windows): release rebuild source pins and align test fixtures

Release the multipart source directory pin as soon as the upload head is
decoded during online index rebuild. On Windows the pin denies delete
sharing, so keeping it while indexing made a concurrent abort fail with a
sharing violation.

Align three test fixtures with Windows path and handle semantics:
- heal bucket pass targets use the endpoint string form, matching
  production disk endpoint identity;
- scanner erasure probe injection keys are normalized with path_join_buf
  so they match the scanner's lookup keys;
- the targets queue budget test changes mtime through a writable handle,
  which Windows requires.

* test(scanner): look up failed objects by the scanner's key form

The scanner records failed objects under keys built with path_join_buf,
which uses `/` separators on every platform. Three erasure probe tests
looked them up with a native `root.join(..)` path, so on Windows the
expected key never matched and the assertions failed. Build the expected
key the same way the scanner does; the assertions still require the exact
key to be present.
2026-11-11 10:39:00 -05:00
唐小鸭 7cfa5c3d28 fix(ecstore): name blocked migration records; port fixtures to Windows (#9261)
* test: bypass the disk independence check in path-only topology tests

These tests parse endpoints whose local paths are never created. Unix
leaves missing paths out of the physical-disk independence check, but
Windows resolves them to their volume, so every disk counted as one
shared device and the parsing assertions never ran. The check keeps its
own coverage in the dedicated shared-volume tests.

* fix(ecstore): name the target record when a migration write fails

The existence probe reports a blocked record with its path, but the
commit path returns bare disk errors. On Windows a regular file in the
record's place reads as missing, so migration only fails at the write,
and startup reported "File access denied" without naming the record.

* test: make app fixtures portable to native Windows

- Token- and cache-shaped keys use Windows-legal spellings at the same
  sort position, so pagination still runs without relaxing key admission.
- The TLS root test falls back to a directory junction only when a
  symlink needs a privilege the runner lacks, and asserts the root is a link.
- Snapshot timestamps step by 1 us, which Windows SystemTime can represent.
- The receiver disk URL uses a slash-led path instead of a raw drive path.
2026-11-11 10:38:01 -05:00
Chris An c44eebfe1c test(kms): use absolute fixture paths on Windows (#9274) 2026-11-11 10:35:54 -05:00
Chris An d62d63cdea test(sftp): match structured Windows host-key warning (#9273) 2026-11-11 10:35:00 -05:00
Chris An e5691e7d1a refactor(auth): share authorization core with owned entry points (#9281)
* feat(server): add the RUSTFS_S3_STACK selector and dual-stack e2e lane (#8411)

* refactor(server): group the s3s-compat tower layers under legacy_compat (#8410)

Move the ten tower layers that patch s3s behaviour out of
rustfs/src/server/layer.rs into rustfs/src/server/legacy_compat/, one file
per layer with its tests. Each file header names the s3s behaviour it
patches, the gateway mechanism that replaces it (Replaced by:), and the task
that removes it (Removed by: T2.12, rustfs/backlog#2771). The layers keep
their code, their order in external_service_stack!, and their behaviour; no
layer is deleted.

Add server::http::tests::stack_census, which renders the ordered service
chain of the real external_service_stack! expansion and compares it with a
committed constant, so a reorder, insertion or removal is a visible diff.

Route the two s3s error types the moved layers use through a
storage_api::server::legacy_compat shim so the split adds no s3s-importing
file, and lower the s3s footprint baselines to the measured values (files 209
to 208, s3_error! lines 1586 to 1585). Keep the server runtime-source bypass
guard covering the moved code.

Refs rustfs/backlog#2787, rustfs/backlog#2734

* feat(s3-types): add the RustFS-owned S3 error type and s3_error! macro (#8406)

`rustfs-s3-types` now provides `S3ErrorCode` (generated by build.rs from
src/codes.txt, the grep-derived call surface, plus `Custom`), `S3Error`,
`S3Result`, `StdError` and the `s3_error` macro with the same call syntax
as s3s, so the T1.2 codemod is a `use` swap. Statuses are hand-written and
pinned to the s3s behaviour RustFS ships today by an oracle test that uses
s3s as a dev-dependency only. The `From` conversions for the legacy edge
live behind the off-by-default `compat-s3s` feature (the orphan rule
forbids them in rustfs/src/error.rs); `rustfs` enables it.

Refs rustfs/backlog#2742, rustfs/backlog#2734

* ci: add the per-crate test-count ratchet (#8404)

Quick Checks now fails any PR that lowers the number of test attributes
in a workspace crate or adds #[ignore], measured per crate against the
exact rows of .config/test-count-baseline.txt (rustfs/backlog#2734
section 1 rule 3). The counter widens the spec's literal
#[(tokio::)?test] to the tokio attributes that carry arguments and both
spellings of #[test_case(...)], and anchors at line start so
commented-out attributes do not count. The initial baseline covers 52
crates, 21077 tests and 164 ignored.

Refs rustfs/backlog#2738
Refs rustfs/backlog#2734

* test(ecstore): pin the persisted bucket-config XML bytes for 13 families (#8408)

Before any DTO or codec is replaced, pin the exact bytes that
deserialize followed by serialize produces for every persisted
bucket-configuration XML family in its three source shapes (AWS
documentation, MinIO-written, RustFS-written), and the bytes RustFS
writes today for a typed value. Byte equality only: a structural
comparison would let a codec replacement change the persisted encoding
unnoticed.

The generator script refuses to overwrite existing goldens without
--force, refuses on dirty codec paths, cannot report success from its
write mode, and records the generating commit in the fixtures README.

Refs rustfs/backlog#2744, rustfs/backlog#2734

* ci: raise the test-count baseline to the integration tree (#8426)

* ci: make the s3s footprint baseline exact and lower-only (#8403)

* ci: make the s3s footprint baseline exact and lower-only

The s3s footprint guard compared three counters against ceilings held in
the script itself, so a ceiling could sit above the tree and a drift was
fixable by raising it (rustfs/rustfs#5964 raised the s3_error! ceiling
to go green). The three values now live in
.config/s3s-footprint-baseline.txt as files, s3_error_lines and
ecstore_files, the script reads only that file, and a measurement that
differs from its baseline in either direction fails: growth prints
"s3s footprint grew: <measured> > <baseline>", shrinkage prints
"lower the baseline to <measured> in this PR". The counting method is
unchanged; the baseline is the current measurement of
gateway/integration (209 / 1585 / 36). The header records the end state:
once all three reach 0, T4.2 switches the script to failing on any
occurrence.

Refs rustfs/backlog#2739, rustfs/backlog#2734

* ci: add the s3s edge allowlist mode to the footprint guard (#8405)

`scripts/check_s3s_footprint.sh --mode allowlist` fails on every Rust
file that references s3s (`s3s::` or `s3s_sigv4`) and is not matched by
an entry of the new `.config/s3s-edge-allowlist.txt`; `--dry-run` lists
those files and exits 0. The allowlist carries the initial edge from the
spec, and the script rejects an entry that names or matches anything
under crates/ other than crates/e2e_test. The baseline mode and the
argument-less CI invocation are unchanged.

Refs rustfs/backlog#2741, rustfs/backlog#2734

* test(ecstore): reach the bucket-config codec through the test facade (#8428)

* refactor(s3-types): own the HTTP Body and drop s3s from three crates (#8431)

Add rustfs_s3_types::Body, a BoxBody<Bytes, StdError> wrapper with
empty() and From<Bytes | Vec<u8> | String | &'static str>, and the
legacy From conversions to and from s3s::Body behind compat-s3s.

The signer is now generic over the request body: it only reads and
writes the request head, so it needs no body type and its callers keep
theirs. s3-client sends its requests with the owned Body, madmin signs a
bodiless head, and targets drops its two S3Request/S3Response header
wrappers in favour of extract_params_header.

rustfs-signer, rustfs-madmin and rustfs-targets no longer depend on
s3s; the s3s footprint baseline drops from 209 to 202 files.

Refs rustfs/backlog#2747, rustfs/backlog#2734

* refactor(ecstore): move S3ErrorCode to rustfs-s3-types by codemod (#8432)

* test(ecstore): pin the InvalidRange responses of new_getobjectreader

Both error branches of new_getobjectreader build an ErrorResponse with S3ErrorCode::InvalidRange, and no test observed either code. Pin the code, its wire name and the location fields of each branch before the code type moves to rustfs-s3-types (rustfs/backlog#2743), so a changed code on either branch fails a test. The ecstore test-count row rises by the two tests.

* chore(scripts): add the s3s-to-rustfs-s3-types error codemod

Task T1.2 of rustfs/backlog#2734 (rustfs/backlog#2743) moves S3Error, S3ErrorCode, S3Result and s3_error! from s3s to rustfs-s3-types by script, one crate closure per PR. The codemod rewrites use trees (splitting mixed s3s imports) and qualified code paths, never comments or literals, adds the rustfs-s3-types dependency, drops s3s only from a crate that no longer names it, refuses every file of the s3s edge allowlist, takes --hold for a file whose type is fixed outside the closure, and prints a per-crate census. A second run changes nothing. Its self-test runs under make script-tests.

* refactor(ecstore): take S3ErrorCode from rustfs-s3-types

Output of scripts/codemods/s3_error_to_s3_types.py over crates/ecstore, crates/lifecycle, crates/scanner and crates/replication (rustfs/backlog#2743 batches 1 and 2). The ErrorResponse code field, the two InvalidRange constructions in new_getobjectreader and an unused tier import now use rustfs_s3_types::S3ErrorCode; the variant names and wire names are unchanged. lifecycle, scanner and replication reference none of the four error names. warm_backend.rs and warm_backend_s3.rs are held: they compare against rustfs_s3_client::ErrorResponse.code, which stays the s3s type until s3-client moves.

* ci(footprint): count only s3_error! lines still on s3s's macro

s3_error_lines counted every line holding the text `s3_error!`, which
also matches the RustFS-owned `rustfs_s3_types::s3_error!`. A codemod
that moves call sites onto rustfs-s3-types therefore could not lower it
(rustfs/backlog#2743, decision 3; rustfs/rustfs#8432, gotcha 2).

The counter now applies a textual, per-file and per-line rule that errs
on the side of counting (full rule in the script header):
- occurrences spelled `rustfs_s3_types::s3_error!` never count;
- a line holding `s3s::s3_error!` always counts;
- any other `s3_error!` line counts unless its file is cleared, that is,
  it imports the macro from rustfs_s3_types by name on a line starting
  with the use item (or defines it with `macro_rules! s3_error`) and does
  not also bring s3s's macro in (`use s3s::s3_error`, a `use s3s::{..}`
  tree naming it or a crate-root `*`, `use s3s::*`, or
  `#[macro_use] extern crate s3s`);
- fail closed: a file matching neither form (`use super::*`, a
  crate-local re-export, a renamed or glob rustfs_s3_types import, a
  commented-out import, a comment-only mention) counts.

Census of the 1585 lines on 96fc06776: 1478 in 64 files that import the
macro from s3s; 107 in 15 files matching neither form (105 through
`use super::*`, 1 qualified `s3s::s3_error!`, 1 comment line); none in
files that import it from rustfs_s3_types or define it. Every line stays
counted, so the baseline remains s3_error_lines=1585 (files=202 and
ecstore_files=36 unchanged). Rewriting rustfs/src/auth.rs's import to
rustfs_s3_types lowers the new counter to 1567; the old one stayed 1585.

`--self-test` (wired into `make script-tests`) copies the script into
22 fixture trees: 6 positive cases and 16 negative ones (lines that must
not count, forms that must not clear a file, and a missing baseline,
missing key or failing classification search that must fail). Each tree
has a control line that must count, and each run gets s3s text on stdin.
Seven cases were red against the old counter. 19 mutants each turn their
case red: old definition, flipped import test, dropped fail-closed
branch, dropped e2e_test exclusion, dropped s3s veto, dropped qualified
s3s rule, dropped rustfs-qualified strip, unanchored rustfs import,
renamed import accepted, nested glob taken as root glob, single-line
s3s trees only, dropped s3s glob forms, dropped macro_use form, dropped
definition clearing, ignored classification rg errors, error-line search
without its path, classification searches without their path, missing
baseline skipped, missing key skipped.

* test(e2e): compare two RUSTFS_S3_STACK servers over every S3 operation (#8435)

Add dual_stack_differential to e2e_test. It starts two rustfs processes
from the same binary, stack A on RUSTFS_S3_STACK=legacy and stack B on
RUSTFS_E2E_STACK_B (default legacy until rustfs/backlog#2734 T2.5),
sends both the same aws-sdk-s3 script covering the 73 methods of
impl S3 for FS (a normal request and a 4xx negative each), and compares
status, error code, an allowlist of response headers and a few typed
body fields. Error message text is never compared. Each run writes
<target>/dual-stack/<run>.json with compared_ops and the (op, field,
a, b) differences; any difference fails the test.

differential_detects_an_injected_difference starts B with
RUSTFS_REGION=eu-west-1 and requires GetBucketLocation to be reported.
The comparator is pure and unit-tested without a server.

Wire the module into the e2e-smoke filter and update the e2e-smoke and
e2e-full selection digests and the e2e_test test-count row.

Refs rustfs/backlog#2740

* feat(app): add the RustFS-owned RequestEnvelope and its s3s constructor (#8437)

* feat(app): add the RustFS-owned RequestEnvelope and its s3s constructor

* ci: classify the s3s edge adapters in storage/s3_compat as interface

* feat(s3-types): add Body::from_stream and move Swift off s3s::Body (#8438)

rustfs/backlog#2747 batch 3. `Body::from_stream` wraps a chunk stream of
unknown length over `StreamBody`/`BoxBody`: one data frame per chunk, the
first error ends the body, the size hint stays unknown, and dropping the
body drops the stream. Parity tests pin it to the s3s `StreamingBlob::wrap`
it replaces, at the `http_body` level and byte for byte through hyper.

The Swift handlers in rustfs-protocols build their responses as the owned
`Body`. `SwiftService` converts them into the wrapped S3 service's response
body type through `From<Body>`, so rustfs keeps serving `s3s::Body` at the
legacy edge unchanged. Request bodies stay Send-only (`UnsyncBoxBody`); the
two storage writers and the DLO/SLO segment streams get `Sync` from
`sync_wrapper::SyncStream`, which is what s3s's `http_body_unsync` mutex
provided before. The DTO and error uses of s3s in protocols are untouched
(T1.5 and T1.2).

* refactor(protocols): move error types to rustfs-s3-types by codemod (#8439)

T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and
the error items of protocols move off s3s by scripts/codemods, run with
--hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it
downcasts the backend error, whose runtime type is still s3s::S3Error.

- rustfs/src/protocols/client.rs: the two webdav session methods now
  return rustfs_s3_types::S3Result and convert the s3s error with From.
- codemod: a child module that calls s3_error! through `use super::*`
  gets its own `use rustfs_s3_types::s3_error;` once its parent imports
  that macro by name; otherwise it is listed as GLOB.
- s3-types: a table test pins code, status and the s3s error the legacy
  edge renders for every variant a moved call site names.
- s3_error_lines 1585 -> 1584 (a lifecycle comment reworded).

* test(connect): restore one-second drive request budgets

Apply the exact test-only repair already landed on main in 5972a09cc3. The filesystem probe and production CLI success test request one second and keep their existing one-second duration and parameter checks. All outcome, I/O, error, cleanup and resource-limit assertions remain.

Scope: rustfs/tests/connect_perf_drive.rs only. Patch-id: cb0a1dfc8f68a67727525adea7458e2cee7d09f8.

Validation environment: LANG=en_US.UTF-8, CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0, exclusive APFS-cloned CARGO_TARGET_DIR.

Quick Checks: every command from .github/actions/quick-checks/action.yml, with captured output and exit status below.

$ shellcheck --version && actionlint
ShellCheck - shell script analysis tool
version: 0.11.0
license: GNU General Public License, version 3
website: https://www.shellcheck.net
exit_code=0

$ cargo fmt --all --check
exit_code=0

$ ./scripts/check_unsafe_code_allowances.sh
exit_code=0

$ ./scripts/check_layer_dependencies.sh
Layer dependency guard passed (no new reverse dependencies/cycles).
exit_code=0

$ ./scripts/check_architecture_migration_rules.sh
Architecture migration rules passed.
exit_code=0

$ ./scripts/check_logging_guardrails.sh
✅ Logging guardrails check passed
exit_code=0

$ ./scripts/check_error_other_format_ratchet.sh
error(format!) ratchet passed (total call sites: 582).
exit_code=0

$ ./scripts/check_no_tokio_io_uring.sh
exit_code=0

$ ./scripts/check_extension_schema_boundaries.sh
Extension schema boundaries passed.
exit_code=0

$ ./scripts/check_body_cache_whitelist.sh
Body-cache whitelist guard passed.
exit_code=0

$ ./scripts/check_s3s_footprint.sh
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)
✅ s3s footprint baseline check passed
exit_code=0

$ ./scripts/check_test_count_ratchet.sh
test-count ratchet OK: 52 crates, 21222 tests, 164 ignored match .config/test-count-baseline.txt
exit_code=0

$ ./scripts/check_fips_wording.sh
FIPS wording guard passed (README.md and CHANGELOG.md contain no forbidden claims).
KMS wording guard passed (crates/kms claims no Vault KV2 Transit wrapping).
exit_code=0

$ ./scripts/check_embedded_secrets.sh
Embedded secret guard passed (no private key material or provider credential literal in the tree).
exit_code=0

$ make script-tests
Running script tests...
entrypoint credential policy tests passed
internode grpc A/B bench env tests passed
object batch benchmark enhanced tests passed
PASS package service script contract
hotpath warp A/B gate tests passed
hotpath warp ABBA tests passed
Scanner validation artifacts written to /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.1pFe6Kl1Yu/out
Scanner validation artifacts written to /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.1pFe6Kl1Yu/out-missing-pid
.............................
----------------------------------------------------------------------
Ran 29 tests in 181.286s

OK
{"decision":"verified","pending_gates":["G01","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G02"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-E"}
{"decision":"verified","pending_gates":["G01","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G02"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-E"}
{"decision":"verified","pending_gates":["G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G01"}
{"decision":"verified","pending_gates":["G01","G02","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G03"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-L"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G14"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G14"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G10"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P1"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P2"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P3"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G05"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G06"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-D"}
Status-and-outcome release descriptor verified: /private/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp8rwn1pe0/descriptor/release-bundle-status-outcome.json
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G05"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G06"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-D"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G07"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G08"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P4"}
exact 1MiB handoff ABBA tests passed
pinned paired ABBA bench tests passed
==> /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/bin/path_containment (-max_total_time=7, -seed=123456789, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/explicit-artifacts/path_containment/, corpus=/private/tmp/gw-orchestrator-2734/ci-drive/fuzz/corpus/path_containment)
-seed=123456789
target=path_containment
seed=123456789
max_total_time=7
git_revision=4d15b816236630b2c42f321ffc9ed3654edc94e3
git_dirty=true
runner_mode=prebuilt
==> /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/bin/path_containment (-max_total_time=1, -seed=2322617706, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/auto-artifacts/path_containment/, corpus=/private/tmp/gw-orchestrator-2734/ci-drive/fuzz/corpus/path_containment)
-seed=2322617706
==> cargo +nightly fuzz run path_containment (-max_total_time=9, -seed=987654321, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/cargo-artifacts/path_containment/)
+nightly
fuzz
run
path_containment
-seed=987654321
seed=987654321
runner_mode=cargo-fuzz
fuzz runner seed manifest tests passed
✅ scripts/python_bin.sh resolver checks passed
self-test ok: clean fixture with every exemption present passes
self-test ok: all 11 pattern families fire
self-test ok: moving a pinned source is reported
Embedded secret guard self-test passed.
self-test ok: p1-s3s-import: 'use s3s::s3_error;' counts every line, the rustfs_s3_types module's too (s3_error_lines=3)
self-test ok: p2-s3s-mixed-tree: a multi-line s3s use tree naming s3_error counts (s3_error_lines=3)
self-test ok: p3-s3s-glob: a crate-root glob of s3s, bare or inside a tree, counts (s3_error_lines=5)
self-test ok: p4-macro-use: '#[macro_use] extern crate s3s' counts (s3_error_lines=3)
self-test ok: p5-qualified-s3s: a qualified s3s::s3_error! line counts in a cleared file (s3_error_lines=2)
self-test ok: p6-exact-pass: a baseline equal to the tree passes (exit 0)
self-test ok: n1-rustfs-import: 'use rustfs_s3_types::s3_error;' alone counts nothing (s3_error_lines=1)
self-test ok: n2-rustfs-mixed-tree: a rustfs_s3_types tree plus an s3s tree without the macro counts nothing (s3_error_lines=1)
self-test ok: n3-defines: the file defining the macro counts nothing (s3_error_lines=1)
self-test ok: n4-qualified-rustfs: a qualified rustfs_s3_types::s3_error! line counts nothing (s3_error_lines=1)
self-test ok: n5-renamed-call: a call through a renamed macro is not text-matched (out of scope) (s3_error_lines=1)
self-test ok: n6-comment-string: a comment or string mentioning s3_error! in a cleared file counts nothing (s3_error_lines=1)
self-test ok: n7-e2e-excluded: crates/e2e_test is excluded (s3_error_lines=1)
self-test ok: n8-super-glob: an unrecognised import form ('use super::*;') counts (s3_error_lines=3)
self-test ok: n9-facade: a crate-local re-export counts (s3_error_lines=2)
self-test ok: n10-renamed-import: a renamed rustfs_s3_types import does not clear s3_error! lines (s3_error_lines=2)
self-test ok: n11-commented-import: a commented-out rustfs_s3_types import does not clear (s3_error_lines=2)
self-test ok: n12-rustfs-glob: a glob import of rustfs_s3_types does not clear (s3_error_lines=2)
self-test ok: n13-comment-only: a comment mentioning s3_error! in a file with no import counts (s3_error_lines=2)
self-test ok: f1-baseline-missing: a missing baseline file fails (exit 1)
self-test ok: f2-key-missing: a baseline without s3_error_lines fails (exit 1)
self-test ok: f3-rg-fails: a failing classification search fails the guard (exit 1)
✅ s3s footprint self-test passed
test_core_gate_rejects_missing_ignored_filtered_and_corrupt_inputs (__main__.SelfTests.test_core_gate_rejects_missing_ignored_filtered_and_corrupt_inputs) ... ok
test_e2e_requires_registration (__main__.SelfTests.test_e2e_requires_registration) ... ok
test_fuzz_runtime_uses_matrix_target (__main__.SelfTests.test_fuzz_runtime_uses_matrix_target) ... ok
test_ilm_lane_keeps_the_measured_cargo_build_budget (__main__.SelfTests.test_ilm_lane_keeps_the_measured_cargo_build_budget) ... ok
test_profile_listing_binds_platform_digest (__main__.SelfTests.test_profile_listing_binds_platform_digest) ... ok
test_profile_listing_enforces_selection (__main__.SelfTests.test_profile_listing_enforces_selection) ... ok
test_quick_checks_commands_propagate_failure (__main__.SelfTests.test_quick_checks_commands_propagate_failure) ... ok
test_quick_checks_rejects_caller_and_execution_bypasses (__main__.SelfTests.test_quick_checks_rejects_caller_and_execution_bypasses) ... ok
test_runner_readiness_fails_closed (__main__.SelfTests.test_runner_readiness_fails_closed) ... ok
test_s3_runner_rejects_unbounded_failure_locals (__main__.SelfTests.test_s3_runner_rejects_unbounded_failure_locals) ... ok
test_scanner_heal_begin_requires_embedded_source_provenance (__main__.SelfTests.test_scanner_heal_begin_requires_embedded_source_provenance) ... ok
test_scanner_heal_case_does_not_approve_pending_release (__main__.SelfTests.test_scanner_heal_case_does_not_approve_pending_release) ... ok
test_scanner_heal_case_lane_is_required_for_runner_profile (__main__.SelfTests.test_scanner_heal_case_lane_is_required_for_runner_profile) ... ok
test_scanner_heal_case_only_schema_cannot_approve_release (__main__.SelfTests.test_scanner_heal_case_only_schema_cannot_approve_release) ... ok
test_scanner_heal_crash_case_rejects_restart_oracle (__main__.SelfTests.test_scanner_heal_crash_case_rejects_restart_oracle) ... ok
test_scanner_heal_ec84_case_rejects_wrong_erasure_geometry (__main__.SelfTests.test_scanner_heal_ec84_case_rejects_wrong_erasure_geometry) ... ok
test_scanner_heal_finish_collects_oracles_from_registry (__main__.SelfTests.test_scanner_heal_finish_collects_oracles_from_registry) ... ok
test_scanner_heal_multipool_case_accepts_deferred_optional_outage_write (__main__.SelfTests.test_scanner_heal_multipool_case_accepts_deferred_optional_outage_write) ... ok
test_scanner_heal_pending_gate_cannot_map_to_implemented_lane (__main__.SelfTests.test_scanner_heal_pending_gate_cannot_map_to_implemented_lane) ... ok
test_scanner_heal_receipts_reject_reuse_and_missing_builds (__main__.SelfTests.test_scanner_heal_receipts_reject_reuse_and_missing_builds) ... ok
test_scanner_heal_rejects_boolean_fractional_and_out_of_geometry_integers (__main__.SelfTests.test_scanner_heal_rejects_boolean_fractional_and_out_of_geometry_integers) ... ok
test_scanner_heal_rejects_broken_execution_and_artifacts (__main__.SelfTests.test_scanner_heal_rejects_broken_execution_and_artifacts) ... ok
test_scanner_heal_rejects_copied_junit_and_wrong_suite_build (__main__.SelfTests.test_scanner_heal_rejects_copied_junit_and_wrong_suite_build) ... ok
test_scanner_heal_release_bundle_accepts_complete_measured_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_accepts_complete_measured_evidence) ... ok
test_scanner_heal_release_bundle_assembler_copies_measured_artifacts (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_copies_measured_artifacts) ... ok
test_scanner_heal_release_bundle_assembler_merges_measured_descriptors (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_merges_measured_descriptors) ... ok
test_scanner_heal_release_bundle_assembler_rejects_duplicate_descriptor_gate (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_duplicate_descriptor_gate) ... ok
test_scanner_heal_release_bundle_assembler_rejects_fixture_descriptor (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_fixture_descriptor) ... ok
test_scanner_heal_release_bundle_assembler_rejects_hash_mismatch (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_hash_mismatch) ... ok
test_scanner_heal_release_bundle_assembler_rejects_mixed_descriptor_revisions (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_mixed_descriptor_revisions) ... ok
test_scanner_heal_release_bundle_enforces_topology_duration_profile_and_versions (__main__.SelfTests.test_scanner_heal_release_bundle_enforces_topology_duration_profile_and_versions) ... ok
test_scanner_heal_release_bundle_fixture_writer_is_not_release_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_fixture_writer_is_not_release_evidence) ... ok
test_scanner_heal_release_bundle_gate_accepts_single_measured_gate (__main__.SelfTests.test_scanner_heal_release_bundle_gate_accepts_single_measured_gate) ... ok
test_scanner_heal_release_bundle_gate_rejects_missing_field (__main__.SelfTests.test_scanner_heal_release_bundle_gate_rejects_missing_field) ... ok
test_scanner_heal_release_bundle_gate_rejects_unknown_gate (__main__.SelfTests.test_scanner_heal_release_bundle_gate_rejects_unknown_gate) ... ok
test_scanner_heal_release_bundle_rejects_synthetic_or_missing_fields (__main__.SelfTests.test_scanner_heal_release_bundle_rejects_synthetic_or_missing_fields) ... ok
test_scanner_heal_release_bundle_requires_domain_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_requires_domain_evidence) ... ok
test_scanner_heal_release_bundle_requires_field_provenance (__main__.SelfTests.test_scanner_heal_release_bundle_requires_field_provenance) ... ok
test_scanner_heal_release_bundle_requires_json_artifact_provenance (__main__.SelfTests.test_scanner_heal_release_bundle_requires_json_artifact_provenance) ... ok
test_scanner_heal_release_bundle_requires_mixed_version_field_roles (__main__.SelfTests.test_scanner_heal_release_bundle_requires_mixed_version_field_roles) ... ok
test_scanner_heal_release_bundle_requires_same_gate_measurement_window (__main__.SelfTests.test_scanner_heal_release_bundle_requires_same_gate_measurement_window) ... ok
test_scanner_heal_release_consumes_multiple_registry_oracles (__main__.SelfTests.test_scanner_heal_release_consumes_multiple_registry_oracles) ... ok
test_scanner_heal_release_matrix_lanes_remain_pending_without_real_evidence (__main__.SelfTests.test_scanner_heal_release_matrix_lanes_remain_pending_without_real_evidence) ... ok
test_scanner_heal_release_requirements_cannot_be_cleared (__main__.SelfTests.test_scanner_heal_release_requirements_cannot_be_cleared) ... ok
test_scanner_heal_release_status_rejects_focused_case_run (__main__.SelfTests.test_scanner_heal_release_status_rejects_focused_case_run) ... ok
test_scanner_heal_release_status_rejects_synthetic_case (__main__.SelfTests.test_scanner_heal_release_status_rejects_synthetic_case) ... ok
test_scanner_heal_required_evidence_fields_cannot_be_removed (__main__.SelfTests.test_scanner_heal_required_evidence_fields_cannot_be_removed) ... ok
test_scheduled_alerts_require_completion_watchdog (__main__.SelfTests.test_scheduled_alerts_require_completion_watchdog) ... ok
test_update_profile_selection_changes_only_requested_platform (__main__.SelfTests.test_update_profile_selection_changes_only_requested_platform) ... ok
test_validate_includes_quick_checks (__main__.SelfTests.test_validate_includes_quick_checks) ... ok
test_vault_tests_require_cross_process_serialization (__main__.SelfTests.test_vault_tests_require_cross_process_serialization) ... ok
test_workflow_readiness_requires_dependency_probe (__main__.SelfTests.test_workflow_readiness_requires_dependency_probe) ... ok

----------------------------------------------------------------------
Ran 52 tests in 14.301s

OK
e2e-full selection OK: 1 tests, sha256=29e30e6ed2ed38d9e5b0a7cdd9536abf5c719f7021a02b578ae1381563a3c273
...................................
----------------------------------------------------------------------
Ran 35 tests in 1.161s

OK
...................
----------------------------------------------------------------------
Ran 19 tests in 17.397s

OK
..........
----------------------------------------------------------------------
Ran 10 tests in 0.183s

OK
test_actual_selector_bootstrap_uses_base_policy_and_fails_closed (__main__.SelfTests.test_actual_selector_bootstrap_uses_base_policy_and_fails_closed) ... ok
test_documentation_paths_do_not_hide_build_or_fixture_changes (__main__.SelfTests.test_documentation_paths_do_not_hide_build_or_fixture_changes) ... ok
test_e2e_scope_keeps_production_and_shared_configuration_full (__main__.SelfTests.test_e2e_scope_keeps_production_and_shared_configuration_full) ... ok
test_e2e_shortcut_rejects_new_direct_renamed_and_target_dependencies (__main__.SelfTests.test_e2e_shortcut_rejects_new_direct_renamed_and_target_dependencies) ... ok
test_e2e_shortcut_traverses_implicit_path_dependencies (__main__.SelfTests.test_e2e_shortcut_traverses_implicit_path_dependencies) ... ok
test_event_contract_requires_complete_candidate_and_optional_lanes (__main__.SelfTests.test_event_contract_requires_complete_candidate_and_optional_lanes) ... ok
test_every_wrong_result_missing_job_or_selection_fails_closed (__main__.SelfTests.test_every_wrong_result_missing_job_or_selection_fails_closed) ... ok
test_full_e2e_gate_preserves_workflow_branch_and_event_scope (__main__.SelfTests.test_full_e2e_gate_preserves_workflow_branch_and_event_scope) ... ok
test_git_range_includes_deleted_source_and_rename_origins (__main__.SelfTests.test_git_range_includes_deleted_source_and_rename_origins) ... ok
test_job_ids_and_display_names_cannot_hide_validation (__main__.SelfTests.test_job_ids_and_display_names_cannot_hide_validation) ... ok
test_report_only_stack_lane_mirrors_e2e_smoke_outside_the_aggregate (__main__.SelfTests.test_report_only_stack_lane_mirrors_e2e_smoke_outside_the_aggregate) ... ok
test_repository_wiring_and_missing_dependency_regression (__main__.SelfTests.test_repository_wiring_and_missing_dependency_regression) ... ok
test_verify_command_preserves_failures (__main__.SelfTests.test_verify_command_preserves_failures) ... ok

----------------------------------------------------------------------
Ran 13 tests in 3.280s

OK
security coverage self-test passed
test_api_failures_preserve_other_evidence_and_never_enter_grace (__main__.SelfTests.test_api_failures_preserve_other_evidence_and_never_enter_grace) ... ok
test_attempt_outcome_does_not_replace_recent_success (__main__.SelfTests.test_attempt_outcome_does_not_replace_recent_success) ... ok
test_cancelled_attempt_cannot_refresh_expired_success (__main__.SelfTests.test_cancelled_attempt_cannot_refresh_expired_success) ... ok
test_cli_requires_the_repository_default_branch (__main__.SelfTests.test_cli_requires_the_repository_default_branch) ... ok
test_config_rejects_duplicate_and_invalid_entries (__main__.SelfTests.test_config_rejects_duplicate_and_invalid_entries) ... ok
test_freshness_boundaries (__main__.SelfTests.test_freshness_boundaries) ... ok
test_grace_requires_two_successful_queries_with_no_history (__main__.SelfTests.test_grace_requires_two_successful_queries_with_no_history) ... ok
test_invalid_api_evidence_fails_closed (__main__.SelfTests.test_invalid_api_evidence_fails_closed) ... ok
test_report_retains_every_workflow (__main__.SelfTests.test_report_retains_every_workflow) ... ok
test_requests_filter_schedule_default_branch_and_success_on_server (__main__.SelfTests.test_requests_filter_schedule_default_branch_and_success_on_server) ... ok

----------------------------------------------------------------------
Ran 10 tests in 0.051s

OK
..............................
----------------------------------------------------------------------
Ran 30 tests in 22.354s

OK
..........
----------------------------------------------------------------------
Ran 10 tests in 2.969s

OK
.................................
----------------------------------------------------------------------
Ran 33 tests in 0.160s

OK
Online functional runner labels: pf-testing
.......
----------------------------------------------------------------------
Ran 7 tests in 0.002s

OK
.......
----------------------------------------------------------------------
Ran 7 tests in 0.001s

OK
....
----------------------------------------------------------------------
Ran 4 tests in 0.128s

OK
...
----------------------------------------------------------------------
Ran 3 tests in 0.547s

OK
PASS: cold stampede script self-test
exit_code=0

$ python3 ./scripts/test_object_generation_protocol_model.py
test_a_recovery_barrier_cannot_choose_an_empty_value (__main__.ObjectGenerationProtocolModel.test_a_recovery_barrier_cannot_choose_an_empty_value) ... ok
test_all_bounded_two_proposer_interleavings_choose_at_most_one_value (__main__.ObjectGenerationProtocolModel.test_all_bounded_two_proposer_interleavings_choose_at_most_one_value) ... ok
test_an_unavailable_decision_quorum_cannot_choose_a_candidate (__main__.ObjectGenerationProtocolModel.test_an_unavailable_decision_quorum_cannot_choose_a_candidate) ... ok
test_counter_allocation_requires_persistence_and_is_consumed_once (__main__.ObjectGenerationProtocolModel.test_counter_allocation_requires_persistence_and_is_consumed_once) ... ok
test_counter_restart_burns_persisted_allocations_before_or_after_issue (__main__.ObjectGenerationProtocolModel.test_counter_restart_burns_persisted_allocations_before_or_after_issue) ... ok
test_crash_after_promise_before_reply_does_not_restore_an_older_ballot (__main__.ObjectGenerationProtocolModel.test_crash_after_promise_before_reply_does_not_restore_an_older_ballot) ... ok
test_duplicate_or_unconfigured_voters_cannot_form_a_quorum (__main__.ObjectGenerationProtocolModel.test_duplicate_or_unconfigured_voters_cannot_form_a_quorum) ... ok
test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision (__main__.ObjectGenerationProtocolModel.test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision) ... ok
test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected (__main__.ObjectGenerationProtocolModel.test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected) ... ok
test_forgetting_tombstone_authority_allows_an_aba_create (__main__.ObjectGenerationProtocolModel.test_forgetting_tombstone_authority_allows_an_aba_create) ... ok
test_issuing_before_persistence_reuses_a_ballot_after_restart (__main__.ObjectGenerationProtocolModel.test_issuing_before_persistence_reuses_a_ballot_after_restart) ... ok
test_lost_ack_and_restart_preserve_the_chosen_operation (__main__.ObjectGenerationProtocolModel.test_lost_ack_and_restart_preserve_the_chosen_operation) ... ok
test_minority_accepted_work_is_adopted_and_finished (__main__.ObjectGenerationProtocolModel.test_minority_accepted_work_is_adopted_and_finished) ... ok
test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample (__main__.ObjectGenerationProtocolModel.test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample) ... ok
test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it (__main__.ObjectGenerationProtocolModel.test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it) ... ok
test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor (__main__.ObjectGenerationProtocolModel.test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor) ... ok
test_replacing_voters_with_empty_state_can_choose_a_conflicting_value (__main__.ObjectGenerationProtocolModel.test_replacing_voters_with_empty_state_can_choose_a_conflicting_value) ... ok
test_skipping_predecessor_recovery_chooses_a_broken_revision_chain (__main__.ObjectGenerationProtocolModel.test_skipping_predecessor_recovery_chooses_a_broken_revision_chain) ... ok
test_stale_publication_rollback_and_gc_cannot_change_the_winner (__main__.ObjectGenerationProtocolModel.test_stale_publication_rollback_and_gc_cannot_change_the_winner) ... ok
test_superseded_operation_identity_cannot_be_reused_for_different_input (__main__.ObjectGenerationProtocolModel.test_superseded_operation_identity_cannot_be_reused_for_different_input) ... ok
test_superseded_operation_retry_returns_its_original_successor (__main__.ObjectGenerationProtocolModel.test_superseded_operation_retry_returns_its_original_successor) ... ok
test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision (__main__.ObjectGenerationProtocolModel.test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision) ... ok
test_tombstone_rejects_a_delayed_never_present_create (__main__.ObjectGenerationProtocolModel.test_tombstone_rejects_a_delayed_never_present_create) ... ok
test_unlearned_predecessor_is_recovered_before_the_second_slot (__main__.ObjectGenerationProtocolModel.test_unlearned_predecessor_is_recovered_before_the_second_slot) ... ok

----------------------------------------------------------------------
Ran 24 tests in 0.958s

OK
model: states=51023, transitions=167520, terminal=732, chosen_candidates=2
negative control: double-choice trace=((0, 'prepare', 0), (0, 'prepare', 1), (0, 'prepare', 2), (0, 'accept', 0), (0, 'accept', 1), (0, 'accept', 2), (1, 'prepare', 0), (1, 'prepare', 1), (1, 'prepare', 2), (1, 'accept', 0), (1, 'accept', 1), (1, 'accept', 2))
multi-slot round schedules: 640
exit_code=0

$ python3 ./scripts/check_test_wiring.py
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired
exit_code=0

$ ./scripts/check_no_planning_docs.sh
exit_code=0

$ ./scripts/check_uring_lane_lib_only.sh
OK: no 'uring_'-matching test functions under crates/ecstore/tests; --lib is safe
exit_code=0

Initial complete gate
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
 Nextest run ID e07fc5b9-c6c8-4cdd-ae61-54839bf24437 with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.016s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.011s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.009s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.009s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  16.950s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   4.025s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.008s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.007s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.035s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  21.087s] 10 tests run: 10 passed, 0 skipped
exit_code=0

Restored complete gate
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
 Nextest run ID 233b7307-5361-4eb7-94d5-b384f17d2d6a with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.009s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.008s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.008s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.008s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  19.518s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   4.146s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.009s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.018s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.009s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.054s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  23.787s] 10 tests run: 10 passed, 0 skipped
exit_code=0

Mutation M1: success_measurement reports max(actual elapsed, requested budget + 1), forcing 1001 ms for the one-second request. The retained duration assertion at rustfs/tests/connect_perf_drive.rs:105 fails.
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast -E 'test(real_local_write_read_is_measured_and_scratch_is_removed)'
info: experimental features enabled: setup-scripts
   Compiling rustfs v1.0.1 (/private/tmp/gw-orchestrator-2734/ci-drive/rustfs)
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs") generated 1 warning
warning: `rustfs` (bin "rustfs-cli") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 7m 06s
────────────
 Nextest run ID 9f618777-7bf6-4cbc-8803-dd7f50f449ed with nextest profile: ci
    Starting 1 test across 1 binary (9 tests skipped)
        FAIL [   0.016s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
  stdout ───

    running 1 test
    test real_local_write_read_is_measured_and_scratch_is_removed ... FAILED

    failures:

    failures:
        real_local_write_read_is_measured_and_scratch_is_removed

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 0.01s

  stderr ───

    thread 'real_local_write_read_is_measured_and_scratch_is_removed' (106986846) panicked at rustfs/tests/connect_perf_drive.rs:105:5:
    assertion failed: (1..=1_000).contains(&data.duration_millis)
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

────────────
     Summary [   0.017s] 1 test run: 0 passed, 1 failed, 9 skipped
        FAIL [   0.016s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
error: test run failed
exit_code=100

Mutation M2: target_parameters reports requested duration + 1. The in-process assertion at rustfs/tests/connect_perf_drive.rs:111 and CLI JSON assertion at rustfs/tests/connect_perf_drive.rs:527 both reject 1001 against 1000.
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast -E 'test(real_local_write_read_is_measured_and_scratch_is_removed) | test(production_cli_measures_and_signs_exact_binary_provenance)'
info: experimental features enabled: setup-scripts
   Compiling rustfs v1.0.1 (/private/tmp/gw-orchestrator-2734/ci-drive/rustfs)
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs") generated 1 warning
warning: `rustfs` (bin "rustfs-cli") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 7m 20s
────────────
 Nextest run ID b73d46ab-c910-4e1e-8063-7fc7e332ec15 with nextest profile: ci
    Starting 2 tests across 1 binary (8 tests skipped)
        FAIL [  13.968s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
  stdout ───

    running 1 test
    test production_cli_measures_and_signs_exact_binary_provenance ... FAILED

    failures:

    failures:
        production_cli_measures_and_signs_exact_binary_provenance

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 13.96s

  stderr ───

    thread 'production_cli_measures_and_signs_exact_binary_provenance' (107015462) panicked at rustfs/tests/connect_perf_drive.rs:527:5:
    assertion `left == right` failed
      left: Number(1001)
     right: 1000
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

        FAIL [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
  stdout ───

    running 1 test
    test real_local_write_read_is_measured_and_scratch_is_removed ... FAILED

    failures:

    failures:
        real_local_write_read_is_measured_and_scratch_is_removed

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 0.01s

  stderr ───

    thread 'real_local_write_read_is_measured_and_scratch_is_removed' (107016667) panicked at rustfs/tests/connect_perf_drive.rs:111:5:
    assertion `left == right` failed
      left: 1001
     right: 1000
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

────────────
     Summary [  13.985s] 2 tests run: 0 passed, 2 failed, 8 skipped
        FAIL [  13.968s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        FAIL [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
error: test run failed
exit_code=100

Transient production mutations were restored byte-for-byte before the complete green rerun. Original/restored SHA256: 8d389e6d8836c041aab8c6a93e56addd65ba3f70b33264e1c74f639d0572a2da.

$ git diff --check

exit_code=0

$ git diff --cached --check

exit_code=0

Independent orchestrator acceptance
Tested code tree: e67c72fb01

Tested tree: 16fbbd2ea4
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

$ LANG=en_US.UTF-8 CARGO_BUILD_JOBS=1 CARGO_INCREMENTAL=0 cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
info: experimental features enabled: setup-scripts
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs-cli") generated 1 warning
warning: `rustfs` (bin "rustfs") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.69s
────────────
 Nextest run ID cd1edf4c-f1c0-456d-ad1a-76cc00ce05c9 with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.012s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.010s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.010s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.009s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  20.328s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   5.527s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.013s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.019s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.011s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.049s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  25.991s] 10 tests run: 10 passed, 0 skipped

exit_code=0

All 54 tracked Cargo.toml files parse without duplicate keys. git diff --check passed. The production source remains unchanged; the two deliberate implementation mutations above were restored before this complete green run.

* fix(test): compare legacy body wire parity as raw bytes

Lossy UTF-8 comparison concealed a changed binary response byte.
Compare the already captured Vec<u8> values directly.

Validation:
- Binary mutant: migrated payload 0x84 vs legacy 0x83; old comparator
  passed 1/1, raw comparator failed 0/1 (exit 100); restored green.
- Non-Swift body-drain mutant: untouched-body delegation test failed
  with empty vs payload (exit 100); restored handler without a diff.
- cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols
  --features rustfs-protocols/swift --no-fail-fast:
  543 tests run, 543 passed, 0 skipped after both mutants restored.
- cargo fmt --all --check and git diff --check passed.

Orchestrator acceptance (LANG=en_US.UTF-8):
cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
Summary [2.445s] 543 tests run: 543 passed, 0 skipped.

All shared quick checks on the final code tree:
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

Final independent acceptance after the drive-test CI repair
Parent: a99d8f4f89
Tested code tree: 93cbd96bf0

Tested tree: 075ae5161c2b62a96812bc1e3c904a7663b4f2f8
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

$ LANG=en_US.UTF-8 CARGO_BUILD_JOBS=1 CARGO_INCREMENTAL=0 cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
    Finished `test` profile [unoptimized + debuginfo] target(s) in 10m 08s
 Nextest run ID 5c7ff102-ec6d-479a-86e0-810859b28bbf with nextest profile: ci
    Starting 543 tests across 8 binaries
     Summary [   6.729s] 543 tests run: 543 passed, 0 skipped
exit_code=0

The only source change is the raw-byte wire comparator. Cargo manifests and the ratchet baselines remain unchanged from the independently validated parent. The invalid-byte and consumed-request mutations remain recorded above; their implementation changes were restored.

* fix: repair integration checks and IPC test readiness

Wait for the RPC/locks test sockets to pass the existing owner and mode
checks inside the original 5s readiness window and 10ms poll interval.

Use the existing admin S3 carrier for the new recovery and quota error
calls. Regenerate the test-count baseline for main's 118 existing tests
across seven crates. Pair the private pagination helper's raw-scan inputs
and remove the redundant Copy clone from the retirement event.

Keep the original assertions, responses, authorization, locks and storage
operations. The pagination body and raw-scan expressions at all five calls
are byte-identical; reinserting only the Copy clone restores the entire
base heal file. All ignored baseline values and s3s counts stay unchanged.

Validation base: c47432e353
Validation tree: f081084bc5
Patch SHA-256: 2dcf0028691bfb2772edfb38e262829b04f4151832dcce823c9dd9f7434b52de
Trace production-prefix SHA-256: fd994f218a21429d62866f4efb8eee1ee57b58f4a51865aa35d05a54ef1ccb45

All commands used LANG=en_US.UTF-8, CARGO_BUILD_JOBS=1,
CARGO_INCREMENTAL=0 and CARGO_NET_OFFLINE=true in one exclusive target.
Every gate retained the same tree and six source hashes before and after.
54 Cargo.toml files parse without duplicate keys.

cargo clippy -p rustfs --all-targets -- -D warnings
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 12m 50s
Exit: 0. Wall time: 770.855s.

The original 19 quick-check steps:
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
Wall time: 286.337s.

cargo nextest run --profile ci -p rustfs --no-fail-fast
     Summary [ 217.017s] 5382 tests run: 5382 passed (4 leaky), 40 skipped
Exit: 0. Wall time: 1324.414s.

cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
     Summary [   2.189s] 543 tests run: 543 passed, 0 skipped
Exit: 0. Wall time: 123.192s.

cargo nextest run --profile ci -p rustfs-ecstore list_objects_paginate --no-fail-fast
     Summary [   0.170s] 8 tests run: 8 passed, 5921 skipped
Exit: 0. Wall time: 261.324s.

cargo nextest run --profile ci -p rustfs-heal durable_mrf_anchor_retires_only_after_exact_bucket_generation_is_deleted --no-fail-fast
     Summary [   0.805s] 1 test run: 1 passed, 835 skipped
Exit: 0. Wall time: 199.801s.

Baseline regeneration:
./scripts/check_test_count_ratchet.sh --update
wrote .config/test-count-baseline.txt: 52 crates, 21340 tests, 164 ignored
Test-only increases: ecstore48, heal3, iam14, io-metrics2, obs3, policy14,
rustfs34. Existing ignored values and all other rows are byte-identical.
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)

Historical runtime red controls on f622 (same retained two assertions):
cargo nextest run --profile ci -p rustfs --lib -E 'test(=connect::diagnostics::trace_runtime::tests::local_top_rpc_reads_service_http_completions_and_signs_offline) | test(=connect::diagnostics::trace_runtime::tests::local_top_locks_reads_service_manager_and_signs_offline)' --no-fail-fast
Original path readiness with a temporary 500ms bind/seal gap:
     Summary [   0.863s] 2 tests run: 0 passed, 2 failed, 5113 skipped
Exit: 100.
Sealed readiness in that same gap:
     Summary [   5.621s] 2 tests run: 2 passed, 5113 skipped
Exit: 0.
Compiled M1: replace the helper predicate with path.exists():
     Summary [   0.870s] 2 tests run: 0 passed, 2 failed, 5113 skipped
Exit: 100.
Restore the sealed predicate byte-for-byte:
     Summary [   5.640s] 2 tests run: 2 passed, 5113 skipped
Exit: 0.
Both reds observe mode755 with owner_matches=true and fail at the original
RPC/locks unwraps with StateSecurity. M1 failure locations: trace_runtime.rs
2547:29 and 2713:29. No new assertions were introduced.

cargo nextest run --profile ci -p rustfs --lib -E 'test(connect::diagnostics::trace_runtime::tests::)' --no-fail-fast
     Summary [  20.748s] 28 tests run: 28 passed (1 leaky), 5087 skipped
Exit: 0. This security sweep is historical; the full final unit gate above
covers the actual c474 candidate.
The first disk-stopped compile ran no tests and is excluded from red credit.
All temporary seal instrumentation is removed from the final source.

Final unit cleanup markers: 4 leaky, 40 original skips (exit0). The ld
__eh_frame linker warning also appears verbatim in the original f622 run
and the earlier c474 run; there are no new source warnings.

* fix: remove redundant pending closure in disk test

Pass the capture-free pending future factory as a function item in the existing slow-operation test. This fixes the redundant_closure error at disk_store.rs:2914 that stopped strict Clippy before the feature tests could run.

The single-line test edit preserves the factory's result, warning and timeout behavior. Reversing the replacement reproduces the parent file byte for byte. All 220 assertion lines and the production prefix are unchanged; no tests, assertions, guards, profiles or baselines were added or relaxed.

Original Linux CI evidence (run 37888552268 at 53245be9f):
- RIO job113685483992: strict Clippy exit101, redundant closure at disk_store.rs:2914; feature tests skipped.
- Workspace Test and Lint job113685484058: same strict Clippy exit101; nextest and doc tests skipped.
These are lint failures, not runtime mutation kills or known flakes.

Native macOS arm64 validation on tree7a5bf47e7533b29479bc85207416102f1337c566, LANG=en_US.UTF-8, one build job, CARGO_INCREMENTAL=0 and offline dependencies:

shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
All 19 checks: exit0, 312.161s.

$ cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 20m 11s
exit0, 1213.210s total.

$ cargo nextest run --profile ci -p rustfs-ecstore --features rio-v2 slow_disk_operation_warns_once_while_pending_and_preserves_result --no-fail-fast
Finished `test` profile [unoptimized + debuginfo] target(s) in 4m 34s
PASS [0.026s] rustfs-ecstore disk::disk_store::tests::slow_disk_operation_warns_once_while_pending_and_preserves_result
Summary [   0.170s] 1 test run: 1 passed, 5943 skipped
exit0, 281.920s total; 5943 existing filter skips.

warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
This linker warning is retained; flags, profiles and allowances are unchanged.

54 tracked Cargo.toml files parsed without duplicate keys. git diff --check passed. No new assertions were introduced, so no new runtime mutation is claimed. Native compilation and the existing helper test do not substitute for actual Linux RIO execution; the new integration CI run still needs to recover before the RPC merge.

Preservation and output hashes:
parent file SHA256: 8e59ee745e3aaaa22848b943d3b10f6353b515956282cde1f3af2c394200c9ac
final file SHA256: b69ba3237061b161448ae8b62b615885d5770b98a5b4a2479bcfbd70f1bd1f17
unchanged production prefix SHA256: 7903a6e288252c420231242d7b58a25fa2a7140ea3dd430cbe7f0dde11369dcb
RIO Linux failure log SHA256:421addb9868fd88e4109a922311e54f91f0dfd70862f8ba7fcaba193482ed90b
Workspace Linux failure log SHA256:e0aea8d0f014c022252a9534968e083c98cfc1fd760da445ecd04ec3f522e26a
19 output SHA256:35a54f28d707956ed1edb29d492c4dfe8a9fe753d103b111b5a2f1b5c41c2b98
RIO Clippy output SHA256:d6046e3868f2b3992fe11cdd04e9e98eb50a87d1234d9a70dbfeb154ff61638d
slow-operation output SHA256:605b7b258d2fd09c6ef2ab2ffc90d433d0509480f1e45125d20c823759e83608

Refs rustfs/backlog#2734.

* test(ecstore): isolate tier-free recovery fixture startup

The fixture constructor can take the production recovery lock and wait for 60 seconds before this test installs its queue and seeds metadata. Disable its background recovery only while creating this fixture, then restore the environment before calling the existing production recovery entrypoint.

Only the existing test setup changes. Its production entrypoint, default lock parameters, original assertions, 30-second timeout, cancellation and cleanup remain intact. No tests or assertions were added. Production and store/init.rs are byte-identical to the parent.

Independent acceptance ran on merge tree 8ec358cb42ba9ebb3bcfeafcef048a3f0366355b; the runner header reports parent HEAD, while before/after index and source hashes confirm this candidate. Worker task branch d8055766b8 has the same tree.

Every command used LANG=en_US.UTF-8. Task compilation also used CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0, CARGO_NET_OFFLINE=true, an independent target, and empty RUSTC_WRAPPER/RUSTC_WORKSPACE_WRAPPER to keep compilers in its owned process group without changing repository configuration.

Command:
shellcheck --version && actionlint
Exit 0; observed output excerpt:
version: 0.11.0
Log SHA-256: fc9eec16b4191d058a295813ca9afbe9d2b37b59975da860170726be162a3924

Command:
cargo fmt --all --check
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_unsafe_code_allowances.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_layer_dependencies.sh
Exit 0; observed output excerpt:
Layer dependency guard passed (no new reverse dependencies/cycles).
Log SHA-256: 69c6cb1896c0586b28d1167380d3260d99d8017a4265c8df5a2348f289e93a12

Command:
./scripts/check_architecture_migration_rules.sh
Exit 0; observed output excerpt:
Architecture migration rules passed.
Log SHA-256: 28790e70d070607ed4de0704f292a65030be92ec224ef106d4b72fd8296f896e

Command:
./scripts/check_logging_guardrails.sh
Exit 0; observed output excerpt:
✅ Logging guardrails check passed
Log SHA-256: 7adf41abff8ed8f7617decf86498888ddde78937099dd9ce47f6d359dfc2d317

Command:
./scripts/check_error_other_format_ratchet.sh
Exit 0; observed output excerpt:
error(format!) ratchet passed (total call sites: 579).
Log SHA-256: 66172aef18f150ba0605af122e26ffcf91f6dd86f43e38b05e2e982ce8d783ee

Command:
./scripts/check_no_tokio_io_uring.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_extension_schema_boundaries.sh
Exit 0; observed output excerpt:
Extension schema boundaries passed.
Log SHA-256: f13d0acd0e2ce5d9815241b12b4dab7d01cc11b9b65bcf6dfb7812c878532a51

Command:
./scripts/check_body_cache_whitelist.sh
Exit 0; observed output excerpt:
Body-cache whitelist guard passed.
Log SHA-256: 6b828847b0383950ee6eef9cd846d0c322c945e47efd4c7d1ad881178810d6bb

Command:
./scripts/check_s3s_footprint.sh
Exit 0; observed output excerpt:
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)
✅ s3s footprint baseline check passed
Log SHA-256: 47f53d04ef645618563fe1f22abc64667f19078f1974feedcca46e4f2c016844

Command:
./scripts/check_test_count_ratchet.sh
Exit 0; observed output excerpt:
test-count ratchet OK: 52 crates, 21340 tests, 164 ignored match .config/test-count-baseline.txt
Log SHA-256: 618a5e330e933de0c6ff484546df401457024a88bb5986d11d588794e33dab60

Command:
./scripts/check_fips_wording.sh
Exit 0; observed output excerpt:
FIPS wording guard passed (README.md and CHANGELOG.md contain no forbidden claims).
KMS wording guard passed (crates/kms claims no Vault KV2 Transit wrapping).
Log SHA-256: 99221cc9979c480ec4f46f50d76c52b18c72fe499f12e15e4f4a22a1b8bfb140

Command:
./scripts/check_embedded_secrets.sh
Exit 0; observed output excerpt:
Embedded secret guard passed (no private key material or provider credential literal in the tree).
Log SHA-256: 437a3ceba32094d2901889965ad191e411aad32cdb18bf0b8659e818011c406b

Command:
make script-tests
Exit 0; observed output excerpt:
test_cli_requires_the_repository_default_branch (__main__.SelfTests.test_cli_requires_the_repository_default_branch) ... ok
test_config_rejects_duplicate_and_invalid_entries (__main__.SelfTests.test_config_rejects_duplicate_and_invalid_entries) ... ok
test_freshness_boundaries (__main__.SelfTests.test_freshness_boundaries) ... ok
test_grace_requires_two_successful_queries_with_no_history (__main__.SelfTests.test_grace_requires_two_successful_queries_with_no_history) ... ok
test_invalid_api_evidence_fails_closed (__main__.SelfTests.test_invalid_api_evidence_fails_closed) ... ok
test_report_retains_every_workflow (__main__.SelfTests.test_report_retains_every_workflow) ... ok
test_requests_filter_schedule_default_branch_and_success_on_server (__main__.SelfTests.test_requests_filter_schedule_default_branch_and_success_on_server) ... ok
Ran 10 tests in 0.042s
OK
Ran 30 tests in 20.627s
OK
Ran 10 tests in 2.268s
OK
Ran 33 tests in 0.127s
OK
Ran 7 tests in 0.002s
OK
Ran 7 tests in 0.001s
OK
Ran 4 tests in 0.108s
OK
Ran 3 tests in 0.564s
OK
PASS: cold stampede script self-test
Log SHA-256: 75fda2d069a373c63667b908baaf99e6591aab9099a739b0c4c22ceb6da8d68c

Command:
python3 ./scripts/test_object_generation_protocol_model.py
Exit 0; observed output excerpt:
test_an_unavailable_decision_quorum_cannot_choose_a_candidate (__main__.ObjectGenerationProtocolModel.test_an_unavailable_decision_quorum_cannot_choose_a_candidate) ... ok
test_counter_allocation_requires_persistence_and_is_consumed_once (__main__.ObjectGenerationProtocolModel.test_counter_allocation_requires_persistence_and_is_consumed_once) ... ok
test_counter_restart_burns_persisted_allocations_before_or_after_issue (__main__.ObjectGenerationProtocolModel.test_counter_restart_burns_persisted_allocations_before_or_after_issue) ... ok
test_crash_after_promise_before_reply_does_not_restore_an_older_ballot (__main__.ObjectGenerationProtocolModel.test_crash_after_promise_before_reply_does_not_restore_an_older_ballot) ... ok
test_duplicate_or_unconfigured_voters_cannot_form_a_quorum (__main__.ObjectGenerationProtocolModel.test_duplicate_or_unconfigured_voters_cannot_form_a_quorum) ... ok
test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision (__main__.ObjectGenerationProtocolModel.test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision) ... ok
test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected (__main__.ObjectGenerationProtocolModel.test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected) ... ok
test_forgetting_tombstone_authority_allows_an_aba_create (__main__.ObjectGenerationProtocolModel.test_forgetting_tombstone_authority_allows_an_aba_create) ... ok
test_issuing_before_persistence_reuses_a_ballot_after_restart (__main__.ObjectGenerationProtocolModel.test_issuing_before_persistence_reuses_a_ballot_after_restart) ... ok
test_lost_ack_and_restart_preserve_the_chosen_operation (__main__.ObjectGenerationProtocolModel.test_lost_ack_and_restart_preserve_the_chosen_operation) ... ok
test_minority_accepted_work_is_adopted_and_finished (__main__.ObjectGenerationProtocolModel.test_minority_accepted_work_is_adopted_and_finished) ... ok
test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample (__main__.ObjectGenerationProtocolModel.test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample) ... ok
test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it (__main__.ObjectGenerationProtocolModel.test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it) ... ok
test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor (__main__.ObjectGenerationProtocolModel.test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor) ... ok
test_replacing_voters_with_empty_state_can_choose_a_conflicting_value (__main__.ObjectGenerationProtocolModel.test_replacing_voters_with_empty_state_can_choose_a_conflicting_value) ... ok
test_skipping_predecessor_recovery_chooses_a_broken_revision_chain (__main__.ObjectGenerationProtocolModel.test_skipping_predecessor_recovery_chooses_a_broken_revision_chain) ... ok
test_stale_publication_rollback_and_gc_cannot_change_the_winner (__main__.ObjectGenerationProtocolModel.test_stale_publication_rollback_and_gc_cannot_change_the_winner) ... ok
test_superseded_operation_identity_cannot_be_reused_for_different_input (__main__.ObjectGenerationProtocolModel.test_superseded_operation_identity_cannot_be_reused_for_different_input) ... ok
test_superseded_operation_retry_returns_its_original_successor (__main__.ObjectGenerationProtocolModel.test_superseded_operation_retry_returns_its_original_successor) ... ok
test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision (__main__.ObjectGenerationProtocolModel.test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision) ... ok
test_tombstone_rejects_a_delayed_never_present_create (__main__.ObjectGenerationProtocolModel.test_tombstone_rejects_a_delayed_never_present_create) ... ok
test_unlearned_predecessor_is_recovered_before_the_second_slot (__main__.ObjectGenerationProtocolModel.test_unlearned_predecessor_is_recovered_before_the_second_slot) ... ok
Ran 24 tests in 0.929s
OK
Log SHA-256: 0b4be926efb0609d497ae58930340153f01957b606206cbbdb3e0d47eb29d3a2

Command:
python3 ./scripts/check_test_wiring.py
Exit 0; observed output excerpt:
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired
Log SHA-256: 21f676f458451525535ab0ef73d6a9c0d9774e16d76a4d19e396ad34936dd39c

Command:
./scripts/check_no_planning_docs.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_uring_lane_lib_only.sh
Exit 0; observed output excerpt:
OK: no 'uring_'-matching test functions under crates/ecstore/tests; --lib is safe
Log SHA-256: 43f3f851e27ec2ab9e34d6b05fefdede38af992699c473904bfe6f53a46d52e3

Command:
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast
Exit 0; wall 234.399s.
Observed output:
    Finished `test` profile [unoptimized + debuginfo] target(s) in 3m 49s
    Starting 1 test across 1 binary (5871 tests skipped)
  SETUP PASS [   0.006s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"'
        PASS [   0.440s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item
     Summary [   0.586s] 1 test run: 1 passed, 5871 skipped
Log SHA-256: 34e973f86e8dc2102c1a0e483e0ac802ce7df3d1b40a603932f62c0805c4de27

Command:
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over) | test(=store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence) | test(=store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable)' --no-fail-fast
Exit 0; wall 4.010s.
Observed output:
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.43s
    Starting 3 tests across 1 binary (5869 tests skipped)
  SETUP PASS [   0.005s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"'
        PASS [   0.068s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over
        PASS [   0.637s] rustfs-ecstore store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence
        PASS [   0.846s] rustfs-ecstore store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable
     Summary [   0.987s] 3 tests run: 3 passed, 5869 skipped
Log SHA-256: b5da2a927ff18ca9aa217762f7186a6d4371bd9e8feedd7b77940e10b64dffac

Command:
cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
Exit 0; wall 1084.508s.
Observed output:
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 18m 03s
Log SHA-256: 28f98be7970faaa71cd118da1da4e6e2e3d23b9dda7e6449eaa7a37e741260b5

Compiled mutation command (unchanged for each red and restored run):
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast

M1: reverse only this test-local constructor flag from false to true. The mutant compiled, then failed the original recovered-item queue assertion after 30.481s. Restoring the entire observed candidate passed in 0.460s. Red log 55bf3b1535529ac0bcfb8071f9b56f9f40631007a85c315367e7e93786d65293; restored log 204d5fbe4574a7917f5e740a795d864fe28a7e7fb5e33d27b0a9cf68e2f17f52.

M2: replace only the actual awaited enqueue with false, consuming its original inputs. The mutant compiled, then failed the same original queue assertion after 30.516s. Restoring the entire observed candidate passed in 0.457s. Red log 89186330413215e231e2b06bf013cc31097ea8f4a106d91421a36d6c8986a7fd; restored log e848b42d2302401afa5dbcd9e37bf76d2e56a96ab6de5a488513b997f53c5e9b.

All temporary observation and mutation code was removed before final acceptance. Historical baseline and diagnostic failures remain recorded; they are not mutation credit. The separate Object Lock intent-retry diagnostic still fails, and these native macOS checks do not establish Linux CI recovery.

* refactor(rpc): serve internode rpc bodies with the owned Body (#8441)

* test(rpc): pin the wire bytes of every internode rpc response body

rustfs/backlog#2747 batch 4, step 1 of 2, on the s3s bodies the rpc paths
serve today. Each response body builder in storage/rpc/http_service.rs is
served through hyper's HTTP/1.1 server and the raw bytes the peer reads are
compared with goldens captured from this commit: status, content-length or
chunked framing, chunk boundaries, the terminating chunk, HEAD, and the cut
connection a failure after data produces. The capability responses carry a
per-process proof, so their length header and payload are checked against
the body they encode instead.

The read_file_stream response builder moves into read_file_stream_response
so the test reaches the same code the handler runs; behaviour is unchanged.

* refactor(rpc): serve internode rpc bodies with the owned Body

rustfs/backlog#2747 batch 4, step 2 of 2. The internode rpc HTTP paths
build their responses with rustfs_s3_types::Body: Body::from_stream
replaces Body::from(StreamingBlob::wrap(..)) at the six stream sites, and
the in-memory and empty bodies keep their constructors. InternodeRpcService
converts its responses into the wrapped service's response body through
From<Body>, so the server keeps composing it with the s3s service as
before. The wire-byte goldens from the previous commit pass unchanged;
storage/rpc no longer names s3s.

* fix(rpc): simplify wire test response builder type

Name the boxed response builder used by the capability wire test so
all-target Clippy can check it without a type-complexity warning.

Validation (LANG=en_US.UTF-8, CARGO_INCREMENTAL=0):
- cargo clippy -p rustfs --all-targets -- -D warnings: exit 0.
- cargo nextest run --profile ci -p rustfs --lib -E 'test(/storage::rpc::/)': 279 tests run, 279 passed, 4842 skipped.
- All 19 quick-check commands, including script-tests, both Python scripts and cargo fmt --all --check: exit 0.
- Footprint baseline regenerated: files=194, s3_error_lines=1584, ecstore_files=36; test-count baseline regenerated: 52 crates, 21228 tests, 164 ignored.
- All 54 Cargo.toml files parse without duplicate keys.

No assertion was added or changed by this lint fix. R1-R8 are retained
in the existing PR verification record.

* fix(ecstore): resume fenced Object Lock creation intents

* fix(ecstore): bound creation retry futures on the stack

* fix(ecstore): require native proof for creation retries

* fix(ecstore): bound bucket creation stack usage (#8466)

* fix(ecstore): retain creation fences through rollback

Keep lifecycle, metadata-transaction, and namespace guards around rollback and metadata cleanup. Preserve the original publication error when a guard is lost. Build the timestamp retry fixture through the real interrupted producer so it checks a valid pending creation.

Validated source tree: 4e8e207999.
Two independent source reviews found no blocking regression.

Validation (LANG=en_US.UTF-8):
- bash -c 'shellcheck --version && actionlint' — exit 0
- cargo fmt --all --check — exit 0
- ./scripts/check_unsafe_code_allowances.sh — exit 0
- ./scripts/check_layer_dependencies.sh — exit 0
- ./scripts/check_architecture_migration_rules.sh — exit 0
- ./scripts/check_logging_guardrails.sh — exit 0
- ./scripts/check_error_other_format_ratchet.sh — exit 0
- ./scripts/check_no_tokio_io_uring.sh — exit 0
- ./scripts/check_extension_schema_boundaries.sh — exit 0
- ./scripts/check_body_cache_whitelist.sh — exit 0
- ./scripts/check_s3s_footprint.sh — exit 0
- ./scripts/check_test_count_ratchet.sh — exit 0
- ./scripts/check_fips_wording.sh — exit 0
- ./scripts/check_embedded_secrets.sh — exit 0
- make script-tests — exit 0
- python3 ./scripts/test_object_generation_protocol_model.py — exit 0
- python3 ./scripts/check_test_wiring.py — exit 0
- ./scripts/check_no_planning_docs.sh — exit 0
- ./scripts/check_uring_lane_lib_only.sh — exit 0
- cargo nextest run --profile ci -p rustfs-ecstore --lib --locked --no-fail-fast — 5894 passed, 0 skipped
- cargo check -p rustfs-ecstore --all-targets --locked — exit 0
- cargo nextest run --profile ci -p rustfs --lib --locked --no-fail-fast -E 'test(=admin::handlers::table_catalog::tests::cold_strong_registration_completes_while_other_bucket_backup_waits) | test(=admin::handlers::table_catalog::tests::commit_backend_borrowed_put_preserves_owner_authorization_and_precondition) | test(=admin::handlers::table_catalog::tests::disable_table_bucket_preserves_objects_and_can_be_reenabled) | test(=admin::handlers::table_catalog::tests::load_table_handler_negotiates_vended_credentials_without_breaking_metadata_only_callers) | test(=admin::handlers::table_catalog::tests::namespace_property_handler_updates_object_backed_catalog_and_maps_errors) | test(=app::object::get::tests::execute_get_object_resumes_from_relocated_pool_without_splicing_body) | test(=table_catalog::store::tests::catalog_writer_loss_prevents_recovery_storage_commit)' — 7 passed
- cargo nextest run --profile ci -p rustfs --lib --locked --no-fail-fast -E 'test(unsupported_remote_disk_)' — 2 passed

Additional fault-injection evidence remains incomplete. Timestamp M1 reached its original assertion; its restore run is being collected. M2–M5 and the three planned rollback faults have not run. An automated risk check interrupted the timestamp proof task; no assertions or validation settings were weakened.

* fix(ecstore): box nested bucket rollback state

Keep the guarded rollback future behind a heap allocation so bucket creation does not store its nested state inline. The rollback body, all three fence references and pending-proof revocation order are unchanged.

The same three original Linux tests overflowed on both e611 CI attempts. Local format, ecstore test compilation and all 19 quick-check steps pass; the original Linux feature lanes still need to validate this candidate.

* test(ecstore): isolate rollback guard loss fixtures

Run the three rollback metadata-cleanup cases in separate test processes so reused /proc/self/fd aliases from earlier cases cannot invalidate their creation witnesses. Keep the original fixture, guards, assertions and io_uring test selection, and verify that each child runs the intended case successfully.

Update the generated test-count baseline for the added macro arm.

Validation: cargo check -p rustfs-ecstore --tests; cargo fmt --all --check; all 19 quick-check commands, including make script-tests and the two Python checks. Linux io_uring acceptance and fault controls remain pending.

* refactor(replication): move the replication config onto gateway types (#8443)

* wip: T1.4 replication family

* refactor(replication): move the replication config onto gateway types

The replication configuration family leaves s3s: rustfs-replication no longer depends on s3s, and ECStore bucket metadata, the scanner and the rustfs app/admin layers hold rustfs_gateway_types::persistence::PersistedReplicationConfiguration. The persisted XML goes through the gateway persistence codec behind a new BucketConfigXml trait (the 39 byte goldens pass with unchanged fixtures); the legacy s3s edge converts once in storage/ecfs.rs through storage/s3_compat/replication.rs; the scanner data usage cache keeps its MessagePack bytes through a serde mirror pinned by bytes recorded from s3s 5761ddfe. rustfs-gateway-types 0.32.0 is pinned once (rev 772d3385). Refs rustfs/backlog#2745.

* test: use S3 serializer for list response tests

* fix(ci): restore integration test compilation

Gate the diagnostic test import with its feature, initialize the conditional
MetadataWriteStore fields explicitly, and decode the scanner replication fixture
through its existing cache boundary. Preserve the original fixture and assertions.

Validation was run on tree 544acb80c0 at 6003d4b8.
LANG=en_US.UTF-8; no source changes follow validation.

Quick checks: 19/19 passed.
- shellcheck --version && actionlint: exit 0
- cargo fmt --all --check: exit 0
- ./scripts/check_unsafe_code_allowances.sh: exit 0
- ./scripts/check_layer_dependencies.sh: exit 0
- ./scripts/check_architecture_migration_rules.sh: exit 0
- ./scripts/check_logging_guardrails.sh: exit 0
- ./scripts/check_error_other_format_ratchet.sh: exit 0
- ./scripts/check_no_tokio_io_uring.sh: exit 0
- ./scripts/check_extension_schema_boundaries.sh: exit 0
- ./scripts/check_body_cache_whitelist.sh: exit 0
- ./scripts/check_s3s_footprint.sh: exit 0
- ./scripts/check_test_count_ratchet.sh: exit 0
- ./scripts/check_fips_wording.sh: exit 0
- ./scripts/check_embedded_secrets.sh: exit 0
- make script-tests: exit 0
- python3 ./scripts/test_object_generation_protocol_model.py: exit 0
- python3 ./scripts/check_test_wiring.py: exit 0
- ./scripts/check_no_planning_docs.sh: exit 0
- ./scripts/check_uring_lane_lib_only.sh: exit 0

Darwin validation:
- cargo clippy --locked -p rustfs-ecstore -p rustfs-scanner --all-targets -- -D warnings: exit 0
- cargo nextest run --locked -p rustfs-ecstore -p rustfs-scanner --lib -E 'test(config::com::tests::save_config_preserves_retryable_system_volume_errors) | test(scanner_io::tests::replication_maintenance_refreshes_compacted_stats_across_cycles_and_retries)' --test-threads 1 --no-fail-fast: exit 0, 2/2 passed
- cargo clippy --locked -p rustfs-ecstore -p rustfs-scanner --all-targets --features rustfs-ecstore/diagnostic-read-version-caller,rustfs-scanner/diagnostic-read-version-caller -- -D warnings: exit 101
- cargo nextest run --locked -p rustfs-ecstore -p rustfs-scanner --lib --features rustfs-ecstore/diagnostic-read-version-caller,rustfs-scanner/diagnostic-read-version-caller -E 'test(config::com::tests::save_config_preserves_retryable_system_volume_errors) | test(scanner_io::tests::replication_maintenance_refreshes_compacted_stats_across_cycles_and_retries) | test(cluster::rpc::runtime_sources::tests::) | test(config::com::tests::diagnostic_config_write_retains_origin_and_api_for_results_and_cancellation)' --test-threads 1 --no-fail-fast: exit 0, 14/14 passed

Diagnostic strict Clippy remains red on four unchanged production lints:
runtime_sources.rs if_same_then_else and three redundant_locals reports in
set_disk/core/metadata_read.rs and set_disk/read.rs. Their source ranges and
feature wiring match the base; the base was not compiled separately.

Linux aarch64 validation (Rust/Clippy 1.99.0, UID 501, jobs=1):
- cargo clippy --all-targets -- -D warnings: exit 0; Finished dev profile in 68m 12s
- cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings: exit 0; Finished dev profile in 49m 45s
Source HEAD/tree and clean status matched before and after both commands.
No compiler or stack overrides were used. protoc 35.1 was installed from its
verified ARM64 release; flatc was absent and not invoked by the workspace build scripts.
The local environment differs from hosted CI.

* fix(ecstore): recover delete batches with missing versions

Treat an explicit non-null version as a no-op when it is absent from both recovery sources. Keep rejecting missing null versions, delete-marker mismatches and conflicting local generations.

Add returning-disk and partial-receipt regressions, extend journal boundary checks, and regenerate the ecstore test-count baseline. Production faults exercise the 24 added assertion sites and 7 semantic checks: 33 faults failed at the expected sites and passed after full restoration.

Validation (LANG=en_US.UTF-8, one build job):

cargo nextest run --locked -p rustfs-ecstore --lib -E 'test(delete_recovery) | test(quorum_delete_cleanup_coordinator_binds_single_and_batch) | test(crash_delete_retry_batch_reclaims_inline_and_reserved_external)' --test-threads 1 --no-fail-fast
Summary [  34.580s] 69 tests run: 69 passed, 6058 skipped; exit 0

cargo clippy --locked -p rustfs-ecstore --all-targets -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 5m 37s; exit 0

All 19 final Quick Checks completed with exit 0:

- shellcheck --version && actionlint
- cargo fmt --all --check
- ./scripts/check_unsafe_code_allowances.sh
- ./scripts/check_layer_dependencies.sh
- ./scripts/check_architecture_migration_rules.sh
- ./scripts/check_logging_guardrails.sh
- ./scripts/check_error_other_format_ratchet.sh
- ./scripts/check_no_tokio_io_uring.sh
- ./scripts/check_extension_schema_boundaries.sh
- ./scripts/check_body_cache_whitelist.sh
- ./scripts/check_s3s_footprint.sh
- ./scripts/check_test_count_ratchet.sh
- ./scripts/check_fips_wording.sh
- ./scripts/check_embedded_secrets.sh
- make script-tests
- python3 ./scripts/test_object_generation_protocol_model.py
- python3 ./scripts/check_test_wiring.py
- ./scripts/check_no_planning_docs.sh
- ./scripts/check_uring_lane_lib_only.sh

test-count ratchet OK: 52 crates, 22009 tests, 165 ignored match .config/test-count-baseline.txt
PASS: cold stampede script self-test
model: states=51023, transitions=167520, terminal=732, chosen_candidates=2
multi-slot round schedules: 640
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired

* fix(ecstore): retry tier commit intent lock contention

Retry transient namespace lock timeouts within the existing three-attempt
CAS budget. Reload and verify the original mutation identity after
contention, and preserve the final typed error when attempts are exhausted.
Cover save/read contention, exhausted retries, replacement identities and
CAS reconciliation without changing the existing timeout or lock policy.

Validation:
- cargo nextest run --locked -p rustfs-ecstore --features test-util --lib -E 'test(tier_mutation)' --test-threads 1 --no-fail-fast: 52 passed.
- cargo nextest run --locked -p rustfs-ecstore --features test-util --lib -E 'test(intent_advance_) | test(=services::tier::tier::tests::caller_cancellation_after_durable_prepare_does_not_hide_the_mutation)' --test-threads 1 --no-fail-fast: 5 passed.
- cargo clippy --locked -p rustfs-ecstore --all-targets -- -D warnings: passed.
- cargo clippy --locked -p rustfs-ecstore --all-targets --features test-util -- -D warnings: passed.
- 27 mutation pairs failed at the intended assertions and passed after restoring the production code.
- bash scripts/check_test_count_ratchet.sh --update, then bash scripts/check_test_count_ratchet.sh: 52 crates, 22013 tests, 165 ignored; ecstore increases from 6581 to 6585.

* refactor(lifecycle): replace s3s domain types with Gateway DTOs

Keep lifecycle rules, restore requests, Object Lock, and scanner cached tags in domain types. Convert at the existing HTTP boundary while preserving legacy XML, scanner cache bytes, date ordering, and tombstones.

Linux test validation before the final import-path-only adjustment:

cargo nextest run -p rustfs-ecstore bucket_config_goldens
39 passed; 6211 skipped by selector.

cargo nextest run --lib -p rustfs -E 'test(/test_zero_rule_lifecycle_tombstone_recognition/) | test(/test_local_lifecycle_staleness_axis_selection/) | test(/test_lifecycle_staleness_preserves_legacy_submillisecond_axis/) | test(/test_merge_incoming_lifecycle/) | test(/test_lifecycle_expiry_statement_matrix/) | test(/test_lifecycle_expiry_subset_xml/) | test(/put_bucket_lifecycle_validation_errors_keep_their_s3_code/) | test(/execute_put_bucket_lifecycle_configuration_rejects_missing_configuration/) | test(/execute_restore_object_rejects_missing_restore_request/) | test(/execute_restore_object_rejects_select_type/) | test(/execute_restore_object_maps_failures_to_typed_s3_errors/) | test(/execute_restore_object_returns_internal_error_when_store_uninitialized/) | test(/restore_request_conversion_preserves_legacy_output_values/) | test(/restore_request_conversion_preserves_presence_and_raw_enums/) | test(/restore_request_conversion_distinguishes_absent_and_empty_containers/)'
25 passed; 5318 skipped by selector.

cargo nextest run --profile ci -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-replication -p rustfs-scanner
7872 passed; 21 skipped under the existing CI profile.

Previously verified mutation controls: 39 production faults, 41 RED/GREEN pairs.

cargo clippy --locked -p rustfs -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-scanner --all-targets -- -D warnings
Passed in 40m 45s before the import-path adjustment, then in 1m 59s on
the final source with warnings denied. The final adjustment imports the
existing helper through the object domain facade; fmt and the architecture
check also passed.

Part of rustfs/backlog#2745. The remaining ecstore configuration families follow separately.

* refactor(admin): move error types to rustfs-s3-types (#8440)

* refactor(protocols): move error types to rustfs-s3-types by codemod

T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and
the error items of protocols move off s3s by scripts/codemods, run with
--hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it
downcasts the backend error, whose runtime type is still s3s::S3Error.

- rustfs/src/protocols/client.rs: the two webdav session methods now
  return rustfs_s3_types::S3Result and convert the s3s error with From.
- codemod: a child module that calls s3_error! through `use super::*`
  gets its own `use rustfs_s3_types::s3_error;` once its parent imports
  that macro by name; otherwise it is listed as GLOB.
- s3-types: a table test pins code, status and the s3s error the legacy
  edge renders for every variant a moved call site names.
- s3_error_lines 1585 -> 1584 (a lifecycle comment reworded).

* refactor(admin): move error types to rustfs-s3-types (WIP, no build)

T1.2 batch 4 (rustfs/backlog#2743), work in progress: rustfs check
still fails with 16 errors (listed in the PR body).

- codemod: --facade PATH treats a crate-local re-export of s3s's error
  items like s3s for the four names (9 new self-tests, 9 mutants red).
- codemod output for rustfs/src/admin and rustfs/src/site_replication
  (--facade crate::storage_api::site_replication::s3); 6 glob children
  get an explicit rustfs_s3_types::s3_error import.
- admin/router.rs: the Operation trait returns rustfs_s3_types::S3Result.
- error.rs: From<ApiError> builds the rustfs-s3-types error; the s3s
  conversion is derived from it.

* fix(admin): convert error types at the s3s boundary

Batch 4 of T1.2 left the admin and site_replication modules returning
rustfs_s3_types::S3Error while their callers and callees on the storage
and auth side still speak s3s::S3Error. Convert at each edge with the
existing From impls in rustfs-s3-types (compat_s3s.rs) instead of
changing any code, message or status:

- admin/auth.rs, admin/handlers/object_zip_download.rs,
  admin/handlers/table_catalog/mod.rs: map the s3s result returned by
  check_key_valid / authorize_request / authorize_internal_object_request
  into the admin error type with .map_err(Into::into).
- storage/ecfs.rs: the s3s-facing side maps the admin site_replication
  gate results the other way.
- admin/handlers/pools.rs: ApiError.code stays s3s::S3ErrorCode, so the
  rustfs code is converted with .into() when the struct is built, in
  production and in the two tests that build or compare it.
- admin/handlers/heal.rs, admin/runtime_sources.rs: the router returns
  s3s errors (S3Route impl); the expected rustfs code is converted with
  .into() before the comparison.
- admin/handlers/table_catalog/mod.rs: iceberg_rest_error takes
  &'static str so the Cow<'static, str> custom code borrows it; every
  caller already passes an ICEBERG_ERROR_* constant or a &'static str.
- admin/handlers/quota.rs: drop a now same-type .into() after s3_error!.
- storage_api.rs, site_replication/mod.rs: drop re-exports and an import
  that no longer have users (clippy -D warnings).

cargo check -p rustfs --all-targets and cargo clippy -p rustfs
--all-targets -- -D warnings are clean; the s3s footprint baseline is
unchanged at files=172, s3_error_lines=303, ecstore_files=28.

Refs rustfs/backlog#2743, rustfs/backlog#2734.

* refactor(ecstore): move bucket configs onto gateway persisted types (#8709)

* refactor(ecstore): move bucket configs onto gateway persisted types

Finish the ECStore batch of T1.4 (rustfs/backlog#2745). The remaining
eleven bucket configuration families (Website, CORS, Tagging, Versioning,
ObjectLock, Encryption, Notification, Logging, Accelerate, RequestPayment,
PublicAccessBlock) now read and write their stored XML through the
rustfs-gateway-types persistence and cors_tagging codecs instead of the
s3s codec, so the legacy s3s bucket-config codec and its
s3gate-metadata-xml compat marker are removed.

- ecstore: BucketConfigXml is implemented by the gateway codec for every
  family; metadata, metadata_sys, versioning, object lock, replication and
  lifecycle consumers take the persisted types. Region moves to
  rustfs_s3_types::Region, and the ETag conversion to s3s moves to the
  rustfs edge with its tests. Only warm_backend.rs and warm_backend_s3.rs
  still reference s3s, held for T1.5 (rustfs/backlog#2746).
- scanner: object lock cache, item actions and data usage use the
  persisted types.
- rustfs: the legacy S3 edge converts each family between s3s DTOs and the
  persisted types in storage/storage_api.rs by writing one codec's XML and
  reading it with the other; Tagging converts field by field.
- Bump the rustfs-gateway-types pin to 21283e87.

The 39 persisted-byte goldens are unchanged and pass, so a document the
old codec wrote reads back and re-encodes to the same bytes. s3s footprint:
files 181 -> 146, ecstore_files 28 -> 2, s3_error_lines unchanged.

* test(app): prepare the PutBucketEncryption config from the request

The uninitialized-store test passed an empty persisted configuration, which the use case now validates before it reaches the store check, so it answered MalformedXML instead of InternalError. Convert the request's configuration the way the S3 edge does.

* style(app): format the PutBucketEncryption test

* refactor(s3-client): move remote error codes and DTO enums off s3s (#8783)

Part of T1.5 (rustfs/backlog#2746), batch 1.

* refactor(s3select): run queries on gateway DTO request shapes (#8794)

* refactor(s3select): run queries on an engine request built from gateway DTO shapes

Part of T1.5 (rustfs/backlog#2746), batch 2.

* fix(s3select): keep the select request conversion in the app layer

The layer guard forbids app -> storage; the conversion moves from storage/s3_compat/select.rs to app/select_request.rs unchanged.

* fix(heal): retry transient metadata conflicts (#8812)

* fix(heal): retry transient metadata conflicts

* chore: update heal test count baseline

* test: reserve nextest capacity for progress probes (#8818)

* test(e2e): use a non-default region for the differential negative control

Since main's region-aware CreateBucket and GetBucketLocation (#8739, #8743), us-east-1 answers an empty LocationConstraint and 200 for an owned re-create, so the control no longer reported its own region. Also refresh the e2e-full linux selection from the CI listing (682 tests).

* refactor(protocols): move StorageBackend to gateway types (#8825)

* refactor(protocols): move storage backend to gateway types

* test: update e2e test count baseline

* fix: preserve copy source keys and unknown lengths

* test: update protocols test count baseline

* fix(protocols): preserve typed SFTP backend errors

* test(protocols): use gateway list input in policy tests

* fix(protocols): clear clippy failures after T1.5 merge (#8831)

fix(protocols): clear clippy failures after merge

* fix(ecstore): finalize restore metadata after local copy-back (#8836)

* fix(ecstore): finalize restore metadata after local copy-back

* test(ecstore): ratchet added restore finalization cases

* fix(ecstore): keep restore copyback compatible with older peers

* fix(ecstore): gate restore copyback on fleet compatibility

* fix(ecstore): retain transition identity during restore copyback

* fix(ecstore): route committed restore copyback locally

* fix restore copyback cleanup

* fix(ecstore): honor restore probe test fallback

* refactor(storage): migrate option errors to domain types (#8839)

* refactor(s3): migrate private multipart validation errors (#8844)

* refactor(auth): derive policy conditions from AuthzFacts (#8442)

* refactor(auth): extract authorization facts

* test(e2e): update smoke selection baseline

* test(e2e): update full selection baseline

* refactor(admin): migrate health and prefix usage to owned requests (#8887)

* refactor(admin): add owned health router boundary

* refactor(admin): migrate prefix usage to owned request

* refactor(admin): complete the owned admin request boundary (#8949)

* refactor(admin): complete the owned admin request boundary

Every type implementing the owned admin `Operation` now serves the legacy
router through one blanket impl that converts the s3s request and response
at that single edge, so handlers register with `AdminOperation(&Handler {})`
and need no adapter statics; `LegacyAdminAdapter` is removed.

`AdminRequest` gains the accessors handlers read today (`remote_addr`,
`client_info`, `request_context`, `req_info`, `req_info_mut`,
`install_req_info`, `envelope_mut`, `take_body`), the request helpers gain
envelope forms (`authenticate_envelope_credentials`,
`object_store_from_envelope`, the table-catalog `*_from_envelope` helpers,
`KmsAdminAudit::from_envelope`, `AccountAuditContext::from_envelope`,
`CallerIdentity::resolve_envelope`), `read_compatible_admin_body` accepts
either body type, and the owned `Body` gains `store_all_limited` and
`Default`. `admin::operation::test_support::TestRequest` builds owned
requests for handler tests. Unused accessors carry a per-method dead-code
expectation that each migrating handler batch removes.

No handler behavior changes; the remaining migrations follow in batches.

* chore(admin): add the T1.12 admin handler codemod

The script rewrites an admin handler file from the s3s request and response
types to the owned boundary (imports, signatures, request field and helper
calls, body access, response construction, and test request literals) and
leaves a TODO comment at every site it cannot rewrite mechanically. It is
idempotent and is removed once T1.12 completes.

* refactor(admin): migrate replication and KMS handlers to owned requests (#8951)

Move the T1.12 batch 2 admin handlers (site_replication, kms_keys, event,
replication, diagnostics, on_demand_migration, audit, account, kms_rekey,
kms_key_lifecycle, object_data_cache, tls_debug, integrity) from the s3s
request/response types to the owned AdminRequest/AdminResponse boundary.
Handler logic, authorization order, error codes and response bytes are
unchanged; the request/response plumbing now goes through the envelope
accessors and helpers introduced by the owned boundary.

The shared response helpers (utils::json_response, utils::empty_response,
handlers::admin_json_response, target_descriptor::build_json_response,
profile::authorize_profile_request) and the site replication repair
executor still answer with the s3s types, so the migrated files carry
owned-boundary forms of those helpers and convert the repair response at
its call edge. The streaming console-log body and the two devnull drains
move from the s3s byte stream to the owned Body stream/frame API.

Drop the dead_code expectations for the owned accessors and envelope
helpers this batch starts using, and lower the s3s footprint baseline to
the measured 99 files.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* refactor(admin): add owned forms of the shared admin response helpers

Handler batches were copying `json_response`, `empty_response`,
`admin_json_response`, and `build_json_response` into each migrated file
because the shared helpers still return the s3s response. Provide owned
forms next to each legacy helper (`owned_json_response`,
`owned_empty_response`, `owned_admin_json_response`,
`owned_build_json_response`, `authorize_profile_envelope`) with parity
tests against the legacy shape, derive `Debug` for `AdminResponse` so
tests can use `expect_err`, and fix three codemod rules: strip owned
string conversions in test credentials, annotate non-literal URI parses,
and leave a rebound request parameter immutable.

* refactor(admin): use the shared owned response helpers in batch 2 (#8954)

Replace the per-file owned-boundary copies of json_response,
empty_response, admin_json_response, build_json_response and
authorize_profile_request in the batch 2 admin handlers with the shared
owned_* helpers and authorize_profile_envelope that the boundary branch
now provides, and restore expect_err in the handler tests now that
AdminResponse implements Debug; the `.err().expect(..)` stand-ins were
failing the base branch's clippy step (clippy::err_expect under
-D warnings). The site replication repair call edge
keeps its conversion because the repair executor still answers with the
s3s response type.

Drop the dead_code expectations on the shared owned helpers this batch
now uses in non-test code.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* fix(admin): preserve legacy body limit error responses

* refactor(admin): move policy, ILM and batch 3 handlers to owned requests (#8956)

Batch 3 of T1.12 (rustfs/backlog#2788, parent rustfs/backlog#2734) moves
14 admin handler files to AdminRequest/AdminResponse and the owned
Operation trait: ilm_transition, policies, idp_compat, config_admin,
plugins_instances, profile_admin, group, heal, object_zip_download,
kms_key_metadata, durability, inspect_archive, sts and
gateway_key_inventory. Handler logic, authorization order, error codes
and response bytes are unchanged; test request literals use
test_support::TestRequest.

The handlers use the shared owned helpers (owned_json_response,
owned_admin_json_response, owned_build_json_response,
authorize_profile_envelope). The dead_code expectations on the
AdminRequest accessors and RequestEnvelope::uri_mut they now exercise
are removed. The legacy target_descriptor::build_json_response keeps
only its equivalence test caller and carries the matching expectation
until it retires.

object_zip_download builds the s3s request the S3 gate still reads from
the owned request facts; heal authenticates through
authenticate_envelope_credentials; the trace, inspect-archive and zip
bodies use Body::from_stream. extract_plugin_instance_filters takes the
URI so extensions.rs keeps calling it, and the two idp_compat
delegations stay on the legacy trait until their delegates migrate.

The s3s footprint baseline drops from 99 to 90 files.

* test(admin): anchor the replication diff body contract on the call

The source-contract test looked for `let body = read_compatible_admin_body(`
on one line; rustfmt wrapped that binding after the handler moved to the
owned request boundary, so the marker was no longer found. Anchor on the
call itself so formatting cannot change what the contract checks.

* refactor(admin): migrate table catalog and batch 1 to owned requests (#8955)

* fix(admin): preserve catalog body limit response bytes

* refactor(admin): migrate identity and cluster handlers to owned requests (#8966)

* refactor(admin): migrate identity and cluster handlers to owned requests

Move the T1.12 batch 4 admin handlers (user, service_account, oidc, tier,
system, pools, quota, kms_dynamic, kms_backup, rebalance, module_switch,
bucket_meta, cluster_snapshot, realtime, bucket_recovery, profile) from the
s3s request/response types to the owned AdminRequest/AdminResponse
boundary. All 90 impl Operation blocks now take AdminRequest and return
AdminResponse; handler logic, authorization order, error codes and response
bytes are unchanged. The handlers use the shared owned helpers
(owned_json_response, owned_admin_json_response, authorize_profile_envelope,
the envelope-form runtime sources and audit constructors) instead of local
copies.

profile.rs drops the s3s forms of authorize_profile_request and
profile_not_implemented_response now that profile_admin.rs is on the
envelope gate too; profile_not_implemented_response answers with an
AdminResponse. The realtime metrics stream moves from the s3s ByteStream
adapter to Body::from_stream, and the rebalance idle-404 document still
renders through s3s::S3Error::to_http_response with its body converted at
the edge.

The idp_compat delegators ImportIamV2 and AddServiceAccountLdap move to the
owned trait as well: with their targets on both traits, the method-call
delegation became ambiguous between the owned impl and the router blanket
impl, so they now take the owned request directly.

KmsAdminAudit::from_request loses its last production caller with this batch
and is deleted; its comparison test now asserts the envelope facts directly.

Lower the s3s footprint baseline to the measured 68 files.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* refactor(admin): remove the legacy admin helpers batch 4 superseded

Batch 4 moved the last production callers of the s3s-typed admin helpers
to their owned forms, which left `admin_json_response`, `json_response`,
`authenticate_request`, `authorize_admin_request`, `object_store_from_req`
and `object_store_from_extensions` unused, along with the `Body`,
`S3Request` and `S3Response` re-exports of the admin `s3` facade. Remove
them. The owned forms keep their documentation; the behavior tests of the
removed `json_response` now exercise `owned_json_response`, and the
compat-prefix test asserts the owned response directly.

Two tests existed only to compare a removed legacy helper with its owned
twin (`owned_json_response_matches_the_legacy_shape`) or to cover the
removed legacy gate (`authorize_admin_request_without_credentials_is_rejected`);
they go with the helpers, and the owned equivalents stay. The rustfs
test-count row drops from 5957 to 5955 and the s3s footprint from 68 to 66
files.

* refactor(admin): drop the legacy target descriptor JSON response helper (#8968)

`target_descriptor::build_json_response` returned the s3s response and
had no production caller left once the event, audit and plugin instance
handlers moved to `owned_build_json_response`; it survived only for the
legacy/owned equivalence test. Remove it, give the owned helper its own
documentation, and make the test assert the owned response directly:
status, `application/json`, the echoed `x-request-id` when present and
only then, and the body bytes. The file no longer imports s3s, so the
footprint baseline drops from 66 to 65 files.

* test(ecstore): pin native history on cancellation under the cancellable queue

main moved the local write path onto the disk blocking queue, which skips an
operation once its waiter is dropped, so a borrowed write cancelled while it is
queued no longer creates its file. The integration-only native publication
test still expected the queued open to run. Assert the new execution model
instead (the open never runs) while keeping its safety claims: the Started
record is taken before queueing and the cancelled publication never regrants
native history.

* chore(gateway): raise test-count rows for tests merged from main

* test(s3select): use the owned JSON input types in merged planner tests

* fix(ecstore): scope the merged direct-writer test hook to test builds

* fix(ecstore): create parents before admitted truncate opens

Align the heal tests with removal-disabled preservation and local-only outcome
fields, and refresh the e2e-full selection for the synced tests.

* chore(gateway): update test-count baseline after main sync

Main #9241 removed the node pool-metadata readiness cache with 8 tests
and added 6, so the rustfs row drops from 6440 to 6438. The ecstore and
s3select rows rise for tests added by #9237, #9239 and #9241.

* feat(scripts): add junit_diff.py for dual-stack JUnit comparison (#9275)

* refactor(auth): share authorization core with owned entry points

Extract the policy decision behind authorize_request into a core that both the legacy s3s request and the RustFS-owned RequestEnvelope feed, and move the zip-create and table-catalog admin callers to the owned entry points (authorize_owned_request, authorize_owned_internal_object_request). Request-body policy inputs added on main since the candidate base (POST form headers, copy source, legal hold and tagging bodies) become named envelope fields so both edges reach the same decision. Refs rustfs/backlog#2749.

* ci(gateway): make the test-count ratchet a floor, not exact (#9278)

ci(gateway): make the test-count ratchet a floor instead of an exact baseline

Tests below a row still fail; tests above it, a member without a row and ignores below the row now pass with a note. Ignores above a row still fail. Adds contract tests wired into make script-tests.

* chore(gateway): record ignored microbenchmarks merged from main (#9290)

chore(gateway): record the ignored microbenchmarks merged from main

main added three throughput microbenchmarks that are ignored by design
(#9217): one each in rustfs-checksums, rustfs-rio and rustfs-utils. The
test-count ratchet treats a higher ignored count as a failure until the
row is raised, so raise those three rows to 1 and tighten the test rows
to the merged tree with --update.

* fix(auth): keep owned authorization in the app layer and fail closed

Move the RequestEnvelope impl of AuthorizationRequest and the owned entry points into the app layer so storage depends only on the request-facts trait (layer guard). The owned entry points now hoist signed x-amz-meta-* query parameters like the legacy edge and refuse actions whose decision needs a request-body input the envelope does not carry. Revert unrelated TODO rewording and redact PostObjectPolicyHeaders debug output.

* test(auth): move owned parity tests to the app layer

Owned-versus-legacy parity, hoist and fail-closed tests now live in app/owned_authorization_test.rs beside the envelope's facts impl, so storage names no app type in test code either. Private storage steps are reached through a test-only parity_test_support module. Also fix the S3Error conversion CI reported in prepare_owned_authorization.

* ci(s3-tests): add reference parity lane against pinned 1.0.5 (#9288)

* test(auth): read storage-private parity facts through test accessors

The parity tests moved to the app layer can no longer read ObjectTagConditions and TableDataPlanePublicationGuards fields directly; parity_test_support exposes read-only accessors instead. Drop an unused import.

* test(auth): owned and legacy entry points reach the same decisions

Decision-level parity over IAM allow and explicit deny, owner, anonymous on private and public buckets, bucket-policy explicit deny, aws:SourceIp from the peer and from a verified client, and one allow and one deny per body-derived input (legal hold, tagging, retention mode and retain-until, resolved copy source, POST form fields).

* test(auth): satisfy clippy on the parity test facade

* test(auth): report every decision mismatch in the parity matrix

* test(auth): cover multipart staging in the owned/legacy decision matrix

* fix(table-catalog): restore owned request lines the main merge reverted

The merge onto main replaced two lines with main's text: the warehouse location check again wrapped an already owned request in legacy_request_view, and an error conversion was re-added. Restore the branch's version of the file.

* fix(auth): make an absent caller secret unrepresentable as a key

RequestCredentials::secret_key now returns Option<&str>: None when the edge kept the secret, and an empty handed-over secret is refused at construction. Secret comparisons go through secret_key_matches, which never matches an absent secret, and readers that need the value fail closed with an internal error.

* fix(auth): keep the test-only credentials rebuild out of the lib build

* fix(app): drop the PutObjectInput re-export main now provides

---------

Co-authored-by: rustfs-cla-assistant[bot] <273121464+rustfs-cla-assistant[bot]@users.noreply.github.com>
2026-11-11 09:43:29 -05:00
Chris An 46594e2dfd test(e2e): pinned release binary as differential reference side (#9285)
* feat(server): add the RUSTFS_S3_STACK selector and dual-stack e2e lane (#8411)

* refactor(server): group the s3s-compat tower layers under legacy_compat (#8410)

Move the ten tower layers that patch s3s behaviour out of
rustfs/src/server/layer.rs into rustfs/src/server/legacy_compat/, one file
per layer with its tests. Each file header names the s3s behaviour it
patches, the gateway mechanism that replaces it (Replaced by:), and the task
that removes it (Removed by: T2.12, rustfs/backlog#2771). The layers keep
their code, their order in external_service_stack!, and their behaviour; no
layer is deleted.

Add server::http::tests::stack_census, which renders the ordered service
chain of the real external_service_stack! expansion and compares it with a
committed constant, so a reorder, insertion or removal is a visible diff.

Route the two s3s error types the moved layers use through a
storage_api::server::legacy_compat shim so the split adds no s3s-importing
file, and lower the s3s footprint baselines to the measured values (files 209
to 208, s3_error! lines 1586 to 1585). Keep the server runtime-source bypass
guard covering the moved code.

Refs rustfs/backlog#2787, rustfs/backlog#2734

* feat(s3-types): add the RustFS-owned S3 error type and s3_error! macro (#8406)

`rustfs-s3-types` now provides `S3ErrorCode` (generated by build.rs from
src/codes.txt, the grep-derived call surface, plus `Custom`), `S3Error`,
`S3Result`, `StdError` and the `s3_error` macro with the same call syntax
as s3s, so the T1.2 codemod is a `use` swap. Statuses are hand-written and
pinned to the s3s behaviour RustFS ships today by an oracle test that uses
s3s as a dev-dependency only. The `From` conversions for the legacy edge
live behind the off-by-default `compat-s3s` feature (the orphan rule
forbids them in rustfs/src/error.rs); `rustfs` enables it.

Refs rustfs/backlog#2742, rustfs/backlog#2734

* ci: add the per-crate test-count ratchet (#8404)

Quick Checks now fails any PR that lowers the number of test attributes
in a workspace crate or adds #[ignore], measured per crate against the
exact rows of .config/test-count-baseline.txt (rustfs/backlog#2734
section 1 rule 3). The counter widens the spec's literal
#[(tokio::)?test] to the tokio attributes that carry arguments and both
spellings of #[test_case(...)], and anchors at line start so
commented-out attributes do not count. The initial baseline covers 52
crates, 21077 tests and 164 ignored.

Refs rustfs/backlog#2738
Refs rustfs/backlog#2734

* test(ecstore): pin the persisted bucket-config XML bytes for 13 families (#8408)

Before any DTO or codec is replaced, pin the exact bytes that
deserialize followed by serialize produces for every persisted
bucket-configuration XML family in its three source shapes (AWS
documentation, MinIO-written, RustFS-written), and the bytes RustFS
writes today for a typed value. Byte equality only: a structural
comparison would let a codec replacement change the persisted encoding
unnoticed.

The generator script refuses to overwrite existing goldens without
--force, refuses on dirty codec paths, cannot report success from its
write mode, and records the generating commit in the fixtures README.

Refs rustfs/backlog#2744, rustfs/backlog#2734

* ci: raise the test-count baseline to the integration tree (#8426)

* ci: make the s3s footprint baseline exact and lower-only (#8403)

* ci: make the s3s footprint baseline exact and lower-only

The s3s footprint guard compared three counters against ceilings held in
the script itself, so a ceiling could sit above the tree and a drift was
fixable by raising it (rustfs/rustfs#5964 raised the s3_error! ceiling
to go green). The three values now live in
.config/s3s-footprint-baseline.txt as files, s3_error_lines and
ecstore_files, the script reads only that file, and a measurement that
differs from its baseline in either direction fails: growth prints
"s3s footprint grew: <measured> > <baseline>", shrinkage prints
"lower the baseline to <measured> in this PR". The counting method is
unchanged; the baseline is the current measurement of
gateway/integration (209 / 1585 / 36). The header records the end state:
once all three reach 0, T4.2 switches the script to failing on any
occurrence.

Refs rustfs/backlog#2739, rustfs/backlog#2734

* ci: add the s3s edge allowlist mode to the footprint guard (#8405)

`scripts/check_s3s_footprint.sh --mode allowlist` fails on every Rust
file that references s3s (`s3s::` or `s3s_sigv4`) and is not matched by
an entry of the new `.config/s3s-edge-allowlist.txt`; `--dry-run` lists
those files and exits 0. The allowlist carries the initial edge from the
spec, and the script rejects an entry that names or matches anything
under crates/ other than crates/e2e_test. The baseline mode and the
argument-less CI invocation are unchanged.

Refs rustfs/backlog#2741, rustfs/backlog#2734

* test(ecstore): reach the bucket-config codec through the test facade (#8428)

* refactor(s3-types): own the HTTP Body and drop s3s from three crates (#8431)

Add rustfs_s3_types::Body, a BoxBody<Bytes, StdError> wrapper with
empty() and From<Bytes | Vec<u8> | String | &'static str>, and the
legacy From conversions to and from s3s::Body behind compat-s3s.

The signer is now generic over the request body: it only reads and
writes the request head, so it needs no body type and its callers keep
theirs. s3-client sends its requests with the owned Body, madmin signs a
bodiless head, and targets drops its two S3Request/S3Response header
wrappers in favour of extract_params_header.

rustfs-signer, rustfs-madmin and rustfs-targets no longer depend on
s3s; the s3s footprint baseline drops from 209 to 202 files.

Refs rustfs/backlog#2747, rustfs/backlog#2734

* refactor(ecstore): move S3ErrorCode to rustfs-s3-types by codemod (#8432)

* test(ecstore): pin the InvalidRange responses of new_getobjectreader

Both error branches of new_getobjectreader build an ErrorResponse with S3ErrorCode::InvalidRange, and no test observed either code. Pin the code, its wire name and the location fields of each branch before the code type moves to rustfs-s3-types (rustfs/backlog#2743), so a changed code on either branch fails a test. The ecstore test-count row rises by the two tests.

* chore(scripts): add the s3s-to-rustfs-s3-types error codemod

Task T1.2 of rustfs/backlog#2734 (rustfs/backlog#2743) moves S3Error, S3ErrorCode, S3Result and s3_error! from s3s to rustfs-s3-types by script, one crate closure per PR. The codemod rewrites use trees (splitting mixed s3s imports) and qualified code paths, never comments or literals, adds the rustfs-s3-types dependency, drops s3s only from a crate that no longer names it, refuses every file of the s3s edge allowlist, takes --hold for a file whose type is fixed outside the closure, and prints a per-crate census. A second run changes nothing. Its self-test runs under make script-tests.

* refactor(ecstore): take S3ErrorCode from rustfs-s3-types

Output of scripts/codemods/s3_error_to_s3_types.py over crates/ecstore, crates/lifecycle, crates/scanner and crates/replication (rustfs/backlog#2743 batches 1 and 2). The ErrorResponse code field, the two InvalidRange constructions in new_getobjectreader and an unused tier import now use rustfs_s3_types::S3ErrorCode; the variant names and wire names are unchanged. lifecycle, scanner and replication reference none of the four error names. warm_backend.rs and warm_backend_s3.rs are held: they compare against rustfs_s3_client::ErrorResponse.code, which stays the s3s type until s3-client moves.

* ci(footprint): count only s3_error! lines still on s3s's macro

s3_error_lines counted every line holding the text `s3_error!`, which
also matches the RustFS-owned `rustfs_s3_types::s3_error!`. A codemod
that moves call sites onto rustfs-s3-types therefore could not lower it
(rustfs/backlog#2743, decision 3; rustfs/rustfs#8432, gotcha 2).

The counter now applies a textual, per-file and per-line rule that errs
on the side of counting (full rule in the script header):
- occurrences spelled `rustfs_s3_types::s3_error!` never count;
- a line holding `s3s::s3_error!` always counts;
- any other `s3_error!` line counts unless its file is cleared, that is,
  it imports the macro from rustfs_s3_types by name on a line starting
  with the use item (or defines it with `macro_rules! s3_error`) and does
  not also bring s3s's macro in (`use s3s::s3_error`, a `use s3s::{..}`
  tree naming it or a crate-root `*`, `use s3s::*`, or
  `#[macro_use] extern crate s3s`);
- fail closed: a file matching neither form (`use super::*`, a
  crate-local re-export, a renamed or glob rustfs_s3_types import, a
  commented-out import, a comment-only mention) counts.

Census of the 1585 lines on 96fc06776: 1478 in 64 files that import the
macro from s3s; 107 in 15 files matching neither form (105 through
`use super::*`, 1 qualified `s3s::s3_error!`, 1 comment line); none in
files that import it from rustfs_s3_types or define it. Every line stays
counted, so the baseline remains s3_error_lines=1585 (files=202 and
ecstore_files=36 unchanged). Rewriting rustfs/src/auth.rs's import to
rustfs_s3_types lowers the new counter to 1567; the old one stayed 1585.

`--self-test` (wired into `make script-tests`) copies the script into
22 fixture trees: 6 positive cases and 16 negative ones (lines that must
not count, forms that must not clear a file, and a missing baseline,
missing key or failing classification search that must fail). Each tree
has a control line that must count, and each run gets s3s text on stdin.
Seven cases were red against the old counter. 19 mutants each turn their
case red: old definition, flipped import test, dropped fail-closed
branch, dropped e2e_test exclusion, dropped s3s veto, dropped qualified
s3s rule, dropped rustfs-qualified strip, unanchored rustfs import,
renamed import accepted, nested glob taken as root glob, single-line
s3s trees only, dropped s3s glob forms, dropped macro_use form, dropped
definition clearing, ignored classification rg errors, error-line search
without its path, classification searches without their path, missing
baseline skipped, missing key skipped.

* test(e2e): compare two RUSTFS_S3_STACK servers over every S3 operation (#8435)

Add dual_stack_differential to e2e_test. It starts two rustfs processes
from the same binary, stack A on RUSTFS_S3_STACK=legacy and stack B on
RUSTFS_E2E_STACK_B (default legacy until rustfs/backlog#2734 T2.5),
sends both the same aws-sdk-s3 script covering the 73 methods of
impl S3 for FS (a normal request and a 4xx negative each), and compares
status, error code, an allowlist of response headers and a few typed
body fields. Error message text is never compared. Each run writes
<target>/dual-stack/<run>.json with compared_ops and the (op, field,
a, b) differences; any difference fails the test.

differential_detects_an_injected_difference starts B with
RUSTFS_REGION=eu-west-1 and requires GetBucketLocation to be reported.
The comparator is pure and unit-tested without a server.

Wire the module into the e2e-smoke filter and update the e2e-smoke and
e2e-full selection digests and the e2e_test test-count row.

Refs rustfs/backlog#2740

* feat(app): add the RustFS-owned RequestEnvelope and its s3s constructor (#8437)

* feat(app): add the RustFS-owned RequestEnvelope and its s3s constructor

* ci: classify the s3s edge adapters in storage/s3_compat as interface

* feat(s3-types): add Body::from_stream and move Swift off s3s::Body (#8438)

rustfs/backlog#2747 batch 3. `Body::from_stream` wraps a chunk stream of
unknown length over `StreamBody`/`BoxBody`: one data frame per chunk, the
first error ends the body, the size hint stays unknown, and dropping the
body drops the stream. Parity tests pin it to the s3s `StreamingBlob::wrap`
it replaces, at the `http_body` level and byte for byte through hyper.

The Swift handlers in rustfs-protocols build their responses as the owned
`Body`. `SwiftService` converts them into the wrapped S3 service's response
body type through `From<Body>`, so rustfs keeps serving `s3s::Body` at the
legacy edge unchanged. Request bodies stay Send-only (`UnsyncBoxBody`); the
two storage writers and the DLO/SLO segment streams get `Sync` from
`sync_wrapper::SyncStream`, which is what s3s's `http_body_unsync` mutex
provided before. The DTO and error uses of s3s in protocols are untouched
(T1.5 and T1.2).

* refactor(protocols): move error types to rustfs-s3-types by codemod (#8439)

T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and
the error items of protocols move off s3s by scripts/codemods, run with
--hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it
downcasts the backend error, whose runtime type is still s3s::S3Error.

- rustfs/src/protocols/client.rs: the two webdav session methods now
  return rustfs_s3_types::S3Result and convert the s3s error with From.
- codemod: a child module that calls s3_error! through `use super::*`
  gets its own `use rustfs_s3_types::s3_error;` once its parent imports
  that macro by name; otherwise it is listed as GLOB.
- s3-types: a table test pins code, status and the s3s error the legacy
  edge renders for every variant a moved call site names.
- s3_error_lines 1585 -> 1584 (a lifecycle comment reworded).

* test(connect): restore one-second drive request budgets

Apply the exact test-only repair already landed on main in 5972a09cc3. The filesystem probe and production CLI success test request one second and keep their existing one-second duration and parameter checks. All outcome, I/O, error, cleanup and resource-limit assertions remain.

Scope: rustfs/tests/connect_perf_drive.rs only. Patch-id: cb0a1dfc8f68a67727525adea7458e2cee7d09f8.

Validation environment: LANG=en_US.UTF-8, CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0, exclusive APFS-cloned CARGO_TARGET_DIR.

Quick Checks: every command from .github/actions/quick-checks/action.yml, with captured output and exit status below.

$ shellcheck --version && actionlint
ShellCheck - shell script analysis tool
version: 0.11.0
license: GNU General Public License, version 3
website: https://www.shellcheck.net
exit_code=0

$ cargo fmt --all --check
exit_code=0

$ ./scripts/check_unsafe_code_allowances.sh
exit_code=0

$ ./scripts/check_layer_dependencies.sh
Layer dependency guard passed (no new reverse dependencies/cycles).
exit_code=0

$ ./scripts/check_architecture_migration_rules.sh
Architecture migration rules passed.
exit_code=0

$ ./scripts/check_logging_guardrails.sh
✅ Logging guardrails check passed
exit_code=0

$ ./scripts/check_error_other_format_ratchet.sh
error(format!) ratchet passed (total call sites: 582).
exit_code=0

$ ./scripts/check_no_tokio_io_uring.sh
exit_code=0

$ ./scripts/check_extension_schema_boundaries.sh
Extension schema boundaries passed.
exit_code=0

$ ./scripts/check_body_cache_whitelist.sh
Body-cache whitelist guard passed.
exit_code=0

$ ./scripts/check_s3s_footprint.sh
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)
✅ s3s footprint baseline check passed
exit_code=0

$ ./scripts/check_test_count_ratchet.sh
test-count ratchet OK: 52 crates, 21222 tests, 164 ignored match .config/test-count-baseline.txt
exit_code=0

$ ./scripts/check_fips_wording.sh
FIPS wording guard passed (README.md and CHANGELOG.md contain no forbidden claims).
KMS wording guard passed (crates/kms claims no Vault KV2 Transit wrapping).
exit_code=0

$ ./scripts/check_embedded_secrets.sh
Embedded secret guard passed (no private key material or provider credential literal in the tree).
exit_code=0

$ make script-tests
Running script tests...
entrypoint credential policy tests passed
internode grpc A/B bench env tests passed
object batch benchmark enhanced tests passed
PASS package service script contract
hotpath warp A/B gate tests passed
hotpath warp ABBA tests passed
Scanner validation artifacts written to /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.1pFe6Kl1Yu/out
Scanner validation artifacts written to /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.1pFe6Kl1Yu/out-missing-pid
.............................
----------------------------------------------------------------------
Ran 29 tests in 181.286s

OK
{"decision":"verified","pending_gates":["G01","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G02"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-E"}
{"decision":"verified","pending_gates":["G01","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G02"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-E"}
{"decision":"verified","pending_gates":["G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G01"}
{"decision":"verified","pending_gates":["G01","G02","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G03"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-L"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G14"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G14"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G10"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P1"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P2"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P3"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G05"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G06"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-D"}
Status-and-outcome release descriptor verified: /private/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp8rwn1pe0/descriptor/release-bundle-status-outcome.json
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G05"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G06"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-D"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G07"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G08"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P4"}
exact 1MiB handoff ABBA tests passed
pinned paired ABBA bench tests passed
==> /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/bin/path_containment (-max_total_time=7, -seed=123456789, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/explicit-artifacts/path_containment/, corpus=/private/tmp/gw-orchestrator-2734/ci-drive/fuzz/corpus/path_containment)
-seed=123456789
target=path_containment
seed=123456789
max_total_time=7
git_revision=4d15b816236630b2c42f321ffc9ed3654edc94e3
git_dirty=true
runner_mode=prebuilt
==> /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/bin/path_containment (-max_total_time=1, -seed=2322617706, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/auto-artifacts/path_containment/, corpus=/private/tmp/gw-orchestrator-2734/ci-drive/fuzz/corpus/path_containment)
-seed=2322617706
==> cargo +nightly fuzz run path_containment (-max_total_time=9, -seed=987654321, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/cargo-artifacts/path_containment/)
+nightly
fuzz
run
path_containment
-seed=987654321
seed=987654321
runner_mode=cargo-fuzz
fuzz runner seed manifest tests passed
✅ scripts/python_bin.sh resolver checks passed
self-test ok: clean fixture with every exemption present passes
self-test ok: all 11 pattern families fire
self-test ok: moving a pinned source is reported
Embedded secret guard self-test passed.
self-test ok: p1-s3s-import: 'use s3s::s3_error;' counts every line, the rustfs_s3_types module's too (s3_error_lines=3)
self-test ok: p2-s3s-mixed-tree: a multi-line s3s use tree naming s3_error counts (s3_error_lines=3)
self-test ok: p3-s3s-glob: a crate-root glob of s3s, bare or inside a tree, counts (s3_error_lines=5)
self-test ok: p4-macro-use: '#[macro_use] extern crate s3s' counts (s3_error_lines=3)
self-test ok: p5-qualified-s3s: a qualified s3s::s3_error! line counts in a cleared file (s3_error_lines=2)
self-test ok: p6-exact-pass: a baseline equal to the tree passes (exit 0)
self-test ok: n1-rustfs-import: 'use rustfs_s3_types::s3_error;' alone counts nothing (s3_error_lines=1)
self-test ok: n2-rustfs-mixed-tree: a rustfs_s3_types tree plus an s3s tree without the macro counts nothing (s3_error_lines=1)
self-test ok: n3-defines: the file defining the macro counts nothing (s3_error_lines=1)
self-test ok: n4-qualified-rustfs: a qualified rustfs_s3_types::s3_error! line counts nothing (s3_error_lines=1)
self-test ok: n5-renamed-call: a call through a renamed macro is not text-matched (out of scope) (s3_error_lines=1)
self-test ok: n6-comment-string: a comment or string mentioning s3_error! in a cleared file counts nothing (s3_error_lines=1)
self-test ok: n7-e2e-excluded: crates/e2e_test is excluded (s3_error_lines=1)
self-test ok: n8-super-glob: an unrecognised import form ('use super::*;') counts (s3_error_lines=3)
self-test ok: n9-facade: a crate-local re-export counts (s3_error_lines=2)
self-test ok: n10-renamed-import: a renamed rustfs_s3_types import does not clear s3_error! lines (s3_error_lines=2)
self-test ok: n11-commented-import: a commented-out rustfs_s3_types import does not clear (s3_error_lines=2)
self-test ok: n12-rustfs-glob: a glob import of rustfs_s3_types does not clear (s3_error_lines=2)
self-test ok: n13-comment-only: a comment mentioning s3_error! in a file with no import counts (s3_error_lines=2)
self-test ok: f1-baseline-missing: a missing baseline file fails (exit 1)
self-test ok: f2-key-missing: a baseline without s3_error_lines fails (exit 1)
self-test ok: f3-rg-fails: a failing classification search fails the guard (exit 1)
✅ s3s footprint self-test passed
test_core_gate_rejects_missing_ignored_filtered_and_corrupt_inputs (__main__.SelfTests.test_core_gate_rejects_missing_ignored_filtered_and_corrupt_inputs) ... ok
test_e2e_requires_registration (__main__.SelfTests.test_e2e_requires_registration) ... ok
test_fuzz_runtime_uses_matrix_target (__main__.SelfTests.test_fuzz_runtime_uses_matrix_target) ... ok
test_ilm_lane_keeps_the_measured_cargo_build_budget (__main__.SelfTests.test_ilm_lane_keeps_the_measured_cargo_build_budget) ... ok
test_profile_listing_binds_platform_digest (__main__.SelfTests.test_profile_listing_binds_platform_digest) ... ok
test_profile_listing_enforces_selection (__main__.SelfTests.test_profile_listing_enforces_selection) ... ok
test_quick_checks_commands_propagate_failure (__main__.SelfTests.test_quick_checks_commands_propagate_failure) ... ok
test_quick_checks_rejects_caller_and_execution_bypasses (__main__.SelfTests.test_quick_checks_rejects_caller_and_execution_bypasses) ... ok
test_runner_readiness_fails_closed (__main__.SelfTests.test_runner_readiness_fails_closed) ... ok
test_s3_runner_rejects_unbounded_failure_locals (__main__.SelfTests.test_s3_runner_rejects_unbounded_failure_locals) ... ok
test_scanner_heal_begin_requires_embedded_source_provenance (__main__.SelfTests.test_scanner_heal_begin_requires_embedded_source_provenance) ... ok
test_scanner_heal_case_does_not_approve_pending_release (__main__.SelfTests.test_scanner_heal_case_does_not_approve_pending_release) ... ok
test_scanner_heal_case_lane_is_required_for_runner_profile (__main__.SelfTests.test_scanner_heal_case_lane_is_required_for_runner_profile) ... ok
test_scanner_heal_case_only_schema_cannot_approve_release (__main__.SelfTests.test_scanner_heal_case_only_schema_cannot_approve_release) ... ok
test_scanner_heal_crash_case_rejects_restart_oracle (__main__.SelfTests.test_scanner_heal_crash_case_rejects_restart_oracle) ... ok
test_scanner_heal_ec84_case_rejects_wrong_erasure_geometry (__main__.SelfTests.test_scanner_heal_ec84_case_rejects_wrong_erasure_geometry) ... ok
test_scanner_heal_finish_collects_oracles_from_registry (__main__.SelfTests.test_scanner_heal_finish_collects_oracles_from_registry) ... ok
test_scanner_heal_multipool_case_accepts_deferred_optional_outage_write (__main__.SelfTests.test_scanner_heal_multipool_case_accepts_deferred_optional_outage_write) ... ok
test_scanner_heal_pending_gate_cannot_map_to_implemented_lane (__main__.SelfTests.test_scanner_heal_pending_gate_cannot_map_to_implemented_lane) ... ok
test_scanner_heal_receipts_reject_reuse_and_missing_builds (__main__.SelfTests.test_scanner_heal_receipts_reject_reuse_and_missing_builds) ... ok
test_scanner_heal_rejects_boolean_fractional_and_out_of_geometry_integers (__main__.SelfTests.test_scanner_heal_rejects_boolean_fractional_and_out_of_geometry_integers) ... ok
test_scanner_heal_rejects_broken_execution_and_artifacts (__main__.SelfTests.test_scanner_heal_rejects_broken_execution_and_artifacts) ... ok
test_scanner_heal_rejects_copied_junit_and_wrong_suite_build (__main__.SelfTests.test_scanner_heal_rejects_copied_junit_and_wrong_suite_build) ... ok
test_scanner_heal_release_bundle_accepts_complete_measured_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_accepts_complete_measured_evidence) ... ok
test_scanner_heal_release_bundle_assembler_copies_measured_artifacts (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_copies_measured_artifacts) ... ok
test_scanner_heal_release_bundle_assembler_merges_measured_descriptors (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_merges_measured_descriptors) ... ok
test_scanner_heal_release_bundle_assembler_rejects_duplicate_descriptor_gate (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_duplicate_descriptor_gate) ... ok
test_scanner_heal_release_bundle_assembler_rejects_fixture_descriptor (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_fixture_descriptor) ... ok
test_scanner_heal_release_bundle_assembler_rejects_hash_mismatch (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_hash_mismatch) ... ok
test_scanner_heal_release_bundle_assembler_rejects_mixed_descriptor_revisions (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_mixed_descriptor_revisions) ... ok
test_scanner_heal_release_bundle_enforces_topology_duration_profile_and_versions (__main__.SelfTests.test_scanner_heal_release_bundle_enforces_topology_duration_profile_and_versions) ... ok
test_scanner_heal_release_bundle_fixture_writer_is_not_release_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_fixture_writer_is_not_release_evidence) ... ok
test_scanner_heal_release_bundle_gate_accepts_single_measured_gate (__main__.SelfTests.test_scanner_heal_release_bundle_gate_accepts_single_measured_gate) ... ok
test_scanner_heal_release_bundle_gate_rejects_missing_field (__main__.SelfTests.test_scanner_heal_release_bundle_gate_rejects_missing_field) ... ok
test_scanner_heal_release_bundle_gate_rejects_unknown_gate (__main__.SelfTests.test_scanner_heal_release_bundle_gate_rejects_unknown_gate) ... ok
test_scanner_heal_release_bundle_rejects_synthetic_or_missing_fields (__main__.SelfTests.test_scanner_heal_release_bundle_rejects_synthetic_or_missing_fields) ... ok
test_scanner_heal_release_bundle_requires_domain_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_requires_domain_evidence) ... ok
test_scanner_heal_release_bundle_requires_field_provenance (__main__.SelfTests.test_scanner_heal_release_bundle_requires_field_provenance) ... ok
test_scanner_heal_release_bundle_requires_json_artifact_provenance (__main__.SelfTests.test_scanner_heal_release_bundle_requires_json_artifact_provenance) ... ok
test_scanner_heal_release_bundle_requires_mixed_version_field_roles (__main__.SelfTests.test_scanner_heal_release_bundle_requires_mixed_version_field_roles) ... ok
test_scanner_heal_release_bundle_requires_same_gate_measurement_window (__main__.SelfTests.test_scanner_heal_release_bundle_requires_same_gate_measurement_window) ... ok
test_scanner_heal_release_consumes_multiple_registry_oracles (__main__.SelfTests.test_scanner_heal_release_consumes_multiple_registry_oracles) ... ok
test_scanner_heal_release_matrix_lanes_remain_pending_without_real_evidence (__main__.SelfTests.test_scanner_heal_release_matrix_lanes_remain_pending_without_real_evidence) ... ok
test_scanner_heal_release_requirements_cannot_be_cleared (__main__.SelfTests.test_scanner_heal_release_requirements_cannot_be_cleared) ... ok
test_scanner_heal_release_status_rejects_focused_case_run (__main__.SelfTests.test_scanner_heal_release_status_rejects_focused_case_run) ... ok
test_scanner_heal_release_status_rejects_synthetic_case (__main__.SelfTests.test_scanner_heal_release_status_rejects_synthetic_case) ... ok
test_scanner_heal_required_evidence_fields_cannot_be_removed (__main__.SelfTests.test_scanner_heal_required_evidence_fields_cannot_be_removed) ... ok
test_scheduled_alerts_require_completion_watchdog (__main__.SelfTests.test_scheduled_alerts_require_completion_watchdog) ... ok
test_update_profile_selection_changes_only_requested_platform (__main__.SelfTests.test_update_profile_selection_changes_only_requested_platform) ... ok
test_validate_includes_quick_checks (__main__.SelfTests.test_validate_includes_quick_checks) ... ok
test_vault_tests_require_cross_process_serialization (__main__.SelfTests.test_vault_tests_require_cross_process_serialization) ... ok
test_workflow_readiness_requires_dependency_probe (__main__.SelfTests.test_workflow_readiness_requires_dependency_probe) ... ok

----------------------------------------------------------------------
Ran 52 tests in 14.301s

OK
e2e-full selection OK: 1 tests, sha256=29e30e6ed2ed38d9e5b0a7cdd9536abf5c719f7021a02b578ae1381563a3c273
...................................
----------------------------------------------------------------------
Ran 35 tests in 1.161s

OK
...................
----------------------------------------------------------------------
Ran 19 tests in 17.397s

OK
..........
----------------------------------------------------------------------
Ran 10 tests in 0.183s

OK
test_actual_selector_bootstrap_uses_base_policy_and_fails_closed (__main__.SelfTests.test_actual_selector_bootstrap_uses_base_policy_and_fails_closed) ... ok
test_documentation_paths_do_not_hide_build_or_fixture_changes (__main__.SelfTests.test_documentation_paths_do_not_hide_build_or_fixture_changes) ... ok
test_e2e_scope_keeps_production_and_shared_configuration_full (__main__.SelfTests.test_e2e_scope_keeps_production_and_shared_configuration_full) ... ok
test_e2e_shortcut_rejects_new_direct_renamed_and_target_dependencies (__main__.SelfTests.test_e2e_shortcut_rejects_new_direct_renamed_and_target_dependencies) ... ok
test_e2e_shortcut_traverses_implicit_path_dependencies (__main__.SelfTests.test_e2e_shortcut_traverses_implicit_path_dependencies) ... ok
test_event_contract_requires_complete_candidate_and_optional_lanes (__main__.SelfTests.test_event_contract_requires_complete_candidate_and_optional_lanes) ... ok
test_every_wrong_result_missing_job_or_selection_fails_closed (__main__.SelfTests.test_every_wrong_result_missing_job_or_selection_fails_closed) ... ok
test_full_e2e_gate_preserves_workflow_branch_and_event_scope (__main__.SelfTests.test_full_e2e_gate_preserves_workflow_branch_and_event_scope) ... ok
test_git_range_includes_deleted_source_and_rename_origins (__main__.SelfTests.test_git_range_includes_deleted_source_and_rename_origins) ... ok
test_job_ids_and_display_names_cannot_hide_validation (__main__.SelfTests.test_job_ids_and_display_names_cannot_hide_validation) ... ok
test_report_only_stack_lane_mirrors_e2e_smoke_outside_the_aggregate (__main__.SelfTests.test_report_only_stack_lane_mirrors_e2e_smoke_outside_the_aggregate) ... ok
test_repository_wiring_and_missing_dependency_regression (__main__.SelfTests.test_repository_wiring_and_missing_dependency_regression) ... ok
test_verify_command_preserves_failures (__main__.SelfTests.test_verify_command_preserves_failures) ... ok

----------------------------------------------------------------------
Ran 13 tests in 3.280s

OK
security coverage self-test passed
test_api_failures_preserve_other_evidence_and_never_enter_grace (__main__.SelfTests.test_api_failures_preserve_other_evidence_and_never_enter_grace) ... ok
test_attempt_outcome_does_not_replace_recent_success (__main__.SelfTests.test_attempt_outcome_does_not_replace_recent_success) ... ok
test_cancelled_attempt_cannot_refresh_expired_success (__main__.SelfTests.test_cancelled_attempt_cannot_refresh_expired_success) ... ok
test_cli_requires_the_repository_default_branch (__main__.SelfTests.test_cli_requires_the_repository_default_branch) ... ok
test_config_rejects_duplicate_and_invalid_entries (__main__.SelfTests.test_config_rejects_duplicate_and_invalid_entries) ... ok
test_freshness_boundaries (__main__.SelfTests.test_freshness_boundaries) ... ok
test_grace_requires_two_successful_queries_with_no_history (__main__.SelfTests.test_grace_requires_two_successful_queries_with_no_history) ... ok
test_invalid_api_evidence_fails_closed (__main__.SelfTests.test_invalid_api_evidence_fails_closed) ... ok
test_report_retains_every_workflow (__main__.SelfTests.test_report_retains_every_workflow) ... ok
test_requests_filter_schedule_default_branch_and_success_on_server (__main__.SelfTests.test_requests_filter_schedule_default_branch_and_success_on_server) ... ok

----------------------------------------------------------------------
Ran 10 tests in 0.051s

OK
..............................
----------------------------------------------------------------------
Ran 30 tests in 22.354s

OK
..........
----------------------------------------------------------------------
Ran 10 tests in 2.969s

OK
.................................
----------------------------------------------------------------------
Ran 33 tests in 0.160s

OK
Online functional runner labels: pf-testing
.......
----------------------------------------------------------------------
Ran 7 tests in 0.002s

OK
.......
----------------------------------------------------------------------
Ran 7 tests in 0.001s

OK
....
----------------------------------------------------------------------
Ran 4 tests in 0.128s

OK
...
----------------------------------------------------------------------
Ran 3 tests in 0.547s

OK
PASS: cold stampede script self-test
exit_code=0

$ python3 ./scripts/test_object_generation_protocol_model.py
test_a_recovery_barrier_cannot_choose_an_empty_value (__main__.ObjectGenerationProtocolModel.test_a_recovery_barrier_cannot_choose_an_empty_value) ... ok
test_all_bounded_two_proposer_interleavings_choose_at_most_one_value (__main__.ObjectGenerationProtocolModel.test_all_bounded_two_proposer_interleavings_choose_at_most_one_value) ... ok
test_an_unavailable_decision_quorum_cannot_choose_a_candidate (__main__.ObjectGenerationProtocolModel.test_an_unavailable_decision_quorum_cannot_choose_a_candidate) ... ok
test_counter_allocation_requires_persistence_and_is_consumed_once (__main__.ObjectGenerationProtocolModel.test_counter_allocation_requires_persistence_and_is_consumed_once) ... ok
test_counter_restart_burns_persisted_allocations_before_or_after_issue (__main__.ObjectGenerationProtocolModel.test_counter_restart_burns_persisted_allocations_before_or_after_issue) ... ok
test_crash_after_promise_before_reply_does_not_restore_an_older_ballot (__main__.ObjectGenerationProtocolModel.test_crash_after_promise_before_reply_does_not_restore_an_older_ballot) ... ok
test_duplicate_or_unconfigured_voters_cannot_form_a_quorum (__main__.ObjectGenerationProtocolModel.test_duplicate_or_unconfigured_voters_cannot_form_a_quorum) ... ok
test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision (__main__.ObjectGenerationProtocolModel.test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision) ... ok
test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected (__main__.ObjectGenerationProtocolModel.test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected) ... ok
test_forgetting_tombstone_authority_allows_an_aba_create (__main__.ObjectGenerationProtocolModel.test_forgetting_tombstone_authority_allows_an_aba_create) ... ok
test_issuing_before_persistence_reuses_a_ballot_after_restart (__main__.ObjectGenerationProtocolModel.test_issuing_before_persistence_reuses_a_ballot_after_restart) ... ok
test_lost_ack_and_restart_preserve_the_chosen_operation (__main__.ObjectGenerationProtocolModel.test_lost_ack_and_restart_preserve_the_chosen_operation) ... ok
test_minority_accepted_work_is_adopted_and_finished (__main__.ObjectGenerationProtocolModel.test_minority_accepted_work_is_adopted_and_finished) ... ok
test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample (__main__.ObjectGenerationProtocolModel.test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample) ... ok
test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it (__main__.ObjectGenerationProtocolModel.test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it) ... ok
test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor (__main__.ObjectGenerationProtocolModel.test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor) ... ok
test_replacing_voters_with_empty_state_can_choose_a_conflicting_value (__main__.ObjectGenerationProtocolModel.test_replacing_voters_with_empty_state_can_choose_a_conflicting_value) ... ok
test_skipping_predecessor_recovery_chooses_a_broken_revision_chain (__main__.ObjectGenerationProtocolModel.test_skipping_predecessor_recovery_chooses_a_broken_revision_chain) ... ok
test_stale_publication_rollback_and_gc_cannot_change_the_winner (__main__.ObjectGenerationProtocolModel.test_stale_publication_rollback_and_gc_cannot_change_the_winner) ... ok
test_superseded_operation_identity_cannot_be_reused_for_different_input (__main__.ObjectGenerationProtocolModel.test_superseded_operation_identity_cannot_be_reused_for_different_input) ... ok
test_superseded_operation_retry_returns_its_original_successor (__main__.ObjectGenerationProtocolModel.test_superseded_operation_retry_returns_its_original_successor) ... ok
test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision (__main__.ObjectGenerationProtocolModel.test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision) ... ok
test_tombstone_rejects_a_delayed_never_present_create (__main__.ObjectGenerationProtocolModel.test_tombstone_rejects_a_delayed_never_present_create) ... ok
test_unlearned_predecessor_is_recovered_before_the_second_slot (__main__.ObjectGenerationProtocolModel.test_unlearned_predecessor_is_recovered_before_the_second_slot) ... ok

----------------------------------------------------------------------
Ran 24 tests in 0.958s

OK
model: states=51023, transitions=167520, terminal=732, chosen_candidates=2
negative control: double-choice trace=((0, 'prepare', 0), (0, 'prepare', 1), (0, 'prepare', 2), (0, 'accept', 0), (0, 'accept', 1), (0, 'accept', 2), (1, 'prepare', 0), (1, 'prepare', 1), (1, 'prepare', 2), (1, 'accept', 0), (1, 'accept', 1), (1, 'accept', 2))
multi-slot round schedules: 640
exit_code=0

$ python3 ./scripts/check_test_wiring.py
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired
exit_code=0

$ ./scripts/check_no_planning_docs.sh
exit_code=0

$ ./scripts/check_uring_lane_lib_only.sh
OK: no 'uring_'-matching test functions under crates/ecstore/tests; --lib is safe
exit_code=0

Initial complete gate
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
 Nextest run ID e07fc5b9-c6c8-4cdd-ae61-54839bf24437 with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.016s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.011s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.009s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.009s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  16.950s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   4.025s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.008s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.007s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.035s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  21.087s] 10 tests run: 10 passed, 0 skipped
exit_code=0

Restored complete gate
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
 Nextest run ID 233b7307-5361-4eb7-94d5-b384f17d2d6a with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.009s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.008s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.008s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.008s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  19.518s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   4.146s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.009s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.018s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.009s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.054s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  23.787s] 10 tests run: 10 passed, 0 skipped
exit_code=0

Mutation M1: success_measurement reports max(actual elapsed, requested budget + 1), forcing 1001 ms for the one-second request. The retained duration assertion at rustfs/tests/connect_perf_drive.rs:105 fails.
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast -E 'test(real_local_write_read_is_measured_and_scratch_is_removed)'
info: experimental features enabled: setup-scripts
   Compiling rustfs v1.0.1 (/private/tmp/gw-orchestrator-2734/ci-drive/rustfs)
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs") generated 1 warning
warning: `rustfs` (bin "rustfs-cli") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 7m 06s
────────────
 Nextest run ID 9f618777-7bf6-4cbc-8803-dd7f50f449ed with nextest profile: ci
    Starting 1 test across 1 binary (9 tests skipped)
        FAIL [   0.016s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
  stdout ───

    running 1 test
    test real_local_write_read_is_measured_and_scratch_is_removed ... FAILED

    failures:

    failures:
        real_local_write_read_is_measured_and_scratch_is_removed

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 0.01s

  stderr ───

    thread 'real_local_write_read_is_measured_and_scratch_is_removed' (106986846) panicked at rustfs/tests/connect_perf_drive.rs:105:5:
    assertion failed: (1..=1_000).contains(&data.duration_millis)
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

────────────
     Summary [   0.017s] 1 test run: 0 passed, 1 failed, 9 skipped
        FAIL [   0.016s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
error: test run failed
exit_code=100

Mutation M2: target_parameters reports requested duration + 1. The in-process assertion at rustfs/tests/connect_perf_drive.rs:111 and CLI JSON assertion at rustfs/tests/connect_perf_drive.rs:527 both reject 1001 against 1000.
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast -E 'test(real_local_write_read_is_measured_and_scratch_is_removed) | test(production_cli_measures_and_signs_exact_binary_provenance)'
info: experimental features enabled: setup-scripts
   Compiling rustfs v1.0.1 (/private/tmp/gw-orchestrator-2734/ci-drive/rustfs)
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs") generated 1 warning
warning: `rustfs` (bin "rustfs-cli") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 7m 20s
────────────
 Nextest run ID b73d46ab-c910-4e1e-8063-7fc7e332ec15 with nextest profile: ci
    Starting 2 tests across 1 binary (8 tests skipped)
        FAIL [  13.968s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
  stdout ───

    running 1 test
    test production_cli_measures_and_signs_exact_binary_provenance ... FAILED

    failures:

    failures:
        production_cli_measures_and_signs_exact_binary_provenance

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 13.96s

  stderr ───

    thread 'production_cli_measures_and_signs_exact_binary_provenance' (107015462) panicked at rustfs/tests/connect_perf_drive.rs:527:5:
    assertion `left == right` failed
      left: Number(1001)
     right: 1000
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

        FAIL [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
  stdout ───

    running 1 test
    test real_local_write_read_is_measured_and_scratch_is_removed ... FAILED

    failures:

    failures:
        real_local_write_read_is_measured_and_scratch_is_removed

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 0.01s

  stderr ───

    thread 'real_local_write_read_is_measured_and_scratch_is_removed' (107016667) panicked at rustfs/tests/connect_perf_drive.rs:111:5:
    assertion `left == right` failed
      left: 1001
     right: 1000
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

────────────
     Summary [  13.985s] 2 tests run: 0 passed, 2 failed, 8 skipped
        FAIL [  13.968s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        FAIL [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
error: test run failed
exit_code=100

Transient production mutations were restored byte-for-byte before the complete green rerun. Original/restored SHA256: 8d389e6d8836c041aab8c6a93e56addd65ba3f70b33264e1c74f639d0572a2da.

$ git diff --check

exit_code=0

$ git diff --cached --check

exit_code=0

Independent orchestrator acceptance
Tested code tree: e67c72fb01

Tested tree: 16fbbd2ea4
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

$ LANG=en_US.UTF-8 CARGO_BUILD_JOBS=1 CARGO_INCREMENTAL=0 cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
info: experimental features enabled: setup-scripts
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs-cli") generated 1 warning
warning: `rustfs` (bin "rustfs") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.69s
────────────
 Nextest run ID cd1edf4c-f1c0-456d-ad1a-76cc00ce05c9 with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.012s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.010s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.010s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.009s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  20.328s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   5.527s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.013s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.019s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.011s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.049s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  25.991s] 10 tests run: 10 passed, 0 skipped

exit_code=0

All 54 tracked Cargo.toml files parse without duplicate keys. git diff --check passed. The production source remains unchanged; the two deliberate implementation mutations above were restored before this complete green run.

* fix(test): compare legacy body wire parity as raw bytes

Lossy UTF-8 comparison concealed a changed binary response byte.
Compare the already captured Vec<u8> values directly.

Validation:
- Binary mutant: migrated payload 0x84 vs legacy 0x83; old comparator
  passed 1/1, raw comparator failed 0/1 (exit 100); restored green.
- Non-Swift body-drain mutant: untouched-body delegation test failed
  with empty vs payload (exit 100); restored handler without a diff.
- cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols
  --features rustfs-protocols/swift --no-fail-fast:
  543 tests run, 543 passed, 0 skipped after both mutants restored.
- cargo fmt --all --check and git diff --check passed.

Orchestrator acceptance (LANG=en_US.UTF-8):
cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
Summary [2.445s] 543 tests run: 543 passed, 0 skipped.

All shared quick checks on the final code tree:
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

Final independent acceptance after the drive-test CI repair
Parent: a99d8f4f89
Tested code tree: 93cbd96bf0

Tested tree: 075ae5161c2b62a96812bc1e3c904a7663b4f2f8
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

$ LANG=en_US.UTF-8 CARGO_BUILD_JOBS=1 CARGO_INCREMENTAL=0 cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
    Finished `test` profile [unoptimized + debuginfo] target(s) in 10m 08s
 Nextest run ID 5c7ff102-ec6d-479a-86e0-810859b28bbf with nextest profile: ci
    Starting 543 tests across 8 binaries
     Summary [   6.729s] 543 tests run: 543 passed, 0 skipped
exit_code=0

The only source change is the raw-byte wire comparator. Cargo manifests and the ratchet baselines remain unchanged from the independently validated parent. The invalid-byte and consumed-request mutations remain recorded above; their implementation changes were restored.

* fix: repair integration checks and IPC test readiness

Wait for the RPC/locks test sockets to pass the existing owner and mode
checks inside the original 5s readiness window and 10ms poll interval.

Use the existing admin S3 carrier for the new recovery and quota error
calls. Regenerate the test-count baseline for main's 118 existing tests
across seven crates. Pair the private pagination helper's raw-scan inputs
and remove the redundant Copy clone from the retirement event.

Keep the original assertions, responses, authorization, locks and storage
operations. The pagination body and raw-scan expressions at all five calls
are byte-identical; reinserting only the Copy clone restores the entire
base heal file. All ignored baseline values and s3s counts stay unchanged.

Validation base: c47432e353
Validation tree: f081084bc5
Patch SHA-256: 2dcf0028691bfb2772edfb38e262829b04f4151832dcce823c9dd9f7434b52de
Trace production-prefix SHA-256: fd994f218a21429d62866f4efb8eee1ee57b58f4a51865aa35d05a54ef1ccb45

All commands used LANG=en_US.UTF-8, CARGO_BUILD_JOBS=1,
CARGO_INCREMENTAL=0 and CARGO_NET_OFFLINE=true in one exclusive target.
Every gate retained the same tree and six source hashes before and after.
54 Cargo.toml files parse without duplicate keys.

cargo clippy -p rustfs --all-targets -- -D warnings
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 12m 50s
Exit: 0. Wall time: 770.855s.

The original 19 quick-check steps:
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
Wall time: 286.337s.

cargo nextest run --profile ci -p rustfs --no-fail-fast
     Summary [ 217.017s] 5382 tests run: 5382 passed (4 leaky), 40 skipped
Exit: 0. Wall time: 1324.414s.

cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
     Summary [   2.189s] 543 tests run: 543 passed, 0 skipped
Exit: 0. Wall time: 123.192s.

cargo nextest run --profile ci -p rustfs-ecstore list_objects_paginate --no-fail-fast
     Summary [   0.170s] 8 tests run: 8 passed, 5921 skipped
Exit: 0. Wall time: 261.324s.

cargo nextest run --profile ci -p rustfs-heal durable_mrf_anchor_retires_only_after_exact_bucket_generation_is_deleted --no-fail-fast
     Summary [   0.805s] 1 test run: 1 passed, 835 skipped
Exit: 0. Wall time: 199.801s.

Baseline regeneration:
./scripts/check_test_count_ratchet.sh --update
wrote .config/test-count-baseline.txt: 52 crates, 21340 tests, 164 ignored
Test-only increases: ecstore48, heal3, iam14, io-metrics2, obs3, policy14,
rustfs34. Existing ignored values and all other rows are byte-identical.
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)

Historical runtime red controls on f622 (same retained two assertions):
cargo nextest run --profile ci -p rustfs --lib -E 'test(=connect::diagnostics::trace_runtime::tests::local_top_rpc_reads_service_http_completions_and_signs_offline) | test(=connect::diagnostics::trace_runtime::tests::local_top_locks_reads_service_manager_and_signs_offline)' --no-fail-fast
Original path readiness with a temporary 500ms bind/seal gap:
     Summary [   0.863s] 2 tests run: 0 passed, 2 failed, 5113 skipped
Exit: 100.
Sealed readiness in that same gap:
     Summary [   5.621s] 2 tests run: 2 passed, 5113 skipped
Exit: 0.
Compiled M1: replace the helper predicate with path.exists():
     Summary [   0.870s] 2 tests run: 0 passed, 2 failed, 5113 skipped
Exit: 100.
Restore the sealed predicate byte-for-byte:
     Summary [   5.640s] 2 tests run: 2 passed, 5113 skipped
Exit: 0.
Both reds observe mode755 with owner_matches=true and fail at the original
RPC/locks unwraps with StateSecurity. M1 failure locations: trace_runtime.rs
2547:29 and 2713:29. No new assertions were introduced.

cargo nextest run --profile ci -p rustfs --lib -E 'test(connect::diagnostics::trace_runtime::tests::)' --no-fail-fast
     Summary [  20.748s] 28 tests run: 28 passed (1 leaky), 5087 skipped
Exit: 0. This security sweep is historical; the full final unit gate above
covers the actual c474 candidate.
The first disk-stopped compile ran no tests and is excluded from red credit.
All temporary seal instrumentation is removed from the final source.

Final unit cleanup markers: 4 leaky, 40 original skips (exit0). The ld
__eh_frame linker warning also appears verbatim in the original f622 run
and the earlier c474 run; there are no new source warnings.

* fix: remove redundant pending closure in disk test

Pass the capture-free pending future factory as a function item in the existing slow-operation test. This fixes the redundant_closure error at disk_store.rs:2914 that stopped strict Clippy before the feature tests could run.

The single-line test edit preserves the factory's result, warning and timeout behavior. Reversing the replacement reproduces the parent file byte for byte. All 220 assertion lines and the production prefix are unchanged; no tests, assertions, guards, profiles or baselines were added or relaxed.

Original Linux CI evidence (run 37888552268 at 53245be9f):
- RIO job113685483992: strict Clippy exit101, redundant closure at disk_store.rs:2914; feature tests skipped.
- Workspace Test and Lint job113685484058: same strict Clippy exit101; nextest and doc tests skipped.
These are lint failures, not runtime mutation kills or known flakes.

Native macOS arm64 validation on tree7a5bf47e7533b29479bc85207416102f1337c566, LANG=en_US.UTF-8, one build job, CARGO_INCREMENTAL=0 and offline dependencies:

shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
All 19 checks: exit0, 312.161s.

$ cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 20m 11s
exit0, 1213.210s total.

$ cargo nextest run --profile ci -p rustfs-ecstore --features rio-v2 slow_disk_operation_warns_once_while_pending_and_preserves_result --no-fail-fast
Finished `test` profile [unoptimized + debuginfo] target(s) in 4m 34s
PASS [0.026s] rustfs-ecstore disk::disk_store::tests::slow_disk_operation_warns_once_while_pending_and_preserves_result
Summary [   0.170s] 1 test run: 1 passed, 5943 skipped
exit0, 281.920s total; 5943 existing filter skips.

warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
This linker warning is retained; flags, profiles and allowances are unchanged.

54 tracked Cargo.toml files parsed without duplicate keys. git diff --check passed. No new assertions were introduced, so no new runtime mutation is claimed. Native compilation and the existing helper test do not substitute for actual Linux RIO execution; the new integration CI run still needs to recover before the RPC merge.

Preservation and output hashes:
parent file SHA256: 8e59ee745e3aaaa22848b943d3b10f6353b515956282cde1f3af2c394200c9ac
final file SHA256: b69ba3237061b161448ae8b62b615885d5770b98a5b4a2479bcfbd70f1bd1f17
unchanged production prefix SHA256: 7903a6e288252c420231242d7b58a25fa2a7140ea3dd430cbe7f0dde11369dcb
RIO Linux failure log SHA256:421addb9868fd88e4109a922311e54f91f0dfd70862f8ba7fcaba193482ed90b
Workspace Linux failure log SHA256:e0aea8d0f014c022252a9534968e083c98cfc1fd760da445ecd04ec3f522e26a
19 output SHA256:35a54f28d707956ed1edb29d492c4dfe8a9fe753d103b111b5a2f1b5c41c2b98
RIO Clippy output SHA256:d6046e3868f2b3992fe11cdd04e9e98eb50a87d1234d9a70dbfeb154ff61638d
slow-operation output SHA256:605b7b258d2fd09c6ef2ab2ffc90d433d0509480f1e45125d20c823759e83608

Refs rustfs/backlog#2734.

* test(ecstore): isolate tier-free recovery fixture startup

The fixture constructor can take the production recovery lock and wait for 60 seconds before this test installs its queue and seeds metadata. Disable its background recovery only while creating this fixture, then restore the environment before calling the existing production recovery entrypoint.

Only the existing test setup changes. Its production entrypoint, default lock parameters, original assertions, 30-second timeout, cancellation and cleanup remain intact. No tests or assertions were added. Production and store/init.rs are byte-identical to the parent.

Independent acceptance ran on merge tree 8ec358cb42ba9ebb3bcfeafcef048a3f0366355b; the runner header reports parent HEAD, while before/after index and source hashes confirm this candidate. Worker task branch d8055766b8 has the same tree.

Every command used LANG=en_US.UTF-8. Task compilation also used CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0, CARGO_NET_OFFLINE=true, an independent target, and empty RUSTC_WRAPPER/RUSTC_WORKSPACE_WRAPPER to keep compilers in its owned process group without changing repository configuration.

Command:
shellcheck --version && actionlint
Exit 0; observed output excerpt:
version: 0.11.0
Log SHA-256: fc9eec16b4191d058a295813ca9afbe9d2b37b59975da860170726be162a3924

Command:
cargo fmt --all --check
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_unsafe_code_allowances.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_layer_dependencies.sh
Exit 0; observed output excerpt:
Layer dependency guard passed (no new reverse dependencies/cycles).
Log SHA-256: 69c6cb1896c0586b28d1167380d3260d99d8017a4265c8df5a2348f289e93a12

Command:
./scripts/check_architecture_migration_rules.sh
Exit 0; observed output excerpt:
Architecture migration rules passed.
Log SHA-256: 28790e70d070607ed4de0704f292a65030be92ec224ef106d4b72fd8296f896e

Command:
./scripts/check_logging_guardrails.sh
Exit 0; observed output excerpt:
✅ Logging guardrails check passed
Log SHA-256: 7adf41abff8ed8f7617decf86498888ddde78937099dd9ce47f6d359dfc2d317

Command:
./scripts/check_error_other_format_ratchet.sh
Exit 0; observed output excerpt:
error(format!) ratchet passed (total call sites: 579).
Log SHA-256: 66172aef18f150ba0605af122e26ffcf91f6dd86f43e38b05e2e982ce8d783ee

Command:
./scripts/check_no_tokio_io_uring.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_extension_schema_boundaries.sh
Exit 0; observed output excerpt:
Extension schema boundaries passed.
Log SHA-256: f13d0acd0e2ce5d9815241b12b4dab7d01cc11b9b65bcf6dfb7812c878532a51

Command:
./scripts/check_body_cache_whitelist.sh
Exit 0; observed output excerpt:
Body-cache whitelist guard passed.
Log SHA-256: 6b828847b0383950ee6eef9cd846d0c322c945e47efd4c7d1ad881178810d6bb

Command:
./scripts/check_s3s_footprint.sh
Exit 0; observed output excerpt:
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)
✅ s3s footprint baseline check passed
Log SHA-256: 47f53d04ef645618563fe1f22abc64667f19078f1974feedcca46e4f2c016844

Command:
./scripts/check_test_count_ratchet.sh
Exit 0; observed output excerpt:
test-count ratchet OK: 52 crates, 21340 tests, 164 ignored match .config/test-count-baseline.txt
Log SHA-256: 618a5e330e933de0c6ff484546df401457024a88bb5986d11d588794e33dab60

Command:
./scripts/check_fips_wording.sh
Exit 0; observed output excerpt:
FIPS wording guard passed (README.md and CHANGELOG.md contain no forbidden claims).
KMS wording guard passed (crates/kms claims no Vault KV2 Transit wrapping).
Log SHA-256: 99221cc9979c480ec4f46f50d76c52b18c72fe499f12e15e4f4a22a1b8bfb140

Command:
./scripts/check_embedded_secrets.sh
Exit 0; observed output excerpt:
Embedded secret guard passed (no private key material or provider credential literal in the tree).
Log SHA-256: 437a3ceba32094d2901889965ad191e411aad32cdb18bf0b8659e818011c406b

Command:
make script-tests
Exit 0; observed output excerpt:
test_cli_requires_the_repository_default_branch (__main__.SelfTests.test_cli_requires_the_repository_default_branch) ... ok
test_config_rejects_duplicate_and_invalid_entries (__main__.SelfTests.test_config_rejects_duplicate_and_invalid_entries) ... ok
test_freshness_boundaries (__main__.SelfTests.test_freshness_boundaries) ... ok
test_grace_requires_two_successful_queries_with_no_history (__main__.SelfTests.test_grace_requires_two_successful_queries_with_no_history) ... ok
test_invalid_api_evidence_fails_closed (__main__.SelfTests.test_invalid_api_evidence_fails_closed) ... ok
test_report_retains_every_workflow (__main__.SelfTests.test_report_retains_every_workflow) ... ok
test_requests_filter_schedule_default_branch_and_success_on_server (__main__.SelfTests.test_requests_filter_schedule_default_branch_and_success_on_server) ... ok
Ran 10 tests in 0.042s
OK
Ran 30 tests in 20.627s
OK
Ran 10 tests in 2.268s
OK
Ran 33 tests in 0.127s
OK
Ran 7 tests in 0.002s
OK
Ran 7 tests in 0.001s
OK
Ran 4 tests in 0.108s
OK
Ran 3 tests in 0.564s
OK
PASS: cold stampede script self-test
Log SHA-256: 75fda2d069a373c63667b908baaf99e6591aab9099a739b0c4c22ceb6da8d68c

Command:
python3 ./scripts/test_object_generation_protocol_model.py
Exit 0; observed output excerpt:
test_an_unavailable_decision_quorum_cannot_choose_a_candidate (__main__.ObjectGenerationProtocolModel.test_an_unavailable_decision_quorum_cannot_choose_a_candidate) ... ok
test_counter_allocation_requires_persistence_and_is_consumed_once (__main__.ObjectGenerationProtocolModel.test_counter_allocation_requires_persistence_and_is_consumed_once) ... ok
test_counter_restart_burns_persisted_allocations_before_or_after_issue (__main__.ObjectGenerationProtocolModel.test_counter_restart_burns_persisted_allocations_before_or_after_issue) ... ok
test_crash_after_promise_before_reply_does_not_restore_an_older_ballot (__main__.ObjectGenerationProtocolModel.test_crash_after_promise_before_reply_does_not_restore_an_older_ballot) ... ok
test_duplicate_or_unconfigured_voters_cannot_form_a_quorum (__main__.ObjectGenerationProtocolModel.test_duplicate_or_unconfigured_voters_cannot_form_a_quorum) ... ok
test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision (__main__.ObjectGenerationProtocolModel.test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision) ... ok
test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected (__main__.ObjectGenerationProtocolModel.test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected) ... ok
test_forgetting_tombstone_authority_allows_an_aba_create (__main__.ObjectGenerationProtocolModel.test_forgetting_tombstone_authority_allows_an_aba_create) ... ok
test_issuing_before_persistence_reuses_a_ballot_after_restart (__main__.ObjectGenerationProtocolModel.test_issuing_before_persistence_reuses_a_ballot_after_restart) ... ok
test_lost_ack_and_restart_preserve_the_chosen_operation (__main__.ObjectGenerationProtocolModel.test_lost_ack_and_restart_preserve_the_chosen_operation) ... ok
test_minority_accepted_work_is_adopted_and_finished (__main__.ObjectGenerationProtocolModel.test_minority_accepted_work_is_adopted_and_finished) ... ok
test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample (__main__.ObjectGenerationProtocolModel.test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample) ... ok
test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it (__main__.ObjectGenerationProtocolModel.test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it) ... ok
test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor (__main__.ObjectGenerationProtocolModel.test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor) ... ok
test_replacing_voters_with_empty_state_can_choose_a_conflicting_value (__main__.ObjectGenerationProtocolModel.test_replacing_voters_with_empty_state_can_choose_a_conflicting_value) ... ok
test_skipping_predecessor_recovery_chooses_a_broken_revision_chain (__main__.ObjectGenerationProtocolModel.test_skipping_predecessor_recovery_chooses_a_broken_revision_chain) ... ok
test_stale_publication_rollback_and_gc_cannot_change_the_winner (__main__.ObjectGenerationProtocolModel.test_stale_publication_rollback_and_gc_cannot_change_the_winner) ... ok
test_superseded_operation_identity_cannot_be_reused_for_different_input (__main__.ObjectGenerationProtocolModel.test_superseded_operation_identity_cannot_be_reused_for_different_input) ... ok
test_superseded_operation_retry_returns_its_original_successor (__main__.ObjectGenerationProtocolModel.test_superseded_operation_retry_returns_its_original_successor) ... ok
test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision (__main__.ObjectGenerationProtocolModel.test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision) ... ok
test_tombstone_rejects_a_delayed_never_present_create (__main__.ObjectGenerationProtocolModel.test_tombstone_rejects_a_delayed_never_present_create) ... ok
test_unlearned_predecessor_is_recovered_before_the_second_slot (__main__.ObjectGenerationProtocolModel.test_unlearned_predecessor_is_recovered_before_the_second_slot) ... ok
Ran 24 tests in 0.929s
OK
Log SHA-256: 0b4be926efb0609d497ae58930340153f01957b606206cbbdb3e0d47eb29d3a2

Command:
python3 ./scripts/check_test_wiring.py
Exit 0; observed output excerpt:
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired
Log SHA-256: 21f676f458451525535ab0ef73d6a9c0d9774e16d76a4d19e396ad34936dd39c

Command:
./scripts/check_no_planning_docs.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_uring_lane_lib_only.sh
Exit 0; observed output excerpt:
OK: no 'uring_'-matching test functions under crates/ecstore/tests; --lib is safe
Log SHA-256: 43f3f851e27ec2ab9e34d6b05fefdede38af992699c473904bfe6f53a46d52e3

Command:
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast
Exit 0; wall 234.399s.
Observed output:
    Finished `test` profile [unoptimized + debuginfo] target(s) in 3m 49s
    Starting 1 test across 1 binary (5871 tests skipped)
  SETUP PASS [   0.006s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"'
        PASS [   0.440s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item
     Summary [   0.586s] 1 test run: 1 passed, 5871 skipped
Log SHA-256: 34e973f86e8dc2102c1a0e483e0ac802ce7df3d1b40a603932f62c0805c4de27

Command:
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over) | test(=store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence) | test(=store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable)' --no-fail-fast
Exit 0; wall 4.010s.
Observed output:
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.43s
    Starting 3 tests across 1 binary (5869 tests skipped)
  SETUP PASS [   0.005s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"'
        PASS [   0.068s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over
        PASS [   0.637s] rustfs-ecstore store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence
        PASS [   0.846s] rustfs-ecstore store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable
     Summary [   0.987s] 3 tests run: 3 passed, 5869 skipped
Log SHA-256: b5da2a927ff18ca9aa217762f7186a6d4371bd9e8feedd7b77940e10b64dffac

Command:
cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
Exit 0; wall 1084.508s.
Observed output:
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 18m 03s
Log SHA-256: 28f98be7970faaa71cd118da1da4e6e2e3d23b9dda7e6449eaa7a37e741260b5

Compiled mutation command (unchanged for each red and restored run):
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast

M1: reverse only this test-local constructor flag from false to true. The mutant compiled, then failed the original recovered-item queue assertion after 30.481s. Restoring the entire observed candidate passed in 0.460s. Red log 55bf3b1535529ac0bcfb8071f9b56f9f40631007a85c315367e7e93786d65293; restored log 204d5fbe4574a7917f5e740a795d864fe28a7e7fb5e33d27b0a9cf68e2f17f52.

M2: replace only the actual awaited enqueue with false, consuming its original inputs. The mutant compiled, then failed the same original queue assertion after 30.516s. Restoring the entire observed candidate passed in 0.457s. Red log 89186330413215e231e2b06bf013cc31097ea8f4a106d91421a36d6c8986a7fd; restored log e848b42d2302401afa5dbcd9e37bf76d2e56a96ab6de5a488513b997f53c5e9b.

All temporary observation and mutation code was removed before final acceptance. Historical baseline and diagnostic failures remain recorded; they are not mutation credit. The separate Object Lock intent-retry diagnostic still fails, and these native macOS checks do not establish Linux CI recovery.

* refactor(rpc): serve internode rpc bodies with the owned Body (#8441)

* test(rpc): pin the wire bytes of every internode rpc response body

rustfs/backlog#2747 batch 4, step 1 of 2, on the s3s bodies the rpc paths
serve today. Each response body builder in storage/rpc/http_service.rs is
served through hyper's HTTP/1.1 server and the raw bytes the peer reads are
compared with goldens captured from this commit: status, content-length or
chunked framing, chunk boundaries, the terminating chunk, HEAD, and the cut
connection a failure after data produces. The capability responses carry a
per-process proof, so their length header and payload are checked against
the body they encode instead.

The read_file_stream response builder moves into read_file_stream_response
so the test reaches the same code the handler runs; behaviour is unchanged.

* refactor(rpc): serve internode rpc bodies with the owned Body

rustfs/backlog#2747 batch 4, step 2 of 2. The internode rpc HTTP paths
build their responses with rustfs_s3_types::Body: Body::from_stream
replaces Body::from(StreamingBlob::wrap(..)) at the six stream sites, and
the in-memory and empty bodies keep their constructors. InternodeRpcService
converts its responses into the wrapped service's response body through
From<Body>, so the server keeps composing it with the s3s service as
before. The wire-byte goldens from the previous commit pass unchanged;
storage/rpc no longer names s3s.

* fix(rpc): simplify wire test response builder type

Name the boxed response builder used by the capability wire test so
all-target Clippy can check it without a type-complexity warning.

Validation (LANG=en_US.UTF-8, CARGO_INCREMENTAL=0):
- cargo clippy -p rustfs --all-targets -- -D warnings: exit 0.
- cargo nextest run --profile ci -p rustfs --lib -E 'test(/storage::rpc::/)': 279 tests run, 279 passed, 4842 skipped.
- All 19 quick-check commands, including script-tests, both Python scripts and cargo fmt --all --check: exit 0.
- Footprint baseline regenerated: files=194, s3_error_lines=1584, ecstore_files=36; test-count baseline regenerated: 52 crates, 21228 tests, 164 ignored.
- All 54 Cargo.toml files parse without duplicate keys.

No assertion was added or changed by this lint fix. R1-R8 are retained
in the existing PR verification record.

* fix(ecstore): resume fenced Object Lock creation intents

* fix(ecstore): bound creation retry futures on the stack

* fix(ecstore): require native proof for creation retries

* fix(ecstore): bound bucket creation stack usage (#8466)

* fix(ecstore): retain creation fences through rollback

Keep lifecycle, metadata-transaction, and namespace guards around rollback and metadata cleanup. Preserve the original publication error when a guard is lost. Build the timestamp retry fixture through the real interrupted producer so it checks a valid pending creation.

Validated source tree: 4e8e207999.
Two independent source reviews found no blocking regression.

Validation (LANG=en_US.UTF-8):
- bash -c 'shellcheck --version && actionlint' — exit 0
- cargo fmt --all --check — exit 0
- ./scripts/check_unsafe_code_allowances.sh — exit 0
- ./scripts/check_layer_dependencies.sh — exit 0
- ./scripts/check_architecture_migration_rules.sh — exit 0
- ./scripts/check_logging_guardrails.sh — exit 0
- ./scripts/check_error_other_format_ratchet.sh — exit 0
- ./scripts/check_no_tokio_io_uring.sh — exit 0
- ./scripts/check_extension_schema_boundaries.sh — exit 0
- ./scripts/check_body_cache_whitelist.sh — exit 0
- ./scripts/check_s3s_footprint.sh — exit 0
- ./scripts/check_test_count_ratchet.sh — exit 0
- ./scripts/check_fips_wording.sh — exit 0
- ./scripts/check_embedded_secrets.sh — exit 0
- make script-tests — exit 0
- python3 ./scripts/test_object_generation_protocol_model.py — exit 0
- python3 ./scripts/check_test_wiring.py — exit 0
- ./scripts/check_no_planning_docs.sh — exit 0
- ./scripts/check_uring_lane_lib_only.sh — exit 0
- cargo nextest run --profile ci -p rustfs-ecstore --lib --locked --no-fail-fast — 5894 passed, 0 skipped
- cargo check -p rustfs-ecstore --all-targets --locked — exit 0
- cargo nextest run --profile ci -p rustfs --lib --locked --no-fail-fast -E 'test(=admin::handlers::table_catalog::tests::cold_strong_registration_completes_while_other_bucket_backup_waits) | test(=admin::handlers::table_catalog::tests::commit_backend_borrowed_put_preserves_owner_authorization_and_precondition) | test(=admin::handlers::table_catalog::tests::disable_table_bucket_preserves_objects_and_can_be_reenabled) | test(=admin::handlers::table_catalog::tests::load_table_handler_negotiates_vended_credentials_without_breaking_metadata_only_callers) | test(=admin::handlers::table_catalog::tests::namespace_property_handler_updates_object_backed_catalog_and_maps_errors) | test(=app::object::get::tests::execute_get_object_resumes_from_relocated_pool_without_splicing_body) | test(=table_catalog::store::tests::catalog_writer_loss_prevents_recovery_storage_commit)' — 7 passed
- cargo nextest run --profile ci -p rustfs --lib --locked --no-fail-fast -E 'test(unsupported_remote_disk_)' — 2 passed

Additional fault-injection evidence remains incomplete. Timestamp M1 reached its original assertion; its restore run is being collected. M2–M5 and the three planned rollback faults have not run. An automated risk check interrupted the timestamp proof task; no assertions or validation settings were weakened.

* fix(ecstore): box nested bucket rollback state

Keep the guarded rollback future behind a heap allocation so bucket creation does not store its nested state inline. The rollback body, all three fence references and pending-proof revocation order are unchanged.

The same three original Linux tests overflowed on both e611 CI attempts. Local format, ecstore test compilation and all 19 quick-check steps pass; the original Linux feature lanes still need to validate this candidate.

* test(ecstore): isolate rollback guard loss fixtures

Run the three rollback metadata-cleanup cases in separate test processes so reused /proc/self/fd aliases from earlier cases cannot invalidate their creation witnesses. Keep the original fixture, guards, assertions and io_uring test selection, and verify that each child runs the intended case successfully.

Update the generated test-count baseline for the added macro arm.

Validation: cargo check -p rustfs-ecstore --tests; cargo fmt --all --check; all 19 quick-check commands, including make script-tests and the two Python checks. Linux io_uring acceptance and fault controls remain pending.

* refactor(replication): move the replication config onto gateway types (#8443)

* wip: T1.4 replication family

* refactor(replication): move the replication config onto gateway types

The replication configuration family leaves s3s: rustfs-replication no longer depends on s3s, and ECStore bucket metadata, the scanner and the rustfs app/admin layers hold rustfs_gateway_types::persistence::PersistedReplicationConfiguration. The persisted XML goes through the gateway persistence codec behind a new BucketConfigXml trait (the 39 byte goldens pass with unchanged fixtures); the legacy s3s edge converts once in storage/ecfs.rs through storage/s3_compat/replication.rs; the scanner data usage cache keeps its MessagePack bytes through a serde mirror pinned by bytes recorded from s3s 5761ddfe. rustfs-gateway-types 0.32.0 is pinned once (rev 772d3385). Refs rustfs/backlog#2745.

* test: use S3 serializer for list response tests

* fix(ci): restore integration test compilation

Gate the diagnostic test import with its feature, initialize the conditional
MetadataWriteStore fields explicitly, and decode the scanner replication fixture
through its existing cache boundary. Preserve the original fixture and assertions.

Validation was run on tree 544acb80c0 at 6003d4b8.
LANG=en_US.UTF-8; no source changes follow validation.

Quick checks: 19/19 passed.
- shellcheck --version && actionlint: exit 0
- cargo fmt --all --check: exit 0
- ./scripts/check_unsafe_code_allowances.sh: exit 0
- ./scripts/check_layer_dependencies.sh: exit 0
- ./scripts/check_architecture_migration_rules.sh: exit 0
- ./scripts/check_logging_guardrails.sh: exit 0
- ./scripts/check_error_other_format_ratchet.sh: exit 0
- ./scripts/check_no_tokio_io_uring.sh: exit 0
- ./scripts/check_extension_schema_boundaries.sh: exit 0
- ./scripts/check_body_cache_whitelist.sh: exit 0
- ./scripts/check_s3s_footprint.sh: exit 0
- ./scripts/check_test_count_ratchet.sh: exit 0
- ./scripts/check_fips_wording.sh: exit 0
- ./scripts/check_embedded_secrets.sh: exit 0
- make script-tests: exit 0
- python3 ./scripts/test_object_generation_protocol_model.py: exit 0
- python3 ./scripts/check_test_wiring.py: exit 0
- ./scripts/check_no_planning_docs.sh: exit 0
- ./scripts/check_uring_lane_lib_only.sh: exit 0

Darwin validation:
- cargo clippy --locked -p rustfs-ecstore -p rustfs-scanner --all-targets -- -D warnings: exit 0
- cargo nextest run --locked -p rustfs-ecstore -p rustfs-scanner --lib -E 'test(config::com::tests::save_config_preserves_retryable_system_volume_errors) | test(scanner_io::tests::replication_maintenance_refreshes_compacted_stats_across_cycles_and_retries)' --test-threads 1 --no-fail-fast: exit 0, 2/2 passed
- cargo clippy --locked -p rustfs-ecstore -p rustfs-scanner --all-targets --features rustfs-ecstore/diagnostic-read-version-caller,rustfs-scanner/diagnostic-read-version-caller -- -D warnings: exit 101
- cargo nextest run --locked -p rustfs-ecstore -p rustfs-scanner --lib --features rustfs-ecstore/diagnostic-read-version-caller,rustfs-scanner/diagnostic-read-version-caller -E 'test(config::com::tests::save_config_preserves_retryable_system_volume_errors) | test(scanner_io::tests::replication_maintenance_refreshes_compacted_stats_across_cycles_and_retries) | test(cluster::rpc::runtime_sources::tests::) | test(config::com::tests::diagnostic_config_write_retains_origin_and_api_for_results_and_cancellation)' --test-threads 1 --no-fail-fast: exit 0, 14/14 passed

Diagnostic strict Clippy remains red on four unchanged production lints:
runtime_sources.rs if_same_then_else and three redundant_locals reports in
set_disk/core/metadata_read.rs and set_disk/read.rs. Their source ranges and
feature wiring match the base; the base was not compiled separately.

Linux aarch64 validation (Rust/Clippy 1.99.0, UID 501, jobs=1):
- cargo clippy --all-targets -- -D warnings: exit 0; Finished dev profile in 68m 12s
- cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings: exit 0; Finished dev profile in 49m 45s
Source HEAD/tree and clean status matched before and after both commands.
No compiler or stack overrides were used. protoc 35.1 was installed from its
verified ARM64 release; flatc was absent and not invoked by the workspace build scripts.
The local environment differs from hosted CI.

* fix(ecstore): recover delete batches with missing versions

Treat an explicit non-null version as a no-op when it is absent from both recovery sources. Keep rejecting missing null versions, delete-marker mismatches and conflicting local generations.

Add returning-disk and partial-receipt regressions, extend journal boundary checks, and regenerate the ecstore test-count baseline. Production faults exercise the 24 added assertion sites and 7 semantic checks: 33 faults failed at the expected sites and passed after full restoration.

Validation (LANG=en_US.UTF-8, one build job):

cargo nextest run --locked -p rustfs-ecstore --lib -E 'test(delete_recovery) | test(quorum_delete_cleanup_coordinator_binds_single_and_batch) | test(crash_delete_retry_batch_reclaims_inline_and_reserved_external)' --test-threads 1 --no-fail-fast
Summary [  34.580s] 69 tests run: 69 passed, 6058 skipped; exit 0

cargo clippy --locked -p rustfs-ecstore --all-targets -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 5m 37s; exit 0

All 19 final Quick Checks completed with exit 0:

- shellcheck --version && actionlint
- cargo fmt --all --check
- ./scripts/check_unsafe_code_allowances.sh
- ./scripts/check_layer_dependencies.sh
- ./scripts/check_architecture_migration_rules.sh
- ./scripts/check_logging_guardrails.sh
- ./scripts/check_error_other_format_ratchet.sh
- ./scripts/check_no_tokio_io_uring.sh
- ./scripts/check_extension_schema_boundaries.sh
- ./scripts/check_body_cache_whitelist.sh
- ./scripts/check_s3s_footprint.sh
- ./scripts/check_test_count_ratchet.sh
- ./scripts/check_fips_wording.sh
- ./scripts/check_embedded_secrets.sh
- make script-tests
- python3 ./scripts/test_object_generation_protocol_model.py
- python3 ./scripts/check_test_wiring.py
- ./scripts/check_no_planning_docs.sh
- ./scripts/check_uring_lane_lib_only.sh

test-count ratchet OK: 52 crates, 22009 tests, 165 ignored match .config/test-count-baseline.txt
PASS: cold stampede script self-test
model: states=51023, transitions=167520, terminal=732, chosen_candidates=2
multi-slot round schedules: 640
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired

* fix(ecstore): retry tier commit intent lock contention

Retry transient namespace lock timeouts within the existing three-attempt
CAS budget. Reload and verify the original mutation identity after
contention, and preserve the final typed error when attempts are exhausted.
Cover save/read contention, exhausted retries, replacement identities and
CAS reconciliation without changing the existing timeout or lock policy.

Validation:
- cargo nextest run --locked -p rustfs-ecstore --features test-util --lib -E 'test(tier_mutation)' --test-threads 1 --no-fail-fast: 52 passed.
- cargo nextest run --locked -p rustfs-ecstore --features test-util --lib -E 'test(intent_advance_) | test(=services::tier::tier::tests::caller_cancellation_after_durable_prepare_does_not_hide_the_mutation)' --test-threads 1 --no-fail-fast: 5 passed.
- cargo clippy --locked -p rustfs-ecstore --all-targets -- -D warnings: passed.
- cargo clippy --locked -p rustfs-ecstore --all-targets --features test-util -- -D warnings: passed.
- 27 mutation pairs failed at the intended assertions and passed after restoring the production code.
- bash scripts/check_test_count_ratchet.sh --update, then bash scripts/check_test_count_ratchet.sh: 52 crates, 22013 tests, 165 ignored; ecstore increases from 6581 to 6585.

* refactor(lifecycle): replace s3s domain types with Gateway DTOs

Keep lifecycle rules, restore requests, Object Lock, and scanner cached tags in domain types. Convert at the existing HTTP boundary while preserving legacy XML, scanner cache bytes, date ordering, and tombstones.

Linux test validation before the final import-path-only adjustment:

cargo nextest run -p rustfs-ecstore bucket_config_goldens
39 passed; 6211 skipped by selector.

cargo nextest run --lib -p rustfs -E 'test(/test_zero_rule_lifecycle_tombstone_recognition/) | test(/test_local_lifecycle_staleness_axis_selection/) | test(/test_lifecycle_staleness_preserves_legacy_submillisecond_axis/) | test(/test_merge_incoming_lifecycle/) | test(/test_lifecycle_expiry_statement_matrix/) | test(/test_lifecycle_expiry_subset_xml/) | test(/put_bucket_lifecycle_validation_errors_keep_their_s3_code/) | test(/execute_put_bucket_lifecycle_configuration_rejects_missing_configuration/) | test(/execute_restore_object_rejects_missing_restore_request/) | test(/execute_restore_object_rejects_select_type/) | test(/execute_restore_object_maps_failures_to_typed_s3_errors/) | test(/execute_restore_object_returns_internal_error_when_store_uninitialized/) | test(/restore_request_conversion_preserves_legacy_output_values/) | test(/restore_request_conversion_preserves_presence_and_raw_enums/) | test(/restore_request_conversion_distinguishes_absent_and_empty_containers/)'
25 passed; 5318 skipped by selector.

cargo nextest run --profile ci -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-replication -p rustfs-scanner
7872 passed; 21 skipped under the existing CI profile.

Previously verified mutation controls: 39 production faults, 41 RED/GREEN pairs.

cargo clippy --locked -p rustfs -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-scanner --all-targets -- -D warnings
Passed in 40m 45s before the import-path adjustment, then in 1m 59s on
the final source with warnings denied. The final adjustment imports the
existing helper through the object domain facade; fmt and the architecture
check also passed.

Part of rustfs/backlog#2745. The remaining ecstore configuration families follow separately.

* refactor(admin): move error types to rustfs-s3-types (#8440)

* refactor(protocols): move error types to rustfs-s3-types by codemod

T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and
the error items of protocols move off s3s by scripts/codemods, run with
--hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it
downcasts the backend error, whose runtime type is still s3s::S3Error.

- rustfs/src/protocols/client.rs: the two webdav session methods now
  return rustfs_s3_types::S3Result and convert the s3s error with From.
- codemod: a child module that calls s3_error! through `use super::*`
  gets its own `use rustfs_s3_types::s3_error;` once its parent imports
  that macro by name; otherwise it is listed as GLOB.
- s3-types: a table test pins code, status and the s3s error the legacy
  edge renders for every variant a moved call site names.
- s3_error_lines 1585 -> 1584 (a lifecycle comment reworded).

* refactor(admin): move error types to rustfs-s3-types (WIP, no build)

T1.2 batch 4 (rustfs/backlog#2743), work in progress: rustfs check
still fails with 16 errors (listed in the PR body).

- codemod: --facade PATH treats a crate-local re-export of s3s's error
  items like s3s for the four names (9 new self-tests, 9 mutants red).
- codemod output for rustfs/src/admin and rustfs/src/site_replication
  (--facade crate::storage_api::site_replication::s3); 6 glob children
  get an explicit rustfs_s3_types::s3_error import.
- admin/router.rs: the Operation trait returns rustfs_s3_types::S3Result.
- error.rs: From<ApiError> builds the rustfs-s3-types error; the s3s
  conversion is derived from it.

* fix(admin): convert error types at the s3s boundary

Batch 4 of T1.2 left the admin and site_replication modules returning
rustfs_s3_types::S3Error while their callers and callees on the storage
and auth side still speak s3s::S3Error. Convert at each edge with the
existing From impls in rustfs-s3-types (compat_s3s.rs) instead of
changing any code, message or status:

- admin/auth.rs, admin/handlers/object_zip_download.rs,
  admin/handlers/table_catalog/mod.rs: map the s3s result returned by
  check_key_valid / authorize_request / authorize_internal_object_request
  into the admin error type with .map_err(Into::into).
- storage/ecfs.rs: the s3s-facing side maps the admin site_replication
  gate results the other way.
- admin/handlers/pools.rs: ApiError.code stays s3s::S3ErrorCode, so the
  rustfs code is converted with .into() when the struct is built, in
  production and in the two tests that build or compare it.
- admin/handlers/heal.rs, admin/runtime_sources.rs: the router returns
  s3s errors (S3Route impl); the expected rustfs code is converted with
  .into() before the comparison.
- admin/handlers/table_catalog/mod.rs: iceberg_rest_error takes
  &'static str so the Cow<'static, str> custom code borrows it; every
  caller already passes an ICEBERG_ERROR_* constant or a &'static str.
- admin/handlers/quota.rs: drop a now same-type .into() after s3_error!.
- storage_api.rs, site_replication/mod.rs: drop re-exports and an import
  that no longer have users (clippy -D warnings).

cargo check -p rustfs --all-targets and cargo clippy -p rustfs
--all-targets -- -D warnings are clean; the s3s footprint baseline is
unchanged at files=172, s3_error_lines=303, ecstore_files=28.

Refs rustfs/backlog#2743, rustfs/backlog#2734.

* refactor(ecstore): move bucket configs onto gateway persisted types (#8709)

* refactor(ecstore): move bucket configs onto gateway persisted types

Finish the ECStore batch of T1.4 (rustfs/backlog#2745). The remaining
eleven bucket configuration families (Website, CORS, Tagging, Versioning,
ObjectLock, Encryption, Notification, Logging, Accelerate, RequestPayment,
PublicAccessBlock) now read and write their stored XML through the
rustfs-gateway-types persistence and cors_tagging codecs instead of the
s3s codec, so the legacy s3s bucket-config codec and its
s3gate-metadata-xml compat marker are removed.

- ecstore: BucketConfigXml is implemented by the gateway codec for every
  family; metadata, metadata_sys, versioning, object lock, replication and
  lifecycle consumers take the persisted types. Region moves to
  rustfs_s3_types::Region, and the ETag conversion to s3s moves to the
  rustfs edge with its tests. Only warm_backend.rs and warm_backend_s3.rs
  still reference s3s, held for T1.5 (rustfs/backlog#2746).
- scanner: object lock cache, item actions and data usage use the
  persisted types.
- rustfs: the legacy S3 edge converts each family between s3s DTOs and the
  persisted types in storage/storage_api.rs by writing one codec's XML and
  reading it with the other; Tagging converts field by field.
- Bump the rustfs-gateway-types pin to 21283e87.

The 39 persisted-byte goldens are unchanged and pass, so a document the
old codec wrote reads back and re-encodes to the same bytes. s3s footprint:
files 181 -> 146, ecstore_files 28 -> 2, s3_error_lines unchanged.

* test(app): prepare the PutBucketEncryption config from the request

The uninitialized-store test passed an empty persisted configuration, which the use case now validates before it reaches the store check, so it answered MalformedXML instead of InternalError. Convert the request's configuration the way the S3 edge does.

* style(app): format the PutBucketEncryption test

* refactor(s3-client): move remote error codes and DTO enums off s3s (#8783)

Part of T1.5 (rustfs/backlog#2746), batch 1.

* refactor(s3select): run queries on gateway DTO request shapes (#8794)

* refactor(s3select): run queries on an engine request built from gateway DTO shapes

Part of T1.5 (rustfs/backlog#2746), batch 2.

* fix(s3select): keep the select request conversion in the app layer

The layer guard forbids app -> storage; the conversion moves from storage/s3_compat/select.rs to app/select_request.rs unchanged.

* fix(heal): retry transient metadata conflicts (#8812)

* fix(heal): retry transient metadata conflicts

* chore: update heal test count baseline

* test: reserve nextest capacity for progress probes (#8818)

* test(e2e): use a non-default region for the differential negative control

Since main's region-aware CreateBucket and GetBucketLocation (#8739, #8743), us-east-1 answers an empty LocationConstraint and 200 for an owned re-create, so the control no longer reported its own region. Also refresh the e2e-full linux selection from the CI listing (682 tests).

* refactor(protocols): move StorageBackend to gateway types (#8825)

* refactor(protocols): move storage backend to gateway types

* test: update e2e test count baseline

* fix: preserve copy source keys and unknown lengths

* test: update protocols test count baseline

* fix(protocols): preserve typed SFTP backend errors

* test(protocols): use gateway list input in policy tests

* fix(protocols): clear clippy failures after T1.5 merge (#8831)

fix(protocols): clear clippy failures after merge

* fix(ecstore): finalize restore metadata after local copy-back (#8836)

* fix(ecstore): finalize restore metadata after local copy-back

* test(ecstore): ratchet added restore finalization cases

* fix(ecstore): keep restore copyback compatible with older peers

* fix(ecstore): gate restore copyback on fleet compatibility

* fix(ecstore): retain transition identity during restore copyback

* fix(ecstore): route committed restore copyback locally

* fix restore copyback cleanup

* fix(ecstore): honor restore probe test fallback

* refactor(storage): migrate option errors to domain types (#8839)

* refactor(s3): migrate private multipart validation errors (#8844)

* refactor(auth): derive policy conditions from AuthzFacts (#8442)

* refactor(auth): extract authorization facts

* test(e2e): update smoke selection baseline

* test(e2e): update full selection baseline

* refactor(admin): migrate health and prefix usage to owned requests (#8887)

* refactor(admin): add owned health router boundary

* refactor(admin): migrate prefix usage to owned request

* refactor(admin): complete the owned admin request boundary (#8949)

* refactor(admin): complete the owned admin request boundary

Every type implementing the owned admin `Operation` now serves the legacy
router through one blanket impl that converts the s3s request and response
at that single edge, so handlers register with `AdminOperation(&Handler {})`
and need no adapter statics; `LegacyAdminAdapter` is removed.

`AdminRequest` gains the accessors handlers read today (`remote_addr`,
`client_info`, `request_context`, `req_info`, `req_info_mut`,
`install_req_info`, `envelope_mut`, `take_body`), the request helpers gain
envelope forms (`authenticate_envelope_credentials`,
`object_store_from_envelope`, the table-catalog `*_from_envelope` helpers,
`KmsAdminAudit::from_envelope`, `AccountAuditContext::from_envelope`,
`CallerIdentity::resolve_envelope`), `read_compatible_admin_body` accepts
either body type, and the owned `Body` gains `store_all_limited` and
`Default`. `admin::operation::test_support::TestRequest` builds owned
requests for handler tests. Unused accessors carry a per-method dead-code
expectation that each migrating handler batch removes.

No handler behavior changes; the remaining migrations follow in batches.

* chore(admin): add the T1.12 admin handler codemod

The script rewrites an admin handler file from the s3s request and response
types to the owned boundary (imports, signatures, request field and helper
calls, body access, response construction, and test request literals) and
leaves a TODO comment at every site it cannot rewrite mechanically. It is
idempotent and is removed once T1.12 completes.

* refactor(admin): migrate replication and KMS handlers to owned requests (#8951)

Move the T1.12 batch 2 admin handlers (site_replication, kms_keys, event,
replication, diagnostics, on_demand_migration, audit, account, kms_rekey,
kms_key_lifecycle, object_data_cache, tls_debug, integrity) from the s3s
request/response types to the owned AdminRequest/AdminResponse boundary.
Handler logic, authorization order, error codes and response bytes are
unchanged; the request/response plumbing now goes through the envelope
accessors and helpers introduced by the owned boundary.

The shared response helpers (utils::json_response, utils::empty_response,
handlers::admin_json_response, target_descriptor::build_json_response,
profile::authorize_profile_request) and the site replication repair
executor still answer with the s3s types, so the migrated files carry
owned-boundary forms of those helpers and convert the repair response at
its call edge. The streaming console-log body and the two devnull drains
move from the s3s byte stream to the owned Body stream/frame API.

Drop the dead_code expectations for the owned accessors and envelope
helpers this batch starts using, and lower the s3s footprint baseline to
the measured 99 files.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* refactor(admin): add owned forms of the shared admin response helpers

Handler batches were copying `json_response`, `empty_response`,
`admin_json_response`, and `build_json_response` into each migrated file
because the shared helpers still return the s3s response. Provide owned
forms next to each legacy helper (`owned_json_response`,
`owned_empty_response`, `owned_admin_json_response`,
`owned_build_json_response`, `authorize_profile_envelope`) with parity
tests against the legacy shape, derive `Debug` for `AdminResponse` so
tests can use `expect_err`, and fix three codemod rules: strip owned
string conversions in test credentials, annotate non-literal URI parses,
and leave a rebound request parameter immutable.

* refactor(admin): use the shared owned response helpers in batch 2 (#8954)

Replace the per-file owned-boundary copies of json_response,
empty_response, admin_json_response, build_json_response and
authorize_profile_request in the batch 2 admin handlers with the shared
owned_* helpers and authorize_profile_envelope that the boundary branch
now provides, and restore expect_err in the handler tests now that
AdminResponse implements Debug; the `.err().expect(..)` stand-ins were
failing the base branch's clippy step (clippy::err_expect under
-D warnings). The site replication repair call edge
keeps its conversion because the repair executor still answers with the
s3s response type.

Drop the dead_code expectations on the shared owned helpers this batch
now uses in non-test code.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* fix(admin): preserve legacy body limit error responses

* refactor(admin): move policy, ILM and batch 3 handlers to owned requests (#8956)

Batch 3 of T1.12 (rustfs/backlog#2788, parent rustfs/backlog#2734) moves
14 admin handler files to AdminRequest/AdminResponse and the owned
Operation trait: ilm_transition, policies, idp_compat, config_admin,
plugins_instances, profile_admin, group, heal, object_zip_download,
kms_key_metadata, durability, inspect_archive, sts and
gateway_key_inventory. Handler logic, authorization order, error codes
and response bytes are unchanged; test request literals use
test_support::TestRequest.

The handlers use the shared owned helpers (owned_json_response,
owned_admin_json_response, owned_build_json_response,
authorize_profile_envelope). The dead_code expectations on the
AdminRequest accessors and RequestEnvelope::uri_mut they now exercise
are removed. The legacy target_descriptor::build_json_response keeps
only its equivalence test caller and carries the matching expectation
until it retires.

object_zip_download builds the s3s request the S3 gate still reads from
the owned request facts; heal authenticates through
authenticate_envelope_credentials; the trace, inspect-archive and zip
bodies use Body::from_stream. extract_plugin_instance_filters takes the
URI so extensions.rs keeps calling it, and the two idp_compat
delegations stay on the legacy trait until their delegates migrate.

The s3s footprint baseline drops from 99 to 90 files.

* test(admin): anchor the replication diff body contract on the call

The source-contract test looked for `let body = read_compatible_admin_body(`
on one line; rustfmt wrapped that binding after the handler moved to the
owned request boundary, so the marker was no longer found. Anchor on the
call itself so formatting cannot change what the contract checks.

* refactor(admin): migrate table catalog and batch 1 to owned requests (#8955)

* fix(admin): preserve catalog body limit response bytes

* refactor(admin): migrate identity and cluster handlers to owned requests (#8966)

* refactor(admin): migrate identity and cluster handlers to owned requests

Move the T1.12 batch 4 admin handlers (user, service_account, oidc, tier,
system, pools, quota, kms_dynamic, kms_backup, rebalance, module_switch,
bucket_meta, cluster_snapshot, realtime, bucket_recovery, profile) from the
s3s request/response types to the owned AdminRequest/AdminResponse
boundary. All 90 impl Operation blocks now take AdminRequest and return
AdminResponse; handler logic, authorization order, error codes and response
bytes are unchanged. The handlers use the shared owned helpers
(owned_json_response, owned_admin_json_response, authorize_profile_envelope,
the envelope-form runtime sources and audit constructors) instead of local
copies.

profile.rs drops the s3s forms of authorize_profile_request and
profile_not_implemented_response now that profile_admin.rs is on the
envelope gate too; profile_not_implemented_response answers with an
AdminResponse. The realtime metrics stream moves from the s3s ByteStream
adapter to Body::from_stream, and the rebalance idle-404 document still
renders through s3s::S3Error::to_http_response with its body converted at
the edge.

The idp_compat delegators ImportIamV2 and AddServiceAccountLdap move to the
owned trait as well: with their targets on both traits, the method-call
delegation became ambiguous between the owned impl and the router blanket
impl, so they now take the owned request directly.

KmsAdminAudit::from_request loses its last production caller with this batch
and is deleted; its comparison test now asserts the envelope facts directly.

Lower the s3s footprint baseline to the measured 68 files.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* refactor(admin): remove the legacy admin helpers batch 4 superseded

Batch 4 moved the last production callers of the s3s-typed admin helpers
to their owned forms, which left `admin_json_response`, `json_response`,
`authenticate_request`, `authorize_admin_request`, `object_store_from_req`
and `object_store_from_extensions` unused, along with the `Body`,
`S3Request` and `S3Response` re-exports of the admin `s3` facade. Remove
them. The owned forms keep their documentation; the behavior tests of the
removed `json_response` now exercise `owned_json_response`, and the
compat-prefix test asserts the owned response directly.

Two tests existed only to compare a removed legacy helper with its owned
twin (`owned_json_response_matches_the_legacy_shape`) or to cover the
removed legacy gate (`authorize_admin_request_without_credentials_is_rejected`);
they go with the helpers, and the owned equivalents stay. The rustfs
test-count row drops from 5957 to 5955 and the s3s footprint from 68 to 66
files.

* refactor(admin): drop the legacy target descriptor JSON response helper (#8968)

`target_descriptor::build_json_response` returned the s3s response and
had no production caller left once the event, audit and plugin instance
handlers moved to `owned_build_json_response`; it survived only for the
legacy/owned equivalence test. Remove it, give the owned helper its own
documentation, and make the test assert the owned response directly:
status, `application/json`, the echoed `x-request-id` when present and
only then, and the body bytes. The file no longer imports s3s, so the
footprint baseline drops from 66 to 65 files.

* test(ecstore): pin native history on cancellation under the cancellable queue

main moved the local write path onto the disk blocking queue, which skips an
operation once its waiter is dropped, so a borrowed write cancelled while it is
queued no longer creates its file. The integration-only native publication
test still expected the queued open to run. Assert the new execution model
instead (the open never runs) while keeping its safety claims: the Started
record is taken before queueing and the cancelled publication never regrants
native history.

* chore(gateway): raise test-count rows for tests merged from main

* test(s3select): use the owned JSON input types in merged planner tests

* fix(ecstore): scope the merged direct-writer test hook to test builds

* fix(ecstore): create parents before admitted truncate opens

Align the heal tests with removal-disabled preservation and local-only outcome
fields, and refresh the e2e-full selection for the synced tests.

* chore(gateway): update test-count baseline after main sync

Main #9241 removed the node pool-metadata readiness cache with 8 tests
and added 6, so the rustfs row drops from 6440 to 6438. The ecstore and
s3select rows rise for tests added by #9237, #9239 and #9241.

* feat(scripts): add junit_diff.py for dual-stack JUnit comparison (#9275)

* test(e2e): allow a pinned release binary as differential reference side

RUSTFS_E2E_REFERENCE_BINARY makes side A of the dual-stack differential a released rustfs binary, started without RUSTFS_S3_STACK. A missing, non-file or non-executable path fails the run instead of falling back to the same-binary mode. The report records which side is the reference and each side's --version output.

* ci(gateway): make the test-count ratchet a floor, not exact (#9278)

ci(gateway): make the test-count ratchet a floor instead of an exact baseline

Tests below a row still fail; tests above it, a member without a row and ignores below the row now pass with a note. Ignores above a row still fail. Adds contract tests wired into make script-tests.

* chore(gateway): record ignored microbenchmarks merged from main (#9290)

chore(gateway): record the ignored microbenchmarks merged from main

main added three throughput microbenchmarks that are ignored by design
(#9217): one each in rustfs-checksums, rustfs-rio and rustfs-utils. The
test-count ratchet treats a higher ignored count as a failure until the
row is raised, so raise those three rows to 1 and tighten the test rows
to the merged tree with --update.

* test(e2e): update smoke selection for reference-binary unit tests

* ci(s3-tests): add reference parity lane against pinned 1.0.5 (#9288)

* chore: restore main's Cargo and baseline files after merging main

---------

Co-authored-by: rustfs-cla-assistant[bot] <273121464+rustfs-cla-assistant[bot]@users.noreply.github.com>
2026-11-11 08:28:14 -05:00
Chris An 9fe0ff6d80 feat(gateway-adapter): audit refusals the gateway enforces (#9298)
* feat(server): add the RUSTFS_S3_STACK selector and dual-stack e2e lane (#8411)

* refactor(server): group the s3s-compat tower layers under legacy_compat (#8410)

Move the ten tower layers that patch s3s behaviour out of
rustfs/src/server/layer.rs into rustfs/src/server/legacy_compat/, one file
per layer with its tests. Each file header names the s3s behaviour it
patches, the gateway mechanism that replaces it (Replaced by:), and the task
that removes it (Removed by: T2.12, rustfs/backlog#2771). The layers keep
their code, their order in external_service_stack!, and their behaviour; no
layer is deleted.

Add server::http::tests::stack_census, which renders the ordered service
chain of the real external_service_stack! expansion and compares it with a
committed constant, so a reorder, insertion or removal is a visible diff.

Route the two s3s error types the moved layers use through a
storage_api::server::legacy_compat shim so the split adds no s3s-importing
file, and lower the s3s footprint baselines to the measured values (files 209
to 208, s3_error! lines 1586 to 1585). Keep the server runtime-source bypass
guard covering the moved code.

Refs rustfs/backlog#2787, rustfs/backlog#2734

* feat(s3-types): add the RustFS-owned S3 error type and s3_error! macro (#8406)

`rustfs-s3-types` now provides `S3ErrorCode` (generated by build.rs from
src/codes.txt, the grep-derived call surface, plus `Custom`), `S3Error`,
`S3Result`, `StdError` and the `s3_error` macro with the same call syntax
as s3s, so the T1.2 codemod is a `use` swap. Statuses are hand-written and
pinned to the s3s behaviour RustFS ships today by an oracle test that uses
s3s as a dev-dependency only. The `From` conversions for the legacy edge
live behind the off-by-default `compat-s3s` feature (the orphan rule
forbids them in rustfs/src/error.rs); `rustfs` enables it.

Refs rustfs/backlog#2742, rustfs/backlog#2734

* ci: add the per-crate test-count ratchet (#8404)

Quick Checks now fails any PR that lowers the number of test attributes
in a workspace crate or adds #[ignore], measured per crate against the
exact rows of .config/test-count-baseline.txt (rustfs/backlog#2734
section 1 rule 3). The counter widens the spec's literal
#[(tokio::)?test] to the tokio attributes that carry arguments and both
spellings of #[test_case(...)], and anchors at line start so
commented-out attributes do not count. The initial baseline covers 52
crates, 21077 tests and 164 ignored.

Refs rustfs/backlog#2738
Refs rustfs/backlog#2734

* test(ecstore): pin the persisted bucket-config XML bytes for 13 families (#8408)

Before any DTO or codec is replaced, pin the exact bytes that
deserialize followed by serialize produces for every persisted
bucket-configuration XML family in its three source shapes (AWS
documentation, MinIO-written, RustFS-written), and the bytes RustFS
writes today for a typed value. Byte equality only: a structural
comparison would let a codec replacement change the persisted encoding
unnoticed.

The generator script refuses to overwrite existing goldens without
--force, refuses on dirty codec paths, cannot report success from its
write mode, and records the generating commit in the fixtures README.

Refs rustfs/backlog#2744, rustfs/backlog#2734

* ci: raise the test-count baseline to the integration tree (#8426)

* ci: make the s3s footprint baseline exact and lower-only (#8403)

* ci: make the s3s footprint baseline exact and lower-only

The s3s footprint guard compared three counters against ceilings held in
the script itself, so a ceiling could sit above the tree and a drift was
fixable by raising it (rustfs/rustfs#5964 raised the s3_error! ceiling
to go green). The three values now live in
.config/s3s-footprint-baseline.txt as files, s3_error_lines and
ecstore_files, the script reads only that file, and a measurement that
differs from its baseline in either direction fails: growth prints
"s3s footprint grew: <measured> > <baseline>", shrinkage prints
"lower the baseline to <measured> in this PR". The counting method is
unchanged; the baseline is the current measurement of
gateway/integration (209 / 1585 / 36). The header records the end state:
once all three reach 0, T4.2 switches the script to failing on any
occurrence.

Refs rustfs/backlog#2739, rustfs/backlog#2734

* ci: add the s3s edge allowlist mode to the footprint guard (#8405)

`scripts/check_s3s_footprint.sh --mode allowlist` fails on every Rust
file that references s3s (`s3s::` or `s3s_sigv4`) and is not matched by
an entry of the new `.config/s3s-edge-allowlist.txt`; `--dry-run` lists
those files and exits 0. The allowlist carries the initial edge from the
spec, and the script rejects an entry that names or matches anything
under crates/ other than crates/e2e_test. The baseline mode and the
argument-less CI invocation are unchanged.

Refs rustfs/backlog#2741, rustfs/backlog#2734

* test(ecstore): reach the bucket-config codec through the test facade (#8428)

* refactor(s3-types): own the HTTP Body and drop s3s from three crates (#8431)

Add rustfs_s3_types::Body, a BoxBody<Bytes, StdError> wrapper with
empty() and From<Bytes | Vec<u8> | String | &'static str>, and the
legacy From conversions to and from s3s::Body behind compat-s3s.

The signer is now generic over the request body: it only reads and
writes the request head, so it needs no body type and its callers keep
theirs. s3-client sends its requests with the owned Body, madmin signs a
bodiless head, and targets drops its two S3Request/S3Response header
wrappers in favour of extract_params_header.

rustfs-signer, rustfs-madmin and rustfs-targets no longer depend on
s3s; the s3s footprint baseline drops from 209 to 202 files.

Refs rustfs/backlog#2747, rustfs/backlog#2734

* refactor(ecstore): move S3ErrorCode to rustfs-s3-types by codemod (#8432)

* test(ecstore): pin the InvalidRange responses of new_getobjectreader

Both error branches of new_getobjectreader build an ErrorResponse with S3ErrorCode::InvalidRange, and no test observed either code. Pin the code, its wire name and the location fields of each branch before the code type moves to rustfs-s3-types (rustfs/backlog#2743), so a changed code on either branch fails a test. The ecstore test-count row rises by the two tests.

* chore(scripts): add the s3s-to-rustfs-s3-types error codemod

Task T1.2 of rustfs/backlog#2734 (rustfs/backlog#2743) moves S3Error, S3ErrorCode, S3Result and s3_error! from s3s to rustfs-s3-types by script, one crate closure per PR. The codemod rewrites use trees (splitting mixed s3s imports) and qualified code paths, never comments or literals, adds the rustfs-s3-types dependency, drops s3s only from a crate that no longer names it, refuses every file of the s3s edge allowlist, takes --hold for a file whose type is fixed outside the closure, and prints a per-crate census. A second run changes nothing. Its self-test runs under make script-tests.

* refactor(ecstore): take S3ErrorCode from rustfs-s3-types

Output of scripts/codemods/s3_error_to_s3_types.py over crates/ecstore, crates/lifecycle, crates/scanner and crates/replication (rustfs/backlog#2743 batches 1 and 2). The ErrorResponse code field, the two InvalidRange constructions in new_getobjectreader and an unused tier import now use rustfs_s3_types::S3ErrorCode; the variant names and wire names are unchanged. lifecycle, scanner and replication reference none of the four error names. warm_backend.rs and warm_backend_s3.rs are held: they compare against rustfs_s3_client::ErrorResponse.code, which stays the s3s type until s3-client moves.

* ci(footprint): count only s3_error! lines still on s3s's macro

s3_error_lines counted every line holding the text `s3_error!`, which
also matches the RustFS-owned `rustfs_s3_types::s3_error!`. A codemod
that moves call sites onto rustfs-s3-types therefore could not lower it
(rustfs/backlog#2743, decision 3; rustfs/rustfs#8432, gotcha 2).

The counter now applies a textual, per-file and per-line rule that errs
on the side of counting (full rule in the script header):
- occurrences spelled `rustfs_s3_types::s3_error!` never count;
- a line holding `s3s::s3_error!` always counts;
- any other `s3_error!` line counts unless its file is cleared, that is,
  it imports the macro from rustfs_s3_types by name on a line starting
  with the use item (or defines it with `macro_rules! s3_error`) and does
  not also bring s3s's macro in (`use s3s::s3_error`, a `use s3s::{..}`
  tree naming it or a crate-root `*`, `use s3s::*`, or
  `#[macro_use] extern crate s3s`);
- fail closed: a file matching neither form (`use super::*`, a
  crate-local re-export, a renamed or glob rustfs_s3_types import, a
  commented-out import, a comment-only mention) counts.

Census of the 1585 lines on 96fc06776: 1478 in 64 files that import the
macro from s3s; 107 in 15 files matching neither form (105 through
`use super::*`, 1 qualified `s3s::s3_error!`, 1 comment line); none in
files that import it from rustfs_s3_types or define it. Every line stays
counted, so the baseline remains s3_error_lines=1585 (files=202 and
ecstore_files=36 unchanged). Rewriting rustfs/src/auth.rs's import to
rustfs_s3_types lowers the new counter to 1567; the old one stayed 1585.

`--self-test` (wired into `make script-tests`) copies the script into
22 fixture trees: 6 positive cases and 16 negative ones (lines that must
not count, forms that must not clear a file, and a missing baseline,
missing key or failing classification search that must fail). Each tree
has a control line that must count, and each run gets s3s text on stdin.
Seven cases were red against the old counter. 19 mutants each turn their
case red: old definition, flipped import test, dropped fail-closed
branch, dropped e2e_test exclusion, dropped s3s veto, dropped qualified
s3s rule, dropped rustfs-qualified strip, unanchored rustfs import,
renamed import accepted, nested glob taken as root glob, single-line
s3s trees only, dropped s3s glob forms, dropped macro_use form, dropped
definition clearing, ignored classification rg errors, error-line search
without its path, classification searches without their path, missing
baseline skipped, missing key skipped.

* test(e2e): compare two RUSTFS_S3_STACK servers over every S3 operation (#8435)

Add dual_stack_differential to e2e_test. It starts two rustfs processes
from the same binary, stack A on RUSTFS_S3_STACK=legacy and stack B on
RUSTFS_E2E_STACK_B (default legacy until rustfs/backlog#2734 T2.5),
sends both the same aws-sdk-s3 script covering the 73 methods of
impl S3 for FS (a normal request and a 4xx negative each), and compares
status, error code, an allowlist of response headers and a few typed
body fields. Error message text is never compared. Each run writes
<target>/dual-stack/<run>.json with compared_ops and the (op, field,
a, b) differences; any difference fails the test.

differential_detects_an_injected_difference starts B with
RUSTFS_REGION=eu-west-1 and requires GetBucketLocation to be reported.
The comparator is pure and unit-tested without a server.

Wire the module into the e2e-smoke filter and update the e2e-smoke and
e2e-full selection digests and the e2e_test test-count row.

Refs rustfs/backlog#2740

* feat(app): add the RustFS-owned RequestEnvelope and its s3s constructor (#8437)

* feat(app): add the RustFS-owned RequestEnvelope and its s3s constructor

* ci: classify the s3s edge adapters in storage/s3_compat as interface

* feat(s3-types): add Body::from_stream and move Swift off s3s::Body (#8438)

rustfs/backlog#2747 batch 3. `Body::from_stream` wraps a chunk stream of
unknown length over `StreamBody`/`BoxBody`: one data frame per chunk, the
first error ends the body, the size hint stays unknown, and dropping the
body drops the stream. Parity tests pin it to the s3s `StreamingBlob::wrap`
it replaces, at the `http_body` level and byte for byte through hyper.

The Swift handlers in rustfs-protocols build their responses as the owned
`Body`. `SwiftService` converts them into the wrapped S3 service's response
body type through `From<Body>`, so rustfs keeps serving `s3s::Body` at the
legacy edge unchanged. Request bodies stay Send-only (`UnsyncBoxBody`); the
two storage writers and the DLO/SLO segment streams get `Sync` from
`sync_wrapper::SyncStream`, which is what s3s's `http_body_unsync` mutex
provided before. The DTO and error uses of s3s in protocols are untouched
(T1.5 and T1.2).

* refactor(protocols): move error types to rustfs-s3-types by codemod (#8439)

T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and
the error items of protocols move off s3s by scripts/codemods, run with
--hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it
downcasts the backend error, whose runtime type is still s3s::S3Error.

- rustfs/src/protocols/client.rs: the two webdav session methods now
  return rustfs_s3_types::S3Result and convert the s3s error with From.
- codemod: a child module that calls s3_error! through `use super::*`
  gets its own `use rustfs_s3_types::s3_error;` once its parent imports
  that macro by name; otherwise it is listed as GLOB.
- s3-types: a table test pins code, status and the s3s error the legacy
  edge renders for every variant a moved call site names.
- s3_error_lines 1585 -> 1584 (a lifecycle comment reworded).

* test(connect): restore one-second drive request budgets

Apply the exact test-only repair already landed on main in 5972a09cc3. The filesystem probe and production CLI success test request one second and keep their existing one-second duration and parameter checks. All outcome, I/O, error, cleanup and resource-limit assertions remain.

Scope: rustfs/tests/connect_perf_drive.rs only. Patch-id: cb0a1dfc8f68a67727525adea7458e2cee7d09f8.

Validation environment: LANG=en_US.UTF-8, CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0, exclusive APFS-cloned CARGO_TARGET_DIR.

Quick Checks: every command from .github/actions/quick-checks/action.yml, with captured output and exit status below.

$ shellcheck --version && actionlint
ShellCheck - shell script analysis tool
version: 0.11.0
license: GNU General Public License, version 3
website: https://www.shellcheck.net
exit_code=0

$ cargo fmt --all --check
exit_code=0

$ ./scripts/check_unsafe_code_allowances.sh
exit_code=0

$ ./scripts/check_layer_dependencies.sh
Layer dependency guard passed (no new reverse dependencies/cycles).
exit_code=0

$ ./scripts/check_architecture_migration_rules.sh
Architecture migration rules passed.
exit_code=0

$ ./scripts/check_logging_guardrails.sh
✅ Logging guardrails check passed
exit_code=0

$ ./scripts/check_error_other_format_ratchet.sh
error(format!) ratchet passed (total call sites: 582).
exit_code=0

$ ./scripts/check_no_tokio_io_uring.sh
exit_code=0

$ ./scripts/check_extension_schema_boundaries.sh
Extension schema boundaries passed.
exit_code=0

$ ./scripts/check_body_cache_whitelist.sh
Body-cache whitelist guard passed.
exit_code=0

$ ./scripts/check_s3s_footprint.sh
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)
✅ s3s footprint baseline check passed
exit_code=0

$ ./scripts/check_test_count_ratchet.sh
test-count ratchet OK: 52 crates, 21222 tests, 164 ignored match .config/test-count-baseline.txt
exit_code=0

$ ./scripts/check_fips_wording.sh
FIPS wording guard passed (README.md and CHANGELOG.md contain no forbidden claims).
KMS wording guard passed (crates/kms claims no Vault KV2 Transit wrapping).
exit_code=0

$ ./scripts/check_embedded_secrets.sh
Embedded secret guard passed (no private key material or provider credential literal in the tree).
exit_code=0

$ make script-tests
Running script tests...
entrypoint credential policy tests passed
internode grpc A/B bench env tests passed
object batch benchmark enhanced tests passed
PASS package service script contract
hotpath warp A/B gate tests passed
hotpath warp ABBA tests passed
Scanner validation artifacts written to /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.1pFe6Kl1Yu/out
Scanner validation artifacts written to /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.1pFe6Kl1Yu/out-missing-pid
.............................
----------------------------------------------------------------------
Ran 29 tests in 181.286s

OK
{"decision":"verified","pending_gates":["G01","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G02"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-E"}
{"decision":"verified","pending_gates":["G01","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G02"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-E"}
{"decision":"verified","pending_gates":["G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G01"}
{"decision":"verified","pending_gates":["G01","G02","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G03"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-L"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G14"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G14"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G10"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P1"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P2"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P3"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G05"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G06"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-D"}
Status-and-outcome release descriptor verified: /private/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp8rwn1pe0/descriptor/release-bundle-status-outcome.json
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G05"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G06"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-D"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G07"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G08"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P4"}
exact 1MiB handoff ABBA tests passed
pinned paired ABBA bench tests passed
==> /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/bin/path_containment (-max_total_time=7, -seed=123456789, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/explicit-artifacts/path_containment/, corpus=/private/tmp/gw-orchestrator-2734/ci-drive/fuzz/corpus/path_containment)
-seed=123456789
target=path_containment
seed=123456789
max_total_time=7
git_revision=4d15b816236630b2c42f321ffc9ed3654edc94e3
git_dirty=true
runner_mode=prebuilt
==> /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/bin/path_containment (-max_total_time=1, -seed=2322617706, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/auto-artifacts/path_containment/, corpus=/private/tmp/gw-orchestrator-2734/ci-drive/fuzz/corpus/path_containment)
-seed=2322617706
==> cargo +nightly fuzz run path_containment (-max_total_time=9, -seed=987654321, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/cargo-artifacts/path_containment/)
+nightly
fuzz
run
path_containment
-seed=987654321
seed=987654321
runner_mode=cargo-fuzz
fuzz runner seed manifest tests passed
✅ scripts/python_bin.sh resolver checks passed
self-test ok: clean fixture with every exemption present passes
self-test ok: all 11 pattern families fire
self-test ok: moving a pinned source is reported
Embedded secret guard self-test passed.
self-test ok: p1-s3s-import: 'use s3s::s3_error;' counts every line, the rustfs_s3_types module's too (s3_error_lines=3)
self-test ok: p2-s3s-mixed-tree: a multi-line s3s use tree naming s3_error counts (s3_error_lines=3)
self-test ok: p3-s3s-glob: a crate-root glob of s3s, bare or inside a tree, counts (s3_error_lines=5)
self-test ok: p4-macro-use: '#[macro_use] extern crate s3s' counts (s3_error_lines=3)
self-test ok: p5-qualified-s3s: a qualified s3s::s3_error! line counts in a cleared file (s3_error_lines=2)
self-test ok: p6-exact-pass: a baseline equal to the tree passes (exit 0)
self-test ok: n1-rustfs-import: 'use rustfs_s3_types::s3_error;' alone counts nothing (s3_error_lines=1)
self-test ok: n2-rustfs-mixed-tree: a rustfs_s3_types tree plus an s3s tree without the macro counts nothing (s3_error_lines=1)
self-test ok: n3-defines: the file defining the macro counts nothing (s3_error_lines=1)
self-test ok: n4-qualified-rustfs: a qualified rustfs_s3_types::s3_error! line counts nothing (s3_error_lines=1)
self-test ok: n5-renamed-call: a call through a renamed macro is not text-matched (out of scope) (s3_error_lines=1)
self-test ok: n6-comment-string: a comment or string mentioning s3_error! in a cleared file counts nothing (s3_error_lines=1)
self-test ok: n7-e2e-excluded: crates/e2e_test is excluded (s3_error_lines=1)
self-test ok: n8-super-glob: an unrecognised import form ('use super::*;') counts (s3_error_lines=3)
self-test ok: n9-facade: a crate-local re-export counts (s3_error_lines=2)
self-test ok: n10-renamed-import: a renamed rustfs_s3_types import does not clear s3_error! lines (s3_error_lines=2)
self-test ok: n11-commented-import: a commented-out rustfs_s3_types import does not clear (s3_error_lines=2)
self-test ok: n12-rustfs-glob: a glob import of rustfs_s3_types does not clear (s3_error_lines=2)
self-test ok: n13-comment-only: a comment mentioning s3_error! in a file with no import counts (s3_error_lines=2)
self-test ok: f1-baseline-missing: a missing baseline file fails (exit 1)
self-test ok: f2-key-missing: a baseline without s3_error_lines fails (exit 1)
self-test ok: f3-rg-fails: a failing classification search fails the guard (exit 1)
✅ s3s footprint self-test passed
test_core_gate_rejects_missing_ignored_filtered_and_corrupt_inputs (__main__.SelfTests.test_core_gate_rejects_missing_ignored_filtered_and_corrupt_inputs) ... ok
test_e2e_requires_registration (__main__.SelfTests.test_e2e_requires_registration) ... ok
test_fuzz_runtime_uses_matrix_target (__main__.SelfTests.test_fuzz_runtime_uses_matrix_target) ... ok
test_ilm_lane_keeps_the_measured_cargo_build_budget (__main__.SelfTests.test_ilm_lane_keeps_the_measured_cargo_build_budget) ... ok
test_profile_listing_binds_platform_digest (__main__.SelfTests.test_profile_listing_binds_platform_digest) ... ok
test_profile_listing_enforces_selection (__main__.SelfTests.test_profile_listing_enforces_selection) ... ok
test_quick_checks_commands_propagate_failure (__main__.SelfTests.test_quick_checks_commands_propagate_failure) ... ok
test_quick_checks_rejects_caller_and_execution_bypasses (__main__.SelfTests.test_quick_checks_rejects_caller_and_execution_bypasses) ... ok
test_runner_readiness_fails_closed (__main__.SelfTests.test_runner_readiness_fails_closed) ... ok
test_s3_runner_rejects_unbounded_failure_locals (__main__.SelfTests.test_s3_runner_rejects_unbounded_failure_locals) ... ok
test_scanner_heal_begin_requires_embedded_source_provenance (__main__.SelfTests.test_scanner_heal_begin_requires_embedded_source_provenance) ... ok
test_scanner_heal_case_does_not_approve_pending_release (__main__.SelfTests.test_scanner_heal_case_does_not_approve_pending_release) ... ok
test_scanner_heal_case_lane_is_required_for_runner_profile (__main__.SelfTests.test_scanner_heal_case_lane_is_required_for_runner_profile) ... ok
test_scanner_heal_case_only_schema_cannot_approve_release (__main__.SelfTests.test_scanner_heal_case_only_schema_cannot_approve_release) ... ok
test_scanner_heal_crash_case_rejects_restart_oracle (__main__.SelfTests.test_scanner_heal_crash_case_rejects_restart_oracle) ... ok
test_scanner_heal_ec84_case_rejects_wrong_erasure_geometry (__main__.SelfTests.test_scanner_heal_ec84_case_rejects_wrong_erasure_geometry) ... ok
test_scanner_heal_finish_collects_oracles_from_registry (__main__.SelfTests.test_scanner_heal_finish_collects_oracles_from_registry) ... ok
test_scanner_heal_multipool_case_accepts_deferred_optional_outage_write (__main__.SelfTests.test_scanner_heal_multipool_case_accepts_deferred_optional_outage_write) ... ok
test_scanner_heal_pending_gate_cannot_map_to_implemented_lane (__main__.SelfTests.test_scanner_heal_pending_gate_cannot_map_to_implemented_lane) ... ok
test_scanner_heal_receipts_reject_reuse_and_missing_builds (__main__.SelfTests.test_scanner_heal_receipts_reject_reuse_and_missing_builds) ... ok
test_scanner_heal_rejects_boolean_fractional_and_out_of_geometry_integers (__main__.SelfTests.test_scanner_heal_rejects_boolean_fractional_and_out_of_geometry_integers) ... ok
test_scanner_heal_rejects_broken_execution_and_artifacts (__main__.SelfTests.test_scanner_heal_rejects_broken_execution_and_artifacts) ... ok
test_scanner_heal_rejects_copied_junit_and_wrong_suite_build (__main__.SelfTests.test_scanner_heal_rejects_copied_junit_and_wrong_suite_build) ... ok
test_scanner_heal_release_bundle_accepts_complete_measured_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_accepts_complete_measured_evidence) ... ok
test_scanner_heal_release_bundle_assembler_copies_measured_artifacts (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_copies_measured_artifacts) ... ok
test_scanner_heal_release_bundle_assembler_merges_measured_descriptors (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_merges_measured_descriptors) ... ok
test_scanner_heal_release_bundle_assembler_rejects_duplicate_descriptor_gate (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_duplicate_descriptor_gate) ... ok
test_scanner_heal_release_bundle_assembler_rejects_fixture_descriptor (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_fixture_descriptor) ... ok
test_scanner_heal_release_bundle_assembler_rejects_hash_mismatch (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_hash_mismatch) ... ok
test_scanner_heal_release_bundle_assembler_rejects_mixed_descriptor_revisions (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_mixed_descriptor_revisions) ... ok
test_scanner_heal_release_bundle_enforces_topology_duration_profile_and_versions (__main__.SelfTests.test_scanner_heal_release_bundle_enforces_topology_duration_profile_and_versions) ... ok
test_scanner_heal_release_bundle_fixture_writer_is_not_release_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_fixture_writer_is_not_release_evidence) ... ok
test_scanner_heal_release_bundle_gate_accepts_single_measured_gate (__main__.SelfTests.test_scanner_heal_release_bundle_gate_accepts_single_measured_gate) ... ok
test_scanner_heal_release_bundle_gate_rejects_missing_field (__main__.SelfTests.test_scanner_heal_release_bundle_gate_rejects_missing_field) ... ok
test_scanner_heal_release_bundle_gate_rejects_unknown_gate (__main__.SelfTests.test_scanner_heal_release_bundle_gate_rejects_unknown_gate) ... ok
test_scanner_heal_release_bundle_rejects_synthetic_or_missing_fields (__main__.SelfTests.test_scanner_heal_release_bundle_rejects_synthetic_or_missing_fields) ... ok
test_scanner_heal_release_bundle_requires_domain_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_requires_domain_evidence) ... ok
test_scanner_heal_release_bundle_requires_field_provenance (__main__.SelfTests.test_scanner_heal_release_bundle_requires_field_provenance) ... ok
test_scanner_heal_release_bundle_requires_json_artifact_provenance (__main__.SelfTests.test_scanner_heal_release_bundle_requires_json_artifact_provenance) ... ok
test_scanner_heal_release_bundle_requires_mixed_version_field_roles (__main__.SelfTests.test_scanner_heal_release_bundle_requires_mixed_version_field_roles) ... ok
test_scanner_heal_release_bundle_requires_same_gate_measurement_window (__main__.SelfTests.test_scanner_heal_release_bundle_requires_same_gate_measurement_window) ... ok
test_scanner_heal_release_consumes_multiple_registry_oracles (__main__.SelfTests.test_scanner_heal_release_consumes_multiple_registry_oracles) ... ok
test_scanner_heal_release_matrix_lanes_remain_pending_without_real_evidence (__main__.SelfTests.test_scanner_heal_release_matrix_lanes_remain_pending_without_real_evidence) ... ok
test_scanner_heal_release_requirements_cannot_be_cleared (__main__.SelfTests.test_scanner_heal_release_requirements_cannot_be_cleared) ... ok
test_scanner_heal_release_status_rejects_focused_case_run (__main__.SelfTests.test_scanner_heal_release_status_rejects_focused_case_run) ... ok
test_scanner_heal_release_status_rejects_synthetic_case (__main__.SelfTests.test_scanner_heal_release_status_rejects_synthetic_case) ... ok
test_scanner_heal_required_evidence_fields_cannot_be_removed (__main__.SelfTests.test_scanner_heal_required_evidence_fields_cannot_be_removed) ... ok
test_scheduled_alerts_require_completion_watchdog (__main__.SelfTests.test_scheduled_alerts_require_completion_watchdog) ... ok
test_update_profile_selection_changes_only_requested_platform (__main__.SelfTests.test_update_profile_selection_changes_only_requested_platform) ... ok
test_validate_includes_quick_checks (__main__.SelfTests.test_validate_includes_quick_checks) ... ok
test_vault_tests_require_cross_process_serialization (__main__.SelfTests.test_vault_tests_require_cross_process_serialization) ... ok
test_workflow_readiness_requires_dependency_probe (__main__.SelfTests.test_workflow_readiness_requires_dependency_probe) ... ok

----------------------------------------------------------------------
Ran 52 tests in 14.301s

OK
e2e-full selection OK: 1 tests, sha256=29e30e6ed2ed38d9e5b0a7cdd9536abf5c719f7021a02b578ae1381563a3c273
...................................
----------------------------------------------------------------------
Ran 35 tests in 1.161s

OK
...................
----------------------------------------------------------------------
Ran 19 tests in 17.397s

OK
..........
----------------------------------------------------------------------
Ran 10 tests in 0.183s

OK
test_actual_selector_bootstrap_uses_base_policy_and_fails_closed (__main__.SelfTests.test_actual_selector_bootstrap_uses_base_policy_and_fails_closed) ... ok
test_documentation_paths_do_not_hide_build_or_fixture_changes (__main__.SelfTests.test_documentation_paths_do_not_hide_build_or_fixture_changes) ... ok
test_e2e_scope_keeps_production_and_shared_configuration_full (__main__.SelfTests.test_e2e_scope_keeps_production_and_shared_configuration_full) ... ok
test_e2e_shortcut_rejects_new_direct_renamed_and_target_dependencies (__main__.SelfTests.test_e2e_shortcut_rejects_new_direct_renamed_and_target_dependencies) ... ok
test_e2e_shortcut_traverses_implicit_path_dependencies (__main__.SelfTests.test_e2e_shortcut_traverses_implicit_path_dependencies) ... ok
test_event_contract_requires_complete_candidate_and_optional_lanes (__main__.SelfTests.test_event_contract_requires_complete_candidate_and_optional_lanes) ... ok
test_every_wrong_result_missing_job_or_selection_fails_closed (__main__.SelfTests.test_every_wrong_result_missing_job_or_selection_fails_closed) ... ok
test_full_e2e_gate_preserves_workflow_branch_and_event_scope (__main__.SelfTests.test_full_e2e_gate_preserves_workflow_branch_and_event_scope) ... ok
test_git_range_includes_deleted_source_and_rename_origins (__main__.SelfTests.test_git_range_includes_deleted_source_and_rename_origins) ... ok
test_job_ids_and_display_names_cannot_hide_validation (__main__.SelfTests.test_job_ids_and_display_names_cannot_hide_validation) ... ok
test_report_only_stack_lane_mirrors_e2e_smoke_outside_the_aggregate (__main__.SelfTests.test_report_only_stack_lane_mirrors_e2e_smoke_outside_the_aggregate) ... ok
test_repository_wiring_and_missing_dependency_regression (__main__.SelfTests.test_repository_wiring_and_missing_dependency_regression) ... ok
test_verify_command_preserves_failures (__main__.SelfTests.test_verify_command_preserves_failures) ... ok

----------------------------------------------------------------------
Ran 13 tests in 3.280s

OK
security coverage self-test passed
test_api_failures_preserve_other_evidence_and_never_enter_grace (__main__.SelfTests.test_api_failures_preserve_other_evidence_and_never_enter_grace) ... ok
test_attempt_outcome_does_not_replace_recent_success (__main__.SelfTests.test_attempt_outcome_does_not_replace_recent_success) ... ok
test_cancelled_attempt_cannot_refresh_expired_success (__main__.SelfTests.test_cancelled_attempt_cannot_refresh_expired_success) ... ok
test_cli_requires_the_repository_default_branch (__main__.SelfTests.test_cli_requires_the_repository_default_branch) ... ok
test_config_rejects_duplicate_and_invalid_entries (__main__.SelfTests.test_config_rejects_duplicate_and_invalid_entries) ... ok
test_freshness_boundaries (__main__.SelfTests.test_freshness_boundaries) ... ok
test_grace_requires_two_successful_queries_with_no_history (__main__.SelfTests.test_grace_requires_two_successful_queries_with_no_history) ... ok
test_invalid_api_evidence_fails_closed (__main__.SelfTests.test_invalid_api_evidence_fails_closed) ... ok
test_report_retains_every_workflow (__main__.SelfTests.test_report_retains_every_workflow) ... ok
test_requests_filter_schedule_default_branch_and_success_on_server (__main__.SelfTests.test_requests_filter_schedule_default_branch_and_success_on_server) ... ok

----------------------------------------------------------------------
Ran 10 tests in 0.051s

OK
..............................
----------------------------------------------------------------------
Ran 30 tests in 22.354s

OK
..........
----------------------------------------------------------------------
Ran 10 tests in 2.969s

OK
.................................
----------------------------------------------------------------------
Ran 33 tests in 0.160s

OK
Online functional runner labels: pf-testing
.......
----------------------------------------------------------------------
Ran 7 tests in 0.002s

OK
.......
----------------------------------------------------------------------
Ran 7 tests in 0.001s

OK
....
----------------------------------------------------------------------
Ran 4 tests in 0.128s

OK
...
----------------------------------------------------------------------
Ran 3 tests in 0.547s

OK
PASS: cold stampede script self-test
exit_code=0

$ python3 ./scripts/test_object_generation_protocol_model.py
test_a_recovery_barrier_cannot_choose_an_empty_value (__main__.ObjectGenerationProtocolModel.test_a_recovery_barrier_cannot_choose_an_empty_value) ... ok
test_all_bounded_two_proposer_interleavings_choose_at_most_one_value (__main__.ObjectGenerationProtocolModel.test_all_bounded_two_proposer_interleavings_choose_at_most_one_value) ... ok
test_an_unavailable_decision_quorum_cannot_choose_a_candidate (__main__.ObjectGenerationProtocolModel.test_an_unavailable_decision_quorum_cannot_choose_a_candidate) ... ok
test_counter_allocation_requires_persistence_and_is_consumed_once (__main__.ObjectGenerationProtocolModel.test_counter_allocation_requires_persistence_and_is_consumed_once) ... ok
test_counter_restart_burns_persisted_allocations_before_or_after_issue (__main__.ObjectGenerationProtocolModel.test_counter_restart_burns_persisted_allocations_before_or_after_issue) ... ok
test_crash_after_promise_before_reply_does_not_restore_an_older_ballot (__main__.ObjectGenerationProtocolModel.test_crash_after_promise_before_reply_does_not_restore_an_older_ballot) ... ok
test_duplicate_or_unconfigured_voters_cannot_form_a_quorum (__main__.ObjectGenerationProtocolModel.test_duplicate_or_unconfigured_voters_cannot_form_a_quorum) ... ok
test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision (__main__.ObjectGenerationProtocolModel.test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision) ... ok
test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected (__main__.ObjectGenerationProtocolModel.test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected) ... ok
test_forgetting_tombstone_authority_allows_an_aba_create (__main__.ObjectGenerationProtocolModel.test_forgetting_tombstone_authority_allows_an_aba_create) ... ok
test_issuing_before_persistence_reuses_a_ballot_after_restart (__main__.ObjectGenerationProtocolModel.test_issuing_before_persistence_reuses_a_ballot_after_restart) ... ok
test_lost_ack_and_restart_preserve_the_chosen_operation (__main__.ObjectGenerationProtocolModel.test_lost_ack_and_restart_preserve_the_chosen_operation) ... ok
test_minority_accepted_work_is_adopted_and_finished (__main__.ObjectGenerationProtocolModel.test_minority_accepted_work_is_adopted_and_finished) ... ok
test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample (__main__.ObjectGenerationProtocolModel.test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample) ... ok
test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it (__main__.ObjectGenerationProtocolModel.test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it) ... ok
test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor (__main__.ObjectGenerationProtocolModel.test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor) ... ok
test_replacing_voters_with_empty_state_can_choose_a_conflicting_value (__main__.ObjectGenerationProtocolModel.test_replacing_voters_with_empty_state_can_choose_a_conflicting_value) ... ok
test_skipping_predecessor_recovery_chooses_a_broken_revision_chain (__main__.ObjectGenerationProtocolModel.test_skipping_predecessor_recovery_chooses_a_broken_revision_chain) ... ok
test_stale_publication_rollback_and_gc_cannot_change_the_winner (__main__.ObjectGenerationProtocolModel.test_stale_publication_rollback_and_gc_cannot_change_the_winner) ... ok
test_superseded_operation_identity_cannot_be_reused_for_different_input (__main__.ObjectGenerationProtocolModel.test_superseded_operation_identity_cannot_be_reused_for_different_input) ... ok
test_superseded_operation_retry_returns_its_original_successor (__main__.ObjectGenerationProtocolModel.test_superseded_operation_retry_returns_its_original_successor) ... ok
test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision (__main__.ObjectGenerationProtocolModel.test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision) ... ok
test_tombstone_rejects_a_delayed_never_present_create (__main__.ObjectGenerationProtocolModel.test_tombstone_rejects_a_delayed_never_present_create) ... ok
test_unlearned_predecessor_is_recovered_before_the_second_slot (__main__.ObjectGenerationProtocolModel.test_unlearned_predecessor_is_recovered_before_the_second_slot) ... ok

----------------------------------------------------------------------
Ran 24 tests in 0.958s

OK
model: states=51023, transitions=167520, terminal=732, chosen_candidates=2
negative control: double-choice trace=((0, 'prepare', 0), (0, 'prepare', 1), (0, 'prepare', 2), (0, 'accept', 0), (0, 'accept', 1), (0, 'accept', 2), (1, 'prepare', 0), (1, 'prepare', 1), (1, 'prepare', 2), (1, 'accept', 0), (1, 'accept', 1), (1, 'accept', 2))
multi-slot round schedules: 640
exit_code=0

$ python3 ./scripts/check_test_wiring.py
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired
exit_code=0

$ ./scripts/check_no_planning_docs.sh
exit_code=0

$ ./scripts/check_uring_lane_lib_only.sh
OK: no 'uring_'-matching test functions under crates/ecstore/tests; --lib is safe
exit_code=0

Initial complete gate
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
 Nextest run ID e07fc5b9-c6c8-4cdd-ae61-54839bf24437 with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.016s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.011s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.009s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.009s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  16.950s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   4.025s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.008s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.007s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.035s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  21.087s] 10 tests run: 10 passed, 0 skipped
exit_code=0

Restored complete gate
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
 Nextest run ID 233b7307-5361-4eb7-94d5-b384f17d2d6a with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.009s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.008s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.008s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.008s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  19.518s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   4.146s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.009s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.018s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.009s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.054s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  23.787s] 10 tests run: 10 passed, 0 skipped
exit_code=0

Mutation M1: success_measurement reports max(actual elapsed, requested budget + 1), forcing 1001 ms for the one-second request. The retained duration assertion at rustfs/tests/connect_perf_drive.rs:105 fails.
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast -E 'test(real_local_write_read_is_measured_and_scratch_is_removed)'
info: experimental features enabled: setup-scripts
   Compiling rustfs v1.0.1 (/private/tmp/gw-orchestrator-2734/ci-drive/rustfs)
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs") generated 1 warning
warning: `rustfs` (bin "rustfs-cli") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 7m 06s
────────────
 Nextest run ID 9f618777-7bf6-4cbc-8803-dd7f50f449ed with nextest profile: ci
    Starting 1 test across 1 binary (9 tests skipped)
        FAIL [   0.016s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
  stdout ───

    running 1 test
    test real_local_write_read_is_measured_and_scratch_is_removed ... FAILED

    failures:

    failures:
        real_local_write_read_is_measured_and_scratch_is_removed

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 0.01s

  stderr ───

    thread 'real_local_write_read_is_measured_and_scratch_is_removed' (106986846) panicked at rustfs/tests/connect_perf_drive.rs:105:5:
    assertion failed: (1..=1_000).contains(&data.duration_millis)
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

────────────
     Summary [   0.017s] 1 test run: 0 passed, 1 failed, 9 skipped
        FAIL [   0.016s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
error: test run failed
exit_code=100

Mutation M2: target_parameters reports requested duration + 1. The in-process assertion at rustfs/tests/connect_perf_drive.rs:111 and CLI JSON assertion at rustfs/tests/connect_perf_drive.rs:527 both reject 1001 against 1000.
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast -E 'test(real_local_write_read_is_measured_and_scratch_is_removed) | test(production_cli_measures_and_signs_exact_binary_provenance)'
info: experimental features enabled: setup-scripts
   Compiling rustfs v1.0.1 (/private/tmp/gw-orchestrator-2734/ci-drive/rustfs)
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs") generated 1 warning
warning: `rustfs` (bin "rustfs-cli") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 7m 20s
────────────
 Nextest run ID b73d46ab-c910-4e1e-8063-7fc7e332ec15 with nextest profile: ci
    Starting 2 tests across 1 binary (8 tests skipped)
        FAIL [  13.968s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
  stdout ───

    running 1 test
    test production_cli_measures_and_signs_exact_binary_provenance ... FAILED

    failures:

    failures:
        production_cli_measures_and_signs_exact_binary_provenance

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 13.96s

  stderr ───

    thread 'production_cli_measures_and_signs_exact_binary_provenance' (107015462) panicked at rustfs/tests/connect_perf_drive.rs:527:5:
    assertion `left == right` failed
      left: Number(1001)
     right: 1000
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

        FAIL [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
  stdout ───

    running 1 test
    test real_local_write_read_is_measured_and_scratch_is_removed ... FAILED

    failures:

    failures:
        real_local_write_read_is_measured_and_scratch_is_removed

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 0.01s

  stderr ───

    thread 'real_local_write_read_is_measured_and_scratch_is_removed' (107016667) panicked at rustfs/tests/connect_perf_drive.rs:111:5:
    assertion `left == right` failed
      left: 1001
     right: 1000
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

────────────
     Summary [  13.985s] 2 tests run: 0 passed, 2 failed, 8 skipped
        FAIL [  13.968s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        FAIL [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
error: test run failed
exit_code=100

Transient production mutations were restored byte-for-byte before the complete green rerun. Original/restored SHA256: 8d389e6d8836c041aab8c6a93e56addd65ba3f70b33264e1c74f639d0572a2da.

$ git diff --check

exit_code=0

$ git diff --cached --check

exit_code=0

Independent orchestrator acceptance
Tested code tree: e67c72fb01

Tested tree: 16fbbd2ea4
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

$ LANG=en_US.UTF-8 CARGO_BUILD_JOBS=1 CARGO_INCREMENTAL=0 cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
info: experimental features enabled: setup-scripts
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs-cli") generated 1 warning
warning: `rustfs` (bin "rustfs") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.69s
────────────
 Nextest run ID cd1edf4c-f1c0-456d-ad1a-76cc00ce05c9 with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.012s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.010s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.010s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.009s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  20.328s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   5.527s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.013s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.019s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.011s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.049s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  25.991s] 10 tests run: 10 passed, 0 skipped

exit_code=0

All 54 tracked Cargo.toml files parse without duplicate keys. git diff --check passed. The production source remains unchanged; the two deliberate implementation mutations above were restored before this complete green run.

* fix(test): compare legacy body wire parity as raw bytes

Lossy UTF-8 comparison concealed a changed binary response byte.
Compare the already captured Vec<u8> values directly.

Validation:
- Binary mutant: migrated payload 0x84 vs legacy 0x83; old comparator
  passed 1/1, raw comparator failed 0/1 (exit 100); restored green.
- Non-Swift body-drain mutant: untouched-body delegation test failed
  with empty vs payload (exit 100); restored handler without a diff.
- cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols
  --features rustfs-protocols/swift --no-fail-fast:
  543 tests run, 543 passed, 0 skipped after both mutants restored.
- cargo fmt --all --check and git diff --check passed.

Orchestrator acceptance (LANG=en_US.UTF-8):
cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
Summary [2.445s] 543 tests run: 543 passed, 0 skipped.

All shared quick checks on the final code tree:
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

Final independent acceptance after the drive-test CI repair
Parent: a99d8f4f89
Tested code tree: 93cbd96bf0

Tested tree: 075ae5161c2b62a96812bc1e3c904a7663b4f2f8
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

$ LANG=en_US.UTF-8 CARGO_BUILD_JOBS=1 CARGO_INCREMENTAL=0 cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
    Finished `test` profile [unoptimized + debuginfo] target(s) in 10m 08s
 Nextest run ID 5c7ff102-ec6d-479a-86e0-810859b28bbf with nextest profile: ci
    Starting 543 tests across 8 binaries
     Summary [   6.729s] 543 tests run: 543 passed, 0 skipped
exit_code=0

The only source change is the raw-byte wire comparator. Cargo manifests and the ratchet baselines remain unchanged from the independently validated parent. The invalid-byte and consumed-request mutations remain recorded above; their implementation changes were restored.

* fix: repair integration checks and IPC test readiness

Wait for the RPC/locks test sockets to pass the existing owner and mode
checks inside the original 5s readiness window and 10ms poll interval.

Use the existing admin S3 carrier for the new recovery and quota error
calls. Regenerate the test-count baseline for main's 118 existing tests
across seven crates. Pair the private pagination helper's raw-scan inputs
and remove the redundant Copy clone from the retirement event.

Keep the original assertions, responses, authorization, locks and storage
operations. The pagination body and raw-scan expressions at all five calls
are byte-identical; reinserting only the Copy clone restores the entire
base heal file. All ignored baseline values and s3s counts stay unchanged.

Validation base: c47432e353
Validation tree: f081084bc5
Patch SHA-256: 2dcf0028691bfb2772edfb38e262829b04f4151832dcce823c9dd9f7434b52de
Trace production-prefix SHA-256: fd994f218a21429d62866f4efb8eee1ee57b58f4a51865aa35d05a54ef1ccb45

All commands used LANG=en_US.UTF-8, CARGO_BUILD_JOBS=1,
CARGO_INCREMENTAL=0 and CARGO_NET_OFFLINE=true in one exclusive target.
Every gate retained the same tree and six source hashes before and after.
54 Cargo.toml files parse without duplicate keys.

cargo clippy -p rustfs --all-targets -- -D warnings
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 12m 50s
Exit: 0. Wall time: 770.855s.

The original 19 quick-check steps:
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
Wall time: 286.337s.

cargo nextest run --profile ci -p rustfs --no-fail-fast
     Summary [ 217.017s] 5382 tests run: 5382 passed (4 leaky), 40 skipped
Exit: 0. Wall time: 1324.414s.

cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
     Summary [   2.189s] 543 tests run: 543 passed, 0 skipped
Exit: 0. Wall time: 123.192s.

cargo nextest run --profile ci -p rustfs-ecstore list_objects_paginate --no-fail-fast
     Summary [   0.170s] 8 tests run: 8 passed, 5921 skipped
Exit: 0. Wall time: 261.324s.

cargo nextest run --profile ci -p rustfs-heal durable_mrf_anchor_retires_only_after_exact_bucket_generation_is_deleted --no-fail-fast
     Summary [   0.805s] 1 test run: 1 passed, 835 skipped
Exit: 0. Wall time: 199.801s.

Baseline regeneration:
./scripts/check_test_count_ratchet.sh --update
wrote .config/test-count-baseline.txt: 52 crates, 21340 tests, 164 ignored
Test-only increases: ecstore48, heal3, iam14, io-metrics2, obs3, policy14,
rustfs34. Existing ignored values and all other rows are byte-identical.
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)

Historical runtime red controls on f622 (same retained two assertions):
cargo nextest run --profile ci -p rustfs --lib -E 'test(=connect::diagnostics::trace_runtime::tests::local_top_rpc_reads_service_http_completions_and_signs_offline) | test(=connect::diagnostics::trace_runtime::tests::local_top_locks_reads_service_manager_and_signs_offline)' --no-fail-fast
Original path readiness with a temporary 500ms bind/seal gap:
     Summary [   0.863s] 2 tests run: 0 passed, 2 failed, 5113 skipped
Exit: 100.
Sealed readiness in that same gap:
     Summary [   5.621s] 2 tests run: 2 passed, 5113 skipped
Exit: 0.
Compiled M1: replace the helper predicate with path.exists():
     Summary [   0.870s] 2 tests run: 0 passed, 2 failed, 5113 skipped
Exit: 100.
Restore the sealed predicate byte-for-byte:
     Summary [   5.640s] 2 tests run: 2 passed, 5113 skipped
Exit: 0.
Both reds observe mode755 with owner_matches=true and fail at the original
RPC/locks unwraps with StateSecurity. M1 failure locations: trace_runtime.rs
2547:29 and 2713:29. No new assertions were introduced.

cargo nextest run --profile ci -p rustfs --lib -E 'test(connect::diagnostics::trace_runtime::tests::)' --no-fail-fast
     Summary [  20.748s] 28 tests run: 28 passed (1 leaky), 5087 skipped
Exit: 0. This security sweep is historical; the full final unit gate above
covers the actual c474 candidate.
The first disk-stopped compile ran no tests and is excluded from red credit.
All temporary seal instrumentation is removed from the final source.

Final unit cleanup markers: 4 leaky, 40 original skips (exit0). The ld
__eh_frame linker warning also appears verbatim in the original f622 run
and the earlier c474 run; there are no new source warnings.

* fix: remove redundant pending closure in disk test

Pass the capture-free pending future factory as a function item in the existing slow-operation test. This fixes the redundant_closure error at disk_store.rs:2914 that stopped strict Clippy before the feature tests could run.

The single-line test edit preserves the factory's result, warning and timeout behavior. Reversing the replacement reproduces the parent file byte for byte. All 220 assertion lines and the production prefix are unchanged; no tests, assertions, guards, profiles or baselines were added or relaxed.

Original Linux CI evidence (run 37888552268 at 53245be9f):
- RIO job113685483992: strict Clippy exit101, redundant closure at disk_store.rs:2914; feature tests skipped.
- Workspace Test and Lint job113685484058: same strict Clippy exit101; nextest and doc tests skipped.
These are lint failures, not runtime mutation kills or known flakes.

Native macOS arm64 validation on tree7a5bf47e7533b29479bc85207416102f1337c566, LANG=en_US.UTF-8, one build job, CARGO_INCREMENTAL=0 and offline dependencies:

shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
All 19 checks: exit0, 312.161s.

$ cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 20m 11s
exit0, 1213.210s total.

$ cargo nextest run --profile ci -p rustfs-ecstore --features rio-v2 slow_disk_operation_warns_once_while_pending_and_preserves_result --no-fail-fast
Finished `test` profile [unoptimized + debuginfo] target(s) in 4m 34s
PASS [0.026s] rustfs-ecstore disk::disk_store::tests::slow_disk_operation_warns_once_while_pending_and_preserves_result
Summary [   0.170s] 1 test run: 1 passed, 5943 skipped
exit0, 281.920s total; 5943 existing filter skips.

warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
This linker warning is retained; flags, profiles and allowances are unchanged.

54 tracked Cargo.toml files parsed without duplicate keys. git diff --check passed. No new assertions were introduced, so no new runtime mutation is claimed. Native compilation and the existing helper test do not substitute for actual Linux RIO execution; the new integration CI run still needs to recover before the RPC merge.

Preservation and output hashes:
parent file SHA256: 8e59ee745e3aaaa22848b943d3b10f6353b515956282cde1f3af2c394200c9ac
final file SHA256: b69ba3237061b161448ae8b62b615885d5770b98a5b4a2479bcfbd70f1bd1f17
unchanged production prefix SHA256: 7903a6e288252c420231242d7b58a25fa2a7140ea3dd430cbe7f0dde11369dcb
RIO Linux failure log SHA256:421addb9868fd88e4109a922311e54f91f0dfd70862f8ba7fcaba193482ed90b
Workspace Linux failure log SHA256:e0aea8d0f014c022252a9534968e083c98cfc1fd760da445ecd04ec3f522e26a
19 output SHA256:35a54f28d707956ed1edb29d492c4dfe8a9fe753d103b111b5a2f1b5c41c2b98
RIO Clippy output SHA256:d6046e3868f2b3992fe11cdd04e9e98eb50a87d1234d9a70dbfeb154ff61638d
slow-operation output SHA256:605b7b258d2fd09c6ef2ab2ffc90d433d0509480f1e45125d20c823759e83608

Refs rustfs/backlog#2734.

* test(ecstore): isolate tier-free recovery fixture startup

The fixture constructor can take the production recovery lock and wait for 60 seconds before this test installs its queue and seeds metadata. Disable its background recovery only while creating this fixture, then restore the environment before calling the existing production recovery entrypoint.

Only the existing test setup changes. Its production entrypoint, default lock parameters, original assertions, 30-second timeout, cancellation and cleanup remain intact. No tests or assertions were added. Production and store/init.rs are byte-identical to the parent.

Independent acceptance ran on merge tree 8ec358cb42ba9ebb3bcfeafcef048a3f0366355b; the runner header reports parent HEAD, while before/after index and source hashes confirm this candidate. Worker task branch d8055766b8 has the same tree.

Every command used LANG=en_US.UTF-8. Task compilation also used CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0, CARGO_NET_OFFLINE=true, an independent target, and empty RUSTC_WRAPPER/RUSTC_WORKSPACE_WRAPPER to keep compilers in its owned process group without changing repository configuration.

Command:
shellcheck --version && actionlint
Exit 0; observed output excerpt:
version: 0.11.0
Log SHA-256: fc9eec16b4191d058a295813ca9afbe9d2b37b59975da860170726be162a3924

Command:
cargo fmt --all --check
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_unsafe_code_allowances.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_layer_dependencies.sh
Exit 0; observed output excerpt:
Layer dependency guard passed (no new reverse dependencies/cycles).
Log SHA-256: 69c6cb1896c0586b28d1167380d3260d99d8017a4265c8df5a2348f289e93a12

Command:
./scripts/check_architecture_migration_rules.sh
Exit 0; observed output excerpt:
Architecture migration rules passed.
Log SHA-256: 28790e70d070607ed4de0704f292a65030be92ec224ef106d4b72fd8296f896e

Command:
./scripts/check_logging_guardrails.sh
Exit 0; observed output excerpt:
✅ Logging guardrails check passed
Log SHA-256: 7adf41abff8ed8f7617decf86498888ddde78937099dd9ce47f6d359dfc2d317

Command:
./scripts/check_error_other_format_ratchet.sh
Exit 0; observed output excerpt:
error(format!) ratchet passed (total call sites: 579).
Log SHA-256: 66172aef18f150ba0605af122e26ffcf91f6dd86f43e38b05e2e982ce8d783ee

Command:
./scripts/check_no_tokio_io_uring.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_extension_schema_boundaries.sh
Exit 0; observed output excerpt:
Extension schema boundaries passed.
Log SHA-256: f13d0acd0e2ce5d9815241b12b4dab7d01cc11b9b65bcf6dfb7812c878532a51

Command:
./scripts/check_body_cache_whitelist.sh
Exit 0; observed output excerpt:
Body-cache whitelist guard passed.
Log SHA-256: 6b828847b0383950ee6eef9cd846d0c322c945e47efd4c7d1ad881178810d6bb

Command:
./scripts/check_s3s_footprint.sh
Exit 0; observed output excerpt:
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)
✅ s3s footprint baseline check passed
Log SHA-256: 47f53d04ef645618563fe1f22abc64667f19078f1974feedcca46e4f2c016844

Command:
./scripts/check_test_count_ratchet.sh
Exit 0; observed output excerpt:
test-count ratchet OK: 52 crates, 21340 tests, 164 ignored match .config/test-count-baseline.txt
Log SHA-256: 618a5e330e933de0c6ff484546df401457024a88bb5986d11d588794e33dab60

Command:
./scripts/check_fips_wording.sh
Exit 0; observed output excerpt:
FIPS wording guard passed (README.md and CHANGELOG.md contain no forbidden claims).
KMS wording guard passed (crates/kms claims no Vault KV2 Transit wrapping).
Log SHA-256: 99221cc9979c480ec4f46f50d76c52b18c72fe499f12e15e4f4a22a1b8bfb140

Command:
./scripts/check_embedded_secrets.sh
Exit 0; observed output excerpt:
Embedded secret guard passed (no private key material or provider credential literal in the tree).
Log SHA-256: 437a3ceba32094d2901889965ad191e411aad32cdb18bf0b8659e818011c406b

Command:
make script-tests
Exit 0; observed output excerpt:
test_cli_requires_the_repository_default_branch (__main__.SelfTests.test_cli_requires_the_repository_default_branch) ... ok
test_config_rejects_duplicate_and_invalid_entries (__main__.SelfTests.test_config_rejects_duplicate_and_invalid_entries) ... ok
test_freshness_boundaries (__main__.SelfTests.test_freshness_boundaries) ... ok
test_grace_requires_two_successful_queries_with_no_history (__main__.SelfTests.test_grace_requires_two_successful_queries_with_no_history) ... ok
test_invalid_api_evidence_fails_closed (__main__.SelfTests.test_invalid_api_evidence_fails_closed) ... ok
test_report_retains_every_workflow (__main__.SelfTests.test_report_retains_every_workflow) ... ok
test_requests_filter_schedule_default_branch_and_success_on_server (__main__.SelfTests.test_requests_filter_schedule_default_branch_and_success_on_server) ... ok
Ran 10 tests in 0.042s
OK
Ran 30 tests in 20.627s
OK
Ran 10 tests in 2.268s
OK
Ran 33 tests in 0.127s
OK
Ran 7 tests in 0.002s
OK
Ran 7 tests in 0.001s
OK
Ran 4 tests in 0.108s
OK
Ran 3 tests in 0.564s
OK
PASS: cold stampede script self-test
Log SHA-256: 75fda2d069a373c63667b908baaf99e6591aab9099a739b0c4c22ceb6da8d68c

Command:
python3 ./scripts/test_object_generation_protocol_model.py
Exit 0; observed output excerpt:
test_an_unavailable_decision_quorum_cannot_choose_a_candidate (__main__.ObjectGenerationProtocolModel.test_an_unavailable_decision_quorum_cannot_choose_a_candidate) ... ok
test_counter_allocation_requires_persistence_and_is_consumed_once (__main__.ObjectGenerationProtocolModel.test_counter_allocation_requires_persistence_and_is_consumed_once) ... ok
test_counter_restart_burns_persisted_allocations_before_or_after_issue (__main__.ObjectGenerationProtocolModel.test_counter_restart_burns_persisted_allocations_before_or_after_issue) ... ok
test_crash_after_promise_before_reply_does_not_restore_an_older_ballot (__main__.ObjectGenerationProtocolModel.test_crash_after_promise_before_reply_does_not_restore_an_older_ballot) ... ok
test_duplicate_or_unconfigured_voters_cannot_form_a_quorum (__main__.ObjectGenerationProtocolModel.test_duplicate_or_unconfigured_voters_cannot_form_a_quorum) ... ok
test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision (__main__.ObjectGenerationProtocolModel.test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision) ... ok
test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected (__main__.ObjectGenerationProtocolModel.test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected) ... ok
test_forgetting_tombstone_authority_allows_an_aba_create (__main__.ObjectGenerationProtocolModel.test_forgetting_tombstone_authority_allows_an_aba_create) ... ok
test_issuing_before_persistence_reuses_a_ballot_after_restart (__main__.ObjectGenerationProtocolModel.test_issuing_before_persistence_reuses_a_ballot_after_restart) ... ok
test_lost_ack_and_restart_preserve_the_chosen_operation (__main__.ObjectGenerationProtocolModel.test_lost_ack_and_restart_preserve_the_chosen_operation) ... ok
test_minority_accepted_work_is_adopted_and_finished (__main__.ObjectGenerationProtocolModel.test_minority_accepted_work_is_adopted_and_finished) ... ok
test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample (__main__.ObjectGenerationProtocolModel.test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample) ... ok
test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it (__main__.ObjectGenerationProtocolModel.test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it) ... ok
test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor (__main__.ObjectGenerationProtocolModel.test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor) ... ok
test_replacing_voters_with_empty_state_can_choose_a_conflicting_value (__main__.ObjectGenerationProtocolModel.test_replacing_voters_with_empty_state_can_choose_a_conflicting_value) ... ok
test_skipping_predecessor_recovery_chooses_a_broken_revision_chain (__main__.ObjectGenerationProtocolModel.test_skipping_predecessor_recovery_chooses_a_broken_revision_chain) ... ok
test_stale_publication_rollback_and_gc_cannot_change_the_winner (__main__.ObjectGenerationProtocolModel.test_stale_publication_rollback_and_gc_cannot_change_the_winner) ... ok
test_superseded_operation_identity_cannot_be_reused_for_different_input (__main__.ObjectGenerationProtocolModel.test_superseded_operation_identity_cannot_be_reused_for_different_input) ... ok
test_superseded_operation_retry_returns_its_original_successor (__main__.ObjectGenerationProtocolModel.test_superseded_operation_retry_returns_its_original_successor) ... ok
test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision (__main__.ObjectGenerationProtocolModel.test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision) ... ok
test_tombstone_rejects_a_delayed_never_present_create (__main__.ObjectGenerationProtocolModel.test_tombstone_rejects_a_delayed_never_present_create) ... ok
test_unlearned_predecessor_is_recovered_before_the_second_slot (__main__.ObjectGenerationProtocolModel.test_unlearned_predecessor_is_recovered_before_the_second_slot) ... ok
Ran 24 tests in 0.929s
OK
Log SHA-256: 0b4be926efb0609d497ae58930340153f01957b606206cbbdb3e0d47eb29d3a2

Command:
python3 ./scripts/check_test_wiring.py
Exit 0; observed output excerpt:
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired
Log SHA-256: 21f676f458451525535ab0ef73d6a9c0d9774e16d76a4d19e396ad34936dd39c

Command:
./scripts/check_no_planning_docs.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_uring_lane_lib_only.sh
Exit 0; observed output excerpt:
OK: no 'uring_'-matching test functions under crates/ecstore/tests; --lib is safe
Log SHA-256: 43f3f851e27ec2ab9e34d6b05fefdede38af992699c473904bfe6f53a46d52e3

Command:
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast
Exit 0; wall 234.399s.
Observed output:
    Finished `test` profile [unoptimized + debuginfo] target(s) in 3m 49s
    Starting 1 test across 1 binary (5871 tests skipped)
  SETUP PASS [   0.006s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"'
        PASS [   0.440s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item
     Summary [   0.586s] 1 test run: 1 passed, 5871 skipped
Log SHA-256: 34e973f86e8dc2102c1a0e483e0ac802ce7df3d1b40a603932f62c0805c4de27

Command:
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over) | test(=store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence) | test(=store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable)' --no-fail-fast
Exit 0; wall 4.010s.
Observed output:
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.43s
    Starting 3 tests across 1 binary (5869 tests skipped)
  SETUP PASS [   0.005s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"'
        PASS [   0.068s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over
        PASS [   0.637s] rustfs-ecstore store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence
        PASS [   0.846s] rustfs-ecstore store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable
     Summary [   0.987s] 3 tests run: 3 passed, 5869 skipped
Log SHA-256: b5da2a927ff18ca9aa217762f7186a6d4371bd9e8feedd7b77940e10b64dffac

Command:
cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
Exit 0; wall 1084.508s.
Observed output:
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 18m 03s
Log SHA-256: 28f98be7970faaa71cd118da1da4e6e2e3d23b9dda7e6449eaa7a37e741260b5

Compiled mutation command (unchanged for each red and restored run):
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast

M1: reverse only this test-local constructor flag from false to true. The mutant compiled, then failed the original recovered-item queue assertion after 30.481s. Restoring the entire observed candidate passed in 0.460s. Red log 55bf3b1535529ac0bcfb8071f9b56f9f40631007a85c315367e7e93786d65293; restored log 204d5fbe4574a7917f5e740a795d864fe28a7e7fb5e33d27b0a9cf68e2f17f52.

M2: replace only the actual awaited enqueue with false, consuming its original inputs. The mutant compiled, then failed the same original queue assertion after 30.516s. Restoring the entire observed candidate passed in 0.457s. Red log 89186330413215e231e2b06bf013cc31097ea8f4a106d91421a36d6c8986a7fd; restored log e848b42d2302401afa5dbcd9e37bf76d2e56a96ab6de5a488513b997f53c5e9b.

All temporary observation and mutation code was removed before final acceptance. Historical baseline and diagnostic failures remain recorded; they are not mutation credit. The separate Object Lock intent-retry diagnostic still fails, and these native macOS checks do not establish Linux CI recovery.

* refactor(rpc): serve internode rpc bodies with the owned Body (#8441)

* test(rpc): pin the wire bytes of every internode rpc response body

rustfs/backlog#2747 batch 4, step 1 of 2, on the s3s bodies the rpc paths
serve today. Each response body builder in storage/rpc/http_service.rs is
served through hyper's HTTP/1.1 server and the raw bytes the peer reads are
compared with goldens captured from this commit: status, content-length or
chunked framing, chunk boundaries, the terminating chunk, HEAD, and the cut
connection a failure after data produces. The capability responses carry a
per-process proof, so their length header and payload are checked against
the body they encode instead.

The read_file_stream response builder moves into read_file_stream_response
so the test reaches the same code the handler runs; behaviour is unchanged.

* refactor(rpc): serve internode rpc bodies with the owned Body

rustfs/backlog#2747 batch 4, step 2 of 2. The internode rpc HTTP paths
build their responses with rustfs_s3_types::Body: Body::from_stream
replaces Body::from(StreamingBlob::wrap(..)) at the six stream sites, and
the in-memory and empty bodies keep their constructors. InternodeRpcService
converts its responses into the wrapped service's response body through
From<Body>, so the server keeps composing it with the s3s service as
before. The wire-byte goldens from the previous commit pass unchanged;
storage/rpc no longer names s3s.

* fix(rpc): simplify wire test response builder type

Name the boxed response builder used by the capability wire test so
all-target Clippy can check it without a type-complexity warning.

Validation (LANG=en_US.UTF-8, CARGO_INCREMENTAL=0):
- cargo clippy -p rustfs --all-targets -- -D warnings: exit 0.
- cargo nextest run --profile ci -p rustfs --lib -E 'test(/storage::rpc::/)': 279 tests run, 279 passed, 4842 skipped.
- All 19 quick-check commands, including script-tests, both Python scripts and cargo fmt --all --check: exit 0.
- Footprint baseline regenerated: files=194, s3_error_lines=1584, ecstore_files=36; test-count baseline regenerated: 52 crates, 21228 tests, 164 ignored.
- All 54 Cargo.toml files parse without duplicate keys.

No assertion was added or changed by this lint fix. R1-R8 are retained
in the existing PR verification record.

* fix(ecstore): resume fenced Object Lock creation intents

* fix(ecstore): bound creation retry futures on the stack

* fix(ecstore): require native proof for creation retries

* fix(ecstore): bound bucket creation stack usage (#8466)

* fix(ecstore): retain creation fences through rollback

Keep lifecycle, metadata-transaction, and namespace guards around rollback and metadata cleanup. Preserve the original publication error when a guard is lost. Build the timestamp retry fixture through the real interrupted producer so it checks a valid pending creation.

Validated source tree: 4e8e207999.
Two independent source reviews found no blocking regression.

Validation (LANG=en_US.UTF-8):
- bash -c 'shellcheck --version && actionlint' — exit 0
- cargo fmt --all --check — exit 0
- ./scripts/check_unsafe_code_allowances.sh — exit 0
- ./scripts/check_layer_dependencies.sh — exit 0
- ./scripts/check_architecture_migration_rules.sh — exit 0
- ./scripts/check_logging_guardrails.sh — exit 0
- ./scripts/check_error_other_format_ratchet.sh — exit 0
- ./scripts/check_no_tokio_io_uring.sh — exit 0
- ./scripts/check_extension_schema_boundaries.sh — exit 0
- ./scripts/check_body_cache_whitelist.sh — exit 0
- ./scripts/check_s3s_footprint.sh — exit 0
- ./scripts/check_test_count_ratchet.sh — exit 0
- ./scripts/check_fips_wording.sh — exit 0
- ./scripts/check_embedded_secrets.sh — exit 0
- make script-tests — exit 0
- python3 ./scripts/test_object_generation_protocol_model.py — exit 0
- python3 ./scripts/check_test_wiring.py — exit 0
- ./scripts/check_no_planning_docs.sh — exit 0
- ./scripts/check_uring_lane_lib_only.sh — exit 0
- cargo nextest run --profile ci -p rustfs-ecstore --lib --locked --no-fail-fast — 5894 passed, 0 skipped
- cargo check -p rustfs-ecstore --all-targets --locked — exit 0
- cargo nextest run --profile ci -p rustfs --lib --locked --no-fail-fast -E 'test(=admin::handlers::table_catalog::tests::cold_strong_registration_completes_while_other_bucket_backup_waits) | test(=admin::handlers::table_catalog::tests::commit_backend_borrowed_put_preserves_owner_authorization_and_precondition) | test(=admin::handlers::table_catalog::tests::disable_table_bucket_preserves_objects_and_can_be_reenabled) | test(=admin::handlers::table_catalog::tests::load_table_handler_negotiates_vended_credentials_without_breaking_metadata_only_callers) | test(=admin::handlers::table_catalog::tests::namespace_property_handler_updates_object_backed_catalog_and_maps_errors) | test(=app::object::get::tests::execute_get_object_resumes_from_relocated_pool_without_splicing_body) | test(=table_catalog::store::tests::catalog_writer_loss_prevents_recovery_storage_commit)' — 7 passed
- cargo nextest run --profile ci -p rustfs --lib --locked --no-fail-fast -E 'test(unsupported_remote_disk_)' — 2 passed

Additional fault-injection evidence remains incomplete. Timestamp M1 reached its original assertion; its restore run is being collected. M2–M5 and the three planned rollback faults have not run. An automated risk check interrupted the timestamp proof task; no assertions or validation settings were weakened.

* fix(ecstore): box nested bucket rollback state

Keep the guarded rollback future behind a heap allocation so bucket creation does not store its nested state inline. The rollback body, all three fence references and pending-proof revocation order are unchanged.

The same three original Linux tests overflowed on both e611 CI attempts. Local format, ecstore test compilation and all 19 quick-check steps pass; the original Linux feature lanes still need to validate this candidate.

* test(ecstore): isolate rollback guard loss fixtures

Run the three rollback metadata-cleanup cases in separate test processes so reused /proc/self/fd aliases from earlier cases cannot invalidate their creation witnesses. Keep the original fixture, guards, assertions and io_uring test selection, and verify that each child runs the intended case successfully.

Update the generated test-count baseline for the added macro arm.

Validation: cargo check -p rustfs-ecstore --tests; cargo fmt --all --check; all 19 quick-check commands, including make script-tests and the two Python checks. Linux io_uring acceptance and fault controls remain pending.

* refactor(replication): move the replication config onto gateway types (#8443)

* wip: T1.4 replication family

* refactor(replication): move the replication config onto gateway types

The replication configuration family leaves s3s: rustfs-replication no longer depends on s3s, and ECStore bucket metadata, the scanner and the rustfs app/admin layers hold rustfs_gateway_types::persistence::PersistedReplicationConfiguration. The persisted XML goes through the gateway persistence codec behind a new BucketConfigXml trait (the 39 byte goldens pass with unchanged fixtures); the legacy s3s edge converts once in storage/ecfs.rs through storage/s3_compat/replication.rs; the scanner data usage cache keeps its MessagePack bytes through a serde mirror pinned by bytes recorded from s3s 5761ddfe. rustfs-gateway-types 0.32.0 is pinned once (rev 772d3385). Refs rustfs/backlog#2745.

* test: use S3 serializer for list response tests

* fix(ci): restore integration test compilation

Gate the diagnostic test import with its feature, initialize the conditional
MetadataWriteStore fields explicitly, and decode the scanner replication fixture
through its existing cache boundary. Preserve the original fixture and assertions.

Validation was run on tree 544acb80c0 at 6003d4b8.
LANG=en_US.UTF-8; no source changes follow validation.

Quick checks: 19/19 passed.
- shellcheck --version && actionlint: exit 0
- cargo fmt --all --check: exit 0
- ./scripts/check_unsafe_code_allowances.sh: exit 0
- ./scripts/check_layer_dependencies.sh: exit 0
- ./scripts/check_architecture_migration_rules.sh: exit 0
- ./scripts/check_logging_guardrails.sh: exit 0
- ./scripts/check_error_other_format_ratchet.sh: exit 0
- ./scripts/check_no_tokio_io_uring.sh: exit 0
- ./scripts/check_extension_schema_boundaries.sh: exit 0
- ./scripts/check_body_cache_whitelist.sh: exit 0
- ./scripts/check_s3s_footprint.sh: exit 0
- ./scripts/check_test_count_ratchet.sh: exit 0
- ./scripts/check_fips_wording.sh: exit 0
- ./scripts/check_embedded_secrets.sh: exit 0
- make script-tests: exit 0
- python3 ./scripts/test_object_generation_protocol_model.py: exit 0
- python3 ./scripts/check_test_wiring.py: exit 0
- ./scripts/check_no_planning_docs.sh: exit 0
- ./scripts/check_uring_lane_lib_only.sh: exit 0

Darwin validation:
- cargo clippy --locked -p rustfs-ecstore -p rustfs-scanner --all-targets -- -D warnings: exit 0
- cargo nextest run --locked -p rustfs-ecstore -p rustfs-scanner --lib -E 'test(config::com::tests::save_config_preserves_retryable_system_volume_errors) | test(scanner_io::tests::replication_maintenance_refreshes_compacted_stats_across_cycles_and_retries)' --test-threads 1 --no-fail-fast: exit 0, 2/2 passed
- cargo clippy --locked -p rustfs-ecstore -p rustfs-scanner --all-targets --features rustfs-ecstore/diagnostic-read-version-caller,rustfs-scanner/diagnostic-read-version-caller -- -D warnings: exit 101
- cargo nextest run --locked -p rustfs-ecstore -p rustfs-scanner --lib --features rustfs-ecstore/diagnostic-read-version-caller,rustfs-scanner/diagnostic-read-version-caller -E 'test(config::com::tests::save_config_preserves_retryable_system_volume_errors) | test(scanner_io::tests::replication_maintenance_refreshes_compacted_stats_across_cycles_and_retries) | test(cluster::rpc::runtime_sources::tests::) | test(config::com::tests::diagnostic_config_write_retains_origin_and_api_for_results_and_cancellation)' --test-threads 1 --no-fail-fast: exit 0, 14/14 passed

Diagnostic strict Clippy remains red on four unchanged production lints:
runtime_sources.rs if_same_then_else and three redundant_locals reports in
set_disk/core/metadata_read.rs and set_disk/read.rs. Their source ranges and
feature wiring match the base; the base was not compiled separately.

Linux aarch64 validation (Rust/Clippy 1.99.0, UID 501, jobs=1):
- cargo clippy --all-targets -- -D warnings: exit 0; Finished dev profile in 68m 12s
- cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings: exit 0; Finished dev profile in 49m 45s
Source HEAD/tree and clean status matched before and after both commands.
No compiler or stack overrides were used. protoc 35.1 was installed from its
verified ARM64 release; flatc was absent and not invoked by the workspace build scripts.
The local environment differs from hosted CI.

* fix(ecstore): recover delete batches with missing versions

Treat an explicit non-null version as a no-op when it is absent from both recovery sources. Keep rejecting missing null versions, delete-marker mismatches and conflicting local generations.

Add returning-disk and partial-receipt regressions, extend journal boundary checks, and regenerate the ecstore test-count baseline. Production faults exercise the 24 added assertion sites and 7 semantic checks: 33 faults failed at the expected sites and passed after full restoration.

Validation (LANG=en_US.UTF-8, one build job):

cargo nextest run --locked -p rustfs-ecstore --lib -E 'test(delete_recovery) | test(quorum_delete_cleanup_coordinator_binds_single_and_batch) | test(crash_delete_retry_batch_reclaims_inline_and_reserved_external)' --test-threads 1 --no-fail-fast
Summary [  34.580s] 69 tests run: 69 passed, 6058 skipped; exit 0

cargo clippy --locked -p rustfs-ecstore --all-targets -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 5m 37s; exit 0

All 19 final Quick Checks completed with exit 0:

- shellcheck --version && actionlint
- cargo fmt --all --check
- ./scripts/check_unsafe_code_allowances.sh
- ./scripts/check_layer_dependencies.sh
- ./scripts/check_architecture_migration_rules.sh
- ./scripts/check_logging_guardrails.sh
- ./scripts/check_error_other_format_ratchet.sh
- ./scripts/check_no_tokio_io_uring.sh
- ./scripts/check_extension_schema_boundaries.sh
- ./scripts/check_body_cache_whitelist.sh
- ./scripts/check_s3s_footprint.sh
- ./scripts/check_test_count_ratchet.sh
- ./scripts/check_fips_wording.sh
- ./scripts/check_embedded_secrets.sh
- make script-tests
- python3 ./scripts/test_object_generation_protocol_model.py
- python3 ./scripts/check_test_wiring.py
- ./scripts/check_no_planning_docs.sh
- ./scripts/check_uring_lane_lib_only.sh

test-count ratchet OK: 52 crates, 22009 tests, 165 ignored match .config/test-count-baseline.txt
PASS: cold stampede script self-test
model: states=51023, transitions=167520, terminal=732, chosen_candidates=2
multi-slot round schedules: 640
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired

* fix(ecstore): retry tier commit intent lock contention

Retry transient namespace lock timeouts within the existing three-attempt
CAS budget. Reload and verify the original mutation identity after
contention, and preserve the final typed error when attempts are exhausted.
Cover save/read contention, exhausted retries, replacement identities and
CAS reconciliation without changing the existing timeout or lock policy.

Validation:
- cargo nextest run --locked -p rustfs-ecstore --features test-util --lib -E 'test(tier_mutation)' --test-threads 1 --no-fail-fast: 52 passed.
- cargo nextest run --locked -p rustfs-ecstore --features test-util --lib -E 'test(intent_advance_) | test(=services::tier::tier::tests::caller_cancellation_after_durable_prepare_does_not_hide_the_mutation)' --test-threads 1 --no-fail-fast: 5 passed.
- cargo clippy --locked -p rustfs-ecstore --all-targets -- -D warnings: passed.
- cargo clippy --locked -p rustfs-ecstore --all-targets --features test-util -- -D warnings: passed.
- 27 mutation pairs failed at the intended assertions and passed after restoring the production code.
- bash scripts/check_test_count_ratchet.sh --update, then bash scripts/check_test_count_ratchet.sh: 52 crates, 22013 tests, 165 ignored; ecstore increases from 6581 to 6585.

* refactor(lifecycle): replace s3s domain types with Gateway DTOs

Keep lifecycle rules, restore requests, Object Lock, and scanner cached tags in domain types. Convert at the existing HTTP boundary while preserving legacy XML, scanner cache bytes, date ordering, and tombstones.

Linux test validation before the final import-path-only adjustment:

cargo nextest run -p rustfs-ecstore bucket_config_goldens
39 passed; 6211 skipped by selector.

cargo nextest run --lib -p rustfs -E 'test(/test_zero_rule_lifecycle_tombstone_recognition/) | test(/test_local_lifecycle_staleness_axis_selection/) | test(/test_lifecycle_staleness_preserves_legacy_submillisecond_axis/) | test(/test_merge_incoming_lifecycle/) | test(/test_lifecycle_expiry_statement_matrix/) | test(/test_lifecycle_expiry_subset_xml/) | test(/put_bucket_lifecycle_validation_errors_keep_their_s3_code/) | test(/execute_put_bucket_lifecycle_configuration_rejects_missing_configuration/) | test(/execute_restore_object_rejects_missing_restore_request/) | test(/execute_restore_object_rejects_select_type/) | test(/execute_restore_object_maps_failures_to_typed_s3_errors/) | test(/execute_restore_object_returns_internal_error_when_store_uninitialized/) | test(/restore_request_conversion_preserves_legacy_output_values/) | test(/restore_request_conversion_preserves_presence_and_raw_enums/) | test(/restore_request_conversion_distinguishes_absent_and_empty_containers/)'
25 passed; 5318 skipped by selector.

cargo nextest run --profile ci -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-replication -p rustfs-scanner
7872 passed; 21 skipped under the existing CI profile.

Previously verified mutation controls: 39 production faults, 41 RED/GREEN pairs.

cargo clippy --locked -p rustfs -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-scanner --all-targets -- -D warnings
Passed in 40m 45s before the import-path adjustment, then in 1m 59s on
the final source with warnings denied. The final adjustment imports the
existing helper through the object domain facade; fmt and the architecture
check also passed.

Part of rustfs/backlog#2745. The remaining ecstore configuration families follow separately.

* refactor(admin): move error types to rustfs-s3-types (#8440)

* refactor(protocols): move error types to rustfs-s3-types by codemod

T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and
the error items of protocols move off s3s by scripts/codemods, run with
--hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it
downcasts the backend error, whose runtime type is still s3s::S3Error.

- rustfs/src/protocols/client.rs: the two webdav session methods now
  return rustfs_s3_types::S3Result and convert the s3s error with From.
- codemod: a child module that calls s3_error! through `use super::*`
  gets its own `use rustfs_s3_types::s3_error;` once its parent imports
  that macro by name; otherwise it is listed as GLOB.
- s3-types: a table test pins code, status and the s3s error the legacy
  edge renders for every variant a moved call site names.
- s3_error_lines 1585 -> 1584 (a lifecycle comment reworded).

* refactor(admin): move error types to rustfs-s3-types (WIP, no build)

T1.2 batch 4 (rustfs/backlog#2743), work in progress: rustfs check
still fails with 16 errors (listed in the PR body).

- codemod: --facade PATH treats a crate-local re-export of s3s's error
  items like s3s for the four names (9 new self-tests, 9 mutants red).
- codemod output for rustfs/src/admin and rustfs/src/site_replication
  (--facade crate::storage_api::site_replication::s3); 6 glob children
  get an explicit rustfs_s3_types::s3_error import.
- admin/router.rs: the Operation trait returns rustfs_s3_types::S3Result.
- error.rs: From<ApiError> builds the rustfs-s3-types error; the s3s
  conversion is derived from it.

* fix(admin): convert error types at the s3s boundary

Batch 4 of T1.2 left the admin and site_replication modules returning
rustfs_s3_types::S3Error while their callers and callees on the storage
and auth side still speak s3s::S3Error. Convert at each edge with the
existing From impls in rustfs-s3-types (compat_s3s.rs) instead of
changing any code, message or status:

- admin/auth.rs, admin/handlers/object_zip_download.rs,
  admin/handlers/table_catalog/mod.rs: map the s3s result returned by
  check_key_valid / authorize_request / authorize_internal_object_request
  into the admin error type with .map_err(Into::into).
- storage/ecfs.rs: the s3s-facing side maps the admin site_replication
  gate results the other way.
- admin/handlers/pools.rs: ApiError.code stays s3s::S3ErrorCode, so the
  rustfs code is converted with .into() when the struct is built, in
  production and in the two tests that build or compare it.
- admin/handlers/heal.rs, admin/runtime_sources.rs: the router returns
  s3s errors (S3Route impl); the expected rustfs code is converted with
  .into() before the comparison.
- admin/handlers/table_catalog/mod.rs: iceberg_rest_error takes
  &'static str so the Cow<'static, str> custom code borrows it; every
  caller already passes an ICEBERG_ERROR_* constant or a &'static str.
- admin/handlers/quota.rs: drop a now same-type .into() after s3_error!.
- storage_api.rs, site_replication/mod.rs: drop re-exports and an import
  that no longer have users (clippy -D warnings).

cargo check -p rustfs --all-targets and cargo clippy -p rustfs
--all-targets -- -D warnings are clean; the s3s footprint baseline is
unchanged at files=172, s3_error_lines=303, ecstore_files=28.

Refs rustfs/backlog#2743, rustfs/backlog#2734.

* refactor(ecstore): move bucket configs onto gateway persisted types (#8709)

* refactor(ecstore): move bucket configs onto gateway persisted types

Finish the ECStore batch of T1.4 (rustfs/backlog#2745). The remaining
eleven bucket configuration families (Website, CORS, Tagging, Versioning,
ObjectLock, Encryption, Notification, Logging, Accelerate, RequestPayment,
PublicAccessBlock) now read and write their stored XML through the
rustfs-gateway-types persistence and cors_tagging codecs instead of the
s3s codec, so the legacy s3s bucket-config codec and its
s3gate-metadata-xml compat marker are removed.

- ecstore: BucketConfigXml is implemented by the gateway codec for every
  family; metadata, metadata_sys, versioning, object lock, replication and
  lifecycle consumers take the persisted types. Region moves to
  rustfs_s3_types::Region, and the ETag conversion to s3s moves to the
  rustfs edge with its tests. Only warm_backend.rs and warm_backend_s3.rs
  still reference s3s, held for T1.5 (rustfs/backlog#2746).
- scanner: object lock cache, item actions and data usage use the
  persisted types.
- rustfs: the legacy S3 edge converts each family between s3s DTOs and the
  persisted types in storage/storage_api.rs by writing one codec's XML and
  reading it with the other; Tagging converts field by field.
- Bump the rustfs-gateway-types pin to 21283e87.

The 39 persisted-byte goldens are unchanged and pass, so a document the
old codec wrote reads back and re-encodes to the same bytes. s3s footprint:
files 181 -> 146, ecstore_files 28 -> 2, s3_error_lines unchanged.

* test(app): prepare the PutBucketEncryption config from the request

The uninitialized-store test passed an empty persisted configuration, which the use case now validates before it reaches the store check, so it answered MalformedXML instead of InternalError. Convert the request's configuration the way the S3 edge does.

* style(app): format the PutBucketEncryption test

* refactor(s3-client): move remote error codes and DTO enums off s3s (#8783)

Part of T1.5 (rustfs/backlog#2746), batch 1.

* refactor(s3select): run queries on gateway DTO request shapes (#8794)

* refactor(s3select): run queries on an engine request built from gateway DTO shapes

Part of T1.5 (rustfs/backlog#2746), batch 2.

* fix(s3select): keep the select request conversion in the app layer

The layer guard forbids app -> storage; the conversion moves from storage/s3_compat/select.rs to app/select_request.rs unchanged.

* fix(heal): retry transient metadata conflicts (#8812)

* fix(heal): retry transient metadata conflicts

* chore: update heal test count baseline

* test: reserve nextest capacity for progress probes (#8818)

* test(e2e): use a non-default region for the differential negative control

Since main's region-aware CreateBucket and GetBucketLocation (#8739, #8743), us-east-1 answers an empty LocationConstraint and 200 for an owned re-create, so the control no longer reported its own region. Also refresh the e2e-full linux selection from the CI listing (682 tests).

* refactor(protocols): move StorageBackend to gateway types (#8825)

* refactor(protocols): move storage backend to gateway types

* test: update e2e test count baseline

* fix: preserve copy source keys and unknown lengths

* test: update protocols test count baseline

* fix(protocols): preserve typed SFTP backend errors

* test(protocols): use gateway list input in policy tests

* fix(protocols): clear clippy failures after T1.5 merge (#8831)

fix(protocols): clear clippy failures after merge

* fix(ecstore): finalize restore metadata after local copy-back (#8836)

* fix(ecstore): finalize restore metadata after local copy-back

* test(ecstore): ratchet added restore finalization cases

* fix(ecstore): keep restore copyback compatible with older peers

* fix(ecstore): gate restore copyback on fleet compatibility

* fix(ecstore): retain transition identity during restore copyback

* fix(ecstore): route committed restore copyback locally

* fix restore copyback cleanup

* fix(ecstore): honor restore probe test fallback

* refactor(storage): migrate option errors to domain types (#8839)

* refactor(s3): migrate private multipart validation errors (#8844)

* refactor(auth): derive policy conditions from AuthzFacts (#8442)

* refactor(auth): extract authorization facts

* test(e2e): update smoke selection baseline

* test(e2e): update full selection baseline

* refactor(admin): migrate health and prefix usage to owned requests (#8887)

* refactor(admin): add owned health router boundary

* refactor(admin): migrate prefix usage to owned request

* refactor(admin): complete the owned admin request boundary (#8949)

* refactor(admin): complete the owned admin request boundary

Every type implementing the owned admin `Operation` now serves the legacy
router through one blanket impl that converts the s3s request and response
at that single edge, so handlers register with `AdminOperation(&Handler {})`
and need no adapter statics; `LegacyAdminAdapter` is removed.

`AdminRequest` gains the accessors handlers read today (`remote_addr`,
`client_info`, `request_context`, `req_info`, `req_info_mut`,
`install_req_info`, `envelope_mut`, `take_body`), the request helpers gain
envelope forms (`authenticate_envelope_credentials`,
`object_store_from_envelope`, the table-catalog `*_from_envelope` helpers,
`KmsAdminAudit::from_envelope`, `AccountAuditContext::from_envelope`,
`CallerIdentity::resolve_envelope`), `read_compatible_admin_body` accepts
either body type, and the owned `Body` gains `store_all_limited` and
`Default`. `admin::operation::test_support::TestRequest` builds owned
requests for handler tests. Unused accessors carry a per-method dead-code
expectation that each migrating handler batch removes.

No handler behavior changes; the remaining migrations follow in batches.

* chore(admin): add the T1.12 admin handler codemod

The script rewrites an admin handler file from the s3s request and response
types to the owned boundary (imports, signatures, request field and helper
calls, body access, response construction, and test request literals) and
leaves a TODO comment at every site it cannot rewrite mechanically. It is
idempotent and is removed once T1.12 completes.

* refactor(admin): migrate replication and KMS handlers to owned requests (#8951)

Move the T1.12 batch 2 admin handlers (site_replication, kms_keys, event,
replication, diagnostics, on_demand_migration, audit, account, kms_rekey,
kms_key_lifecycle, object_data_cache, tls_debug, integrity) from the s3s
request/response types to the owned AdminRequest/AdminResponse boundary.
Handler logic, authorization order, error codes and response bytes are
unchanged; the request/response plumbing now goes through the envelope
accessors and helpers introduced by the owned boundary.

The shared response helpers (utils::json_response, utils::empty_response,
handlers::admin_json_response, target_descriptor::build_json_response,
profile::authorize_profile_request) and the site replication repair
executor still answer with the s3s types, so the migrated files carry
owned-boundary forms of those helpers and convert the repair response at
its call edge. The streaming console-log body and the two devnull drains
move from the s3s byte stream to the owned Body stream/frame API.

Drop the dead_code expectations for the owned accessors and envelope
helpers this batch starts using, and lower the s3s footprint baseline to
the measured 99 files.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* refactor(admin): add owned forms of the shared admin response helpers

Handler batches were copying `json_response`, `empty_response`,
`admin_json_response`, and `build_json_response` into each migrated file
because the shared helpers still return the s3s response. Provide owned
forms next to each legacy helper (`owned_json_response`,
`owned_empty_response`, `owned_admin_json_response`,
`owned_build_json_response`, `authorize_profile_envelope`) with parity
tests against the legacy shape, derive `Debug` for `AdminResponse` so
tests can use `expect_err`, and fix three codemod rules: strip owned
string conversions in test credentials, annotate non-literal URI parses,
and leave a rebound request parameter immutable.

* refactor(admin): use the shared owned response helpers in batch 2 (#8954)

Replace the per-file owned-boundary copies of json_response,
empty_response, admin_json_response, build_json_response and
authorize_profile_request in the batch 2 admin handlers with the shared
owned_* helpers and authorize_profile_envelope that the boundary branch
now provides, and restore expect_err in the handler tests now that
AdminResponse implements Debug; the `.err().expect(..)` stand-ins were
failing the base branch's clippy step (clippy::err_expect under
-D warnings). The site replication repair call edge
keeps its conversion because the repair executor still answers with the
s3s response type.

Drop the dead_code expectations on the shared owned helpers this batch
now uses in non-test code.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* fix(admin): preserve legacy body limit error responses

* refactor(admin): move policy, ILM and batch 3 handlers to owned requests (#8956)

Batch 3 of T1.12 (rustfs/backlog#2788, parent rustfs/backlog#2734) moves
14 admin handler files to AdminRequest/AdminResponse and the owned
Operation trait: ilm_transition, policies, idp_compat, config_admin,
plugins_instances, profile_admin, group, heal, object_zip_download,
kms_key_metadata, durability, inspect_archive, sts and
gateway_key_inventory. Handler logic, authorization order, error codes
and response bytes are unchanged; test request literals use
test_support::TestRequest.

The handlers use the shared owned helpers (owned_json_response,
owned_admin_json_response, owned_build_json_response,
authorize_profile_envelope). The dead_code expectations on the
AdminRequest accessors and RequestEnvelope::uri_mut they now exercise
are removed. The legacy target_descriptor::build_json_response keeps
only its equivalence test caller and carries the matching expectation
until it retires.

object_zip_download builds the s3s request the S3 gate still reads from
the owned request facts; heal authenticates through
authenticate_envelope_credentials; the trace, inspect-archive and zip
bodies use Body::from_stream. extract_plugin_instance_filters takes the
URI so extensions.rs keeps calling it, and the two idp_compat
delegations stay on the legacy trait until their delegates migrate.

The s3s footprint baseline drops from 99 to 90 files.

* test(admin): anchor the replication diff body contract on the call

The source-contract test looked for `let body = read_compatible_admin_body(`
on one line; rustfmt wrapped that binding after the handler moved to the
owned request boundary, so the marker was no longer found. Anchor on the
call itself so formatting cannot change what the contract checks.

* refactor(admin): migrate table catalog and batch 1 to owned requests (#8955)

* fix(admin): preserve catalog body limit response bytes

* refactor(admin): migrate identity and cluster handlers to owned requests (#8966)

* refactor(admin): migrate identity and cluster handlers to owned requests

Move the T1.12 batch 4 admin handlers (user, service_account, oidc, tier,
system, pools, quota, kms_dynamic, kms_backup, rebalance, module_switch,
bucket_meta, cluster_snapshot, realtime, bucket_recovery, profile) from the
s3s request/response types to the owned AdminRequest/AdminResponse
boundary. All 90 impl Operation blocks now take AdminRequest and return
AdminResponse; handler logic, authorization order, error codes and response
bytes are unchanged. The handlers use the shared owned helpers
(owned_json_response, owned_admin_json_response, authorize_profile_envelope,
the envelope-form runtime sources and audit constructors) instead of local
copies.

profile.rs drops the s3s forms of authorize_profile_request and
profile_not_implemented_response now that profile_admin.rs is on the
envelope gate too; profile_not_implemented_response answers with an
AdminResponse. The realtime metrics stream moves from the s3s ByteStream
adapter to Body::from_stream, and the rebalance idle-404 document still
renders through s3s::S3Error::to_http_response with its body converted at
the edge.

The idp_compat delegators ImportIamV2 and AddServiceAccountLdap move to the
owned trait as well: with their targets on both traits, the method-call
delegation became ambiguous between the owned impl and the router blanket
impl, so they now take the owned request directly.

KmsAdminAudit::from_request loses its last production caller with this batch
and is deleted; its comparison test now asserts the envelope facts directly.

Lower the s3s footprint baseline to the measured 68 files.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* refactor(admin): remove the legacy admin helpers batch 4 superseded

Batch 4 moved the last production callers of the s3s-typed admin helpers
to their owned forms, which left `admin_json_response`, `json_response`,
`authenticate_request`, `authorize_admin_request`, `object_store_from_req`
and `object_store_from_extensions` unused, along with the `Body`,
`S3Request` and `S3Response` re-exports of the admin `s3` facade. Remove
them. The owned forms keep their documentation; the behavior tests of the
removed `json_response` now exercise `owned_json_response`, and the
compat-prefix test asserts the owned response directly.

Two tests existed only to compare a removed legacy helper with its owned
twin (`owned_json_response_matches_the_legacy_shape`) or to cover the
removed legacy gate (`authorize_admin_request_without_credentials_is_rejected`);
they go with the helpers, and the owned equivalents stay. The rustfs
test-count row drops from 5957 to 5955 and the s3s footprint from 68 to 66
files.

* refactor(admin): drop the legacy target descriptor JSON response helper (#8968)

`target_descriptor::build_json_response` returned the s3s response and
had no production caller left once the event, audit and plugin instance
handlers moved to `owned_build_json_response`; it survived only for the
legacy/owned equivalence test. Remove it, give the owned helper its own
documentation, and make the test assert the owned response directly:
status, `application/json`, the echoed `x-request-id` when present and
only then, and the body bytes. The file no longer imports s3s, so the
footprint baseline drops from 66 to 65 files.

* test(ecstore): pin native history on cancellation under the cancellable queue

main moved the local write path onto the disk blocking queue, which skips an
operation once its waiter is dropped, so a borrowed write cancelled while it is
queued no longer creates its file. The integration-only native publication
test still expected the queued open to run. Assert the new execution model
instead (the open never runs) while keeping its safety claims: the Started
record is taken before queueing and the cancelled publication never regrants
native history.

* chore(gateway): raise test-count rows for tests merged from main

* test(s3select): use the owned JSON input types in merged planner tests

* fix(ecstore): scope the merged direct-writer test hook to test builds

* fix(ecstore): create parents before admitted truncate opens

Align the heal tests with removal-disabled preservation and local-only outcome
fields, and refresh the e2e-full selection for the synced tests.

* chore(gateway): update test-count baseline after main sync

Main #9241 removed the node pool-metadata readiness cache with 8 tests
and added 6, so the rustfs row drops from 6440 to 6438. The ecstore and
s3select rows rise for tests added by #9237, #9239 and #9241.

* feat(gateway-adapter): add IAM-backed CredentialProvider

Add the ring-2 crate rustfs-gateway-adapter with IamCredentials, the
gateway CredentialProvider answered by RustFS IAM (rustfs/backlog#2760,
credentials half; PolicySource is deferred).

- Pin rustfs-gateway at the existing gateway rev with
  default-features = false so the server crate stays out of the build.
- IamKeyLookup is a narrow IAM trait implemented for IamSys so the
  adapter is testable without an in-memory IAM backend.
- Root credentials are injected at construction and resolve before IAM
  is ready.
- Mapping: keystone: prefix -> NotFound (deliberate change from the
  legacy empty secret); IAM not ready or not initialized -> Unavailable;
  unknown key or NoSuch{User,ServiceAccount,TempAccount,Account} ->
  NotFound; other IAM errors -> Backend; invalid or "off" principals
  resolve disabled; STS credentials carry a SessionBinding, and an STS
  credential without an expiry is a Backend fault rather than a
  never-ending session.
- No authorization in lookup; no Debug on secret-holding types; IAM
  error text is dropped after classification.

* feat(scripts): add junit_diff.py for dual-stack JUnit comparison (#9275)

* feat(gateway-adapter): add IAM-backed PolicySource

Add IamPolicySource, the gateway PolicySource answered by RustFS IAM
(rustfs/backlog#2760, policy half). One snapshot per request carries a
RustfsPrincipal: principal kind, account, effective username, parent
user, credential groups, live group memberships, token claims and
is_owner.

- Anonymous callers get an empty snapshot without asking IAM.
- IAM not ready, not initialized or failing is PolicyError::unavailable,
  never an empty snapshot.
- Unknown, disabled and session-kind-mismatched principals are refused
  with constant labels; unreadable tokens are malformed.
- Claims are read from the stored token (root secret for STS, the
  account's own secret for service accounts), and is_owner mirrors the
  legacy has_root_access rule.
- RustfsPrincipal holds no secret or session token and has no Debug.

* ci(gateway): make the test-count ratchet a floor, not exact (#9278)

ci(gateway): make the test-count ratchet a floor instead of an exact baseline

Tests below a row still fail; tests above it, a member without a row and ignores below the row now pass with a note. Ignores above a row still fail. Adds contract tests wired into make script-tests.

* chore(gateway): record ignored microbenchmarks merged from main (#9290)

chore(gateway): record the ignored microbenchmarks merged from main

main added three throughput microbenchmarks that are ignored by design
(#9217): one each in rustfs-checksums, rustfs-rio and rustfs-utils. The
test-count ratchet treats a higher ignored count as a failure until the
row is raised, so raise those three rows to 1 and tighten the test rows
to the merged tree with --update.

* feat(gateway-adapter): add Observer and AuthzAuditSink adapters

Map the gateway's answered-request event onto the labels of
rustfs_s3_http_requests_total, bounded as the legacy counter bounds them,
and write one audit record per authorization decision (allow, deny and
indeterminate) in the shape of the legacy authorization-denial entry,
with the gateway's verified caller as accessKey.

Both adapters stop at RustFS-side sink traits. The legacy metric and
audit field names, with the file:line that emits each and the owner
that supplies it under the gateway, are committed as
tests/fixtures/observability-contract.json and held to the code.

* fix(gateway-adapter): share live root and refuse short secrets

Address the security review of the T2.1 adapters (rustfs/backlog#2760).

- Add RootCredentials (validated: root secret and STS signing key at
  least MIN_SECRET_KEY_LEN bytes), the RootSource trait and the settable
  RootCell. IamAdapters::new is now the only constructor of
  IamCredentials and IamPolicySource, so both read one live root source
  per request; no root source answers unavailable.
- A looked-up identity whose secret is shorter than 8 bytes is NotFound
  in the credential provider and DISABLED_PRINCIPAL in the policy
  source, whatever the lookup reports as valid.
- STS claims are read with the STS signing key from the root source
  instead of assuming it equals the root secret.
- Cover the off-status-alone branch, the OIDC owner exclusion, service
  accounts of root, the name live groups are read under, and another
  session's token.

* ci(s3-tests): add reference parity lane against pinned 1.0.5 (#9288)

* fix(gateway-adapter): effective secret length and unsplittable pair

Address the confirmation review of the T2.1 adapters
(rustfs/backlog#2760).

- One rule for every secret: its length after trailing NUL bytes are
  removed must be at least MIN_SECRET_KEY_LEN. HMAC zero-pads its key,
  so trailing NULs add nothing; interior NULs count. Applied to the root
  secret, the STS signing key and every looked-up credential.
- IamAdapters keeps both halves private and hands them to a builder
  together through the consuming install(builder, make_authenticator);
  compile-fail doctests show a mixed pair cannot be assembled.
- Any Fn() -> Option<Arc<RootCredentials>> is a RootSource, so the
  binary reads its live root accessor directly; RootCell is now a
  test-only helper.

* refactor(gateway-adapter): audit only refusals the gateway enforced

Drop the Observer adapter: request metrics stay with the outer tower
layer and the use-case code on both stacks, and the gateway already
logs every refusal it makes, so nothing is left for an observer to add.

Rework the AuthzAuditSink adapter to record only a Deny or
Indeterminate at an enforcing stage. An Allow and the GetObject
missing-object visibility probe produce no record. The audit switch
and the denial rate limiter are asked before a record is built, and
the record carries the legacy denial entry's fields only.

Remove the observability contract fixture and its tests.

* chore(gateway-adapter): drop the dependencies the observer needed

* docs(gateway-adapter): state what install guarantees about the provider handle

---------

Co-authored-by: rustfs-cla-assistant[bot] <273121464+rustfs-cla-assistant[bot]@users.noreply.github.com>
2026-11-11 08:27:16 -05:00
Chris An 8125128d05 test(app): split lifecycle tests by owner and add use-case seams (#9316)
* feat(server): add the RUSTFS_S3_STACK selector and dual-stack e2e lane (#8411)

* refactor(server): group the s3s-compat tower layers under legacy_compat (#8410)

Move the ten tower layers that patch s3s behaviour out of
rustfs/src/server/layer.rs into rustfs/src/server/legacy_compat/, one file
per layer with its tests. Each file header names the s3s behaviour it
patches, the gateway mechanism that replaces it (Replaced by:), and the task
that removes it (Removed by: T2.12, rustfs/backlog#2771). The layers keep
their code, their order in external_service_stack!, and their behaviour; no
layer is deleted.

Add server::http::tests::stack_census, which renders the ordered service
chain of the real external_service_stack! expansion and compares it with a
committed constant, so a reorder, insertion or removal is a visible diff.

Route the two s3s error types the moved layers use through a
storage_api::server::legacy_compat shim so the split adds no s3s-importing
file, and lower the s3s footprint baselines to the measured values (files 209
to 208, s3_error! lines 1586 to 1585). Keep the server runtime-source bypass
guard covering the moved code.

Refs rustfs/backlog#2787, rustfs/backlog#2734

* feat(s3-types): add the RustFS-owned S3 error type and s3_error! macro (#8406)

`rustfs-s3-types` now provides `S3ErrorCode` (generated by build.rs from
src/codes.txt, the grep-derived call surface, plus `Custom`), `S3Error`,
`S3Result`, `StdError` and the `s3_error` macro with the same call syntax
as s3s, so the T1.2 codemod is a `use` swap. Statuses are hand-written and
pinned to the s3s behaviour RustFS ships today by an oracle test that uses
s3s as a dev-dependency only. The `From` conversions for the legacy edge
live behind the off-by-default `compat-s3s` feature (the orphan rule
forbids them in rustfs/src/error.rs); `rustfs` enables it.

Refs rustfs/backlog#2742, rustfs/backlog#2734

* ci: add the per-crate test-count ratchet (#8404)

Quick Checks now fails any PR that lowers the number of test attributes
in a workspace crate or adds #[ignore], measured per crate against the
exact rows of .config/test-count-baseline.txt (rustfs/backlog#2734
section 1 rule 3). The counter widens the spec's literal
#[(tokio::)?test] to the tokio attributes that carry arguments and both
spellings of #[test_case(...)], and anchors at line start so
commented-out attributes do not count. The initial baseline covers 52
crates, 21077 tests and 164 ignored.

Refs rustfs/backlog#2738
Refs rustfs/backlog#2734

* test(ecstore): pin the persisted bucket-config XML bytes for 13 families (#8408)

Before any DTO or codec is replaced, pin the exact bytes that
deserialize followed by serialize produces for every persisted
bucket-configuration XML family in its three source shapes (AWS
documentation, MinIO-written, RustFS-written), and the bytes RustFS
writes today for a typed value. Byte equality only: a structural
comparison would let a codec replacement change the persisted encoding
unnoticed.

The generator script refuses to overwrite existing goldens without
--force, refuses on dirty codec paths, cannot report success from its
write mode, and records the generating commit in the fixtures README.

Refs rustfs/backlog#2744, rustfs/backlog#2734

* ci: raise the test-count baseline to the integration tree (#8426)

* ci: make the s3s footprint baseline exact and lower-only (#8403)

* ci: make the s3s footprint baseline exact and lower-only

The s3s footprint guard compared three counters against ceilings held in
the script itself, so a ceiling could sit above the tree and a drift was
fixable by raising it (rustfs/rustfs#5964 raised the s3_error! ceiling
to go green). The three values now live in
.config/s3s-footprint-baseline.txt as files, s3_error_lines and
ecstore_files, the script reads only that file, and a measurement that
differs from its baseline in either direction fails: growth prints
"s3s footprint grew: <measured> > <baseline>", shrinkage prints
"lower the baseline to <measured> in this PR". The counting method is
unchanged; the baseline is the current measurement of
gateway/integration (209 / 1585 / 36). The header records the end state:
once all three reach 0, T4.2 switches the script to failing on any
occurrence.

Refs rustfs/backlog#2739, rustfs/backlog#2734

* ci: add the s3s edge allowlist mode to the footprint guard (#8405)

`scripts/check_s3s_footprint.sh --mode allowlist` fails on every Rust
file that references s3s (`s3s::` or `s3s_sigv4`) and is not matched by
an entry of the new `.config/s3s-edge-allowlist.txt`; `--dry-run` lists
those files and exits 0. The allowlist carries the initial edge from the
spec, and the script rejects an entry that names or matches anything
under crates/ other than crates/e2e_test. The baseline mode and the
argument-less CI invocation are unchanged.

Refs rustfs/backlog#2741, rustfs/backlog#2734

* test(ecstore): reach the bucket-config codec through the test facade (#8428)

* refactor(s3-types): own the HTTP Body and drop s3s from three crates (#8431)

Add rustfs_s3_types::Body, a BoxBody<Bytes, StdError> wrapper with
empty() and From<Bytes | Vec<u8> | String | &'static str>, and the
legacy From conversions to and from s3s::Body behind compat-s3s.

The signer is now generic over the request body: it only reads and
writes the request head, so it needs no body type and its callers keep
theirs. s3-client sends its requests with the owned Body, madmin signs a
bodiless head, and targets drops its two S3Request/S3Response header
wrappers in favour of extract_params_header.

rustfs-signer, rustfs-madmin and rustfs-targets no longer depend on
s3s; the s3s footprint baseline drops from 209 to 202 files.

Refs rustfs/backlog#2747, rustfs/backlog#2734

* refactor(ecstore): move S3ErrorCode to rustfs-s3-types by codemod (#8432)

* test(ecstore): pin the InvalidRange responses of new_getobjectreader

Both error branches of new_getobjectreader build an ErrorResponse with S3ErrorCode::InvalidRange, and no test observed either code. Pin the code, its wire name and the location fields of each branch before the code type moves to rustfs-s3-types (rustfs/backlog#2743), so a changed code on either branch fails a test. The ecstore test-count row rises by the two tests.

* chore(scripts): add the s3s-to-rustfs-s3-types error codemod

Task T1.2 of rustfs/backlog#2734 (rustfs/backlog#2743) moves S3Error, S3ErrorCode, S3Result and s3_error! from s3s to rustfs-s3-types by script, one crate closure per PR. The codemod rewrites use trees (splitting mixed s3s imports) and qualified code paths, never comments or literals, adds the rustfs-s3-types dependency, drops s3s only from a crate that no longer names it, refuses every file of the s3s edge allowlist, takes --hold for a file whose type is fixed outside the closure, and prints a per-crate census. A second run changes nothing. Its self-test runs under make script-tests.

* refactor(ecstore): take S3ErrorCode from rustfs-s3-types

Output of scripts/codemods/s3_error_to_s3_types.py over crates/ecstore, crates/lifecycle, crates/scanner and crates/replication (rustfs/backlog#2743 batches 1 and 2). The ErrorResponse code field, the two InvalidRange constructions in new_getobjectreader and an unused tier import now use rustfs_s3_types::S3ErrorCode; the variant names and wire names are unchanged. lifecycle, scanner and replication reference none of the four error names. warm_backend.rs and warm_backend_s3.rs are held: they compare against rustfs_s3_client::ErrorResponse.code, which stays the s3s type until s3-client moves.

* ci(footprint): count only s3_error! lines still on s3s's macro

s3_error_lines counted every line holding the text `s3_error!`, which
also matches the RustFS-owned `rustfs_s3_types::s3_error!`. A codemod
that moves call sites onto rustfs-s3-types therefore could not lower it
(rustfs/backlog#2743, decision 3; rustfs/rustfs#8432, gotcha 2).

The counter now applies a textual, per-file and per-line rule that errs
on the side of counting (full rule in the script header):
- occurrences spelled `rustfs_s3_types::s3_error!` never count;
- a line holding `s3s::s3_error!` always counts;
- any other `s3_error!` line counts unless its file is cleared, that is,
  it imports the macro from rustfs_s3_types by name on a line starting
  with the use item (or defines it with `macro_rules! s3_error`) and does
  not also bring s3s's macro in (`use s3s::s3_error`, a `use s3s::{..}`
  tree naming it or a crate-root `*`, `use s3s::*`, or
  `#[macro_use] extern crate s3s`);
- fail closed: a file matching neither form (`use super::*`, a
  crate-local re-export, a renamed or glob rustfs_s3_types import, a
  commented-out import, a comment-only mention) counts.

Census of the 1585 lines on 96fc06776: 1478 in 64 files that import the
macro from s3s; 107 in 15 files matching neither form (105 through
`use super::*`, 1 qualified `s3s::s3_error!`, 1 comment line); none in
files that import it from rustfs_s3_types or define it. Every line stays
counted, so the baseline remains s3_error_lines=1585 (files=202 and
ecstore_files=36 unchanged). Rewriting rustfs/src/auth.rs's import to
rustfs_s3_types lowers the new counter to 1567; the old one stayed 1585.

`--self-test` (wired into `make script-tests`) copies the script into
22 fixture trees: 6 positive cases and 16 negative ones (lines that must
not count, forms that must not clear a file, and a missing baseline,
missing key or failing classification search that must fail). Each tree
has a control line that must count, and each run gets s3s text on stdin.
Seven cases were red against the old counter. 19 mutants each turn their
case red: old definition, flipped import test, dropped fail-closed
branch, dropped e2e_test exclusion, dropped s3s veto, dropped qualified
s3s rule, dropped rustfs-qualified strip, unanchored rustfs import,
renamed import accepted, nested glob taken as root glob, single-line
s3s trees only, dropped s3s glob forms, dropped macro_use form, dropped
definition clearing, ignored classification rg errors, error-line search
without its path, classification searches without their path, missing
baseline skipped, missing key skipped.

* test(e2e): compare two RUSTFS_S3_STACK servers over every S3 operation (#8435)

Add dual_stack_differential to e2e_test. It starts two rustfs processes
from the same binary, stack A on RUSTFS_S3_STACK=legacy and stack B on
RUSTFS_E2E_STACK_B (default legacy until rustfs/backlog#2734 T2.5),
sends both the same aws-sdk-s3 script covering the 73 methods of
impl S3 for FS (a normal request and a 4xx negative each), and compares
status, error code, an allowlist of response headers and a few typed
body fields. Error message text is never compared. Each run writes
<target>/dual-stack/<run>.json with compared_ops and the (op, field,
a, b) differences; any difference fails the test.

differential_detects_an_injected_difference starts B with
RUSTFS_REGION=eu-west-1 and requires GetBucketLocation to be reported.
The comparator is pure and unit-tested without a server.

Wire the module into the e2e-smoke filter and update the e2e-smoke and
e2e-full selection digests and the e2e_test test-count row.

Refs rustfs/backlog#2740

* feat(app): add the RustFS-owned RequestEnvelope and its s3s constructor (#8437)

* feat(app): add the RustFS-owned RequestEnvelope and its s3s constructor

* ci: classify the s3s edge adapters in storage/s3_compat as interface

* feat(s3-types): add Body::from_stream and move Swift off s3s::Body (#8438)

rustfs/backlog#2747 batch 3. `Body::from_stream` wraps a chunk stream of
unknown length over `StreamBody`/`BoxBody`: one data frame per chunk, the
first error ends the body, the size hint stays unknown, and dropping the
body drops the stream. Parity tests pin it to the s3s `StreamingBlob::wrap`
it replaces, at the `http_body` level and byte for byte through hyper.

The Swift handlers in rustfs-protocols build their responses as the owned
`Body`. `SwiftService` converts them into the wrapped S3 service's response
body type through `From<Body>`, so rustfs keeps serving `s3s::Body` at the
legacy edge unchanged. Request bodies stay Send-only (`UnsyncBoxBody`); the
two storage writers and the DLO/SLO segment streams get `Sync` from
`sync_wrapper::SyncStream`, which is what s3s's `http_body_unsync` mutex
provided before. The DTO and error uses of s3s in protocols are untouched
(T1.5 and T1.2).

* refactor(protocols): move error types to rustfs-s3-types by codemod (#8439)

T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and
the error items of protocols move off s3s by scripts/codemods, run with
--hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it
downcasts the backend error, whose runtime type is still s3s::S3Error.

- rustfs/src/protocols/client.rs: the two webdav session methods now
  return rustfs_s3_types::S3Result and convert the s3s error with From.
- codemod: a child module that calls s3_error! through `use super::*`
  gets its own `use rustfs_s3_types::s3_error;` once its parent imports
  that macro by name; otherwise it is listed as GLOB.
- s3-types: a table test pins code, status and the s3s error the legacy
  edge renders for every variant a moved call site names.
- s3_error_lines 1585 -> 1584 (a lifecycle comment reworded).

* test(connect): restore one-second drive request budgets

Apply the exact test-only repair already landed on main in 5972a09cc3. The filesystem probe and production CLI success test request one second and keep their existing one-second duration and parameter checks. All outcome, I/O, error, cleanup and resource-limit assertions remain.

Scope: rustfs/tests/connect_perf_drive.rs only. Patch-id: cb0a1dfc8f68a67727525adea7458e2cee7d09f8.

Validation environment: LANG=en_US.UTF-8, CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0, exclusive APFS-cloned CARGO_TARGET_DIR.

Quick Checks: every command from .github/actions/quick-checks/action.yml, with captured output and exit status below.

$ shellcheck --version && actionlint
ShellCheck - shell script analysis tool
version: 0.11.0
license: GNU General Public License, version 3
website: https://www.shellcheck.net
exit_code=0

$ cargo fmt --all --check
exit_code=0

$ ./scripts/check_unsafe_code_allowances.sh
exit_code=0

$ ./scripts/check_layer_dependencies.sh
Layer dependency guard passed (no new reverse dependencies/cycles).
exit_code=0

$ ./scripts/check_architecture_migration_rules.sh
Architecture migration rules passed.
exit_code=0

$ ./scripts/check_logging_guardrails.sh
✅ Logging guardrails check passed
exit_code=0

$ ./scripts/check_error_other_format_ratchet.sh
error(format!) ratchet passed (total call sites: 582).
exit_code=0

$ ./scripts/check_no_tokio_io_uring.sh
exit_code=0

$ ./scripts/check_extension_schema_boundaries.sh
Extension schema boundaries passed.
exit_code=0

$ ./scripts/check_body_cache_whitelist.sh
Body-cache whitelist guard passed.
exit_code=0

$ ./scripts/check_s3s_footprint.sh
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)
✅ s3s footprint baseline check passed
exit_code=0

$ ./scripts/check_test_count_ratchet.sh
test-count ratchet OK: 52 crates, 21222 tests, 164 ignored match .config/test-count-baseline.txt
exit_code=0

$ ./scripts/check_fips_wording.sh
FIPS wording guard passed (README.md and CHANGELOG.md contain no forbidden claims).
KMS wording guard passed (crates/kms claims no Vault KV2 Transit wrapping).
exit_code=0

$ ./scripts/check_embedded_secrets.sh
Embedded secret guard passed (no private key material or provider credential literal in the tree).
exit_code=0

$ make script-tests
Running script tests...
entrypoint credential policy tests passed
internode grpc A/B bench env tests passed
object batch benchmark enhanced tests passed
PASS package service script contract
hotpath warp A/B gate tests passed
hotpath warp ABBA tests passed
Scanner validation artifacts written to /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.1pFe6Kl1Yu/out
Scanner validation artifacts written to /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.1pFe6Kl1Yu/out-missing-pid
.............................
----------------------------------------------------------------------
Ran 29 tests in 181.286s

OK
{"decision":"verified","pending_gates":["G01","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G02"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-E"}
{"decision":"verified","pending_gates":["G01","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G02"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-E"}
{"decision":"verified","pending_gates":["G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G01"}
{"decision":"verified","pending_gates":["G01","G02","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G03"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-L"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G14"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G14"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G10"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P1"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P2"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P3"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G05"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G06"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-D"}
Status-and-outcome release descriptor verified: /private/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp8rwn1pe0/descriptor/release-bundle-status-outcome.json
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G05"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G06"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-D"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G07"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G08"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P4"}
exact 1MiB handoff ABBA tests passed
pinned paired ABBA bench tests passed
==> /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/bin/path_containment (-max_total_time=7, -seed=123456789, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/explicit-artifacts/path_containment/, corpus=/private/tmp/gw-orchestrator-2734/ci-drive/fuzz/corpus/path_containment)
-seed=123456789
target=path_containment
seed=123456789
max_total_time=7
git_revision=4d15b816236630b2c42f321ffc9ed3654edc94e3
git_dirty=true
runner_mode=prebuilt
==> /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/bin/path_containment (-max_total_time=1, -seed=2322617706, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/auto-artifacts/path_containment/, corpus=/private/tmp/gw-orchestrator-2734/ci-drive/fuzz/corpus/path_containment)
-seed=2322617706
==> cargo +nightly fuzz run path_containment (-max_total_time=9, -seed=987654321, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/cargo-artifacts/path_containment/)
+nightly
fuzz
run
path_containment
-seed=987654321
seed=987654321
runner_mode=cargo-fuzz
fuzz runner seed manifest tests passed
✅ scripts/python_bin.sh resolver checks passed
self-test ok: clean fixture with every exemption present passes
self-test ok: all 11 pattern families fire
self-test ok: moving a pinned source is reported
Embedded secret guard self-test passed.
self-test ok: p1-s3s-import: 'use s3s::s3_error;' counts every line, the rustfs_s3_types module's too (s3_error_lines=3)
self-test ok: p2-s3s-mixed-tree: a multi-line s3s use tree naming s3_error counts (s3_error_lines=3)
self-test ok: p3-s3s-glob: a crate-root glob of s3s, bare or inside a tree, counts (s3_error_lines=5)
self-test ok: p4-macro-use: '#[macro_use] extern crate s3s' counts (s3_error_lines=3)
self-test ok: p5-qualified-s3s: a qualified s3s::s3_error! line counts in a cleared file (s3_error_lines=2)
self-test ok: p6-exact-pass: a baseline equal to the tree passes (exit 0)
self-test ok: n1-rustfs-import: 'use rustfs_s3_types::s3_error;' alone counts nothing (s3_error_lines=1)
self-test ok: n2-rustfs-mixed-tree: a rustfs_s3_types tree plus an s3s tree without the macro counts nothing (s3_error_lines=1)
self-test ok: n3-defines: the file defining the macro counts nothing (s3_error_lines=1)
self-test ok: n4-qualified-rustfs: a qualified rustfs_s3_types::s3_error! line counts nothing (s3_error_lines=1)
self-test ok: n5-renamed-call: a call through a renamed macro is not text-matched (out of scope) (s3_error_lines=1)
self-test ok: n6-comment-string: a comment or string mentioning s3_error! in a cleared file counts nothing (s3_error_lines=1)
self-test ok: n7-e2e-excluded: crates/e2e_test is excluded (s3_error_lines=1)
self-test ok: n8-super-glob: an unrecognised import form ('use super::*;') counts (s3_error_lines=3)
self-test ok: n9-facade: a crate-local re-export counts (s3_error_lines=2)
self-test ok: n10-renamed-import: a renamed rustfs_s3_types import does not clear s3_error! lines (s3_error_lines=2)
self-test ok: n11-commented-import: a commented-out rustfs_s3_types import does not clear (s3_error_lines=2)
self-test ok: n12-rustfs-glob: a glob import of rustfs_s3_types does not clear (s3_error_lines=2)
self-test ok: n13-comment-only: a comment mentioning s3_error! in a file with no import counts (s3_error_lines=2)
self-test ok: f1-baseline-missing: a missing baseline file fails (exit 1)
self-test ok: f2-key-missing: a baseline without s3_error_lines fails (exit 1)
self-test ok: f3-rg-fails: a failing classification search fails the guard (exit 1)
✅ s3s footprint self-test passed
test_core_gate_rejects_missing_ignored_filtered_and_corrupt_inputs (__main__.SelfTests.test_core_gate_rejects_missing_ignored_filtered_and_corrupt_inputs) ... ok
test_e2e_requires_registration (__main__.SelfTests.test_e2e_requires_registration) ... ok
test_fuzz_runtime_uses_matrix_target (__main__.SelfTests.test_fuzz_runtime_uses_matrix_target) ... ok
test_ilm_lane_keeps_the_measured_cargo_build_budget (__main__.SelfTests.test_ilm_lane_keeps_the_measured_cargo_build_budget) ... ok
test_profile_listing_binds_platform_digest (__main__.SelfTests.test_profile_listing_binds_platform_digest) ... ok
test_profile_listing_enforces_selection (__main__.SelfTests.test_profile_listing_enforces_selection) ... ok
test_quick_checks_commands_propagate_failure (__main__.SelfTests.test_quick_checks_commands_propagate_failure) ... ok
test_quick_checks_rejects_caller_and_execution_bypasses (__main__.SelfTests.test_quick_checks_rejects_caller_and_execution_bypasses) ... ok
test_runner_readiness_fails_closed (__main__.SelfTests.test_runner_readiness_fails_closed) ... ok
test_s3_runner_rejects_unbounded_failure_locals (__main__.SelfTests.test_s3_runner_rejects_unbounded_failure_locals) ... ok
test_scanner_heal_begin_requires_embedded_source_provenance (__main__.SelfTests.test_scanner_heal_begin_requires_embedded_source_provenance) ... ok
test_scanner_heal_case_does_not_approve_pending_release (__main__.SelfTests.test_scanner_heal_case_does_not_approve_pending_release) ... ok
test_scanner_heal_case_lane_is_required_for_runner_profile (__main__.SelfTests.test_scanner_heal_case_lane_is_required_for_runner_profile) ... ok
test_scanner_heal_case_only_schema_cannot_approve_release (__main__.SelfTests.test_scanner_heal_case_only_schema_cannot_approve_release) ... ok
test_scanner_heal_crash_case_rejects_restart_oracle (__main__.SelfTests.test_scanner_heal_crash_case_rejects_restart_oracle) ... ok
test_scanner_heal_ec84_case_rejects_wrong_erasure_geometry (__main__.SelfTests.test_scanner_heal_ec84_case_rejects_wrong_erasure_geometry) ... ok
test_scanner_heal_finish_collects_oracles_from_registry (__main__.SelfTests.test_scanner_heal_finish_collects_oracles_from_registry) ... ok
test_scanner_heal_multipool_case_accepts_deferred_optional_outage_write (__main__.SelfTests.test_scanner_heal_multipool_case_accepts_deferred_optional_outage_write) ... ok
test_scanner_heal_pending_gate_cannot_map_to_implemented_lane (__main__.SelfTests.test_scanner_heal_pending_gate_cannot_map_to_implemented_lane) ... ok
test_scanner_heal_receipts_reject_reuse_and_missing_builds (__main__.SelfTests.test_scanner_heal_receipts_reject_reuse_and_missing_builds) ... ok
test_scanner_heal_rejects_boolean_fractional_and_out_of_geometry_integers (__main__.SelfTests.test_scanner_heal_rejects_boolean_fractional_and_out_of_geometry_integers) ... ok
test_scanner_heal_rejects_broken_execution_and_artifacts (__main__.SelfTests.test_scanner_heal_rejects_broken_execution_and_artifacts) ... ok
test_scanner_heal_rejects_copied_junit_and_wrong_suite_build (__main__.SelfTests.test_scanner_heal_rejects_copied_junit_and_wrong_suite_build) ... ok
test_scanner_heal_release_bundle_accepts_complete_measured_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_accepts_complete_measured_evidence) ... ok
test_scanner_heal_release_bundle_assembler_copies_measured_artifacts (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_copies_measured_artifacts) ... ok
test_scanner_heal_release_bundle_assembler_merges_measured_descriptors (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_merges_measured_descriptors) ... ok
test_scanner_heal_release_bundle_assembler_rejects_duplicate_descriptor_gate (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_duplicate_descriptor_gate) ... ok
test_scanner_heal_release_bundle_assembler_rejects_fixture_descriptor (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_fixture_descriptor) ... ok
test_scanner_heal_release_bundle_assembler_rejects_hash_mismatch (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_hash_mismatch) ... ok
test_scanner_heal_release_bundle_assembler_rejects_mixed_descriptor_revisions (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_mixed_descriptor_revisions) ... ok
test_scanner_heal_release_bundle_enforces_topology_duration_profile_and_versions (__main__.SelfTests.test_scanner_heal_release_bundle_enforces_topology_duration_profile_and_versions) ... ok
test_scanner_heal_release_bundle_fixture_writer_is_not_release_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_fixture_writer_is_not_release_evidence) ... ok
test_scanner_heal_release_bundle_gate_accepts_single_measured_gate (__main__.SelfTests.test_scanner_heal_release_bundle_gate_accepts_single_measured_gate) ... ok
test_scanner_heal_release_bundle_gate_rejects_missing_field (__main__.SelfTests.test_scanner_heal_release_bundle_gate_rejects_missing_field) ... ok
test_scanner_heal_release_bundle_gate_rejects_unknown_gate (__main__.SelfTests.test_scanner_heal_release_bundle_gate_rejects_unknown_gate) ... ok
test_scanner_heal_release_bundle_rejects_synthetic_or_missing_fields (__main__.SelfTests.test_scanner_heal_release_bundle_rejects_synthetic_or_missing_fields) ... ok
test_scanner_heal_release_bundle_requires_domain_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_requires_domain_evidence) ... ok
test_scanner_heal_release_bundle_requires_field_provenance (__main__.SelfTests.test_scanner_heal_release_bundle_requires_field_provenance) ... ok
test_scanner_heal_release_bundle_requires_json_artifact_provenance (__main__.SelfTests.test_scanner_heal_release_bundle_requires_json_artifact_provenance) ... ok
test_scanner_heal_release_bundle_requires_mixed_version_field_roles (__main__.SelfTests.test_scanner_heal_release_bundle_requires_mixed_version_field_roles) ... ok
test_scanner_heal_release_bundle_requires_same_gate_measurement_window (__main__.SelfTests.test_scanner_heal_release_bundle_requires_same_gate_measurement_window) ... ok
test_scanner_heal_release_consumes_multiple_registry_oracles (__main__.SelfTests.test_scanner_heal_release_consumes_multiple_registry_oracles) ... ok
test_scanner_heal_release_matrix_lanes_remain_pending_without_real_evidence (__main__.SelfTests.test_scanner_heal_release_matrix_lanes_remain_pending_without_real_evidence) ... ok
test_scanner_heal_release_requirements_cannot_be_cleared (__main__.SelfTests.test_scanner_heal_release_requirements_cannot_be_cleared) ... ok
test_scanner_heal_release_status_rejects_focused_case_run (__main__.SelfTests.test_scanner_heal_release_status_rejects_focused_case_run) ... ok
test_scanner_heal_release_status_rejects_synthetic_case (__main__.SelfTests.test_scanner_heal_release_status_rejects_synthetic_case) ... ok
test_scanner_heal_required_evidence_fields_cannot_be_removed (__main__.SelfTests.test_scanner_heal_required_evidence_fields_cannot_be_removed) ... ok
test_scheduled_alerts_require_completion_watchdog (__main__.SelfTests.test_scheduled_alerts_require_completion_watchdog) ... ok
test_update_profile_selection_changes_only_requested_platform (__main__.SelfTests.test_update_profile_selection_changes_only_requested_platform) ... ok
test_validate_includes_quick_checks (__main__.SelfTests.test_validate_includes_quick_checks) ... ok
test_vault_tests_require_cross_process_serialization (__main__.SelfTests.test_vault_tests_require_cross_process_serialization) ... ok
test_workflow_readiness_requires_dependency_probe (__main__.SelfTests.test_workflow_readiness_requires_dependency_probe) ... ok

----------------------------------------------------------------------
Ran 52 tests in 14.301s

OK
e2e-full selection OK: 1 tests, sha256=29e30e6ed2ed38d9e5b0a7cdd9536abf5c719f7021a02b578ae1381563a3c273
...................................
----------------------------------------------------------------------
Ran 35 tests in 1.161s

OK
...................
----------------------------------------------------------------------
Ran 19 tests in 17.397s

OK
..........
----------------------------------------------------------------------
Ran 10 tests in 0.183s

OK
test_actual_selector_bootstrap_uses_base_policy_and_fails_closed (__main__.SelfTests.test_actual_selector_bootstrap_uses_base_policy_and_fails_closed) ... ok
test_documentation_paths_do_not_hide_build_or_fixture_changes (__main__.SelfTests.test_documentation_paths_do_not_hide_build_or_fixture_changes) ... ok
test_e2e_scope_keeps_production_and_shared_configuration_full (__main__.SelfTests.test_e2e_scope_keeps_production_and_shared_configuration_full) ... ok
test_e2e_shortcut_rejects_new_direct_renamed_and_target_dependencies (__main__.SelfTests.test_e2e_shortcut_rejects_new_direct_renamed_and_target_dependencies) ... ok
test_e2e_shortcut_traverses_implicit_path_dependencies (__main__.SelfTests.test_e2e_shortcut_traverses_implicit_path_dependencies) ... ok
test_event_contract_requires_complete_candidate_and_optional_lanes (__main__.SelfTests.test_event_contract_requires_complete_candidate_and_optional_lanes) ... ok
test_every_wrong_result_missing_job_or_selection_fails_closed (__main__.SelfTests.test_every_wrong_result_missing_job_or_selection_fails_closed) ... ok
test_full_e2e_gate_preserves_workflow_branch_and_event_scope (__main__.SelfTests.test_full_e2e_gate_preserves_workflow_branch_and_event_scope) ... ok
test_git_range_includes_deleted_source_and_rename_origins (__main__.SelfTests.test_git_range_includes_deleted_source_and_rename_origins) ... ok
test_job_ids_and_display_names_cannot_hide_validation (__main__.SelfTests.test_job_ids_and_display_names_cannot_hide_validation) ... ok
test_report_only_stack_lane_mirrors_e2e_smoke_outside_the_aggregate (__main__.SelfTests.test_report_only_stack_lane_mirrors_e2e_smoke_outside_the_aggregate) ... ok
test_repository_wiring_and_missing_dependency_regression (__main__.SelfTests.test_repository_wiring_and_missing_dependency_regression) ... ok
test_verify_command_preserves_failures (__main__.SelfTests.test_verify_command_preserves_failures) ... ok

----------------------------------------------------------------------
Ran 13 tests in 3.280s

OK
security coverage self-test passed
test_api_failures_preserve_other_evidence_and_never_enter_grace (__main__.SelfTests.test_api_failures_preserve_other_evidence_and_never_enter_grace) ... ok
test_attempt_outcome_does_not_replace_recent_success (__main__.SelfTests.test_attempt_outcome_does_not_replace_recent_success) ... ok
test_cancelled_attempt_cannot_refresh_expired_success (__main__.SelfTests.test_cancelled_attempt_cannot_refresh_expired_success) ... ok
test_cli_requires_the_repository_default_branch (__main__.SelfTests.test_cli_requires_the_repository_default_branch) ... ok
test_config_rejects_duplicate_and_invalid_entries (__main__.SelfTests.test_config_rejects_duplicate_and_invalid_entries) ... ok
test_freshness_boundaries (__main__.SelfTests.test_freshness_boundaries) ... ok
test_grace_requires_two_successful_queries_with_no_history (__main__.SelfTests.test_grace_requires_two_successful_queries_with_no_history) ... ok
test_invalid_api_evidence_fails_closed (__main__.SelfTests.test_invalid_api_evidence_fails_closed) ... ok
test_report_retains_every_workflow (__main__.SelfTests.test_report_retains_every_workflow) ... ok
test_requests_filter_schedule_default_branch_and_success_on_server (__main__.SelfTests.test_requests_filter_schedule_default_branch_and_success_on_server) ... ok

----------------------------------------------------------------------
Ran 10 tests in 0.051s

OK
..............................
----------------------------------------------------------------------
Ran 30 tests in 22.354s

OK
..........
----------------------------------------------------------------------
Ran 10 tests in 2.969s

OK
.................................
----------------------------------------------------------------------
Ran 33 tests in 0.160s

OK
Online functional runner labels: pf-testing
.......
----------------------------------------------------------------------
Ran 7 tests in 0.002s

OK
.......
----------------------------------------------------------------------
Ran 7 tests in 0.001s

OK
....
----------------------------------------------------------------------
Ran 4 tests in 0.128s

OK
...
----------------------------------------------------------------------
Ran 3 tests in 0.547s

OK
PASS: cold stampede script self-test
exit_code=0

$ python3 ./scripts/test_object_generation_protocol_model.py
test_a_recovery_barrier_cannot_choose_an_empty_value (__main__.ObjectGenerationProtocolModel.test_a_recovery_barrier_cannot_choose_an_empty_value) ... ok
test_all_bounded_two_proposer_interleavings_choose_at_most_one_value (__main__.ObjectGenerationProtocolModel.test_all_bounded_two_proposer_interleavings_choose_at_most_one_value) ... ok
test_an_unavailable_decision_quorum_cannot_choose_a_candidate (__main__.ObjectGenerationProtocolModel.test_an_unavailable_decision_quorum_cannot_choose_a_candidate) ... ok
test_counter_allocation_requires_persistence_and_is_consumed_once (__main__.ObjectGenerationProtocolModel.test_counter_allocation_requires_persistence_and_is_consumed_once) ... ok
test_counter_restart_burns_persisted_allocations_before_or_after_issue (__main__.ObjectGenerationProtocolModel.test_counter_restart_burns_persisted_allocations_before_or_after_issue) ... ok
test_crash_after_promise_before_reply_does_not_restore_an_older_ballot (__main__.ObjectGenerationProtocolModel.test_crash_after_promise_before_reply_does_not_restore_an_older_ballot) ... ok
test_duplicate_or_unconfigured_voters_cannot_form_a_quorum (__main__.ObjectGenerationProtocolModel.test_duplicate_or_unconfigured_voters_cannot_form_a_quorum) ... ok
test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision (__main__.ObjectGenerationProtocolModel.test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision) ... ok
test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected (__main__.ObjectGenerationProtocolModel.test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected) ... ok
test_forgetting_tombstone_authority_allows_an_aba_create (__main__.ObjectGenerationProtocolModel.test_forgetting_tombstone_authority_allows_an_aba_create) ... ok
test_issuing_before_persistence_reuses_a_ballot_after_restart (__main__.ObjectGenerationProtocolModel.test_issuing_before_persistence_reuses_a_ballot_after_restart) ... ok
test_lost_ack_and_restart_preserve_the_chosen_operation (__main__.ObjectGenerationProtocolModel.test_lost_ack_and_restart_preserve_the_chosen_operation) ... ok
test_minority_accepted_work_is_adopted_and_finished (__main__.ObjectGenerationProtocolModel.test_minority_accepted_work_is_adopted_and_finished) ... ok
test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample (__main__.ObjectGenerationProtocolModel.test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample) ... ok
test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it (__main__.ObjectGenerationProtocolModel.test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it) ... ok
test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor (__main__.ObjectGenerationProtocolModel.test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor) ... ok
test_replacing_voters_with_empty_state_can_choose_a_conflicting_value (__main__.ObjectGenerationProtocolModel.test_replacing_voters_with_empty_state_can_choose_a_conflicting_value) ... ok
test_skipping_predecessor_recovery_chooses_a_broken_revision_chain (__main__.ObjectGenerationProtocolModel.test_skipping_predecessor_recovery_chooses_a_broken_revision_chain) ... ok
test_stale_publication_rollback_and_gc_cannot_change_the_winner (__main__.ObjectGenerationProtocolModel.test_stale_publication_rollback_and_gc_cannot_change_the_winner) ... ok
test_superseded_operation_identity_cannot_be_reused_for_different_input (__main__.ObjectGenerationProtocolModel.test_superseded_operation_identity_cannot_be_reused_for_different_input) ... ok
test_superseded_operation_retry_returns_its_original_successor (__main__.ObjectGenerationProtocolModel.test_superseded_operation_retry_returns_its_original_successor) ... ok
test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision (__main__.ObjectGenerationProtocolModel.test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision) ... ok
test_tombstone_rejects_a_delayed_never_present_create (__main__.ObjectGenerationProtocolModel.test_tombstone_rejects_a_delayed_never_present_create) ... ok
test_unlearned_predecessor_is_recovered_before_the_second_slot (__main__.ObjectGenerationProtocolModel.test_unlearned_predecessor_is_recovered_before_the_second_slot) ... ok

----------------------------------------------------------------------
Ran 24 tests in 0.958s

OK
model: states=51023, transitions=167520, terminal=732, chosen_candidates=2
negative control: double-choice trace=((0, 'prepare', 0), (0, 'prepare', 1), (0, 'prepare', 2), (0, 'accept', 0), (0, 'accept', 1), (0, 'accept', 2), (1, 'prepare', 0), (1, 'prepare', 1), (1, 'prepare', 2), (1, 'accept', 0), (1, 'accept', 1), (1, 'accept', 2))
multi-slot round schedules: 640
exit_code=0

$ python3 ./scripts/check_test_wiring.py
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired
exit_code=0

$ ./scripts/check_no_planning_docs.sh
exit_code=0

$ ./scripts/check_uring_lane_lib_only.sh
OK: no 'uring_'-matching test functions under crates/ecstore/tests; --lib is safe
exit_code=0

Initial complete gate
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
 Nextest run ID e07fc5b9-c6c8-4cdd-ae61-54839bf24437 with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.016s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.011s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.009s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.009s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  16.950s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   4.025s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.008s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.007s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.035s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  21.087s] 10 tests run: 10 passed, 0 skipped
exit_code=0

Restored complete gate
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
 Nextest run ID 233b7307-5361-4eb7-94d5-b384f17d2d6a with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.009s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.008s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.008s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.008s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  19.518s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   4.146s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.009s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.018s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.009s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.054s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  23.787s] 10 tests run: 10 passed, 0 skipped
exit_code=0

Mutation M1: success_measurement reports max(actual elapsed, requested budget + 1), forcing 1001 ms for the one-second request. The retained duration assertion at rustfs/tests/connect_perf_drive.rs:105 fails.
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast -E 'test(real_local_write_read_is_measured_and_scratch_is_removed)'
info: experimental features enabled: setup-scripts
   Compiling rustfs v1.0.1 (/private/tmp/gw-orchestrator-2734/ci-drive/rustfs)
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs") generated 1 warning
warning: `rustfs` (bin "rustfs-cli") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 7m 06s
────────────
 Nextest run ID 9f618777-7bf6-4cbc-8803-dd7f50f449ed with nextest profile: ci
    Starting 1 test across 1 binary (9 tests skipped)
        FAIL [   0.016s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
  stdout ───

    running 1 test
    test real_local_write_read_is_measured_and_scratch_is_removed ... FAILED

    failures:

    failures:
        real_local_write_read_is_measured_and_scratch_is_removed

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 0.01s

  stderr ───

    thread 'real_local_write_read_is_measured_and_scratch_is_removed' (106986846) panicked at rustfs/tests/connect_perf_drive.rs:105:5:
    assertion failed: (1..=1_000).contains(&data.duration_millis)
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

────────────
     Summary [   0.017s] 1 test run: 0 passed, 1 failed, 9 skipped
        FAIL [   0.016s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
error: test run failed
exit_code=100

Mutation M2: target_parameters reports requested duration + 1. The in-process assertion at rustfs/tests/connect_perf_drive.rs:111 and CLI JSON assertion at rustfs/tests/connect_perf_drive.rs:527 both reject 1001 against 1000.
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast -E 'test(real_local_write_read_is_measured_and_scratch_is_removed) | test(production_cli_measures_and_signs_exact_binary_provenance)'
info: experimental features enabled: setup-scripts
   Compiling rustfs v1.0.1 (/private/tmp/gw-orchestrator-2734/ci-drive/rustfs)
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs") generated 1 warning
warning: `rustfs` (bin "rustfs-cli") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 7m 20s
────────────
 Nextest run ID b73d46ab-c910-4e1e-8063-7fc7e332ec15 with nextest profile: ci
    Starting 2 tests across 1 binary (8 tests skipped)
        FAIL [  13.968s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
  stdout ───

    running 1 test
    test production_cli_measures_and_signs_exact_binary_provenance ... FAILED

    failures:

    failures:
        production_cli_measures_and_signs_exact_binary_provenance

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 13.96s

  stderr ───

    thread 'production_cli_measures_and_signs_exact_binary_provenance' (107015462) panicked at rustfs/tests/connect_perf_drive.rs:527:5:
    assertion `left == right` failed
      left: Number(1001)
     right: 1000
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

        FAIL [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
  stdout ───

    running 1 test
    test real_local_write_read_is_measured_and_scratch_is_removed ... FAILED

    failures:

    failures:
        real_local_write_read_is_measured_and_scratch_is_removed

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 0.01s

  stderr ───

    thread 'real_local_write_read_is_measured_and_scratch_is_removed' (107016667) panicked at rustfs/tests/connect_perf_drive.rs:111:5:
    assertion `left == right` failed
      left: 1001
     right: 1000
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

────────────
     Summary [  13.985s] 2 tests run: 0 passed, 2 failed, 8 skipped
        FAIL [  13.968s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        FAIL [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
error: test run failed
exit_code=100

Transient production mutations were restored byte-for-byte before the complete green rerun. Original/restored SHA256: 8d389e6d8836c041aab8c6a93e56addd65ba3f70b33264e1c74f639d0572a2da.

$ git diff --check

exit_code=0

$ git diff --cached --check

exit_code=0

Independent orchestrator acceptance
Tested code tree: e67c72fb01

Tested tree: 16fbbd2ea4
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

$ LANG=en_US.UTF-8 CARGO_BUILD_JOBS=1 CARGO_INCREMENTAL=0 cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
info: experimental features enabled: setup-scripts
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs-cli") generated 1 warning
warning: `rustfs` (bin "rustfs") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.69s
────────────
 Nextest run ID cd1edf4c-f1c0-456d-ad1a-76cc00ce05c9 with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.012s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.010s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.010s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.009s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  20.328s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   5.527s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.013s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.019s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.011s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.049s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  25.991s] 10 tests run: 10 passed, 0 skipped

exit_code=0

All 54 tracked Cargo.toml files parse without duplicate keys. git diff --check passed. The production source remains unchanged; the two deliberate implementation mutations above were restored before this complete green run.

* fix(test): compare legacy body wire parity as raw bytes

Lossy UTF-8 comparison concealed a changed binary response byte.
Compare the already captured Vec<u8> values directly.

Validation:
- Binary mutant: migrated payload 0x84 vs legacy 0x83; old comparator
  passed 1/1, raw comparator failed 0/1 (exit 100); restored green.
- Non-Swift body-drain mutant: untouched-body delegation test failed
  with empty vs payload (exit 100); restored handler without a diff.
- cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols
  --features rustfs-protocols/swift --no-fail-fast:
  543 tests run, 543 passed, 0 skipped after both mutants restored.
- cargo fmt --all --check and git diff --check passed.

Orchestrator acceptance (LANG=en_US.UTF-8):
cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
Summary [2.445s] 543 tests run: 543 passed, 0 skipped.

All shared quick checks on the final code tree:
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

Final independent acceptance after the drive-test CI repair
Parent: a99d8f4f89
Tested code tree: 93cbd96bf0

Tested tree: 075ae5161c2b62a96812bc1e3c904a7663b4f2f8
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

$ LANG=en_US.UTF-8 CARGO_BUILD_JOBS=1 CARGO_INCREMENTAL=0 cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
    Finished `test` profile [unoptimized + debuginfo] target(s) in 10m 08s
 Nextest run ID 5c7ff102-ec6d-479a-86e0-810859b28bbf with nextest profile: ci
    Starting 543 tests across 8 binaries
     Summary [   6.729s] 543 tests run: 543 passed, 0 skipped
exit_code=0

The only source change is the raw-byte wire comparator. Cargo manifests and the ratchet baselines remain unchanged from the independently validated parent. The invalid-byte and consumed-request mutations remain recorded above; their implementation changes were restored.

* fix: repair integration checks and IPC test readiness

Wait for the RPC/locks test sockets to pass the existing owner and mode
checks inside the original 5s readiness window and 10ms poll interval.

Use the existing admin S3 carrier for the new recovery and quota error
calls. Regenerate the test-count baseline for main's 118 existing tests
across seven crates. Pair the private pagination helper's raw-scan inputs
and remove the redundant Copy clone from the retirement event.

Keep the original assertions, responses, authorization, locks and storage
operations. The pagination body and raw-scan expressions at all five calls
are byte-identical; reinserting only the Copy clone restores the entire
base heal file. All ignored baseline values and s3s counts stay unchanged.

Validation base: c47432e353
Validation tree: f081084bc5
Patch SHA-256: 2dcf0028691bfb2772edfb38e262829b04f4151832dcce823c9dd9f7434b52de
Trace production-prefix SHA-256: fd994f218a21429d62866f4efb8eee1ee57b58f4a51865aa35d05a54ef1ccb45

All commands used LANG=en_US.UTF-8, CARGO_BUILD_JOBS=1,
CARGO_INCREMENTAL=0 and CARGO_NET_OFFLINE=true in one exclusive target.
Every gate retained the same tree and six source hashes before and after.
54 Cargo.toml files parse without duplicate keys.

cargo clippy -p rustfs --all-targets -- -D warnings
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 12m 50s
Exit: 0. Wall time: 770.855s.

The original 19 quick-check steps:
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
Wall time: 286.337s.

cargo nextest run --profile ci -p rustfs --no-fail-fast
     Summary [ 217.017s] 5382 tests run: 5382 passed (4 leaky), 40 skipped
Exit: 0. Wall time: 1324.414s.

cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
     Summary [   2.189s] 543 tests run: 543 passed, 0 skipped
Exit: 0. Wall time: 123.192s.

cargo nextest run --profile ci -p rustfs-ecstore list_objects_paginate --no-fail-fast
     Summary [   0.170s] 8 tests run: 8 passed, 5921 skipped
Exit: 0. Wall time: 261.324s.

cargo nextest run --profile ci -p rustfs-heal durable_mrf_anchor_retires_only_after_exact_bucket_generation_is_deleted --no-fail-fast
     Summary [   0.805s] 1 test run: 1 passed, 835 skipped
Exit: 0. Wall time: 199.801s.

Baseline regeneration:
./scripts/check_test_count_ratchet.sh --update
wrote .config/test-count-baseline.txt: 52 crates, 21340 tests, 164 ignored
Test-only increases: ecstore48, heal3, iam14, io-metrics2, obs3, policy14,
rustfs34. Existing ignored values and all other rows are byte-identical.
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)

Historical runtime red controls on f622 (same retained two assertions):
cargo nextest run --profile ci -p rustfs --lib -E 'test(=connect::diagnostics::trace_runtime::tests::local_top_rpc_reads_service_http_completions_and_signs_offline) | test(=connect::diagnostics::trace_runtime::tests::local_top_locks_reads_service_manager_and_signs_offline)' --no-fail-fast
Original path readiness with a temporary 500ms bind/seal gap:
     Summary [   0.863s] 2 tests run: 0 passed, 2 failed, 5113 skipped
Exit: 100.
Sealed readiness in that same gap:
     Summary [   5.621s] 2 tests run: 2 passed, 5113 skipped
Exit: 0.
Compiled M1: replace the helper predicate with path.exists():
     Summary [   0.870s] 2 tests run: 0 passed, 2 failed, 5113 skipped
Exit: 100.
Restore the sealed predicate byte-for-byte:
     Summary [   5.640s] 2 tests run: 2 passed, 5113 skipped
Exit: 0.
Both reds observe mode755 with owner_matches=true and fail at the original
RPC/locks unwraps with StateSecurity. M1 failure locations: trace_runtime.rs
2547:29 and 2713:29. No new assertions were introduced.

cargo nextest run --profile ci -p rustfs --lib -E 'test(connect::diagnostics::trace_runtime::tests::)' --no-fail-fast
     Summary [  20.748s] 28 tests run: 28 passed (1 leaky), 5087 skipped
Exit: 0. This security sweep is historical; the full final unit gate above
covers the actual c474 candidate.
The first disk-stopped compile ran no tests and is excluded from red credit.
All temporary seal instrumentation is removed from the final source.

Final unit cleanup markers: 4 leaky, 40 original skips (exit0). The ld
__eh_frame linker warning also appears verbatim in the original f622 run
and the earlier c474 run; there are no new source warnings.

* fix: remove redundant pending closure in disk test

Pass the capture-free pending future factory as a function item in the existing slow-operation test. This fixes the redundant_closure error at disk_store.rs:2914 that stopped strict Clippy before the feature tests could run.

The single-line test edit preserves the factory's result, warning and timeout behavior. Reversing the replacement reproduces the parent file byte for byte. All 220 assertion lines and the production prefix are unchanged; no tests, assertions, guards, profiles or baselines were added or relaxed.

Original Linux CI evidence (run 37888552268 at 53245be9f):
- RIO job113685483992: strict Clippy exit101, redundant closure at disk_store.rs:2914; feature tests skipped.
- Workspace Test and Lint job113685484058: same strict Clippy exit101; nextest and doc tests skipped.
These are lint failures, not runtime mutation kills or known flakes.

Native macOS arm64 validation on tree7a5bf47e7533b29479bc85207416102f1337c566, LANG=en_US.UTF-8, one build job, CARGO_INCREMENTAL=0 and offline dependencies:

shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
All 19 checks: exit0, 312.161s.

$ cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 20m 11s
exit0, 1213.210s total.

$ cargo nextest run --profile ci -p rustfs-ecstore --features rio-v2 slow_disk_operation_warns_once_while_pending_and_preserves_result --no-fail-fast
Finished `test` profile [unoptimized + debuginfo] target(s) in 4m 34s
PASS [0.026s] rustfs-ecstore disk::disk_store::tests::slow_disk_operation_warns_once_while_pending_and_preserves_result
Summary [   0.170s] 1 test run: 1 passed, 5943 skipped
exit0, 281.920s total; 5943 existing filter skips.

warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
This linker warning is retained; flags, profiles and allowances are unchanged.

54 tracked Cargo.toml files parsed without duplicate keys. git diff --check passed. No new assertions were introduced, so no new runtime mutation is claimed. Native compilation and the existing helper test do not substitute for actual Linux RIO execution; the new integration CI run still needs to recover before the RPC merge.

Preservation and output hashes:
parent file SHA256: 8e59ee745e3aaaa22848b943d3b10f6353b515956282cde1f3af2c394200c9ac
final file SHA256: b69ba3237061b161448ae8b62b615885d5770b98a5b4a2479bcfbd70f1bd1f17
unchanged production prefix SHA256: 7903a6e288252c420231242d7b58a25fa2a7140ea3dd430cbe7f0dde11369dcb
RIO Linux failure log SHA256:421addb9868fd88e4109a922311e54f91f0dfd70862f8ba7fcaba193482ed90b
Workspace Linux failure log SHA256:e0aea8d0f014c022252a9534968e083c98cfc1fd760da445ecd04ec3f522e26a
19 output SHA256:35a54f28d707956ed1edb29d492c4dfe8a9fe753d103b111b5a2f1b5c41c2b98
RIO Clippy output SHA256:d6046e3868f2b3992fe11cdd04e9e98eb50a87d1234d9a70dbfeb154ff61638d
slow-operation output SHA256:605b7b258d2fd09c6ef2ab2ffc90d433d0509480f1e45125d20c823759e83608

Refs rustfs/backlog#2734.

* test(ecstore): isolate tier-free recovery fixture startup

The fixture constructor can take the production recovery lock and wait for 60 seconds before this test installs its queue and seeds metadata. Disable its background recovery only while creating this fixture, then restore the environment before calling the existing production recovery entrypoint.

Only the existing test setup changes. Its production entrypoint, default lock parameters, original assertions, 30-second timeout, cancellation and cleanup remain intact. No tests or assertions were added. Production and store/init.rs are byte-identical to the parent.

Independent acceptance ran on merge tree 8ec358cb42ba9ebb3bcfeafcef048a3f0366355b; the runner header reports parent HEAD, while before/after index and source hashes confirm this candidate. Worker task branch d8055766b8 has the same tree.

Every command used LANG=en_US.UTF-8. Task compilation also used CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0, CARGO_NET_OFFLINE=true, an independent target, and empty RUSTC_WRAPPER/RUSTC_WORKSPACE_WRAPPER to keep compilers in its owned process group without changing repository configuration.

Command:
shellcheck --version && actionlint
Exit 0; observed output excerpt:
version: 0.11.0
Log SHA-256: fc9eec16b4191d058a295813ca9afbe9d2b37b59975da860170726be162a3924

Command:
cargo fmt --all --check
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_unsafe_code_allowances.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_layer_dependencies.sh
Exit 0; observed output excerpt:
Layer dependency guard passed (no new reverse dependencies/cycles).
Log SHA-256: 69c6cb1896c0586b28d1167380d3260d99d8017a4265c8df5a2348f289e93a12

Command:
./scripts/check_architecture_migration_rules.sh
Exit 0; observed output excerpt:
Architecture migration rules passed.
Log SHA-256: 28790e70d070607ed4de0704f292a65030be92ec224ef106d4b72fd8296f896e

Command:
./scripts/check_logging_guardrails.sh
Exit 0; observed output excerpt:
✅ Logging guardrails check passed
Log SHA-256: 7adf41abff8ed8f7617decf86498888ddde78937099dd9ce47f6d359dfc2d317

Command:
./scripts/check_error_other_format_ratchet.sh
Exit 0; observed output excerpt:
error(format!) ratchet passed (total call sites: 579).
Log SHA-256: 66172aef18f150ba0605af122e26ffcf91f6dd86f43e38b05e2e982ce8d783ee

Command:
./scripts/check_no_tokio_io_uring.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_extension_schema_boundaries.sh
Exit 0; observed output excerpt:
Extension schema boundaries passed.
Log SHA-256: f13d0acd0e2ce5d9815241b12b4dab7d01cc11b9b65bcf6dfb7812c878532a51

Command:
./scripts/check_body_cache_whitelist.sh
Exit 0; observed output excerpt:
Body-cache whitelist guard passed.
Log SHA-256: 6b828847b0383950ee6eef9cd846d0c322c945e47efd4c7d1ad881178810d6bb

Command:
./scripts/check_s3s_footprint.sh
Exit 0; observed output excerpt:
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)
✅ s3s footprint baseline check passed
Log SHA-256: 47f53d04ef645618563fe1f22abc64667f19078f1974feedcca46e4f2c016844

Command:
./scripts/check_test_count_ratchet.sh
Exit 0; observed output excerpt:
test-count ratchet OK: 52 crates, 21340 tests, 164 ignored match .config/test-count-baseline.txt
Log SHA-256: 618a5e330e933de0c6ff484546df401457024a88bb5986d11d588794e33dab60

Command:
./scripts/check_fips_wording.sh
Exit 0; observed output excerpt:
FIPS wording guard passed (README.md and CHANGELOG.md contain no forbidden claims).
KMS wording guard passed (crates/kms claims no Vault KV2 Transit wrapping).
Log SHA-256: 99221cc9979c480ec4f46f50d76c52b18c72fe499f12e15e4f4a22a1b8bfb140

Command:
./scripts/check_embedded_secrets.sh
Exit 0; observed output excerpt:
Embedded secret guard passed (no private key material or provider credential literal in the tree).
Log SHA-256: 437a3ceba32094d2901889965ad191e411aad32cdb18bf0b8659e818011c406b

Command:
make script-tests
Exit 0; observed output excerpt:
test_cli_requires_the_repository_default_branch (__main__.SelfTests.test_cli_requires_the_repository_default_branch) ... ok
test_config_rejects_duplicate_and_invalid_entries (__main__.SelfTests.test_config_rejects_duplicate_and_invalid_entries) ... ok
test_freshness_boundaries (__main__.SelfTests.test_freshness_boundaries) ... ok
test_grace_requires_two_successful_queries_with_no_history (__main__.SelfTests.test_grace_requires_two_successful_queries_with_no_history) ... ok
test_invalid_api_evidence_fails_closed (__main__.SelfTests.test_invalid_api_evidence_fails_closed) ... ok
test_report_retains_every_workflow (__main__.SelfTests.test_report_retains_every_workflow) ... ok
test_requests_filter_schedule_default_branch_and_success_on_server (__main__.SelfTests.test_requests_filter_schedule_default_branch_and_success_on_server) ... ok
Ran 10 tests in 0.042s
OK
Ran 30 tests in 20.627s
OK
Ran 10 tests in 2.268s
OK
Ran 33 tests in 0.127s
OK
Ran 7 tests in 0.002s
OK
Ran 7 tests in 0.001s
OK
Ran 4 tests in 0.108s
OK
Ran 3 tests in 0.564s
OK
PASS: cold stampede script self-test
Log SHA-256: 75fda2d069a373c63667b908baaf99e6591aab9099a739b0c4c22ceb6da8d68c

Command:
python3 ./scripts/test_object_generation_protocol_model.py
Exit 0; observed output excerpt:
test_an_unavailable_decision_quorum_cannot_choose_a_candidate (__main__.ObjectGenerationProtocolModel.test_an_unavailable_decision_quorum_cannot_choose_a_candidate) ... ok
test_counter_allocation_requires_persistence_and_is_consumed_once (__main__.ObjectGenerationProtocolModel.test_counter_allocation_requires_persistence_and_is_consumed_once) ... ok
test_counter_restart_burns_persisted_allocations_before_or_after_issue (__main__.ObjectGenerationProtocolModel.test_counter_restart_burns_persisted_allocations_before_or_after_issue) ... ok
test_crash_after_promise_before_reply_does_not_restore_an_older_ballot (__main__.ObjectGenerationProtocolModel.test_crash_after_promise_before_reply_does_not_restore_an_older_ballot) ... ok
test_duplicate_or_unconfigured_voters_cannot_form_a_quorum (__main__.ObjectGenerationProtocolModel.test_duplicate_or_unconfigured_voters_cannot_form_a_quorum) ... ok
test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision (__main__.ObjectGenerationProtocolModel.test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision) ... ok
test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected (__main__.ObjectGenerationProtocolModel.test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected) ... ok
test_forgetting_tombstone_authority_allows_an_aba_create (__main__.ObjectGenerationProtocolModel.test_forgetting_tombstone_authority_allows_an_aba_create) ... ok
test_issuing_before_persistence_reuses_a_ballot_after_restart (__main__.ObjectGenerationProtocolModel.test_issuing_before_persistence_reuses_a_ballot_after_restart) ... ok
test_lost_ack_and_restart_preserve_the_chosen_operation (__main__.ObjectGenerationProtocolModel.test_lost_ack_and_restart_preserve_the_chosen_operation) ... ok
test_minority_accepted_work_is_adopted_and_finished (__main__.ObjectGenerationProtocolModel.test_minority_accepted_work_is_adopted_and_finished) ... ok
test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample (__main__.ObjectGenerationProtocolModel.test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample) ... ok
test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it (__main__.ObjectGenerationProtocolModel.test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it) ... ok
test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor (__main__.ObjectGenerationProtocolModel.test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor) ... ok
test_replacing_voters_with_empty_state_can_choose_a_conflicting_value (__main__.ObjectGenerationProtocolModel.test_replacing_voters_with_empty_state_can_choose_a_conflicting_value) ... ok
test_skipping_predecessor_recovery_chooses_a_broken_revision_chain (__main__.ObjectGenerationProtocolModel.test_skipping_predecessor_recovery_chooses_a_broken_revision_chain) ... ok
test_stale_publication_rollback_and_gc_cannot_change_the_winner (__main__.ObjectGenerationProtocolModel.test_stale_publication_rollback_and_gc_cannot_change_the_winner) ... ok
test_superseded_operation_identity_cannot_be_reused_for_different_input (__main__.ObjectGenerationProtocolModel.test_superseded_operation_identity_cannot_be_reused_for_different_input) ... ok
test_superseded_operation_retry_returns_its_original_successor (__main__.ObjectGenerationProtocolModel.test_superseded_operation_retry_returns_its_original_successor) ... ok
test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision (__main__.ObjectGenerationProtocolModel.test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision) ... ok
test_tombstone_rejects_a_delayed_never_present_create (__main__.ObjectGenerationProtocolModel.test_tombstone_rejects_a_delayed_never_present_create) ... ok
test_unlearned_predecessor_is_recovered_before_the_second_slot (__main__.ObjectGenerationProtocolModel.test_unlearned_predecessor_is_recovered_before_the_second_slot) ... ok
Ran 24 tests in 0.929s
OK
Log SHA-256: 0b4be926efb0609d497ae58930340153f01957b606206cbbdb3e0d47eb29d3a2

Command:
python3 ./scripts/check_test_wiring.py
Exit 0; observed output excerpt:
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired
Log SHA-256: 21f676f458451525535ab0ef73d6a9c0d9774e16d76a4d19e396ad34936dd39c

Command:
./scripts/check_no_planning_docs.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_uring_lane_lib_only.sh
Exit 0; observed output excerpt:
OK: no 'uring_'-matching test functions under crates/ecstore/tests; --lib is safe
Log SHA-256: 43f3f851e27ec2ab9e34d6b05fefdede38af992699c473904bfe6f53a46d52e3

Command:
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast
Exit 0; wall 234.399s.
Observed output:
    Finished `test` profile [unoptimized + debuginfo] target(s) in 3m 49s
    Starting 1 test across 1 binary (5871 tests skipped)
  SETUP PASS [   0.006s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"'
        PASS [   0.440s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item
     Summary [   0.586s] 1 test run: 1 passed, 5871 skipped
Log SHA-256: 34e973f86e8dc2102c1a0e483e0ac802ce7df3d1b40a603932f62c0805c4de27

Command:
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over) | test(=store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence) | test(=store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable)' --no-fail-fast
Exit 0; wall 4.010s.
Observed output:
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.43s
    Starting 3 tests across 1 binary (5869 tests skipped)
  SETUP PASS [   0.005s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"'
        PASS [   0.068s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over
        PASS [   0.637s] rustfs-ecstore store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence
        PASS [   0.846s] rustfs-ecstore store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable
     Summary [   0.987s] 3 tests run: 3 passed, 5869 skipped
Log SHA-256: b5da2a927ff18ca9aa217762f7186a6d4371bd9e8feedd7b77940e10b64dffac

Command:
cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
Exit 0; wall 1084.508s.
Observed output:
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 18m 03s
Log SHA-256: 28f98be7970faaa71cd118da1da4e6e2e3d23b9dda7e6449eaa7a37e741260b5

Compiled mutation command (unchanged for each red and restored run):
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast

M1: reverse only this test-local constructor flag from false to true. The mutant compiled, then failed the original recovered-item queue assertion after 30.481s. Restoring the entire observed candidate passed in 0.460s. Red log 55bf3b1535529ac0bcfb8071f9b56f9f40631007a85c315367e7e93786d65293; restored log 204d5fbe4574a7917f5e740a795d864fe28a7e7fb5e33d27b0a9cf68e2f17f52.

M2: replace only the actual awaited enqueue with false, consuming its original inputs. The mutant compiled, then failed the same original queue assertion after 30.516s. Restoring the entire observed candidate passed in 0.457s. Red log 89186330413215e231e2b06bf013cc31097ea8f4a106d91421a36d6c8986a7fd; restored log e848b42d2302401afa5dbcd9e37bf76d2e56a96ab6de5a488513b997f53c5e9b.

All temporary observation and mutation code was removed before final acceptance. Historical baseline and diagnostic failures remain recorded; they are not mutation credit. The separate Object Lock intent-retry diagnostic still fails, and these native macOS checks do not establish Linux CI recovery.

* refactor(rpc): serve internode rpc bodies with the owned Body (#8441)

* test(rpc): pin the wire bytes of every internode rpc response body

rustfs/backlog#2747 batch 4, step 1 of 2, on the s3s bodies the rpc paths
serve today. Each response body builder in storage/rpc/http_service.rs is
served through hyper's HTTP/1.1 server and the raw bytes the peer reads are
compared with goldens captured from this commit: status, content-length or
chunked framing, chunk boundaries, the terminating chunk, HEAD, and the cut
connection a failure after data produces. The capability responses carry a
per-process proof, so their length header and payload are checked against
the body they encode instead.

The read_file_stream response builder moves into read_file_stream_response
so the test reaches the same code the handler runs; behaviour is unchanged.

* refactor(rpc): serve internode rpc bodies with the owned Body

rustfs/backlog#2747 batch 4, step 2 of 2. The internode rpc HTTP paths
build their responses with rustfs_s3_types::Body: Body::from_stream
replaces Body::from(StreamingBlob::wrap(..)) at the six stream sites, and
the in-memory and empty bodies keep their constructors. InternodeRpcService
converts its responses into the wrapped service's response body through
From<Body>, so the server keeps composing it with the s3s service as
before. The wire-byte goldens from the previous commit pass unchanged;
storage/rpc no longer names s3s.

* fix(rpc): simplify wire test response builder type

Name the boxed response builder used by the capability wire test so
all-target Clippy can check it without a type-complexity warning.

Validation (LANG=en_US.UTF-8, CARGO_INCREMENTAL=0):
- cargo clippy -p rustfs --all-targets -- -D warnings: exit 0.
- cargo nextest run --profile ci -p rustfs --lib -E 'test(/storage::rpc::/)': 279 tests run, 279 passed, 4842 skipped.
- All 19 quick-check commands, including script-tests, both Python scripts and cargo fmt --all --check: exit 0.
- Footprint baseline regenerated: files=194, s3_error_lines=1584, ecstore_files=36; test-count baseline regenerated: 52 crates, 21228 tests, 164 ignored.
- All 54 Cargo.toml files parse without duplicate keys.

No assertion was added or changed by this lint fix. R1-R8 are retained
in the existing PR verification record.

* fix(ecstore): resume fenced Object Lock creation intents

* fix(ecstore): bound creation retry futures on the stack

* fix(ecstore): require native proof for creation retries

* fix(ecstore): bound bucket creation stack usage (#8466)

* fix(ecstore): retain creation fences through rollback

Keep lifecycle, metadata-transaction, and namespace guards around rollback and metadata cleanup. Preserve the original publication error when a guard is lost. Build the timestamp retry fixture through the real interrupted producer so it checks a valid pending creation.

Validated source tree: 4e8e207999.
Two independent source reviews found no blocking regression.

Validation (LANG=en_US.UTF-8):
- bash -c 'shellcheck --version && actionlint' — exit 0
- cargo fmt --all --check — exit 0
- ./scripts/check_unsafe_code_allowances.sh — exit 0
- ./scripts/check_layer_dependencies.sh — exit 0
- ./scripts/check_architecture_migration_rules.sh — exit 0
- ./scripts/check_logging_guardrails.sh — exit 0
- ./scripts/check_error_other_format_ratchet.sh — exit 0
- ./scripts/check_no_tokio_io_uring.sh — exit 0
- ./scripts/check_extension_schema_boundaries.sh — exit 0
- ./scripts/check_body_cache_whitelist.sh — exit 0
- ./scripts/check_s3s_footprint.sh — exit 0
- ./scripts/check_test_count_ratchet.sh — exit 0
- ./scripts/check_fips_wording.sh — exit 0
- ./scripts/check_embedded_secrets.sh — exit 0
- make script-tests — exit 0
- python3 ./scripts/test_object_generation_protocol_model.py — exit 0
- python3 ./scripts/check_test_wiring.py — exit 0
- ./scripts/check_no_planning_docs.sh — exit 0
- ./scripts/check_uring_lane_lib_only.sh — exit 0
- cargo nextest run --profile ci -p rustfs-ecstore --lib --locked --no-fail-fast — 5894 passed, 0 skipped
- cargo check -p rustfs-ecstore --all-targets --locked — exit 0
- cargo nextest run --profile ci -p rustfs --lib --locked --no-fail-fast -E 'test(=admin::handlers::table_catalog::tests::cold_strong_registration_completes_while_other_bucket_backup_waits) | test(=admin::handlers::table_catalog::tests::commit_backend_borrowed_put_preserves_owner_authorization_and_precondition) | test(=admin::handlers::table_catalog::tests::disable_table_bucket_preserves_objects_and_can_be_reenabled) | test(=admin::handlers::table_catalog::tests::load_table_handler_negotiates_vended_credentials_without_breaking_metadata_only_callers) | test(=admin::handlers::table_catalog::tests::namespace_property_handler_updates_object_backed_catalog_and_maps_errors) | test(=app::object::get::tests::execute_get_object_resumes_from_relocated_pool_without_splicing_body) | test(=table_catalog::store::tests::catalog_writer_loss_prevents_recovery_storage_commit)' — 7 passed
- cargo nextest run --profile ci -p rustfs --lib --locked --no-fail-fast -E 'test(unsupported_remote_disk_)' — 2 passed

Additional fault-injection evidence remains incomplete. Timestamp M1 reached its original assertion; its restore run is being collected. M2–M5 and the three planned rollback faults have not run. An automated risk check interrupted the timestamp proof task; no assertions or validation settings were weakened.

* fix(ecstore): box nested bucket rollback state

Keep the guarded rollback future behind a heap allocation so bucket creation does not store its nested state inline. The rollback body, all three fence references and pending-proof revocation order are unchanged.

The same three original Linux tests overflowed on both e611 CI attempts. Local format, ecstore test compilation and all 19 quick-check steps pass; the original Linux feature lanes still need to validate this candidate.

* test(ecstore): isolate rollback guard loss fixtures

Run the three rollback metadata-cleanup cases in separate test processes so reused /proc/self/fd aliases from earlier cases cannot invalidate their creation witnesses. Keep the original fixture, guards, assertions and io_uring test selection, and verify that each child runs the intended case successfully.

Update the generated test-count baseline for the added macro arm.

Validation: cargo check -p rustfs-ecstore --tests; cargo fmt --all --check; all 19 quick-check commands, including make script-tests and the two Python checks. Linux io_uring acceptance and fault controls remain pending.

* refactor(replication): move the replication config onto gateway types (#8443)

* wip: T1.4 replication family

* refactor(replication): move the replication config onto gateway types

The replication configuration family leaves s3s: rustfs-replication no longer depends on s3s, and ECStore bucket metadata, the scanner and the rustfs app/admin layers hold rustfs_gateway_types::persistence::PersistedReplicationConfiguration. The persisted XML goes through the gateway persistence codec behind a new BucketConfigXml trait (the 39 byte goldens pass with unchanged fixtures); the legacy s3s edge converts once in storage/ecfs.rs through storage/s3_compat/replication.rs; the scanner data usage cache keeps its MessagePack bytes through a serde mirror pinned by bytes recorded from s3s 5761ddfe. rustfs-gateway-types 0.32.0 is pinned once (rev 772d3385). Refs rustfs/backlog#2745.

* test: use S3 serializer for list response tests

* fix(ci): restore integration test compilation

Gate the diagnostic test import with its feature, initialize the conditional
MetadataWriteStore fields explicitly, and decode the scanner replication fixture
through its existing cache boundary. Preserve the original fixture and assertions.

Validation was run on tree 544acb80c0 at 6003d4b8.
LANG=en_US.UTF-8; no source changes follow validation.

Quick checks: 19/19 passed.
- shellcheck --version && actionlint: exit 0
- cargo fmt --all --check: exit 0
- ./scripts/check_unsafe_code_allowances.sh: exit 0
- ./scripts/check_layer_dependencies.sh: exit 0
- ./scripts/check_architecture_migration_rules.sh: exit 0
- ./scripts/check_logging_guardrails.sh: exit 0
- ./scripts/check_error_other_format_ratchet.sh: exit 0
- ./scripts/check_no_tokio_io_uring.sh: exit 0
- ./scripts/check_extension_schema_boundaries.sh: exit 0
- ./scripts/check_body_cache_whitelist.sh: exit 0
- ./scripts/check_s3s_footprint.sh: exit 0
- ./scripts/check_test_count_ratchet.sh: exit 0
- ./scripts/check_fips_wording.sh: exit 0
- ./scripts/check_embedded_secrets.sh: exit 0
- make script-tests: exit 0
- python3 ./scripts/test_object_generation_protocol_model.py: exit 0
- python3 ./scripts/check_test_wiring.py: exit 0
- ./scripts/check_no_planning_docs.sh: exit 0
- ./scripts/check_uring_lane_lib_only.sh: exit 0

Darwin validation:
- cargo clippy --locked -p rustfs-ecstore -p rustfs-scanner --all-targets -- -D warnings: exit 0
- cargo nextest run --locked -p rustfs-ecstore -p rustfs-scanner --lib -E 'test(config::com::tests::save_config_preserves_retryable_system_volume_errors) | test(scanner_io::tests::replication_maintenance_refreshes_compacted_stats_across_cycles_and_retries)' --test-threads 1 --no-fail-fast: exit 0, 2/2 passed
- cargo clippy --locked -p rustfs-ecstore -p rustfs-scanner --all-targets --features rustfs-ecstore/diagnostic-read-version-caller,rustfs-scanner/diagnostic-read-version-caller -- -D warnings: exit 101
- cargo nextest run --locked -p rustfs-ecstore -p rustfs-scanner --lib --features rustfs-ecstore/diagnostic-read-version-caller,rustfs-scanner/diagnostic-read-version-caller -E 'test(config::com::tests::save_config_preserves_retryable_system_volume_errors) | test(scanner_io::tests::replication_maintenance_refreshes_compacted_stats_across_cycles_and_retries) | test(cluster::rpc::runtime_sources::tests::) | test(config::com::tests::diagnostic_config_write_retains_origin_and_api_for_results_and_cancellation)' --test-threads 1 --no-fail-fast: exit 0, 14/14 passed

Diagnostic strict Clippy remains red on four unchanged production lints:
runtime_sources.rs if_same_then_else and three redundant_locals reports in
set_disk/core/metadata_read.rs and set_disk/read.rs. Their source ranges and
feature wiring match the base; the base was not compiled separately.

Linux aarch64 validation (Rust/Clippy 1.99.0, UID 501, jobs=1):
- cargo clippy --all-targets -- -D warnings: exit 0; Finished dev profile in 68m 12s
- cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings: exit 0; Finished dev profile in 49m 45s
Source HEAD/tree and clean status matched before and after both commands.
No compiler or stack overrides were used. protoc 35.1 was installed from its
verified ARM64 release; flatc was absent and not invoked by the workspace build scripts.
The local environment differs from hosted CI.

* fix(ecstore): recover delete batches with missing versions

Treat an explicit non-null version as a no-op when it is absent from both recovery sources. Keep rejecting missing null versions, delete-marker mismatches and conflicting local generations.

Add returning-disk and partial-receipt regressions, extend journal boundary checks, and regenerate the ecstore test-count baseline. Production faults exercise the 24 added assertion sites and 7 semantic checks: 33 faults failed at the expected sites and passed after full restoration.

Validation (LANG=en_US.UTF-8, one build job):

cargo nextest run --locked -p rustfs-ecstore --lib -E 'test(delete_recovery) | test(quorum_delete_cleanup_coordinator_binds_single_and_batch) | test(crash_delete_retry_batch_reclaims_inline_and_reserved_external)' --test-threads 1 --no-fail-fast
Summary [  34.580s] 69 tests run: 69 passed, 6058 skipped; exit 0

cargo clippy --locked -p rustfs-ecstore --all-targets -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 5m 37s; exit 0

All 19 final Quick Checks completed with exit 0:

- shellcheck --version && actionlint
- cargo fmt --all --check
- ./scripts/check_unsafe_code_allowances.sh
- ./scripts/check_layer_dependencies.sh
- ./scripts/check_architecture_migration_rules.sh
- ./scripts/check_logging_guardrails.sh
- ./scripts/check_error_other_format_ratchet.sh
- ./scripts/check_no_tokio_io_uring.sh
- ./scripts/check_extension_schema_boundaries.sh
- ./scripts/check_body_cache_whitelist.sh
- ./scripts/check_s3s_footprint.sh
- ./scripts/check_test_count_ratchet.sh
- ./scripts/check_fips_wording.sh
- ./scripts/check_embedded_secrets.sh
- make script-tests
- python3 ./scripts/test_object_generation_protocol_model.py
- python3 ./scripts/check_test_wiring.py
- ./scripts/check_no_planning_docs.sh
- ./scripts/check_uring_lane_lib_only.sh

test-count ratchet OK: 52 crates, 22009 tests, 165 ignored match .config/test-count-baseline.txt
PASS: cold stampede script self-test
model: states=51023, transitions=167520, terminal=732, chosen_candidates=2
multi-slot round schedules: 640
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired

* fix(ecstore): retry tier commit intent lock contention

Retry transient namespace lock timeouts within the existing three-attempt
CAS budget. Reload and verify the original mutation identity after
contention, and preserve the final typed error when attempts are exhausted.
Cover save/read contention, exhausted retries, replacement identities and
CAS reconciliation without changing the existing timeout or lock policy.

Validation:
- cargo nextest run --locked -p rustfs-ecstore --features test-util --lib -E 'test(tier_mutation)' --test-threads 1 --no-fail-fast: 52 passed.
- cargo nextest run --locked -p rustfs-ecstore --features test-util --lib -E 'test(intent_advance_) | test(=services::tier::tier::tests::caller_cancellation_after_durable_prepare_does_not_hide_the_mutation)' --test-threads 1 --no-fail-fast: 5 passed.
- cargo clippy --locked -p rustfs-ecstore --all-targets -- -D warnings: passed.
- cargo clippy --locked -p rustfs-ecstore --all-targets --features test-util -- -D warnings: passed.
- 27 mutation pairs failed at the intended assertions and passed after restoring the production code.
- bash scripts/check_test_count_ratchet.sh --update, then bash scripts/check_test_count_ratchet.sh: 52 crates, 22013 tests, 165 ignored; ecstore increases from 6581 to 6585.

* refactor(lifecycle): replace s3s domain types with Gateway DTOs

Keep lifecycle rules, restore requests, Object Lock, and scanner cached tags in domain types. Convert at the existing HTTP boundary while preserving legacy XML, scanner cache bytes, date ordering, and tombstones.

Linux test validation before the final import-path-only adjustment:

cargo nextest run -p rustfs-ecstore bucket_config_goldens
39 passed; 6211 skipped by selector.

cargo nextest run --lib -p rustfs -E 'test(/test_zero_rule_lifecycle_tombstone_recognition/) | test(/test_local_lifecycle_staleness_axis_selection/) | test(/test_lifecycle_staleness_preserves_legacy_submillisecond_axis/) | test(/test_merge_incoming_lifecycle/) | test(/test_lifecycle_expiry_statement_matrix/) | test(/test_lifecycle_expiry_subset_xml/) | test(/put_bucket_lifecycle_validation_errors_keep_their_s3_code/) | test(/execute_put_bucket_lifecycle_configuration_rejects_missing_configuration/) | test(/execute_restore_object_rejects_missing_restore_request/) | test(/execute_restore_object_rejects_select_type/) | test(/execute_restore_object_maps_failures_to_typed_s3_errors/) | test(/execute_restore_object_returns_internal_error_when_store_uninitialized/) | test(/restore_request_conversion_preserves_legacy_output_values/) | test(/restore_request_conversion_preserves_presence_and_raw_enums/) | test(/restore_request_conversion_distinguishes_absent_and_empty_containers/)'
25 passed; 5318 skipped by selector.

cargo nextest run --profile ci -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-replication -p rustfs-scanner
7872 passed; 21 skipped under the existing CI profile.

Previously verified mutation controls: 39 production faults, 41 RED/GREEN pairs.

cargo clippy --locked -p rustfs -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-scanner --all-targets -- -D warnings
Passed in 40m 45s before the import-path adjustment, then in 1m 59s on
the final source with warnings denied. The final adjustment imports the
existing helper through the object domain facade; fmt and the architecture
check also passed.

Part of rustfs/backlog#2745. The remaining ecstore configuration families follow separately.

* refactor(admin): move error types to rustfs-s3-types (#8440)

* refactor(protocols): move error types to rustfs-s3-types by codemod

T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and
the error items of protocols move off s3s by scripts/codemods, run with
--hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it
downcasts the backend error, whose runtime type is still s3s::S3Error.

- rustfs/src/protocols/client.rs: the two webdav session methods now
  return rustfs_s3_types::S3Result and convert the s3s error with From.
- codemod: a child module that calls s3_error! through `use super::*`
  gets its own `use rustfs_s3_types::s3_error;` once its parent imports
  that macro by name; otherwise it is listed as GLOB.
- s3-types: a table test pins code, status and the s3s error the legacy
  edge renders for every variant a moved call site names.
- s3_error_lines 1585 -> 1584 (a lifecycle comment reworded).

* refactor(admin): move error types to rustfs-s3-types (WIP, no build)

T1.2 batch 4 (rustfs/backlog#2743), work in progress: rustfs check
still fails with 16 errors (listed in the PR body).

- codemod: --facade PATH treats a crate-local re-export of s3s's error
  items like s3s for the four names (9 new self-tests, 9 mutants red).
- codemod output for rustfs/src/admin and rustfs/src/site_replication
  (--facade crate::storage_api::site_replication::s3); 6 glob children
  get an explicit rustfs_s3_types::s3_error import.
- admin/router.rs: the Operation trait returns rustfs_s3_types::S3Result.
- error.rs: From<ApiError> builds the rustfs-s3-types error; the s3s
  conversion is derived from it.

* fix(admin): convert error types at the s3s boundary

Batch 4 of T1.2 left the admin and site_replication modules returning
rustfs_s3_types::S3Error while their callers and callees on the storage
and auth side still speak s3s::S3Error. Convert at each edge with the
existing From impls in rustfs-s3-types (compat_s3s.rs) instead of
changing any code, message or status:

- admin/auth.rs, admin/handlers/object_zip_download.rs,
  admin/handlers/table_catalog/mod.rs: map the s3s result returned by
  check_key_valid / authorize_request / authorize_internal_object_request
  into the admin error type with .map_err(Into::into).
- storage/ecfs.rs: the s3s-facing side maps the admin site_replication
  gate results the other way.
- admin/handlers/pools.rs: ApiError.code stays s3s::S3ErrorCode, so the
  rustfs code is converted with .into() when the struct is built, in
  production and in the two tests that build or compare it.
- admin/handlers/heal.rs, admin/runtime_sources.rs: the router returns
  s3s errors (S3Route impl); the expected rustfs code is converted with
  .into() before the comparison.
- admin/handlers/table_catalog/mod.rs: iceberg_rest_error takes
  &'static str so the Cow<'static, str> custom code borrows it; every
  caller already passes an ICEBERG_ERROR_* constant or a &'static str.
- admin/handlers/quota.rs: drop a now same-type .into() after s3_error!.
- storage_api.rs, site_replication/mod.rs: drop re-exports and an import
  that no longer have users (clippy -D warnings).

cargo check -p rustfs --all-targets and cargo clippy -p rustfs
--all-targets -- -D warnings are clean; the s3s footprint baseline is
unchanged at files=172, s3_error_lines=303, ecstore_files=28.

Refs rustfs/backlog#2743, rustfs/backlog#2734.

* refactor(ecstore): move bucket configs onto gateway persisted types (#8709)

* refactor(ecstore): move bucket configs onto gateway persisted types

Finish the ECStore batch of T1.4 (rustfs/backlog#2745). The remaining
eleven bucket configuration families (Website, CORS, Tagging, Versioning,
ObjectLock, Encryption, Notification, Logging, Accelerate, RequestPayment,
PublicAccessBlock) now read and write their stored XML through the
rustfs-gateway-types persistence and cors_tagging codecs instead of the
s3s codec, so the legacy s3s bucket-config codec and its
s3gate-metadata-xml compat marker are removed.

- ecstore: BucketConfigXml is implemented by the gateway codec for every
  family; metadata, metadata_sys, versioning, object lock, replication and
  lifecycle consumers take the persisted types. Region moves to
  rustfs_s3_types::Region, and the ETag conversion to s3s moves to the
  rustfs edge with its tests. Only warm_backend.rs and warm_backend_s3.rs
  still reference s3s, held for T1.5 (rustfs/backlog#2746).
- scanner: object lock cache, item actions and data usage use the
  persisted types.
- rustfs: the legacy S3 edge converts each family between s3s DTOs and the
  persisted types in storage/storage_api.rs by writing one codec's XML and
  reading it with the other; Tagging converts field by field.
- Bump the rustfs-gateway-types pin to 21283e87.

The 39 persisted-byte goldens are unchanged and pass, so a document the
old codec wrote reads back and re-encodes to the same bytes. s3s footprint:
files 181 -> 146, ecstore_files 28 -> 2, s3_error_lines unchanged.

* test(app): prepare the PutBucketEncryption config from the request

The uninitialized-store test passed an empty persisted configuration, which the use case now validates before it reaches the store check, so it answered MalformedXML instead of InternalError. Convert the request's configuration the way the S3 edge does.

* style(app): format the PutBucketEncryption test

* refactor(s3-client): move remote error codes and DTO enums off s3s (#8783)

Part of T1.5 (rustfs/backlog#2746), batch 1.

* refactor(s3select): run queries on gateway DTO request shapes (#8794)

* refactor(s3select): run queries on an engine request built from gateway DTO shapes

Part of T1.5 (rustfs/backlog#2746), batch 2.

* fix(s3select): keep the select request conversion in the app layer

The layer guard forbids app -> storage; the conversion moves from storage/s3_compat/select.rs to app/select_request.rs unchanged.

* fix(heal): retry transient metadata conflicts (#8812)

* fix(heal): retry transient metadata conflicts

* chore: update heal test count baseline

* test: reserve nextest capacity for progress probes (#8818)

* test(e2e): use a non-default region for the differential negative control

Since main's region-aware CreateBucket and GetBucketLocation (#8739, #8743), us-east-1 answers an empty LocationConstraint and 200 for an owned re-create, so the control no longer reported its own region. Also refresh the e2e-full linux selection from the CI listing (682 tests).

* refactor(protocols): move StorageBackend to gateway types (#8825)

* refactor(protocols): move storage backend to gateway types

* test: update e2e test count baseline

* fix: preserve copy source keys and unknown lengths

* test: update protocols test count baseline

* fix(protocols): preserve typed SFTP backend errors

* test(protocols): use gateway list input in policy tests

* fix(protocols): clear clippy failures after T1.5 merge (#8831)

fix(protocols): clear clippy failures after merge

* fix(ecstore): finalize restore metadata after local copy-back (#8836)

* fix(ecstore): finalize restore metadata after local copy-back

* test(ecstore): ratchet added restore finalization cases

* fix(ecstore): keep restore copyback compatible with older peers

* fix(ecstore): gate restore copyback on fleet compatibility

* fix(ecstore): retain transition identity during restore copyback

* fix(ecstore): route committed restore copyback locally

* fix restore copyback cleanup

* fix(ecstore): honor restore probe test fallback

* refactor(storage): migrate option errors to domain types (#8839)

* refactor(s3): migrate private multipart validation errors (#8844)

* refactor(auth): derive policy conditions from AuthzFacts (#8442)

* refactor(auth): extract authorization facts

* test(e2e): update smoke selection baseline

* test(e2e): update full selection baseline

* refactor(admin): migrate health and prefix usage to owned requests (#8887)

* refactor(admin): add owned health router boundary

* refactor(admin): migrate prefix usage to owned request

* refactor(admin): complete the owned admin request boundary (#8949)

* refactor(admin): complete the owned admin request boundary

Every type implementing the owned admin `Operation` now serves the legacy
router through one blanket impl that converts the s3s request and response
at that single edge, so handlers register with `AdminOperation(&Handler {})`
and need no adapter statics; `LegacyAdminAdapter` is removed.

`AdminRequest` gains the accessors handlers read today (`remote_addr`,
`client_info`, `request_context`, `req_info`, `req_info_mut`,
`install_req_info`, `envelope_mut`, `take_body`), the request helpers gain
envelope forms (`authenticate_envelope_credentials`,
`object_store_from_envelope`, the table-catalog `*_from_envelope` helpers,
`KmsAdminAudit::from_envelope`, `AccountAuditContext::from_envelope`,
`CallerIdentity::resolve_envelope`), `read_compatible_admin_body` accepts
either body type, and the owned `Body` gains `store_all_limited` and
`Default`. `admin::operation::test_support::TestRequest` builds owned
requests for handler tests. Unused accessors carry a per-method dead-code
expectation that each migrating handler batch removes.

No handler behavior changes; the remaining migrations follow in batches.

* chore(admin): add the T1.12 admin handler codemod

The script rewrites an admin handler file from the s3s request and response
types to the owned boundary (imports, signatures, request field and helper
calls, body access, response construction, and test request literals) and
leaves a TODO comment at every site it cannot rewrite mechanically. It is
idempotent and is removed once T1.12 completes.

* refactor(admin): migrate replication and KMS handlers to owned requests (#8951)

Move the T1.12 batch 2 admin handlers (site_replication, kms_keys, event,
replication, diagnostics, on_demand_migration, audit, account, kms_rekey,
kms_key_lifecycle, object_data_cache, tls_debug, integrity) from the s3s
request/response types to the owned AdminRequest/AdminResponse boundary.
Handler logic, authorization order, error codes and response bytes are
unchanged; the request/response plumbing now goes through the envelope
accessors and helpers introduced by the owned boundary.

The shared response helpers (utils::json_response, utils::empty_response,
handlers::admin_json_response, target_descriptor::build_json_response,
profile::authorize_profile_request) and the site replication repair
executor still answer with the s3s types, so the migrated files carry
owned-boundary forms of those helpers and convert the repair response at
its call edge. The streaming console-log body and the two devnull drains
move from the s3s byte stream to the owned Body stream/frame API.

Drop the dead_code expectations for the owned accessors and envelope
helpers this batch starts using, and lower the s3s footprint baseline to
the measured 99 files.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* refactor(admin): add owned forms of the shared admin response helpers

Handler batches were copying `json_response`, `empty_response`,
`admin_json_response`, and `build_json_response` into each migrated file
because the shared helpers still return the s3s response. Provide owned
forms next to each legacy helper (`owned_json_response`,
`owned_empty_response`, `owned_admin_json_response`,
`owned_build_json_response`, `authorize_profile_envelope`) with parity
tests against the legacy shape, derive `Debug` for `AdminResponse` so
tests can use `expect_err`, and fix three codemod rules: strip owned
string conversions in test credentials, annotate non-literal URI parses,
and leave a rebound request parameter immutable.

* refactor(admin): use the shared owned response helpers in batch 2 (#8954)

Replace the per-file owned-boundary copies of json_response,
empty_response, admin_json_response, build_json_response and
authorize_profile_request in the batch 2 admin handlers with the shared
owned_* helpers and authorize_profile_envelope that the boundary branch
now provides, and restore expect_err in the handler tests now that
AdminResponse implements Debug; the `.err().expect(..)` stand-ins were
failing the base branch's clippy step (clippy::err_expect under
-D warnings). The site replication repair call edge
keeps its conversion because the repair executor still answers with the
s3s response type.

Drop the dead_code expectations on the shared owned helpers this batch
now uses in non-test code.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* fix(admin): preserve legacy body limit error responses

* refactor(admin): move policy, ILM and batch 3 handlers to owned requests (#8956)

Batch 3 of T1.12 (rustfs/backlog#2788, parent rustfs/backlog#2734) moves
14 admin handler files to AdminRequest/AdminResponse and the owned
Operation trait: ilm_transition, policies, idp_compat, config_admin,
plugins_instances, profile_admin, group, heal, object_zip_download,
kms_key_metadata, durability, inspect_archive, sts and
gateway_key_inventory. Handler logic, authorization order, error codes
and response bytes are unchanged; test request literals use
test_support::TestRequest.

The handlers use the shared owned helpers (owned_json_response,
owned_admin_json_response, owned_build_json_response,
authorize_profile_envelope). The dead_code expectations on the
AdminRequest accessors and RequestEnvelope::uri_mut they now exercise
are removed. The legacy target_descriptor::build_json_response keeps
only its equivalence test caller and carries the matching expectation
until it retires.

object_zip_download builds the s3s request the S3 gate still reads from
the owned request facts; heal authenticates through
authenticate_envelope_credentials; the trace, inspect-archive and zip
bodies use Body::from_stream. extract_plugin_instance_filters takes the
URI so extensions.rs keeps calling it, and the two idp_compat
delegations stay on the legacy trait until their delegates migrate.

The s3s footprint baseline drops from 99 to 90 files.

* test(admin): anchor the replication diff body contract on the call

The source-contract test looked for `let body = read_compatible_admin_body(`
on one line; rustfmt wrapped that binding after the handler moved to the
owned request boundary, so the marker was no longer found. Anchor on the
call itself so formatting cannot change what the contract checks.

* refactor(admin): migrate table catalog and batch 1 to owned requests (#8955)

* fix(admin): preserve catalog body limit response bytes

* refactor(admin): migrate identity and cluster handlers to owned requests (#8966)

* refactor(admin): migrate identity and cluster handlers to owned requests

Move the T1.12 batch 4 admin handlers (user, service_account, oidc, tier,
system, pools, quota, kms_dynamic, kms_backup, rebalance, module_switch,
bucket_meta, cluster_snapshot, realtime, bucket_recovery, profile) from the
s3s request/response types to the owned AdminRequest/AdminResponse
boundary. All 90 impl Operation blocks now take AdminRequest and return
AdminResponse; handler logic, authorization order, error codes and response
bytes are unchanged. The handlers use the shared owned helpers
(owned_json_response, owned_admin_json_response, authorize_profile_envelope,
the envelope-form runtime sources and audit constructors) instead of local
copies.

profile.rs drops the s3s forms of authorize_profile_request and
profile_not_implemented_response now that profile_admin.rs is on the
envelope gate too; profile_not_implemented_response answers with an
AdminResponse. The realtime metrics stream moves from the s3s ByteStream
adapter to Body::from_stream, and the rebalance idle-404 document still
renders through s3s::S3Error::to_http_response with its body converted at
the edge.

The idp_compat delegators ImportIamV2 and AddServiceAccountLdap move to the
owned trait as well: with their targets on both traits, the method-call
delegation became ambiguous between the owned impl and the router blanket
impl, so they now take the owned request directly.

KmsAdminAudit::from_request loses its last production caller with this batch
and is deleted; its comparison test now asserts the envelope facts directly.

Lower the s3s footprint baseline to the measured 68 files.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* refactor(admin): remove the legacy admin helpers batch 4 superseded

Batch 4 moved the last production callers of the s3s-typed admin helpers
to their owned forms, which left `admin_json_response`, `json_response`,
`authenticate_request`, `authorize_admin_request`, `object_store_from_req`
and `object_store_from_extensions` unused, along with the `Body`,
`S3Request` and `S3Response` re-exports of the admin `s3` facade. Remove
them. The owned forms keep their documentation; the behavior tests of the
removed `json_response` now exercise `owned_json_response`, and the
compat-prefix test asserts the owned response directly.

Two tests existed only to compare a removed legacy helper with its owned
twin (`owned_json_response_matches_the_legacy_shape`) or to cover the
removed legacy gate (`authorize_admin_request_without_credentials_is_rejected`);
they go with the helpers, and the owned equivalents stay. The rustfs
test-count row drops from 5957 to 5955 and the s3s footprint from 68 to 66
files.

* refactor(admin): drop the legacy target descriptor JSON response helper (#8968)

`target_descriptor::build_json_response` returned the s3s response and
had no production caller left once the event, audit and plugin instance
handlers moved to `owned_build_json_response`; it survived only for the
legacy/owned equivalence test. Remove it, give the owned helper its own
documentation, and make the test assert the owned response directly:
status, `application/json`, the echoed `x-request-id` when present and
only then, and the body bytes. The file no longer imports s3s, so the
footprint baseline drops from 66 to 65 files.

* test(ecstore): pin native history on cancellation under the cancellable queue

main moved the local write path onto the disk blocking queue, which skips an
operation once its waiter is dropped, so a borrowed write cancelled while it is
queued no longer creates its file. The integration-only native publication
test still expected the queued open to run. Assert the new execution model
instead (the open never runs) while keeping its safety claims: the Started
record is taken before queueing and the cancelled publication never regrants
native history.

* chore(gateway): raise test-count rows for tests merged from main

* test(s3select): use the owned JSON input types in merged planner tests

* fix(ecstore): scope the merged direct-writer test hook to test builds

* fix(ecstore): create parents before admitted truncate opens

Align the heal tests with removal-disabled preservation and local-only outcome
fields, and refresh the e2e-full selection for the synced tests.

* chore(gateway): update test-count baseline after main sync

Main #9241 removed the node pool-metadata readiness cache with 8 tests
and added 6, so the rustfs row drops from 6440 to 6438. The ecstore and
s3select rows rise for tests added by #9237, #9239 and #9241.

* feat(scripts): add junit_diff.py for dual-stack JUnit comparison (#9275)

* ci(gateway): make the test-count ratchet a floor, not exact (#9278)

ci(gateway): make the test-count ratchet a floor instead of an exact baseline

Tests below a row still fail; tests above it, a member without a row and ignores below the row now pass with a note. Ignores above a row still fail. Adds contract tests wired into make script-tests.

* chore(gateway): record ignored microbenchmarks merged from main (#9290)

chore(gateway): record the ignored microbenchmarks merged from main

main added three throughput microbenchmarks that are ignored by design
(#9217): one each in rustfs-checksums, rustfs-rio and rustfs-utils. The
test-count ratchet treats a higher ignored count as a failure until the
row is raised, so raise those three rows to 1 and tighten the test rows
to the merged tree with --update.

* ci(s3-tests): add reference parity lane against pinned 1.0.5 (#9288)

* test(app): add per-operation test seams for oracle use-case calls

* chore(codemods): add lifecycle transition test split codemod

* test(app): split lifecycle transition tests by operation owner

* test: call oracle use-cases through test seams in cross-owner tests

* ci: guard cross-owner tests to call use-cases through test seams

---------

Co-authored-by: rustfs-cla-assistant[bot] <273121464+rustfs-cla-assistant[bot]@users.noreply.github.com>
2026-11-11 06:22:25 -05:00
Chris An ff84f2b5bf fix(ci): fix mint runner and Dockerfile.source rc arch (#9310)
* fix(ci): run the mint lane on ubuntu-latest, which has a Docker daemon

* fix(docker): let BuildKit populate TARGETARCH for the rc download

* fix(docker): derive the rc architecture when TARGETARCH is not set
2026-11-11 06:18:31 -05:00
Chris An def5418168 test(e2e): expect ServiceUnavailable for degraded ListParts after #9149 (#9313)
* test(e2e): expect ServiceUnavailable for degraded ListParts after #9149

PR #9149 made ServiceUnavailable the default API code for storage
read-quorum failures and limited SlowDownRead to GET/HEAD object reads.
ListParts maps its storage error through ApiError::from, so a degraded
ListParts now answers 503 ServiceUnavailable. The two distributed
acceptance cases still expected the previous SlowDownRead code and have
failed every nightly e2e-distributed run since 2026-11-05.

The 503 status and the fail-closed (non-empty-list) assertions are
unchanged; only the error code tracks the intended contract.

* style(e2e): apply rustfmt to the ListParts error-code assertion
2026-11-11 06:18:24 -05:00
Chris An 85f5cc9ead feat(gateway-adapter): host, bucket-owner and request-id adapters (#9289)
* feat(server): add the RUSTFS_S3_STACK selector and dual-stack e2e lane (#8411)

* refactor(server): group the s3s-compat tower layers under legacy_compat (#8410)

Move the ten tower layers that patch s3s behaviour out of
rustfs/src/server/layer.rs into rustfs/src/server/legacy_compat/, one file
per layer with its tests. Each file header names the s3s behaviour it
patches, the gateway mechanism that replaces it (Replaced by:), and the task
that removes it (Removed by: T2.12, rustfs/backlog#2771). The layers keep
their code, their order in external_service_stack!, and their behaviour; no
layer is deleted.

Add server::http::tests::stack_census, which renders the ordered service
chain of the real external_service_stack! expansion and compares it with a
committed constant, so a reorder, insertion or removal is a visible diff.

Route the two s3s error types the moved layers use through a
storage_api::server::legacy_compat shim so the split adds no s3s-importing
file, and lower the s3s footprint baselines to the measured values (files 209
to 208, s3_error! lines 1586 to 1585). Keep the server runtime-source bypass
guard covering the moved code.

Refs rustfs/backlog#2787, rustfs/backlog#2734

* feat(s3-types): add the RustFS-owned S3 error type and s3_error! macro (#8406)

`rustfs-s3-types` now provides `S3ErrorCode` (generated by build.rs from
src/codes.txt, the grep-derived call surface, plus `Custom`), `S3Error`,
`S3Result`, `StdError` and the `s3_error` macro with the same call syntax
as s3s, so the T1.2 codemod is a `use` swap. Statuses are hand-written and
pinned to the s3s behaviour RustFS ships today by an oracle test that uses
s3s as a dev-dependency only. The `From` conversions for the legacy edge
live behind the off-by-default `compat-s3s` feature (the orphan rule
forbids them in rustfs/src/error.rs); `rustfs` enables it.

Refs rustfs/backlog#2742, rustfs/backlog#2734

* ci: add the per-crate test-count ratchet (#8404)

Quick Checks now fails any PR that lowers the number of test attributes
in a workspace crate or adds #[ignore], measured per crate against the
exact rows of .config/test-count-baseline.txt (rustfs/backlog#2734
section 1 rule 3). The counter widens the spec's literal
#[(tokio::)?test] to the tokio attributes that carry arguments and both
spellings of #[test_case(...)], and anchors at line start so
commented-out attributes do not count. The initial baseline covers 52
crates, 21077 tests and 164 ignored.

Refs rustfs/backlog#2738
Refs rustfs/backlog#2734

* test(ecstore): pin the persisted bucket-config XML bytes for 13 families (#8408)

Before any DTO or codec is replaced, pin the exact bytes that
deserialize followed by serialize produces for every persisted
bucket-configuration XML family in its three source shapes (AWS
documentation, MinIO-written, RustFS-written), and the bytes RustFS
writes today for a typed value. Byte equality only: a structural
comparison would let a codec replacement change the persisted encoding
unnoticed.

The generator script refuses to overwrite existing goldens without
--force, refuses on dirty codec paths, cannot report success from its
write mode, and records the generating commit in the fixtures README.

Refs rustfs/backlog#2744, rustfs/backlog#2734

* ci: raise the test-count baseline to the integration tree (#8426)

* ci: make the s3s footprint baseline exact and lower-only (#8403)

* ci: make the s3s footprint baseline exact and lower-only

The s3s footprint guard compared three counters against ceilings held in
the script itself, so a ceiling could sit above the tree and a drift was
fixable by raising it (rustfs/rustfs#5964 raised the s3_error! ceiling
to go green). The three values now live in
.config/s3s-footprint-baseline.txt as files, s3_error_lines and
ecstore_files, the script reads only that file, and a measurement that
differs from its baseline in either direction fails: growth prints
"s3s footprint grew: <measured> > <baseline>", shrinkage prints
"lower the baseline to <measured> in this PR". The counting method is
unchanged; the baseline is the current measurement of
gateway/integration (209 / 1585 / 36). The header records the end state:
once all three reach 0, T4.2 switches the script to failing on any
occurrence.

Refs rustfs/backlog#2739, rustfs/backlog#2734

* ci: add the s3s edge allowlist mode to the footprint guard (#8405)

`scripts/check_s3s_footprint.sh --mode allowlist` fails on every Rust
file that references s3s (`s3s::` or `s3s_sigv4`) and is not matched by
an entry of the new `.config/s3s-edge-allowlist.txt`; `--dry-run` lists
those files and exits 0. The allowlist carries the initial edge from the
spec, and the script rejects an entry that names or matches anything
under crates/ other than crates/e2e_test. The baseline mode and the
argument-less CI invocation are unchanged.

Refs rustfs/backlog#2741, rustfs/backlog#2734

* test(ecstore): reach the bucket-config codec through the test facade (#8428)

* refactor(s3-types): own the HTTP Body and drop s3s from three crates (#8431)

Add rustfs_s3_types::Body, a BoxBody<Bytes, StdError> wrapper with
empty() and From<Bytes | Vec<u8> | String | &'static str>, and the
legacy From conversions to and from s3s::Body behind compat-s3s.

The signer is now generic over the request body: it only reads and
writes the request head, so it needs no body type and its callers keep
theirs. s3-client sends its requests with the owned Body, madmin signs a
bodiless head, and targets drops its two S3Request/S3Response header
wrappers in favour of extract_params_header.

rustfs-signer, rustfs-madmin and rustfs-targets no longer depend on
s3s; the s3s footprint baseline drops from 209 to 202 files.

Refs rustfs/backlog#2747, rustfs/backlog#2734

* refactor(ecstore): move S3ErrorCode to rustfs-s3-types by codemod (#8432)

* test(ecstore): pin the InvalidRange responses of new_getobjectreader

Both error branches of new_getobjectreader build an ErrorResponse with S3ErrorCode::InvalidRange, and no test observed either code. Pin the code, its wire name and the location fields of each branch before the code type moves to rustfs-s3-types (rustfs/backlog#2743), so a changed code on either branch fails a test. The ecstore test-count row rises by the two tests.

* chore(scripts): add the s3s-to-rustfs-s3-types error codemod

Task T1.2 of rustfs/backlog#2734 (rustfs/backlog#2743) moves S3Error, S3ErrorCode, S3Result and s3_error! from s3s to rustfs-s3-types by script, one crate closure per PR. The codemod rewrites use trees (splitting mixed s3s imports) and qualified code paths, never comments or literals, adds the rustfs-s3-types dependency, drops s3s only from a crate that no longer names it, refuses every file of the s3s edge allowlist, takes --hold for a file whose type is fixed outside the closure, and prints a per-crate census. A second run changes nothing. Its self-test runs under make script-tests.

* refactor(ecstore): take S3ErrorCode from rustfs-s3-types

Output of scripts/codemods/s3_error_to_s3_types.py over crates/ecstore, crates/lifecycle, crates/scanner and crates/replication (rustfs/backlog#2743 batches 1 and 2). The ErrorResponse code field, the two InvalidRange constructions in new_getobjectreader and an unused tier import now use rustfs_s3_types::S3ErrorCode; the variant names and wire names are unchanged. lifecycle, scanner and replication reference none of the four error names. warm_backend.rs and warm_backend_s3.rs are held: they compare against rustfs_s3_client::ErrorResponse.code, which stays the s3s type until s3-client moves.

* ci(footprint): count only s3_error! lines still on s3s's macro

s3_error_lines counted every line holding the text `s3_error!`, which
also matches the RustFS-owned `rustfs_s3_types::s3_error!`. A codemod
that moves call sites onto rustfs-s3-types therefore could not lower it
(rustfs/backlog#2743, decision 3; rustfs/rustfs#8432, gotcha 2).

The counter now applies a textual, per-file and per-line rule that errs
on the side of counting (full rule in the script header):
- occurrences spelled `rustfs_s3_types::s3_error!` never count;
- a line holding `s3s::s3_error!` always counts;
- any other `s3_error!` line counts unless its file is cleared, that is,
  it imports the macro from rustfs_s3_types by name on a line starting
  with the use item (or defines it with `macro_rules! s3_error`) and does
  not also bring s3s's macro in (`use s3s::s3_error`, a `use s3s::{..}`
  tree naming it or a crate-root `*`, `use s3s::*`, or
  `#[macro_use] extern crate s3s`);
- fail closed: a file matching neither form (`use super::*`, a
  crate-local re-export, a renamed or glob rustfs_s3_types import, a
  commented-out import, a comment-only mention) counts.

Census of the 1585 lines on 96fc06776: 1478 in 64 files that import the
macro from s3s; 107 in 15 files matching neither form (105 through
`use super::*`, 1 qualified `s3s::s3_error!`, 1 comment line); none in
files that import it from rustfs_s3_types or define it. Every line stays
counted, so the baseline remains s3_error_lines=1585 (files=202 and
ecstore_files=36 unchanged). Rewriting rustfs/src/auth.rs's import to
rustfs_s3_types lowers the new counter to 1567; the old one stayed 1585.

`--self-test` (wired into `make script-tests`) copies the script into
22 fixture trees: 6 positive cases and 16 negative ones (lines that must
not count, forms that must not clear a file, and a missing baseline,
missing key or failing classification search that must fail). Each tree
has a control line that must count, and each run gets s3s text on stdin.
Seven cases were red against the old counter. 19 mutants each turn their
case red: old definition, flipped import test, dropped fail-closed
branch, dropped e2e_test exclusion, dropped s3s veto, dropped qualified
s3s rule, dropped rustfs-qualified strip, unanchored rustfs import,
renamed import accepted, nested glob taken as root glob, single-line
s3s trees only, dropped s3s glob forms, dropped macro_use form, dropped
definition clearing, ignored classification rg errors, error-line search
without its path, classification searches without their path, missing
baseline skipped, missing key skipped.

* test(e2e): compare two RUSTFS_S3_STACK servers over every S3 operation (#8435)

Add dual_stack_differential to e2e_test. It starts two rustfs processes
from the same binary, stack A on RUSTFS_S3_STACK=legacy and stack B on
RUSTFS_E2E_STACK_B (default legacy until rustfs/backlog#2734 T2.5),
sends both the same aws-sdk-s3 script covering the 73 methods of
impl S3 for FS (a normal request and a 4xx negative each), and compares
status, error code, an allowlist of response headers and a few typed
body fields. Error message text is never compared. Each run writes
<target>/dual-stack/<run>.json with compared_ops and the (op, field,
a, b) differences; any difference fails the test.

differential_detects_an_injected_difference starts B with
RUSTFS_REGION=eu-west-1 and requires GetBucketLocation to be reported.
The comparator is pure and unit-tested without a server.

Wire the module into the e2e-smoke filter and update the e2e-smoke and
e2e-full selection digests and the e2e_test test-count row.

Refs rustfs/backlog#2740

* feat(app): add the RustFS-owned RequestEnvelope and its s3s constructor (#8437)

* feat(app): add the RustFS-owned RequestEnvelope and its s3s constructor

* ci: classify the s3s edge adapters in storage/s3_compat as interface

* feat(s3-types): add Body::from_stream and move Swift off s3s::Body (#8438)

rustfs/backlog#2747 batch 3. `Body::from_stream` wraps a chunk stream of
unknown length over `StreamBody`/`BoxBody`: one data frame per chunk, the
first error ends the body, the size hint stays unknown, and dropping the
body drops the stream. Parity tests pin it to the s3s `StreamingBlob::wrap`
it replaces, at the `http_body` level and byte for byte through hyper.

The Swift handlers in rustfs-protocols build their responses as the owned
`Body`. `SwiftService` converts them into the wrapped S3 service's response
body type through `From<Body>`, so rustfs keeps serving `s3s::Body` at the
legacy edge unchanged. Request bodies stay Send-only (`UnsyncBoxBody`); the
two storage writers and the DLO/SLO segment streams get `Sync` from
`sync_wrapper::SyncStream`, which is what s3s's `http_body_unsync` mutex
provided before. The DTO and error uses of s3s in protocols are untouched
(T1.5 and T1.2).

* refactor(protocols): move error types to rustfs-s3-types by codemod (#8439)

T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and
the error items of protocols move off s3s by scripts/codemods, run with
--hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it
downcasts the backend error, whose runtime type is still s3s::S3Error.

- rustfs/src/protocols/client.rs: the two webdav session methods now
  return rustfs_s3_types::S3Result and convert the s3s error with From.
- codemod: a child module that calls s3_error! through `use super::*`
  gets its own `use rustfs_s3_types::s3_error;` once its parent imports
  that macro by name; otherwise it is listed as GLOB.
- s3-types: a table test pins code, status and the s3s error the legacy
  edge renders for every variant a moved call site names.
- s3_error_lines 1585 -> 1584 (a lifecycle comment reworded).

* test(connect): restore one-second drive request budgets

Apply the exact test-only repair already landed on main in 5972a09cc3. The filesystem probe and production CLI success test request one second and keep their existing one-second duration and parameter checks. All outcome, I/O, error, cleanup and resource-limit assertions remain.

Scope: rustfs/tests/connect_perf_drive.rs only. Patch-id: cb0a1dfc8f68a67727525adea7458e2cee7d09f8.

Validation environment: LANG=en_US.UTF-8, CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0, exclusive APFS-cloned CARGO_TARGET_DIR.

Quick Checks: every command from .github/actions/quick-checks/action.yml, with captured output and exit status below.

$ shellcheck --version && actionlint
ShellCheck - shell script analysis tool
version: 0.11.0
license: GNU General Public License, version 3
website: https://www.shellcheck.net
exit_code=0

$ cargo fmt --all --check
exit_code=0

$ ./scripts/check_unsafe_code_allowances.sh
exit_code=0

$ ./scripts/check_layer_dependencies.sh
Layer dependency guard passed (no new reverse dependencies/cycles).
exit_code=0

$ ./scripts/check_architecture_migration_rules.sh
Architecture migration rules passed.
exit_code=0

$ ./scripts/check_logging_guardrails.sh
✅ Logging guardrails check passed
exit_code=0

$ ./scripts/check_error_other_format_ratchet.sh
error(format!) ratchet passed (total call sites: 582).
exit_code=0

$ ./scripts/check_no_tokio_io_uring.sh
exit_code=0

$ ./scripts/check_extension_schema_boundaries.sh
Extension schema boundaries passed.
exit_code=0

$ ./scripts/check_body_cache_whitelist.sh
Body-cache whitelist guard passed.
exit_code=0

$ ./scripts/check_s3s_footprint.sh
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)
✅ s3s footprint baseline check passed
exit_code=0

$ ./scripts/check_test_count_ratchet.sh
test-count ratchet OK: 52 crates, 21222 tests, 164 ignored match .config/test-count-baseline.txt
exit_code=0

$ ./scripts/check_fips_wording.sh
FIPS wording guard passed (README.md and CHANGELOG.md contain no forbidden claims).
KMS wording guard passed (crates/kms claims no Vault KV2 Transit wrapping).
exit_code=0

$ ./scripts/check_embedded_secrets.sh
Embedded secret guard passed (no private key material or provider credential literal in the tree).
exit_code=0

$ make script-tests
Running script tests...
entrypoint credential policy tests passed
internode grpc A/B bench env tests passed
object batch benchmark enhanced tests passed
PASS package service script contract
hotpath warp A/B gate tests passed
hotpath warp ABBA tests passed
Scanner validation artifacts written to /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.1pFe6Kl1Yu/out
Scanner validation artifacts written to /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.1pFe6Kl1Yu/out-missing-pid
.............................
----------------------------------------------------------------------
Ran 29 tests in 181.286s

OK
{"decision":"verified","pending_gates":["G01","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G02"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-E"}
{"decision":"verified","pending_gates":["G01","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G02"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-E"}
{"decision":"verified","pending_gates":["G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G01"}
{"decision":"verified","pending_gates":["G01","G02","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G03"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-L"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G14"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G14"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G10"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P1"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P2"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P3"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G05"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G06"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-D"}
Status-and-outcome release descriptor verified: /private/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp8rwn1pe0/descriptor/release-bundle-status-outcome.json
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G05"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G06"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-D"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G07"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G08"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P4"}
exact 1MiB handoff ABBA tests passed
pinned paired ABBA bench tests passed
==> /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/bin/path_containment (-max_total_time=7, -seed=123456789, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/explicit-artifacts/path_containment/, corpus=/private/tmp/gw-orchestrator-2734/ci-drive/fuzz/corpus/path_containment)
-seed=123456789
target=path_containment
seed=123456789
max_total_time=7
git_revision=4d15b816236630b2c42f321ffc9ed3654edc94e3
git_dirty=true
runner_mode=prebuilt
==> /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/bin/path_containment (-max_total_time=1, -seed=2322617706, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/auto-artifacts/path_containment/, corpus=/private/tmp/gw-orchestrator-2734/ci-drive/fuzz/corpus/path_containment)
-seed=2322617706
==> cargo +nightly fuzz run path_containment (-max_total_time=9, -seed=987654321, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/cargo-artifacts/path_containment/)
+nightly
fuzz
run
path_containment
-seed=987654321
seed=987654321
runner_mode=cargo-fuzz
fuzz runner seed manifest tests passed
✅ scripts/python_bin.sh resolver checks passed
self-test ok: clean fixture with every exemption present passes
self-test ok: all 11 pattern families fire
self-test ok: moving a pinned source is reported
Embedded secret guard self-test passed.
self-test ok: p1-s3s-import: 'use s3s::s3_error;' counts every line, the rustfs_s3_types module's too (s3_error_lines=3)
self-test ok: p2-s3s-mixed-tree: a multi-line s3s use tree naming s3_error counts (s3_error_lines=3)
self-test ok: p3-s3s-glob: a crate-root glob of s3s, bare or inside a tree, counts (s3_error_lines=5)
self-test ok: p4-macro-use: '#[macro_use] extern crate s3s' counts (s3_error_lines=3)
self-test ok: p5-qualified-s3s: a qualified s3s::s3_error! line counts in a cleared file (s3_error_lines=2)
self-test ok: p6-exact-pass: a baseline equal to the tree passes (exit 0)
self-test ok: n1-rustfs-import: 'use rustfs_s3_types::s3_error;' alone counts nothing (s3_error_lines=1)
self-test ok: n2-rustfs-mixed-tree: a rustfs_s3_types tree plus an s3s tree without the macro counts nothing (s3_error_lines=1)
self-test ok: n3-defines: the file defining the macro counts nothing (s3_error_lines=1)
self-test ok: n4-qualified-rustfs: a qualified rustfs_s3_types::s3_error! line counts nothing (s3_error_lines=1)
self-test ok: n5-renamed-call: a call through a renamed macro is not text-matched (out of scope) (s3_error_lines=1)
self-test ok: n6-comment-string: a comment or string mentioning s3_error! in a cleared file counts nothing (s3_error_lines=1)
self-test ok: n7-e2e-excluded: crates/e2e_test is excluded (s3_error_lines=1)
self-test ok: n8-super-glob: an unrecognised import form ('use super::*;') counts (s3_error_lines=3)
self-test ok: n9-facade: a crate-local re-export counts (s3_error_lines=2)
self-test ok: n10-renamed-import: a renamed rustfs_s3_types import does not clear s3_error! lines (s3_error_lines=2)
self-test ok: n11-commented-import: a commented-out rustfs_s3_types import does not clear (s3_error_lines=2)
self-test ok: n12-rustfs-glob: a glob import of rustfs_s3_types does not clear (s3_error_lines=2)
self-test ok: n13-comment-only: a comment mentioning s3_error! in a file with no import counts (s3_error_lines=2)
self-test ok: f1-baseline-missing: a missing baseline file fails (exit 1)
self-test ok: f2-key-missing: a baseline without s3_error_lines fails (exit 1)
self-test ok: f3-rg-fails: a failing classification search fails the guard (exit 1)
✅ s3s footprint self-test passed
test_core_gate_rejects_missing_ignored_filtered_and_corrupt_inputs (__main__.SelfTests.test_core_gate_rejects_missing_ignored_filtered_and_corrupt_inputs) ... ok
test_e2e_requires_registration (__main__.SelfTests.test_e2e_requires_registration) ... ok
test_fuzz_runtime_uses_matrix_target (__main__.SelfTests.test_fuzz_runtime_uses_matrix_target) ... ok
test_ilm_lane_keeps_the_measured_cargo_build_budget (__main__.SelfTests.test_ilm_lane_keeps_the_measured_cargo_build_budget) ... ok
test_profile_listing_binds_platform_digest (__main__.SelfTests.test_profile_listing_binds_platform_digest) ... ok
test_profile_listing_enforces_selection (__main__.SelfTests.test_profile_listing_enforces_selection) ... ok
test_quick_checks_commands_propagate_failure (__main__.SelfTests.test_quick_checks_commands_propagate_failure) ... ok
test_quick_checks_rejects_caller_and_execution_bypasses (__main__.SelfTests.test_quick_checks_rejects_caller_and_execution_bypasses) ... ok
test_runner_readiness_fails_closed (__main__.SelfTests.test_runner_readiness_fails_closed) ... ok
test_s3_runner_rejects_unbounded_failure_locals (__main__.SelfTests.test_s3_runner_rejects_unbounded_failure_locals) ... ok
test_scanner_heal_begin_requires_embedded_source_provenance (__main__.SelfTests.test_scanner_heal_begin_requires_embedded_source_provenance) ... ok
test_scanner_heal_case_does_not_approve_pending_release (__main__.SelfTests.test_scanner_heal_case_does_not_approve_pending_release) ... ok
test_scanner_heal_case_lane_is_required_for_runner_profile (__main__.SelfTests.test_scanner_heal_case_lane_is_required_for_runner_profile) ... ok
test_scanner_heal_case_only_schema_cannot_approve_release (__main__.SelfTests.test_scanner_heal_case_only_schema_cannot_approve_release) ... ok
test_scanner_heal_crash_case_rejects_restart_oracle (__main__.SelfTests.test_scanner_heal_crash_case_rejects_restart_oracle) ... ok
test_scanner_heal_ec84_case_rejects_wrong_erasure_geometry (__main__.SelfTests.test_scanner_heal_ec84_case_rejects_wrong_erasure_geometry) ... ok
test_scanner_heal_finish_collects_oracles_from_registry (__main__.SelfTests.test_scanner_heal_finish_collects_oracles_from_registry) ... ok
test_scanner_heal_multipool_case_accepts_deferred_optional_outage_write (__main__.SelfTests.test_scanner_heal_multipool_case_accepts_deferred_optional_outage_write) ... ok
test_scanner_heal_pending_gate_cannot_map_to_implemented_lane (__main__.SelfTests.test_scanner_heal_pending_gate_cannot_map_to_implemented_lane) ... ok
test_scanner_heal_receipts_reject_reuse_and_missing_builds (__main__.SelfTests.test_scanner_heal_receipts_reject_reuse_and_missing_builds) ... ok
test_scanner_heal_rejects_boolean_fractional_and_out_of_geometry_integers (__main__.SelfTests.test_scanner_heal_rejects_boolean_fractional_and_out_of_geometry_integers) ... ok
test_scanner_heal_rejects_broken_execution_and_artifacts (__main__.SelfTests.test_scanner_heal_rejects_broken_execution_and_artifacts) ... ok
test_scanner_heal_rejects_copied_junit_and_wrong_suite_build (__main__.SelfTests.test_scanner_heal_rejects_copied_junit_and_wrong_suite_build) ... ok
test_scanner_heal_release_bundle_accepts_complete_measured_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_accepts_complete_measured_evidence) ... ok
test_scanner_heal_release_bundle_assembler_copies_measured_artifacts (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_copies_measured_artifacts) ... ok
test_scanner_heal_release_bundle_assembler_merges_measured_descriptors (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_merges_measured_descriptors) ... ok
test_scanner_heal_release_bundle_assembler_rejects_duplicate_descriptor_gate (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_duplicate_descriptor_gate) ... ok
test_scanner_heal_release_bundle_assembler_rejects_fixture_descriptor (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_fixture_descriptor) ... ok
test_scanner_heal_release_bundle_assembler_rejects_hash_mismatch (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_hash_mismatch) ... ok
test_scanner_heal_release_bundle_assembler_rejects_mixed_descriptor_revisions (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_mixed_descriptor_revisions) ... ok
test_scanner_heal_release_bundle_enforces_topology_duration_profile_and_versions (__main__.SelfTests.test_scanner_heal_release_bundle_enforces_topology_duration_profile_and_versions) ... ok
test_scanner_heal_release_bundle_fixture_writer_is_not_release_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_fixture_writer_is_not_release_evidence) ... ok
test_scanner_heal_release_bundle_gate_accepts_single_measured_gate (__main__.SelfTests.test_scanner_heal_release_bundle_gate_accepts_single_measured_gate) ... ok
test_scanner_heal_release_bundle_gate_rejects_missing_field (__main__.SelfTests.test_scanner_heal_release_bundle_gate_rejects_missing_field) ... ok
test_scanner_heal_release_bundle_gate_rejects_unknown_gate (__main__.SelfTests.test_scanner_heal_release_bundle_gate_rejects_unknown_gate) ... ok
test_scanner_heal_release_bundle_rejects_synthetic_or_missing_fields (__main__.SelfTests.test_scanner_heal_release_bundle_rejects_synthetic_or_missing_fields) ... ok
test_scanner_heal_release_bundle_requires_domain_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_requires_domain_evidence) ... ok
test_scanner_heal_release_bundle_requires_field_provenance (__main__.SelfTests.test_scanner_heal_release_bundle_requires_field_provenance) ... ok
test_scanner_heal_release_bundle_requires_json_artifact_provenance (__main__.SelfTests.test_scanner_heal_release_bundle_requires_json_artifact_provenance) ... ok
test_scanner_heal_release_bundle_requires_mixed_version_field_roles (__main__.SelfTests.test_scanner_heal_release_bundle_requires_mixed_version_field_roles) ... ok
test_scanner_heal_release_bundle_requires_same_gate_measurement_window (__main__.SelfTests.test_scanner_heal_release_bundle_requires_same_gate_measurement_window) ... ok
test_scanner_heal_release_consumes_multiple_registry_oracles (__main__.SelfTests.test_scanner_heal_release_consumes_multiple_registry_oracles) ... ok
test_scanner_heal_release_matrix_lanes_remain_pending_without_real_evidence (__main__.SelfTests.test_scanner_heal_release_matrix_lanes_remain_pending_without_real_evidence) ... ok
test_scanner_heal_release_requirements_cannot_be_cleared (__main__.SelfTests.test_scanner_heal_release_requirements_cannot_be_cleared) ... ok
test_scanner_heal_release_status_rejects_focused_case_run (__main__.SelfTests.test_scanner_heal_release_status_rejects_focused_case_run) ... ok
test_scanner_heal_release_status_rejects_synthetic_case (__main__.SelfTests.test_scanner_heal_release_status_rejects_synthetic_case) ... ok
test_scanner_heal_required_evidence_fields_cannot_be_removed (__main__.SelfTests.test_scanner_heal_required_evidence_fields_cannot_be_removed) ... ok
test_scheduled_alerts_require_completion_watchdog (__main__.SelfTests.test_scheduled_alerts_require_completion_watchdog) ... ok
test_update_profile_selection_changes_only_requested_platform (__main__.SelfTests.test_update_profile_selection_changes_only_requested_platform) ... ok
test_validate_includes_quick_checks (__main__.SelfTests.test_validate_includes_quick_checks) ... ok
test_vault_tests_require_cross_process_serialization (__main__.SelfTests.test_vault_tests_require_cross_process_serialization) ... ok
test_workflow_readiness_requires_dependency_probe (__main__.SelfTests.test_workflow_readiness_requires_dependency_probe) ... ok

----------------------------------------------------------------------
Ran 52 tests in 14.301s

OK
e2e-full selection OK: 1 tests, sha256=29e30e6ed2ed38d9e5b0a7cdd9536abf5c719f7021a02b578ae1381563a3c273
...................................
----------------------------------------------------------------------
Ran 35 tests in 1.161s

OK
...................
----------------------------------------------------------------------
Ran 19 tests in 17.397s

OK
..........
----------------------------------------------------------------------
Ran 10 tests in 0.183s

OK
test_actual_selector_bootstrap_uses_base_policy_and_fails_closed (__main__.SelfTests.test_actual_selector_bootstrap_uses_base_policy_and_fails_closed) ... ok
test_documentation_paths_do_not_hide_build_or_fixture_changes (__main__.SelfTests.test_documentation_paths_do_not_hide_build_or_fixture_changes) ... ok
test_e2e_scope_keeps_production_and_shared_configuration_full (__main__.SelfTests.test_e2e_scope_keeps_production_and_shared_configuration_full) ... ok
test_e2e_shortcut_rejects_new_direct_renamed_and_target_dependencies (__main__.SelfTests.test_e2e_shortcut_rejects_new_direct_renamed_and_target_dependencies) ... ok
test_e2e_shortcut_traverses_implicit_path_dependencies (__main__.SelfTests.test_e2e_shortcut_traverses_implicit_path_dependencies) ... ok
test_event_contract_requires_complete_candidate_and_optional_lanes (__main__.SelfTests.test_event_contract_requires_complete_candidate_and_optional_lanes) ... ok
test_every_wrong_result_missing_job_or_selection_fails_closed (__main__.SelfTests.test_every_wrong_result_missing_job_or_selection_fails_closed) ... ok
test_full_e2e_gate_preserves_workflow_branch_and_event_scope (__main__.SelfTests.test_full_e2e_gate_preserves_workflow_branch_and_event_scope) ... ok
test_git_range_includes_deleted_source_and_rename_origins (__main__.SelfTests.test_git_range_includes_deleted_source_and_rename_origins) ... ok
test_job_ids_and_display_names_cannot_hide_validation (__main__.SelfTests.test_job_ids_and_display_names_cannot_hide_validation) ... ok
test_report_only_stack_lane_mirrors_e2e_smoke_outside_the_aggregate (__main__.SelfTests.test_report_only_stack_lane_mirrors_e2e_smoke_outside_the_aggregate) ... ok
test_repository_wiring_and_missing_dependency_regression (__main__.SelfTests.test_repository_wiring_and_missing_dependency_regression) ... ok
test_verify_command_preserves_failures (__main__.SelfTests.test_verify_command_preserves_failures) ... ok

----------------------------------------------------------------------
Ran 13 tests in 3.280s

OK
security coverage self-test passed
test_api_failures_preserve_other_evidence_and_never_enter_grace (__main__.SelfTests.test_api_failures_preserve_other_evidence_and_never_enter_grace) ... ok
test_attempt_outcome_does_not_replace_recent_success (__main__.SelfTests.test_attempt_outcome_does_not_replace_recent_success) ... ok
test_cancelled_attempt_cannot_refresh_expired_success (__main__.SelfTests.test_cancelled_attempt_cannot_refresh_expired_success) ... ok
test_cli_requires_the_repository_default_branch (__main__.SelfTests.test_cli_requires_the_repository_default_branch) ... ok
test_config_rejects_duplicate_and_invalid_entries (__main__.SelfTests.test_config_rejects_duplicate_and_invalid_entries) ... ok
test_freshness_boundaries (__main__.SelfTests.test_freshness_boundaries) ... ok
test_grace_requires_two_successful_queries_with_no_history (__main__.SelfTests.test_grace_requires_two_successful_queries_with_no_history) ... ok
test_invalid_api_evidence_fails_closed (__main__.SelfTests.test_invalid_api_evidence_fails_closed) ... ok
test_report_retains_every_workflow (__main__.SelfTests.test_report_retains_every_workflow) ... ok
test_requests_filter_schedule_default_branch_and_success_on_server (__main__.SelfTests.test_requests_filter_schedule_default_branch_and_success_on_server) ... ok

----------------------------------------------------------------------
Ran 10 tests in 0.051s

OK
..............................
----------------------------------------------------------------------
Ran 30 tests in 22.354s

OK
..........
----------------------------------------------------------------------
Ran 10 tests in 2.969s

OK
.................................
----------------------------------------------------------------------
Ran 33 tests in 0.160s

OK
Online functional runner labels: pf-testing
.......
----------------------------------------------------------------------
Ran 7 tests in 0.002s

OK
.......
----------------------------------------------------------------------
Ran 7 tests in 0.001s

OK
....
----------------------------------------------------------------------
Ran 4 tests in 0.128s

OK
...
----------------------------------------------------------------------
Ran 3 tests in 0.547s

OK
PASS: cold stampede script self-test
exit_code=0

$ python3 ./scripts/test_object_generation_protocol_model.py
test_a_recovery_barrier_cannot_choose_an_empty_value (__main__.ObjectGenerationProtocolModel.test_a_recovery_barrier_cannot_choose_an_empty_value) ... ok
test_all_bounded_two_proposer_interleavings_choose_at_most_one_value (__main__.ObjectGenerationProtocolModel.test_all_bounded_two_proposer_interleavings_choose_at_most_one_value) ... ok
test_an_unavailable_decision_quorum_cannot_choose_a_candidate (__main__.ObjectGenerationProtocolModel.test_an_unavailable_decision_quorum_cannot_choose_a_candidate) ... ok
test_counter_allocation_requires_persistence_and_is_consumed_once (__main__.ObjectGenerationProtocolModel.test_counter_allocation_requires_persistence_and_is_consumed_once) ... ok
test_counter_restart_burns_persisted_allocations_before_or_after_issue (__main__.ObjectGenerationProtocolModel.test_counter_restart_burns_persisted_allocations_before_or_after_issue) ... ok
test_crash_after_promise_before_reply_does_not_restore_an_older_ballot (__main__.ObjectGenerationProtocolModel.test_crash_after_promise_before_reply_does_not_restore_an_older_ballot) ... ok
test_duplicate_or_unconfigured_voters_cannot_form_a_quorum (__main__.ObjectGenerationProtocolModel.test_duplicate_or_unconfigured_voters_cannot_form_a_quorum) ... ok
test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision (__main__.ObjectGenerationProtocolModel.test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision) ... ok
test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected (__main__.ObjectGenerationProtocolModel.test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected) ... ok
test_forgetting_tombstone_authority_allows_an_aba_create (__main__.ObjectGenerationProtocolModel.test_forgetting_tombstone_authority_allows_an_aba_create) ... ok
test_issuing_before_persistence_reuses_a_ballot_after_restart (__main__.ObjectGenerationProtocolModel.test_issuing_before_persistence_reuses_a_ballot_after_restart) ... ok
test_lost_ack_and_restart_preserve_the_chosen_operation (__main__.ObjectGenerationProtocolModel.test_lost_ack_and_restart_preserve_the_chosen_operation) ... ok
test_minority_accepted_work_is_adopted_and_finished (__main__.ObjectGenerationProtocolModel.test_minority_accepted_work_is_adopted_and_finished) ... ok
test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample (__main__.ObjectGenerationProtocolModel.test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample) ... ok
test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it (__main__.ObjectGenerationProtocolModel.test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it) ... ok
test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor (__main__.ObjectGenerationProtocolModel.test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor) ... ok
test_replacing_voters_with_empty_state_can_choose_a_conflicting_value (__main__.ObjectGenerationProtocolModel.test_replacing_voters_with_empty_state_can_choose_a_conflicting_value) ... ok
test_skipping_predecessor_recovery_chooses_a_broken_revision_chain (__main__.ObjectGenerationProtocolModel.test_skipping_predecessor_recovery_chooses_a_broken_revision_chain) ... ok
test_stale_publication_rollback_and_gc_cannot_change_the_winner (__main__.ObjectGenerationProtocolModel.test_stale_publication_rollback_and_gc_cannot_change_the_winner) ... ok
test_superseded_operation_identity_cannot_be_reused_for_different_input (__main__.ObjectGenerationProtocolModel.test_superseded_operation_identity_cannot_be_reused_for_different_input) ... ok
test_superseded_operation_retry_returns_its_original_successor (__main__.ObjectGenerationProtocolModel.test_superseded_operation_retry_returns_its_original_successor) ... ok
test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision (__main__.ObjectGenerationProtocolModel.test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision) ... ok
test_tombstone_rejects_a_delayed_never_present_create (__main__.ObjectGenerationProtocolModel.test_tombstone_rejects_a_delayed_never_present_create) ... ok
test_unlearned_predecessor_is_recovered_before_the_second_slot (__main__.ObjectGenerationProtocolModel.test_unlearned_predecessor_is_recovered_before_the_second_slot) ... ok

----------------------------------------------------------------------
Ran 24 tests in 0.958s

OK
model: states=51023, transitions=167520, terminal=732, chosen_candidates=2
negative control: double-choice trace=((0, 'prepare', 0), (0, 'prepare', 1), (0, 'prepare', 2), (0, 'accept', 0), (0, 'accept', 1), (0, 'accept', 2), (1, 'prepare', 0), (1, 'prepare', 1), (1, 'prepare', 2), (1, 'accept', 0), (1, 'accept', 1), (1, 'accept', 2))
multi-slot round schedules: 640
exit_code=0

$ python3 ./scripts/check_test_wiring.py
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired
exit_code=0

$ ./scripts/check_no_planning_docs.sh
exit_code=0

$ ./scripts/check_uring_lane_lib_only.sh
OK: no 'uring_'-matching test functions under crates/ecstore/tests; --lib is safe
exit_code=0

Initial complete gate
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
 Nextest run ID e07fc5b9-c6c8-4cdd-ae61-54839bf24437 with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.016s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.011s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.009s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.009s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  16.950s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   4.025s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.008s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.007s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.035s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  21.087s] 10 tests run: 10 passed, 0 skipped
exit_code=0

Restored complete gate
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
 Nextest run ID 233b7307-5361-4eb7-94d5-b384f17d2d6a with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.009s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.008s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.008s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.008s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  19.518s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   4.146s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.009s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.018s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.009s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.054s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  23.787s] 10 tests run: 10 passed, 0 skipped
exit_code=0

Mutation M1: success_measurement reports max(actual elapsed, requested budget + 1), forcing 1001 ms for the one-second request. The retained duration assertion at rustfs/tests/connect_perf_drive.rs:105 fails.
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast -E 'test(real_local_write_read_is_measured_and_scratch_is_removed)'
info: experimental features enabled: setup-scripts
   Compiling rustfs v1.0.1 (/private/tmp/gw-orchestrator-2734/ci-drive/rustfs)
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs") generated 1 warning
warning: `rustfs` (bin "rustfs-cli") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 7m 06s
────────────
 Nextest run ID 9f618777-7bf6-4cbc-8803-dd7f50f449ed with nextest profile: ci
    Starting 1 test across 1 binary (9 tests skipped)
        FAIL [   0.016s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
  stdout ───

    running 1 test
    test real_local_write_read_is_measured_and_scratch_is_removed ... FAILED

    failures:

    failures:
        real_local_write_read_is_measured_and_scratch_is_removed

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 0.01s

  stderr ───

    thread 'real_local_write_read_is_measured_and_scratch_is_removed' (106986846) panicked at rustfs/tests/connect_perf_drive.rs:105:5:
    assertion failed: (1..=1_000).contains(&data.duration_millis)
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

────────────
     Summary [   0.017s] 1 test run: 0 passed, 1 failed, 9 skipped
        FAIL [   0.016s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
error: test run failed
exit_code=100

Mutation M2: target_parameters reports requested duration + 1. The in-process assertion at rustfs/tests/connect_perf_drive.rs:111 and CLI JSON assertion at rustfs/tests/connect_perf_drive.rs:527 both reject 1001 against 1000.
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast -E 'test(real_local_write_read_is_measured_and_scratch_is_removed) | test(production_cli_measures_and_signs_exact_binary_provenance)'
info: experimental features enabled: setup-scripts
   Compiling rustfs v1.0.1 (/private/tmp/gw-orchestrator-2734/ci-drive/rustfs)
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs") generated 1 warning
warning: `rustfs` (bin "rustfs-cli") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 7m 20s
────────────
 Nextest run ID b73d46ab-c910-4e1e-8063-7fc7e332ec15 with nextest profile: ci
    Starting 2 tests across 1 binary (8 tests skipped)
        FAIL [  13.968s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
  stdout ───

    running 1 test
    test production_cli_measures_and_signs_exact_binary_provenance ... FAILED

    failures:

    failures:
        production_cli_measures_and_signs_exact_binary_provenance

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 13.96s

  stderr ───

    thread 'production_cli_measures_and_signs_exact_binary_provenance' (107015462) panicked at rustfs/tests/connect_perf_drive.rs:527:5:
    assertion `left == right` failed
      left: Number(1001)
     right: 1000
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

        FAIL [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
  stdout ───

    running 1 test
    test real_local_write_read_is_measured_and_scratch_is_removed ... FAILED

    failures:

    failures:
        real_local_write_read_is_measured_and_scratch_is_removed

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 0.01s

  stderr ───

    thread 'real_local_write_read_is_measured_and_scratch_is_removed' (107016667) panicked at rustfs/tests/connect_perf_drive.rs:111:5:
    assertion `left == right` failed
      left: 1001
     right: 1000
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

────────────
     Summary [  13.985s] 2 tests run: 0 passed, 2 failed, 8 skipped
        FAIL [  13.968s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        FAIL [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
error: test run failed
exit_code=100

Transient production mutations were restored byte-for-byte before the complete green rerun. Original/restored SHA256: 8d389e6d8836c041aab8c6a93e56addd65ba3f70b33264e1c74f639d0572a2da.

$ git diff --check

exit_code=0

$ git diff --cached --check

exit_code=0

Independent orchestrator acceptance
Tested code tree: e67c72fb01

Tested tree: 16fbbd2ea4
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

$ LANG=en_US.UTF-8 CARGO_BUILD_JOBS=1 CARGO_INCREMENTAL=0 cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
info: experimental features enabled: setup-scripts
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs-cli") generated 1 warning
warning: `rustfs` (bin "rustfs") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.69s
────────────
 Nextest run ID cd1edf4c-f1c0-456d-ad1a-76cc00ce05c9 with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.012s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.010s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.010s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.009s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  20.328s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   5.527s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.013s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.019s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.011s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.049s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  25.991s] 10 tests run: 10 passed, 0 skipped

exit_code=0

All 54 tracked Cargo.toml files parse without duplicate keys. git diff --check passed. The production source remains unchanged; the two deliberate implementation mutations above were restored before this complete green run.

* fix(test): compare legacy body wire parity as raw bytes

Lossy UTF-8 comparison concealed a changed binary response byte.
Compare the already captured Vec<u8> values directly.

Validation:
- Binary mutant: migrated payload 0x84 vs legacy 0x83; old comparator
  passed 1/1, raw comparator failed 0/1 (exit 100); restored green.
- Non-Swift body-drain mutant: untouched-body delegation test failed
  with empty vs payload (exit 100); restored handler without a diff.
- cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols
  --features rustfs-protocols/swift --no-fail-fast:
  543 tests run, 543 passed, 0 skipped after both mutants restored.
- cargo fmt --all --check and git diff --check passed.

Orchestrator acceptance (LANG=en_US.UTF-8):
cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
Summary [2.445s] 543 tests run: 543 passed, 0 skipped.

All shared quick checks on the final code tree:
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

Final independent acceptance after the drive-test CI repair
Parent: a99d8f4f89
Tested code tree: 93cbd96bf0

Tested tree: 075ae5161c2b62a96812bc1e3c904a7663b4f2f8
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

$ LANG=en_US.UTF-8 CARGO_BUILD_JOBS=1 CARGO_INCREMENTAL=0 cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
    Finished `test` profile [unoptimized + debuginfo] target(s) in 10m 08s
 Nextest run ID 5c7ff102-ec6d-479a-86e0-810859b28bbf with nextest profile: ci
    Starting 543 tests across 8 binaries
     Summary [   6.729s] 543 tests run: 543 passed, 0 skipped
exit_code=0

The only source change is the raw-byte wire comparator. Cargo manifests and the ratchet baselines remain unchanged from the independently validated parent. The invalid-byte and consumed-request mutations remain recorded above; their implementation changes were restored.

* fix: repair integration checks and IPC test readiness

Wait for the RPC/locks test sockets to pass the existing owner and mode
checks inside the original 5s readiness window and 10ms poll interval.

Use the existing admin S3 carrier for the new recovery and quota error
calls. Regenerate the test-count baseline for main's 118 existing tests
across seven crates. Pair the private pagination helper's raw-scan inputs
and remove the redundant Copy clone from the retirement event.

Keep the original assertions, responses, authorization, locks and storage
operations. The pagination body and raw-scan expressions at all five calls
are byte-identical; reinserting only the Copy clone restores the entire
base heal file. All ignored baseline values and s3s counts stay unchanged.

Validation base: c47432e353
Validation tree: f081084bc5
Patch SHA-256: 2dcf0028691bfb2772edfb38e262829b04f4151832dcce823c9dd9f7434b52de
Trace production-prefix SHA-256: fd994f218a21429d62866f4efb8eee1ee57b58f4a51865aa35d05a54ef1ccb45

All commands used LANG=en_US.UTF-8, CARGO_BUILD_JOBS=1,
CARGO_INCREMENTAL=0 and CARGO_NET_OFFLINE=true in one exclusive target.
Every gate retained the same tree and six source hashes before and after.
54 Cargo.toml files parse without duplicate keys.

cargo clippy -p rustfs --all-targets -- -D warnings
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 12m 50s
Exit: 0. Wall time: 770.855s.

The original 19 quick-check steps:
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
Wall time: 286.337s.

cargo nextest run --profile ci -p rustfs --no-fail-fast
     Summary [ 217.017s] 5382 tests run: 5382 passed (4 leaky), 40 skipped
Exit: 0. Wall time: 1324.414s.

cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
     Summary [   2.189s] 543 tests run: 543 passed, 0 skipped
Exit: 0. Wall time: 123.192s.

cargo nextest run --profile ci -p rustfs-ecstore list_objects_paginate --no-fail-fast
     Summary [   0.170s] 8 tests run: 8 passed, 5921 skipped
Exit: 0. Wall time: 261.324s.

cargo nextest run --profile ci -p rustfs-heal durable_mrf_anchor_retires_only_after_exact_bucket_generation_is_deleted --no-fail-fast
     Summary [   0.805s] 1 test run: 1 passed, 835 skipped
Exit: 0. Wall time: 199.801s.

Baseline regeneration:
./scripts/check_test_count_ratchet.sh --update
wrote .config/test-count-baseline.txt: 52 crates, 21340 tests, 164 ignored
Test-only increases: ecstore48, heal3, iam14, io-metrics2, obs3, policy14,
rustfs34. Existing ignored values and all other rows are byte-identical.
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)

Historical runtime red controls on f622 (same retained two assertions):
cargo nextest run --profile ci -p rustfs --lib -E 'test(=connect::diagnostics::trace_runtime::tests::local_top_rpc_reads_service_http_completions_and_signs_offline) | test(=connect::diagnostics::trace_runtime::tests::local_top_locks_reads_service_manager_and_signs_offline)' --no-fail-fast
Original path readiness with a temporary 500ms bind/seal gap:
     Summary [   0.863s] 2 tests run: 0 passed, 2 failed, 5113 skipped
Exit: 100.
Sealed readiness in that same gap:
     Summary [   5.621s] 2 tests run: 2 passed, 5113 skipped
Exit: 0.
Compiled M1: replace the helper predicate with path.exists():
     Summary [   0.870s] 2 tests run: 0 passed, 2 failed, 5113 skipped
Exit: 100.
Restore the sealed predicate byte-for-byte:
     Summary [   5.640s] 2 tests run: 2 passed, 5113 skipped
Exit: 0.
Both reds observe mode755 with owner_matches=true and fail at the original
RPC/locks unwraps with StateSecurity. M1 failure locations: trace_runtime.rs
2547:29 and 2713:29. No new assertions were introduced.

cargo nextest run --profile ci -p rustfs --lib -E 'test(connect::diagnostics::trace_runtime::tests::)' --no-fail-fast
     Summary [  20.748s] 28 tests run: 28 passed (1 leaky), 5087 skipped
Exit: 0. This security sweep is historical; the full final unit gate above
covers the actual c474 candidate.
The first disk-stopped compile ran no tests and is excluded from red credit.
All temporary seal instrumentation is removed from the final source.

Final unit cleanup markers: 4 leaky, 40 original skips (exit0). The ld
__eh_frame linker warning also appears verbatim in the original f622 run
and the earlier c474 run; there are no new source warnings.

* fix: remove redundant pending closure in disk test

Pass the capture-free pending future factory as a function item in the existing slow-operation test. This fixes the redundant_closure error at disk_store.rs:2914 that stopped strict Clippy before the feature tests could run.

The single-line test edit preserves the factory's result, warning and timeout behavior. Reversing the replacement reproduces the parent file byte for byte. All 220 assertion lines and the production prefix are unchanged; no tests, assertions, guards, profiles or baselines were added or relaxed.

Original Linux CI evidence (run 37888552268 at 53245be9f):
- RIO job113685483992: strict Clippy exit101, redundant closure at disk_store.rs:2914; feature tests skipped.
- Workspace Test and Lint job113685484058: same strict Clippy exit101; nextest and doc tests skipped.
These are lint failures, not runtime mutation kills or known flakes.

Native macOS arm64 validation on tree7a5bf47e7533b29479bc85207416102f1337c566, LANG=en_US.UTF-8, one build job, CARGO_INCREMENTAL=0 and offline dependencies:

shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
All 19 checks: exit0, 312.161s.

$ cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 20m 11s
exit0, 1213.210s total.

$ cargo nextest run --profile ci -p rustfs-ecstore --features rio-v2 slow_disk_operation_warns_once_while_pending_and_preserves_result --no-fail-fast
Finished `test` profile [unoptimized + debuginfo] target(s) in 4m 34s
PASS [0.026s] rustfs-ecstore disk::disk_store::tests::slow_disk_operation_warns_once_while_pending_and_preserves_result
Summary [   0.170s] 1 test run: 1 passed, 5943 skipped
exit0, 281.920s total; 5943 existing filter skips.

warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
This linker warning is retained; flags, profiles and allowances are unchanged.

54 tracked Cargo.toml files parsed without duplicate keys. git diff --check passed. No new assertions were introduced, so no new runtime mutation is claimed. Native compilation and the existing helper test do not substitute for actual Linux RIO execution; the new integration CI run still needs to recover before the RPC merge.

Preservation and output hashes:
parent file SHA256: 8e59ee745e3aaaa22848b943d3b10f6353b515956282cde1f3af2c394200c9ac
final file SHA256: b69ba3237061b161448ae8b62b615885d5770b98a5b4a2479bcfbd70f1bd1f17
unchanged production prefix SHA256: 7903a6e288252c420231242d7b58a25fa2a7140ea3dd430cbe7f0dde11369dcb
RIO Linux failure log SHA256:421addb9868fd88e4109a922311e54f91f0dfd70862f8ba7fcaba193482ed90b
Workspace Linux failure log SHA256:e0aea8d0f014c022252a9534968e083c98cfc1fd760da445ecd04ec3f522e26a
19 output SHA256:35a54f28d707956ed1edb29d492c4dfe8a9fe753d103b111b5a2f1b5c41c2b98
RIO Clippy output SHA256:d6046e3868f2b3992fe11cdd04e9e98eb50a87d1234d9a70dbfeb154ff61638d
slow-operation output SHA256:605b7b258d2fd09c6ef2ab2ffc90d433d0509480f1e45125d20c823759e83608

Refs rustfs/backlog#2734.

* test(ecstore): isolate tier-free recovery fixture startup

The fixture constructor can take the production recovery lock and wait for 60 seconds before this test installs its queue and seeds metadata. Disable its background recovery only while creating this fixture, then restore the environment before calling the existing production recovery entrypoint.

Only the existing test setup changes. Its production entrypoint, default lock parameters, original assertions, 30-second timeout, cancellation and cleanup remain intact. No tests or assertions were added. Production and store/init.rs are byte-identical to the parent.

Independent acceptance ran on merge tree 8ec358cb42ba9ebb3bcfeafcef048a3f0366355b; the runner header reports parent HEAD, while before/after index and source hashes confirm this candidate. Worker task branch d8055766b8 has the same tree.

Every command used LANG=en_US.UTF-8. Task compilation also used CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0, CARGO_NET_OFFLINE=true, an independent target, and empty RUSTC_WRAPPER/RUSTC_WORKSPACE_WRAPPER to keep compilers in its owned process group without changing repository configuration.

Command:
shellcheck --version && actionlint
Exit 0; observed output excerpt:
version: 0.11.0
Log SHA-256: fc9eec16b4191d058a295813ca9afbe9d2b37b59975da860170726be162a3924

Command:
cargo fmt --all --check
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_unsafe_code_allowances.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_layer_dependencies.sh
Exit 0; observed output excerpt:
Layer dependency guard passed (no new reverse dependencies/cycles).
Log SHA-256: 69c6cb1896c0586b28d1167380d3260d99d8017a4265c8df5a2348f289e93a12

Command:
./scripts/check_architecture_migration_rules.sh
Exit 0; observed output excerpt:
Architecture migration rules passed.
Log SHA-256: 28790e70d070607ed4de0704f292a65030be92ec224ef106d4b72fd8296f896e

Command:
./scripts/check_logging_guardrails.sh
Exit 0; observed output excerpt:
✅ Logging guardrails check passed
Log SHA-256: 7adf41abff8ed8f7617decf86498888ddde78937099dd9ce47f6d359dfc2d317

Command:
./scripts/check_error_other_format_ratchet.sh
Exit 0; observed output excerpt:
error(format!) ratchet passed (total call sites: 579).
Log SHA-256: 66172aef18f150ba0605af122e26ffcf91f6dd86f43e38b05e2e982ce8d783ee

Command:
./scripts/check_no_tokio_io_uring.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_extension_schema_boundaries.sh
Exit 0; observed output excerpt:
Extension schema boundaries passed.
Log SHA-256: f13d0acd0e2ce5d9815241b12b4dab7d01cc11b9b65bcf6dfb7812c878532a51

Command:
./scripts/check_body_cache_whitelist.sh
Exit 0; observed output excerpt:
Body-cache whitelist guard passed.
Log SHA-256: 6b828847b0383950ee6eef9cd846d0c322c945e47efd4c7d1ad881178810d6bb

Command:
./scripts/check_s3s_footprint.sh
Exit 0; observed output excerpt:
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)
✅ s3s footprint baseline check passed
Log SHA-256: 47f53d04ef645618563fe1f22abc64667f19078f1974feedcca46e4f2c016844

Command:
./scripts/check_test_count_ratchet.sh
Exit 0; observed output excerpt:
test-count ratchet OK: 52 crates, 21340 tests, 164 ignored match .config/test-count-baseline.txt
Log SHA-256: 618a5e330e933de0c6ff484546df401457024a88bb5986d11d588794e33dab60

Command:
./scripts/check_fips_wording.sh
Exit 0; observed output excerpt:
FIPS wording guard passed (README.md and CHANGELOG.md contain no forbidden claims).
KMS wording guard passed (crates/kms claims no Vault KV2 Transit wrapping).
Log SHA-256: 99221cc9979c480ec4f46f50d76c52b18c72fe499f12e15e4f4a22a1b8bfb140

Command:
./scripts/check_embedded_secrets.sh
Exit 0; observed output excerpt:
Embedded secret guard passed (no private key material or provider credential literal in the tree).
Log SHA-256: 437a3ceba32094d2901889965ad191e411aad32cdb18bf0b8659e818011c406b

Command:
make script-tests
Exit 0; observed output excerpt:
test_cli_requires_the_repository_default_branch (__main__.SelfTests.test_cli_requires_the_repository_default_branch) ... ok
test_config_rejects_duplicate_and_invalid_entries (__main__.SelfTests.test_config_rejects_duplicate_and_invalid_entries) ... ok
test_freshness_boundaries (__main__.SelfTests.test_freshness_boundaries) ... ok
test_grace_requires_two_successful_queries_with_no_history (__main__.SelfTests.test_grace_requires_two_successful_queries_with_no_history) ... ok
test_invalid_api_evidence_fails_closed (__main__.SelfTests.test_invalid_api_evidence_fails_closed) ... ok
test_report_retains_every_workflow (__main__.SelfTests.test_report_retains_every_workflow) ... ok
test_requests_filter_schedule_default_branch_and_success_on_server (__main__.SelfTests.test_requests_filter_schedule_default_branch_and_success_on_server) ... ok
Ran 10 tests in 0.042s
OK
Ran 30 tests in 20.627s
OK
Ran 10 tests in 2.268s
OK
Ran 33 tests in 0.127s
OK
Ran 7 tests in 0.002s
OK
Ran 7 tests in 0.001s
OK
Ran 4 tests in 0.108s
OK
Ran 3 tests in 0.564s
OK
PASS: cold stampede script self-test
Log SHA-256: 75fda2d069a373c63667b908baaf99e6591aab9099a739b0c4c22ceb6da8d68c

Command:
python3 ./scripts/test_object_generation_protocol_model.py
Exit 0; observed output excerpt:
test_an_unavailable_decision_quorum_cannot_choose_a_candidate (__main__.ObjectGenerationProtocolModel.test_an_unavailable_decision_quorum_cannot_choose_a_candidate) ... ok
test_counter_allocation_requires_persistence_and_is_consumed_once (__main__.ObjectGenerationProtocolModel.test_counter_allocation_requires_persistence_and_is_consumed_once) ... ok
test_counter_restart_burns_persisted_allocations_before_or_after_issue (__main__.ObjectGenerationProtocolModel.test_counter_restart_burns_persisted_allocations_before_or_after_issue) ... ok
test_crash_after_promise_before_reply_does_not_restore_an_older_ballot (__main__.ObjectGenerationProtocolModel.test_crash_after_promise_before_reply_does_not_restore_an_older_ballot) ... ok
test_duplicate_or_unconfigured_voters_cannot_form_a_quorum (__main__.ObjectGenerationProtocolModel.test_duplicate_or_unconfigured_voters_cannot_form_a_quorum) ... ok
test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision (__main__.ObjectGenerationProtocolModel.test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision) ... ok
test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected (__main__.ObjectGenerationProtocolModel.test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected) ... ok
test_forgetting_tombstone_authority_allows_an_aba_create (__main__.ObjectGenerationProtocolModel.test_forgetting_tombstone_authority_allows_an_aba_create) ... ok
test_issuing_before_persistence_reuses_a_ballot_after_restart (__main__.ObjectGenerationProtocolModel.test_issuing_before_persistence_reuses_a_ballot_after_restart) ... ok
test_lost_ack_and_restart_preserve_the_chosen_operation (__main__.ObjectGenerationProtocolModel.test_lost_ack_and_restart_preserve_the_chosen_operation) ... ok
test_minority_accepted_work_is_adopted_and_finished (__main__.ObjectGenerationProtocolModel.test_minority_accepted_work_is_adopted_and_finished) ... ok
test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample (__main__.ObjectGenerationProtocolModel.test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample) ... ok
test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it (__main__.ObjectGenerationProtocolModel.test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it) ... ok
test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor (__main__.ObjectGenerationProtocolModel.test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor) ... ok
test_replacing_voters_with_empty_state_can_choose_a_conflicting_value (__main__.ObjectGenerationProtocolModel.test_replacing_voters_with_empty_state_can_choose_a_conflicting_value) ... ok
test_skipping_predecessor_recovery_chooses_a_broken_revision_chain (__main__.ObjectGenerationProtocolModel.test_skipping_predecessor_recovery_chooses_a_broken_revision_chain) ... ok
test_stale_publication_rollback_and_gc_cannot_change_the_winner (__main__.ObjectGenerationProtocolModel.test_stale_publication_rollback_and_gc_cannot_change_the_winner) ... ok
test_superseded_operation_identity_cannot_be_reused_for_different_input (__main__.ObjectGenerationProtocolModel.test_superseded_operation_identity_cannot_be_reused_for_different_input) ... ok
test_superseded_operation_retry_returns_its_original_successor (__main__.ObjectGenerationProtocolModel.test_superseded_operation_retry_returns_its_original_successor) ... ok
test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision (__main__.ObjectGenerationProtocolModel.test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision) ... ok
test_tombstone_rejects_a_delayed_never_present_create (__main__.ObjectGenerationProtocolModel.test_tombstone_rejects_a_delayed_never_present_create) ... ok
test_unlearned_predecessor_is_recovered_before_the_second_slot (__main__.ObjectGenerationProtocolModel.test_unlearned_predecessor_is_recovered_before_the_second_slot) ... ok
Ran 24 tests in 0.929s
OK
Log SHA-256: 0b4be926efb0609d497ae58930340153f01957b606206cbbdb3e0d47eb29d3a2

Command:
python3 ./scripts/check_test_wiring.py
Exit 0; observed output excerpt:
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired
Log SHA-256: 21f676f458451525535ab0ef73d6a9c0d9774e16d76a4d19e396ad34936dd39c

Command:
./scripts/check_no_planning_docs.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_uring_lane_lib_only.sh
Exit 0; observed output excerpt:
OK: no 'uring_'-matching test functions under crates/ecstore/tests; --lib is safe
Log SHA-256: 43f3f851e27ec2ab9e34d6b05fefdede38af992699c473904bfe6f53a46d52e3

Command:
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast
Exit 0; wall 234.399s.
Observed output:
    Finished `test` profile [unoptimized + debuginfo] target(s) in 3m 49s
    Starting 1 test across 1 binary (5871 tests skipped)
  SETUP PASS [   0.006s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"'
        PASS [   0.440s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item
     Summary [   0.586s] 1 test run: 1 passed, 5871 skipped
Log SHA-256: 34e973f86e8dc2102c1a0e483e0ac802ce7df3d1b40a603932f62c0805c4de27

Command:
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over) | test(=store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence) | test(=store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable)' --no-fail-fast
Exit 0; wall 4.010s.
Observed output:
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.43s
    Starting 3 tests across 1 binary (5869 tests skipped)
  SETUP PASS [   0.005s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"'
        PASS [   0.068s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over
        PASS [   0.637s] rustfs-ecstore store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence
        PASS [   0.846s] rustfs-ecstore store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable
     Summary [   0.987s] 3 tests run: 3 passed, 5869 skipped
Log SHA-256: b5da2a927ff18ca9aa217762f7186a6d4371bd9e8feedd7b77940e10b64dffac

Command:
cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
Exit 0; wall 1084.508s.
Observed output:
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 18m 03s
Log SHA-256: 28f98be7970faaa71cd118da1da4e6e2e3d23b9dda7e6449eaa7a37e741260b5

Compiled mutation command (unchanged for each red and restored run):
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast

M1: reverse only this test-local constructor flag from false to true. The mutant compiled, then failed the original recovered-item queue assertion after 30.481s. Restoring the entire observed candidate passed in 0.460s. Red log 55bf3b1535529ac0bcfb8071f9b56f9f40631007a85c315367e7e93786d65293; restored log 204d5fbe4574a7917f5e740a795d864fe28a7e7fb5e33d27b0a9cf68e2f17f52.

M2: replace only the actual awaited enqueue with false, consuming its original inputs. The mutant compiled, then failed the same original queue assertion after 30.516s. Restoring the entire observed candidate passed in 0.457s. Red log 89186330413215e231e2b06bf013cc31097ea8f4a106d91421a36d6c8986a7fd; restored log e848b42d2302401afa5dbcd9e37bf76d2e56a96ab6de5a488513b997f53c5e9b.

All temporary observation and mutation code was removed before final acceptance. Historical baseline and diagnostic failures remain recorded; they are not mutation credit. The separate Object Lock intent-retry diagnostic still fails, and these native macOS checks do not establish Linux CI recovery.

* refactor(rpc): serve internode rpc bodies with the owned Body (#8441)

* test(rpc): pin the wire bytes of every internode rpc response body

rustfs/backlog#2747 batch 4, step 1 of 2, on the s3s bodies the rpc paths
serve today. Each response body builder in storage/rpc/http_service.rs is
served through hyper's HTTP/1.1 server and the raw bytes the peer reads are
compared with goldens captured from this commit: status, content-length or
chunked framing, chunk boundaries, the terminating chunk, HEAD, and the cut
connection a failure after data produces. The capability responses carry a
per-process proof, so their length header and payload are checked against
the body they encode instead.

The read_file_stream response builder moves into read_file_stream_response
so the test reaches the same code the handler runs; behaviour is unchanged.

* refactor(rpc): serve internode rpc bodies with the owned Body

rustfs/backlog#2747 batch 4, step 2 of 2. The internode rpc HTTP paths
build their responses with rustfs_s3_types::Body: Body::from_stream
replaces Body::from(StreamingBlob::wrap(..)) at the six stream sites, and
the in-memory and empty bodies keep their constructors. InternodeRpcService
converts its responses into the wrapped service's response body through
From<Body>, so the server keeps composing it with the s3s service as
before. The wire-byte goldens from the previous commit pass unchanged;
storage/rpc no longer names s3s.

* fix(rpc): simplify wire test response builder type

Name the boxed response builder used by the capability wire test so
all-target Clippy can check it without a type-complexity warning.

Validation (LANG=en_US.UTF-8, CARGO_INCREMENTAL=0):
- cargo clippy -p rustfs --all-targets -- -D warnings: exit 0.
- cargo nextest run --profile ci -p rustfs --lib -E 'test(/storage::rpc::/)': 279 tests run, 279 passed, 4842 skipped.
- All 19 quick-check commands, including script-tests, both Python scripts and cargo fmt --all --check: exit 0.
- Footprint baseline regenerated: files=194, s3_error_lines=1584, ecstore_files=36; test-count baseline regenerated: 52 crates, 21228 tests, 164 ignored.
- All 54 Cargo.toml files parse without duplicate keys.

No assertion was added or changed by this lint fix. R1-R8 are retained
in the existing PR verification record.

* fix(ecstore): resume fenced Object Lock creation intents

* fix(ecstore): bound creation retry futures on the stack

* fix(ecstore): require native proof for creation retries

* fix(ecstore): bound bucket creation stack usage (#8466)

* fix(ecstore): retain creation fences through rollback

Keep lifecycle, metadata-transaction, and namespace guards around rollback and metadata cleanup. Preserve the original publication error when a guard is lost. Build the timestamp retry fixture through the real interrupted producer so it checks a valid pending creation.

Validated source tree: 4e8e207999.
Two independent source reviews found no blocking regression.

Validation (LANG=en_US.UTF-8):
- bash -c 'shellcheck --version && actionlint' — exit 0
- cargo fmt --all --check — exit 0
- ./scripts/check_unsafe_code_allowances.sh — exit 0
- ./scripts/check_layer_dependencies.sh — exit 0
- ./scripts/check_architecture_migration_rules.sh — exit 0
- ./scripts/check_logging_guardrails.sh — exit 0
- ./scripts/check_error_other_format_ratchet.sh — exit 0
- ./scripts/check_no_tokio_io_uring.sh — exit 0
- ./scripts/check_extension_schema_boundaries.sh — exit 0
- ./scripts/check_body_cache_whitelist.sh — exit 0
- ./scripts/check_s3s_footprint.sh — exit 0
- ./scripts/check_test_count_ratchet.sh — exit 0
- ./scripts/check_fips_wording.sh — exit 0
- ./scripts/check_embedded_secrets.sh — exit 0
- make script-tests — exit 0
- python3 ./scripts/test_object_generation_protocol_model.py — exit 0
- python3 ./scripts/check_test_wiring.py — exit 0
- ./scripts/check_no_planning_docs.sh — exit 0
- ./scripts/check_uring_lane_lib_only.sh — exit 0
- cargo nextest run --profile ci -p rustfs-ecstore --lib --locked --no-fail-fast — 5894 passed, 0 skipped
- cargo check -p rustfs-ecstore --all-targets --locked — exit 0
- cargo nextest run --profile ci -p rustfs --lib --locked --no-fail-fast -E 'test(=admin::handlers::table_catalog::tests::cold_strong_registration_completes_while_other_bucket_backup_waits) | test(=admin::handlers::table_catalog::tests::commit_backend_borrowed_put_preserves_owner_authorization_and_precondition) | test(=admin::handlers::table_catalog::tests::disable_table_bucket_preserves_objects_and_can_be_reenabled) | test(=admin::handlers::table_catalog::tests::load_table_handler_negotiates_vended_credentials_without_breaking_metadata_only_callers) | test(=admin::handlers::table_catalog::tests::namespace_property_handler_updates_object_backed_catalog_and_maps_errors) | test(=app::object::get::tests::execute_get_object_resumes_from_relocated_pool_without_splicing_body) | test(=table_catalog::store::tests::catalog_writer_loss_prevents_recovery_storage_commit)' — 7 passed
- cargo nextest run --profile ci -p rustfs --lib --locked --no-fail-fast -E 'test(unsupported_remote_disk_)' — 2 passed

Additional fault-injection evidence remains incomplete. Timestamp M1 reached its original assertion; its restore run is being collected. M2–M5 and the three planned rollback faults have not run. An automated risk check interrupted the timestamp proof task; no assertions or validation settings were weakened.

* fix(ecstore): box nested bucket rollback state

Keep the guarded rollback future behind a heap allocation so bucket creation does not store its nested state inline. The rollback body, all three fence references and pending-proof revocation order are unchanged.

The same three original Linux tests overflowed on both e611 CI attempts. Local format, ecstore test compilation and all 19 quick-check steps pass; the original Linux feature lanes still need to validate this candidate.

* test(ecstore): isolate rollback guard loss fixtures

Run the three rollback metadata-cleanup cases in separate test processes so reused /proc/self/fd aliases from earlier cases cannot invalidate their creation witnesses. Keep the original fixture, guards, assertions and io_uring test selection, and verify that each child runs the intended case successfully.

Update the generated test-count baseline for the added macro arm.

Validation: cargo check -p rustfs-ecstore --tests; cargo fmt --all --check; all 19 quick-check commands, including make script-tests and the two Python checks. Linux io_uring acceptance and fault controls remain pending.

* refactor(replication): move the replication config onto gateway types (#8443)

* wip: T1.4 replication family

* refactor(replication): move the replication config onto gateway types

The replication configuration family leaves s3s: rustfs-replication no longer depends on s3s, and ECStore bucket metadata, the scanner and the rustfs app/admin layers hold rustfs_gateway_types::persistence::PersistedReplicationConfiguration. The persisted XML goes through the gateway persistence codec behind a new BucketConfigXml trait (the 39 byte goldens pass with unchanged fixtures); the legacy s3s edge converts once in storage/ecfs.rs through storage/s3_compat/replication.rs; the scanner data usage cache keeps its MessagePack bytes through a serde mirror pinned by bytes recorded from s3s 5761ddfe. rustfs-gateway-types 0.32.0 is pinned once (rev 772d3385). Refs rustfs/backlog#2745.

* test: use S3 serializer for list response tests

* fix(ci): restore integration test compilation

Gate the diagnostic test import with its feature, initialize the conditional
MetadataWriteStore fields explicitly, and decode the scanner replication fixture
through its existing cache boundary. Preserve the original fixture and assertions.

Validation was run on tree 544acb80c0 at 6003d4b8.
LANG=en_US.UTF-8; no source changes follow validation.

Quick checks: 19/19 passed.
- shellcheck --version && actionlint: exit 0
- cargo fmt --all --check: exit 0
- ./scripts/check_unsafe_code_allowances.sh: exit 0
- ./scripts/check_layer_dependencies.sh: exit 0
- ./scripts/check_architecture_migration_rules.sh: exit 0
- ./scripts/check_logging_guardrails.sh: exit 0
- ./scripts/check_error_other_format_ratchet.sh: exit 0
- ./scripts/check_no_tokio_io_uring.sh: exit 0
- ./scripts/check_extension_schema_boundaries.sh: exit 0
- ./scripts/check_body_cache_whitelist.sh: exit 0
- ./scripts/check_s3s_footprint.sh: exit 0
- ./scripts/check_test_count_ratchet.sh: exit 0
- ./scripts/check_fips_wording.sh: exit 0
- ./scripts/check_embedded_secrets.sh: exit 0
- make script-tests: exit 0
- python3 ./scripts/test_object_generation_protocol_model.py: exit 0
- python3 ./scripts/check_test_wiring.py: exit 0
- ./scripts/check_no_planning_docs.sh: exit 0
- ./scripts/check_uring_lane_lib_only.sh: exit 0

Darwin validation:
- cargo clippy --locked -p rustfs-ecstore -p rustfs-scanner --all-targets -- -D warnings: exit 0
- cargo nextest run --locked -p rustfs-ecstore -p rustfs-scanner --lib -E 'test(config::com::tests::save_config_preserves_retryable_system_volume_errors) | test(scanner_io::tests::replication_maintenance_refreshes_compacted_stats_across_cycles_and_retries)' --test-threads 1 --no-fail-fast: exit 0, 2/2 passed
- cargo clippy --locked -p rustfs-ecstore -p rustfs-scanner --all-targets --features rustfs-ecstore/diagnostic-read-version-caller,rustfs-scanner/diagnostic-read-version-caller -- -D warnings: exit 101
- cargo nextest run --locked -p rustfs-ecstore -p rustfs-scanner --lib --features rustfs-ecstore/diagnostic-read-version-caller,rustfs-scanner/diagnostic-read-version-caller -E 'test(config::com::tests::save_config_preserves_retryable_system_volume_errors) | test(scanner_io::tests::replication_maintenance_refreshes_compacted_stats_across_cycles_and_retries) | test(cluster::rpc::runtime_sources::tests::) | test(config::com::tests::diagnostic_config_write_retains_origin_and_api_for_results_and_cancellation)' --test-threads 1 --no-fail-fast: exit 0, 14/14 passed

Diagnostic strict Clippy remains red on four unchanged production lints:
runtime_sources.rs if_same_then_else and three redundant_locals reports in
set_disk/core/metadata_read.rs and set_disk/read.rs. Their source ranges and
feature wiring match the base; the base was not compiled separately.

Linux aarch64 validation (Rust/Clippy 1.99.0, UID 501, jobs=1):
- cargo clippy --all-targets -- -D warnings: exit 0; Finished dev profile in 68m 12s
- cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings: exit 0; Finished dev profile in 49m 45s
Source HEAD/tree and clean status matched before and after both commands.
No compiler or stack overrides were used. protoc 35.1 was installed from its
verified ARM64 release; flatc was absent and not invoked by the workspace build scripts.
The local environment differs from hosted CI.

* fix(ecstore): recover delete batches with missing versions

Treat an explicit non-null version as a no-op when it is absent from both recovery sources. Keep rejecting missing null versions, delete-marker mismatches and conflicting local generations.

Add returning-disk and partial-receipt regressions, extend journal boundary checks, and regenerate the ecstore test-count baseline. Production faults exercise the 24 added assertion sites and 7 semantic checks: 33 faults failed at the expected sites and passed after full restoration.

Validation (LANG=en_US.UTF-8, one build job):

cargo nextest run --locked -p rustfs-ecstore --lib -E 'test(delete_recovery) | test(quorum_delete_cleanup_coordinator_binds_single_and_batch) | test(crash_delete_retry_batch_reclaims_inline_and_reserved_external)' --test-threads 1 --no-fail-fast
Summary [  34.580s] 69 tests run: 69 passed, 6058 skipped; exit 0

cargo clippy --locked -p rustfs-ecstore --all-targets -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 5m 37s; exit 0

All 19 final Quick Checks completed with exit 0:

- shellcheck --version && actionlint
- cargo fmt --all --check
- ./scripts/check_unsafe_code_allowances.sh
- ./scripts/check_layer_dependencies.sh
- ./scripts/check_architecture_migration_rules.sh
- ./scripts/check_logging_guardrails.sh
- ./scripts/check_error_other_format_ratchet.sh
- ./scripts/check_no_tokio_io_uring.sh
- ./scripts/check_extension_schema_boundaries.sh
- ./scripts/check_body_cache_whitelist.sh
- ./scripts/check_s3s_footprint.sh
- ./scripts/check_test_count_ratchet.sh
- ./scripts/check_fips_wording.sh
- ./scripts/check_embedded_secrets.sh
- make script-tests
- python3 ./scripts/test_object_generation_protocol_model.py
- python3 ./scripts/check_test_wiring.py
- ./scripts/check_no_planning_docs.sh
- ./scripts/check_uring_lane_lib_only.sh

test-count ratchet OK: 52 crates, 22009 tests, 165 ignored match .config/test-count-baseline.txt
PASS: cold stampede script self-test
model: states=51023, transitions=167520, terminal=732, chosen_candidates=2
multi-slot round schedules: 640
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired

* fix(ecstore): retry tier commit intent lock contention

Retry transient namespace lock timeouts within the existing three-attempt
CAS budget. Reload and verify the original mutation identity after
contention, and preserve the final typed error when attempts are exhausted.
Cover save/read contention, exhausted retries, replacement identities and
CAS reconciliation without changing the existing timeout or lock policy.

Validation:
- cargo nextest run --locked -p rustfs-ecstore --features test-util --lib -E 'test(tier_mutation)' --test-threads 1 --no-fail-fast: 52 passed.
- cargo nextest run --locked -p rustfs-ecstore --features test-util --lib -E 'test(intent_advance_) | test(=services::tier::tier::tests::caller_cancellation_after_durable_prepare_does_not_hide_the_mutation)' --test-threads 1 --no-fail-fast: 5 passed.
- cargo clippy --locked -p rustfs-ecstore --all-targets -- -D warnings: passed.
- cargo clippy --locked -p rustfs-ecstore --all-targets --features test-util -- -D warnings: passed.
- 27 mutation pairs failed at the intended assertions and passed after restoring the production code.
- bash scripts/check_test_count_ratchet.sh --update, then bash scripts/check_test_count_ratchet.sh: 52 crates, 22013 tests, 165 ignored; ecstore increases from 6581 to 6585.

* refactor(lifecycle): replace s3s domain types with Gateway DTOs

Keep lifecycle rules, restore requests, Object Lock, and scanner cached tags in domain types. Convert at the existing HTTP boundary while preserving legacy XML, scanner cache bytes, date ordering, and tombstones.

Linux test validation before the final import-path-only adjustment:

cargo nextest run -p rustfs-ecstore bucket_config_goldens
39 passed; 6211 skipped by selector.

cargo nextest run --lib -p rustfs -E 'test(/test_zero_rule_lifecycle_tombstone_recognition/) | test(/test_local_lifecycle_staleness_axis_selection/) | test(/test_lifecycle_staleness_preserves_legacy_submillisecond_axis/) | test(/test_merge_incoming_lifecycle/) | test(/test_lifecycle_expiry_statement_matrix/) | test(/test_lifecycle_expiry_subset_xml/) | test(/put_bucket_lifecycle_validation_errors_keep_their_s3_code/) | test(/execute_put_bucket_lifecycle_configuration_rejects_missing_configuration/) | test(/execute_restore_object_rejects_missing_restore_request/) | test(/execute_restore_object_rejects_select_type/) | test(/execute_restore_object_maps_failures_to_typed_s3_errors/) | test(/execute_restore_object_returns_internal_error_when_store_uninitialized/) | test(/restore_request_conversion_preserves_legacy_output_values/) | test(/restore_request_conversion_preserves_presence_and_raw_enums/) | test(/restore_request_conversion_distinguishes_absent_and_empty_containers/)'
25 passed; 5318 skipped by selector.

cargo nextest run --profile ci -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-replication -p rustfs-scanner
7872 passed; 21 skipped under the existing CI profile.

Previously verified mutation controls: 39 production faults, 41 RED/GREEN pairs.

cargo clippy --locked -p rustfs -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-scanner --all-targets -- -D warnings
Passed in 40m 45s before the import-path adjustment, then in 1m 59s on
the final source with warnings denied. The final adjustment imports the
existing helper through the object domain facade; fmt and the architecture
check also passed.

Part of rustfs/backlog#2745. The remaining ecstore configuration families follow separately.

* refactor(admin): move error types to rustfs-s3-types (#8440)

* refactor(protocols): move error types to rustfs-s3-types by codemod

T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and
the error items of protocols move off s3s by scripts/codemods, run with
--hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it
downcasts the backend error, whose runtime type is still s3s::S3Error.

- rustfs/src/protocols/client.rs: the two webdav session methods now
  return rustfs_s3_types::S3Result and convert the s3s error with From.
- codemod: a child module that calls s3_error! through `use super::*`
  gets its own `use rustfs_s3_types::s3_error;` once its parent imports
  that macro by name; otherwise it is listed as GLOB.
- s3-types: a table test pins code, status and the s3s error the legacy
  edge renders for every variant a moved call site names.
- s3_error_lines 1585 -> 1584 (a lifecycle comment reworded).

* refactor(admin): move error types to rustfs-s3-types (WIP, no build)

T1.2 batch 4 (rustfs/backlog#2743), work in progress: rustfs check
still fails with 16 errors (listed in the PR body).

- codemod: --facade PATH treats a crate-local re-export of s3s's error
  items like s3s for the four names (9 new self-tests, 9 mutants red).
- codemod output for rustfs/src/admin and rustfs/src/site_replication
  (--facade crate::storage_api::site_replication::s3); 6 glob children
  get an explicit rustfs_s3_types::s3_error import.
- admin/router.rs: the Operation trait returns rustfs_s3_types::S3Result.
- error.rs: From<ApiError> builds the rustfs-s3-types error; the s3s
  conversion is derived from it.

* fix(admin): convert error types at the s3s boundary

Batch 4 of T1.2 left the admin and site_replication modules returning
rustfs_s3_types::S3Error while their callers and callees on the storage
and auth side still speak s3s::S3Error. Convert at each edge with the
existing From impls in rustfs-s3-types (compat_s3s.rs) instead of
changing any code, message or status:

- admin/auth.rs, admin/handlers/object_zip_download.rs,
  admin/handlers/table_catalog/mod.rs: map the s3s result returned by
  check_key_valid / authorize_request / authorize_internal_object_request
  into the admin error type with .map_err(Into::into).
- storage/ecfs.rs: the s3s-facing side maps the admin site_replication
  gate results the other way.
- admin/handlers/pools.rs: ApiError.code stays s3s::S3ErrorCode, so the
  rustfs code is converted with .into() when the struct is built, in
  production and in the two tests that build or compare it.
- admin/handlers/heal.rs, admin/runtime_sources.rs: the router returns
  s3s errors (S3Route impl); the expected rustfs code is converted with
  .into() before the comparison.
- admin/handlers/table_catalog/mod.rs: iceberg_rest_error takes
  &'static str so the Cow<'static, str> custom code borrows it; every
  caller already passes an ICEBERG_ERROR_* constant or a &'static str.
- admin/handlers/quota.rs: drop a now same-type .into() after s3_error!.
- storage_api.rs, site_replication/mod.rs: drop re-exports and an import
  that no longer have users (clippy -D warnings).

cargo check -p rustfs --all-targets and cargo clippy -p rustfs
--all-targets -- -D warnings are clean; the s3s footprint baseline is
unchanged at files=172, s3_error_lines=303, ecstore_files=28.

Refs rustfs/backlog#2743, rustfs/backlog#2734.

* refactor(ecstore): move bucket configs onto gateway persisted types (#8709)

* refactor(ecstore): move bucket configs onto gateway persisted types

Finish the ECStore batch of T1.4 (rustfs/backlog#2745). The remaining
eleven bucket configuration families (Website, CORS, Tagging, Versioning,
ObjectLock, Encryption, Notification, Logging, Accelerate, RequestPayment,
PublicAccessBlock) now read and write their stored XML through the
rustfs-gateway-types persistence and cors_tagging codecs instead of the
s3s codec, so the legacy s3s bucket-config codec and its
s3gate-metadata-xml compat marker are removed.

- ecstore: BucketConfigXml is implemented by the gateway codec for every
  family; metadata, metadata_sys, versioning, object lock, replication and
  lifecycle consumers take the persisted types. Region moves to
  rustfs_s3_types::Region, and the ETag conversion to s3s moves to the
  rustfs edge with its tests. Only warm_backend.rs and warm_backend_s3.rs
  still reference s3s, held for T1.5 (rustfs/backlog#2746).
- scanner: object lock cache, item actions and data usage use the
  persisted types.
- rustfs: the legacy S3 edge converts each family between s3s DTOs and the
  persisted types in storage/storage_api.rs by writing one codec's XML and
  reading it with the other; Tagging converts field by field.
- Bump the rustfs-gateway-types pin to 21283e87.

The 39 persisted-byte goldens are unchanged and pass, so a document the
old codec wrote reads back and re-encodes to the same bytes. s3s footprint:
files 181 -> 146, ecstore_files 28 -> 2, s3_error_lines unchanged.

* test(app): prepare the PutBucketEncryption config from the request

The uninitialized-store test passed an empty persisted configuration, which the use case now validates before it reaches the store check, so it answered MalformedXML instead of InternalError. Convert the request's configuration the way the S3 edge does.

* style(app): format the PutBucketEncryption test

* refactor(s3-client): move remote error codes and DTO enums off s3s (#8783)

Part of T1.5 (rustfs/backlog#2746), batch 1.

* refactor(s3select): run queries on gateway DTO request shapes (#8794)

* refactor(s3select): run queries on an engine request built from gateway DTO shapes

Part of T1.5 (rustfs/backlog#2746), batch 2.

* fix(s3select): keep the select request conversion in the app layer

The layer guard forbids app -> storage; the conversion moves from storage/s3_compat/select.rs to app/select_request.rs unchanged.

* fix(heal): retry transient metadata conflicts (#8812)

* fix(heal): retry transient metadata conflicts

* chore: update heal test count baseline

* test: reserve nextest capacity for progress probes (#8818)

* test(e2e): use a non-default region for the differential negative control

Since main's region-aware CreateBucket and GetBucketLocation (#8739, #8743), us-east-1 answers an empty LocationConstraint and 200 for an owned re-create, so the control no longer reported its own region. Also refresh the e2e-full linux selection from the CI listing (682 tests).

* refactor(protocols): move StorageBackend to gateway types (#8825)

* refactor(protocols): move storage backend to gateway types

* test: update e2e test count baseline

* fix: preserve copy source keys and unknown lengths

* test: update protocols test count baseline

* fix(protocols): preserve typed SFTP backend errors

* test(protocols): use gateway list input in policy tests

* fix(protocols): clear clippy failures after T1.5 merge (#8831)

fix(protocols): clear clippy failures after merge

* fix(ecstore): finalize restore metadata after local copy-back (#8836)

* fix(ecstore): finalize restore metadata after local copy-back

* test(ecstore): ratchet added restore finalization cases

* fix(ecstore): keep restore copyback compatible with older peers

* fix(ecstore): gate restore copyback on fleet compatibility

* fix(ecstore): retain transition identity during restore copyback

* fix(ecstore): route committed restore copyback locally

* fix restore copyback cleanup

* fix(ecstore): honor restore probe test fallback

* refactor(storage): migrate option errors to domain types (#8839)

* refactor(s3): migrate private multipart validation errors (#8844)

* refactor(auth): derive policy conditions from AuthzFacts (#8442)

* refactor(auth): extract authorization facts

* test(e2e): update smoke selection baseline

* test(e2e): update full selection baseline

* refactor(admin): migrate health and prefix usage to owned requests (#8887)

* refactor(admin): add owned health router boundary

* refactor(admin): migrate prefix usage to owned request

* refactor(admin): complete the owned admin request boundary (#8949)

* refactor(admin): complete the owned admin request boundary

Every type implementing the owned admin `Operation` now serves the legacy
router through one blanket impl that converts the s3s request and response
at that single edge, so handlers register with `AdminOperation(&Handler {})`
and need no adapter statics; `LegacyAdminAdapter` is removed.

`AdminRequest` gains the accessors handlers read today (`remote_addr`,
`client_info`, `request_context`, `req_info`, `req_info_mut`,
`install_req_info`, `envelope_mut`, `take_body`), the request helpers gain
envelope forms (`authenticate_envelope_credentials`,
`object_store_from_envelope`, the table-catalog `*_from_envelope` helpers,
`KmsAdminAudit::from_envelope`, `AccountAuditContext::from_envelope`,
`CallerIdentity::resolve_envelope`), `read_compatible_admin_body` accepts
either body type, and the owned `Body` gains `store_all_limited` and
`Default`. `admin::operation::test_support::TestRequest` builds owned
requests for handler tests. Unused accessors carry a per-method dead-code
expectation that each migrating handler batch removes.

No handler behavior changes; the remaining migrations follow in batches.

* chore(admin): add the T1.12 admin handler codemod

The script rewrites an admin handler file from the s3s request and response
types to the owned boundary (imports, signatures, request field and helper
calls, body access, response construction, and test request literals) and
leaves a TODO comment at every site it cannot rewrite mechanically. It is
idempotent and is removed once T1.12 completes.

* refactor(admin): migrate replication and KMS handlers to owned requests (#8951)

Move the T1.12 batch 2 admin handlers (site_replication, kms_keys, event,
replication, diagnostics, on_demand_migration, audit, account, kms_rekey,
kms_key_lifecycle, object_data_cache, tls_debug, integrity) from the s3s
request/response types to the owned AdminRequest/AdminResponse boundary.
Handler logic, authorization order, error codes and response bytes are
unchanged; the request/response plumbing now goes through the envelope
accessors and helpers introduced by the owned boundary.

The shared response helpers (utils::json_response, utils::empty_response,
handlers::admin_json_response, target_descriptor::build_json_response,
profile::authorize_profile_request) and the site replication repair
executor still answer with the s3s types, so the migrated files carry
owned-boundary forms of those helpers and convert the repair response at
its call edge. The streaming console-log body and the two devnull drains
move from the s3s byte stream to the owned Body stream/frame API.

Drop the dead_code expectations for the owned accessors and envelope
helpers this batch starts using, and lower the s3s footprint baseline to
the measured 99 files.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* refactor(admin): add owned forms of the shared admin response helpers

Handler batches were copying `json_response`, `empty_response`,
`admin_json_response`, and `build_json_response` into each migrated file
because the shared helpers still return the s3s response. Provide owned
forms next to each legacy helper (`owned_json_response`,
`owned_empty_response`, `owned_admin_json_response`,
`owned_build_json_response`, `authorize_profile_envelope`) with parity
tests against the legacy shape, derive `Debug` for `AdminResponse` so
tests can use `expect_err`, and fix three codemod rules: strip owned
string conversions in test credentials, annotate non-literal URI parses,
and leave a rebound request parameter immutable.

* refactor(admin): use the shared owned response helpers in batch 2 (#8954)

Replace the per-file owned-boundary copies of json_response,
empty_response, admin_json_response, build_json_response and
authorize_profile_request in the batch 2 admin handlers with the shared
owned_* helpers and authorize_profile_envelope that the boundary branch
now provides, and restore expect_err in the handler tests now that
AdminResponse implements Debug; the `.err().expect(..)` stand-ins were
failing the base branch's clippy step (clippy::err_expect under
-D warnings). The site replication repair call edge
keeps its conversion because the repair executor still answers with the
s3s response type.

Drop the dead_code expectations on the shared owned helpers this batch
now uses in non-test code.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* fix(admin): preserve legacy body limit error responses

* refactor(admin): move policy, ILM and batch 3 handlers to owned requests (#8956)

Batch 3 of T1.12 (rustfs/backlog#2788, parent rustfs/backlog#2734) moves
14 admin handler files to AdminRequest/AdminResponse and the owned
Operation trait: ilm_transition, policies, idp_compat, config_admin,
plugins_instances, profile_admin, group, heal, object_zip_download,
kms_key_metadata, durability, inspect_archive, sts and
gateway_key_inventory. Handler logic, authorization order, error codes
and response bytes are unchanged; test request literals use
test_support::TestRequest.

The handlers use the shared owned helpers (owned_json_response,
owned_admin_json_response, owned_build_json_response,
authorize_profile_envelope). The dead_code expectations on the
AdminRequest accessors and RequestEnvelope::uri_mut they now exercise
are removed. The legacy target_descriptor::build_json_response keeps
only its equivalence test caller and carries the matching expectation
until it retires.

object_zip_download builds the s3s request the S3 gate still reads from
the owned request facts; heal authenticates through
authenticate_envelope_credentials; the trace, inspect-archive and zip
bodies use Body::from_stream. extract_plugin_instance_filters takes the
URI so extensions.rs keeps calling it, and the two idp_compat
delegations stay on the legacy trait until their delegates migrate.

The s3s footprint baseline drops from 99 to 90 files.

* test(admin): anchor the replication diff body contract on the call

The source-contract test looked for `let body = read_compatible_admin_body(`
on one line; rustfmt wrapped that binding after the handler moved to the
owned request boundary, so the marker was no longer found. Anchor on the
call itself so formatting cannot change what the contract checks.

* refactor(admin): migrate table catalog and batch 1 to owned requests (#8955)

* fix(admin): preserve catalog body limit response bytes

* refactor(admin): migrate identity and cluster handlers to owned requests (#8966)

* refactor(admin): migrate identity and cluster handlers to owned requests

Move the T1.12 batch 4 admin handlers (user, service_account, oidc, tier,
system, pools, quota, kms_dynamic, kms_backup, rebalance, module_switch,
bucket_meta, cluster_snapshot, realtime, bucket_recovery, profile) from the
s3s request/response types to the owned AdminRequest/AdminResponse
boundary. All 90 impl Operation blocks now take AdminRequest and return
AdminResponse; handler logic, authorization order, error codes and response
bytes are unchanged. The handlers use the shared owned helpers
(owned_json_response, owned_admin_json_response, authorize_profile_envelope,
the envelope-form runtime sources and audit constructors) instead of local
copies.

profile.rs drops the s3s forms of authorize_profile_request and
profile_not_implemented_response now that profile_admin.rs is on the
envelope gate too; profile_not_implemented_response answers with an
AdminResponse. The realtime metrics stream moves from the s3s ByteStream
adapter to Body::from_stream, and the rebalance idle-404 document still
renders through s3s::S3Error::to_http_response with its body converted at
the edge.

The idp_compat delegators ImportIamV2 and AddServiceAccountLdap move to the
owned trait as well: with their targets on both traits, the method-call
delegation became ambiguous between the owned impl and the router blanket
impl, so they now take the owned request directly.

KmsAdminAudit::from_request loses its last production caller with this batch
and is deleted; its comparison test now asserts the envelope facts directly.

Lower the s3s footprint baseline to the measured 68 files.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* refactor(admin): remove the legacy admin helpers batch 4 superseded

Batch 4 moved the last production callers of the s3s-typed admin helpers
to their owned forms, which left `admin_json_response`, `json_response`,
`authenticate_request`, `authorize_admin_request`, `object_store_from_req`
and `object_store_from_extensions` unused, along with the `Body`,
`S3Request` and `S3Response` re-exports of the admin `s3` facade. Remove
them. The owned forms keep their documentation; the behavior tests of the
removed `json_response` now exercise `owned_json_response`, and the
compat-prefix test asserts the owned response directly.

Two tests existed only to compare a removed legacy helper with its owned
twin (`owned_json_response_matches_the_legacy_shape`) or to cover the
removed legacy gate (`authorize_admin_request_without_credentials_is_rejected`);
they go with the helpers, and the owned equivalents stay. The rustfs
test-count row drops from 5957 to 5955 and the s3s footprint from 68 to 66
files.

* refactor(admin): drop the legacy target descriptor JSON response helper (#8968)

`target_descriptor::build_json_response` returned the s3s response and
had no production caller left once the event, audit and plugin instance
handlers moved to `owned_build_json_response`; it survived only for the
legacy/owned equivalence test. Remove it, give the owned helper its own
documentation, and make the test assert the owned response directly:
status, `application/json`, the echoed `x-request-id` when present and
only then, and the body bytes. The file no longer imports s3s, so the
footprint baseline drops from 66 to 65 files.

* test(ecstore): pin native history on cancellation under the cancellable queue

main moved the local write path onto the disk blocking queue, which skips an
operation once its waiter is dropped, so a borrowed write cancelled while it is
queued no longer creates its file. The integration-only native publication
test still expected the queued open to run. Assert the new execution model
instead (the open never runs) while keeping its safety claims: the Started
record is taken before queueing and the cancelled publication never regrants
native history.

* chore(gateway): raise test-count rows for tests merged from main

* test(s3select): use the owned JSON input types in merged planner tests

* fix(ecstore): scope the merged direct-writer test hook to test builds

* fix(ecstore): create parents before admitted truncate opens

Align the heal tests with removal-disabled preservation and local-only outcome
fields, and refresh the e2e-full selection for the synced tests.

* chore(gateway): update test-count baseline after main sync

Main #9241 removed the node pool-metadata readiness cache with 8 tests
and added 6, so the rustfs row drops from 6440 to 6438. The ecstore and
s3select rows rise for tests added by #9237, #9239 and #9241.

* feat(gateway-adapter): add IAM-backed CredentialProvider

Add the ring-2 crate rustfs-gateway-adapter with IamCredentials, the
gateway CredentialProvider answered by RustFS IAM (rustfs/backlog#2760,
credentials half; PolicySource is deferred).

- Pin rustfs-gateway at the existing gateway rev with
  default-features = false so the server crate stays out of the build.
- IamKeyLookup is a narrow IAM trait implemented for IamSys so the
  adapter is testable without an in-memory IAM backend.
- Root credentials are injected at construction and resolve before IAM
  is ready.
- Mapping: keystone: prefix -> NotFound (deliberate change from the
  legacy empty secret); IAM not ready or not initialized -> Unavailable;
  unknown key or NoSuch{User,ServiceAccount,TempAccount,Account} ->
  NotFound; other IAM errors -> Backend; invalid or "off" principals
  resolve disabled; STS credentials carry a SessionBinding, and an STS
  credential without an expiry is a Backend fault rather than a
  never-ending session.
- No authorization in lookup; no Debug on secret-holding types; IAM
  error text is dropped after classification.

* feat(scripts): add junit_diff.py for dual-stack JUnit comparison (#9275)

* feat(gateway-adapter): add IAM-backed PolicySource

Add IamPolicySource, the gateway PolicySource answered by RustFS IAM
(rustfs/backlog#2760, policy half). One snapshot per request carries a
RustfsPrincipal: principal kind, account, effective username, parent
user, credential groups, live group memberships, token claims and
is_owner.

- Anonymous callers get an empty snapshot without asking IAM.
- IAM not ready, not initialized or failing is PolicyError::unavailable,
  never an empty snapshot.
- Unknown, disabled and session-kind-mismatched principals are refused
  with constant labels; unreadable tokens are malformed.
- Claims are read from the stored token (root secret for STS, the
  account's own secret for service accounts), and is_owner mirrors the
  legacy has_root_access rule.
- RustfsPrincipal holds no secret or session token and has no Debug.

* ci(gateway): make the test-count ratchet a floor, not exact (#9278)

ci(gateway): make the test-count ratchet a floor instead of an exact baseline

Tests below a row still fail; tests above it, a member without a row and ignores below the row now pass with a note. Ignores above a row still fail. Adds contract tests wired into make script-tests.

* feat(gateway-adapter): add host, bucket-owner and request-id adapters

* fix(gateway-adapter): avoid ecstore path in doc comment

* fix(gateway-adapter): keep the s3s path pattern out of docs

* chore(gateway): record ignored microbenchmarks merged from main (#9290)

chore(gateway): record the ignored microbenchmarks merged from main

main added three throughput microbenchmarks that are ignored by design
(#9217): one each in rustfs-checksums, rustfs-rio and rustfs-utils. The
test-count ratchet treats a higher ignored count as a failure until the
row is raised, so raise those three rows to 1 and tighten the test rows
to the merged tree with --update.

* test(gateway-adapter): make the test carrier Clone for Extensions::insert

* fix(gateway-adapter): share live root and refuse short secrets

Address the security review of the T2.1 adapters (rustfs/backlog#2760).

- Add RootCredentials (validated: root secret and STS signing key at
  least MIN_SECRET_KEY_LEN bytes), the RootSource trait and the settable
  RootCell. IamAdapters::new is now the only constructor of
  IamCredentials and IamPolicySource, so both read one live root source
  per request; no root source answers unavailable.
- A looked-up identity whose secret is shorter than 8 bytes is NotFound
  in the credential provider and DISABLED_PRINCIPAL in the policy
  source, whatever the lookup reports as valid.
- STS claims are read with the STS signing key from the root source
  instead of assuming it equals the root secret.
- Cover the off-status-alone branch, the OIDC owner exclusion, service
  accounts of root, the name live groups are read under, and another
  session's token.

* ci(s3-tests): add reference parity lane against pinned 1.0.5 (#9288)

* fix(gateway-adapter): effective secret length and unsplittable pair

Address the confirmation review of the T2.1 adapters
(rustfs/backlog#2760).

- One rule for every secret: its length after trailing NUL bytes are
  removed must be at least MIN_SECRET_KEY_LEN. HMAC zero-pads its key,
  so trailing NULs add nothing; interior NULs count. Applied to the root
  secret, the STS signing key and every looked-up credential.
- IamAdapters keeps both halves private and hands them to a builder
  together through the consuming install(builder, make_authenticator);
  compile-fail doctests show a mixed pair cannot be assembled.
- Any Fn() -> Option<Arc<RootCredentials>> is a RootSource, so the
  binary reads its live root accessor directly; RootCell is now a
  test-only helper.

* test(gateway-adapter): use bucket names the rules accept in host tests

* refactor(gateway-adapter): never refuse on request id; owner is the reported constant

* docs(gateway-adapter): state what install guarantees about the provider handle

---------

Co-authored-by: rustfs-cla-assistant[bot] <273121464+rustfs-cla-assistant[bot]@users.noreply.github.com>
2026-11-11 06:16:20 -05:00
hector 75292c6224 ci: add manual packaging of the origin/main tip (#9291)
Dispatch-only workflow that compiles the current main commit with a
caller-supplied version and uploads zip, deb, and rpm artifacts.

Co-authored-by: hector <majinghe@users.noreply.github.com>
2026-11-11 04:55:02 +00:00
Chris An 16e3071f84 feat(gateway-adapter): IAM CredentialProvider and PolicySource (#9284)
* feat(server): add the RUSTFS_S3_STACK selector and dual-stack e2e lane (#8411)

* refactor(server): group the s3s-compat tower layers under legacy_compat (#8410)

Move the ten tower layers that patch s3s behaviour out of
rustfs/src/server/layer.rs into rustfs/src/server/legacy_compat/, one file
per layer with its tests. Each file header names the s3s behaviour it
patches, the gateway mechanism that replaces it (Replaced by:), and the task
that removes it (Removed by: T2.12, rustfs/backlog#2771). The layers keep
their code, their order in external_service_stack!, and their behaviour; no
layer is deleted.

Add server::http::tests::stack_census, which renders the ordered service
chain of the real external_service_stack! expansion and compares it with a
committed constant, so a reorder, insertion or removal is a visible diff.

Route the two s3s error types the moved layers use through a
storage_api::server::legacy_compat shim so the split adds no s3s-importing
file, and lower the s3s footprint baselines to the measured values (files 209
to 208, s3_error! lines 1586 to 1585). Keep the server runtime-source bypass
guard covering the moved code.

Refs rustfs/backlog#2787, rustfs/backlog#2734

* feat(s3-types): add the RustFS-owned S3 error type and s3_error! macro (#8406)

`rustfs-s3-types` now provides `S3ErrorCode` (generated by build.rs from
src/codes.txt, the grep-derived call surface, plus `Custom`), `S3Error`,
`S3Result`, `StdError` and the `s3_error` macro with the same call syntax
as s3s, so the T1.2 codemod is a `use` swap. Statuses are hand-written and
pinned to the s3s behaviour RustFS ships today by an oracle test that uses
s3s as a dev-dependency only. The `From` conversions for the legacy edge
live behind the off-by-default `compat-s3s` feature (the orphan rule
forbids them in rustfs/src/error.rs); `rustfs` enables it.

Refs rustfs/backlog#2742, rustfs/backlog#2734

* ci: add the per-crate test-count ratchet (#8404)

Quick Checks now fails any PR that lowers the number of test attributes
in a workspace crate or adds #[ignore], measured per crate against the
exact rows of .config/test-count-baseline.txt (rustfs/backlog#2734
section 1 rule 3). The counter widens the spec's literal
#[(tokio::)?test] to the tokio attributes that carry arguments and both
spellings of #[test_case(...)], and anchors at line start so
commented-out attributes do not count. The initial baseline covers 52
crates, 21077 tests and 164 ignored.

Refs rustfs/backlog#2738
Refs rustfs/backlog#2734

* test(ecstore): pin the persisted bucket-config XML bytes for 13 families (#8408)

Before any DTO or codec is replaced, pin the exact bytes that
deserialize followed by serialize produces for every persisted
bucket-configuration XML family in its three source shapes (AWS
documentation, MinIO-written, RustFS-written), and the bytes RustFS
writes today for a typed value. Byte equality only: a structural
comparison would let a codec replacement change the persisted encoding
unnoticed.

The generator script refuses to overwrite existing goldens without
--force, refuses on dirty codec paths, cannot report success from its
write mode, and records the generating commit in the fixtures README.

Refs rustfs/backlog#2744, rustfs/backlog#2734

* ci: raise the test-count baseline to the integration tree (#8426)

* ci: make the s3s footprint baseline exact and lower-only (#8403)

* ci: make the s3s footprint baseline exact and lower-only

The s3s footprint guard compared three counters against ceilings held in
the script itself, so a ceiling could sit above the tree and a drift was
fixable by raising it (rustfs/rustfs#5964 raised the s3_error! ceiling
to go green). The three values now live in
.config/s3s-footprint-baseline.txt as files, s3_error_lines and
ecstore_files, the script reads only that file, and a measurement that
differs from its baseline in either direction fails: growth prints
"s3s footprint grew: <measured> > <baseline>", shrinkage prints
"lower the baseline to <measured> in this PR". The counting method is
unchanged; the baseline is the current measurement of
gateway/integration (209 / 1585 / 36). The header records the end state:
once all three reach 0, T4.2 switches the script to failing on any
occurrence.

Refs rustfs/backlog#2739, rustfs/backlog#2734

* ci: add the s3s edge allowlist mode to the footprint guard (#8405)

`scripts/check_s3s_footprint.sh --mode allowlist` fails on every Rust
file that references s3s (`s3s::` or `s3s_sigv4`) and is not matched by
an entry of the new `.config/s3s-edge-allowlist.txt`; `--dry-run` lists
those files and exits 0. The allowlist carries the initial edge from the
spec, and the script rejects an entry that names or matches anything
under crates/ other than crates/e2e_test. The baseline mode and the
argument-less CI invocation are unchanged.

Refs rustfs/backlog#2741, rustfs/backlog#2734

* test(ecstore): reach the bucket-config codec through the test facade (#8428)

* refactor(s3-types): own the HTTP Body and drop s3s from three crates (#8431)

Add rustfs_s3_types::Body, a BoxBody<Bytes, StdError> wrapper with
empty() and From<Bytes | Vec<u8> | String | &'static str>, and the
legacy From conversions to and from s3s::Body behind compat-s3s.

The signer is now generic over the request body: it only reads and
writes the request head, so it needs no body type and its callers keep
theirs. s3-client sends its requests with the owned Body, madmin signs a
bodiless head, and targets drops its two S3Request/S3Response header
wrappers in favour of extract_params_header.

rustfs-signer, rustfs-madmin and rustfs-targets no longer depend on
s3s; the s3s footprint baseline drops from 209 to 202 files.

Refs rustfs/backlog#2747, rustfs/backlog#2734

* refactor(ecstore): move S3ErrorCode to rustfs-s3-types by codemod (#8432)

* test(ecstore): pin the InvalidRange responses of new_getobjectreader

Both error branches of new_getobjectreader build an ErrorResponse with S3ErrorCode::InvalidRange, and no test observed either code. Pin the code, its wire name and the location fields of each branch before the code type moves to rustfs-s3-types (rustfs/backlog#2743), so a changed code on either branch fails a test. The ecstore test-count row rises by the two tests.

* chore(scripts): add the s3s-to-rustfs-s3-types error codemod

Task T1.2 of rustfs/backlog#2734 (rustfs/backlog#2743) moves S3Error, S3ErrorCode, S3Result and s3_error! from s3s to rustfs-s3-types by script, one crate closure per PR. The codemod rewrites use trees (splitting mixed s3s imports) and qualified code paths, never comments or literals, adds the rustfs-s3-types dependency, drops s3s only from a crate that no longer names it, refuses every file of the s3s edge allowlist, takes --hold for a file whose type is fixed outside the closure, and prints a per-crate census. A second run changes nothing. Its self-test runs under make script-tests.

* refactor(ecstore): take S3ErrorCode from rustfs-s3-types

Output of scripts/codemods/s3_error_to_s3_types.py over crates/ecstore, crates/lifecycle, crates/scanner and crates/replication (rustfs/backlog#2743 batches 1 and 2). The ErrorResponse code field, the two InvalidRange constructions in new_getobjectreader and an unused tier import now use rustfs_s3_types::S3ErrorCode; the variant names and wire names are unchanged. lifecycle, scanner and replication reference none of the four error names. warm_backend.rs and warm_backend_s3.rs are held: they compare against rustfs_s3_client::ErrorResponse.code, which stays the s3s type until s3-client moves.

* ci(footprint): count only s3_error! lines still on s3s's macro

s3_error_lines counted every line holding the text `s3_error!`, which
also matches the RustFS-owned `rustfs_s3_types::s3_error!`. A codemod
that moves call sites onto rustfs-s3-types therefore could not lower it
(rustfs/backlog#2743, decision 3; rustfs/rustfs#8432, gotcha 2).

The counter now applies a textual, per-file and per-line rule that errs
on the side of counting (full rule in the script header):
- occurrences spelled `rustfs_s3_types::s3_error!` never count;
- a line holding `s3s::s3_error!` always counts;
- any other `s3_error!` line counts unless its file is cleared, that is,
  it imports the macro from rustfs_s3_types by name on a line starting
  with the use item (or defines it with `macro_rules! s3_error`) and does
  not also bring s3s's macro in (`use s3s::s3_error`, a `use s3s::{..}`
  tree naming it or a crate-root `*`, `use s3s::*`, or
  `#[macro_use] extern crate s3s`);
- fail closed: a file matching neither form (`use super::*`, a
  crate-local re-export, a renamed or glob rustfs_s3_types import, a
  commented-out import, a comment-only mention) counts.

Census of the 1585 lines on 96fc06776: 1478 in 64 files that import the
macro from s3s; 107 in 15 files matching neither form (105 through
`use super::*`, 1 qualified `s3s::s3_error!`, 1 comment line); none in
files that import it from rustfs_s3_types or define it. Every line stays
counted, so the baseline remains s3_error_lines=1585 (files=202 and
ecstore_files=36 unchanged). Rewriting rustfs/src/auth.rs's import to
rustfs_s3_types lowers the new counter to 1567; the old one stayed 1585.

`--self-test` (wired into `make script-tests`) copies the script into
22 fixture trees: 6 positive cases and 16 negative ones (lines that must
not count, forms that must not clear a file, and a missing baseline,
missing key or failing classification search that must fail). Each tree
has a control line that must count, and each run gets s3s text on stdin.
Seven cases were red against the old counter. 19 mutants each turn their
case red: old definition, flipped import test, dropped fail-closed
branch, dropped e2e_test exclusion, dropped s3s veto, dropped qualified
s3s rule, dropped rustfs-qualified strip, unanchored rustfs import,
renamed import accepted, nested glob taken as root glob, single-line
s3s trees only, dropped s3s glob forms, dropped macro_use form, dropped
definition clearing, ignored classification rg errors, error-line search
without its path, classification searches without their path, missing
baseline skipped, missing key skipped.

* test(e2e): compare two RUSTFS_S3_STACK servers over every S3 operation (#8435)

Add dual_stack_differential to e2e_test. It starts two rustfs processes
from the same binary, stack A on RUSTFS_S3_STACK=legacy and stack B on
RUSTFS_E2E_STACK_B (default legacy until rustfs/backlog#2734 T2.5),
sends both the same aws-sdk-s3 script covering the 73 methods of
impl S3 for FS (a normal request and a 4xx negative each), and compares
status, error code, an allowlist of response headers and a few typed
body fields. Error message text is never compared. Each run writes
<target>/dual-stack/<run>.json with compared_ops and the (op, field,
a, b) differences; any difference fails the test.

differential_detects_an_injected_difference starts B with
RUSTFS_REGION=eu-west-1 and requires GetBucketLocation to be reported.
The comparator is pure and unit-tested without a server.

Wire the module into the e2e-smoke filter and update the e2e-smoke and
e2e-full selection digests and the e2e_test test-count row.

Refs rustfs/backlog#2740

* feat(app): add the RustFS-owned RequestEnvelope and its s3s constructor (#8437)

* feat(app): add the RustFS-owned RequestEnvelope and its s3s constructor

* ci: classify the s3s edge adapters in storage/s3_compat as interface

* feat(s3-types): add Body::from_stream and move Swift off s3s::Body (#8438)

rustfs/backlog#2747 batch 3. `Body::from_stream` wraps a chunk stream of
unknown length over `StreamBody`/`BoxBody`: one data frame per chunk, the
first error ends the body, the size hint stays unknown, and dropping the
body drops the stream. Parity tests pin it to the s3s `StreamingBlob::wrap`
it replaces, at the `http_body` level and byte for byte through hyper.

The Swift handlers in rustfs-protocols build their responses as the owned
`Body`. `SwiftService` converts them into the wrapped S3 service's response
body type through `From<Body>`, so rustfs keeps serving `s3s::Body` at the
legacy edge unchanged. Request bodies stay Send-only (`UnsyncBoxBody`); the
two storage writers and the DLO/SLO segment streams get `Sync` from
`sync_wrapper::SyncStream`, which is what s3s's `http_body_unsync` mutex
provided before. The DTO and error uses of s3s in protocols are untouched
(T1.5 and T1.2).

* refactor(protocols): move error types to rustfs-s3-types by codemod (#8439)

T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and
the error items of protocols move off s3s by scripts/codemods, run with
--hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it
downcasts the backend error, whose runtime type is still s3s::S3Error.

- rustfs/src/protocols/client.rs: the two webdav session methods now
  return rustfs_s3_types::S3Result and convert the s3s error with From.
- codemod: a child module that calls s3_error! through `use super::*`
  gets its own `use rustfs_s3_types::s3_error;` once its parent imports
  that macro by name; otherwise it is listed as GLOB.
- s3-types: a table test pins code, status and the s3s error the legacy
  edge renders for every variant a moved call site names.
- s3_error_lines 1585 -> 1584 (a lifecycle comment reworded).

* test(connect): restore one-second drive request budgets

Apply the exact test-only repair already landed on main in 5972a09cc3. The filesystem probe and production CLI success test request one second and keep their existing one-second duration and parameter checks. All outcome, I/O, error, cleanup and resource-limit assertions remain.

Scope: rustfs/tests/connect_perf_drive.rs only. Patch-id: cb0a1dfc8f68a67727525adea7458e2cee7d09f8.

Validation environment: LANG=en_US.UTF-8, CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0, exclusive APFS-cloned CARGO_TARGET_DIR.

Quick Checks: every command from .github/actions/quick-checks/action.yml, with captured output and exit status below.

$ shellcheck --version && actionlint
ShellCheck - shell script analysis tool
version: 0.11.0
license: GNU General Public License, version 3
website: https://www.shellcheck.net
exit_code=0

$ cargo fmt --all --check
exit_code=0

$ ./scripts/check_unsafe_code_allowances.sh
exit_code=0

$ ./scripts/check_layer_dependencies.sh
Layer dependency guard passed (no new reverse dependencies/cycles).
exit_code=0

$ ./scripts/check_architecture_migration_rules.sh
Architecture migration rules passed.
exit_code=0

$ ./scripts/check_logging_guardrails.sh
✅ Logging guardrails check passed
exit_code=0

$ ./scripts/check_error_other_format_ratchet.sh
error(format!) ratchet passed (total call sites: 582).
exit_code=0

$ ./scripts/check_no_tokio_io_uring.sh
exit_code=0

$ ./scripts/check_extension_schema_boundaries.sh
Extension schema boundaries passed.
exit_code=0

$ ./scripts/check_body_cache_whitelist.sh
Body-cache whitelist guard passed.
exit_code=0

$ ./scripts/check_s3s_footprint.sh
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)
✅ s3s footprint baseline check passed
exit_code=0

$ ./scripts/check_test_count_ratchet.sh
test-count ratchet OK: 52 crates, 21222 tests, 164 ignored match .config/test-count-baseline.txt
exit_code=0

$ ./scripts/check_fips_wording.sh
FIPS wording guard passed (README.md and CHANGELOG.md contain no forbidden claims).
KMS wording guard passed (crates/kms claims no Vault KV2 Transit wrapping).
exit_code=0

$ ./scripts/check_embedded_secrets.sh
Embedded secret guard passed (no private key material or provider credential literal in the tree).
exit_code=0

$ make script-tests
Running script tests...
entrypoint credential policy tests passed
internode grpc A/B bench env tests passed
object batch benchmark enhanced tests passed
PASS package service script contract
hotpath warp A/B gate tests passed
hotpath warp ABBA tests passed
Scanner validation artifacts written to /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.1pFe6Kl1Yu/out
Scanner validation artifacts written to /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.1pFe6Kl1Yu/out-missing-pid
.............................
----------------------------------------------------------------------
Ran 29 tests in 181.286s

OK
{"decision":"verified","pending_gates":["G01","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G02"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-E"}
{"decision":"verified","pending_gates":["G01","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G02"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-E"}
{"decision":"verified","pending_gates":["G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G01"}
{"decision":"verified","pending_gates":["G01","G02","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G03"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-L"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G14"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G14"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G10"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P1"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P2"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P3"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G05"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G06"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-D"}
Status-and-outcome release descriptor verified: /private/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp8rwn1pe0/descriptor/release-bundle-status-outcome.json
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G05"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G06"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-D"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G07"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G08"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P4"}
exact 1MiB handoff ABBA tests passed
pinned paired ABBA bench tests passed
==> /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/bin/path_containment (-max_total_time=7, -seed=123456789, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/explicit-artifacts/path_containment/, corpus=/private/tmp/gw-orchestrator-2734/ci-drive/fuzz/corpus/path_containment)
-seed=123456789
target=path_containment
seed=123456789
max_total_time=7
git_revision=4d15b816236630b2c42f321ffc9ed3654edc94e3
git_dirty=true
runner_mode=prebuilt
==> /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/bin/path_containment (-max_total_time=1, -seed=2322617706, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/auto-artifacts/path_containment/, corpus=/private/tmp/gw-orchestrator-2734/ci-drive/fuzz/corpus/path_containment)
-seed=2322617706
==> cargo +nightly fuzz run path_containment (-max_total_time=9, -seed=987654321, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/cargo-artifacts/path_containment/)
+nightly
fuzz
run
path_containment
-seed=987654321
seed=987654321
runner_mode=cargo-fuzz
fuzz runner seed manifest tests passed
✅ scripts/python_bin.sh resolver checks passed
self-test ok: clean fixture with every exemption present passes
self-test ok: all 11 pattern families fire
self-test ok: moving a pinned source is reported
Embedded secret guard self-test passed.
self-test ok: p1-s3s-import: 'use s3s::s3_error;' counts every line, the rustfs_s3_types module's too (s3_error_lines=3)
self-test ok: p2-s3s-mixed-tree: a multi-line s3s use tree naming s3_error counts (s3_error_lines=3)
self-test ok: p3-s3s-glob: a crate-root glob of s3s, bare or inside a tree, counts (s3_error_lines=5)
self-test ok: p4-macro-use: '#[macro_use] extern crate s3s' counts (s3_error_lines=3)
self-test ok: p5-qualified-s3s: a qualified s3s::s3_error! line counts in a cleared file (s3_error_lines=2)
self-test ok: p6-exact-pass: a baseline equal to the tree passes (exit 0)
self-test ok: n1-rustfs-import: 'use rustfs_s3_types::s3_error;' alone counts nothing (s3_error_lines=1)
self-test ok: n2-rustfs-mixed-tree: a rustfs_s3_types tree plus an s3s tree without the macro counts nothing (s3_error_lines=1)
self-test ok: n3-defines: the file defining the macro counts nothing (s3_error_lines=1)
self-test ok: n4-qualified-rustfs: a qualified rustfs_s3_types::s3_error! line counts nothing (s3_error_lines=1)
self-test ok: n5-renamed-call: a call through a renamed macro is not text-matched (out of scope) (s3_error_lines=1)
self-test ok: n6-comment-string: a comment or string mentioning s3_error! in a cleared file counts nothing (s3_error_lines=1)
self-test ok: n7-e2e-excluded: crates/e2e_test is excluded (s3_error_lines=1)
self-test ok: n8-super-glob: an unrecognised import form ('use super::*;') counts (s3_error_lines=3)
self-test ok: n9-facade: a crate-local re-export counts (s3_error_lines=2)
self-test ok: n10-renamed-import: a renamed rustfs_s3_types import does not clear s3_error! lines (s3_error_lines=2)
self-test ok: n11-commented-import: a commented-out rustfs_s3_types import does not clear (s3_error_lines=2)
self-test ok: n12-rustfs-glob: a glob import of rustfs_s3_types does not clear (s3_error_lines=2)
self-test ok: n13-comment-only: a comment mentioning s3_error! in a file with no import counts (s3_error_lines=2)
self-test ok: f1-baseline-missing: a missing baseline file fails (exit 1)
self-test ok: f2-key-missing: a baseline without s3_error_lines fails (exit 1)
self-test ok: f3-rg-fails: a failing classification search fails the guard (exit 1)
✅ s3s footprint self-test passed
test_core_gate_rejects_missing_ignored_filtered_and_corrupt_inputs (__main__.SelfTests.test_core_gate_rejects_missing_ignored_filtered_and_corrupt_inputs) ... ok
test_e2e_requires_registration (__main__.SelfTests.test_e2e_requires_registration) ... ok
test_fuzz_runtime_uses_matrix_target (__main__.SelfTests.test_fuzz_runtime_uses_matrix_target) ... ok
test_ilm_lane_keeps_the_measured_cargo_build_budget (__main__.SelfTests.test_ilm_lane_keeps_the_measured_cargo_build_budget) ... ok
test_profile_listing_binds_platform_digest (__main__.SelfTests.test_profile_listing_binds_platform_digest) ... ok
test_profile_listing_enforces_selection (__main__.SelfTests.test_profile_listing_enforces_selection) ... ok
test_quick_checks_commands_propagate_failure (__main__.SelfTests.test_quick_checks_commands_propagate_failure) ... ok
test_quick_checks_rejects_caller_and_execution_bypasses (__main__.SelfTests.test_quick_checks_rejects_caller_and_execution_bypasses) ... ok
test_runner_readiness_fails_closed (__main__.SelfTests.test_runner_readiness_fails_closed) ... ok
test_s3_runner_rejects_unbounded_failure_locals (__main__.SelfTests.test_s3_runner_rejects_unbounded_failure_locals) ... ok
test_scanner_heal_begin_requires_embedded_source_provenance (__main__.SelfTests.test_scanner_heal_begin_requires_embedded_source_provenance) ... ok
test_scanner_heal_case_does_not_approve_pending_release (__main__.SelfTests.test_scanner_heal_case_does_not_approve_pending_release) ... ok
test_scanner_heal_case_lane_is_required_for_runner_profile (__main__.SelfTests.test_scanner_heal_case_lane_is_required_for_runner_profile) ... ok
test_scanner_heal_case_only_schema_cannot_approve_release (__main__.SelfTests.test_scanner_heal_case_only_schema_cannot_approve_release) ... ok
test_scanner_heal_crash_case_rejects_restart_oracle (__main__.SelfTests.test_scanner_heal_crash_case_rejects_restart_oracle) ... ok
test_scanner_heal_ec84_case_rejects_wrong_erasure_geometry (__main__.SelfTests.test_scanner_heal_ec84_case_rejects_wrong_erasure_geometry) ... ok
test_scanner_heal_finish_collects_oracles_from_registry (__main__.SelfTests.test_scanner_heal_finish_collects_oracles_from_registry) ... ok
test_scanner_heal_multipool_case_accepts_deferred_optional_outage_write (__main__.SelfTests.test_scanner_heal_multipool_case_accepts_deferred_optional_outage_write) ... ok
test_scanner_heal_pending_gate_cannot_map_to_implemented_lane (__main__.SelfTests.test_scanner_heal_pending_gate_cannot_map_to_implemented_lane) ... ok
test_scanner_heal_receipts_reject_reuse_and_missing_builds (__main__.SelfTests.test_scanner_heal_receipts_reject_reuse_and_missing_builds) ... ok
test_scanner_heal_rejects_boolean_fractional_and_out_of_geometry_integers (__main__.SelfTests.test_scanner_heal_rejects_boolean_fractional_and_out_of_geometry_integers) ... ok
test_scanner_heal_rejects_broken_execution_and_artifacts (__main__.SelfTests.test_scanner_heal_rejects_broken_execution_and_artifacts) ... ok
test_scanner_heal_rejects_copied_junit_and_wrong_suite_build (__main__.SelfTests.test_scanner_heal_rejects_copied_junit_and_wrong_suite_build) ... ok
test_scanner_heal_release_bundle_accepts_complete_measured_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_accepts_complete_measured_evidence) ... ok
test_scanner_heal_release_bundle_assembler_copies_measured_artifacts (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_copies_measured_artifacts) ... ok
test_scanner_heal_release_bundle_assembler_merges_measured_descriptors (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_merges_measured_descriptors) ... ok
test_scanner_heal_release_bundle_assembler_rejects_duplicate_descriptor_gate (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_duplicate_descriptor_gate) ... ok
test_scanner_heal_release_bundle_assembler_rejects_fixture_descriptor (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_fixture_descriptor) ... ok
test_scanner_heal_release_bundle_assembler_rejects_hash_mismatch (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_hash_mismatch) ... ok
test_scanner_heal_release_bundle_assembler_rejects_mixed_descriptor_revisions (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_mixed_descriptor_revisions) ... ok
test_scanner_heal_release_bundle_enforces_topology_duration_profile_and_versions (__main__.SelfTests.test_scanner_heal_release_bundle_enforces_topology_duration_profile_and_versions) ... ok
test_scanner_heal_release_bundle_fixture_writer_is_not_release_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_fixture_writer_is_not_release_evidence) ... ok
test_scanner_heal_release_bundle_gate_accepts_single_measured_gate (__main__.SelfTests.test_scanner_heal_release_bundle_gate_accepts_single_measured_gate) ... ok
test_scanner_heal_release_bundle_gate_rejects_missing_field (__main__.SelfTests.test_scanner_heal_release_bundle_gate_rejects_missing_field) ... ok
test_scanner_heal_release_bundle_gate_rejects_unknown_gate (__main__.SelfTests.test_scanner_heal_release_bundle_gate_rejects_unknown_gate) ... ok
test_scanner_heal_release_bundle_rejects_synthetic_or_missing_fields (__main__.SelfTests.test_scanner_heal_release_bundle_rejects_synthetic_or_missing_fields) ... ok
test_scanner_heal_release_bundle_requires_domain_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_requires_domain_evidence) ... ok
test_scanner_heal_release_bundle_requires_field_provenance (__main__.SelfTests.test_scanner_heal_release_bundle_requires_field_provenance) ... ok
test_scanner_heal_release_bundle_requires_json_artifact_provenance (__main__.SelfTests.test_scanner_heal_release_bundle_requires_json_artifact_provenance) ... ok
test_scanner_heal_release_bundle_requires_mixed_version_field_roles (__main__.SelfTests.test_scanner_heal_release_bundle_requires_mixed_version_field_roles) ... ok
test_scanner_heal_release_bundle_requires_same_gate_measurement_window (__main__.SelfTests.test_scanner_heal_release_bundle_requires_same_gate_measurement_window) ... ok
test_scanner_heal_release_consumes_multiple_registry_oracles (__main__.SelfTests.test_scanner_heal_release_consumes_multiple_registry_oracles) ... ok
test_scanner_heal_release_matrix_lanes_remain_pending_without_real_evidence (__main__.SelfTests.test_scanner_heal_release_matrix_lanes_remain_pending_without_real_evidence) ... ok
test_scanner_heal_release_requirements_cannot_be_cleared (__main__.SelfTests.test_scanner_heal_release_requirements_cannot_be_cleared) ... ok
test_scanner_heal_release_status_rejects_focused_case_run (__main__.SelfTests.test_scanner_heal_release_status_rejects_focused_case_run) ... ok
test_scanner_heal_release_status_rejects_synthetic_case (__main__.SelfTests.test_scanner_heal_release_status_rejects_synthetic_case) ... ok
test_scanner_heal_required_evidence_fields_cannot_be_removed (__main__.SelfTests.test_scanner_heal_required_evidence_fields_cannot_be_removed) ... ok
test_scheduled_alerts_require_completion_watchdog (__main__.SelfTests.test_scheduled_alerts_require_completion_watchdog) ... ok
test_update_profile_selection_changes_only_requested_platform (__main__.SelfTests.test_update_profile_selection_changes_only_requested_platform) ... ok
test_validate_includes_quick_checks (__main__.SelfTests.test_validate_includes_quick_checks) ... ok
test_vault_tests_require_cross_process_serialization (__main__.SelfTests.test_vault_tests_require_cross_process_serialization) ... ok
test_workflow_readiness_requires_dependency_probe (__main__.SelfTests.test_workflow_readiness_requires_dependency_probe) ... ok

----------------------------------------------------------------------
Ran 52 tests in 14.301s

OK
e2e-full selection OK: 1 tests, sha256=29e30e6ed2ed38d9e5b0a7cdd9536abf5c719f7021a02b578ae1381563a3c273
...................................
----------------------------------------------------------------------
Ran 35 tests in 1.161s

OK
...................
----------------------------------------------------------------------
Ran 19 tests in 17.397s

OK
..........
----------------------------------------------------------------------
Ran 10 tests in 0.183s

OK
test_actual_selector_bootstrap_uses_base_policy_and_fails_closed (__main__.SelfTests.test_actual_selector_bootstrap_uses_base_policy_and_fails_closed) ... ok
test_documentation_paths_do_not_hide_build_or_fixture_changes (__main__.SelfTests.test_documentation_paths_do_not_hide_build_or_fixture_changes) ... ok
test_e2e_scope_keeps_production_and_shared_configuration_full (__main__.SelfTests.test_e2e_scope_keeps_production_and_shared_configuration_full) ... ok
test_e2e_shortcut_rejects_new_direct_renamed_and_target_dependencies (__main__.SelfTests.test_e2e_shortcut_rejects_new_direct_renamed_and_target_dependencies) ... ok
test_e2e_shortcut_traverses_implicit_path_dependencies (__main__.SelfTests.test_e2e_shortcut_traverses_implicit_path_dependencies) ... ok
test_event_contract_requires_complete_candidate_and_optional_lanes (__main__.SelfTests.test_event_contract_requires_complete_candidate_and_optional_lanes) ... ok
test_every_wrong_result_missing_job_or_selection_fails_closed (__main__.SelfTests.test_every_wrong_result_missing_job_or_selection_fails_closed) ... ok
test_full_e2e_gate_preserves_workflow_branch_and_event_scope (__main__.SelfTests.test_full_e2e_gate_preserves_workflow_branch_and_event_scope) ... ok
test_git_range_includes_deleted_source_and_rename_origins (__main__.SelfTests.test_git_range_includes_deleted_source_and_rename_origins) ... ok
test_job_ids_and_display_names_cannot_hide_validation (__main__.SelfTests.test_job_ids_and_display_names_cannot_hide_validation) ... ok
test_report_only_stack_lane_mirrors_e2e_smoke_outside_the_aggregate (__main__.SelfTests.test_report_only_stack_lane_mirrors_e2e_smoke_outside_the_aggregate) ... ok
test_repository_wiring_and_missing_dependency_regression (__main__.SelfTests.test_repository_wiring_and_missing_dependency_regression) ... ok
test_verify_command_preserves_failures (__main__.SelfTests.test_verify_command_preserves_failures) ... ok

----------------------------------------------------------------------
Ran 13 tests in 3.280s

OK
security coverage self-test passed
test_api_failures_preserve_other_evidence_and_never_enter_grace (__main__.SelfTests.test_api_failures_preserve_other_evidence_and_never_enter_grace) ... ok
test_attempt_outcome_does_not_replace_recent_success (__main__.SelfTests.test_attempt_outcome_does_not_replace_recent_success) ... ok
test_cancelled_attempt_cannot_refresh_expired_success (__main__.SelfTests.test_cancelled_attempt_cannot_refresh_expired_success) ... ok
test_cli_requires_the_repository_default_branch (__main__.SelfTests.test_cli_requires_the_repository_default_branch) ... ok
test_config_rejects_duplicate_and_invalid_entries (__main__.SelfTests.test_config_rejects_duplicate_and_invalid_entries) ... ok
test_freshness_boundaries (__main__.SelfTests.test_freshness_boundaries) ... ok
test_grace_requires_two_successful_queries_with_no_history (__main__.SelfTests.test_grace_requires_two_successful_queries_with_no_history) ... ok
test_invalid_api_evidence_fails_closed (__main__.SelfTests.test_invalid_api_evidence_fails_closed) ... ok
test_report_retains_every_workflow (__main__.SelfTests.test_report_retains_every_workflow) ... ok
test_requests_filter_schedule_default_branch_and_success_on_server (__main__.SelfTests.test_requests_filter_schedule_default_branch_and_success_on_server) ... ok

----------------------------------------------------------------------
Ran 10 tests in 0.051s

OK
..............................
----------------------------------------------------------------------
Ran 30 tests in 22.354s

OK
..........
----------------------------------------------------------------------
Ran 10 tests in 2.969s

OK
.................................
----------------------------------------------------------------------
Ran 33 tests in 0.160s

OK
Online functional runner labels: pf-testing
.......
----------------------------------------------------------------------
Ran 7 tests in 0.002s

OK
.......
----------------------------------------------------------------------
Ran 7 tests in 0.001s

OK
....
----------------------------------------------------------------------
Ran 4 tests in 0.128s

OK
...
----------------------------------------------------------------------
Ran 3 tests in 0.547s

OK
PASS: cold stampede script self-test
exit_code=0

$ python3 ./scripts/test_object_generation_protocol_model.py
test_a_recovery_barrier_cannot_choose_an_empty_value (__main__.ObjectGenerationProtocolModel.test_a_recovery_barrier_cannot_choose_an_empty_value) ... ok
test_all_bounded_two_proposer_interleavings_choose_at_most_one_value (__main__.ObjectGenerationProtocolModel.test_all_bounded_two_proposer_interleavings_choose_at_most_one_value) ... ok
test_an_unavailable_decision_quorum_cannot_choose_a_candidate (__main__.ObjectGenerationProtocolModel.test_an_unavailable_decision_quorum_cannot_choose_a_candidate) ... ok
test_counter_allocation_requires_persistence_and_is_consumed_once (__main__.ObjectGenerationProtocolModel.test_counter_allocation_requires_persistence_and_is_consumed_once) ... ok
test_counter_restart_burns_persisted_allocations_before_or_after_issue (__main__.ObjectGenerationProtocolModel.test_counter_restart_burns_persisted_allocations_before_or_after_issue) ... ok
test_crash_after_promise_before_reply_does_not_restore_an_older_ballot (__main__.ObjectGenerationProtocolModel.test_crash_after_promise_before_reply_does_not_restore_an_older_ballot) ... ok
test_duplicate_or_unconfigured_voters_cannot_form_a_quorum (__main__.ObjectGenerationProtocolModel.test_duplicate_or_unconfigured_voters_cannot_form_a_quorum) ... ok
test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision (__main__.ObjectGenerationProtocolModel.test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision) ... ok
test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected (__main__.ObjectGenerationProtocolModel.test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected) ... ok
test_forgetting_tombstone_authority_allows_an_aba_create (__main__.ObjectGenerationProtocolModel.test_forgetting_tombstone_authority_allows_an_aba_create) ... ok
test_issuing_before_persistence_reuses_a_ballot_after_restart (__main__.ObjectGenerationProtocolModel.test_issuing_before_persistence_reuses_a_ballot_after_restart) ... ok
test_lost_ack_and_restart_preserve_the_chosen_operation (__main__.ObjectGenerationProtocolModel.test_lost_ack_and_restart_preserve_the_chosen_operation) ... ok
test_minority_accepted_work_is_adopted_and_finished (__main__.ObjectGenerationProtocolModel.test_minority_accepted_work_is_adopted_and_finished) ... ok
test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample (__main__.ObjectGenerationProtocolModel.test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample) ... ok
test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it (__main__.ObjectGenerationProtocolModel.test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it) ... ok
test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor (__main__.ObjectGenerationProtocolModel.test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor) ... ok
test_replacing_voters_with_empty_state_can_choose_a_conflicting_value (__main__.ObjectGenerationProtocolModel.test_replacing_voters_with_empty_state_can_choose_a_conflicting_value) ... ok
test_skipping_predecessor_recovery_chooses_a_broken_revision_chain (__main__.ObjectGenerationProtocolModel.test_skipping_predecessor_recovery_chooses_a_broken_revision_chain) ... ok
test_stale_publication_rollback_and_gc_cannot_change_the_winner (__main__.ObjectGenerationProtocolModel.test_stale_publication_rollback_and_gc_cannot_change_the_winner) ... ok
test_superseded_operation_identity_cannot_be_reused_for_different_input (__main__.ObjectGenerationProtocolModel.test_superseded_operation_identity_cannot_be_reused_for_different_input) ... ok
test_superseded_operation_retry_returns_its_original_successor (__main__.ObjectGenerationProtocolModel.test_superseded_operation_retry_returns_its_original_successor) ... ok
test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision (__main__.ObjectGenerationProtocolModel.test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision) ... ok
test_tombstone_rejects_a_delayed_never_present_create (__main__.ObjectGenerationProtocolModel.test_tombstone_rejects_a_delayed_never_present_create) ... ok
test_unlearned_predecessor_is_recovered_before_the_second_slot (__main__.ObjectGenerationProtocolModel.test_unlearned_predecessor_is_recovered_before_the_second_slot) ... ok

----------------------------------------------------------------------
Ran 24 tests in 0.958s

OK
model: states=51023, transitions=167520, terminal=732, chosen_candidates=2
negative control: double-choice trace=((0, 'prepare', 0), (0, 'prepare', 1), (0, 'prepare', 2), (0, 'accept', 0), (0, 'accept', 1), (0, 'accept', 2), (1, 'prepare', 0), (1, 'prepare', 1), (1, 'prepare', 2), (1, 'accept', 0), (1, 'accept', 1), (1, 'accept', 2))
multi-slot round schedules: 640
exit_code=0

$ python3 ./scripts/check_test_wiring.py
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired
exit_code=0

$ ./scripts/check_no_planning_docs.sh
exit_code=0

$ ./scripts/check_uring_lane_lib_only.sh
OK: no 'uring_'-matching test functions under crates/ecstore/tests; --lib is safe
exit_code=0

Initial complete gate
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
 Nextest run ID e07fc5b9-c6c8-4cdd-ae61-54839bf24437 with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.016s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.011s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.009s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.009s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  16.950s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   4.025s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.008s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.007s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.035s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  21.087s] 10 tests run: 10 passed, 0 skipped
exit_code=0

Restored complete gate
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
 Nextest run ID 233b7307-5361-4eb7-94d5-b384f17d2d6a with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.009s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.008s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.008s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.008s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  19.518s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   4.146s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.009s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.018s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.009s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.054s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  23.787s] 10 tests run: 10 passed, 0 skipped
exit_code=0

Mutation M1: success_measurement reports max(actual elapsed, requested budget + 1), forcing 1001 ms for the one-second request. The retained duration assertion at rustfs/tests/connect_perf_drive.rs:105 fails.
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast -E 'test(real_local_write_read_is_measured_and_scratch_is_removed)'
info: experimental features enabled: setup-scripts
   Compiling rustfs v1.0.1 (/private/tmp/gw-orchestrator-2734/ci-drive/rustfs)
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs") generated 1 warning
warning: `rustfs` (bin "rustfs-cli") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 7m 06s
────────────
 Nextest run ID 9f618777-7bf6-4cbc-8803-dd7f50f449ed with nextest profile: ci
    Starting 1 test across 1 binary (9 tests skipped)
        FAIL [   0.016s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
  stdout ───

    running 1 test
    test real_local_write_read_is_measured_and_scratch_is_removed ... FAILED

    failures:

    failures:
        real_local_write_read_is_measured_and_scratch_is_removed

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 0.01s

  stderr ───

    thread 'real_local_write_read_is_measured_and_scratch_is_removed' (106986846) panicked at rustfs/tests/connect_perf_drive.rs:105:5:
    assertion failed: (1..=1_000).contains(&data.duration_millis)
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

────────────
     Summary [   0.017s] 1 test run: 0 passed, 1 failed, 9 skipped
        FAIL [   0.016s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
error: test run failed
exit_code=100

Mutation M2: target_parameters reports requested duration + 1. The in-process assertion at rustfs/tests/connect_perf_drive.rs:111 and CLI JSON assertion at rustfs/tests/connect_perf_drive.rs:527 both reject 1001 against 1000.
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast -E 'test(real_local_write_read_is_measured_and_scratch_is_removed) | test(production_cli_measures_and_signs_exact_binary_provenance)'
info: experimental features enabled: setup-scripts
   Compiling rustfs v1.0.1 (/private/tmp/gw-orchestrator-2734/ci-drive/rustfs)
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs") generated 1 warning
warning: `rustfs` (bin "rustfs-cli") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 7m 20s
────────────
 Nextest run ID b73d46ab-c910-4e1e-8063-7fc7e332ec15 with nextest profile: ci
    Starting 2 tests across 1 binary (8 tests skipped)
        FAIL [  13.968s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
  stdout ───

    running 1 test
    test production_cli_measures_and_signs_exact_binary_provenance ... FAILED

    failures:

    failures:
        production_cli_measures_and_signs_exact_binary_provenance

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 13.96s

  stderr ───

    thread 'production_cli_measures_and_signs_exact_binary_provenance' (107015462) panicked at rustfs/tests/connect_perf_drive.rs:527:5:
    assertion `left == right` failed
      left: Number(1001)
     right: 1000
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

        FAIL [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
  stdout ───

    running 1 test
    test real_local_write_read_is_measured_and_scratch_is_removed ... FAILED

    failures:

    failures:
        real_local_write_read_is_measured_and_scratch_is_removed

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 0.01s

  stderr ───

    thread 'real_local_write_read_is_measured_and_scratch_is_removed' (107016667) panicked at rustfs/tests/connect_perf_drive.rs:111:5:
    assertion `left == right` failed
      left: 1001
     right: 1000
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

────────────
     Summary [  13.985s] 2 tests run: 0 passed, 2 failed, 8 skipped
        FAIL [  13.968s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        FAIL [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
error: test run failed
exit_code=100

Transient production mutations were restored byte-for-byte before the complete green rerun. Original/restored SHA256: 8d389e6d8836c041aab8c6a93e56addd65ba3f70b33264e1c74f639d0572a2da.

$ git diff --check

exit_code=0

$ git diff --cached --check

exit_code=0

Independent orchestrator acceptance
Tested code tree: e67c72fb01

Tested tree: 16fbbd2ea4
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

$ LANG=en_US.UTF-8 CARGO_BUILD_JOBS=1 CARGO_INCREMENTAL=0 cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
info: experimental features enabled: setup-scripts
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs-cli") generated 1 warning
warning: `rustfs` (bin "rustfs") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.69s
────────────
 Nextest run ID cd1edf4c-f1c0-456d-ad1a-76cc00ce05c9 with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.012s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.010s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.010s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.009s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  20.328s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   5.527s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.013s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.019s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.011s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.049s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  25.991s] 10 tests run: 10 passed, 0 skipped

exit_code=0

All 54 tracked Cargo.toml files parse without duplicate keys. git diff --check passed. The production source remains unchanged; the two deliberate implementation mutations above were restored before this complete green run.

* fix(test): compare legacy body wire parity as raw bytes

Lossy UTF-8 comparison concealed a changed binary response byte.
Compare the already captured Vec<u8> values directly.

Validation:
- Binary mutant: migrated payload 0x84 vs legacy 0x83; old comparator
  passed 1/1, raw comparator failed 0/1 (exit 100); restored green.
- Non-Swift body-drain mutant: untouched-body delegation test failed
  with empty vs payload (exit 100); restored handler without a diff.
- cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols
  --features rustfs-protocols/swift --no-fail-fast:
  543 tests run, 543 passed, 0 skipped after both mutants restored.
- cargo fmt --all --check and git diff --check passed.

Orchestrator acceptance (LANG=en_US.UTF-8):
cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
Summary [2.445s] 543 tests run: 543 passed, 0 skipped.

All shared quick checks on the final code tree:
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

Final independent acceptance after the drive-test CI repair
Parent: a99d8f4f89
Tested code tree: 93cbd96bf0

Tested tree: 075ae5161c2b62a96812bc1e3c904a7663b4f2f8
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

$ LANG=en_US.UTF-8 CARGO_BUILD_JOBS=1 CARGO_INCREMENTAL=0 cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
    Finished `test` profile [unoptimized + debuginfo] target(s) in 10m 08s
 Nextest run ID 5c7ff102-ec6d-479a-86e0-810859b28bbf with nextest profile: ci
    Starting 543 tests across 8 binaries
     Summary [   6.729s] 543 tests run: 543 passed, 0 skipped
exit_code=0

The only source change is the raw-byte wire comparator. Cargo manifests and the ratchet baselines remain unchanged from the independently validated parent. The invalid-byte and consumed-request mutations remain recorded above; their implementation changes were restored.

* fix: repair integration checks and IPC test readiness

Wait for the RPC/locks test sockets to pass the existing owner and mode
checks inside the original 5s readiness window and 10ms poll interval.

Use the existing admin S3 carrier for the new recovery and quota error
calls. Regenerate the test-count baseline for main's 118 existing tests
across seven crates. Pair the private pagination helper's raw-scan inputs
and remove the redundant Copy clone from the retirement event.

Keep the original assertions, responses, authorization, locks and storage
operations. The pagination body and raw-scan expressions at all five calls
are byte-identical; reinserting only the Copy clone restores the entire
base heal file. All ignored baseline values and s3s counts stay unchanged.

Validation base: c47432e353
Validation tree: f081084bc5
Patch SHA-256: 2dcf0028691bfb2772edfb38e262829b04f4151832dcce823c9dd9f7434b52de
Trace production-prefix SHA-256: fd994f218a21429d62866f4efb8eee1ee57b58f4a51865aa35d05a54ef1ccb45

All commands used LANG=en_US.UTF-8, CARGO_BUILD_JOBS=1,
CARGO_INCREMENTAL=0 and CARGO_NET_OFFLINE=true in one exclusive target.
Every gate retained the same tree and six source hashes before and after.
54 Cargo.toml files parse without duplicate keys.

cargo clippy -p rustfs --all-targets -- -D warnings
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 12m 50s
Exit: 0. Wall time: 770.855s.

The original 19 quick-check steps:
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
Wall time: 286.337s.

cargo nextest run --profile ci -p rustfs --no-fail-fast
     Summary [ 217.017s] 5382 tests run: 5382 passed (4 leaky), 40 skipped
Exit: 0. Wall time: 1324.414s.

cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
     Summary [   2.189s] 543 tests run: 543 passed, 0 skipped
Exit: 0. Wall time: 123.192s.

cargo nextest run --profile ci -p rustfs-ecstore list_objects_paginate --no-fail-fast
     Summary [   0.170s] 8 tests run: 8 passed, 5921 skipped
Exit: 0. Wall time: 261.324s.

cargo nextest run --profile ci -p rustfs-heal durable_mrf_anchor_retires_only_after_exact_bucket_generation_is_deleted --no-fail-fast
     Summary [   0.805s] 1 test run: 1 passed, 835 skipped
Exit: 0. Wall time: 199.801s.

Baseline regeneration:
./scripts/check_test_count_ratchet.sh --update
wrote .config/test-count-baseline.txt: 52 crates, 21340 tests, 164 ignored
Test-only increases: ecstore48, heal3, iam14, io-metrics2, obs3, policy14,
rustfs34. Existing ignored values and all other rows are byte-identical.
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)

Historical runtime red controls on f622 (same retained two assertions):
cargo nextest run --profile ci -p rustfs --lib -E 'test(=connect::diagnostics::trace_runtime::tests::local_top_rpc_reads_service_http_completions_and_signs_offline) | test(=connect::diagnostics::trace_runtime::tests::local_top_locks_reads_service_manager_and_signs_offline)' --no-fail-fast
Original path readiness with a temporary 500ms bind/seal gap:
     Summary [   0.863s] 2 tests run: 0 passed, 2 failed, 5113 skipped
Exit: 100.
Sealed readiness in that same gap:
     Summary [   5.621s] 2 tests run: 2 passed, 5113 skipped
Exit: 0.
Compiled M1: replace the helper predicate with path.exists():
     Summary [   0.870s] 2 tests run: 0 passed, 2 failed, 5113 skipped
Exit: 100.
Restore the sealed predicate byte-for-byte:
     Summary [   5.640s] 2 tests run: 2 passed, 5113 skipped
Exit: 0.
Both reds observe mode755 with owner_matches=true and fail at the original
RPC/locks unwraps with StateSecurity. M1 failure locations: trace_runtime.rs
2547:29 and 2713:29. No new assertions were introduced.

cargo nextest run --profile ci -p rustfs --lib -E 'test(connect::diagnostics::trace_runtime::tests::)' --no-fail-fast
     Summary [  20.748s] 28 tests run: 28 passed (1 leaky), 5087 skipped
Exit: 0. This security sweep is historical; the full final unit gate above
covers the actual c474 candidate.
The first disk-stopped compile ran no tests and is excluded from red credit.
All temporary seal instrumentation is removed from the final source.

Final unit cleanup markers: 4 leaky, 40 original skips (exit0). The ld
__eh_frame linker warning also appears verbatim in the original f622 run
and the earlier c474 run; there are no new source warnings.

* fix: remove redundant pending closure in disk test

Pass the capture-free pending future factory as a function item in the existing slow-operation test. This fixes the redundant_closure error at disk_store.rs:2914 that stopped strict Clippy before the feature tests could run.

The single-line test edit preserves the factory's result, warning and timeout behavior. Reversing the replacement reproduces the parent file byte for byte. All 220 assertion lines and the production prefix are unchanged; no tests, assertions, guards, profiles or baselines were added or relaxed.

Original Linux CI evidence (run 37888552268 at 53245be9f):
- RIO job113685483992: strict Clippy exit101, redundant closure at disk_store.rs:2914; feature tests skipped.
- Workspace Test and Lint job113685484058: same strict Clippy exit101; nextest and doc tests skipped.
These are lint failures, not runtime mutation kills or known flakes.

Native macOS arm64 validation on tree7a5bf47e7533b29479bc85207416102f1337c566, LANG=en_US.UTF-8, one build job, CARGO_INCREMENTAL=0 and offline dependencies:

shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
All 19 checks: exit0, 312.161s.

$ cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 20m 11s
exit0, 1213.210s total.

$ cargo nextest run --profile ci -p rustfs-ecstore --features rio-v2 slow_disk_operation_warns_once_while_pending_and_preserves_result --no-fail-fast
Finished `test` profile [unoptimized + debuginfo] target(s) in 4m 34s
PASS [0.026s] rustfs-ecstore disk::disk_store::tests::slow_disk_operation_warns_once_while_pending_and_preserves_result
Summary [   0.170s] 1 test run: 1 passed, 5943 skipped
exit0, 281.920s total; 5943 existing filter skips.

warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
This linker warning is retained; flags, profiles and allowances are unchanged.

54 tracked Cargo.toml files parsed without duplicate keys. git diff --check passed. No new assertions were introduced, so no new runtime mutation is claimed. Native compilation and the existing helper test do not substitute for actual Linux RIO execution; the new integration CI run still needs to recover before the RPC merge.

Preservation and output hashes:
parent file SHA256: 8e59ee745e3aaaa22848b943d3b10f6353b515956282cde1f3af2c394200c9ac
final file SHA256: b69ba3237061b161448ae8b62b615885d5770b98a5b4a2479bcfbd70f1bd1f17
unchanged production prefix SHA256: 7903a6e288252c420231242d7b58a25fa2a7140ea3dd430cbe7f0dde11369dcb
RIO Linux failure log SHA256:421addb9868fd88e4109a922311e54f91f0dfd70862f8ba7fcaba193482ed90b
Workspace Linux failure log SHA256:e0aea8d0f014c022252a9534968e083c98cfc1fd760da445ecd04ec3f522e26a
19 output SHA256:35a54f28d707956ed1edb29d492c4dfe8a9fe753d103b111b5a2f1b5c41c2b98
RIO Clippy output SHA256:d6046e3868f2b3992fe11cdd04e9e98eb50a87d1234d9a70dbfeb154ff61638d
slow-operation output SHA256:605b7b258d2fd09c6ef2ab2ffc90d433d0509480f1e45125d20c823759e83608

Refs rustfs/backlog#2734.

* test(ecstore): isolate tier-free recovery fixture startup

The fixture constructor can take the production recovery lock and wait for 60 seconds before this test installs its queue and seeds metadata. Disable its background recovery only while creating this fixture, then restore the environment before calling the existing production recovery entrypoint.

Only the existing test setup changes. Its production entrypoint, default lock parameters, original assertions, 30-second timeout, cancellation and cleanup remain intact. No tests or assertions were added. Production and store/init.rs are byte-identical to the parent.

Independent acceptance ran on merge tree 8ec358cb42ba9ebb3bcfeafcef048a3f0366355b; the runner header reports parent HEAD, while before/after index and source hashes confirm this candidate. Worker task branch d8055766b8 has the same tree.

Every command used LANG=en_US.UTF-8. Task compilation also used CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0, CARGO_NET_OFFLINE=true, an independent target, and empty RUSTC_WRAPPER/RUSTC_WORKSPACE_WRAPPER to keep compilers in its owned process group without changing repository configuration.

Command:
shellcheck --version && actionlint
Exit 0; observed output excerpt:
version: 0.11.0
Log SHA-256: fc9eec16b4191d058a295813ca9afbe9d2b37b59975da860170726be162a3924

Command:
cargo fmt --all --check
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_unsafe_code_allowances.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_layer_dependencies.sh
Exit 0; observed output excerpt:
Layer dependency guard passed (no new reverse dependencies/cycles).
Log SHA-256: 69c6cb1896c0586b28d1167380d3260d99d8017a4265c8df5a2348f289e93a12

Command:
./scripts/check_architecture_migration_rules.sh
Exit 0; observed output excerpt:
Architecture migration rules passed.
Log SHA-256: 28790e70d070607ed4de0704f292a65030be92ec224ef106d4b72fd8296f896e

Command:
./scripts/check_logging_guardrails.sh
Exit 0; observed output excerpt:
✅ Logging guardrails check passed
Log SHA-256: 7adf41abff8ed8f7617decf86498888ddde78937099dd9ce47f6d359dfc2d317

Command:
./scripts/check_error_other_format_ratchet.sh
Exit 0; observed output excerpt:
error(format!) ratchet passed (total call sites: 579).
Log SHA-256: 66172aef18f150ba0605af122e26ffcf91f6dd86f43e38b05e2e982ce8d783ee

Command:
./scripts/check_no_tokio_io_uring.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_extension_schema_boundaries.sh
Exit 0; observed output excerpt:
Extension schema boundaries passed.
Log SHA-256: f13d0acd0e2ce5d9815241b12b4dab7d01cc11b9b65bcf6dfb7812c878532a51

Command:
./scripts/check_body_cache_whitelist.sh
Exit 0; observed output excerpt:
Body-cache whitelist guard passed.
Log SHA-256: 6b828847b0383950ee6eef9cd846d0c322c945e47efd4c7d1ad881178810d6bb

Command:
./scripts/check_s3s_footprint.sh
Exit 0; observed output excerpt:
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)
✅ s3s footprint baseline check passed
Log SHA-256: 47f53d04ef645618563fe1f22abc64667f19078f1974feedcca46e4f2c016844

Command:
./scripts/check_test_count_ratchet.sh
Exit 0; observed output excerpt:
test-count ratchet OK: 52 crates, 21340 tests, 164 ignored match .config/test-count-baseline.txt
Log SHA-256: 618a5e330e933de0c6ff484546df401457024a88bb5986d11d588794e33dab60

Command:
./scripts/check_fips_wording.sh
Exit 0; observed output excerpt:
FIPS wording guard passed (README.md and CHANGELOG.md contain no forbidden claims).
KMS wording guard passed (crates/kms claims no Vault KV2 Transit wrapping).
Log SHA-256: 99221cc9979c480ec4f46f50d76c52b18c72fe499f12e15e4f4a22a1b8bfb140

Command:
./scripts/check_embedded_secrets.sh
Exit 0; observed output excerpt:
Embedded secret guard passed (no private key material or provider credential literal in the tree).
Log SHA-256: 437a3ceba32094d2901889965ad191e411aad32cdb18bf0b8659e818011c406b

Command:
make script-tests
Exit 0; observed output excerpt:
test_cli_requires_the_repository_default_branch (__main__.SelfTests.test_cli_requires_the_repository_default_branch) ... ok
test_config_rejects_duplicate_and_invalid_entries (__main__.SelfTests.test_config_rejects_duplicate_and_invalid_entries) ... ok
test_freshness_boundaries (__main__.SelfTests.test_freshness_boundaries) ... ok
test_grace_requires_two_successful_queries_with_no_history (__main__.SelfTests.test_grace_requires_two_successful_queries_with_no_history) ... ok
test_invalid_api_evidence_fails_closed (__main__.SelfTests.test_invalid_api_evidence_fails_closed) ... ok
test_report_retains_every_workflow (__main__.SelfTests.test_report_retains_every_workflow) ... ok
test_requests_filter_schedule_default_branch_and_success_on_server (__main__.SelfTests.test_requests_filter_schedule_default_branch_and_success_on_server) ... ok
Ran 10 tests in 0.042s
OK
Ran 30 tests in 20.627s
OK
Ran 10 tests in 2.268s
OK
Ran 33 tests in 0.127s
OK
Ran 7 tests in 0.002s
OK
Ran 7 tests in 0.001s
OK
Ran 4 tests in 0.108s
OK
Ran 3 tests in 0.564s
OK
PASS: cold stampede script self-test
Log SHA-256: 75fda2d069a373c63667b908baaf99e6591aab9099a739b0c4c22ceb6da8d68c

Command:
python3 ./scripts/test_object_generation_protocol_model.py
Exit 0; observed output excerpt:
test_an_unavailable_decision_quorum_cannot_choose_a_candidate (__main__.ObjectGenerationProtocolModel.test_an_unavailable_decision_quorum_cannot_choose_a_candidate) ... ok
test_counter_allocation_requires_persistence_and_is_consumed_once (__main__.ObjectGenerationProtocolModel.test_counter_allocation_requires_persistence_and_is_consumed_once) ... ok
test_counter_restart_burns_persisted_allocations_before_or_after_issue (__main__.ObjectGenerationProtocolModel.test_counter_restart_burns_persisted_allocations_before_or_after_issue) ... ok
test_crash_after_promise_before_reply_does_not_restore_an_older_ballot (__main__.ObjectGenerationProtocolModel.test_crash_after_promise_before_reply_does_not_restore_an_older_ballot) ... ok
test_duplicate_or_unconfigured_voters_cannot_form_a_quorum (__main__.ObjectGenerationProtocolModel.test_duplicate_or_unconfigured_voters_cannot_form_a_quorum) ... ok
test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision (__main__.ObjectGenerationProtocolModel.test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision) ... ok
test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected (__main__.ObjectGenerationProtocolModel.test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected) ... ok
test_forgetting_tombstone_authority_allows_an_aba_create (__main__.ObjectGenerationProtocolModel.test_forgetting_tombstone_authority_allows_an_aba_create) ... ok
test_issuing_before_persistence_reuses_a_ballot_after_restart (__main__.ObjectGenerationProtocolModel.test_issuing_before_persistence_reuses_a_ballot_after_restart) ... ok
test_lost_ack_and_restart_preserve_the_chosen_operation (__main__.ObjectGenerationProtocolModel.test_lost_ack_and_restart_preserve_the_chosen_operation) ... ok
test_minority_accepted_work_is_adopted_and_finished (__main__.ObjectGenerationProtocolModel.test_minority_accepted_work_is_adopted_and_finished) ... ok
test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample (__main__.ObjectGenerationProtocolModel.test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample) ... ok
test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it (__main__.ObjectGenerationProtocolModel.test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it) ... ok
test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor (__main__.ObjectGenerationProtocolModel.test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor) ... ok
test_replacing_voters_with_empty_state_can_choose_a_conflicting_value (__main__.ObjectGenerationProtocolModel.test_replacing_voters_with_empty_state_can_choose_a_conflicting_value) ... ok
test_skipping_predecessor_recovery_chooses_a_broken_revision_chain (__main__.ObjectGenerationProtocolModel.test_skipping_predecessor_recovery_chooses_a_broken_revision_chain) ... ok
test_stale_publication_rollback_and_gc_cannot_change_the_winner (__main__.ObjectGenerationProtocolModel.test_stale_publication_rollback_and_gc_cannot_change_the_winner) ... ok
test_superseded_operation_identity_cannot_be_reused_for_different_input (__main__.ObjectGenerationProtocolModel.test_superseded_operation_identity_cannot_be_reused_for_different_input) ... ok
test_superseded_operation_retry_returns_its_original_successor (__main__.ObjectGenerationProtocolModel.test_superseded_operation_retry_returns_its_original_successor) ... ok
test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision (__main__.ObjectGenerationProtocolModel.test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision) ... ok
test_tombstone_rejects_a_delayed_never_present_create (__main__.ObjectGenerationProtocolModel.test_tombstone_rejects_a_delayed_never_present_create) ... ok
test_unlearned_predecessor_is_recovered_before_the_second_slot (__main__.ObjectGenerationProtocolModel.test_unlearned_predecessor_is_recovered_before_the_second_slot) ... ok
Ran 24 tests in 0.929s
OK
Log SHA-256: 0b4be926efb0609d497ae58930340153f01957b606206cbbdb3e0d47eb29d3a2

Command:
python3 ./scripts/check_test_wiring.py
Exit 0; observed output excerpt:
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired
Log SHA-256: 21f676f458451525535ab0ef73d6a9c0d9774e16d76a4d19e396ad34936dd39c

Command:
./scripts/check_no_planning_docs.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_uring_lane_lib_only.sh
Exit 0; observed output excerpt:
OK: no 'uring_'-matching test functions under crates/ecstore/tests; --lib is safe
Log SHA-256: 43f3f851e27ec2ab9e34d6b05fefdede38af992699c473904bfe6f53a46d52e3

Command:
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast
Exit 0; wall 234.399s.
Observed output:
    Finished `test` profile [unoptimized + debuginfo] target(s) in 3m 49s
    Starting 1 test across 1 binary (5871 tests skipped)
  SETUP PASS [   0.006s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"'
        PASS [   0.440s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item
     Summary [   0.586s] 1 test run: 1 passed, 5871 skipped
Log SHA-256: 34e973f86e8dc2102c1a0e483e0ac802ce7df3d1b40a603932f62c0805c4de27

Command:
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over) | test(=store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence) | test(=store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable)' --no-fail-fast
Exit 0; wall 4.010s.
Observed output:
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.43s
    Starting 3 tests across 1 binary (5869 tests skipped)
  SETUP PASS [   0.005s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"'
        PASS [   0.068s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over
        PASS [   0.637s] rustfs-ecstore store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence
        PASS [   0.846s] rustfs-ecstore store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable
     Summary [   0.987s] 3 tests run: 3 passed, 5869 skipped
Log SHA-256: b5da2a927ff18ca9aa217762f7186a6d4371bd9e8feedd7b77940e10b64dffac

Command:
cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
Exit 0; wall 1084.508s.
Observed output:
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 18m 03s
Log SHA-256: 28f98be7970faaa71cd118da1da4e6e2e3d23b9dda7e6449eaa7a37e741260b5

Compiled mutation command (unchanged for each red and restored run):
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast

M1: reverse only this test-local constructor flag from false to true. The mutant compiled, then failed the original recovered-item queue assertion after 30.481s. Restoring the entire observed candidate passed in 0.460s. Red log 55bf3b1535529ac0bcfb8071f9b56f9f40631007a85c315367e7e93786d65293; restored log 204d5fbe4574a7917f5e740a795d864fe28a7e7fb5e33d27b0a9cf68e2f17f52.

M2: replace only the actual awaited enqueue with false, consuming its original inputs. The mutant compiled, then failed the same original queue assertion after 30.516s. Restoring the entire observed candidate passed in 0.457s. Red log 89186330413215e231e2b06bf013cc31097ea8f4a106d91421a36d6c8986a7fd; restored log e848b42d2302401afa5dbcd9e37bf76d2e56a96ab6de5a488513b997f53c5e9b.

All temporary observation and mutation code was removed before final acceptance. Historical baseline and diagnostic failures remain recorded; they are not mutation credit. The separate Object Lock intent-retry diagnostic still fails, and these native macOS checks do not establish Linux CI recovery.

* refactor(rpc): serve internode rpc bodies with the owned Body (#8441)

* test(rpc): pin the wire bytes of every internode rpc response body

rustfs/backlog#2747 batch 4, step 1 of 2, on the s3s bodies the rpc paths
serve today. Each response body builder in storage/rpc/http_service.rs is
served through hyper's HTTP/1.1 server and the raw bytes the peer reads are
compared with goldens captured from this commit: status, content-length or
chunked framing, chunk boundaries, the terminating chunk, HEAD, and the cut
connection a failure after data produces. The capability responses carry a
per-process proof, so their length header and payload are checked against
the body they encode instead.

The read_file_stream response builder moves into read_file_stream_response
so the test reaches the same code the handler runs; behaviour is unchanged.

* refactor(rpc): serve internode rpc bodies with the owned Body

rustfs/backlog#2747 batch 4, step 2 of 2. The internode rpc HTTP paths
build their responses with rustfs_s3_types::Body: Body::from_stream
replaces Body::from(StreamingBlob::wrap(..)) at the six stream sites, and
the in-memory and empty bodies keep their constructors. InternodeRpcService
converts its responses into the wrapped service's response body through
From<Body>, so the server keeps composing it with the s3s service as
before. The wire-byte goldens from the previous commit pass unchanged;
storage/rpc no longer names s3s.

* fix(rpc): simplify wire test response builder type

Name the boxed response builder used by the capability wire test so
all-target Clippy can check it without a type-complexity warning.

Validation (LANG=en_US.UTF-8, CARGO_INCREMENTAL=0):
- cargo clippy -p rustfs --all-targets -- -D warnings: exit 0.
- cargo nextest run --profile ci -p rustfs --lib -E 'test(/storage::rpc::/)': 279 tests run, 279 passed, 4842 skipped.
- All 19 quick-check commands, including script-tests, both Python scripts and cargo fmt --all --check: exit 0.
- Footprint baseline regenerated: files=194, s3_error_lines=1584, ecstore_files=36; test-count baseline regenerated: 52 crates, 21228 tests, 164 ignored.
- All 54 Cargo.toml files parse without duplicate keys.

No assertion was added or changed by this lint fix. R1-R8 are retained
in the existing PR verification record.

* fix(ecstore): resume fenced Object Lock creation intents

* fix(ecstore): bound creation retry futures on the stack

* fix(ecstore): require native proof for creation retries

* fix(ecstore): bound bucket creation stack usage (#8466)

* fix(ecstore): retain creation fences through rollback

Keep lifecycle, metadata-transaction, and namespace guards around rollback and metadata cleanup. Preserve the original publication error when a guard is lost. Build the timestamp retry fixture through the real interrupted producer so it checks a valid pending creation.

Validated source tree: 4e8e207999.
Two independent source reviews found no blocking regression.

Validation (LANG=en_US.UTF-8):
- bash -c 'shellcheck --version && actionlint' — exit 0
- cargo fmt --all --check — exit 0
- ./scripts/check_unsafe_code_allowances.sh — exit 0
- ./scripts/check_layer_dependencies.sh — exit 0
- ./scripts/check_architecture_migration_rules.sh — exit 0
- ./scripts/check_logging_guardrails.sh — exit 0
- ./scripts/check_error_other_format_ratchet.sh — exit 0
- ./scripts/check_no_tokio_io_uring.sh — exit 0
- ./scripts/check_extension_schema_boundaries.sh — exit 0
- ./scripts/check_body_cache_whitelist.sh — exit 0
- ./scripts/check_s3s_footprint.sh — exit 0
- ./scripts/check_test_count_ratchet.sh — exit 0
- ./scripts/check_fips_wording.sh — exit 0
- ./scripts/check_embedded_secrets.sh — exit 0
- make script-tests — exit 0
- python3 ./scripts/test_object_generation_protocol_model.py — exit 0
- python3 ./scripts/check_test_wiring.py — exit 0
- ./scripts/check_no_planning_docs.sh — exit 0
- ./scripts/check_uring_lane_lib_only.sh — exit 0
- cargo nextest run --profile ci -p rustfs-ecstore --lib --locked --no-fail-fast — 5894 passed, 0 skipped
- cargo check -p rustfs-ecstore --all-targets --locked — exit 0
- cargo nextest run --profile ci -p rustfs --lib --locked --no-fail-fast -E 'test(=admin::handlers::table_catalog::tests::cold_strong_registration_completes_while_other_bucket_backup_waits) | test(=admin::handlers::table_catalog::tests::commit_backend_borrowed_put_preserves_owner_authorization_and_precondition) | test(=admin::handlers::table_catalog::tests::disable_table_bucket_preserves_objects_and_can_be_reenabled) | test(=admin::handlers::table_catalog::tests::load_table_handler_negotiates_vended_credentials_without_breaking_metadata_only_callers) | test(=admin::handlers::table_catalog::tests::namespace_property_handler_updates_object_backed_catalog_and_maps_errors) | test(=app::object::get::tests::execute_get_object_resumes_from_relocated_pool_without_splicing_body) | test(=table_catalog::store::tests::catalog_writer_loss_prevents_recovery_storage_commit)' — 7 passed
- cargo nextest run --profile ci -p rustfs --lib --locked --no-fail-fast -E 'test(unsupported_remote_disk_)' — 2 passed

Additional fault-injection evidence remains incomplete. Timestamp M1 reached its original assertion; its restore run is being collected. M2–M5 and the three planned rollback faults have not run. An automated risk check interrupted the timestamp proof task; no assertions or validation settings were weakened.

* fix(ecstore): box nested bucket rollback state

Keep the guarded rollback future behind a heap allocation so bucket creation does not store its nested state inline. The rollback body, all three fence references and pending-proof revocation order are unchanged.

The same three original Linux tests overflowed on both e611 CI attempts. Local format, ecstore test compilation and all 19 quick-check steps pass; the original Linux feature lanes still need to validate this candidate.

* test(ecstore): isolate rollback guard loss fixtures

Run the three rollback metadata-cleanup cases in separate test processes so reused /proc/self/fd aliases from earlier cases cannot invalidate their creation witnesses. Keep the original fixture, guards, assertions and io_uring test selection, and verify that each child runs the intended case successfully.

Update the generated test-count baseline for the added macro arm.

Validation: cargo check -p rustfs-ecstore --tests; cargo fmt --all --check; all 19 quick-check commands, including make script-tests and the two Python checks. Linux io_uring acceptance and fault controls remain pending.

* refactor(replication): move the replication config onto gateway types (#8443)

* wip: T1.4 replication family

* refactor(replication): move the replication config onto gateway types

The replication configuration family leaves s3s: rustfs-replication no longer depends on s3s, and ECStore bucket metadata, the scanner and the rustfs app/admin layers hold rustfs_gateway_types::persistence::PersistedReplicationConfiguration. The persisted XML goes through the gateway persistence codec behind a new BucketConfigXml trait (the 39 byte goldens pass with unchanged fixtures); the legacy s3s edge converts once in storage/ecfs.rs through storage/s3_compat/replication.rs; the scanner data usage cache keeps its MessagePack bytes through a serde mirror pinned by bytes recorded from s3s 5761ddfe. rustfs-gateway-types 0.32.0 is pinned once (rev 772d3385). Refs rustfs/backlog#2745.

* test: use S3 serializer for list response tests

* fix(ci): restore integration test compilation

Gate the diagnostic test import with its feature, initialize the conditional
MetadataWriteStore fields explicitly, and decode the scanner replication fixture
through its existing cache boundary. Preserve the original fixture and assertions.

Validation was run on tree 544acb80c0 at 6003d4b8.
LANG=en_US.UTF-8; no source changes follow validation.

Quick checks: 19/19 passed.
- shellcheck --version && actionlint: exit 0
- cargo fmt --all --check: exit 0
- ./scripts/check_unsafe_code_allowances.sh: exit 0
- ./scripts/check_layer_dependencies.sh: exit 0
- ./scripts/check_architecture_migration_rules.sh: exit 0
- ./scripts/check_logging_guardrails.sh: exit 0
- ./scripts/check_error_other_format_ratchet.sh: exit 0
- ./scripts/check_no_tokio_io_uring.sh: exit 0
- ./scripts/check_extension_schema_boundaries.sh: exit 0
- ./scripts/check_body_cache_whitelist.sh: exit 0
- ./scripts/check_s3s_footprint.sh: exit 0
- ./scripts/check_test_count_ratchet.sh: exit 0
- ./scripts/check_fips_wording.sh: exit 0
- ./scripts/check_embedded_secrets.sh: exit 0
- make script-tests: exit 0
- python3 ./scripts/test_object_generation_protocol_model.py: exit 0
- python3 ./scripts/check_test_wiring.py: exit 0
- ./scripts/check_no_planning_docs.sh: exit 0
- ./scripts/check_uring_lane_lib_only.sh: exit 0

Darwin validation:
- cargo clippy --locked -p rustfs-ecstore -p rustfs-scanner --all-targets -- -D warnings: exit 0
- cargo nextest run --locked -p rustfs-ecstore -p rustfs-scanner --lib -E 'test(config::com::tests::save_config_preserves_retryable_system_volume_errors) | test(scanner_io::tests::replication_maintenance_refreshes_compacted_stats_across_cycles_and_retries)' --test-threads 1 --no-fail-fast: exit 0, 2/2 passed
- cargo clippy --locked -p rustfs-ecstore -p rustfs-scanner --all-targets --features rustfs-ecstore/diagnostic-read-version-caller,rustfs-scanner/diagnostic-read-version-caller -- -D warnings: exit 101
- cargo nextest run --locked -p rustfs-ecstore -p rustfs-scanner --lib --features rustfs-ecstore/diagnostic-read-version-caller,rustfs-scanner/diagnostic-read-version-caller -E 'test(config::com::tests::save_config_preserves_retryable_system_volume_errors) | test(scanner_io::tests::replication_maintenance_refreshes_compacted_stats_across_cycles_and_retries) | test(cluster::rpc::runtime_sources::tests::) | test(config::com::tests::diagnostic_config_write_retains_origin_and_api_for_results_and_cancellation)' --test-threads 1 --no-fail-fast: exit 0, 14/14 passed

Diagnostic strict Clippy remains red on four unchanged production lints:
runtime_sources.rs if_same_then_else and three redundant_locals reports in
set_disk/core/metadata_read.rs and set_disk/read.rs. Their source ranges and
feature wiring match the base; the base was not compiled separately.

Linux aarch64 validation (Rust/Clippy 1.99.0, UID 501, jobs=1):
- cargo clippy --all-targets -- -D warnings: exit 0; Finished dev profile in 68m 12s
- cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings: exit 0; Finished dev profile in 49m 45s
Source HEAD/tree and clean status matched before and after both commands.
No compiler or stack overrides were used. protoc 35.1 was installed from its
verified ARM64 release; flatc was absent and not invoked by the workspace build scripts.
The local environment differs from hosted CI.

* fix(ecstore): recover delete batches with missing versions

Treat an explicit non-null version as a no-op when it is absent from both recovery sources. Keep rejecting missing null versions, delete-marker mismatches and conflicting local generations.

Add returning-disk and partial-receipt regressions, extend journal boundary checks, and regenerate the ecstore test-count baseline. Production faults exercise the 24 added assertion sites and 7 semantic checks: 33 faults failed at the expected sites and passed after full restoration.

Validation (LANG=en_US.UTF-8, one build job):

cargo nextest run --locked -p rustfs-ecstore --lib -E 'test(delete_recovery) | test(quorum_delete_cleanup_coordinator_binds_single_and_batch) | test(crash_delete_retry_batch_reclaims_inline_and_reserved_external)' --test-threads 1 --no-fail-fast
Summary [  34.580s] 69 tests run: 69 passed, 6058 skipped; exit 0

cargo clippy --locked -p rustfs-ecstore --all-targets -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 5m 37s; exit 0

All 19 final Quick Checks completed with exit 0:

- shellcheck --version && actionlint
- cargo fmt --all --check
- ./scripts/check_unsafe_code_allowances.sh
- ./scripts/check_layer_dependencies.sh
- ./scripts/check_architecture_migration_rules.sh
- ./scripts/check_logging_guardrails.sh
- ./scripts/check_error_other_format_ratchet.sh
- ./scripts/check_no_tokio_io_uring.sh
- ./scripts/check_extension_schema_boundaries.sh
- ./scripts/check_body_cache_whitelist.sh
- ./scripts/check_s3s_footprint.sh
- ./scripts/check_test_count_ratchet.sh
- ./scripts/check_fips_wording.sh
- ./scripts/check_embedded_secrets.sh
- make script-tests
- python3 ./scripts/test_object_generation_protocol_model.py
- python3 ./scripts/check_test_wiring.py
- ./scripts/check_no_planning_docs.sh
- ./scripts/check_uring_lane_lib_only.sh

test-count ratchet OK: 52 crates, 22009 tests, 165 ignored match .config/test-count-baseline.txt
PASS: cold stampede script self-test
model: states=51023, transitions=167520, terminal=732, chosen_candidates=2
multi-slot round schedules: 640
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired

* fix(ecstore): retry tier commit intent lock contention

Retry transient namespace lock timeouts within the existing three-attempt
CAS budget. Reload and verify the original mutation identity after
contention, and preserve the final typed error when attempts are exhausted.
Cover save/read contention, exhausted retries, replacement identities and
CAS reconciliation without changing the existing timeout or lock policy.

Validation:
- cargo nextest run --locked -p rustfs-ecstore --features test-util --lib -E 'test(tier_mutation)' --test-threads 1 --no-fail-fast: 52 passed.
- cargo nextest run --locked -p rustfs-ecstore --features test-util --lib -E 'test(intent_advance_) | test(=services::tier::tier::tests::caller_cancellation_after_durable_prepare_does_not_hide_the_mutation)' --test-threads 1 --no-fail-fast: 5 passed.
- cargo clippy --locked -p rustfs-ecstore --all-targets -- -D warnings: passed.
- cargo clippy --locked -p rustfs-ecstore --all-targets --features test-util -- -D warnings: passed.
- 27 mutation pairs failed at the intended assertions and passed after restoring the production code.
- bash scripts/check_test_count_ratchet.sh --update, then bash scripts/check_test_count_ratchet.sh: 52 crates, 22013 tests, 165 ignored; ecstore increases from 6581 to 6585.

* refactor(lifecycle): replace s3s domain types with Gateway DTOs

Keep lifecycle rules, restore requests, Object Lock, and scanner cached tags in domain types. Convert at the existing HTTP boundary while preserving legacy XML, scanner cache bytes, date ordering, and tombstones.

Linux test validation before the final import-path-only adjustment:

cargo nextest run -p rustfs-ecstore bucket_config_goldens
39 passed; 6211 skipped by selector.

cargo nextest run --lib -p rustfs -E 'test(/test_zero_rule_lifecycle_tombstone_recognition/) | test(/test_local_lifecycle_staleness_axis_selection/) | test(/test_lifecycle_staleness_preserves_legacy_submillisecond_axis/) | test(/test_merge_incoming_lifecycle/) | test(/test_lifecycle_expiry_statement_matrix/) | test(/test_lifecycle_expiry_subset_xml/) | test(/put_bucket_lifecycle_validation_errors_keep_their_s3_code/) | test(/execute_put_bucket_lifecycle_configuration_rejects_missing_configuration/) | test(/execute_restore_object_rejects_missing_restore_request/) | test(/execute_restore_object_rejects_select_type/) | test(/execute_restore_object_maps_failures_to_typed_s3_errors/) | test(/execute_restore_object_returns_internal_error_when_store_uninitialized/) | test(/restore_request_conversion_preserves_legacy_output_values/) | test(/restore_request_conversion_preserves_presence_and_raw_enums/) | test(/restore_request_conversion_distinguishes_absent_and_empty_containers/)'
25 passed; 5318 skipped by selector.

cargo nextest run --profile ci -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-replication -p rustfs-scanner
7872 passed; 21 skipped under the existing CI profile.

Previously verified mutation controls: 39 production faults, 41 RED/GREEN pairs.

cargo clippy --locked -p rustfs -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-scanner --all-targets -- -D warnings
Passed in 40m 45s before the import-path adjustment, then in 1m 59s on
the final source with warnings denied. The final adjustment imports the
existing helper through the object domain facade; fmt and the architecture
check also passed.

Part of rustfs/backlog#2745. The remaining ecstore configuration families follow separately.

* refactor(admin): move error types to rustfs-s3-types (#8440)

* refactor(protocols): move error types to rustfs-s3-types by codemod

T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and
the error items of protocols move off s3s by scripts/codemods, run with
--hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it
downcasts the backend error, whose runtime type is still s3s::S3Error.

- rustfs/src/protocols/client.rs: the two webdav session methods now
  return rustfs_s3_types::S3Result and convert the s3s error with From.
- codemod: a child module that calls s3_error! through `use super::*`
  gets its own `use rustfs_s3_types::s3_error;` once its parent imports
  that macro by name; otherwise it is listed as GLOB.
- s3-types: a table test pins code, status and the s3s error the legacy
  edge renders for every variant a moved call site names.
- s3_error_lines 1585 -> 1584 (a lifecycle comment reworded).

* refactor(admin): move error types to rustfs-s3-types (WIP, no build)

T1.2 batch 4 (rustfs/backlog#2743), work in progress: rustfs check
still fails with 16 errors (listed in the PR body).

- codemod: --facade PATH treats a crate-local re-export of s3s's error
  items like s3s for the four names (9 new self-tests, 9 mutants red).
- codemod output for rustfs/src/admin and rustfs/src/site_replication
  (--facade crate::storage_api::site_replication::s3); 6 glob children
  get an explicit rustfs_s3_types::s3_error import.
- admin/router.rs: the Operation trait returns rustfs_s3_types::S3Result.
- error.rs: From<ApiError> builds the rustfs-s3-types error; the s3s
  conversion is derived from it.

* fix(admin): convert error types at the s3s boundary

Batch 4 of T1.2 left the admin and site_replication modules returning
rustfs_s3_types::S3Error while their callers and callees on the storage
and auth side still speak s3s::S3Error. Convert at each edge with the
existing From impls in rustfs-s3-types (compat_s3s.rs) instead of
changing any code, message or status:

- admin/auth.rs, admin/handlers/object_zip_download.rs,
  admin/handlers/table_catalog/mod.rs: map the s3s result returned by
  check_key_valid / authorize_request / authorize_internal_object_request
  into the admin error type with .map_err(Into::into).
- storage/ecfs.rs: the s3s-facing side maps the admin site_replication
  gate results the other way.
- admin/handlers/pools.rs: ApiError.code stays s3s::S3ErrorCode, so the
  rustfs code is converted with .into() when the struct is built, in
  production and in the two tests that build or compare it.
- admin/handlers/heal.rs, admin/runtime_sources.rs: the router returns
  s3s errors (S3Route impl); the expected rustfs code is converted with
  .into() before the comparison.
- admin/handlers/table_catalog/mod.rs: iceberg_rest_error takes
  &'static str so the Cow<'static, str> custom code borrows it; every
  caller already passes an ICEBERG_ERROR_* constant or a &'static str.
- admin/handlers/quota.rs: drop a now same-type .into() after s3_error!.
- storage_api.rs, site_replication/mod.rs: drop re-exports and an import
  that no longer have users (clippy -D warnings).

cargo check -p rustfs --all-targets and cargo clippy -p rustfs
--all-targets -- -D warnings are clean; the s3s footprint baseline is
unchanged at files=172, s3_error_lines=303, ecstore_files=28.

Refs rustfs/backlog#2743, rustfs/backlog#2734.

* refactor(ecstore): move bucket configs onto gateway persisted types (#8709)

* refactor(ecstore): move bucket configs onto gateway persisted types

Finish the ECStore batch of T1.4 (rustfs/backlog#2745). The remaining
eleven bucket configuration families (Website, CORS, Tagging, Versioning,
ObjectLock, Encryption, Notification, Logging, Accelerate, RequestPayment,
PublicAccessBlock) now read and write their stored XML through the
rustfs-gateway-types persistence and cors_tagging codecs instead of the
s3s codec, so the legacy s3s bucket-config codec and its
s3gate-metadata-xml compat marker are removed.

- ecstore: BucketConfigXml is implemented by the gateway codec for every
  family; metadata, metadata_sys, versioning, object lock, replication and
  lifecycle consumers take the persisted types. Region moves to
  rustfs_s3_types::Region, and the ETag conversion to s3s moves to the
  rustfs edge with its tests. Only warm_backend.rs and warm_backend_s3.rs
  still reference s3s, held for T1.5 (rustfs/backlog#2746).
- scanner: object lock cache, item actions and data usage use the
  persisted types.
- rustfs: the legacy S3 edge converts each family between s3s DTOs and the
  persisted types in storage/storage_api.rs by writing one codec's XML and
  reading it with the other; Tagging converts field by field.
- Bump the rustfs-gateway-types pin to 21283e87.

The 39 persisted-byte goldens are unchanged and pass, so a document the
old codec wrote reads back and re-encodes to the same bytes. s3s footprint:
files 181 -> 146, ecstore_files 28 -> 2, s3_error_lines unchanged.

* test(app): prepare the PutBucketEncryption config from the request

The uninitialized-store test passed an empty persisted configuration, which the use case now validates before it reaches the store check, so it answered MalformedXML instead of InternalError. Convert the request's configuration the way the S3 edge does.

* style(app): format the PutBucketEncryption test

* refactor(s3-client): move remote error codes and DTO enums off s3s (#8783)

Part of T1.5 (rustfs/backlog#2746), batch 1.

* refactor(s3select): run queries on gateway DTO request shapes (#8794)

* refactor(s3select): run queries on an engine request built from gateway DTO shapes

Part of T1.5 (rustfs/backlog#2746), batch 2.

* fix(s3select): keep the select request conversion in the app layer

The layer guard forbids app -> storage; the conversion moves from storage/s3_compat/select.rs to app/select_request.rs unchanged.

* fix(heal): retry transient metadata conflicts (#8812)

* fix(heal): retry transient metadata conflicts

* chore: update heal test count baseline

* test: reserve nextest capacity for progress probes (#8818)

* test(e2e): use a non-default region for the differential negative control

Since main's region-aware CreateBucket and GetBucketLocation (#8739, #8743), us-east-1 answers an empty LocationConstraint and 200 for an owned re-create, so the control no longer reported its own region. Also refresh the e2e-full linux selection from the CI listing (682 tests).

* refactor(protocols): move StorageBackend to gateway types (#8825)

* refactor(protocols): move storage backend to gateway types

* test: update e2e test count baseline

* fix: preserve copy source keys and unknown lengths

* test: update protocols test count baseline

* fix(protocols): preserve typed SFTP backend errors

* test(protocols): use gateway list input in policy tests

* fix(protocols): clear clippy failures after T1.5 merge (#8831)

fix(protocols): clear clippy failures after merge

* fix(ecstore): finalize restore metadata after local copy-back (#8836)

* fix(ecstore): finalize restore metadata after local copy-back

* test(ecstore): ratchet added restore finalization cases

* fix(ecstore): keep restore copyback compatible with older peers

* fix(ecstore): gate restore copyback on fleet compatibility

* fix(ecstore): retain transition identity during restore copyback

* fix(ecstore): route committed restore copyback locally

* fix restore copyback cleanup

* fix(ecstore): honor restore probe test fallback

* refactor(storage): migrate option errors to domain types (#8839)

* refactor(s3): migrate private multipart validation errors (#8844)

* refactor(auth): derive policy conditions from AuthzFacts (#8442)

* refactor(auth): extract authorization facts

* test(e2e): update smoke selection baseline

* test(e2e): update full selection baseline

* refactor(admin): migrate health and prefix usage to owned requests (#8887)

* refactor(admin): add owned health router boundary

* refactor(admin): migrate prefix usage to owned request

* refactor(admin): complete the owned admin request boundary (#8949)

* refactor(admin): complete the owned admin request boundary

Every type implementing the owned admin `Operation` now serves the legacy
router through one blanket impl that converts the s3s request and response
at that single edge, so handlers register with `AdminOperation(&Handler {})`
and need no adapter statics; `LegacyAdminAdapter` is removed.

`AdminRequest` gains the accessors handlers read today (`remote_addr`,
`client_info`, `request_context`, `req_info`, `req_info_mut`,
`install_req_info`, `envelope_mut`, `take_body`), the request helpers gain
envelope forms (`authenticate_envelope_credentials`,
`object_store_from_envelope`, the table-catalog `*_from_envelope` helpers,
`KmsAdminAudit::from_envelope`, `AccountAuditContext::from_envelope`,
`CallerIdentity::resolve_envelope`), `read_compatible_admin_body` accepts
either body type, and the owned `Body` gains `store_all_limited` and
`Default`. `admin::operation::test_support::TestRequest` builds owned
requests for handler tests. Unused accessors carry a per-method dead-code
expectation that each migrating handler batch removes.

No handler behavior changes; the remaining migrations follow in batches.

* chore(admin): add the T1.12 admin handler codemod

The script rewrites an admin handler file from the s3s request and response
types to the owned boundary (imports, signatures, request field and helper
calls, body access, response construction, and test request literals) and
leaves a TODO comment at every site it cannot rewrite mechanically. It is
idempotent and is removed once T1.12 completes.

* refactor(admin): migrate replication and KMS handlers to owned requests (#8951)

Move the T1.12 batch 2 admin handlers (site_replication, kms_keys, event,
replication, diagnostics, on_demand_migration, audit, account, kms_rekey,
kms_key_lifecycle, object_data_cache, tls_debug, integrity) from the s3s
request/response types to the owned AdminRequest/AdminResponse boundary.
Handler logic, authorization order, error codes and response bytes are
unchanged; the request/response plumbing now goes through the envelope
accessors and helpers introduced by the owned boundary.

The shared response helpers (utils::json_response, utils::empty_response,
handlers::admin_json_response, target_descriptor::build_json_response,
profile::authorize_profile_request) and the site replication repair
executor still answer with the s3s types, so the migrated files carry
owned-boundary forms of those helpers and convert the repair response at
its call edge. The streaming console-log body and the two devnull drains
move from the s3s byte stream to the owned Body stream/frame API.

Drop the dead_code expectations for the owned accessors and envelope
helpers this batch starts using, and lower the s3s footprint baseline to
the measured 99 files.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* refactor(admin): add owned forms of the shared admin response helpers

Handler batches were copying `json_response`, `empty_response`,
`admin_json_response`, and `build_json_response` into each migrated file
because the shared helpers still return the s3s response. Provide owned
forms next to each legacy helper (`owned_json_response`,
`owned_empty_response`, `owned_admin_json_response`,
`owned_build_json_response`, `authorize_profile_envelope`) with parity
tests against the legacy shape, derive `Debug` for `AdminResponse` so
tests can use `expect_err`, and fix three codemod rules: strip owned
string conversions in test credentials, annotate non-literal URI parses,
and leave a rebound request parameter immutable.

* refactor(admin): use the shared owned response helpers in batch 2 (#8954)

Replace the per-file owned-boundary copies of json_response,
empty_response, admin_json_response, build_json_response and
authorize_profile_request in the batch 2 admin handlers with the shared
owned_* helpers and authorize_profile_envelope that the boundary branch
now provides, and restore expect_err in the handler tests now that
AdminResponse implements Debug; the `.err().expect(..)` stand-ins were
failing the base branch's clippy step (clippy::err_expect under
-D warnings). The site replication repair call edge
keeps its conversion because the repair executor still answers with the
s3s response type.

Drop the dead_code expectations on the shared owned helpers this batch
now uses in non-test code.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* fix(admin): preserve legacy body limit error responses

* refactor(admin): move policy, ILM and batch 3 handlers to owned requests (#8956)

Batch 3 of T1.12 (rustfs/backlog#2788, parent rustfs/backlog#2734) moves
14 admin handler files to AdminRequest/AdminResponse and the owned
Operation trait: ilm_transition, policies, idp_compat, config_admin,
plugins_instances, profile_admin, group, heal, object_zip_download,
kms_key_metadata, durability, inspect_archive, sts and
gateway_key_inventory. Handler logic, authorization order, error codes
and response bytes are unchanged; test request literals use
test_support::TestRequest.

The handlers use the shared owned helpers (owned_json_response,
owned_admin_json_response, owned_build_json_response,
authorize_profile_envelope). The dead_code expectations on the
AdminRequest accessors and RequestEnvelope::uri_mut they now exercise
are removed. The legacy target_descriptor::build_json_response keeps
only its equivalence test caller and carries the matching expectation
until it retires.

object_zip_download builds the s3s request the S3 gate still reads from
the owned request facts; heal authenticates through
authenticate_envelope_credentials; the trace, inspect-archive and zip
bodies use Body::from_stream. extract_plugin_instance_filters takes the
URI so extensions.rs keeps calling it, and the two idp_compat
delegations stay on the legacy trait until their delegates migrate.

The s3s footprint baseline drops from 99 to 90 files.

* test(admin): anchor the replication diff body contract on the call

The source-contract test looked for `let body = read_compatible_admin_body(`
on one line; rustfmt wrapped that binding after the handler moved to the
owned request boundary, so the marker was no longer found. Anchor on the
call itself so formatting cannot change what the contract checks.

* refactor(admin): migrate table catalog and batch 1 to owned requests (#8955)

* fix(admin): preserve catalog body limit response bytes

* refactor(admin): migrate identity and cluster handlers to owned requests (#8966)

* refactor(admin): migrate identity and cluster handlers to owned requests

Move the T1.12 batch 4 admin handlers (user, service_account, oidc, tier,
system, pools, quota, kms_dynamic, kms_backup, rebalance, module_switch,
bucket_meta, cluster_snapshot, realtime, bucket_recovery, profile) from the
s3s request/response types to the owned AdminRequest/AdminResponse
boundary. All 90 impl Operation blocks now take AdminRequest and return
AdminResponse; handler logic, authorization order, error codes and response
bytes are unchanged. The handlers use the shared owned helpers
(owned_json_response, owned_admin_json_response, authorize_profile_envelope,
the envelope-form runtime sources and audit constructors) instead of local
copies.

profile.rs drops the s3s forms of authorize_profile_request and
profile_not_implemented_response now that profile_admin.rs is on the
envelope gate too; profile_not_implemented_response answers with an
AdminResponse. The realtime metrics stream moves from the s3s ByteStream
adapter to Body::from_stream, and the rebalance idle-404 document still
renders through s3s::S3Error::to_http_response with its body converted at
the edge.

The idp_compat delegators ImportIamV2 and AddServiceAccountLdap move to the
owned trait as well: with their targets on both traits, the method-call
delegation became ambiguous between the owned impl and the router blanket
impl, so they now take the owned request directly.

KmsAdminAudit::from_request loses its last production caller with this batch
and is deleted; its comparison test now asserts the envelope facts directly.

Lower the s3s footprint baseline to the measured 68 files.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* refactor(admin): remove the legacy admin helpers batch 4 superseded

Batch 4 moved the last production callers of the s3s-typed admin helpers
to their owned forms, which left `admin_json_response`, `json_response`,
`authenticate_request`, `authorize_admin_request`, `object_store_from_req`
and `object_store_from_extensions` unused, along with the `Body`,
`S3Request` and `S3Response` re-exports of the admin `s3` facade. Remove
them. The owned forms keep their documentation; the behavior tests of the
removed `json_response` now exercise `owned_json_response`, and the
compat-prefix test asserts the owned response directly.

Two tests existed only to compare a removed legacy helper with its owned
twin (`owned_json_response_matches_the_legacy_shape`) or to cover the
removed legacy gate (`authorize_admin_request_without_credentials_is_rejected`);
they go with the helpers, and the owned equivalents stay. The rustfs
test-count row drops from 5957 to 5955 and the s3s footprint from 68 to 66
files.

* refactor(admin): drop the legacy target descriptor JSON response helper (#8968)

`target_descriptor::build_json_response` returned the s3s response and
had no production caller left once the event, audit and plugin instance
handlers moved to `owned_build_json_response`; it survived only for the
legacy/owned equivalence test. Remove it, give the owned helper its own
documentation, and make the test assert the owned response directly:
status, `application/json`, the echoed `x-request-id` when present and
only then, and the body bytes. The file no longer imports s3s, so the
footprint baseline drops from 66 to 65 files.

* test(ecstore): pin native history on cancellation under the cancellable queue

main moved the local write path onto the disk blocking queue, which skips an
operation once its waiter is dropped, so a borrowed write cancelled while it is
queued no longer creates its file. The integration-only native publication
test still expected the queued open to run. Assert the new execution model
instead (the open never runs) while keeping its safety claims: the Started
record is taken before queueing and the cancelled publication never regrants
native history.

* chore(gateway): raise test-count rows for tests merged from main

* test(s3select): use the owned JSON input types in merged planner tests

* fix(ecstore): scope the merged direct-writer test hook to test builds

* fix(ecstore): create parents before admitted truncate opens

Align the heal tests with removal-disabled preservation and local-only outcome
fields, and refresh the e2e-full selection for the synced tests.

* chore(gateway): update test-count baseline after main sync

Main #9241 removed the node pool-metadata readiness cache with 8 tests
and added 6, so the rustfs row drops from 6440 to 6438. The ecstore and
s3select rows rise for tests added by #9237, #9239 and #9241.

* feat(gateway-adapter): add IAM-backed CredentialProvider

Add the ring-2 crate rustfs-gateway-adapter with IamCredentials, the
gateway CredentialProvider answered by RustFS IAM (rustfs/backlog#2760,
credentials half; PolicySource is deferred).

- Pin rustfs-gateway at the existing gateway rev with
  default-features = false so the server crate stays out of the build.
- IamKeyLookup is a narrow IAM trait implemented for IamSys so the
  adapter is testable without an in-memory IAM backend.
- Root credentials are injected at construction and resolve before IAM
  is ready.
- Mapping: keystone: prefix -> NotFound (deliberate change from the
  legacy empty secret); IAM not ready or not initialized -> Unavailable;
  unknown key or NoSuch{User,ServiceAccount,TempAccount,Account} ->
  NotFound; other IAM errors -> Backend; invalid or "off" principals
  resolve disabled; STS credentials carry a SessionBinding, and an STS
  credential without an expiry is a Backend fault rather than a
  never-ending session.
- No authorization in lookup; no Debug on secret-holding types; IAM
  error text is dropped after classification.

* feat(scripts): add junit_diff.py for dual-stack JUnit comparison (#9275)

* feat(gateway-adapter): add IAM-backed PolicySource

Add IamPolicySource, the gateway PolicySource answered by RustFS IAM
(rustfs/backlog#2760, policy half). One snapshot per request carries a
RustfsPrincipal: principal kind, account, effective username, parent
user, credential groups, live group memberships, token claims and
is_owner.

- Anonymous callers get an empty snapshot without asking IAM.
- IAM not ready, not initialized or failing is PolicyError::unavailable,
  never an empty snapshot.
- Unknown, disabled and session-kind-mismatched principals are refused
  with constant labels; unreadable tokens are malformed.
- Claims are read from the stored token (root secret for STS, the
  account's own secret for service accounts), and is_owner mirrors the
  legacy has_root_access rule.
- RustfsPrincipal holds no secret or session token and has no Debug.

* ci(gateway): make the test-count ratchet a floor, not exact (#9278)

ci(gateway): make the test-count ratchet a floor instead of an exact baseline

Tests below a row still fail; tests above it, a member without a row and ignores below the row now pass with a note. Ignores above a row still fail. Adds contract tests wired into make script-tests.

* chore(gateway): record ignored microbenchmarks merged from main (#9290)

chore(gateway): record the ignored microbenchmarks merged from main

main added three throughput microbenchmarks that are ignored by design
(#9217): one each in rustfs-checksums, rustfs-rio and rustfs-utils. The
test-count ratchet treats a higher ignored count as a failure until the
row is raised, so raise those three rows to 1 and tighten the test rows
to the merged tree with --update.

* fix(gateway-adapter): share live root and refuse short secrets

Address the security review of the T2.1 adapters (rustfs/backlog#2760).

- Add RootCredentials (validated: root secret and STS signing key at
  least MIN_SECRET_KEY_LEN bytes), the RootSource trait and the settable
  RootCell. IamAdapters::new is now the only constructor of
  IamCredentials and IamPolicySource, so both read one live root source
  per request; no root source answers unavailable.
- A looked-up identity whose secret is shorter than 8 bytes is NotFound
  in the credential provider and DISABLED_PRINCIPAL in the policy
  source, whatever the lookup reports as valid.
- STS claims are read with the STS signing key from the root source
  instead of assuming it equals the root secret.
- Cover the off-status-alone branch, the OIDC owner exclusion, service
  accounts of root, the name live groups are read under, and another
  session's token.

* ci(s3-tests): add reference parity lane against pinned 1.0.5 (#9288)

* fix(gateway-adapter): effective secret length and unsplittable pair

Address the confirmation review of the T2.1 adapters
(rustfs/backlog#2760).

- One rule for every secret: its length after trailing NUL bytes are
  removed must be at least MIN_SECRET_KEY_LEN. HMAC zero-pads its key,
  so trailing NULs add nothing; interior NULs count. Applied to the root
  secret, the STS signing key and every looked-up credential.
- IamAdapters keeps both halves private and hands them to a builder
  together through the consuming install(builder, make_authenticator);
  compile-fail doctests show a mixed pair cannot be assembled.
- Any Fn() -> Option<Arc<RootCredentials>> is a RootSource, so the
  binary reads its live root accessor directly; RootCell is now a
  test-only helper.

* docs(gateway-adapter): state what install guarantees about the provider handle

---------

Co-authored-by: rustfs-cla-assistant[bot] <273121464+rustfs-cla-assistant[bot]@users.noreply.github.com>
2026-11-11 04:29:40 -05:00
Chris An 04bcb57e6a fix(ecstore): pin multipart index markers while their root is published (#9318)
A multipart index marker is recorded before its staging root is published, and a listing prunes markers whose root is gone. The window in between was covered by a five-minute age check, which cannot prove that the publisher's filesystem work has ended: a publisher stalled past it lost its marker, and the upload it then published stayed out of ListMultipartUploads on that drive until restart.

The publisher now holds an in-memory publication pin from before it records the marker until the filesystem work that publishes the root has returned. The pin travels with that work (the blocking closure or the lease owner chain), so it also covers work that outlives a cancelled caller. Pruning reads the pin first and the staging root second, under the marker lease. The age check and the per-write marker touch are removed.
2026-11-11 03:22:46 -05:00
Chris An 31e1e42ad7 refactor: land the s3s decoupling groundwork on main (#9276)
Lands gateway/integration on main: the guard rails and the domain
decoupling for replacing s3s with rustfs/gateway (rustfs/backlog#2734,
Phase 0 and Phase 1). The S3 edge is still s3s and no wire behaviour
change is intended.

- Guard rails: lower-only s3s footprint baseline and edge allow-list,
  per-crate test-count floor, 39 byte-level goldens for the persisted
  bucket-config XML families, a two-server differential harness over
  all 73 S3 operations, a JUnit differ, and a reference-parity lane
  against the sha256-pinned 1.0.5 binary.
- Owned types: rustfs-s3-types (S3Error, S3ErrorCode, Body),
  RequestEnvelope, AuthzFacts.
- Engine crates off s3s: ecstore, lifecycle, replication, scanner,
  protocols, s3select, s3-client, rpc.
- Admin: error types moved; 300 of 314 Operation impls take the owned
  AdminRequest.
- Server: the s3s-compat tower layers grouped under
  server/legacy_compat; RUSTFS_S3_STACK=legacy|gateway selector, where
  gateway refuses to start until the gateway edge exists.

Fixes made on the branch that change ecstore and heal behaviour are
listed in the pull request description. The per-commit history stays
on gateway/integration.
2026-11-11 02:39:52 -05:00
Chris An 0e402b2534 fix(scanner): retain child objects beneath object prefixes (#9277) 2026-11-11 14:36:53 +08:00
Chris An db39cd5634 fix(ecstore): let missing-key deletes complete on UUID-named folders (#9280)
* fix(ecstore): let missing-key deletes complete on UUID-named folders

Deleting a key that does not exist probes the key's directory for
interrupted delete transactions. The probe was bounded at 4096 entries
per drive and charged every entry inside a UUID-named child, so a key
naming a folder of roughly 1,400 or more UUID-named child keys answered
503 SlowDown on every retry.

Inside a UUID-named child only journal files and data-directory markers
can be evidence, so every other name (a child key's own xl.meta, data
directories or children) is now skipped unread at 1/64 like plain
children of the folder. The per-call allowance of the bounded protocol
rises to 262,144 entries per drive; the resumable background step keeps
4096 and the request budget keeps its sizing, so the per-request total
is unchanged. The probe runs on the durable-mutation lane instead of
the one-worker foreground lane, and stops incomplete when its caller
goes away or after a third of the drive timeout (at most 10 s), so a
slow walk answers SlowDown instead of timing out the drive.

* test(ecstore): keep discovery fixtures meaningful under 1/64 charging

Part files inside a data directory are inert to object-scoped discovery
now, so reclaim_orphan_data_dirs_drains_empty_discovery_pages fills its
first step with the referenced data directories of many versions, which
still cost a full entry each. The charging test drives the walk directly
instead of through the wall-clock-limited disk entry point, so a loaded
machine cannot turn it into an incomplete probe.
2026-11-10 23:24:50 -05:00
Ramakrishna Chilaka 89d01aef4f perf(ecstore): reduce syscalls and directory re-opens in PUT and DELETE (#9264)
Co-authored-by: RustFS <hello@rustfs.com>
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
2026-11-11 00:06:11 +00:00
dependabot[bot] 86c3395d07 chore(deps): bump the dependencies group across 1 directory with 12 updates (#9251)
Bumps the dependencies group with 12 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [async-nats](https://github.com/nats-io/nats.rs) | `0.51.0` | `0.52.0` |
| [tar-codec](https://github.com/astral-sh/astral-codec) | `0.0.15` | `0.0.16` |
| [tar-framing](https://github.com/astral-sh/astral-codec) | `0.0.15` | `0.0.16` |
| [flate2](https://github.com/rust-lang/flate2-rs) | `1.1.10` | `1.1.11` |
| [smallvec](https://github.com/servo/rust-smallvec) | `1.16.4` | `1.16.5` |
| [starshard](https://github.com/houseme/starshard) | `2.4.0` | `2.4.1` |
| [zip](https://github.com/zip-rs/zip2) | `9.3.0` | `9.4.1` |
| [dial9](https://github.com/dial9-rs/dial9) | `0.5.7` | `0.5.8` |
| [rustfs-mimalloc](https://github.com/houseme/rustfs-mimalloc) | `0.7.3` | `0.7.4` |
| [hotpath](https://github.com/hotpath-rs/hotpath) | `0.30.3` | `0.31.1` |
| [rustfs-mimalloc-sys](https://github.com/houseme/rustfs-mimalloc) | `0.7.3` | `0.7.4` |
| [toml](https://github.com/toml-rs/toml) | `1.2.0+spec-1.1.0` | `1.2.1+spec-1.1.0` |



Updates `async-nats` from 0.51.0 to 0.52.0
- [Release notes](https://github.com/nats-io/nats.rs/releases)
- [Commits](https://github.com/nats-io/nats.rs/compare/async-nats/v0.51.0...async-nats/v0.52.0)

Updates `tar-codec` from 0.0.15 to 0.0.16
- [Release notes](https://github.com/astral-sh/astral-codec/releases)
- [Commits](https://github.com/astral-sh/astral-codec/compare/v0.0.15...v0.0.16)

Updates `tar-framing` from 0.0.15 to 0.0.16
- [Release notes](https://github.com/astral-sh/astral-codec/releases)
- [Commits](https://github.com/astral-sh/astral-codec/compare/v0.0.15...v0.0.16)

Updates `flate2` from 1.1.10 to 1.1.11
- [Release notes](https://github.com/rust-lang/flate2-rs/releases)
- [Changelog](https://github.com/rust-lang/flate2-rs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/flate2-rs/compare/1.1.10...v1.1.11)

Updates `smallvec` from 1.16.4 to 1.16.5
- [Release notes](https://github.com/servo/rust-smallvec/releases)
- [Commits](https://github.com/servo/rust-smallvec/compare/v1.16.4...v1.16.5)

Updates `starshard` from 2.4.0 to 2.4.1
- [Changelog](https://github.com/houseme/starshard/blob/main/CHANGELOG.md)
- [Commits](https://github.com/houseme/starshard/compare/2.4.0...2.4.1)

Updates `zip` from 9.3.0 to 9.4.1
- [Release notes](https://github.com/zip-rs/zip2/releases)
- [Changelog](https://github.com/zip-rs/zip2/blob/master/CHANGELOG.md)
- [Commits](https://github.com/zip-rs/zip2/compare/v9.3.0...v9.4.1)

Updates `dial9` from 0.5.7 to 0.5.8
- [Release notes](https://github.com/dial9-rs/dial9/releases)
- [Changelog](https://github.com/dial9-rs/dial9/blob/main/CHANGELOG.md)
- [Commits](https://github.com/dial9-rs/dial9/compare/dial9-v0.5.7...dial9-v0.5.8)

Updates `rustfs-mimalloc` from 0.7.3 to 0.7.4
- [Release notes](https://github.com/houseme/rustfs-mimalloc/releases)
- [Changelog](https://github.com/houseme/rustfs-mimalloc/blob/main/CHANGELOG.md)
- [Commits](https://github.com/houseme/rustfs-mimalloc/compare/v0.7.3...v0.7.4)

Updates `hotpath` from 0.30.3 to 0.31.1
- [Release notes](https://github.com/hotpath-rs/hotpath/releases)
- [Changelog](https://github.com/hotpath-rs/hotpath/blob/main/CHANGELOG.md)
- [Commits](https://github.com/hotpath-rs/hotpath/compare/v0.30.3...v0.31.1)

Updates `rustfs-mimalloc-sys` from 0.7.3 to 0.7.4
- [Release notes](https://github.com/houseme/rustfs-mimalloc/releases)
- [Changelog](https://github.com/houseme/rustfs-mimalloc/blob/main/CHANGELOG.md)
- [Commits](https://github.com/houseme/rustfs-mimalloc/compare/v0.7.3...v0.7.4)

Updates `toml` from 1.2.0+spec-1.1.0 to 1.2.1+spec-1.1.0
- [Commits](https://github.com/toml-rs/toml/compare/toml-v1.2.0...toml-v1.2.1)

---
updated-dependencies:
- dependency-name: async-nats
  dependency-version: 0.52.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
- dependency-name: tar-codec
  dependency-version: 0.0.16
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: tar-framing
  dependency-version: 0.0.16
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: flate2
  dependency-version: 1.1.11
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: smallvec
  dependency-version: 1.16.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: starshard
  dependency-version: 2.4.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: zip
  dependency-version: 9.4.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
- dependency-name: dial9
  dependency-version: 0.5.8
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: rustfs-mimalloc
  dependency-version: 0.7.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: hotpath
  dependency-version: 0.31.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
- dependency-name: rustfs-mimalloc-sys
  dependency-version: 0.7.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: toml
  dependency-version: 1.2.1+spec-1.1.0
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
2026-11-11 00:04:47 +00:00
cxymds c9743f166e fix(s3): normalize leading-slash keys in bulk deletes (#9258)
* fix(s3): normalize leading-slash keys in bulk deletes

* test(s3): align bulk delete reserved-prefix guard expectations
2026-11-10 17:56:00 -05:00
Boyi (Elliot) Li 50707a610c test(utils,rio,checksums): add checksum SIMD equivalence and bench UTs (#9217)
Add unaligned-boundary SIMD vs scalar equivalence unit tests and on-demand (#[ignore]) utility-level throughput microbenchmarks across rustfs-utils, rustfs-rio, and rustfs-checksums:
- crates/utils/src/hash.rs: verify HighwayHasher SIMD (AVX2/SSE4.1/NEON) against highway::PortableHash scalar reference and contiguous hash_encode vs odd-chunked hash_encode_slices across unaligned tail boundaries; provide on-demand throughput microbenchmark.
- crates/rio/src/checksum.rs: verify SIMD-accelerated CRC32, CRC32C, and CRC64_NVME against bitwise scalar reference implementations across vector tail boundaries and measure utility-level throughput across all 10 BASE_CHECKSUM_TYPES.
- crates/checksums/src/lib.rs: verify streaming chunked update + finalize + headers() against one-shot digests across all 9 ChecksumAlgorithm variants for unaligned chunk and payload sizes, plus on-demand throughput microbenchmark.
2026-11-10 17:54:11 -05:00
Patrick Wicki ba3e0e1546 fix(admin): keep measured bucket usage when the overlay is unconverged (#9243)
The single-process usage overlay clears usage_snapshot_complete when it
holds counts it cannot reconcile with the persisted snapshot, for
example writes into a bucket that has no verified baseline. The admin
datausageinfo path then judged the overlaid document as structurally
incomplete and replaced it with an empty one, hiding every bucket the
scanner had measured. Because each write restarts that bucket's scanner
fence, a continuously written bucket kept the whole response empty
indefinitely.

Decide whether the persisted snapshot is usable before the overlay is
applied, and report overlay-induced incompleteness as a non-converged
snapshot instead, as is already done for buckets the scan never reached
(rustfs#5806).
2026-11-10 17:52:06 -05:00
Chris An 95e128336d ci: narrow checks for stable installation metadata (#9269)
* ci: narrow checks for stable installation metadata

* fix: reject release metadata marker collisions
2026-11-10 16:10:04 -05:00
Chris An 8b7bbdb17d chore(release): align installation references for 1.0.5 (#9268) 2026-11-10 14:48:02 -05:00
Chris An 2780bba67b ci: stage release builds and promote verified artifacts (#9265)
* ci: stage release builds and promote verified artifacts

* fix: preserve release package and channel retry contracts

* fix: align installation chart versions with sealed candidates
2026-11-10 13:15:05 -05:00
hector 3a23122104 ci: run macOS release builds on macos-latest-xlarge (#9260)
Use the larger GitHub-hosted macOS runner for the macos-aarch64
release target. The target id, triple, and other platforms stay the same.

Co-authored-by: hector <majinghe@users.noreply.github.com>
2026-11-10 18:55:19 +08:00
Ramakrishna Chilaka 0b1ebeffec perf(ecstore): reduce syscalls in multipart copy, commit, and delete (#9247) 2026-11-10 12:27:10 +08:00
1168 changed files with 172026 additions and 43965 deletions
+21 -26
View File
@@ -1,43 +1,38 @@
---
name: rustfs-release-publish
description: Publish a RustFS version through Console, release-branch CI, preview acceptance, human confirmation, and final publication. Use only for an explicit release request.
description: Publish a RustFS version through frozen Linux candidate acceptance, desktop completion, manifest approval, byte-preserving publication and post-release follow-up. Use only for an explicit release request.
---
# RustFS Release
Require the exact final SemVer target before version edits or publication. If unspecified, inspect current releases/tags and ask with concrete candidates; do not guess the channel or bump. A named bump type resolves to an exact version that still needs confirmation. Report the confirmed target verbatim; resolve any later mismatch before proceeding. Independent read-only preflight may continue.
Require the exact final SemVer target before version edits or publication. If unspecified, inspect releases/tags and ask with concrete candidates; do not guess the channel or bump. Reuse the user's confirmed target and authorized delivery scope. Independent read-only preflight may continue.
## Invariants
- Cargo stores `<target>`, never `-preview.N`. Before publication, installation defaults retain the previous available deliverable. Final and preview binaries embed their own tag names, so version/asset names differ despite identical source.
- Use annotated tags without `v`. Preview names are exactly `<target>-preview.<digits>`; choose the next unused iteration after fetching tags. Preview Releases are prereleases, never Latest; publish only versioned assets/exact Docker tags, never latest assets, R2, Helm, moving Docker channels, or installation/banner updates.
- `release/<target>` owns the candidate. Retain its history, backport only reviewed fixes, and never merge main wholesale or force-push it. Pin `PREVIEW_HASH` to its exact commit with complete passing CI. Preview and final tags both point there; never tag a later main/branch HEAD or insert another bump.
- Phase failures block dependents. Keep preview Releases until final publication and notes generation finish; CI removes Releases while retaining tags. Never move an existing preview or final tag.
- Cargo and every candidate executable contain the final product version. `<target>-preview.<run_number>` is an immutable candidate identifier, not a public tag or displayed binary version.
- Freeze the exact source, reviewed workflow recipe, toolchain, Console, CLI, dependency lock, previous stable version and validation plan. Newer unrelated main or Console releases do not change those inputs.
- Retain `release/<target>` history when a release branch is used; backport only reviewed fixes. Pin `CANDIDATE_SHA` to its exact CI-validated commit. Never force-push a release branch or move a public tag.
- Build Linux first. Complete field acceptance before macOS/Windows and packaging. Full certification and final acceptance are required before publication; a partial Linux PASS grants no release authority.
- Promote the accepted bytes. Final publication never recompiles binaries or rebuilds packages/images. Infrastructure retries reuse successful artifacts; changed source or inputs require a new candidate and affected acceptance.
- Phase failures block dependents. No latest/default-channel update before versioned delivery. Use the [pipeline runbook](../../../docs/operations/release-pipeline.md) for executable entry points, locks and recovery.
## Read by Phase
Read only the current phase's procedure; retain completed evidence across turns. Follow root verification, Git/PR, and authorization rules.
## Read by phase
| Phase | Procedure |
|---|---|
| 0 | Clean tree, fetch main/tags, verify GitHub access, then complete the mandatory [Console release gate](references/console-gate.md). A green Console build alone is insufficient. |
| 1–2 | [Prepare/reuse the release candidate and run exact-SHA branch CI](references/release-candidate.md). Source version changes use `rustfs-release-version-bump` with authorized delivery scope. |
| 3 | [Verify the preview build, assets, channels, and notes](references/preview-release.md). |
| 4–5 | [Run the downloaded binary, Console CRUD, and latest-rc matrix](references/preview-acceptance.md). Every check must pass. |
| Confirmation | Complete the gate below and end the turn. |
| 6 | After fresh confirmation, [publish and verify the final tag](references/final-release.md) at `PREVIEW_HASH`. |
| 7 | [Reconcile Security Advisories, milestones, installation references, and the website; reconcile main and remove the completed release branch](references/post-release-updates.md) only after complete publication. |
| 0 | Clean tree, fetch main/tags, verify access, then complete the mandatory [Console gate](references/console-gate.md). |
| 1–2 | [Prepare the exact source and run branch CI](references/release-candidate.md). Version edits use `rustfs-release-version-bump`. |
| 3 | [Build and verify the Linux candidate](references/preview-release.md). |
| 4–5 | [Run artifact, field, Console and rc acceptance](references/preview-acceptance.md), then complete desktop/package/image/chart validation. |
| Confirmation | Report the complete manifest and obtain the confirmation below. |
| 6 | [Publish the accepted manifest](references/final-release.md). |
| 7 | [Reconcile advisories, milestones, installation references, website and release branch](references/post-release-updates.md). |
## Human Confirmation
## Human confirmation
After Phases 3–5 pass, recheck that the release branch still equals `PREVIEW_HASH` and that the current Console asset tag/digest match the preview's recorded build inputs. Report the target, branch, preview tag/hash, branch CI run, preview Release URL, accepted Console asset identity/result, and rc matrix. Explicitly ask whether to publish the final tag and end the turn without creating/pushing it.
After all acceptance passes, report the target, source SHA, source CI run, Linux and completion run URLs, final manifest SHA-256, accepted Console/CLI identities, compatibility/performance results and backlog report. Explicitly ask whether to publish this exact manifest and end the turn without dispatching publication.
Only a new affirmative user reply to that exact report authorizes Phase 6. The original release request, earlier confirmation, silence, or automated follow-up does not count. Confirmation binds the target, preview tag, hash, and accepted build inputs; a changed value or repeated/failed acceptance cycle invalidates it. Revalidate and obtain fresh confirmation before continuing.
Only a new affirmative reply to that exact report authorizes Phase 6. An earlier general release request, silence or automatic continuation does not count. Reuse confirmation for retries of that same manifest; do not ask again merely because a destination failed. A changed source, input, manifest or acceptance result invalidates it and requires renewed acceptance/confirmation.
## Recovery
## Recovery and reporting
- Before the final tag exists, land fixes on main, backport to the candidate, verify target versions, and restart branch CI/acceptance. Use a new preview iteration once a preview was tagged; main may advance independently.
- If only an external build input changed, record it, create a new preview iteration even when the source SHA is unchanged, and repeat the affected build and artifact acceptance before fresh confirmation. Reuse source checks whose inputs remain valid.
- Once the final tag exists, retry publication jobs for that exact tag. A source fix requires a newly confirmed target, not retagging or a newer preview. Phase 7 retries recheck artifact availability and superseding releases.
- If abandoned after the bump merged, report the unpublished version on main and record whether to revert or let the next release overwrite it. Deleting unpublished candidate branches or preview tags requires an explicit decision. Completed release branches follow Phase 7 cleanup; keep release tags.
Report completed phase results and incomplete work with evidence: Console previous/latest tags, release need, hash and run/Release URLs; target, branch CI/SHA, previews and hash, confirmation state; preview/final assets, latest-channel state, Console/rc results and cleanup; advisory reconciliation results and pending disclosure decisions; both milestone states, installation PR checks/merge state, and website text/link/PR/deployment state. Keep artifact publication separate from follow-up delivery; do not call an open PR a live update. Report deviations and their authorization.
Resume failed infrastructure jobs without rebuilding successful units. Fix product defects on main, backport when needed, and start a new candidate. After a final tag exists, retry the same bytes or use a new patch version; never move the tag. Do not conflate implemented, CI-passed, field-accepted, approved, published and post-release-complete states. Report incomplete work and deviations with evidence; a tag or green build alone is not a completed release.
@@ -4,7 +4,7 @@ Read during Phase 0, before changing RustFS version files or tags. Follow the pa
### Console release gate
Complete this gate before changing any RustFS version file or creating any RustFS tag. RustFS `build.yml` downloads the asset returned by `repos/rustfs/console/releases/latest`, so a successful Console build alone is insufficient.
Complete this gate before changing any RustFS version file or creating any RustFS tag. Candidate creation resolves `repos/rustfs/console/releases/latest` once and freezes the exact asset for every build, so a successful Console build alone is insufficient. After freezing, later Console releases do not silently change or invalidate the accepted inputs.
1. Read the latest published Console tag and compare it with Console `main`:
@@ -1,22 +1,13 @@
# Final Release Publication
# Final release publication
Read only after fresh user confirmation of the preview acceptance report. The [parent confirmation gate](../SKILL.md#human-confirmation) applies.
Read only after fresh confirmation of the complete candidate report. The [parent confirmation gate](../SKILL.md#human-confirmation) applies.
## Phase 6 — Publish the final tag on the validated commit
Dispatch `release-publish.yml` on main with the successful completion run ID and approved `release-manifest.json` SHA-256. Do not create the tag manually. The workflow revalidates source, bytes and trusted certification, then creates the annotated final tag at `CANDIDATE_SHA`. A recipe change fails closed and requires a new candidate; unrelated main changes do not require a rebuild.
No second source version bump. Before creating the final tag, recheck the remote `release/<target>` HEAD equals `PREVIEW_HASH` and the current Console release tag/asset digest still match the accepted preview inputs. If either changed after confirmation, the approval no longer applies; validate the changed inputs with a new preview and obtain fresh confirmation. The final tag goes on the exact commit the preview validated:
Require both versioned publication and channel publication jobs to succeed. Verify the final tag's commit and manifest identity; all six platform ZIPs and compatibility aliases; DEB/RPM, checksums, SBOM, provenance and evidence bundle; both multi-architecture image variants in Docker Hub, GHCR and Quay; and the accepted Helm chart in the chart repository. Download a host-compatible final artifact and confirm its digest is exactly the accepted candidate digest and its version/source output agrees.
```bash
git fetch origin --tags
git rev-parse "<preview-tag>^{commit}" # must equal PREVIEW_HASH — abort if not
git tag -a "<target>" -m "Release <target>" "$PREVIEW_HASH"
git push origin "<target>"
```
For stable releases verify GitHub Latest, R2 download aliases and `latest.json` unless a newer stable release already owns those channels. Prereleases must remain GitHub prereleases and leave stable pointers unchanged. Verify generated release notes compare against the frozen previous stable release, excluding internal previews.
- CI rebuilds from the same source with a different tag name, so the binary self-reports `<target>`. Other inputs can change: `build.yml` reads Console `releases/latest` during each build. Compare the actual Console tag, URL, and verified digest in every final build job with the accepted preview record, and inspect available build provenance for other relevant changes. A pre-tag check cannot prevent a later input change. On a mismatch or missing evidence, repeat affected acceptance against the final artifacts and report the limitation; failed acceptance leaves release validation incomplete even if publication already succeeded. Never move the final tag to conceal a failure.
- Verify the final tag's complete publication path: all matrix and release jobs green; `gh release view "<target>"` shows the full versioned and `-latest` asset set plus checksums, SBOM, and provenance; Docker and Helm workflows succeed; `latest.json` points to `<target>`. A stable target must have `isPrerelease=false` and `isLatest=true`. An alpha/beta/rc target must have `isPrerelease=true`; GitHub does not permit prereleases to be Latest, but the project `latest.json` still advances to the final non-preview target.
- Verify the final Release body contains `## What's Changed` and a Full Changelog link. When `PREVIOUS_DELIVERABLE` exists, the link MUST be `https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<target>` and the baseline MUST equal the preview Release baseline; for example, both `1.0.0-beta.12-preview.1` and `1.0.0-beta.12` compare from `1.0.0-beta.11`.
- Verify the preview cleanup: `cleanup-preview-releases` must succeed and `gh release view "<preview-tag>"` must then report `release not found` for every preview iteration of this target. Each surviving preview tag must still resolve to its own recorded commit; the accepted preview resolves to `PREVIEW_HASH`. If the job failed, delete the leftover Releases manually with `gh release delete "<preview-tag>" --yes` and report it.
- Verify a host-compatible final artifact's checksum and `./rustfs --version`; it must report `<target>` and the expected source SHA. This identity check does not replace affected acceptance when build inputs changed.
On infrastructure failure retry the same workflow/seal; matching destinations are reused. Never clobber an asset, force a tag, rebuild the image or silently accept a changed digest. A source fix after the final tag requires a new patch version. Cross-registry writes are not atomic: report the unfinished destination rather than claiming publication complete.
Never pass `--cleanup-tag` when removing preview Releases. Remove them manually only if `cleanup-preview-releases` failed, after final publication and notes generation. Record deleted Releases and surviving tags.
New candidates create no preview Releases to clean up. For a legacy preview release, preserve its tag and remove its Release only after final delivery and notes are verified. Do not pass `--cleanup-tag`. Complete the [post-release follow-up](post-release-updates.md) separately.
@@ -24,7 +24,7 @@ Reconcile advisories after every completed final release, including Console-only
1. Re-read the published deliverable state before edits or a retry. If a newer release has superseded this target, do not downgrade installation references or the website announcement. Report the superseding release and leave those defaults intact.
2. Invoke `rustfs-release-version-bump` with stage `post-release`, the exact target, and the authorized delivery scope. Prepare the follow-up on current main; do not edit or move either validated release tag. Cargo versions may already be preparing the next target and must remain untouched.
3. Verify the exact Docker image manifest, Release download links, RPM source archive, and Helm repository entry before publishing the installation PR. The chart workflow overrides chart/app versions from the final tag, pins the source checkout to that release, and waits for its default Docker image before packaging.
3. Verify the exact Docker image manifest, Release download links, RPM source archive, and Helm repository entry before publishing the installation PR. The chart is packaged and tested before sealing; publication uploads those same bytes only after versioned Docker images are available.
4. Report the installation PR and its merge state separately from the completed release. Reuse existing merge authorization and repository checks; do not infer new merge authority from this reference.
## rustfs.com top banner
@@ -54,4 +54,4 @@ Use the existing announcement configuration from [website PR #112](https://githu
1. Compare the exact published tag with current main. Audit every release-side commit since their merge base, including inherited backports. An ahead count is a history difference, not proof of missing changes. Use ancestry and patch equivalence, then inspect current code and regression evidence for squashed, adapted, superseded, or reverted changes. Record each commit as present, superseded, release-only, or missing; a positive `git cherry` result alone does not prove a missing fix.
2. Bring missing shared changes to main through a reviewed PR and the required checks before closing the follow-up. Preserve main's later changes; decide version bumps and pinned Console configuration separately. If a history merge is needed, use a merge commit and verify the release commit is an ancestor of main; squash does not establish that ancestry. Do not merge main into the published release branch or move its tags. Report unresolved changes instead of claiming reconciliation is complete.
3. After publication and authorized follow-ups are complete, delete the remote `release/<target>` branch. Immediately before deletion, verify its HEAD still equals the published tag's commit and no open PR targets or originates from it. Preserve a branch with newer work or open PRs and report the blocker. Use a deletion with an expected-old-SHA check to avoid removing a concurrently advanced branch.
4. Verify the remote branch is absent and the final and preview tags still resolve to their recorded commits. Remove a local release branch only if it is task-owned, unchecked out, and has no unpreserved work. Record cleanup separately from artifact publication; an explicitly authorized deletion before reconciliation does not make unresolved main changes complete.
4. Verify the remote branch is absent and the final tag and any legacy preview tags still resolve to their recorded commits. Remove a local release branch only if it is task-owned, unchecked out, and has no unpreserved work. Record cleanup separately from artifact publication; an explicitly authorized deletion before reconciliation does not make unresolved main changes complete.
@@ -1,10 +1,10 @@
# Preview Artifact Acceptance
# Candidate Artifact Acceptance
Read after Phase 3 succeeds. Complete every check below before the parent skill's manual confirmation gate. These checks cover the downloaded artifact, embedded Console, and latest rc client.
Read after Phase 3 succeeds. Complete every check below before the parent skill's manual confirmation gate. These checks cover the downloaded artifact, embedded Console, and the rc release frozen in the candidate inputs.
## Phase 4 — Run the artifact locally, verify the console
Use a unique task-owned scratch directory. Select the versioned archive matching the host OS, architecture, and runtime, download its checksum file, and verify the archive before extraction. Do not substitute a different build when a matching artifact is unavailable; report that acceptance gap. The shell example below assumes the verified executable is at `RUSTFS_BINARY` (`rustfs.exe` on Windows), `SCRATCH` is the task directory, and `S3_PORT` and `CONSOLE_PORT` are distinct available ports. Use equivalent native commands when the shell differs.
Use a unique task-owned scratch directory. Select the versioned archive matching the acceptance host, and verify it against the sealed manifest before extraction. Before desktop completion, use the Linux acceptance environment rather than rebuilding for the local desktop. Do not substitute a different build when a matching artifact is unavailable; report that acceptance gap. The shell example below assumes the verified executable is at `RUSTFS_BINARY` (`rustfs.exe` on Windows), `SCRATCH` is the task directory, and `S3_PORT` and `CONSOLE_PORT` are distinct available ports. Use equivalent native commands when the shell differs.
Bind both listeners to loopback. Choose task-only credentials, set loopback proxy exclusions for the server/client processes, and record the server PID. A port probe does not reserve a port; retry startup with fresh ports on a bind conflict rather than connecting to an existing service.
@@ -24,16 +24,16 @@ PREVIEW_SERVER_PID=$!
Checks (all must pass):
- `"$RUSTFS_BINARY" --version` reports the preview tag name and the short SHA of `PREVIEW_HASH`. Reporting `<target>` without the `-preview.N` suffix means the build did not embed the tag — treat as FAIL and investigate before proceeding.
- `"$RUSTFS_BINARY" --version` reports the exact final `<target>` and full `CANDIDATE_SHA`. A preview suffix, another version or another source is a failure.
- Wait with a bounded readiness deadline while checking the recorded server process remains alive; `curl --noproxy '*' -fsS "$S3_ENDPOINT/health/ready"` must return ready from this process's listener.
- Startup log shows the embedded console being served (this was the regression that `fix(release): require embedded console assets` guards).
- Open `CONSOLE_ENDPOINT` in the browser and log in with the task credentials. The dashboard must render without JS console errors; create a bucket, upload a file, download it back (byte-identical), and delete the object and bucket. Keep the server running for Phase 5.
## Phase 5 — Validate with the latest rc client
## Phase 5 — Validate with the pinned rc client
`rc` is the RustFS CLI client from <https://github.com/rustfs/cli>.
- Resolve the latest release with `gh api repos/rustfs/cli/releases/latest --jq .tag_name`; verify the local client's version or download and verify a host-compatible client into the task directory. Set `RC_BIN` to that executable and record its version/digest. Avoid changing the user's installed client for this check.
- Read the CLI release and asset digests from `inputs.json`; verify the local client's version or download a host-compatible asset from that exact release into the task directory. If the host is a desktop, resolve its asset within the same pinned CLI release and record the digest; do not independently follow `latest`. Set `RC_BIN` to that executable and record its version/digest. Avoid changing the user's installed client for this check.
- Inspect the client's current help for an isolated configuration option and use it when supported. Otherwise choose a unique `PREVIEW_ALIAS` and confirm it is absent before creating it; never overwrite an existing alias. Use task-owned bucket/user names and input/output files. The names below are safe only inside the newly started isolated server.
- Point the client at `S3_ENDPOINT` with loopback proxy exclusions and run the command matrix, recording PASS/FAIL per command:
@@ -60,3 +60,9 @@ cmp <local-file> "$SCRATCH/out"
```
- Any FAIL blocks acceptance. On success or failure, stop only the recorded task process and remove only task-created aliases/data after retaining the needed failure evidence. Do not erase an existing client configuration or unrelated server data.
## Field acceptance and desktop completion
Before accepting the Linux manifest, record Scanner/Heal, multi-pool, memory, major features, high-concurrency 503, performance and previous-version integrity/upgrade results in backlog. Preserve the release-specific agreed limits and baseline; a PASS label without artifact identities or test results does not satisfy acceptance.
Dispatch `release-complete.yml` only after that report passes, using the exact Linux manifest digest and backlog URL. Require native macOS/Windows checks, package extraction, image startup/scans, chart tests and complete certification to pass. Complete any desktop Console/client checks applicable to the release against these same archives. Record the completion run and final manifest SHA-256 before entering the parent confirmation gate. Partial Linux acceptance does not authorize final publication.
@@ -1,25 +1,9 @@
# Preview Release Verification
# Linux candidate verification
Read for Phase 3 after the candidate passed exact-SHA CI. The [parent release invariants](../SKILL.md) apply.
Read for Phase 3 after the source passed exact-SHA CI. The [parent invariants](../SKILL.md) apply; commands and recovery live in the [pipeline runbook](../../../../docs/operations/release-pipeline.md).
## Preview channels
Dispatch `release-candidate.yml` on main with the confirmed final version and `CANDIDATE_SHA`. Record the run ID, frozen `inputs.json`, every Linux archive digest and `linux-manifest.json` SHA-256. Require the four Linux builds, native startup/S3 checks, previous-release upgrade matrix and Linux seal to succeed. Missing, skipped, cancelled or failed required work is incomplete acceptance.
- Use `<target>-preview.N` for every target, e.g. `1.0.0-beta.10-preview.3` or `1.1.0-preview.1`.
- The canonical suffix is exactly `-preview.<digits>`. `build.yml` recognizes it before alpha/beta/rc classification and routes it to the preview-only path; any other tag containing `-preview` fails closed instead of being treated as a release.
- A preview Release MUST be published with `isPrerelease=true` and `isLatest=false`. Any `*-latest` preview asset or preview-triggered `latest.json`, R2, Helm, or moving Docker channel tag is a pipeline failure. Docker images may use the exact preview version tag, including its existing variant suffixes.
- Preview Releases are cleaned up by the `cleanup-preview-releases` job after `publish-release` succeeds for the deliverable tag. It deletes every Release whose tag is exactly `<target>-preview.<digits>` and never passes `--cleanup-tag`, so the tags survive.
Verify the frozen Console identity equals the one accepted in Phase 0. Every matrix job consumes that same digest and exact toolchain. The binary must report the final product version and source commit; its candidate ID is separate. No public tag, Release, registry tag, R2 release path, Helm repository or latest pointer should change during this stage.
## Phase 3 — CI and preview Release verification
- Find and watch the tag build: `gh run list --workflow build.yml --branch "<preview-tag>" --limit 1` then `gh run watch <run-id>`. Every build matrix target must succeed (linux x86_64/aarch64 × musl/gnu, macos-aarch64, windows-x86_64).
- Confirm the preview tag resolves to the Phase 2 CI-validated `PREVIEW_HASH`. If the release branch advances during acceptance, repeat Phases 2–5 with a new preview tag.
- Confirm the Release publication jobs (`create-release`, `upload-release-assets`, and `publish-release`) succeed while `update-latest-version` is skipped.
- Record the inputs actually consumed by each build job: source SHA and the embedded Console release tag, asset URL, and verified SHA256 from the download logs. The workflow reads Console `releases/latest` separately for each build; confirm the matrix used the Console asset accepted in Phase 0. A missing identity or differing asset blocks acceptance until resolved. Retain available toolchain and build provenance with the artifact checksums; the same source SHA alone does not establish equivalent builds.
- Verify `gh release view "<preview-tag>" --json isPrerelease,assets,url`: `isPrerelease` must be `true`, and the Release must contain all 6 versioned platform zips, checksums, SBOM, and provenance with no `-latest` assets. Confirm `gh api repos/{owner}/{repo}/releases/latest --jq .tag_name` does not return `<preview-tag>`.
- Record `PREVIOUS_DELIVERABLE`, selected from published Releases by `publishedAt` after excluding the current tag and every `-preview.N` tag. Verify `gh release view "<preview-tag>" --json body --jq .body` contains `## What's Changed` and, when `PREVIOUS_DELIVERABLE` exists, `**Full Changelog**: https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<preview-tag>`. For a repository with no previous deliverable, verify a Full Changelog link exists and record the GitHub baseline fallback.
- Confirm Helm is skipped. If preview Docker images are published, confirm they use only exact preview tags (and variant suffixes), with no `latest`, `alpha`, `beta`, or `rc` channel updates. Preview validation covers the built RustFS binaries, embedded console, and rc compatibility; it does not authorize advancing any installation default.
## Release notes
- When a previous deliverable exists, GitHub Release notes for the preview and final tags MUST use it as their shared comparison baseline: the most recently published non-preview Release before the target. Internal `-preview.N` Releases are explicitly excluded from that selection, even when they point at the same commit as the final tag — cleanup runs after the notes are generated, so the preview Release is still present and would otherwise be picked as the baseline. If no previous deliverable exists, omit `previous_tag_name` and record that GitHub's default baseline fallback was used.
- Generated Release notes carry a workflow-management marker so retries can repair them. Before manually curating a generated body, remove that marker; unmarked non-placeholder notes are preserved by later workflow runs.
A green Linux run is ready for field acceptance, not publication. The partial certification must have `publish_allowed: false`. Retain the exact Linux manifest for the next stage; a new source/build-input candidate must not inherit the old approval. A newer unrelated main commit does not invalidate this evidence.
@@ -16,22 +16,13 @@ Read for Phases 1–2. The [parent release invariants](../SKILL.md) and Console
- Only when neither the release branch, an unfinished preview tag, nor current main already has `<target>`: if main is at a later version, select and verify a historical commit containing `<target>` rather than downgrading main; block if none exists. Otherwise invoke the `rustfs-release-version-bump` skill with stage `prepare`, the final `<target>` (NOT a preview version), and full GitHub flow (commit/push/PR).
- If a bump PR was needed, get it merged into main and record its merge commit. For a new release branch when main already has `<target>`, identify the main commit that introduced those versions. Phase 2 chooses a release candidate containing that commit; an existing release branch keeps its own history and a newer main HEAD is not required.
Do not wait for the moving latest main HEAD to become green. The release-branch CI in Phase 2 validates the chosen commit before a preview tag is created.
Do not wait for the moving latest main HEAD to become green. The release-branch CI in Phase 2 validates the chosen commit before a candidate build is dispatched.
## Phase 2 — Pin and validate the release candidate, then publish the preview tag
## Phase 2 — Pin and validate the release source
- Use `release/<target>`. On the first attempt, create it from a selected main commit containing the version bump and intended changes; it need not be the latest main HEAD. On a restart, retain the existing branch and backport only reviewed fixes from main. Verify the branch's Cargo versions still equal `<target>`. If adopting an existing preview, run this branch CI before continuing its acceptance; reuse the tag only when it points to the validated SHA.
- The current `ci.yml` runs automatically on main pushes, not release-branch pushes. Dispatch it explicitly with `gh workflow run ci.yml --ref "release/<target>"`. Locate that dispatch run, verify its `head_sha` equals the release branch's remote HEAD, and require the full expected CI job set to pass, including `Quick Checks`, `Test and Lint`, and `End-to-End Tests (full merge gate)`. Treat a missing or skipped required lane as incomplete. A successful PR check or CI run for a different SHA does not qualify. If the branch advances or a run is cancelled, validate the new SHA again. Do not tag a candidate with failing or incomplete CI.
- Use `release/<target>`. On the first attempt, create it from a selected main commit containing the version bump and intended changes; it need not be the latest main HEAD. On a restart, retain the existing branch and backport only reviewed fixes from main. Verify the branch's Cargo versions still equal `<target>`. A legacy preview can anchor source selection, but its tag and old artifacts do not constitute a candidate in the new protocol; freeze and verify a new candidate after the release recipe is aligned.
- The current `ci.yml` runs automatically on main pushes, not release-branch pushes. Dispatch it explicitly with `gh workflow run ci.yml --ref "release/<target>"`. Locate that dispatch run, verify its `head_sha` equals the release branch's remote HEAD, and require the full expected CI job set to pass, including `Quick Checks`, `Test and Lint`, and `End-to-End Tests (full merge gate)`. Treat a missing or skipped required lane as incomplete. A successful PR check or CI run for a different SHA does not qualify. If the branch advances or a run is cancelled, validate the new SHA again. Do not dispatch a candidate with failing or incomplete CI.
- Prefer protecting `release/*` against force pushes and deletion and requiring reviewed backport PRs. Check live rulesets rather than assuming the main ruleset covers release branches; report a missing release-branch rule as a gap.
- Set `PREVIEW_HASH` to the CI-validated release-branch commit and recheck the remote branch HEAD immediately before tagging. Report the branch, commit, and CI run URL; do not derive `PREVIEW_HASH` from a later `origin/main` fetch. When adopting an existing preview tag at this SHA, skip tag creation and continue Phase 3.
```bash
git tag -a "<preview-tag>" -m "Release <preview-tag>" "$PREVIEW_HASH"
git push origin "<preview-tag>"
```
Pushing the tag triggers `.github/workflows/build.yml` ("Build and Release"); `docker.yml` chains off it via `workflow_run`.
The preview run builds versioned artifacts and publishes them in a GitHub prerelease. Docker may publish exact preview image tags. Latest-channel, R2, and Helm publication must be skipped; installation references and the website announcement stay unchanged.
On a restart after a backport, re-run release-branch CI before setting the new `PREVIEW_HASH`. Use the next unused preview iteration when the validated commit differs from an existing preview tag; never move an existing tag.
- Set `CANDIDATE_SHA` to the CI-validated release-branch commit. Record the branch, exact commit and CI run URL. The new candidate protocol starts from a commit whose release recipe matches the reviewed main dispatcher; backport the recipe before freezing an older release branch if needed.
- Dispatch the Linux candidate as described in [candidate verification](preview-release.md). Do not create a preview or final tag here. The final executable already contains `<target>`; publication later promotes those same bytes.
- After a source or build-input change, run the affected source checks and create a new candidate. Reuse the frozen candidate after an infrastructure failure. Do not restart acceptance for unrelated main advances.
@@ -4,18 +4,18 @@ description: Prepare an exact RustFS Cargo version or align installation referen
---
# Release Version Files
Require an exact target; ask if missing or ambiguous. Reject targets containing `-preview`: those are tag-only. Infer `post-release` only from an explicit request or parent workflow; otherwise use `prepare`. Delivery defaults to local edit/verify; commit, push, PR, and merge follow existing authorization and [Git rules](../../references/pull-requests.md).
Require an exact target; ask if missing or ambiguous. Reject targets containing `-preview`: those identify candidates, not product versions or new public tags. Infer `post-release` only from an explicit request or parent workflow; otherwise use `prepare`. Delivery defaults to local edit/verify; commit, push, PR, and merge follow existing authorization and [Git rules](../../references/pull-requests.md).
| Stage | Files and constraints |
|---|---|
| `prepare` | Only `Cargo.toml` and `Cargo.lock`: workspace package, internal dependency, and workspace member versions become `<target>`. External dependencies stay unchanged. |
| `post-release` | Only `README.md`, `README_ZH.md`, `flake.nix`, `helm/rustfs/Chart.yaml`, and `rustfs.spec`. Cargo may already target the next release; leave it untouched. |
Before installation edits, verify the exact non-preview Release, downloadable assets/source archive, pullable `rustfs/rustfs:<target>` image manifest, and published Helm chart. Block on missing artifacts. Do not downgrade defaults if a newer deliverable superseded the target. Preview preparation retains the previous published installation references; follow-up commits never replace either validated tag.
Before installation edits, verify the exact non-preview Release, downloadable assets/source archive, pullable `rustfs/rustfs:<target>` image manifest, and published Helm chart. Block on missing artifacts. Do not downgrade defaults if a newer deliverable superseded the target. Candidate preparation retains the previous published installation references; follow-up commits never replace validated tags or sealed artifacts.
Packaging policy:
- Docker image tags have no `v` prefix. Derive chart/app versions with `scripts/helm_chart_version.sh <target>` (`beta.N -> 0.N.0`; other versions unchanged). Helm CI derives versions from the final tag, so no early chart bump is required.
- Docker image tags have no `v` prefix. Derive chart/app versions with `scripts/helm_chart_version.sh <target>`; both equal the final product SemVer, including prerelease suffixes. Helm CI uses the frozen candidate version before sealing, so no early installation chart bump is required. Previously published chart versions remain unchanged.
- RPM `Version` is numeric; `Release` is the prerelease suffix or `1` for stable. Expanded `Source0` and the unpack directory must resolve to the exact target including its suffix. Add the changelog entry using current git identity, date, and target; never copy an example identity/date.
- Changing these mappings requires explicit confirmation.
+11
View File
@@ -31,3 +31,14 @@
# the flag, so the two can no longer drift apart unnoticed.
#
# For CPU profiling, add `-C force-frame-pointers=yes` to that RUSTFLAGS value.
# Exact internode HTTP/2 PING ACK timeout accounting is also deliberately
# default-off. Hyper exposes the exact timeout only through its unstable tracing
# event, so diagnostic builds must opt in to both the feature and cfg:
#
# RUSTFLAGS="--cfg hyper_unstable_tracing" \
# cargo build -p rustfs --features diagnostic-internode-http2-ping
#
# `crates/protos/build.rs` and `crates/obs/build.rs` fail fast if the feature and
# cfg drift apart. Do not move this cfg into global rustflags: doing so would
# recompile every dependency under an unstable, non-semver build contract.
+10
View File
@@ -0,0 +1,10 @@
# Rulings for the gateway conformance lane (.github/workflows/gateway-acceptance.yml).
#
# One row per case that fails on the GATEWAY stack for a reviewed reason: `case`; `ledger`, the
# rustfs/backlog#2684 entry that accepts the difference (R-0xx or K-n); a one-line `reason`; and
# `failure`, a prefix of the failure the ruling accepts, as the runner's report.json renders it
# (`[expect/status] at /expect/status: exchange #1, attempt 1: expected status 400, observed 403`).
# scripts/check_conformance_rulings.py fails the lane on a failed case without a row, on a ruled
# case whose failures do not start with its `failure`, and on a row whose case no longer fails or
# no longer exists. The legacy stack is a reference run and takes no rulings: the `rustfs` profile
# describes the gateway edge, not the legacy one.
+2 -2
View File
@@ -1,2 +1,2 @@
sha256-darwin=9901a4950099b94d3d2618524d540f1ed4bf6023044903886f98b6c4bf5f30fb
sha256-linux=09fd5f48f5556c9fc576357222dccd7ebb0b90b790798f1834db2ec5d67b4952
sha256-darwin=24a8cfaa1b9125bd69d7d4a7ff752a8ab8ed350c761d90ae8472ef52393e2686
sha256-linux=014257197fbbfbb48cab2864da9e10d96ee877d478fa29d119f8186532ea293d
+2 -2
View File
@@ -1,2 +1,2 @@
sha256-darwin=a2cb67d91eb0d093928371d6e84a84b2dd41b2ba975a28094e81879bd65838ad
sha256-linux=82db0561cf9b111f11782d647d0859c9d2a44c5c68481b5e39c48505828bd392
sha256-darwin=dcc076f8a36320d2fe36b4d2f26e2d54c3b9bc2ff7e14c6e90241f4fd8cf4a33
sha256-linux=229ebc0cbc9b35863600ae60c6fbb548516efe2a2f41f4b5db503b3283c2f64f
+1 -1
View File
@@ -1 +1 @@
sha256=e4ee3898b840a8374cb02c64342cc15f8d7ecd1becaf4b4a2a4657290f9e7ff1
sha256=4eac269f9a59b2f9be4e7a87ea6aa38454f5ebe10ede35c9b8d4218137d2cd35
+1 -1
View File
@@ -1 +1 @@
sha256=5bdd4c2eadbc41625ae3184fff34062e000d328108436259b2b7c4fe2531ecaf
sha256=fd5683273bba1eae3b4402a4e854a5d168ace845bd0e5fc801f8e336485a4394
+20
View File
@@ -70,6 +70,26 @@ s3s-footprint-check: ## Check the s3s dependency footprint ratchet stays frozen
@echo "📦 Checking s3s footprint ratchet..."
./scripts/check_s3s_footprint.sh
.PHONY: release-feature-boundary-check
release-feature-boundary-check: ## Check no release-reachable feature enables a debug-only one
@echo "🚧 Checking release feature boundaries..."
$(RUSTFS_PYTHON_BIN) ./scripts/check_connect_release_boundary.py --features-only
.PHONY: test-count-ratchet-check
test-count-ratchet-check: ## Check no workspace crate fell below its test floor or rose above its #[ignore] ceiling in the committed baseline
@echo "🧪 Checking per-crate test-count ratchet..."
./scripts/check_test_count_ratchet.sh
.PHONY: test-seams-check
test-seams-check: ## Check tests outside an operation's owner call its use-case only through rustfs/src/app/test_seams
@echo "🧪 Checking cross-owner test seams..."
$(RUSTFS_PYTHON_BIN) ./scripts/check_test_seams.py
.PHONY: test-retirements-check
test-retirements-check: ## Check every test retired by a migration slice names a replacement that exists
@echo "🧪 Checking test retirement tables..."
$(RUSTFS_PYTHON_BIN) ./scripts/check_test_retirements.py
.PHONY: fips-wording-check
fips-wording-check: ## Check docs and crates/kms do not over-claim crypto capabilities
@echo "📣 Checking cryptographic capability wording guard..."
+2 -2
View File
@@ -20,11 +20,11 @@ planning-docs-check: ## Check that no planning-type documents are committed
./scripts/check_no_planning_docs.sh
.PHONY: pre-commit
pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check quick-check ## Run fast pre-commit checks without clippy/full tests
pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check release-feature-boundary-check test-count-ratchet-check test-seams-check test-retirements-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check quick-check ## Run fast pre-commit checks without clippy/full tests
@echo "✅ All pre-commit checks passed!"
.PHONY: pre-pr
pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check log-analyzer-rules-check offline-enrollment-e2e-check clippy-check test ## Run full pre-PR checks with clippy and tests
pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check release-feature-boundary-check test-count-ratchet-check test-seams-check test-retirements-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check log-analyzer-rules-check offline-enrollment-e2e-check clippy-check test ## Run full pre-PR checks with clippy and tests
@echo "✅ All pre-PR checks passed!"
.PHONY: dev-check
+29 -1
View File
@@ -22,6 +22,14 @@ TEST_THREADS ?= 1
# differ from CI and .config/nextest.toml test-groups will NOT apply.
.PHONY: script-tests
.PHONY: release-package-tests
release-package-tests: ## Run release package integration on Linux with fpm, dpkg and rpm installed
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_packages.py
.PHONY: release-image-tests
release-image-tests: ## Run OCI integration with Docker, Go and skopeo; set RELEASE_IMAGE_VARIANT and RELEASE_IMAGE_OUTPUT
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_images.py
script-tests: ## Run shell script tests
@echo "Running script tests..."
trap 'status=$$?; printf "ERROR: script-tests failed (exit %s): %s\n" "$$status" "$$BASH_COMMAND" >&2; exit "$$status"' ERR
@@ -30,6 +38,7 @@ script-tests: ## Run shell script tests
./scripts/test_entrypoint_credentials.sh
./scripts/test_entrypoint_directories.sh
./scripts/test_internode_grpc_ab_bench.sh
$(RUSTFS_PYTHON_BIN) ./scripts/test_internode_grpc_window.py
./scripts/test_object_batch_bench_enhanced.sh
./scripts/test_package_service_scripts.sh
./scripts/test_hotpath_warp_ab_gate.sh
@@ -52,18 +61,30 @@ script-tests: ## Run shell script tests
./scripts/test_python_bin.sh
$(RUSTFS_PYTHON_BIN) ./scripts/test_check_doc_paths.py
./scripts/check_embedded_secrets.sh --self-test
./scripts/check_s3s_footprint.sh --self-test
./scripts/test_check_test_count_ratchet.sh
./scripts/test_guard_source_set.sh
./scripts/test_split_guard_rules.sh
$(RUSTFS_PYTHON_BIN) ./scripts/check_test_wiring.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/codemods/s3_error_to_s3_types.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/codemods/split_bucket_usecase.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/codemods/split_ecfs_legacy_edge.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/codemods/split_lifecycle_transition_api_test.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/check_test_seams.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/check_test_retirements.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/test_e2e_binary.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_download_release_asset.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_reference_binary.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_inspect_btf_prototypes.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_build_workflow.py
$(RUSTFS_PYTHON_BIN) -m unittest discover -s scripts/release -p "test_*.py"
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_glibc.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_upgrade_workflow.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_migration_gate_evidence.py
$(RUSTFS_PYTHON_BIN) ./scripts/ci_gate.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/check_connect_release_boundary.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/test_ci_test_artifact.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_ci_test_shards.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_ci_merge_group_proof.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_ci_merge_queue.py
$(RUSTFS_PYTHON_BIN) -m unittest discover -s scripts/ci -p 'test_*.py'
$(RUSTFS_PYTHON_BIN) ./scripts/check_security_coverage.py --self-test
@@ -77,8 +98,15 @@ script-tests: ## Run shell script tests
$(RUSTFS_PYTHON_BIN) ./scripts/test_inspect_validation_runner.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_retain_validation_evidence.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_ci_timing_report.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_junit_diff.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_dual_stack.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_mint_stack_diff.py
$(RUSTFS_PYTHON_BIN) ./scripts/check_conformance_rulings.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/compat_matrix_report.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/s3-tests/test_report_compat.py
$(RUSTFS_PYTHON_BIN) ./scripts/s3-tests/test_runner_tools.py
$(RUSTFS_PYTHON_BIN) ./scripts/s3-tests/test_api_coverage.py
$(RUSTFS_PYTHON_BIN) ./scripts/s3-tests/test_reference_parity.py
bash -n ./scripts/validate_object_data_cache_cold_stampede.sh
$(RUSTFS_PYTHON_BIN) ./scripts/check_object_data_cache_follower_samples.py --self-test
./scripts/validate_object_data_cache_cold_stampede.sh --self-test
+39 -2
View File
@@ -99,6 +99,11 @@ threads-required = "num-test-threads"
filter = '(package(rustfs-ecstore) & (test(/^core::pools_test::decommission_lock_order_tests::(active_decommission_migration_does_not_invert_capacity_and_delete_object(_tag)?_locks|decommission_moves_a_transitioned_version_expired_by_a_delete_all_rule_without_stalling)$/) | test(=services::integrity::tests::audit_migration_and_ack_recovery_preserve_source_and_reject_conflicts))) | (package(rustfs-heal) & test(=heal::mrf_queue::partial_write::tests::durable_retry_submission_error_keeps_checkpoint_and_uses_backoff))'
threads-required = "num-test-threads"
# These eight-disk namespace-owner probes overlap durable cleanup with a lock contender. Reserve capacity so sibling disk fixtures cannot consume their lock and completion budgets.
[[profile.default.overrides]]
filter = 'package(rustfs-ecstore) & test(/^store::object::tests::(canceled_create_retains_usage_cleanup_namespace_owner|batch_delete_excludes_recovery_on_nonzero_pool|delete_excludes_recovery_with_(disjoint|identical|overlapping)_pool_lock_clients)$/)'
threads-required = "num-test-threads"
# These eight-disk decommission fixtures exercise concurrent retries,
# free-version cleanup, and real capacity probes under production lock deadlines.
# Reserve the run's capacity so unrelated IO cannot exhaust those budgets.
@@ -136,8 +141,9 @@ setup = 'ecstore-large-stack'
filter = '(package(rustfs) & test(/^(admin::handlers::replication::target_repair_tests::[a-z_]+|admin::handlers::plugin_attempt::v2_tests::delegation_v2_persists_and_vends_with_real_parent_intersection|admin::handlers::site_replication::tests::durable_iam_state_apply_iam_item_rejects_a_stale_recreate_after_a_replicated_delete|admin::service::federated_identity::tests::(binding_keeps_explicit_iam_mapping_strict|markerless_legacy_oidc_session_crosses_auth_and_iam_deletion_boundaries)|storage::scanner_dirty_journal::tests::(clear_callback_keeps_unknown_coverage_until_all_pending_work_is_confirmed|owned_journals_keep_peer_records_after_interleaved_writes_and_clear))$/)) | (package(rustfs-heal) & ((binary(heal_b5_versioned_regression_test) & test(/^serial_tests::(test_heal_unversioned_bucket_e2e|test_historical_null_recursive_heal_restores_physical_versions|test_recursive_deep_heal_reports_stale_delete_marker_repair)$/)) | (binary(usage_observation_read_repair_test) & test(=usage_observation_read_repair_survives_snapshot_cleanup)))) | (package(rustfs-test-utils) & test(=data_usage_snapshot_tests::data_usage_snapshot_recovery_wires_primary_backup_and_negative_cache))'
setup = 'ecstore-base-stack'
# The multipart refusal fixture composes real-store setup, abort, quota and legal-hold futures.
[[profile.default.scripts]]
filter = 'package(rustfs-ecstore) | package(rustfs-s3select-api) | package(rustfs-scanner) | (package(rustfs) & test(/^(app::multipart_usecase::tests::concurrent_completions_share_durable_bucket_quota_reservations|app::object::delete::tests::compressed_delete_requests_update_observed_usage_without_releasing_quota_floor|app::object::internal_put::tests::internal_multipart_roundtrip_completes_and_abort_leaves_nothing|app::object::restore::tests::execute_restore_object_maps_failures_to_typed_s3_errors|storage::access::tests::(delete_object_access_captures_authorized_bucket_incarnation|copy_operations_reject_recreated_source_bucket_after_authorization|request_slot_keeps_bucket_policy_bound_to_its_store))$/))'
filter = 'package(rustfs-ecstore) | package(rustfs-s3select-api) | package(rustfs-scanner) | (package(rustfs) & test(/^(app::multipart_usecase::complete::tests::completion_absent_binding_refusal_order_baseline|app::multipart_usecase::complete::tests::concurrent_completions_share_durable_bucket_quota_reservations|app::object::delete::tests::compressed_delete_requests_update_observed_usage_without_releasing_quota_floor|app::object::internal_put::tests::internal_multipart_roundtrip_completes_and_abort_leaves_nothing|app::object::restore::tests::execute_restore_object_maps_failures_to_typed_s3_errors|storage::access::tests::(delete_object_access_captures_authorized_bucket_incarnation|copy_operations_reject_recreated_source_bucket_after_authorization|request_slot_keeps_bucket_policy_bound_to_its_store))$/))'
setup = 'ecstore-base-stack'
# These handler/store fixtures and the relocated-pool GET compose deep storage
@@ -356,6 +362,12 @@ threads-required = "num-test-threads"
filter = 'package(rustfs-ecstore) & test(=store::init::tests::dispatch_manifest_rollback_bounded_concurrency_reaches_tail_behind_slow_member)'
threads-required = "num-test-threads"
# These short-deadline real-disk progress probes need the test run's capacity
# for scanner and object-lock handoffs; keep their assertions and deadlines.
[[profile.default.overrides]]
filter = '(package(rustfs-scanner) & (test(=scanner::tests::cycle_persist_failure::later_cycle_post_commit_failure_reloads_durable_state) | test(=scanner::tests::running_main_loop_catches_up_pause_cleared_after_startup_observe))) | (package(rustfs-ecstore) & test(=store::init::tests::restored_transitioned_delete_causally_enqueues_free_version))'
threads-required = "num-test-threads"
# ---------------------------------------------------------------------------
# ci profile — the strict CI gate (ci.yml `cargo nextest run --profile ci`)
# ---------------------------------------------------------------------------
@@ -399,6 +411,11 @@ threads-required = "num-test-threads"
filter = '(package(rustfs-ecstore) & (test(/^core::pools_test::decommission_lock_order_tests::(active_decommission_migration_does_not_invert_capacity_and_delete_object(_tag)?_locks|decommission_moves_a_transitioned_version_expired_by_a_delete_all_rule_without_stalling)$/) | test(=services::integrity::tests::audit_migration_and_ack_recovery_preserve_source_and_reject_conflicts))) | (package(rustfs-heal) & test(=heal::mrf_queue::partial_write::tests::durable_retry_submission_error_keeps_checkpoint_and_uses_backoff))'
threads-required = "num-test-threads"
# These eight-disk namespace-owner probes overlap durable cleanup with a lock contender. Reserve capacity so sibling disk fixtures cannot consume their lock and completion budgets.
[[profile.ci.overrides]]
filter = 'package(rustfs-ecstore) & test(/^store::object::tests::(canceled_create_retains_usage_cleanup_namespace_owner|batch_delete_excludes_recovery_on_nonzero_pool|delete_excludes_recovery_with_(disjoint|identical|overlapping)_pool_lock_clients)$/)'
threads-required = "num-test-threads"
# These eight-disk decommission fixtures exercise concurrent retries,
# free-version cleanup, and real capacity probes under production lock deadlines.
# Reserve the run's capacity so unrelated IO cannot exhaust those budgets.
@@ -424,6 +441,12 @@ threads-required = "num-test-threads"
filter = 'package(rustfs-scanner) & (test(/^scanner::backlog::tests::native_/) | test(=scanner_folder::tests::mrf_ownership::mrf_ownership_manager_completion_preserves_scanner_pending) | test(/^scanner_io::tests::service_cohort::/))'
threads-required = "num-test-threads"
# Keep short-deadline real-disk progress probes from competing with unrelated
# tests for scanner and object-lock capacity. Assertions and deadlines stay on.
[[profile.ci.overrides]]
filter = '(package(rustfs-scanner) & (test(=scanner::tests::cycle_persist_failure::later_cycle_post_commit_failure_reloads_durable_state) | test(=scanner::tests::running_main_loop_catches_up_pause_cleared_after_startup_observe))) | (package(rustfs-ecstore) & test(=store::init::tests::restored_transitioned_delete_causally_enqueues_free_version))'
threads-required = "num-test-threads"
# Emit this expensive fixture's stage timings in CI and use tmpfs when safe.
[[profile.ci.overrides]]
filter = 'package(rustfs-ecstore) & test(=store::init::tests::decommission_durable_ilm_receipt_pagination_fails_closed_on_second_page)'
@@ -443,7 +466,7 @@ filter = '(package(rustfs) & test(/^(admin::handlers::replication::target_repair
setup = 'ecstore-base-stack'
[[profile.ci.scripts]]
filter = 'package(rustfs-ecstore) | package(rustfs-s3select-api) | package(rustfs-scanner) | (package(rustfs) & test(/^(app::multipart_usecase::tests::concurrent_completions_share_durable_bucket_quota_reservations|app::object::delete::tests::compressed_delete_requests_update_observed_usage_without_releasing_quota_floor|app::object::internal_put::tests::internal_multipart_roundtrip_completes_and_abort_leaves_nothing|app::object::restore::tests::execute_restore_object_maps_failures_to_typed_s3_errors|storage::access::tests::(delete_object_access_captures_authorized_bucket_incarnation|copy_operations_reject_recreated_source_bucket_after_authorization|request_slot_keeps_bucket_policy_bound_to_its_store))$/))'
filter = 'package(rustfs-ecstore) | package(rustfs-s3select-api) | package(rustfs-scanner) | (package(rustfs) & test(/^(app::multipart_usecase::complete::tests::completion_absent_binding_refusal_order_baseline|app::multipart_usecase::complete::tests::concurrent_completions_share_durable_bucket_quota_reservations|app::object::delete::tests::compressed_delete_requests_update_observed_usage_without_releasing_quota_floor|app::object::internal_put::tests::internal_multipart_roundtrip_completes_and_abort_leaves_nothing|app::object::restore::tests::execute_restore_object_maps_failures_to_typed_s3_errors|storage::access::tests::(delete_object_access_captures_authorized_bucket_incarnation|copy_operations_reject_recreated_source_bucket_after_authorization|request_slot_keeps_bucket_policy_bound_to_its_store))$/))'
setup = 'ecstore-base-stack'
[[profile.ci.scripts]]
@@ -698,6 +721,14 @@ test-group = 'ecstore-serial-flaky'
# the PR-lane e2e coverage for scanner usage snapshots consumed by quota and
# admin surfaces. It uses the same single-node, random-port, isolated-temp-dir
# fixture as the existing smoke modules.
#
# Dual-stack differential (rustfs/backlog#2740): `dual_stack_differential::`
# starts two single-node servers from the same binary, one per
# RUSTFS_S3_STACK value, on random ports with isolated temp dirs, and runs one
# SDK script covering every operation of `impl S3 for FS` against both. Its
# comparator unit tests need no server. It meets the admission criteria
# unchanged; the module clause admits a new differential test without another
# filter edit.
[profile.e2e-smoke]
default-filter = """
package(e2e_test) & (
@@ -706,6 +737,7 @@ default-filter = """
| test(/^reliant::lifecycle::/)
| test(/^conditional_delete_objects_test::/)
| test(/^reliant::tiering::/)
| test(/^dual_stack_differential::/)
| test(/^upgrade_compatibility_test::upgrade_write_readiness_tests::/)
| test(/^on_demand_migration::(get_basic_test::(get_miss_pulls_inline_and_serves_locally_afterwards|head_miss_answers_from_the_source_without_persisting)|interaction_test::test_odm_admin_config_is_redacted_and_status_counts_match_the_source)$/)
)
@@ -925,6 +957,10 @@ path = "junit.xml"
# parallel-safe — the same property e2e-smoke relies on. The exceptions are the
# 4-disk reliability / degraded-read fault-injection tests, multi-node heal
# clusters, and the fixed-port Vault tests, all serialized below.
# `dual_stack_differential::reference::` holds hermetic unit tests of how the
# differential picks side A (shell-script stand-ins, no server, no reference
# binary). They already run on every pull request in e2e-smoke, so e2e-full
# excludes them to keep its selection digest unchanged.
[profile.e2e-full]
default-filter = """
package(e2e_test)
@@ -935,6 +971,7 @@ default-filter = """
& !test(/^replication_target_matrix_test::/)
& !test(/^on_demand_migration::(concurrency_test|fault_test|interop_test|real_source_test)::/)
& !test(/^upgrade_compatibility_test::upgrade_write_readiness_tests::/)
& !test(/^dual_stack_differential::reference::/)
"""
fail-fast = false
+20
View File
@@ -0,0 +1,20 @@
# Pinned RustFS release used as the "reference" target of parity lanes
# (.github/workflows/reference-parity.yml). This is the single place that names
# the reference release; .github/actions/reference-binary resolves, downloads and
# verifies it, and fails on a missing pin, a missing asset or a digest mismatch.
#
# Bumping the reference is a reviewed event: every per-test difference between
# the candidate and the reference is measured against this exact build.
#
# Each sha256 is the value published in the release's SHA256SUMS asset, which
# matches the digest GitHub reports for the same asset. The action checks both.
schema = 1
repository = "rustfs/rustfs"
version = "1.0.5"
tag = "1.0.5"
[platforms.linux-x86_64-gnu]
asset = "rustfs-linux-x86_64-gnu-v1.0.5.zip"
sha256 = "5addae2f74756bf80104d0975e6347cda843bc2daffdfcd1f0cb80118b95cbd0"
executable = "rustfs"
+24
View File
@@ -0,0 +1,24 @@
# Targets of the s3-tests reference parity lane (.github/workflows/reference-parity.yml).
#
# Exactly one target uses `binary = "reference"` (the release pinned in
# .config/reference-binary.toml). Every other target uses `binary = "candidate"`
# (the binary built from the branch under test) and is compared per node id
# against the reference. All targets run the same harness, test list and server
# configuration; `env` adds only the target's own RUSTFS_* settings.
#
# Adding a target is one more [[targets]] table, for example:
#
# [[targets]]
# name = "candidate-gateway"
# binary = "candidate"
# env = { RUSTFS_S3_STACK = "gateway" }
schema = 1
[[targets]]
name = "reference"
binary = "reference"
[[targets]]
name = "candidate-legacy"
binary = "candidate"
+12
View File
@@ -0,0 +1,12 @@
# Tolerated per-test differences of the s3-tests reference parity lane.
#
# One entry per line, four fields separated by " | ":
#
# <target> | <pytest node id> | <tracking issue URL> | <reason>
#
# <target> is a candidate name from .config/reference-parity-targets.toml, the
# node id is exactly as the lane prints it (s3tests/functional/test_s3.py::...),
# and the issue URL is https://github.com/rustfs/<repo>/issues/<number>.
#
# Same semantics as scripts/junit_diff.py --allow: an entry whose difference no
# longer occurs is stale and fails the lane, so remove it when it is fixed.
+135
View File
@@ -0,0 +1,135 @@
{
"schema_version": 1,
"plan_id": "rustfs-release-validation",
"description": "Certification for immutable candidates. All shipped platforms require native identity and startup; stable and prerelease require the pinned upgrade matrix. Linux-only acceptance never authorizes publication.",
"release_types": [
"preview",
"prerelease",
"stable"
],
"targets": [
{
"os": "linux",
"arch": "x86_64",
"abi": "musl"
},
{
"os": "linux",
"arch": "aarch64",
"abi": "musl"
},
{
"os": "linux",
"arch": "x86_64",
"abi": "gnu"
},
{
"os": "linux",
"arch": "aarch64",
"abi": "gnu"
},
{
"os": "macos",
"arch": "aarch64",
"abi": "none"
},
{
"os": "windows",
"arch": "x86_64",
"abi": "msvc"
}
],
"targets_basis": "Platform matrix of .github/workflows/build.yml (prepare-platform-matrix). gnu and macOS targets build with the pyroscope feature; musl and Windows targets build with no extra features.",
"features": [
"pyroscope"
],
"trusted_workflows": [
{
"repository": "rustfs/rustfs",
"workflow_id": "release-candidate.yml"
},
{
"repository": "rustfs/rustfs",
"workflow_id": "release-complete.yml"
}
],
"trusted_workflows_basis": "Receipts are produced by the reviewed main workflows and fetched by exact run ID with GitHub artifact digest verification. Candidate inputs freeze the harness commit separately from product source.",
"lanes": [
{
"lane": "identity",
"harness_repository": "rustfs/rustfs",
"harness_sha": "candidate.workflow_sha",
"harness_basis": "The exact workflow commit is frozen in candidate inputs; no mutable branch is accepted as a harness identity.",
"timeout_seconds": 1800,
"environment_capabilities": [
"process-exec"
],
"cases": [
"artifact-identity",
"startup"
],
"applies_to": {
"release_types": [
"preview",
"prerelease",
"stable"
],
"platforms": "all"
},
"applies_basis": "Artifact identity and startup checks are mandatory for every release type, preview included, on every shipped target (rustfs/backlog#2662 frozen design item 5)."
},
{
"lane": "upgrade",
"harness_repository": "rustfs/rustfs",
"harness_sha": "candidate.workflow_sha",
"harness_basis": "The exact workflow commit is frozen in candidate inputs; no mutable branch is accepted as a harness identity.",
"timeout_seconds": 5400,
"environment_capabilities": [
"process-exec",
"previous-release-artifact"
],
"cases": [
"direct-upgrade",
"mixed-version-upgrade",
"bucket-config-upgrade",
"bucket-config-rollback",
"multipart-layout-upgrade",
"multipart-layout-baseline"
],
"cases_basis": "Case IDs of the pinned previous-release matrix in .github/workflows/e2e-upgrade.yml (source.version/asset/sha256 and the case list mapped there).",
"applies_to": {
"release_types": [
"prerelease",
"stable"
],
"platforms": [
{
"os": "linux",
"arch": "x86_64",
"abi": "gnu"
}
]
},
"applies_basis": "preview releases use the pre-declared smaller set (identity lane only). The upgrade lane is only exercised on linux-x86_64-gnu, the platform of the pinned source assets."
}
],
"upgrade_sources": {
"basis": "Exactly the two previous-release sources pinned in .github/workflows/e2e-upgrade.yml are exercised today (source.version / source.asset / source.sha256). No other N-1/N-2 compatibility is claimed, and these digests are not inferred from tags.",
"sources": [
{
"version": "1.0.1",
"role": "N-1",
"artifact_name": "rustfs-linux-x86_64-gnu-v1.0.1.zip",
"artifact_sha256": "9742e183cb0dfb0d48fa909797af572fe47b7f827569036baa4156e0d3958ee5"
},
{
"version": "1.0.0-rc.5",
"role": "N-2",
"artifact_name": "rustfs-linux-x86_64-gnu-v1.0.0-rc.5.zip",
"artifact_sha256": "3ee8df71e8edcfada533be452c4135868f697bc515460ae97b027313eade7a3d"
}
],
"rollback_supported": false,
"rollback_basis": "Only the bucket-config-rollback case (rollback_to_previous_release_reads_current_bucket_metadata) is exercised. Full data-plane rollback is not certified and is never a default claim."
}
}
+37
View File
@@ -0,0 +1,37 @@
# Files that may still reference s3s (`s3s::` or `s3s_sigv4`) when Phase 1 of
# the gateway migration ends (rustfs/backlog#2734 task T0.6, rustfs/backlog#2741).
# Consumed only by `scripts/check_s3s_footprint.sh --mode allowlist`.
#
# One entry per line: a repository-relative file path or a git `:(glob)`
# pathspec. `**` matches across directories, `*` stays within one path
# component, and a directory entry matches everything under it. Blank lines
# and lines starting with `#` are ignored; an entry may not contain whitespace,
# so a note belongs on its own comment line.
#
# Every Rust file listed by `git grep -l '\bs3s\(::\|_sigv4\)' -- '*.rs'` that
# no entry matches fails the allowlist mode. Task T1.9 switches CI to that
# mode; until then `--dry-run` prints the files still outside this list, which
# is the Phase 1 tracking list.
#
# Never add crates/** other than crates/e2e_test: the engine crates leave s3s
# entirely, and the script rejects such an entry. Only the s3s-facing edge of
# the rustfs binary and the e2e oracle keep it.
# rustfs binary: the s3s-facing edge that survives Phase 1
rustfs/src/storage/ecfs.rs
rustfs/src/storage/s3_compat/**
rustfs/src/storage/access.rs
rustfs/src/auth.rs
rustfs/src/error.rs
rustfs/src/storage_api.rs
rustfs/src/app/storage_api.rs
rustfs/src/admin/router.rs
rustfs/src/app/metadata_route.rs
rustfs/src/server/http.rs
rustfs/src/server/layer.rs
rustfs/src/server/hybrid.rs
rustfs/src/protocols/client.rs
# e2e oracle: the test target that verifies S3 behaviour against s3s
crates/e2e_test/src/fake_s3_target/**
crates/e2e_test/src/**
+18
View File
@@ -0,0 +1,18 @@
# Baseline for the direct s3s footprint, consumed only by
# scripts/check_s3s_footprint.sh (rustfs/backlog#1677; rustfs/backlog#2734 T0.4;
# comparison rules on rustfs/backlog#2765).
#
# Each line is a ceiling everywhere: a counter above it fails, because new code
# widened the s3s surface; route it through the gateway abstractions instead and
# never raise a value here. Pull requests and merge groups are additionally held
# to their base revision's measurement, so a slice that shrinks the surface need
# not edit this file. A counter below its line passes and the check prints the
# lowered line; the migration coordinator lowers the lines in batches.
#
# files Rust files referencing s3s paths, repo-wide minus crates/e2e_test
# s3_error_lines lines holding s3_error!, same scope, minus those the script
# header's rule clears (rustfs_s3_types's macro); fails closed
# ecstore_files Rust files referencing s3s paths under crates/ecstore/src
files=63
s3_error_lines=209
ecstore_files=0
+67
View File
@@ -0,0 +1,67 @@
# Per-crate test-count baseline consumed by scripts/check_test_count_ratchet.sh
# (rustfs/backlog#2734 section 1 rule 3; T0.3 in rustfs/backlog#2738).
#
# One row per workspace member: <package>, tests, ignored, tab-separated.
# tests counts #[test], #[tokio::test], #[tokio::test(...)] and #[test_case(...)]
# attribute lines under the package directory; ignored counts #[ignore] and
# #[ignore = "..."]. The tests column is a floor and the ignored column is a
# ceiling: CI fails when a crate's tests fall below its row or its ignored count
# rises above it. Extra tests and fewer ignores pass; --update tightens the rows.
# Lowering a tests row or raising an ignored row is a reviewed statement that
# tests were intentionally removed or ignored; justify it in the PR description.
# Regenerate with:
# scripts/check_test_count_ratchet.sh --update
e2e_test 1026 20
rustfs 6449 64
rustfs-audit 70 0
rustfs-checksums 28 1
rustfs-common 36 0
rustfs-concurrency 87 0
rustfs-config 43 0
rustfs-connect-symbol-catalog 6 0
rustfs-credentials 24 0
rustfs-crypto 67 0
rustfs-data-usage 58 0
rustfs-ecstore 7858 2
rustfs-extension-schema 23 0
rustfs-filemeta 361 0
rustfs-gateway-adapter 74 0
rustfs-heal 1128 3
rustfs-heal-contracts 8 0
rustfs-iam 415 0
rustfs-io-core 52 0
rustfs-io-metrics 157 0
rustfs-keystone 31 0
rustfs-kms 716 25
rustfs-license 2 0
rustfs-lifecycle 156 0
rustfs-lock 182 0
rustfs-log-analyzer 91 0
rustfs-madmin 210 0
rustfs-notify 140 0
rustfs-object-capacity 92 0
rustfs-object-data-cache 141 0
rustfs-obs 387 0
rustfs-policy 758 0
rustfs-protocols 770 1
rustfs-protos 75 0
rustfs-replication 209 0
rustfs-rio 225 1
rustfs-rio-v2 34 6
rustfs-s3-client 71 0
rustfs-s3-ops 8 0
rustfs-s3-types 105 0
rustfs-s3select-api 232 0
rustfs-s3select-query 235 2
rustfs-scanner 1392 20
rustfs-scanner-metrics 90 0
rustfs-security-governance 20 0
rustfs-signer 34 0
rustfs-storage-api 34 0
rustfs-targets 612 26
rustfs-test-utils 1 0
rustfs-tls-runtime 44 0
rustfs-trusted-proxies 72 0
rustfs-utils 253 1
rustfs-zip 30 0
zstd 11 0
@@ -0,0 +1,3 @@
# Tests the bucket_access slice retires (rustfs/backlog#2765); format and checks: scripts/check_test_retirements.py.
# <crate> | <module path>::<test fn> | <kind: test, differential or e2e> | <replacement> | <reason>
# The retired test must still exist when its entry is added; the change that deletes it deletes the entry.
@@ -0,0 +1,3 @@
# Tests the bucket_configs slice retires (rustfs/backlog#2765); format and checks: scripts/check_test_retirements.py.
# <crate> | <module path>::<test fn> | <kind: test, differential or e2e> | <replacement> | <reason>
# The retired test must still exist when its entry is added; the change that deletes it deletes the entry.
+3
View File
@@ -0,0 +1,3 @@
# Tests the bucket_core slice retires (rustfs/backlog#2765); format and checks: scripts/check_test_retirements.py.
# <crate> | <module path>::<test fn> | <kind: test, differential or e2e> | <replacement> | <reason>
# The retired test must still exist when its entry is added; the change that deletes it deletes the entry.
@@ -0,0 +1,3 @@
# Tests the bucket_replication_lifecycle slice retires (rustfs/backlog#2765); format and checks: scripts/check_test_retirements.py.
# <crate> | <module path>::<test fn> | <kind: test, differential or e2e> | <replacement> | <reason>
# The retired test must still exist when its entry is added; the change that deletes it deletes the entry.
+3
View File
@@ -0,0 +1,3 @@
# Tests the cutover slice retires (rustfs/backlog#2765); format and checks: scripts/check_test_retirements.py.
# <crate> | <module path>::<test fn> | <kind: test, differential or e2e> | <replacement> | <reason>
# The retired test must still exist when its entry is added; the change that deletes it deletes the entry.
@@ -0,0 +1,3 @@
# Tests the listing_multipart slice retires (rustfs/backlog#2765); format and checks: scripts/check_test_retirements.py.
# <crate> | <module path>::<test fn> | <kind: test, differential or e2e> | <replacement> | <reason>
# The retired test must still exist when its entry is added; the change that deletes it deletes the entry.
@@ -0,0 +1,3 @@
# Tests the object_commit slice retires (rustfs/backlog#2765); format and checks: scripts/check_test_retirements.py.
# <crate> | <module path>::<test fn> | <kind: test, differential or e2e> | <replacement> | <reason>
# The retired test must still exist when its entry is added; the change that deletes it deletes the entry.
+3
View File
@@ -0,0 +1,3 @@
# Tests the object_meta slice retires (rustfs/backlog#2765); format and checks: scripts/check_test_retirements.py.
# <crate> | <module path>::<test fn> | <kind: test, differential or e2e> | <replacement> | <reason>
# The retired test must still exist when its entry is added; the change that deletes it deletes the entry.
+3
View File
@@ -0,0 +1,3 @@
# Tests the object_misc slice retires (rustfs/backlog#2765); format and checks: scripts/check_test_retirements.py.
# <crate> | <module path>::<test fn> | <kind: test, differential or e2e> | <replacement> | <reason>
# The retired test must still exist when its entry is added; the change that deletes it deletes the entry.
+3
View File
@@ -0,0 +1,3 @@
# Tests the object_read slice retires (rustfs/backlog#2765); format and checks: scripts/check_test_retirements.py.
# <crate> | <module path>::<test fn> | <kind: test, differential or e2e> | <replacement> | <reason>
# The retired test must still exist when its entry is added; the change that deletes it deletes the entry.
@@ -0,0 +1,3 @@
# Tests the object_write slice retires (rustfs/backlog#2765); format and checks: scripts/check_test_retirements.py.
# <crate> | <module path>::<test fn> | <kind: test, differential or e2e> | <replacement> | <reason>
# The retired test must still exist when its entry is added; the change that deletes it deletes the entry.
+3
View File
@@ -0,0 +1,3 @@
# Tests the t211 slice retires (rustfs/backlog#2765); format and checks: scripts/check_test_retirements.py.
# <crate> | <module path>::<test fn> | <kind: test, differential or e2e> | <replacement> | <reason>
# The retired test must still exist when its entry is added; the change that deletes it deletes the entry.
+3
View File
@@ -0,0 +1,3 @@
# Tests the t212 slice retires (rustfs/backlog#2765); format and checks: scripts/check_test_retirements.py.
# <crate> | <module path>::<test fn> | <kind: test, differential or e2e> | <replacement> | <reason>
# The retired test must still exist when its entry is added; the change that deletes it deletes the entry.
+6
View File
@@ -140,6 +140,12 @@ With list syntax, use `- RUSTFS_OBS_LOG_DIRECTORY=/var/log/rustfs/` or quote the
---
## Local storage pool expansion
To append a second four-drive pool, set `RUSTFS_VOLUMES: "/data/rustfs0/{0...3} /data/rustfs1/{0...3}"` and keep the original pool's drives, order, and erasure set width. The container prepares each drive directory and passes one argument per pool to RustFS.
If an older image already formatted both ranges together as a single eight-drive pool, list the eight individual paths in their original order to keep that layout when upgrading.
## 🔧 Common Operations
### Cleaning Up
@@ -34,6 +34,10 @@ services:
- RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
- RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
- RUSTFS_DURABILITY_MODE=${RUSTFS_DURABILITY_MODE:-strict}
- RUSTFS_OBJECT_DIRECT_IO_WRITE_ENABLE=${RUSTFS_OBJECT_DIRECT_IO_WRITE_ENABLE:-false}
- RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_ENABLE=${RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_ENABLE:-false}
- RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_WAIT_MICROS=${RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_WAIT_MICROS:-0}
# Internode gRPC optimization knobs (grpc-optimization P0-P3). Defaults match the binary
# defaults, so leaving these unset is a no-op; the A/B driver exports them to toggle a stage.
- RUSTFS_INTERNODE_RPC_TCP_NODELAY=${RUSTFS_INTERNODE_RPC_TCP_NODELAY:-true}
@@ -43,6 +47,8 @@ services:
- RUSTFS_INTERNODE_CHANNEL_ISOLATION=${RUSTFS_INTERNODE_CHANNEL_ISOLATION:-false}
- RUSTFS_INTERNODE_BULK_CHANNELS=${RUSTFS_INTERNODE_BULK_CHANNELS:-2}
- RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY:-false}
- RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED:-false}
- RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED=${RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED:-false}
- RUSTFS_INTERNODE_PREWARM=${RUSTFS_INTERNODE_PREWARM:-false}
- RUSTFS_INTERNODE_OFFLINE_BYPASS=${RUSTFS_INTERNODE_OFFLINE_BYPASS:-false}
- RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS=${RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS:-5}
@@ -56,7 +62,7 @@ services:
- node1_data_2:/data/rustfs2
- node1_data_3:/data/rustfs3
ports:
- "9000:9000"
- "${RUSTFS_NODE1_PORT:-9000}:9000"
networks:
- rustfs-cluster-net
@@ -81,6 +87,10 @@ services:
- RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
- RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
- RUSTFS_DURABILITY_MODE=${RUSTFS_DURABILITY_MODE:-strict}
- RUSTFS_OBJECT_DIRECT_IO_WRITE_ENABLE=${RUSTFS_OBJECT_DIRECT_IO_WRITE_ENABLE:-false}
- RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_ENABLE=${RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_ENABLE:-false}
- RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_WAIT_MICROS=${RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_WAIT_MICROS:-0}
# Internode gRPC optimization knobs (grpc-optimization P0-P3). Defaults match the binary
# defaults, so leaving these unset is a no-op; the A/B driver exports them to toggle a stage.
- RUSTFS_INTERNODE_RPC_TCP_NODELAY=${RUSTFS_INTERNODE_RPC_TCP_NODELAY:-true}
@@ -90,6 +100,8 @@ services:
- RUSTFS_INTERNODE_CHANNEL_ISOLATION=${RUSTFS_INTERNODE_CHANNEL_ISOLATION:-false}
- RUSTFS_INTERNODE_BULK_CHANNELS=${RUSTFS_INTERNODE_BULK_CHANNELS:-2}
- RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY:-false}
- RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED:-false}
- RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED=${RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED:-false}
- RUSTFS_INTERNODE_PREWARM=${RUSTFS_INTERNODE_PREWARM:-false}
- RUSTFS_INTERNODE_OFFLINE_BYPASS=${RUSTFS_INTERNODE_OFFLINE_BYPASS:-false}
- RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS=${RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS:-5}
@@ -103,7 +115,7 @@ services:
- node2_data_2:/data/rustfs2
- node2_data_3:/data/rustfs3
ports:
- "9001:9000"
- "${RUSTFS_NODE2_PORT:-9001}:9000"
networks:
- rustfs-cluster-net
@@ -128,6 +140,10 @@ services:
- RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
- RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
- RUSTFS_DURABILITY_MODE=${RUSTFS_DURABILITY_MODE:-strict}
- RUSTFS_OBJECT_DIRECT_IO_WRITE_ENABLE=${RUSTFS_OBJECT_DIRECT_IO_WRITE_ENABLE:-false}
- RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_ENABLE=${RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_ENABLE:-false}
- RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_WAIT_MICROS=${RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_WAIT_MICROS:-0}
# Internode gRPC optimization knobs (grpc-optimization P0-P3). Defaults match the binary
# defaults, so leaving these unset is a no-op; the A/B driver exports them to toggle a stage.
- RUSTFS_INTERNODE_RPC_TCP_NODELAY=${RUSTFS_INTERNODE_RPC_TCP_NODELAY:-true}
@@ -137,6 +153,8 @@ services:
- RUSTFS_INTERNODE_CHANNEL_ISOLATION=${RUSTFS_INTERNODE_CHANNEL_ISOLATION:-false}
- RUSTFS_INTERNODE_BULK_CHANNELS=${RUSTFS_INTERNODE_BULK_CHANNELS:-2}
- RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY:-false}
- RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED:-false}
- RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED=${RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED:-false}
- RUSTFS_INTERNODE_PREWARM=${RUSTFS_INTERNODE_PREWARM:-false}
- RUSTFS_INTERNODE_OFFLINE_BYPASS=${RUSTFS_INTERNODE_OFFLINE_BYPASS:-false}
- RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS=${RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS:-5}
@@ -150,7 +168,7 @@ services:
- node3_data_2:/data/rustfs2
- node3_data_3:/data/rustfs3
ports:
- "9002:9000"
- "${RUSTFS_NODE3_PORT:-9002}:9000"
networks:
- rustfs-cluster-net
@@ -175,6 +193,10 @@ services:
- RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
- RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
- RUSTFS_DURABILITY_MODE=${RUSTFS_DURABILITY_MODE:-strict}
- RUSTFS_OBJECT_DIRECT_IO_WRITE_ENABLE=${RUSTFS_OBJECT_DIRECT_IO_WRITE_ENABLE:-false}
- RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_ENABLE=${RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_ENABLE:-false}
- RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_WAIT_MICROS=${RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_WAIT_MICROS:-0}
# Internode gRPC optimization knobs (grpc-optimization P0-P3). Defaults match the binary
# defaults, so leaving these unset is a no-op; the A/B driver exports them to toggle a stage.
- RUSTFS_INTERNODE_RPC_TCP_NODELAY=${RUSTFS_INTERNODE_RPC_TCP_NODELAY:-true}
@@ -184,6 +206,8 @@ services:
- RUSTFS_INTERNODE_CHANNEL_ISOLATION=${RUSTFS_INTERNODE_CHANNEL_ISOLATION:-false}
- RUSTFS_INTERNODE_BULK_CHANNELS=${RUSTFS_INTERNODE_BULK_CHANNELS:-2}
- RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY:-false}
- RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED:-false}
- RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED=${RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED:-false}
- RUSTFS_INTERNODE_PREWARM=${RUSTFS_INTERNODE_PREWARM:-false}
- RUSTFS_INTERNODE_OFFLINE_BYPASS=${RUSTFS_INTERNODE_OFFLINE_BYPASS:-false}
- RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS=${RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS:-5}
@@ -197,7 +221,7 @@ services:
- node4_data_2:/data/rustfs2
- node4_data_3:/data/rustfs3
ports:
- "9003:9000"
- "${RUSTFS_NODE4_PORT:-9003}:9000"
networks:
- rustfs-cluster-net
+54
View File
@@ -0,0 +1,54 @@
# Copyright 2026 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name: Both-stack leg
description: Prove which S3 stack one e2e leg's servers selected and upload the leg for the summary (call with if always)
inputs:
profile:
description: The nextest profile the leg ran
required: true
stack:
description: The stack the leg requested, legacy or gateway
required: true
logs:
description: Directory or file holding the servers' output
required: true
runs:
using: composite
steps:
- name: Prove which stack the servers selected
shell: bash
env:
PROFILE: ${{ inputs.profile }}
STACK: ${{ inputs.stack }}
LOGS: ${{ inputs.logs }}
run: |
# Outside the workspace: e2e_binary.py refuses a run whose source tree changed.
mkdir -p "$RUNNER_TEMP/leg"
junit="target/nextest/${PROFILE}/junit.xml"
if [ -s "$junit" ]; then cp "$junit" "$RUNNER_TEMP/leg/junit.xml"; fi
# A gateway leg stopped by its preflight has no output; the preflight's stack.json stays.
if [ -e "$LOGS" ]; then python3 scripts/dual_stack.py stack --expect "$STACK" --out "$RUNNER_TEMP/leg/stack.json" "$LOGS"; fi
- name: Upload the leg
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: dual-stack-leg-${{ inputs.profile }}-${{ inputs.stack }}
overwrite: true
path: ${{ runner.temp }}/leg
if-no-files-found: warn
retention-days: 7
@@ -0,0 +1,56 @@
# Copyright 2026 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name: Both-stack summary
description: Diff the legacy and gateway legs of this run (scripts/junit_diff.py) and upload one dated summary
inputs:
profiles:
description: Comma-separated nextest profiles whose two legs this run uploaded
required: true
runs:
using: composite
steps:
- name: Download the legs
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
pattern: dual-stack-leg-*
path: legs
- name: Diff every profile and write the summary
id: summary
shell: bash
env:
PROFILES: ${{ inputs.profiles }}
run: |
# The shell runs with -e; the exit status is reported after the summary is written.
set +e -u -o pipefail
date="$(date -u +%F)"
echo "date=${date}" >> "$GITHUB_OUTPUT"
mkdir -p legs
python3 scripts/dual_stack.py summary --profiles "$PROFILES" --legs legs --date "$date" --out-dir dual-stack
status=$?
if [ -f dual-stack/summary.md ]; then cat dual-stack/summary.md >> "$GITHUB_STEP_SUMMARY"; fi
exit "$status"
- name: Upload the dated summary
if: always() && steps.summary.outputs.date != ''
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: dual-stack-summary-${{ steps.summary.outputs.date }}
overwrite: true
path: dual-stack
if-no-files-found: error
retention-days: 30
+27 -1
View File
@@ -95,7 +95,33 @@ runs:
- name: Check s3s footprint ratchet
shell: bash
run: ./scripts/check_s3s_footprint.sh
env:
# Pull requests and merge groups compare with their base revision; a
# push or scheduled run holds the tree to the baseline file exactly.
FOOTPRINT_BASE: ${{ github.event.pull_request.base.sha || github.event.merge_group.base_sha }}
run: |
if [[ -z "$FOOTPRINT_BASE" ]]; then
./scripts/check_s3s_footprint.sh
exit
fi
git cat-file -e "$FOOTPRINT_BASE^{commit}" 2>/dev/null || git fetch --no-tags --depth=1 origin "$FOOTPRINT_BASE"
./scripts/check_s3s_footprint.sh --base "$FOOTPRINT_BASE"
- name: Check release feature boundaries
shell: bash
run: python3 scripts/check_connect_release_boundary.py --features-only
- name: Check per-crate test-count ratchet
shell: bash
run: ./scripts/check_test_count_ratchet.sh
- name: Check cross-owner tests use test seams
shell: bash
run: python3 ./scripts/check_test_seams.py
- name: Check retired tests name existing replacements
shell: bash
run: python3 ./scripts/check_test_retirements.py
- name: Check cryptographic capability wording
shell: bash
@@ -0,0 +1,96 @@
# Copyright 2026 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name: "Reference RustFS binary"
description: "Download the pinned reference RustFS release, verify its sha256 and expose the executable path. A missing pin, missing asset or digest mismatch fails the step."
inputs:
pin-file:
description: "Pin file naming the reference release (repository-relative)"
required: false
default: ".config/reference-binary.toml"
platform:
description: "Platform table in the pin file"
required: false
default: "linux-x86_64-gnu"
destination:
description: "Directory that receives the executable; defaults to $RUNNER_TEMP/reference-binary"
required: false
default: ""
outputs:
path:
description: "Absolute path of the verified executable"
value: ${{ steps.unpack.outputs.path }}
version:
description: "Pinned reference version"
value: ${{ steps.pin.outputs.version }}
sha256:
description: "Pinned and verified archive sha256"
value: ${{ steps.pin.outputs.sha256 }}
runs:
using: "composite"
steps:
- name: Resolve reference pin
id: pin
shell: bash
env:
PIN_TOOL: ${{ github.action_path }}/reference_binary.py
PIN_FILE: ${{ inputs.pin-file }}
PIN_PLATFORM: ${{ inputs.platform }}
run: |
set -euo pipefail
resolved="$(python3 "$PIN_TOOL" resolve --pin "$PIN_FILE" --platform "$PIN_PLATFORM")"
printf '%s\n' "$resolved"
printf '%s\n' "$resolved" >> "$GITHUB_OUTPUT"
- name: Download and verify reference archive
id: download
shell: bash
env:
GH_TOKEN: ${{ github.token }}
ASSET_DOWNLOADER: ${{ github.action_path }}/../setup/download_release_asset.py
REF_REPOSITORY: ${{ steps.pin.outputs.repository }}
REF_TAG: ${{ steps.pin.outputs.tag }}
REF_ASSET: ${{ steps.pin.outputs.asset }}
REF_SHA256: ${{ steps.pin.outputs.sha256 }}
run: |
set -euo pipefail
archive="$RUNNER_TEMP/reference-archive/$REF_ASSET"
# Fails unless the release lists exactly one asset of this name whose
# published digest equals the pin and whose downloaded bytes match it.
python3 "$ASSET_DOWNLOADER" --repository "$REF_REPOSITORY" --tag "$REF_TAG" \
--asset "$REF_ASSET" --output "$archive" --sha256 "$REF_SHA256"
echo "archive=$archive" >> "$GITHUB_OUTPUT"
- name: Unpack reference executable
id: unpack
shell: bash
env:
PIN_TOOL: ${{ github.action_path }}/reference_binary.py
REF_ARCHIVE: ${{ steps.download.outputs.archive }}
REF_SHA256: ${{ steps.pin.outputs.sha256 }}
REF_EXECUTABLE: ${{ steps.pin.outputs.executable }}
REF_DESTINATION: ${{ inputs.destination }}
run: |
set -euo pipefail
destination="${REF_DESTINATION:-$RUNNER_TEMP/reference-binary}"
unpacked="$(python3 "$PIN_TOOL" unpack --archive "$REF_ARCHIVE" --sha256 "$REF_SHA256" \
--executable "$REF_EXECUTABLE" --destination "$destination")"
path="${unpacked#path=}"
test -x "$path"
echo "Reference executable: $path"
"$path" --version
echo "path=$path" >> "$GITHUB_OUTPUT"
+144
View File
@@ -0,0 +1,144 @@
#!/usr/bin/env python3
# Copyright 2026 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Resolve and unpack the pinned reference RustFS release binary.
`resolve` reads the pin file and prints the selected platform's identity as
`key=value` lines (GitHub step-output format). `unpack` re-verifies the
downloaded archive digest against the pin and extracts exactly one executable.
Every missing or malformed input is a failure; nothing is skipped.
"""
import argparse
import hashlib
from pathlib import Path
import re
import sys
import tomllib
import zipfile
REPOSITORY = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.-]*/[A-Za-z0-9][A-Za-z0-9_.-]*")
TAG = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.+-]*")
PLATFORM = re.compile(r"[a-z0-9][a-z0-9_-]*")
ASSET = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.+-]*\.zip")
SHA256 = re.compile(r"[0-9a-f]{64}")
EXECUTABLE = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.-]*")
MAX_EXECUTABLE_BYTES = 1024 * 1024 * 1024
class PinError(Exception):
"""A pin, archive or argument that cannot be trusted."""
def field(table, key, pattern, where):
value = table.get(key)
if not isinstance(value, str) or not pattern.fullmatch(value):
raise PinError(f"{where}: missing or invalid '{key}'")
return value
def resolve(pin_path, platform):
if not PLATFORM.fullmatch(platform or ""):
raise PinError("invalid platform name")
try:
data = tomllib.loads(Path(pin_path).read_text(encoding="utf-8"))
except FileNotFoundError as exc:
raise PinError(f"{pin_path}: reference pin file is missing") from exc
except (OSError, UnicodeDecodeError, tomllib.TOMLDecodeError) as exc:
raise PinError(f"{pin_path}: cannot read reference pin: {exc}") from exc
if data.get("schema") != 1 or type(data.get("schema")) is not int:
raise PinError(f"{pin_path}: unsupported pin schema")
identity = {
"repository": field(data, "repository", REPOSITORY, pin_path),
"version": field(data, "version", TAG, pin_path),
"tag": field(data, "tag", TAG, pin_path),
}
platforms = data.get("platforms")
if not isinstance(platforms, dict) or not platforms:
raise PinError(f"{pin_path}: no pinned platforms")
entry = platforms.get(platform)
if not isinstance(entry, dict):
raise PinError(f"{pin_path}: no pin for platform '{platform}'")
where = f"{pin_path} [platforms.{platform}]"
identity["asset"] = field(entry, "asset", ASSET, where)
identity["sha256"] = field(entry, "sha256", SHA256, where)
identity["executable"] = field(entry, "executable", EXECUTABLE, where)
if identity["version"] not in identity["asset"]:
raise PinError(f"{where}: asset name does not carry version {identity['version']}")
unknown = set(entry) - {"asset", "sha256", "executable"}
if unknown:
raise PinError(f"{where}: unknown keys {sorted(unknown)}")
return identity
def file_sha256(path):
with open(path, "rb") as stream:
return hashlib.file_digest(stream, "sha256").hexdigest()
def unpack(archive, sha256, executable, destination):
if not SHA256.fullmatch(sha256 or "") or not EXECUTABLE.fullmatch(executable or ""):
raise PinError("invalid expected digest or executable name")
try:
actual = file_sha256(archive)
except OSError as exc:
raise PinError(f"{archive}: reference archive is missing or unreadable") from exc
if actual != sha256:
raise PinError(f"{archive}: sha256 {actual} does not match the pinned digest")
try:
with zipfile.ZipFile(archive) as bundle:
matches = [info for info in bundle.infolist() if info.filename == executable]
if len(matches) != 1:
raise PinError(f"{archive}: must contain exactly one top-level '{executable}'")
info = matches[0]
if info.is_dir() or not 0 < info.file_size <= MAX_EXECUTABLE_BYTES:
raise PinError(f"{archive}: '{executable}' is not a plausible executable")
destination = Path(destination)
destination.mkdir(parents=True, exist_ok=True)
target = destination / executable
with bundle.open(info) as source, open(target, "wb") as sink:
while chunk := source.read(1024 * 1024):
sink.write(chunk)
except zipfile.BadZipFile as exc:
raise PinError(f"{archive}: not a valid zip archive") from exc
target.chmod(0o755)
return target
def main(argv=None):
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
sub = parser.add_subparsers(dest="mode", required=True)
res = sub.add_parser("resolve")
res.add_argument("--pin", required=True)
res.add_argument("--platform", required=True)
unp = sub.add_parser("unpack")
unp.add_argument("--archive", required=True)
unp.add_argument("--sha256", required=True)
unp.add_argument("--executable", required=True)
unp.add_argument("--destination", required=True)
args = parser.parse_args(argv)
try:
if args.mode == "resolve":
for key, value in resolve(args.pin, args.platform).items():
print(f"{key}={value}")
else:
print(f"path={unpack(args.archive, args.sha256, args.executable, args.destination)}")
except PinError as exc:
print(f"reference binary: {exc}", file=sys.stderr)
return 1
return 0
if __name__ == "__main__":
sys.exit(main())
+6
View File
@@ -63,6 +63,11 @@ inputs:
required: false
default: "true"
outputs:
cache-hit:
description: "'true' when the Rust cache was restored from an exact key match"
value: ${{ steps.rust-cache.outputs.cache-hit }}
runs:
using: "composite"
steps:
@@ -143,6 +148,7 @@ runs:
tool: cargo-nextest@0.9.138
- name: Setup Rust cache
id: rust-cache
uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2
with:
# false is rust-cache's own default. With true, cleanup.ts returns
+208 -22
View File
@@ -296,6 +296,34 @@
"origin": "baseline",
"alert_contract": "scheduled-validation"
},
{
"workflow": ".github/workflows/rdma-feature-check.yml",
"name": "S3-over-RDMA feature check",
"roles": [
"pr",
"periodic",
"manual"
],
"events": [
"pull_request",
"schedule",
"workflow_dispatch"
],
"cadence": "weekly",
"timezone": "UTC",
"schedule": [
"37 5 * * 1"
],
"owner": "@overtrue",
"policy": "active",
"monitor": "workflow",
"max_age_hours": 192,
"result_owner": "rdma-feature-check.yml",
"disposition": "compiles the nixl feature and runs rustfs-nixl unit tests on Linux; the feature is off in every other build",
"failure_owner": "workflow",
"origin": "added",
"alert_contract": "scheduled-validation"
},
{
"workflow": ".github/workflows/rustfs-validation-sweep.yml",
"name": "RustFS Validation Sweep",
@@ -350,24 +378,25 @@
"roles": [
"periodic",
"manual",
"release"
"reusable"
],
"events": [
"push",
"workflow_call",
"schedule",
"workflow_dispatch"
],
"cadence": "nightly",
"timezone": "UTC",
"schedule": [
"13 1 * * *"
"13 1 * * *",
"17 16 * * 0"
],
"owner": "@overtrue",
"policy": "active",
"monitor": "workflow",
"max_age_hours": 36,
"result_owner": "build.yml",
"disposition": "retained independent entry",
"disposition": "Daily cron builds Linux; Sunday 16:17 UTC builds macOS and Windows. Public release writes belong to release-publish.yml.",
"failure_owner": "workflow",
"origin": "baseline",
"alert_contract": "scheduled-validation"
@@ -745,14 +774,13 @@
},
{
"workflow": ".github/workflows/docker.yml",
"name": "Docker Images",
"name": "Build candidate images",
"roles": [
"manual",
"release"
"release",
"reusable"
],
"events": [
"workflow_run",
"workflow_dispatch"
"workflow_call"
],
"cadence": "event",
"timezone": "UTC",
@@ -761,10 +789,35 @@
"policy": "active",
"monitor": "none",
"result_owner": "docker.yml",
"disposition": "retained independent entry",
"disposition": "Builds artifacts only; public release writes belong to release-publish.yml.",
"monitor_reason": "Event-driven or operator-requested work has no periodic freshness obligation.",
"origin": "baseline"
},
{
"workflow": ".github/workflows/dual-stack-nightly.yml",
"name": "dual-stack-nightly",
"roles": [
"periodic",
"manual"
],
"events": [
"workflow_dispatch",
"schedule"
],
"cadence": "nightly",
"timezone": "UTC",
"schedule": [
"13 3 * * *"
],
"owner": "@overtrue",
"policy": "active",
"monitor": "workflow",
"max_age_hours": 36,
"result_owner": "dual-stack-nightly.yml",
"disposition": "both-stack JUnit diff of the two profiles only ci.yml runs; the gateway legs are report-only until T2.10",
"failure_owner": "watchdog",
"origin": "added"
},
{
"workflow": ".github/workflows/e2e-distributed.yml",
"name": "e2e-distributed",
@@ -870,6 +923,33 @@
"failure_owner": "watchdog",
"origin": "baseline"
},
{
"workflow": ".github/workflows/reference-parity.yml",
"name": "Reference Parity",
"roles": [
"pr",
"periodic",
"manual"
],
"events": [
"pull_request",
"schedule",
"workflow_dispatch"
],
"cadence": "nightly",
"timezone": "UTC",
"schedule": [
"37 1 * * *"
],
"owner": "@overtrue",
"policy": "active",
"monitor": "workflow",
"max_age_hours": 36,
"result_owner": "reference-parity.yml",
"disposition": "s3-tests per-node-id parity of branch candidates against the pinned reference release",
"failure_owner": "watchdog",
"origin": "added"
},
{
"workflow": ".github/workflows/functional-chain-health.yml",
"name": "Functional chain health",
@@ -925,6 +1005,31 @@
"origin": "baseline",
"alert_contract": "scheduled-validation"
},
{
"workflow": ".github/workflows/gateway-acceptance.yml",
"name": "Gateway Acceptance",
"roles": [
"periodic",
"manual"
],
"events": [
"schedule",
"workflow_dispatch"
],
"cadence": "nightly",
"timezone": "UTC",
"schedule": [
"29 3 * * *"
],
"owner": "@overtrue",
"policy": "active",
"monitor": "workflow",
"max_age_hours": 36,
"result_owner": "gateway-acceptance.yml",
"disposition": "gateway conformance corpus (rustfs profile) and client matrix against the candidate on both S3 stacks",
"failure_owner": "watchdog",
"origin": "added"
},
{
"workflow": ".github/workflows/gateway-integration-sync.yml",
"name": "Gateway Integration Sync",
@@ -954,14 +1059,13 @@
},
{
"workflow": ".github/workflows/helm-package.yml",
"name": "Publish helm chart to artifacthub",
"name": "Build candidate Helm chart",
"roles": [
"manual",
"release"
"release",
"reusable"
],
"events": [
"workflow_run",
"workflow_dispatch"
"workflow_call"
],
"cadence": "event",
"timezone": "UTC",
@@ -970,7 +1074,7 @@
"policy": "active",
"monitor": "none",
"result_owner": "helm-package.yml",
"disposition": "retained independent entry",
"disposition": "Builds artifacts only; public release writes belong to release-publish.yml.",
"monitor_reason": "Event-driven or operator-requested work has no periodic freshness obligation.",
"origin": "baseline"
},
@@ -996,6 +1100,26 @@
"policy_reason": "Preserve operator-disabled state; enabling requires a separate policy decision.",
"origin": "baseline"
},
{
"workflow": ".github/workflows/manual-main-packages.yml",
"name": "Manual Linux packages",
"roles": [
"manual"
],
"events": [
"workflow_dispatch"
],
"cadence": "event",
"timezone": "UTC",
"schedule": [],
"owner": "@overtrue",
"policy": "active",
"monitor": "none",
"result_owner": "manual-main-packages.yml",
"disposition": "Packages a git tag, or the origin/main tip when the ref input is empty, as workflow artifacts only. No GitHub Release or download-channel writes.",
"monitor_reason": "Operator-requested packaging has no periodic freshness obligation.",
"origin": "manual-main-packages"
},
{
"workflow": ".github/workflows/minio-interop.yml",
"name": "minio-interop",
@@ -1182,14 +1306,13 @@
},
{
"workflow": ".github/workflows/package.yml",
"name": "Package DEB/RPM",
"name": "Build candidate packages",
"roles": [
"manual",
"release"
"release",
"reusable"
],
"events": [
"workflow_run",
"workflow_dispatch"
"workflow_call"
],
"cadence": "event",
"timezone": "UTC",
@@ -1198,7 +1321,7 @@
"policy": "active",
"monitor": "none",
"result_owner": "package.yml",
"disposition": "retained independent entry",
"disposition": "Builds artifacts only; public release writes belong to release-publish.yml.",
"monitor_reason": "Event-driven or operator-requested work has no periodic freshness obligation.",
"origin": "baseline"
},
@@ -1727,6 +1850,69 @@
"disposition": "retained independent entry",
"monitor_reason": "Event-driven or operator-requested work has no periodic freshness obligation.",
"origin": "baseline"
},
{
"workflow": ".github/workflows/release-candidate.yml",
"name": "Release candidate (Linux)",
"roles": [
"manual",
"release"
],
"events": [
"workflow_dispatch"
],
"cadence": "event",
"timezone": "UTC",
"schedule": [],
"owner": "@overtrue",
"policy": "active",
"monitor": "none",
"result_owner": "release-candidate.yml",
"disposition": "Explicit immutable candidate stage",
"monitor_reason": "Operator-requested release stage has no periodic freshness obligation.",
"origin": "release-candidate-migration"
},
{
"workflow": ".github/workflows/release-complete.yml",
"name": "Release candidate (complete)",
"roles": [
"manual",
"release"
],
"events": [
"workflow_dispatch"
],
"cadence": "event",
"timezone": "UTC",
"schedule": [],
"owner": "@overtrue",
"policy": "active",
"monitor": "none",
"result_owner": "release-complete.yml",
"disposition": "Explicit immutable candidate stage",
"monitor_reason": "Operator-requested release stage has no periodic freshness obligation.",
"origin": "release-candidate-migration"
},
{
"workflow": ".github/workflows/release-publish.yml",
"name": "Publish accepted release",
"roles": [
"manual",
"release"
],
"events": [
"workflow_dispatch"
],
"cadence": "event",
"timezone": "UTC",
"schedule": [],
"owner": "@overtrue",
"policy": "active",
"monitor": "none",
"result_owner": "release-publish.yml",
"disposition": "Explicit immutable candidate stage",
"monitor_reason": "Operator-requested release stage has no periodic freshness obligation.",
"origin": "release-candidate-migration"
}
]
}
+12 -4
View File
@@ -5,7 +5,7 @@
## Billing facts
This repository is public. For a public repository, GitHub's standard hosted runners (`ubuntu-latest`, `windows-latest`, `macos-latest`) cost nothing and have no minute cap, whatever plan the organisation is on. GitHub bills only two things: larger hosted runners, which have custom labels and are not used here, and standard runners in private repositories.
This repository is public. For a public repository, GitHub's standard hosted runners (`ubuntu-latest`, `windows-latest`, `macos-latest`) cost nothing and have no minute cap, whatever plan the organisation is on. GitHub bills only two things: larger hosted runners, including the `macos-latest-xlarge` release lane used here, and standard runners in private repositories.
Every other label in this repository is capacity RustFS pays for. ARC pods and lab machines accrue cost while the machine exists. A Ubicloud label is metered for the minutes a job holds its VM:
@@ -13,6 +13,7 @@ Every other label in this repository is capacity RustFS pays for. ARC pods and l
|---|---|---|---|
| `ubuntu-latest` | nobody | GitHub-hosted, Ubuntu 24.04, 4 vCPU, 16 GB RAM, about 14 GB of free disk. Ships a C toolchain, `docker`, `gh`, `curl`, `wget`, `jq`, Python. Can `mount` tmpfs and loop devices; io_uring works | the default for every job |
| `windows-latest`, `macos-latest` | nobody | GitHub-hosted standard runners | Windows and macOS targets |
| `macos-latest-xlarge` | RustFS | GitHub-hosted larger Apple Silicon runner | macOS release builds, deferred until Linux acceptance for candidates |
| `sm-standard-2` | RustFS | ARC pod, Ubuntu 24.04, minimal image: no `gh`, no C toolchain; Docker only on the `dind-` label | only a job that a hosted runner cannot run (see the placement rule) |
| `sm-standard-4` | RustFS | ARC pod, Ubuntu 24.04, 14 CPUs and 28 GB (the label is not the core count), large disk | workspace-scale Cargo builds |
| `ubicloud-standard-16-arm` | RustFS | Ubicloud VM, alias of `ubicloud-standard-16-arm-ubuntu-2404`: Ubuntu 24.04 arm64, 16 vCPU, 48 GB, 300 GB disk. Reached through the `build.yml` matrix, so `actionlint.yaml` does not list it | native aarch64 release builds (`linux-aarch64-musl`, `linux-aarch64-gnu`) |
@@ -32,7 +33,7 @@ A job runs on `ubuntu-latest` unless it has one of these reasons, and the reason
Jobs that do only the following have no reason to be self-hosted: shell or Python scripts, `cargo fmt`, `cargo deny`, typos, actionlint, API calls (`gh`, dependency review, issue and stale bots, CLA checks), downloading an artifact and running a prebuilt binary, or a `cargo build -p rustfs` with its own cache key. `e2e-distributed.yml` and `e2e-upgrade.yml` build the server and the `e2e_test` harness on `ubuntu-latest` inside their budgets, so "it compiles something" alone is not a reason.
The functional chain's `prepare` and `complete-chain` jobs, the performance runner availability check, and runner hygiene use `ubuntu-latest`: they inspect APIs or artifacts, not the lab hosts. The actual functional and performance suites remain on their lab runners. Functional-chain health collection uses `ubuntu-latest` so a lab outage cannot also stop its independent monitor. Docker image packaging also uses `ubuntu-latest` for both musl and glibc variants; it downloads release binaries rather than compiling the workspace, and still sets up QEMU and Buildx explicitly.
The functional chain's `prepare` and `complete-chain` jobs, the performance runner availability check, and runner hygiene use `ubuntu-latest`: they inspect APIs or artifacts, not the lab hosts. The actual functional and performance suites remain on their lab runners. Functional-chain health collection uses `ubuntu-latest` so a lab outage cannot also stop its independent monitor. Docker image packaging also uses `ubuntu-latest` for both musl and glibc variants; it wraps verified candidate binaries rather than compiling the workspace, and still sets up QEMU and Buildx explicitly.
Moving these jobs off AKS reduces capacity demand, not necessarily the Azure bill immediately. VM costs fall only when nodes scale down or smaller capacity replaces them.
@@ -43,6 +44,7 @@ A hosted runner is also the only choice for a few things the pods cannot do: `mo
`Swatinem/rust-cache` builds its key from the OS, architecture, toolchain and lockfile. It does not include the runner label or image, so a cache written on an `sm-standard-4` pod and one written on `ubuntu-latest` would overwrite each other under the same `shared-key`.
- Every `shared-key` has exactly one writer. `cache-warm.yml` owns `ci-dev`, `ci-feat-rio`, `ci-feat-proto` and `ci-uring`; every other `./.github/actions/setup` call states `cache-save-if` explicitly, and `scripts/security/check_cache_save_if.sh` fails the build when one does not.
- The `ci-dev` key ends in a digest of `warm-ci-dev`'s build commands. Changing what that job warms therefore needs a new key, or readers keep restoring the old artifacts: `scripts/ci_gate.py --check-workflow` prints the key the commands require and fails until the writer and every reader in `ci.yml` and `reference-parity.yml` name it.
- All readers of a key run on the same image as its writer. Keys that live on hosted runners say so in their name (`ci-uring` is warmed on `ubuntu-latest`; `ci-e2e-distributed-hosted`, `e2e-upgrade-server-hosted`, `e2e-upgrade-tests-hosted`).
- Moving a job between pools therefore means renaming its key. The new key is empty until a run from `main` (a scheduled run, or a manual dispatch of that workflow from `main`) saves it; until then that lane does a cold build. The `build.yml` aarch64 release legs append `-ubicloud-arm` for this reason. `cache-warm.yml` does not write those keys.
- `CARGO_BUILD_JOBS` and every other `CARGO*`, `CC*`, `CFLAGS*`, `CXX*`, `CMAKE*` and `RUST*` variable is hashed into the key. `ci.yml` sets `CARGO_BUILD_JOBS` at step level for that reason; promoting it to job level rotates the key for every lane that shares it.
@@ -52,7 +54,7 @@ A hosted runner is also the only choice for a few things the pods cannot do: `mo
These are the patterns that have cost real money here without validating anything more:
- An `if: always()` summary or alert job on a self-hosted label inside a `workflow_run`-triggered workflow. The upstream workflow completes many times a day, the real jobs skip, and the summary still starts a pod each time. Gate it on the upstream job having run, and keep it on `ubuntu-latest`.
- Per-push workflows on `main`. A merge triggers the full `ci.yml` matrix, `cache-warm.yml` and the `build.yml` development build; with tens of merges a day that is as much self-hosted time as all pull requests together. Anything that does not verify the merged tree itself belongs on a schedule or behind a `paths` filter.
- Per-push workflows on `main`. A merge triggers cache warming and a `ci.yml` run, which validates the merged tree in full unless a merge-queue run already validated that tree; `build.yml` development builds are scheduled or manual; with tens of merges a day that is as much self-hosted time as all pull requests together. Anything that does not verify the merged tree itself belongs on a schedule or behind a `paths` filter.
- `apt-get install` of tools the hosted image already ships. It is only needed on the minimal pod images.
- A throttled `CARGO_BUILD_JOBS` on a 14-CPU pod. The limit exists for link-storm reasons documented in `ci.yml`; measure before raising it, and measure before lowering it further.
- Tiny jobs on ARC pods. Each one pays roughly a minute of pod start-up before its first step; a compile-free job that takes ten seconds costs more to schedule than to run.
@@ -66,7 +68,7 @@ gh api "repos/rustfs/rustfs/actions/runs/<run_id>/jobs?per_page=100" \
--jq '.jobs[] | "\(.conclusion)\t\(.labels | join(","))\t\(.started_at)\t\(.completed_at)\t\(.name)"'
```
Only rows whose label starts with `sm-`, `dind-`, `ubicloud-`, `smoke-` or `pf-` cost anything. At the October 2026 review, a full pull-request run of `ci.yml` spent about 150 self-hosted minutes, a third of it in jobs that never compiled; treat that as the scale of what a placement mistake costs, not as a current figure.
Rows using `macos-latest-xlarge` or a label starting with `sm-`, `dind-`, `ubicloud-`, `smoke-` or `pf-` use paid capacity. At the October 2026 review, a full pull-request run of `ci.yml` spent about 150 self-hosted minutes, a third of it in jobs that never compiled; treat that as the scale of what a placement mistake costs, not as a current figure.
## Checklist for a new or moved job
@@ -76,3 +78,9 @@ Only rows whose label starts with `sm-`, `dind-`, `ubicloud-`, `smoke-` or `pf-`
- A self-hosted `runs-on` carries a comment naming its reason from the placement rule.
- Every `./.github/actions/setup` call states `cache-save-if`, and a key read from one pool is not written from the other.
- Moving a job does not change its check name, so the `main` ruleset and `docs/testing/ci-gates.md` stay as they are; renaming it does, and then that document's change checklist applies.
## Release stages
The [candidate release pipeline](../../docs/operations/release-pipeline.md) builds Linux first and defers paid desktop capacity until acceptance. The daily `build.yml` schedule selects Linux; the Sunday 16:17 UTC schedule selects macOS and Windows. Manual dispatch can still select any platform list. All six platforms remain required before publication. The pipeline records and promotes the exact tested bytes. Builder workflows have no public-release credentials. Keep periodic development builds, PR verification and field-test chains separate when their contracts differ; do not re-enable disabled workflows as part of a source-file migration.
Larger runners remain billed for public repositories; see [GitHub runner billing](https://docs.github.com/en/actions/concepts/runners/larger-runners#billing). Actual spending comes from organization billing, not runner labels or an estimated build duration.
+149 -8
View File
@@ -23,9 +23,16 @@ on:
- 'deny.toml'
- '.github/actions/**'
- '.github/workflows/**'
- 'scripts/release/create_or_update_release.sh'
- 'scripts/release/package_service_scripts.sh'
- 'scripts/release/package_versions.sh'
- 'scripts/release/**'
- 'scripts/security/generate_release_supply_chain_assets.py'
- '.config/release-validation-plan.json'
- '.github/scheduled-validations.json'
- 'scripts/test_helm_release_workflow.py'
- 'scripts/test_release_packages.py'
- 'scripts/test_release_images.py'
- 'Dockerfile*'
- 'entrypoint.sh'
- 'bootstrap/**'
- 'scripts/test_package_service_scripts.sh'
- 'scripts/test_package_versions.sh'
- 'scripts/test_docker_workflow.py'
@@ -41,9 +48,16 @@ on:
- 'deny.toml'
- '.github/actions/**'
- '.github/workflows/**'
- 'scripts/release/create_or_update_release.sh'
- 'scripts/release/package_service_scripts.sh'
- 'scripts/release/package_versions.sh'
- 'scripts/release/**'
- 'scripts/security/generate_release_supply_chain_assets.py'
- '.config/release-validation-plan.json'
- '.github/scheduled-validations.json'
- 'scripts/test_helm_release_workflow.py'
- 'scripts/test_release_packages.py'
- 'scripts/test_release_images.py'
- 'Dockerfile*'
- 'entrypoint.sh'
- 'bootstrap/**'
- 'scripts/test_package_service_scripts.sh'
- 'scripts/test_package_versions.sh'
- 'scripts/test_docker_workflow.py'
@@ -132,12 +146,38 @@ jobs:
if: github.repository == 'rustfs/rustfs' && (github.event_name != 'pull_request' || github.event.action != 'closed')
runs-on: ubuntu-latest
timeout-minutes: 5
permissions:
contents: read
actions: read
outputs:
release_changed: ${{ steps.scope.outputs.release_changed }}
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Scope native packaging checks to release changes
id: scope
env:
BASE_SHA: ${{ github.event.pull_request.base.sha || github.event.before }}
run: |
python3 - <<'PY'
import os, re, subprocess
event, base = os.environ['GITHUB_EVENT_NAME'], os.environ.get('BASE_SHA', '')
changed = event == 'workflow_dispatch'
if event in ('push', 'pull_request'):
if not re.fullmatch(r'[0-9a-f]{40}', base) or base == '0' * 40:
changed = True
else:
paths = subprocess.check_output(['git', 'diff', '--name-only', base, os.environ['GITHUB_SHA']], text=True).splitlines()
pattern = r'^(scripts/release/|scripts/test_release_|Dockerfile|entrypoint.sh$|bootstrap/|\.dockerignore$|\.config/release-validation-plan.json$|\.github/workflows/(audit|build|docker|package|helm-package|release-[^.]+)\.yml$)'
changed = any(re.match(pattern, path) for path in paths)
with open(os.environ['GITHUB_OUTPUT'], 'a') as output:
output.write(f'release_changed={str(changed).lower()}\n')
PY
- name: Report unpinned GitHub Actions
run: ./scripts/security/check_workflow_pins.sh --enforce
@@ -150,8 +190,35 @@ jobs:
- name: Check checkouts clear their credentials
run: ./scripts/security/check_persist_credentials.sh
- name: Check preview release workflow policy
run: ./scripts/security/check_preview_release_workflow.sh
- name: Check release candidate and promotion contracts
run: python3 -m unittest discover -s scripts/release -p 'test_*.py'
- name: Prepare artifact transport rehearsal
run: |
mkdir -p "$RUNNER_TEMP/release-transport"
printf '{"purpose":"transport contract, not release evidence"}\n' > "$RUNNER_TEMP/release-transport/transport.json"
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: release-transport-contract-${{ github.run_attempt }}
path: ${{ runner.temp }}/release-transport/transport.json
if-no-files-found: error
retention-days: 1
- name: Verify actual GitHub artifact metadata and downloaded bytes
env:
GH_TOKEN: ${{ github.token }}
run: |
python3 - <<'PY'
import os, sys
from pathlib import Path
sys.path.insert(0, 'scripts/release')
import candidate as c
run = c.github(f"repos/{c.REPOSITORY}/actions/runs/{os.environ['GITHUB_RUN_ID']}")
destination = Path(os.environ['RUNNER_TEMP']) / 'release-transport-downloaded'
c.download_artifact(run, f"release-transport-contract-{os.environ['GITHUB_RUN_ATTEMPT']}", destination)
c.require(c.read_json(destination / 'transport.json') == {'purpose': 'transport contract, not release evidence'}, 'artifact round trip changed')
PY
- name: Check tier evidence workflow isolation
run: ./scripts/security/check_tier_artifact_workflow.sh
@@ -162,6 +229,80 @@ jobs:
- name: Check package service lifecycle contract
run: ./scripts/test_package_service_scripts.sh
release-package-contract:
name: Release package contract
needs: workflow-pin-report
if: needs.workflow-pin-report.outputs.release_changed == 'true'
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Install real package managers
run: |
sudo apt-get update
sudo apt-get install -y ruby ruby-dev build-essential rpm fakeroot cpio
sudo gem install fpm --version 1.17.0 --no-document
- name: Exercise accepted and substituted payloads through both recipes
run: python3 scripts/test_release_packages.py
release-image-contract:
name: Release image contract (${{ matrix.variant }})
needs: workflow-pin-report
if: needs.workflow-pin-report.outputs.release_changed == 'true'
runs-on: ubuntu-latest
timeout-minutes: 35
strategy:
fail-fast: false
matrix:
variant: [musl, gnu]
env:
RELEASE_IMAGE_VARIANT: ${{ matrix.variant }}
NO_PROXY: 127.0.0.1,localhost,::1
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Install OCI tools
run: |
echo "RELEASE_IMAGE_OUTPUT=$RUNNER_TEMP/image-contract" >> "$GITHUB_ENV"
echo 'no_proxy=127.0.0.1,localhost,::1' >> "$GITHUB_ENV"
sudo apt-get update
sudo apt-get install -y skopeo
go version
- uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 # v3
- uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
- name: Exercise real OCI packaging, startup, registry retry and conflicts
run: python3 scripts/test_release_images.py
- name: Exercise amd64 OCI archive scanning
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25
env:
TRIVY_PLATFORM: linux/amd64
with:
input: ${{ runner.temp }}/image-contract-scan/image-${{ matrix.variant }}-amd64.tar
format: sarif
output: ${{ runner.temp }}/image-contract/scan-${{ matrix.variant }}-amd64.sarif
exit-code: "0"
- name: Exercise arm64 OCI archive scanning
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25
env:
TRIVY_PLATFORM: linux/arm64
with:
input: ${{ runner.temp }}/image-contract-scan/image-${{ matrix.variant }}-arm64.tar
format: sarif
output: ${{ runner.temp }}/image-contract/scan-${{ matrix.variant }}-arm64.sarif
exit-code: "0"
- name: Verify scan receipts before recording the archive
run: |
python3 - <<'PY'
import os, sys
from pathlib import Path
sys.path.insert(0, 'scripts/release')
import deliver, test_candidate
deliver.image_record(test_candidate.inputs(), os.environ['RELEASE_IMAGE_VARIANT'], Path(os.environ['RELEASE_IMAGE_OUTPUT']))
PY
dependency-review:
name: Dependency Review
runs-on: ubuntu-latest
+215 -471
View File
@@ -12,51 +12,34 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# Build and Release Workflow
#
# This workflow builds RustFS binaries and automatically triggers Docker image builds.
#
# Flow:
# 1. Build binaries for multiple platforms
# 2. Upload binaries to OSS storage
# 3. Trigger docker.yml to build and push images using the uploaded binaries
#
# Platform scope:
# - Development builds come from the nightly schedule and manual dispatch and
# cover the full platform matrix; they feed the dev download channel
# (artifacts/rustfs/dev on R2, the *-main-latest packages). Tags build the
# release matrix. Main pushes do not build: at the usual merge rate most
# per-merge builds were cancelled by the next merge before publishing
# anything, and the dev channel does not need per-merge granularity.
#
# Manual Parameters:
# - publish: Publish binaries/releases for manual runs (default: false)
# - build_docker: Advisory only; manual runs do not trigger Docker publishing
# - platforms: Comma-separated platform IDs or 'all' (default: all)
# Scheduled development artifacts and reusable immutable-candidate builds.
# The daily cron builds Linux. Sunday 16:17 UTC builds macOS and Windows.
# Public release writes live exclusively in release-publish.yml. Candidate callers
# select Linux or desktop and pin release inputs before any matrix starts.
name: Build and Release
on:
push:
tags: [ "*.*.*" ]
workflow_call:
inputs:
candidate_manifest:
description: Frozen candidate inputs; candidates never publish from this workflow
required: true
type: string
platforms:
description: linux, desktop, all, or an explicit platform list
required: true
type: string
schedule:
- cron: "13 1 * * *" # Nightly 01:13 UTC
- cron: "13 1 * * *" # Daily Linux, 01:13 UTC
- cron: "17 16 * * 0" # Weekly macOS and Windows, Sunday 16:17 UTC
workflow_dispatch:
inputs:
publish:
description: "Publish to download channels and GitHub Releases (false keeps only workflow artifacts)"
description: "Publish the development download channel (never a release)"
required: false
default: false
type: boolean
build_docker:
# Advisory only. docker.yml triggers on workflow_run and its job-level
# condition requires the triggering event to be a tag push, so a manual
# dispatch of this workflow never produces images regardless of this
# value. Publishing remains controlled by the explicit publish input.
description: "Build and push Docker images after binary build (ignored: dispatch runs never reach docker.yml)"
required: false
default: true
type: boolean
platforms:
description: "Comma-separated targets or 'all' (e.g. linux-x86_64-musl,macos-aarch64)"
required: false
@@ -65,10 +48,11 @@ on:
permissions:
contents: read
actions: read
concurrency:
group: ${{ github.workflow }}-${{ github.event_name == 'push' && github.ref || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'push' }}
cancel-in-progress: false
env:
CARGO_TERM_COLOR: always
@@ -84,19 +68,39 @@ jobs:
timeout-minutes: 30
outputs:
should_build: ${{ steps.check.outputs.should_build }}
should_publish: ${{ github.event_name != 'workflow_dispatch' || inputs.publish == true }}
should_publish: ${{ inputs.candidate_manifest == '' && github.ref == 'refs/heads/main' && (github.event_name == 'schedule' || inputs.publish == true) }}
build_type: ${{ steps.check.outputs.build_type }}
version: ${{ steps.check.outputs.version }}
short_sha: ${{ steps.check.outputs.short_sha }}
is_prerelease: ${{ steps.check.outputs.is_prerelease }}
candidate_id: ${{ steps.candidate.outputs.candidate_id }}
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
ref: ${{ inputs.candidate_manifest != '' && fromJson(inputs.candidate_manifest).source_sha || github.sha }}
- name: Validate frozen candidate
if: inputs.candidate_manifest != ''
id: candidate
env:
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
run: |
python3 - <<'PYINPUT'
import json, os, sys
sys.path.insert(0, 'scripts/release')
import candidate
value = candidate.validate_inputs(json.loads(os.environ['CANDIDATE_MANIFEST']))
candidate.write_json('candidate-inputs.json', value)
candidate.emit_output('candidate_id', value['candidate_id'])
PYINPUT
python3 scripts/release/candidate.py validate-inputs --inputs candidate-inputs.json --check-checkout
- name: Determine build strategy
id: check
env:
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
run: |
should_build=false
build_type="none"
@@ -107,8 +111,14 @@ jobs:
# Get short SHA for all builds
short_sha=$(git rev-parse --short HEAD)
# Candidates contain the final product version but have no release
# authority. Their only publication is immutable workflow artifacts.
if [[ -n "$CANDIDATE_MANIFEST" ]]; then
should_build=true
build_type="candidate"
version=$(python3 -c 'import json; print(json.load(open("candidate-inputs.json"))["version"])')
# Determine build type based on trigger
if [[ "${{ startsWith(github.ref, 'refs/tags/') }}" == "true" ]]; then
elif [[ "${{ startsWith(github.ref, 'refs/tags/') }}" == "true" ]]; then
# Tag push - preview, release, or prerelease
should_build=true
tag_name="${GITHUB_REF#refs/tags/}"
@@ -155,6 +165,46 @@ jobs:
echo " - Short SHA: $short_sha"
echo " - Is prerelease: $is_prerelease"
prepare-build-inputs:
needs: build-check
if: needs.build-check.outputs.should_build == 'true'
runs-on: ubuntu-latest
timeout-minutes: 15
outputs:
console: ${{ steps.inputs.outputs.console }}
toolchain: ${{ steps.inputs.outputs.toolchain }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
ref: ${{ inputs.candidate_manifest != '' && fromJson(inputs.candidate_manifest).source_sha || github.sha }}
- name: Resolve one Console and toolchain identity
id: inputs
env:
GH_TOKEN: ${{ github.token }}
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
shell: bash
run: |
set -euo pipefail
if [[ -z "$CANDIDATE_MANIFEST" ]]; then
gh api repos/rustfs/console/releases/latest > console-release.json
rustup update stable --no-self-update
fi
python3 - <<'PYINPUT'
import json, os, subprocess, sys
sys.path.insert(0, 'scripts/release')
import candidate
raw = os.environ['CANDIDATE_MANIFEST']
if raw:
frozen = candidate.validate_inputs(json.loads(raw))
console, toolchain = frozen['console'], frozen['toolchain']
else:
console = candidate.console_asset(candidate.read_json('console-release.json'))
toolchain = subprocess.check_output(['rustc', '+stable', '--version'], text=True).split()[1]
candidate.emit_output('console', json.dumps(console, separators=(',', ':')))
candidate.emit_output('toolchain', toolchain)
PYINPUT
# Build RustFS binaries
prepare-platform-matrix:
name: Prepare Platform Matrix
@@ -171,14 +221,33 @@ jobs:
env:
# via env, not interpolation: a dispatch input is free-form text and
# would otherwise be pasted into the script for bash to evaluate.
RAW_PLATFORMS: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.platforms || 'all' }}
# A direct schedule has no platforms input. Choose its subset from
# github.event.schedule, which is fixed when that cron fires, so a
# queued job cannot switch subsets because the runner clock moved.
# Callers that pass platforms keep that list, including a scheduled
# caller: inputs.platforms is set even though event_name stays schedule.
RAW_PLATFORMS: ${{ inputs.platforms || 'linux' }}
HAS_PLATFORM_INPUT: ${{ inputs.platforms && 'true' || 'false' }}
EVENT_NAME: ${{ github.event_name }}
EVENT_SCHEDULE: ${{ github.event.schedule }}
run: |
set -euo pipefail
selected="$RAW_PLATFORMS"
selected="$(echo "${selected}" | tr -d '[:space:]')"
if [[ -z "${selected}" ]]; then
selected="all"
if [[ "${EVENT_NAME}" == "schedule" && "${HAS_PLATFORM_INPUT}" != "true" ]]; then
if [[ "${EVENT_SCHEDULE}" == "13 1 * * *" ]]; then
selected="linux"
elif [[ "${EVENT_SCHEDULE}" == "17 16 * * 0" ]]; then
selected="desktop"
else
echo "Unrecognized schedule cron: ${EVENT_SCHEDULE}" >&2
exit 1
fi
else
selected="${RAW_PLATFORMS}"
selected="$(echo "${selected}" | tr -d '[:space:]')"
if [[ -z "${selected}" ]]; then
selected="all"
fi
fi
# linux-aarch64-* need a real aarch64 host (placement rule: hardware)
@@ -197,6 +266,10 @@ jobs:
{"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":"","features":"","glibc":""}
]}'
if [[ "$selected" == "linux" || "$selected" == "desktop" ]]; then
selected=$(jq -r --arg group "$selected" '.include[] | select(if $group == "linux" then .platform == "linux" else .platform != "linux" end) | .target_id' <<<"$all" | paste -sd ',' -)
fi
if [[ "${selected}" == "all" ]]; then
matrix="$(jq -c . <<<"${all}")"
else
@@ -224,12 +297,13 @@ jobs:
build-rustfs:
# Acceptance workflows select artifacts by this stable platform name.
name: Build RustFS (${{ matrix.target_id }})
needs: [ build-check, prepare-platform-matrix ]
needs: [ build-check, prepare-platform-matrix, prepare-build-inputs ]
if: needs.build-check.outputs.should_build == 'true' && needs.prepare-platform-matrix.result == 'success'
runs-on: ${{ matrix.os }}
timeout-minutes: 180
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
RUSTUP_TOOLCHAIN: ${{ needs.prepare-build-inputs.outputs.toolchain }}
# Release binaries ship without dial9 telemetry and therefore do not need
# --cfg tokio_unstable. Telemetry builds opt in explicitly (see
# `make build-profiling`); crates/obs/build.rs fails the compile if the
@@ -244,11 +318,33 @@ jobs:
with:
persist-credentials: false
fetch-depth: 0
ref: ${{ inputs.candidate_manifest != '' && fromJson(inputs.candidate_manifest).source_sha || github.sha }}
- name: Ensure Python for artifact verification
if: runner.os == 'Linux'
shell: bash
run: |
if ! command -v python3 >/dev/null 2>&1; then
sudo apt-get update
sudo apt-get install -y python3
fi
- name: Reuse completed candidate platform
id: reuse
if: needs.build-check.outputs.build_type == 'candidate'
env:
GH_TOKEN: ${{ github.token }}
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
TARGET: ${{ matrix.target_id }}
shell: bash
run: |
python3 scripts/release/pipeline.py reuse-platform --target "$TARGET"
- name: Setup Rust environment
if: steps.reuse.outputs.reused != 'true'
uses: ./.github/actions/setup
with:
rust-version: stable
rust-version: ${{ needs.prepare-build-inputs.outputs.toolchain }}
target: ${{ matrix.target }}
# rust-cache keys include OS and architecture but not the runner image.
# The Ubicloud aarch64 legs therefore use their own suffix so they
@@ -268,128 +364,35 @@ jobs:
install-test-tools: 'false'
- name: Download static console assets
if: steps.reuse.outputs.reused != 'true'
shell: bash
env:
GITHUB_TOKEN: ${{ github.token }}
GH_TOKEN: ${{ github.token }}
CONSOLE_IDENTITY: ${{ needs.prepare-build-inputs.outputs.console }}
run: |
mkdir -p ./rustfs/static
verify_sha256() {
local expected="$1"
local file="$2"
local actual=""
local actual_line=""
if command -v sha256sum >/dev/null 2>&1; then
actual_line=$(sha256sum -- "$file") || return 1
elif command -v shasum >/dev/null 2>&1; then
actual_line=$(shasum -a 256 -- "$file") || return 1
else
echo "No SHA-256 verification tool found" >&2
return 1
fi
actual="${actual_line%%[[:space:]]*}"
if [ "$actual" = "$expected" ]; then
echo "SHA256 verified OK: $actual"
return 0
else
echo "SHA256 mismatch: expected=$expected actual=$actual" >&2
return 1
fi
}
download_console_assets() {
local curl_bin="curl"
local python_bin="python3"
local console_api="https://api.github.com/repos/rustfs/console/releases/latest"
local console_json="console-release.json"
local console_url
local console_sha256
local curl_auth_args=()
console_tag=""
if [[ "${{ matrix.platform }}" == "windows" ]]; then
curl_bin="curl.exe"
else
chmod +w ./rustfs/static/LICENSE || true
fi
if ! command -v "$python_bin" >/dev/null 2>&1; then
python_bin="python"
fi
if ! command -v "$python_bin" >/dev/null 2>&1; then
echo "No Python interpreter found for release metadata parsing" >&2
return 1
fi
if [[ -n "${GITHUB_TOKEN:-}" ]]; then
curl_auth_args=(-H "Authorization: Bearer ${GITHUB_TOKEN}" -H "X-GitHub-Api-Version: 2022-11-28")
fi
"$curl_bin" "${curl_auth_args[@]}" --fail -L "$console_api" \
-o "$console_json" --retry 3 --retry-delay 5 --max-time 300 || return 1
read -r console_tag console_url console_sha256 < <("$python_bin" - "$console_json" <<'PY'
import json
import re
import sys
with open(sys.argv[1], encoding="utf-8") as handle:
release = json.load(handle)
tag = release.get("tag_name", "") or "unknown"
for asset in release.get("assets", []):
name = asset.get("name", "")
digest = asset.get("digest", "")
url = asset.get("browser_download_url", "")
if name.startswith("rustfs-console-") and name.endswith(".zip") and digest.startswith("sha256:") and url:
sha256 = digest.split(":", 1)[1]
if not re.fullmatch(r"[0-9a-fA-F]{64}", sha256):
raise SystemExit(f"console zip asset has invalid sha256 digest: {sha256}")
sys.stdout.buffer.write(f"{tag} {url} {sha256}\n".encode("utf-8"))
break
else:
raise SystemExit("no console zip asset with sha256 digest found")
PY
) || return 1
if [[ -z "$console_url" || -z "$console_sha256" ]]; then
echo "Console release metadata is missing URL or SHA-256 digest" >&2
return 1
fi
echo "Console release: ${console_tag}"
echo "Downloading console asset: ${console_url}"
"$curl_bin" --fail -L "$console_url" -o console.zip --retry 3 --retry-delay 5 --max-time 300 || return 1
verify_sha256 "$console_sha256" console.zip || return 2
unzip -o console.zip -d ./rustfs/static || return 2
}
if download_console_assets; then
rm -f console.zip console-release.json
else
status=$?
rm -f console.zip console-release.json
if [[ "$status" -eq 2 ]]; then
echo "Console asset integrity verification failed" >&2
fi
echo "Failed to download verified console assets" >&2
exit 1
fi
if [[ ! -s ./rustfs/static/index.html ]]; then
echo "Console asset archive is missing static/index.html" >&2
exit 1
fi
asset_count=$(find ./rustfs/static -type f | wc -l | tr -d '[:space:]')
echo "Console assets ready: version=${console_tag:-unknown}, ${asset_count} files extracted to ./rustfs/static"
python3 - <<'PYCONSOLE'
import json, os, pathlib, subprocess, zipfile
value = json.loads(os.environ['CONSOLE_IDENTITY'])
subprocess.run(['python3', '.github/actions/setup/download_release_asset.py',
'--repository', 'rustfs/console', '--tag', value['tag'], '--asset', value['name'],
'--sha256', value['sha256'], '--output', 'console.zip'], check=True)
destination = pathlib.Path('rustfs/static')
destination.mkdir(parents=True, exist_ok=True)
with zipfile.ZipFile('console.zip') as archive:
for member in archive.infolist():
path = pathlib.PurePosixPath(member.filename)
if path.is_absolute() or '..' in path.parts or '\\' in member.filename:
raise SystemExit('unsafe Console archive member')
archive.extractall(destination)
if not (destination / 'index.html').is_file():
raise SystemExit('Console archive has no index.html')
PYCONSOLE
- name: Build RustFS
if: steps.reuse.outputs.reused != 'true'
shell: bash
env:
RUSTFS_BUILD_VERSION: ${{ needs.build-check.outputs.build_type == 'candidate' && needs.build-check.outputs.version || '' }}
BUILD_TARGET: ${{ matrix.target }}
BUILD_OS: ${{ matrix.platform }}-${{ startsWith(matrix.target, 'aarch64-') && 'aarch64' || 'x86_64' }}
run: |
@@ -440,7 +443,7 @@ jobs:
fi
- name: Verify macOS binary has no dynamic liblzma dependency
if: matrix.platform == 'macos'
if: steps.reuse.outputs.reused != 'true' && (matrix.platform == 'macos')
shell: bash
run: |
binary="target/${{ matrix.target }}/release/rustfs"
@@ -451,7 +454,7 @@ jobs:
"$binary" --version
- name: Generate the CPU symbol catalog
if: endsWith(matrix.target, '-linux-gnu') && matrix.features == 'pyroscope'
if: steps.reuse.outputs.reused != 'true' && (endsWith(matrix.target, '-linux-gnu') && matrix.features == 'pyroscope')
shell: bash
run: |
cargo metadata --locked --format-version 1 --filter-platform '${{ matrix.target }}' \
@@ -468,6 +471,7 @@ jobs:
"$release_dir/rustfs.cpu-symbol-catalog.json" > "$release_dir/rustfs.cpu-symbol-catalog.sha256"
- name: Create release package
if: steps.reuse.outputs.reused != 'true'
id: package
shell: bash
run: |
@@ -668,7 +672,7 @@ jobs:
echo "📊 Version: ${VERSION}"
- name: Verify packaged Linux binary
if: matrix.platform == 'linux'
if: steps.reuse.outputs.reused != 'true' && (matrix.platform == 'linux')
shell: bash
run: |
set -euo pipefail
@@ -697,7 +701,7 @@ jobs:
"$binary_dir/rustfs" --version
- name: Verify packaged console
if: matrix.target == 'x86_64-unknown-linux-gnu' || matrix.target == 'aarch64-unknown-linux-gnu'
if: steps.reuse.outputs.reused != 'true' && (matrix.target == 'x86_64-unknown-linux-gnu' || matrix.target == 'aarch64-unknown-linux-gnu')
shell: bash
run: |
set -euo pipefail
@@ -750,15 +754,49 @@ jobs:
cat "$log_file"
exit 1
- name: Verify native candidate and retain receipts
if: needs.build-check.outputs.build_type == 'candidate' && steps.reuse.outputs.reused != 'true'
env:
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
TARGET: ${{ matrix.target_id }}
ARCHIVE: ${{ steps.package.outputs.package_file }}
shell: bash
run: |
python3 -c 'import os; from pathlib import Path; Path("inputs.json").write_text(os.environ["CANDIDATE_MANIFEST"])'
python3 scripts/release/verify.py native --inputs inputs.json --archive "$ARCHIVE" --target "$TARGET" --output .
- name: Record candidate artifact identity
if: steps.reuse.outputs.reused != 'true' && (needs.build-check.outputs.build_type == 'candidate')
env:
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
TARGET_ID: ${{ matrix.target_id }}
PACKAGE_FILE: ${{ steps.package.outputs.package_file }}
shell: bash
run: |
python3 - <<'PYRECORD'
import json, os, sys
sys.path.insert(0, 'scripts/release')
import candidate
value = candidate.validate_inputs(json.loads(os.environ['CANDIDATE_MANIFEST']))
record = candidate.archive_record(os.environ['PACKAGE_FILE'], os.environ['TARGET_ID'], value)
candidate.write_json('record-' + os.environ['TARGET_ID'] + '.json', record)
PYRECORD
- name: Upload to GitHub artifacts
if: steps.reuse.outputs.reused != 'true'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: ${{ steps.package.outputs.package_name }}
path: "rustfs-*.zip"
retention-days: ${{ startsWith(github.ref, 'refs/tags/') && 30 || 7 }}
name: ${{ needs.build-check.outputs.build_type == 'candidate' && format('candidate-{0}-{1}', needs.build-check.outputs.candidate_id, matrix.target_id) || steps.package.outputs.package_name }}
path: |
rustfs-*.zip
record-${{ matrix.target_id }}.json
identity-${{ matrix.target_id }}*
startup-${{ matrix.target_id }}.log
if-no-files-found: error
retention-days: ${{ (startsWith(github.ref, 'refs/tags/') || needs.build-check.outputs.build_type == 'candidate') && 30 || 7 }}
- name: Upload to Cloudflare R2
if: needs.build-check.outputs.should_publish == 'true' && env.R2_ACCESS_KEY_ID != '' && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease' || needs.build-check.outputs.build_type == 'development')
if: steps.reuse.outputs.reused != 'true' && (needs.build-check.outputs.should_publish == 'true' && env.R2_ACCESS_KEY_ID != '' && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease' || needs.build-check.outputs.build_type == 'development'))
env:
R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
@@ -814,13 +852,18 @@ jobs:
export AWS_SECRET_ACCESS_KEY="$R2_SECRET_ACCESS_KEY"
export AWS_DEFAULT_REGION="auto"
# Determine upload path based on build type
# Determine upload path based on build type. Release and prerelease
# builds only stage their bytes here: the official
# artifacts/rustfs/release/ path is written solely by the
# certification-gated promote-release-artifacts job, from these same
# staged objects. Development builds keep the uncertified dev channel
# and never touch official or latest channels.
if [[ "$BUILD_TYPE" == "development" ]]; then
R2_PATH="s3://${R2_BUCKET}/artifacts/rustfs/dev/"
echo "📤 Uploading development build to R2 dev directory"
else
R2_PATH="s3://${R2_BUCKET}/artifacts/rustfs/release/"
echo "📤 Uploading release build to R2 release directory"
R2_PATH="s3://${R2_BUCKET}/artifacts/rustfs/staging/${GITHUB_RUN_ID}/${GITHUB_RUN_ATTEMPT}/"
echo "📤 Staging release build for certification-gated promotion"
fi
# Upload all rustfs zip files to R2 using S3-compatible API
@@ -835,6 +878,7 @@ jobs:
echo "✅ Upload completed successfully"
# Build summary
build-summary:
name: Build Summary
@@ -869,309 +913,9 @@ jobs:
echo "This run does not trigger Docker publishing."
fi
# Create GitHub Release for every valid release tag, including previews
create-release:
name: Create GitHub Release
needs: [ build-check, build-rustfs ]
if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
permissions:
contents: write
outputs:
release_id: ${{ steps.create.outputs.release_id }}
release_url: ${{ steps.create.outputs.release_url }}
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
fetch-depth: 0
- name: Create GitHub Release
id: create
env:
GH_TOKEN: ${{ github.token }}
shell: bash
run: |
TAG="${{ needs.build-check.outputs.version }}"
VERSION="${{ needs.build-check.outputs.version }}"
IS_PRERELEASE="${{ needs.build-check.outputs.is_prerelease }}"
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
TARGET_COMMITISH=$(git rev-parse --verify "refs/tags/${TAG}^{commit}")
# Determine release type for title
if [[ "$BUILD_TYPE" == "preview" ]]; then
RELEASE_TYPE="preview"
elif [[ "$BUILD_TYPE" == "prerelease" ]]; then
if [[ "$TAG" == *"alpha"* ]]; then
RELEASE_TYPE="alpha"
elif [[ "$TAG" == *"beta"* ]]; then
RELEASE_TYPE="beta"
elif [[ "$TAG" == *"rc"* ]]; then
RELEASE_TYPE="rc"
else
RELEASE_TYPE="prerelease"
fi
else
RELEASE_TYPE="release"
fi
# Create release title
if [[ "$IS_PRERELEASE" == "true" ]]; then
TITLE="RustFS $VERSION (${RELEASE_TYPE})"
else
TITLE="RustFS $VERSION"
fi
./scripts/release/create_or_update_release.sh \
"$TAG" \
"$TARGET_COMMITISH" \
"$TITLE" \
"$IS_PRERELEASE"
# Prepare and upload release assets
upload-release-assets:
name: Upload Release Assets
needs: [ build-check, build-rustfs, create-release ]
if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
permissions:
contents: write
actions: read
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download all build artifacts
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
path: ./artifacts
pattern: rustfs-*
merge-multiple: true
- name: Prepare release assets
id: prepare
shell: bash
run: |
VERSION="${{ needs.build-check.outputs.version }}"
TAG="${{ needs.build-check.outputs.version }}"
mkdir -p ./release-assets
# Copy and verify artifacts (including latest files created during build)
ASSETS_COUNT=0
for file in ./artifacts/*.zip; do
if [[ -f "$file" ]]; then
cp "$file" ./release-assets/
ASSETS_COUNT=$((ASSETS_COUNT + 1))
fi
done
if [[ $ASSETS_COUNT -eq 0 ]]; then
echo "❌ No artifacts found!"
exit 1
fi
cd ./release-assets
# Generate checksums for all files (including latest versions)
if compgen -G "*.zip" >/dev/null; then
sha256sum -- *.zip > SHA256SUMS
sha512sum -- *.zip > SHA512SUMS
fi
cd ..
python3 scripts/security/generate_release_supply_chain_assets.py \
--asset-dir ./release-assets \
--version "$VERSION" \
--tag "$TAG" \
--build-type "${{ needs.build-check.outputs.build_type }}" \
--repository "${{ github.repository }}" \
--ref "${{ github.ref }}" \
--sha "${{ github.sha }}" \
--run-id "${{ github.run_id }}" \
--run-attempt "${{ github.run_attempt }}"
cd ./release-assets
echo "📦 Prepared assets:"
ls -la
echo "🔢 Total asset count: $ASSETS_COUNT"
- name: Upload to GitHub Release
env:
GH_TOKEN: ${{ github.token }}
shell: bash
run: |
TAG="${{ needs.build-check.outputs.version }}"
cd ./release-assets
# Upload all files
for file in *; do
if [[ -f "$file" ]]; then
echo "📤 Uploading $file..."
gh release upload "$TAG" "$file" --clobber
fi
done
echo "✅ All assets uploaded successfully"
# Update latest.json for every release tag (stable and prerelease): the
# project currently ships prerelease tags only, so gating this to stable
# left the version pointer permanently stale. release_type records whether
# the pointed-to version is a prerelease.
update-latest-version:
name: Update Latest Version
needs: [ build-check, publish-release ]
if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- name: Update latest.json
env:
OSS_ACCESS_KEY_ID: ${{ secrets.ALICLOUDOSS_KEY_ID }}
OSS_ACCESS_KEY_SECRET: ${{ secrets.ALICLOUDOSS_KEY_SECRET }}
OSS_REGION: cn-beijing
OSS_ENDPOINT: https://oss-cn-beijing.aliyuncs.com
shell: bash
run: |
if [[ -z "$OSS_ACCESS_KEY_ID" ]]; then
echo "⚠️ OSS credentials not available, skipping latest.json update"
exit 0
fi
VERSION="${{ needs.build-check.outputs.version }}"
TAG="${{ needs.build-check.outputs.version }}"
if [[ "${{ needs.build-check.outputs.build_type }}" == "prerelease" ]]; then
RELEASE_TYPE="prerelease"
else
RELEASE_TYPE="stable"
fi
# Install ossutil
OSSUTIL_VERSION="2.1.1"
OSSUTIL_ZIP="ossutil-${OSSUTIL_VERSION}-linux-amd64.zip"
OSSUTIL_DIR="ossutil-${OSSUTIL_VERSION}-linux-amd64"
OSSUTIL_SHA256="60f3a808cbbdfd4b5269b8f967c6a66f564c9dacff52d93a5d21a588976ab5a2"
curl --fail -L -o "$OSSUTIL_ZIP" "https://gosspublic.alicdn.com/ossutil/v2/${OSSUTIL_VERSION}/${OSSUTIL_ZIP}"
printf '%s %s\n' "$OSSUTIL_SHA256" "$OSSUTIL_ZIP" | sha256sum -c -
unzip "$OSSUTIL_ZIP"
OSSUTIL_BIN="${RUNNER_TEMP}/ossutil"
mv "${OSSUTIL_DIR}/ossutil" "$OSSUTIL_BIN"
rm -rf "$OSSUTIL_DIR" "$OSSUTIL_ZIP"
chmod +x "$OSSUTIL_BIN"
# Create latest.json
cat > latest.json << EOF
{
"version": "${VERSION}",
"tag": "${TAG}",
"release_date": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",
"release_type": "${RELEASE_TYPE}",
"download_url": "https://github.com/${{ github.repository }}/releases/tag/${TAG}"
}
EOF
# Upload to OSS
"$OSSUTIL_BIN" cp latest.json oss://rustfs-version/latest.json --force
echo "✅ Updated latest.json for ${RELEASE_TYPE} release $VERSION"
# Publish release (remove draft status)
publish-release:
name: Publish Release
needs: [ build-check, create-release, upload-release-assets ]
if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
permissions:
contents: write
steps:
- name: Publish release
env:
GH_TOKEN: ${{ github.token }}
shell: bash
run: |
TAG="${{ needs.build-check.outputs.version }}"
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
RELEASE_ID="${{ needs.create-release.outputs.release_id }}"
# Publish the release and correct its channel state on retries.
# Only a stable final release may become GitHub Latest.
if [[ "$BUILD_TYPE" == "release" ]]; then
gh api --method PATCH "repos/${GITHUB_REPOSITORY}/releases/${RELEASE_ID}" \
-F draft=false \
-F prerelease=false \
-f make_latest=true >/dev/null
else
gh api --method PATCH "repos/${GITHUB_REPOSITORY}/releases/${RELEASE_ID}" \
-F draft=false \
-F prerelease=true \
-f make_latest=false >/dev/null
fi
echo "🎉 Released $TAG successfully!"
echo "📄 Release URL: ${{ needs.create-release.outputs.release_url }}"
# Remove the internal preview releases once the deliverable release is live.
# Only the Releases are deleted; the -preview.N tags stay so the validated
# commit remains traceable.
cleanup-preview-releases:
name: Cleanup Preview Releases
needs: [ build-check, publish-release ]
if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
permissions:
contents: write
steps:
- name: Delete preview releases for this target
env:
GH_TOKEN: ${{ github.token }}
shell: bash
run: |
set -euo pipefail
TAG="${{ needs.build-check.outputs.version }}"
RELEASES_JSON="${RUNNER_TEMP}/releases.json"
# Fetch before filtering: a failed listing must abort here instead of
# looking like "nothing to clean up".
gh api --paginate "repos/${GITHUB_REPOSITORY}/releases?per_page=100" > "$RELEASES_JSON"
# Match only <target>-preview.<digits>. String operations, not a
# regex over the tag, so dots in the version cannot widen the match.
DELETED=0
while IFS= read -r preview_tag; do
[[ -n "$preview_tag" ]] || continue
echo "🧹 Deleting preview release $preview_tag (tag kept)"
gh release delete "$preview_tag" --repo "${GITHUB_REPOSITORY}" --yes
DELETED=$((DELETED + 1))
done < <(
jq -r --arg tag "$TAG" '
.[]
| select(.tag_name | startswith($tag + "-preview."))
| select(.tag_name | ltrimstr($tag + "-preview.") | test("^[0-9]+$"))
| .tag_name
' "$RELEASES_JSON"
)
if [[ "$DELETED" -eq 0 ]]; then
echo "ℹ️ No preview releases to clean up for $TAG"
else
echo "✅ Removed $DELETED preview release(s) for $TAG"
fi
alert-on-failure:
name: Alert on scheduled failure
needs: [build-check, prepare-platform-matrix, build-rustfs, build-summary]
needs: [build-check, prepare-build-inputs, prepare-platform-matrix, build-rustfs, build-summary]
if: >-
always() && github.event_name == 'schedule' &&
(contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
+36 -25
View File
@@ -33,22 +33,22 @@
# cache-save-if to anything but 'false' for it; every lane in ci.yml reads.
# scripts/security/check_cache_save_if.sh keeps the declarations explicit.
#
# The builds are supersets of what the reading lanes compile, because a reader
# restores only what the writer saved. Feature resolution matters here: a lane
# built with e2e-test-hooks resolves dependency features differently, which
# changes -Cmetadata, so the plain build does not cover it. See
# rustfs/backlog#1600.
# A reader restores only what the writer saved, and Cargo reuses a dependency
# only when it resolves the same features for it. Every package selection and
# feature set resolves differently and changes -Cmetadata, so each job builds
# its readers' own invocations, not a superset (rustfs/backlog#1600).
#
# Rust-cache never replaces an entry restored from an exact key match: the key
# hashes the lockfile, toolchain and compiler environment, not the commit. A
# job whose restore was an exact hit therefore skips its builds; they could
# only produce what is already stored.
name: Cache Warm
on:
schedule:
# Every six hours; the 22:41 UTC run lands before ci.yml's nightly 00:11
# UTC matrix. A reader restores the newest entry under its key and
# rebuilds whatever its own diff changed, so a cache up to six hours old
# costs a PR the handful of crates merged since. Warming after every merge
# (fourteen completed runs a day, about two sm-standard-4 hours each)
# spent far more than that to avoid it.
# Writes a key that no push created, such as one for a new stable
# toolchain; with every key present the jobs stop after the restore.
- cron: "41 4,10,16,22 * * *"
push:
branches: [ main ]
@@ -96,8 +96,8 @@ env:
RUST_BACKTRACE: 1
jobs:
# Readers: workspace-lint-and-build, workspace-tests (workspace and ILM), build-rustfs-debug-binary,
# e2e-tests, e2e-full.
# Readers: workspace-clippy, workspace-lint-and-build, build-rustfs-debug-binary,
# connect-feature-boundaries (nightly) and reference-parity.yml.
warm-ci-dev:
name: Warm ci-dev
runs-on: sm-standard-4
@@ -111,10 +111,13 @@ jobs:
persist-credentials: false
- name: Setup Rust environment
id: setup
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-dev
# Digest of the build steps below; scripts/ci_gate.py --check-workflow
# names the new key when they change, and every reader must follow.
cache-shared-key: ci-dev-ea7eec74fe14
cache-save-if: 'true'
install-build-packaging-tools: 'false'
@@ -130,7 +133,7 @@ jobs:
# 50% and 25% respectively; if linking dominates instead, the answer is
# mold/lld plus split-debuginfo, which is exactly the part sccache cannot
# touch. Off by default — this doubles the ci-dev build.
- name: Build ci-dev superset (with --timings)
- name: Build the workspace with --timings
if: inputs.emit_timings
env:
CARGO_BUILD_JOBS: "2"
@@ -145,18 +148,18 @@ jobs:
retention-days: 30
if-no-files-found: error
# --all-targets covers the test binaries nextest builds, including
# e2e_test, which the workspace run excludes. The second build adds
# the e2e-test-hooks feature resolution that build-rustfs-debug-binary uses
# and that no lint lane enables.
- name: Build ci-dev superset
# The workspace and E2E test archives of workspace-lint-and-build and
# build-rustfs-debug-binary, then that job's hooks server.
- name: Build ci-dev reader selections
if: steps.setup.outputs.cache-hit != 'true'
env:
# Same limit ci.yml puts on its nextest step: this builds the same
# ~100 workspace test binaries, and three concurrent links saturate the
# self-hosted runner's overlay I/O and can wedge Cargo (#5394).
# Same limit ci.yml puts on its archive steps: three concurrent test
# binary links saturate the self-hosted runner's overlay I/O and can
# wedge Cargo (#5394).
CARGO_BUILD_JOBS: "2"
run: |
cargo build --workspace --all-targets
python3 scripts/ci_test_artifact.py cargo warm --configuration workspace
python3 scripts/ci_test_artifact.py cargo warm --configuration e2e
cargo build -p rustfs --bins --features e2e-test-hooks
# ci.yml runs Clippy on every pull request. Clippy's check-mode artifacts
@@ -165,8 +168,9 @@ jobs:
# touches. A lint failure on main must not stop the warm-up: the cache is
# still saved (cache-on-failure), and the failure belongs to ci.yml.
- name: Warm Clippy check artifacts
if: steps.setup.outputs.cache-hit != 'true'
run: |
if ! cargo clippy --all-targets -- -D warnings; then
if ! python3 scripts/ci_test_artifact.py cargo lint --configuration workspace; then
echo "::warning::cargo clippy reports findings on this commit; check artifacts are still cached"
fi
@@ -213,6 +217,7 @@ jobs:
persist-credentials: false
- name: Setup Rust environment
id: setup
uses: ./.github/actions/setup
with:
rust-version: stable
@@ -221,6 +226,7 @@ jobs:
install-build-packaging-tools: 'false'
- name: Build ci-feat-rio superset
if: steps.setup.outputs.cache-hit != 'true'
env:
# --all-targets links the same test binaries as the reader lane.
CARGO_BUILD_JOBS: "2"
@@ -244,6 +250,7 @@ jobs:
persist-credentials: false
- name: Setup Rust environment
id: setup
uses: ./.github/actions/setup
with:
rust-version: stable
@@ -252,6 +259,7 @@ jobs:
install-build-packaging-tools: 'false'
- name: Build ci-feat-proto superset
if: steps.setup.outputs.cache-hit != 'true'
env:
# Avoid an unbounded burst of concurrent test-binary links (#5394).
CARGO_BUILD_JOBS: "2"
@@ -274,6 +282,7 @@ jobs:
persist-credentials: false
- name: Setup Rust environment
id: setup
uses: ./.github/actions/setup
with:
rust-version: stable
@@ -282,7 +291,9 @@ jobs:
install-build-packaging-tools: 'false'
- name: Install build dependencies
if: steps.setup.outputs.cache-hit != 'true'
run: sudo apt-get update && sudo apt-get install -y protobuf-compiler
- name: Build ci-uring superset
if: steps.setup.outputs.cache-hit != 'true'
run: cargo build -p rustfs-ecstore --all-targets
+264 -120
View File
@@ -16,7 +16,9 @@ name: Continuous Integration
on:
push:
branches: [ main ]
# gateway/integration receives the daily merge of main as a direct push
# (rustfs/backlog#2734), so its pushes must prove the result like main's.
branches: [ main, gateway/integration ]
paths-ignore:
- "**.md"
- "docs/**"
@@ -44,11 +46,6 @@ on:
schedule:
- cron: "11 0 * * *" # Nightly 00:11 UTC, after the 22:41 UTC cache-warm run
workflow_dispatch:
inputs:
rehearse_workspace_shards:
description: Compare one full execution with at most two resource-safe shards of the same archive
type: boolean
default: false
permissions:
contents: read
@@ -86,13 +83,16 @@ jobs:
if: github.event_name != 'pull_request' || github.event.action != 'closed'
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: read
actions: read
outputs:
mode: ${{ steps.scope.outputs.mode }}
plan: ${{ steps.scope.outputs.plan }}
connect: ${{ steps.scope.outputs.connect || 'true' }}
packages: ${{ steps.scope.outputs.packages }}
full_validation: ${{ steps.scope.outputs.full_validation || 'true' }}
rehearse_shards: ${{ steps.scope.outputs.rehearse_shards || 'false' }}
gateway_smoke: ${{ steps.scope.outputs.gateway_smoke || 'true' }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
@@ -103,8 +103,13 @@ jobs:
env:
CI_BASE_SHA: ${{ github.event.pull_request.base.sha || github.event.merge_group.base_sha || github.event.before }}
CI_EVENT_ACTION: ${{ github.event.action }}
CI_REHEARSE_SHARDS: ${{ inputs.rehearse_workspace_shards && 'true' || 'false' }}
# A push whose tree a successful merge-group run already passed skips
# the validation jobs; the lookup answers null on any doubt.
PROOF_TOKEN: ${{ github.event_name == 'push' && github.token || '' }}
run: |
# Without a token (every event but push) the lookup answers null at once.
CI_MERGE_GROUP_PROOF="$(GITHUB_TOKEN="$PROOF_TOKEN" python3 -I scripts/ci_merge_group_proof.py)" || CI_MERGE_GROUP_PROOF=null
export CI_MERGE_GROUP_PROOF
if [[ "$GITHUB_EVENT_NAME" != "pull_request" ]]; then
python3 -I scripts/ci_gate.py select
elif [[ "$CI_BASE_SHA" =~ ^[0-9a-f]{40}$ ]] && git show "$CI_BASE_SHA:scripts/ci_gate.py" > "$RUNNER_TEMP/ci-gate-base.py"; then
@@ -113,8 +118,8 @@ jobs:
printf '%s\n' 'mode=full' >> "$GITHUB_OUTPUT"
echo "Base CI policy unavailable; running the full matrix."
fi
if ! grep -q '^plan=' "$GITHUB_OUTPUT" || ! grep -q '^plan_contract=3$' "$GITHUB_OUTPUT"; then
printf '%s\n' 'mode=full' 'connect=true' 'full_validation=true' 'rehearse_shards=false' 'packages=' 'plan=' >> "$GITHUB_OUTPUT"
if ! grep -q '^plan=' "$GITHUB_OUTPUT" || ! grep -q '^plan_contract=8$' "$GITHUB_OUTPUT"; then
printf '%s\n' 'mode=full' 'connect=true' 'full_validation=true' 'gateway_smoke=true' 'packages=' 'plan=' >> "$GITHUB_OUTPUT"
echo "Base policy does not cover this workflow contract; retaining the full matrix."
fi
- name: Retain verification plan
@@ -168,6 +173,67 @@ jobs:
- name: Run shared quick checks
uses: ./.github/actions/quick-checks
# Clippy runs beside the test-binary build instead of in front of it. The two
# share no compiler output: Clippy writes check-mode metadata, the archive
# needs codegen and linked test binaries, and both restore the third-party
# artifacts of their own mode from ci-dev. As one job the lint held the
# archive, and every test that consumes it, back by its whole duration.
# scripts/ci_gate.py keeps this job in the Test and Lint aggregate and out
# of the archive consumer's dependency chain.
workspace-clippy:
name: Workspace Clippy
if: contains(fromJSON('["full", "e2e"]'), needs.classify-changes.outputs.mode) && (github.event_name != 'pull_request' || github.event.action != 'closed')
needs: [ quick-checks, classify-changes ]
# Reads the ci-dev cache that cache-warm.yml writes on this image, and the
# all-targets check of the workspace does not fit a hosted runner's disk.
runs-on: sm-standard-4
timeout-minutes: 60
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
CI_PACKAGES: ${{ needs.classify-changes.outputs.packages }}
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
# Checkout otherwise writes the token into .git/config, where a PR's
# own build.rs or proc-macro could read it back out.
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
# Read-only, like every other ci-dev reader; cache-warm.yml warms the
# check artifacts of this exact invocation.
cache-shared-key: ci-dev-ea7eec74fe14
cache-save-if: 'false'
install-build-packaging-tools: 'false'
# Clippy gates every event, pull requests included. When it ran only after
# merge, a lint regression merged freely and then failed every main run
# until someone fixed it (10 of 12 main runs on 2026-10-17/18). cache-warm
# writes the same invocation's check artifacts into ci-dev, so a pull
# request re-checks only the crates its diff touches. Main keeps the step
# because pull requests can merge without being up to date with main.
- name: Run clippy lints
run: |
./scripts/ci/resource_sampler.sh start clippy
trap './scripts/ci/resource_sampler.sh stop' EXIT
python3 scripts/ci_test_artifact.py cargo lint --configuration workspace
- name: Upload lint diagnostics
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: workspace-clippy-diagnostics-${{ github.run_number }}-${{ github.run_attempt }}
path: artifacts/workspace
retention-days: 3
if-no-files-found: error
# Keeps its job ID and check name: it is the archive producer that
# workspace-tests and scripts/check_test_wiring.py bind to. The lint half
# is workspace-clippy above; this job builds the test archive, then runs
# the doctests and the log-analyzer anchor check on the same build output.
workspace-lint-and-build:
name: Workspace Lint and Build
if: contains(fromJSON('["full", "e2e"]'), needs.classify-changes.outputs.mode) && (github.event_name != 'pull_request' || github.event.action != 'closed')
@@ -192,7 +258,7 @@ jobs:
# Every lane reads its cache; cache-warm.yml remains the sole writer
# with the matching feature closure. Tests never compete to replace
# shared caches, including when a PR attempt is cancelled.
cache-shared-key: ci-dev
cache-shared-key: ci-dev-ea7eec74fe14
cache-save-if: 'false'
install-build-packaging-tools: 'false'
@@ -209,18 +275,6 @@ jobs:
echo "started_at=$(date --utc --iso-8601=seconds)"
} > artifacts/workspace/run-metadata.txt
# Clippy gates every event, pull requests included. When it ran only after
# merge, a lint regression merged freely and then failed every main run
# until someone fixed it (10 of 12 main runs on 2026-10-17/18). cache-warm
# writes the same invocation's check artifacts into ci-dev, so a pull
# request re-checks only the crates its diff touches. Main keeps the step
# because pull requests can merge without being up to date with main.
- name: Run clippy lints
run: |
./scripts/ci/resource_sampler.sh start clippy
trap './scripts/ci/resource_sampler.sh stop' EXIT
python3 scripts/ci_test_artifact.py cargo lint --configuration workspace
- name: Archive workspace test binaries
env:
# Workspace test compilation has high per-process memory use.
@@ -271,7 +325,7 @@ jobs:
# rustfs/backlog#1289: fail if a seed rule's log anchor no longer exists
# verbatim in the source tree (log message drifted without updating the
# rule). Placed here where the workspace — including the la-dump-anchors
# bin — is already built by the clippy/test steps above.
# bin's dependencies — is already built by the test archive step above.
- name: Check log-analyzer rule anchors
run: ./scripts/check_log_analyzer_rules.sh
@@ -282,8 +336,6 @@ jobs:
# requires the same runner image until the hosted-runtime pilot passes.
runs-on: sm-standard-4
timeout-minutes: 90
outputs:
workspace_shards: ${{ steps.shards.outputs.workspace_shards || '[]' }}
env:
CI_PACKAGES: ${{ needs.classify-changes.outputs.packages }}
steps:
@@ -372,89 +424,6 @@ jobs:
retention-days: 3
if-no-files-found: error
- name: Prepare bounded shards from this candidate's observed durations
id: shards
if: needs.classify-changes.outputs.rehearse_shards == 'true'
run: |
python3 scripts/ci_test_shards.py prepare \
--listing artifacts/workspace/core-test-listing.json \
--junit target/nextest/ci/junit.xml --output artifacts/workspace-shard-plan
- name: Upload same-run shard plan
if: needs.classify-changes.outputs.rehearse_shards == 'true'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: workspace-shard-plan-${{ github.run_attempt }}
path: artifacts/workspace-shard-plan
retention-days: 3
if-no-files-found: error
workspace-shards:
name: Workspace shards (${{ matrix.shard }})
if: needs.classify-changes.outputs.rehearse_shards == 'true'
needs: [classify-changes, workspace-tests]
# Reuse the producer image and checkout path for the full RustFS archive.
runs-on: sm-standard-4
timeout-minutes: 90
strategy:
fail-fast: false
max-parallel: 2
matrix:
shard: ${{ fromJSON(needs.workspace-tests.outputs.workspace_shards || '[]') }}
env:
CI_PACKAGES: ${{ needs.classify-changes.outputs.packages }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- uses: ./.github/actions/test-runtime
- name: Protect Connect test home
run: chmod go-w "$(realpath "$HOME")"
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: workspace-test-binaries-${{ github.run_attempt }}
path: ${{ runner.temp }}
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: workspace-shard-plan-${{ github.run_attempt }}
path: artifacts/workspace-shard-plan
- name: Verify and execute this shard without compilation
env:
SHARD_INDEX: ${{ matrix.shard }}
run: |
python3 scripts/ci_test_artifact.py verify --configuration workspace --archive "$RUNNER_TEMP/workspace-tests.tar.zst"
python3 scripts/ci_test_shards.py execute --plan artifacts/workspace-shard-plan \
--archive "$RUNNER_TEMP/workspace-tests.tar.zst" --index "$SHARD_INDEX" --output artifacts/workspace-shard-result
- name: Retain shard evidence including failures
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: workspace-shard-result-${{ matrix.shard }}-${{ github.run_attempt }}
path: artifacts/workspace-shard-result
retention-days: 3
if-no-files-found: error
workspace-shard-evidence:
name: Workspace shard completeness
if: always() && needs.classify-changes.outputs.rehearse_shards == 'true'
needs: [classify-changes, workspace-tests, workspace-shards]
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: workspace-shard-plan-${{ github.run_attempt }}
path: artifacts/workspace-shard-plan
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
pattern: workspace-shard-result-*-${{ github.run_attempt }}
path: artifacts/workspace-shards
- name: Require the exact union of successful shard executions
run: |
python3 scripts/ci_test_shards.py verify --plan artifacts/workspace-shard-plan --evidence artifacts/workspace-shards
./scripts/check_migration_gate_count.sh evidence artifacts/workspace-shard-plan/listing.json artifacts/workspace-shards/combined.xml
connect-feature-boundaries:
name: Connect Feature Boundaries
@@ -474,7 +443,7 @@ jobs:
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-dev
cache-shared-key: ci-dev-ea7eec74fe14
cache-save-if: 'false'
install-build-packaging-tools: 'false'
install-test-tools: 'false'
@@ -500,7 +469,7 @@ jobs:
# hold. `cargo check --release` is metadata-only and needs no
# link-concurrency limit; it is cold because ci-dev holds no release
# artifacts.
- name: Reject short credentials in release builds
- name: Reject debug-only features in release builds
run: python3 scripts/check_connect_release_boundary.py
- name: Check offline enrollment E2E root boundary
@@ -646,6 +615,10 @@ jobs:
flags: "--features sftp"
- name: http3
flags: "--features http3,swift"
- name: internode-transport
flags: "--features rustfs/internode-transport"
packages: "-p rustfs-ecstore -p rustfs-rio"
tests: "--no-tests fail -E 'test(peer_transport_)'"
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
@@ -671,7 +644,7 @@ jobs:
run: |
./scripts/ci/resource_sampler.sh start clippy
trap './scripts/ci/resource_sampler.sh stop' EXIT
cargo clippy -p rustfs -p rustfs-protocols --all-targets ${{ matrix.features.flags }} -- -D warnings
cargo clippy -p rustfs ${{ matrix.features.packages || '-p rustfs-protocols' }} --all-targets ${{ matrix.features.flags }} -- -D warnings
- name: Run tests with ${{ matrix.features.name }}
id: protocol-tests
@@ -688,7 +661,7 @@ jobs:
mkdir -p artifacts/protocol-tests
./scripts/ci/resource_sampler.sh start nextest
trap './scripts/ci/resource_sampler.sh stop' EXIT
cargo nextest run --profile ci -p rustfs -p rustfs-protocols ${{ matrix.features.flags }} \
cargo nextest run --profile ci -p rustfs ${{ matrix.features.packages || '-p rustfs-protocols' }} ${{ matrix.features.flags }} ${{ matrix.features.tests }} \
2>&1 | tee artifacts/protocol-tests/nextest.log
- name: Upload protocol test reports and diagnostics
@@ -703,6 +676,54 @@ jobs:
retention-days: 3
if-no-files-found: warn
test-and-lint-gateway-stack:
name: Test and Lint (gateway-stack)
# The gateway S3 stack (rustfs/backlog#2734) sits behind the rustfs
# `gateway-stack` feature, which no other pull-request lane lints or tests.
# Unlike the dormant feature lanes above, this one runs for every code
# change: each operation slice puts its handler tests and its refusal-matrix
# row behind the feature, and a lane that ran only on demand would let them
# rot behind a green pull request.
if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed')
needs: [ quick-checks, classify-changes ]
# Self-hosted for build scale and cache ownership: a workspace-scale Cargo
# build that reads the ci-dev cache written on this image.
runs-on: sm-standard-4
timeout-minutes: 90
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-dev-ea7eec74fe14
cache-save-if: 'false'
install-build-packaging-tools: 'false'
# A feature that does not compile fails here; nothing below is skipped
# into a pass.
- name: Run clippy with gateway-stack
run: cargo clippy --locked -p rustfs --all-targets --features gateway-stack -- -D warnings
# The gateway stack's own tests: the assembly, the operation table and
# its completeness, the shared handler sequence, the stack selection, and
# the embedded server driven through the whole middleware stack. The
# filter names modules, never operations.
- name: Run tests with gateway-stack
env:
# The same bounded link concurrency as the other feature lanes.
CARGO_BUILD_JOBS: "2"
run: |
cargo nextest run --profile ci --locked -p rustfs --features gateway-stack --no-tests fail \
--status-level all --final-status-level all \
-E 'test(/gateway_stack|gateway_handlers|server::stack::/) | binary(embedded_gateway_stack_test)'
build-rustfs-debug-binary:
name: Build RustFS Debug Binary
if: contains(fromJSON('["full", "e2e"]'), needs.classify-changes.outputs.mode) && (github.event_name != 'pull_request' || github.event.action != 'closed')
@@ -721,7 +742,7 @@ jobs:
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-dev
cache-shared-key: ci-dev-ea7eec74fe14
cache-save-if: 'false'
install-build-packaging-tools: 'false'
@@ -965,6 +986,8 @@ jobs:
# Retain the producer image and fixture paths until the runtime pilot passes.
runs-on: sm-standard-4
timeout-minutes: 30
env:
RUSTFS_S3_STACK: legacy
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
@@ -1085,18 +1108,137 @@ jobs:
path: ${{ runner.temp }}/rustfs-e2e-*/rustfs.log
retention-days: 3
# The same e2e-smoke suite against the same debug binary with the gateway
# S3 stack selected (rustfs/backlog#2734). Report-only: it is outside the
# Test and Lint aggregate and becomes required at T2.10. The test-binary
# download, identity, membership, run and completeness steps must stay
# identical to e2e-tests; scripts/ci_gate.py enforces that.
#
# When it runs: every push to main or gateway/integration, merge group,
# nightly schedule and manual dispatch, and every pull request that changes a
# gateway-stack input (GATEWAY_SMOKE_PATHS and the rustfs-gateway pins in
# scripts/ci_gate.py, published by Select CI scope as gateway_smoke).
# Why it is skipped on every other pull request: until T2.10 the gateway
# stack does not serve the whole smoke suite, so the lane fails on any
# revision. A pull request that touches none of its inputs can neither cause
# nor fix that failure, and a red check there says nothing about the change.
# Skipped is the honest state; a failure must never be rendered green, so no
# step here may carry continue-on-error. A selection that is missing or
# unreadable runs the lane. At T2.10, when the lane becomes required, it
# must run on every pull request again: delete the selection, do not extend
# it.
e2e-tests-gateway:
name: e2e-smoke (gateway, report-only until T2.10)
needs: [ classify-changes, build-rustfs-debug-binary ]
if: github.event_name != 'pull_request' || needs.classify-changes.outputs.gateway_smoke != 'false'
runs-on: sm-standard-2
timeout-minutes: 30
continue-on-error: true
env:
RUSTFS_S3_STACK: gateway
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Install verification runtime
uses: ./.github/actions/test-runtime
- name: Download E2E test binaries
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: e2e-test-binaries
path: ${{ runner.temp }}
- name: Verify E2E archive identity
run: python3 scripts/ci_test_artifact.py verify --configuration e2e --archive "$RUNNER_TEMP/e2e-tests.tar.zst"
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
- name: Install awscurl
run: |
python3 -m pip install --user --upgrade pip "awscurl==0.44"
echo "AWSCURL_PATH=$HOME/.local/bin/awscurl" >> "$GITHUB_ENV"
- name: Verify awscurl
run: test -x "$AWSCURL_PATH"
# Download after the cache restore so the freshly built binary from the
# build job always wins over anything restored into target/debug.
- name: Download debug binary
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: rustfs-debug-binary
path: target/debug
- name: Make binary executable
run: chmod +x ./target/debug/rustfs
# Build the e2e test graph once. The archive is reused by the smoke
# selection guard, security exact-count check, and run below, avoiding a
# second compile of the same e2e_test target on cold runners (backlog#1645).
- name: Verify e2e smoke test membership
env:
NEXTEST_ARCHIVE: ${{ runner.temp }}/e2e-tests.tar.zst
NEXTEST_LISTING: ${{ runner.temp }}/rustfs-e2e-smoke-list.json
run: |
cargo nextest list --profile e2e-smoke --archive-file "${NEXTEST_ARCHIVE}" --workspace-remap "$GITHUB_WORKSPACE" --extract-to "$GITHUB_WORKSPACE" --extract-overwrite --message-format json > "${NEXTEST_LISTING}"
python3 ./scripts/check_test_wiring.py --check-profile e2e-smoke "${NEXTEST_LISTING}"
./scripts/check_security_smoke_count.sh check "${NEXTEST_LISTING}"
# PR smoke subset of the in-repo e2e suite (backlog#1149 ci-4). The
# profile.e2e-smoke default-filter in .config/nextest.toml is the single
# wiring mechanism for e2e tests in CI — extend that filter instead of
# adding new e2e jobs here. Each test spawns its own rustfs server on a
# random port and reuses the downloaded debug binary above.
- name: Run e2e smoke suite
env:
NEXTEST_ARCHIVE: ${{ runner.temp }}/e2e-tests.tar.zst
RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-e2e-smoke-logs
SAMPLER_INTERVAL_SECS: "15"
run: |
./scripts/ci/resource_sampler.sh start e2e-smoke
trap './scripts/ci/resource_sampler.sh stop' EXIT
python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run --profile e2e-smoke --cargo-metadata target/nextest/cargo-metadata.json --binaries-metadata target/nextest/binaries-metadata.json --workspace-remap "$GITHUB_WORKSPACE" --target-dir-remap "$GITHUB_WORKSPACE/target" \
--status-level all --final-status-level all --failure-output final
- name: Verify e2e smoke execution completeness
run: python3 scripts/ci_test_artifact.py evidence "$RUNNER_TEMP/rustfs-e2e-smoke-list.json" target/nextest/e2e-smoke/junit.xml
- name: Upload e2e smoke diagnostics
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-smoke-gateway-diagnostics-${{ github.run_number }}-${{ github.run_attempt }}
path: |
${{ runner.temp }}/rustfs-e2e-smoke-logs/
${{ runner.temp }}/rustfs-e2e-smoke-list.json
if-no-files-found: warn
- name: Upload e2e smoke JUnit report
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-smoke-gateway-junit-${{ github.run_number }}-${{ github.run_attempt }}
path: target/nextest/e2e-smoke/junit.xml
if-no-files-found: warn
e2e-full:
name: End-to-End Tests (full merge gate)
# Merge gate only (backlog#1149 ci-5): the never-automated user-visible
# suites — KMS, object_lock, multipart_auth, quota, checksum, encryption,
# security-boundary, ... — via the e2e-full nextest profile. Too heavy for
# every PR, so it is gated to main pushes, the merge queue, and manual
# dispatch. protocols / the 7 cluster suites / replication / #[ignore] are
# owned by other lanes (see .config/nextest.toml profile.e2e-full).
# every PR, so it is gated to main and gateway/integration pushes, the merge
# queue, and manual dispatch. protocols / the 7 cluster suites / replication
# / #[ignore] are owned by other lanes (see .config/nextest.toml profile.e2e-full).
if: >-
github.event_name == 'workflow_dispatch' ||
github.event_name == 'merge_group' ||
(github.event_name == 'push' && github.ref == 'refs/heads/main')
(github.event_name == 'push' && (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/gateway/integration'))
needs: [ build-rustfs-debug-binary ]
# Retain the producer image and fixture paths until the runtime pilot passes.
runs-on: sm-standard-4
@@ -1462,15 +1604,15 @@ jobs:
- classify-changes
- typos
- quick-checks
- workspace-clippy
- workspace-lint-and-build
- workspace-tests
- workspace-shards
- workspace-shard-evidence
- core-contract
- connect-feature-boundaries
- fips-provider
- test-and-lint-rio-v2
- test-and-lint-protocols
- test-and-lint-gateway-stack
- build-rustfs-debug-binary
- uring-integration
- e2e-tests
@@ -1500,11 +1642,13 @@ jobs:
- required-checks
- typos
- quick-checks
- workspace-clippy
- workspace-lint-and-build
- workspace-tests
- core-contract
- test-and-lint-rio-v2
- test-and-lint-protocols
- test-and-lint-gateway-stack
- build-rustfs-debug-binary
- build-rustfs-debug-binary-rio-v2
- uring-integration
+71 -571
View File
@@ -1,599 +1,99 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Docker Images Workflow
#
# This workflow builds Docker images using pre-built binaries from the build workflow.
#
# Trigger Types:
# 1. workflow_run: Automatically triggered when "Build and Release" workflow completes
# 2. workflow_dispatch: Manual trigger for standalone Docker builds
#
# Key Features:
# - Only triggers when Linux builds (x86_64 + aarch64) are successful
# - Independent of macOS/Windows build status
# - Uses workflow_run event for precise control
# - Only builds Docker images for releases and prereleases (development builds are skipped)
name: Docker Images
# Permissions needed for workflow_run event and Docker registry access
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Build candidate images
on:
workflow_call:
inputs:
candidate_manifest:
type: string
required: true
permissions:
contents: read
packages: write
on:
# Automatically triggered when build workflow completes
workflow_run:
workflows: [ "Build and Release" ]
types: [ completed ]
# Manual trigger with same parameters for consistency
workflow_dispatch:
inputs:
push_images:
description: "Push images to registries"
required: false
default: true
type: boolean
version:
description: "Version to build (latest, v1.0.0, v1.0.0-alpha1, or v1.0.1-preview.1)"
required: false
default: "latest"
type: string
force_rebuild:
description: "Force rebuild even if binary exists (useful for testing)"
required: false
default: false
type: boolean
env:
CONCLUSION: ${{ github.event.workflow_run.conclusion }}
HEAD_BRANCH: ${{ github.event.workflow_run.head_branch }}
HEAD_SHA: ${{ github.event.workflow_run.head_sha }}
TRIGGERING_EVENT: ${{ github.event.workflow_run.event }}
DOCKERHUB_USERNAME: rustfs
CARGO_TERM_COLOR: always
REGISTRY_DOCKERHUB: rustfs/rustfs
REGISTRY_GHCR: ghcr.io/${{ github.repository }}
REGISTRY_QUAY: quay.io/rustfs/rustfs
DOCKER_PLATFORMS: linux/amd64,linux/arm64
actions: read
jobs:
# Check if we should build Docker images.
# Short-circuit at job level so per-merge development builds (workflow_run
# from a push to main) skip the whole run without spawning a checkout job:
# images are only published for tag builds — build.yml push triggers cover
# only main and tags, so a non-main head_branch is a tag name — and for
# manual dispatch.
build-check:
name: Docker Build Check
if: >-
github.event_name == 'workflow_dispatch' ||
(github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push' &&
github.event.workflow_run.head_branch != 'main')
runs-on: ubuntu-latest
timeout-minutes: 30
outputs:
should_build: ${{ steps.check.outputs.should_build }}
should_push: ${{ steps.check.outputs.should_push }}
build_type: ${{ steps.check.outputs.build_type }}
version: ${{ steps.check.outputs.version }}
short_sha: ${{ steps.check.outputs.short_sha }}
is_prerelease: ${{ steps.check.outputs.is_prerelease }}
create_latest: ${{ steps.check.outputs.create_latest }}
source_ref: ${{ steps.check.outputs.source_ref }}
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
# For workflow_run events, checkout the specific commit that triggered the workflow
ref: ${{ github.event.workflow_run.head_sha || github.sha }}
- name: Check build conditions
id: check
env:
# dispatch inputs via env, not `${{ }}` interpolation: they are
# free-form strings and would otherwise be evaluated by bash.
INPUT_VERSION: ${{ github.event.inputs.version }}
INPUT_PUSH_IMAGES: ${{ github.event.inputs.push_images }}
INPUT_FORCE_REBUILD: ${{ github.event.inputs.force_rebuild }}
run: |
should_build=false
should_push=false
build_type="none"
version=""
short_sha=""
is_prerelease=false
create_latest=false
source_ref="$GITHUB_SHA"
# Pre-GA policy: until the first stable (vX.Y.Z) tag exists, every
# prerelease (alpha/beta/rc) also moves `latest`, so users pulling
# `latest` get the newest test build. Once a stable tag is published
# this returns false and `latest` follows stable releases only.
prerelease_moves_latest() {
local stable_tags
stable_tags=$(git ls-remote --tags --refs origin 2>/dev/null \
| awk '{print $2}' \
| grep -E '^refs/tags/v?[0-9]+\.[0-9]+\.[0-9]+$' || true)
if [[ -z "$stable_tags" ]]; then
return 0
fi
echo "ℹ️ Stable release tag(s) already exist; prereleases no longer update latest"
return 1
}
if [[ "${{ github.event_name }}" == "workflow_run" ]]; then
# Triggered by build workflow completion
echo "🔗 Triggered by build workflow completion"
# Check if the triggering workflow was successful
# If the workflow succeeded, it means ALL builds (including Linux x86_64 and aarch64) succeeded
if [[ "$CONCLUSION" == "success" ]]; then
echo "✅ Build workflow succeeded, all builds including Linux are successful"
should_build=true
should_push=true
else
echo "❌ Build workflow failed (conclusion: $CONCLUSION), skipping Docker build"
should_build=false
fi
# Extract version info from commit message or use commit SHA
# Use Git to generate consistent short SHA (ensures uniqueness like build.yml)
short_sha=$(git rev-parse --short "$HEAD_SHA")
source_ref="$HEAD_SHA"
# Determine build type based on triggering workflow event and ref
triggering_event="$TRIGGERING_EVENT"
head_branch="$HEAD_BRANCH"
echo "🔍 Analyzing triggering workflow:"
echo " 📋 Event: $triggering_event"
echo " 🌿 Head branch: $head_branch"
echo " 📎 Head SHA: $HEAD_SHA"
# Check if this was triggered by a tag push
if [[ "$triggering_event" == "push" ]]; then
# For tag pushes, head_branch will be like "refs/tags/v1.0.0" or just "v1.0.0"
if [[ "$head_branch" == refs/tags/* ]]; then
# Extract tag name from refs/tags/TAG_NAME
tag_name="${head_branch#refs/tags/}"
version="$tag_name"
elif [[ "$head_branch" =~ ^v?[0-9]+\.[0-9]+\.[0-9]+ ]]; then
# Direct tag name like "v1.0.0" or "1.0.0-alpha.1"
version="$head_branch"
elif [[ "$head_branch" == "main" ]]; then
# Regular branch push to main
build_type="development"
version="dev-${short_sha}"
should_build=false
echo "⏭️ Skipping Docker build for development version (main branch push)"
else
# Other branch push
build_type="development"
version="dev-${short_sha}"
should_build=false
echo "⏭️ Skipping Docker build for development version (branch: $head_branch)"
fi
# If we extracted a version (tag), determine release type
if [[ -n "$version" ]] && [[ "$version" != "dev-${short_sha}" ]]; then
# Remove 'v' prefix if present for consistent version format
if [[ "$version" == v* ]]; then
version="${version#v}"
fi
if [[ "$version" =~ -preview\.[0-9]+$ ]]; then
build_type="preview"
is_prerelease=true
echo "🔍 Building Docker image for preview: $version (version tags only)"
elif [[ "$version" == *"-preview"* ]]; then
echo "❌ Invalid preview tag: $version (expected suffix: -preview.<number>)" >&2
exit 1
elif [[ "$version" == *"alpha"* ]] || [[ "$version" == *"beta"* ]] || [[ "$version" == *"rc"* ]]; then
build_type="prerelease"
is_prerelease=true
# Pre-GA policy: prereleases update latest until the first stable tag exists.
if prerelease_moves_latest; then
create_latest=true
echo "🧪 Building Docker image for prerelease: $version (creating latest tag)"
else
echo "🧪 Building Docker image for prerelease: $version"
fi
else
build_type="release"
create_latest=true
echo "🚀 Building Docker image for release: $version"
fi
fi
else
# Non-push events
build_type="development"
version="dev-${short_sha}"
should_build=false
echo "⏭️ Skipping Docker build for development version (event: $triggering_event)"
fi
echo "🔄 Build triggered by workflow_run:"
echo " 📋 Conclusion: $CONCLUSION"
echo " 🌿 Branch: $HEAD_BRANCH"
echo " 📎 SHA: $HEAD_SHA"
echo " 🎯 Event: $TRIGGERING_EVENT"
elif [[ "${{ github.event_name }}" == "workflow_dispatch" ]]; then
# Manual trigger
input_version="$INPUT_VERSION"
version="${input_version}"
should_push="$INPUT_PUSH_IMAGES"
should_build=true
# Get short SHA
short_sha=$(git rev-parse --short HEAD)
echo "🎯 Manual Docker build triggered:"
echo " 📋 Requested version: $input_version"
echo " 🔧 Force rebuild: $INPUT_FORCE_REBUILD"
echo " 🚀 Push images: $should_push"
case "$input_version" in
"latest")
build_type="release"
create_latest=true
echo "🚀 Building with latest stable release version"
;;
*-preview*)
if [[ ! "$input_version" =~ ^v?[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.-]+)?-preview\.[0-9]+$ ]]; then
echo "❌ Invalid preview version: $input_version" >&2
exit 1
fi
build_type="preview"
is_prerelease=true
version="${input_version#v}"
echo "🔍 Building with preview version: $version (version tags only)"
;;
# Prerelease versions (must match first, more specific)
v*alpha*|v*beta*|v*rc*|*alpha*|*beta*|*rc*)
build_type="prerelease"
is_prerelease=true
# Pre-GA policy: prereleases update latest until the first stable tag exists.
if prerelease_moves_latest; then
create_latest=true
echo "🧪 Building with prerelease version: $input_version (creating latest tag)"
else
echo "🧪 Building with prerelease version: $input_version"
fi
;;
# Release versions (match after prereleases, more general)
v[0-9]*|[0-9]*.*.*)
build_type="release"
create_latest=true
echo "📦 Building with specific release version: $input_version"
;;
*)
# Invalid version for Docker build
should_build=false
echo "❌ Invalid version for Docker build: $input_version"
echo "⚠️ Only release versions (latest, v1.0.0, 1.0.0) and prereleases (v1.0.0-alpha1, 1.0.0-beta2) are supported"
;;
esac
if [[ "$should_build" == true && "$input_version" != "latest" ]]; then
tag_ref="refs/tags/$input_version"
if ! git ls-remote --exit-code origin "$tag_ref" >/dev/null 2>&1; then
if [[ "$input_version" == v* ]]; then
tag_ref="refs/tags/${input_version#v}"
else
tag_ref="refs/tags/v$input_version"
fi
fi
if ! git ls-remote --exit-code origin "$tag_ref" >/dev/null 2>&1; then
echo "❌ Release tag not found for Docker build: $input_version"
exit 1
fi
source_ref="$tag_ref"
fi
fi
{
echo "should_build=$should_build"
echo "should_push=$should_push"
echo "build_type=$build_type"
echo "version=$version"
echo "short_sha=$short_sha"
echo "is_prerelease=$is_prerelease"
echo "create_latest=$create_latest"
echo "source_ref=$source_ref"
} >> "$GITHUB_OUTPUT"
echo "🐳 Docker Build Summary:"
echo " - Should build: $should_build"
echo " - Should push: $should_push"
echo " - Build type: $build_type"
echo " - Version: $version"
echo " - Short SHA: $short_sha"
echo " - Is prerelease: $is_prerelease"
echo " - Create latest: $create_latest"
echo " - Source ref: $source_ref"
# Build multi-arch Docker images
# Strategy: Build images using pre-built binaries from dl.rustfs.com
# Supports both release and dev channel binaries based on build context
# Only runs when should_build is true (which includes workflow success check)
build-docker:
name: Build Docker Images
needs: build-check
if: needs.build-check.outputs.should_build == 'true'
runs-on: ubuntu-latest
timeout-minutes: 60
timeout-minutes: 90
strategy:
fail-fast: false
matrix:
include:
- variant: musl
file: Dockerfile
suffix: ""
- variant: glibc
- variant: gnu
file: Dockerfile.glibc
suffix: "-glibc"
env:
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
VARIANT: ${{ matrix.variant }}
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: ${{ fromJson(inputs.candidate_manifest).source_sha }}
persist-credentials: false
ref: ${{ needs.build-check.outputs.source_ref }}
- name: Login to Docker Hub
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
username: ${{ env.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Login to GitHub Container Registry
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
password: ${{ secrets.GHCR_PASSWORD }}
- name: Login to Quay.io
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
registry: quay.io
username: ${{ secrets.QUAY_USERNAME }}
password: ${{ secrets.QUAY_PASSWORD }}
- name: Set up QEMU
uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 # v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
- name: Extract metadata and generate tags
id: meta
- name: Reuse completed image
id: reuse
env:
GH_TOKEN: ${{ github.token }}
run: python3 scripts/release/deliver.py reuse --kind image --target "$VARIANT"
- name: Install OCI tools
if: steps.reuse.outputs.reused != 'true'
run: |
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
VERSION="${{ needs.build-check.outputs.version }}"
CREATE_LATEST="${{ needs.build-check.outputs.create_latest }}"
VARIANT_SUFFIX="${{ matrix.suffix }}"
# Convert version format for Dockerfile compatibility. The former
# DOCKER_CHANNEL was "release" down every branch and was passed as a
# build-arg no Dockerfile declares, so it is gone.
case "$VERSION" in
"latest")
DOCKER_RELEASE="latest"
;;
v*)
# For versioned releases (v1.0.0), remove 'v' prefix for Dockerfile
DOCKER_RELEASE="${VERSION#v}"
;;
*)
# For other versions, pass as-is
DOCKER_RELEASE="${VERSION}"
;;
esac
echo "docker_release=$DOCKER_RELEASE" >> "$GITHUB_OUTPUT"
echo "🐳 Docker build parameters:"
echo " - Original version: $VERSION"
echo " - Docker RELEASE: $DOCKER_RELEASE"
# Generate tags based on build type
# Only support release and prerelease builds (no development builds)
TAG_BASE="${VERSION}${VARIANT_SUFFIX}"
TAGS="${{ env.REGISTRY_DOCKERHUB }}:$TAG_BASE,${{ env.REGISTRY_GHCR }}:$TAG_BASE,${{ env.REGISTRY_QUAY }}:$TAG_BASE"
# Add latest when requested (stable releases, and prereleases before GA)
if [[ "$CREATE_LATEST" == "true" ]]; then
TAGS="$TAGS,${{ env.REGISTRY_DOCKERHUB }}:latest${VARIANT_SUFFIX},${{ env.REGISTRY_GHCR }}:latest${VARIANT_SUFFIX},${{ env.REGISTRY_QUAY }}:latest${VARIANT_SUFFIX}"
fi
# Always add the channel tag for prereleases, independent of latest
if [[ "$BUILD_TYPE" == "prerelease" ]]; then
# Prerelease channel tags (alpha, beta, rc)
if [[ "$VERSION" == *"alpha"* ]]; then
CHANNEL="alpha"
elif [[ "$VERSION" == *"beta"* ]]; then
CHANNEL="beta"
elif [[ "$VERSION" == *"rc"* ]]; then
CHANNEL="rc"
fi
if [[ -n "$CHANNEL" ]]; then
TAGS="$TAGS,${{ env.REGISTRY_DOCKERHUB }}:${CHANNEL}${VARIANT_SUFFIX},${{ env.REGISTRY_GHCR }}:${CHANNEL}${VARIANT_SUFFIX},${{ env.REGISTRY_QUAY }}:${CHANNEL}${VARIANT_SUFFIX}"
fi
fi
# Output tags
echo "tags=$TAGS" >> "$GITHUB_OUTPUT"
# Generate labels
LABELS="org.opencontainers.image.title=RustFS"
LABELS="$LABELS,org.opencontainers.image.description=RustFS distributed object storage system"
LABELS="$LABELS,org.opencontainers.image.version=$VERSION"
SOURCE_REVISION="$(git rev-parse HEAD)"
LABELS="$LABELS,org.opencontainers.image.revision=$SOURCE_REVISION"
LABELS="$LABELS,org.opencontainers.image.source=${{ github.server_url }}/${{ github.repository }}"
LABELS="$LABELS,org.opencontainers.image.created=$(date -u +'%Y-%m-%dT%H:%M:%SZ')"
LABELS="$LABELS,org.opencontainers.image.build-type=$BUILD_TYPE"
echo "labels=$LABELS" >> "$GITHUB_OUTPUT"
echo "🐳 Generated Docker tags:"
echo "$TAGS" | tr ',' '\n' | sed 's/^/ - /'
echo "📋 Build type: $BUILD_TYPE"
echo "🔖 Version: $VERSION"
- name: Build and push Docker image
sudo apt-get update
sudo apt-get install -y skopeo
- uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 # v3
if: steps.reuse.outputs.reused != 'true'
- uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
if: steps.reuse.outputs.reused != 'true'
- name: Supply accepted binaries and pinned CLI
if: steps.reuse.outputs.reused != 'true'
env:
GH_TOKEN: ${{ github.token }}
run: python3 scripts/release/deliver.py image-prepare --target "$VARIANT"
- name: Build OCI archive without registry writes
if: steps.reuse.outputs.reused != 'true'
uses: docker/build-push-action@10e90e3645eae34f1e60eeb005ba3a3d33f178e8 # v6
with:
context: .
file: ${{ matrix.file }}
platforms: ${{ env.DOCKER_PLATFORMS }}
push: ${{ needs.build-check.outputs.should_push == 'true' }}
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
# No layer cache. This build compiles nothing — it downloads a
# release zip and runs apk/apt — so the cache could only save the
# minute or two those take, while creating a correctness problem: with
# RELEASE=latest the binary URL is resolved by curl *inside* a RUN
# layer, and the layer key does not include what that resolved to. A
# rebuild at the same RELEASE value (dispatch with version=latest, or
# a re-run of the same version) would hit the old layer and ship the
# previous release's binary. mode=max also consumed the same 10GB
# Actions cache quota the Rust lanes are fighting over.
#
# Only RELEASE is passed: it is the sole build-arg the Dockerfiles
# declare besides TARGETARCH. BUILDTIME, VERSION, BUILD_TYPE, REVISION
# and CHANNEL were never read by any stage (and BUILDTIME's $(date ...)
# was a literal here, not a shell substitution). BUILD_DATE and VCS_REF
# are declared by the Dockerfiles but deliberately left unset —
# supplying them would change the published image labels.
platforms: linux/amd64,linux/arm64
push: false
provenance: false
outputs: type=oci,dest=delivery/image-${{ matrix.variant }}.tar
build-args: |
RELEASE=${{ steps.meta.outputs.docker_release }}
provenance: true
sbom: true
# Add retry mechanism by splitting the build process
no-cache: false
pull: true
# Note: Manifest creation is no longer needed as we only build one variant
# Multi-arch manifests are automatically created by docker/build-push-action
# Report-only container image vulnerability scan
scan-docker-image:
name: Scan Docker Images
needs: [ build-check, build-docker ]
if: needs.build-check.outputs.should_build == 'true' && needs.build-check.outputs.should_push == 'true'
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
security-events: write
strategy:
fail-fast: false
matrix:
include:
- variant: musl
suffix: ""
- variant: glibc
suffix: "-glibc"
steps:
- name: Login to GitHub Container Registry
# docker/login-action v3
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
password: ${{ secrets.GHCR_PASSWORD }}
- name: Scan image with Trivy
# aquasecurity/trivy-action v0.36.0
BINARY_SOURCE=supplied
RELEASE=${{ fromJson(inputs.candidate_manifest).version }}
VCS_REF=${{ fromJson(inputs.candidate_manifest).source_sha }}
- name: Verify wrapped binaries and startup on both architectures
if: steps.reuse.outputs.reused != 'true'
run: python3 scripts/release/deliver.py image-smoke --target "$VARIANT"
- name: Scan amd64 image before sealing
if: steps.reuse.outputs.reused != 'true'
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25
env:
TRIVY_PLATFORM: linux/amd64
with:
image-ref: ${{ env.REGISTRY_GHCR }}:${{ needs.build-check.outputs.version }}${{ matrix.suffix }}
input: delivery-scan/image-${{ matrix.variant }}-amd64.tar
format: sarif
output: trivy-${{ matrix.variant }}.sarif
ignore-unfixed: true
vuln-type: os,library
severity: CRITICAL,HIGH
output: delivery/scan-${{ matrix.variant }}-amd64.sarif
exit-code: "0" # Keep the repository's report-only vulnerability policy.
- name: Scan arm64 image before sealing
if: steps.reuse.outputs.reused != 'true'
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25
env:
TRIVY_PLATFORM: linux/arm64
with:
input: delivery-scan/image-${{ matrix.variant }}-arm64.tar
format: sarif
output: delivery/scan-${{ matrix.variant }}-arm64.sarif
exit-code: "0"
# Surface findings in the Security tab; an artifact alone is write-only
# reporting nobody reviews.
- name: Upload scan results to code scanning
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4.37.0
- name: Record verified OCI identity
if: steps.reuse.outputs.reused != 'true'
run: python3 scripts/release/deliver.py image-record --target "$VARIANT"
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.reuse.outputs.reused != 'true'
with:
sarif_file: trivy-${{ matrix.variant }}.sarif
category: container-image-${{ matrix.variant }}
- name: Upload container scan report
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: container-image-scan-${{ matrix.variant }}
path: trivy-${{ matrix.variant }}.sarif
name: image-${{ matrix.variant }}
path: delivery/*
if-no-files-found: error
retention-days: 30
# Docker build summary
docker-summary:
name: Docker Build Summary
needs: [ build-check, build-docker ]
if: always() && needs.build-check.outputs.should_build == 'true'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Docker build completion summary
run: |
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
VERSION="${{ needs.build-check.outputs.version }}"
CREATE_LATEST="${{ needs.build-check.outputs.create_latest }}"
echo "🐳 Docker build completed successfully!"
echo "📦 Build type: $BUILD_TYPE"
echo "🔢 Version: $VERSION"
echo "🚀 Strategy: Images using pre-built binaries (release channel only)"
echo ""
case "$BUILD_TYPE" in
"preview")
echo "🔍 Preview Docker image has been built with ${VERSION} tags"
echo "⏭️ Preview images do not update latest or prerelease channels"
;;
"release")
echo "🚀 Release Docker image has been built with ${VERSION} tags"
echo "✅ This image is ready for production use"
if [[ "$CREATE_LATEST" == "true" ]]; then
echo "🏷️ Latest tag has been created for stable release"
fi
;;
"prerelease")
echo "🧪 Prerelease Docker image has been built with ${VERSION} tags"
echo "⚠️ This is a prerelease image - use with caution"
# Prereleases move latest until the first stable tag exists (pre-GA policy).
if [[ "$CREATE_LATEST" == "true" ]]; then
echo "🏷️ Latest tag has been created for prerelease: $VERSION"
else
echo "🚫 Latest tag NOT created for prerelease"
fi
;;
*)
echo "❌ Unexpected build type: $BUILD_TYPE"
;;
esac
compression-level: 0
+242
View File
@@ -0,0 +1,242 @@
# Copyright 2026 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Nightly both-stack diff of the two e2e profiles only ci.yml runs, e2e-smoke
# and e2e-full (rustfs/backlog#2772, T3.1). The other profiles carry a `stack`
# dimension in their own workflows, where the existing lane is the legacy leg.
#
# One build serves every leg. One preflight starts that binary with
# RUSTFS_S3_STACK=gateway (scripts/dual_stack.py preflight); without a proven
# gateway start the gateway legs do not run and the summary says `skipped:
# gateway stack refuses startup: <why>`. Which stack a leg ran is read from the
# servers' own start-up line. The gateway legs are report-only until T2.10.
#
# Left out of both stacks: `dual_stack_differential::` (servers with an explicit
# stack), `fake_s3_target::`, the upgrade write-readiness cases and one deadline
# case (no server), and `distributed_startup_regression_test::` (needs ci.yml's
# hooks-binary manifest). Both e2e-full stacks use ci.yml's pinned mc and
# Vault binaries and network fault-injection tools.
name: dual-stack-nightly
on:
workflow_dispatch:
schedule:
# 03:13 UTC nightly, clear of e2e-nightly (04:29), the ODM lane (05:23) and
# e2e-distributed (05:53).
- cron: "13 3 * * *"
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: ${{ github.event_name != 'schedule' }}
permissions:
contents: read
defaults:
run:
shell: bash
jobs:
build:
name: Build once and probe the gateway stack
if: github.repository == 'rustfs/rustfs'
# The runner and cache of ci.yml's build-rustfs-debug-binary.
runs-on: sm-standard-4
timeout-minutes: 60
outputs:
stacks: ${{ steps.preflight.outputs.stacks }}
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-dev
cache-save-if: 'false'
install-build-packaging-tools: 'false'
- name: Build the rustfs binary
run: python3 scripts/e2e_binary.py build --bins --features e2e-test-hooks
- name: Upload the binary
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: dual-stack-binary
path: |
target/debug/rustfs
target/debug/rustfs.e2e.json
if-no-files-found: error
retention-days: 1
# Without a proven gateway start there is no gateway leg to run.
- name: Gateway preflight
id: preflight
run: |
if python3 scripts/dual_stack.py preflight --binary target/debug/rustfs --out "$RUNNER_TEMP/preflight/stack.json"; then
echo 'stacks=["legacy","gateway"]' >> "$GITHUB_OUTPUT"
else
echo 'stacks=["legacy"]' >> "$GITHUB_OUTPUT"
fi
# The evidence stands in for a gateway leg that is skipped; a leg that runs overwrites it.
- name: Upload the preflight evidence for e2e-smoke
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: dual-stack-leg-e2e-smoke-gateway
path: ${{ runner.temp }}/preflight
if-no-files-found: warn
retention-days: 7
- name: Upload the preflight evidence for e2e-full
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: dual-stack-leg-e2e-full-gateway
path: ${{ runner.temp }}/preflight
if-no-files-found: warn
retention-days: 7
leg:
name: ${{ matrix.profile }} (${{ matrix.stack }}${{ matrix.stack == 'gateway' && ', report-only until T2.10' || '' }})
needs: [ build ]
runs-on: sm-standard-4
timeout-minutes: 120
# Report-only until T2.10: the gateway stack cannot serve the suites yet.
continue-on-error: ${{ matrix.stack == 'gateway' }}
strategy:
fail-fast: false
matrix:
profile: [ e2e-smoke, e2e-full ]
stack: ${{ fromJSON(needs.build.outputs.stacks) }}
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-dev
cache-save-if: 'false'
install-build-packaging-tools: 'false'
- name: Download the binary
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: dual-stack-binary
path: target/debug
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
- name: Install awscurl
run: |
python3 -m pip install --user --upgrade pip "awscurl==0.44"
echo "AWSCURL_PATH=$HOME/.local/bin/awscurl" >> "$GITHUB_ENV"
test -x "$HOME/.local/bin/awscurl"
- name: Install network fault-injection tools
if: matrix.profile == 'e2e-full'
run: |
sudo apt-get install -y iptables
sudo -n iptables --version
# The endpoint-blackhole heal scenario needs CAP_NET_ADMIN. Containerised
# runners can run iptables but not touch the rule set; the test then logs
# a skip instead of failing, so surface that here where it is visible.
if ! sudo -n iptables -w 5 -S OUTPUT >/dev/null 2>&1; then
echo "::warning::iptables cannot read the OUTPUT chain on this runner (no CAP_NET_ADMIN); the endpoint-blackhole heal scenario will be skipped"
fi
- name: Install mc
if: matrix.profile == 'e2e-full'
env:
GH_TOKEN: ${{ github.token }}
MC_VERSION: RELEASE.2025-08-13T08-35-41Z
MC_SHA256: 01f866e9c5f9b87c2b09116fa5d7c06695b106242d829a8bb32990c00312e891
run: |
if ! command -v gh >/dev/null 2>&1; then
sudo apt-get update
sudo apt-get install -y gh
fi
MC_BINARY="mc.linux-amd64.${MC_VERSION}"
python3 .github/actions/setup/download_release_asset.py --repository minio/mc --tag "$MC_VERSION" --asset "$MC_BINARY" --output "$RUNNER_TEMP/mc" --sha256 "$MC_SHA256"
echo "${MC_SHA256} $RUNNER_TEMP/mc" | sha256sum --check --status
chmod +x "$RUNNER_TEMP/mc"
echo "$RUNNER_TEMP" >> "$GITHUB_PATH"
- name: Verify mc
if: matrix.profile == 'e2e-full'
run: mc --version
- name: Install Vault
if: matrix.profile == 'e2e-full'
run: |
VAULT_VERSION="1.17.6"
VAULT_ARCHIVE="vault_${VAULT_VERSION}_linux_amd64.zip"
curl -fsSLo "$RUNNER_TEMP/$VAULT_ARCHIVE" "https://releases.hashicorp.com/vault/${VAULT_VERSION}/${VAULT_ARCHIVE}"
echo "0cddc1fbbb88583b5ba5b845f9f8fae47c6fb39a6d48cd543c6ba6fd3ac1a669 $RUNNER_TEMP/$VAULT_ARCHIVE" | sha256sum --check --status
unzip -q "$RUNNER_TEMP/$VAULT_ARCHIVE" -d "$RUNNER_TEMP/vault-bin"
echo "RUSTFS_TEST_VAULT_BIN=$RUNNER_TEMP/vault-bin/vault" >> "$GITHUB_ENV"
- name: Verify Vault
if: matrix.profile == 'e2e-full'
run: |
"$RUSTFS_TEST_VAULT_BIN" version
- name: Run the profile on the ${{ matrix.stack }} stack
env:
PROFILE: ${{ matrix.profile }}
RUSTFS_S3_STACK: ${{ matrix.stack }}
RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/server-logs
run: |
chmod +x target/debug/rustfs
filter='not test(/^(dual_stack_differential|fake_s3_target|distributed_startup_regression_test)::/)'
filter="$filter & not test(/^upgrade_compatibility_test::upgrade_write_readiness_tests::/)"
filter="$filter & not test(=list_objects_v2_pagination_test::tests::test_list_objects_v2_deadline_rejects_unresponsive_peer)"
python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run --profile "$PROFILE" -p e2e_test -E "$filter"
- name: Prove the stack and upload the leg
if: always()
uses: ./.github/actions/dual-stack-leg
with: { profile: "${{ matrix.profile }}", stack: "${{ matrix.stack }}", logs: "${{ runner.temp }}/server-logs" }
summary:
name: Both-stack diff summary
needs: [ build, leg ]
if: github.repository == 'rustfs/rustfs' && always()
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- uses: ./.github/actions/dual-stack-summary
with:
profiles: e2e-smoke,e2e-full
+33 -2
View File
@@ -53,11 +53,14 @@ concurrency:
jobs:
distributed:
if: github.repository == 'rustfs/rustfs'
name: Distributed 4-node 4-disk e2e
name: Distributed 4-node 4-disk e2e (${{ matrix.stack }}${{ matrix.stack == 'gateway' && ', report-only until T2.10' || '' }})
# GitHub-hosted VM: loop and tmpfs mounts work here. sm-standard-4 is an
# ARC pod and rejects both (`mount -o loop` ENOENT, tmpfs "read-only").
runs-on: ubuntu-latest
timeout-minutes: 180
# The gateway leg is report-only until T2.10.
continue-on-error: ${{ matrix.stack == 'gateway' }}
strategy: { fail-fast: false, matrix: { stack: [legacy, gateway] } }
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
NO_PROXY: 127.0.0.1,localhost
@@ -129,6 +132,11 @@ jobs:
run: |
python3 scripts/e2e_binary.py build --bins --features e2e-test-hooks
# A gateway binary that refuses to start skips the suite; the summary says why.
- name: Gateway preflight
if: matrix.stack == 'gateway'
run: python3 scripts/dual_stack.py preflight --binary target/debug/rustfs --out "$RUNNER_TEMP/leg/stack.json"
- name: Verify distributed e2e membership
env:
NEXTEST_LISTING: ${{ runner.temp }}/rustfs-e2e-distributed-list.json
@@ -140,6 +148,7 @@ jobs:
env:
RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-e2e-distributed-logs
FILTER: ${{ inputs.filter }}
RUSTFS_S3_STACK: ${{ matrix.stack }}
run: |
set -euo pipefail
rm -f target/nextest/e2e-distributed/junit.xml
@@ -150,11 +159,19 @@ jobs:
fi
test -s target/nextest/e2e-distributed/junit.xml
- name: Prove the stack and upload the leg
if: always()
uses: ./.github/actions/dual-stack-leg
with:
profile: e2e-distributed
stack: ${{ matrix.stack }}
logs: ${{ runner.temp }}/rustfs-e2e-distributed-logs
- name: Upload distributed e2e diagnostics
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-distributed-${{ github.run_number }}
name: e2e-distributed-${{ github.run_number }}-${{ matrix.stack }}
path: |
target/nextest/e2e-distributed/junit.xml
${{ runner.temp }}/rustfs-e2e-distributed-list.json
@@ -174,6 +191,20 @@ jobs:
fi
done
dual-stack-summary:
name: Both-stack diff summary
needs: [distributed]
if: github.repository == 'rustfs/rustfs' && always()
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- uses: ./.github/actions/dual-stack-summary
with:
profiles: e2e-distributed
alert-on-failure:
name: Alert on scheduled failure
needs: [distributed]
+82 -11
View File
@@ -54,14 +54,18 @@ concurrency:
jobs:
repl-nightly:
if: github.repository == 'rustfs/rustfs'
name: Replication e2e (nightly)
name: Replication e2e (nightly, ${{ matrix.stack }}${{ matrix.stack == 'gateway' && ', report-only until T2.10' || '' }})
# dual-node tests spawn two full rustfs servers each; the extra headroom
# over the per-PR sm-standard-2 e2e job keeps the parallel process fan-out
# from starving the runner. Mirrors the ILM serial lane's runner class.
runs-on: sm-standard-4
timeout-minutes: 45
# The gateway leg is report-only until T2.10.
continue-on-error: ${{ matrix.stack == 'gateway' }}
strategy: { fail-fast: false, matrix: { stack: [legacy, gateway] } }
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
RUSTFS_BUILD_FEATURES: ${{ matrix.stack == 'gateway' && 'gateway-stack' || '' }}
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
@@ -93,7 +97,12 @@ jobs:
# Build once and carry its source/binary identity into the test invocation.
- name: Build rustfs binary
run: |
python3 scripts/e2e_binary.py build --bins
python3 scripts/e2e_binary.py build --bins --features "$RUSTFS_BUILD_FEATURES"
# A gateway binary that refuses to start skips the suite; the summary says why.
- name: Gateway preflight
if: matrix.stack == 'gateway'
run: python3 scripts/dual_stack.py preflight --binary target/debug/rustfs --out "$RUNNER_TEMP/leg/stack.json"
- name: Verify replication e2e membership
env:
@@ -105,13 +114,22 @@ jobs:
- name: Run replication e2e nightly suite
env:
RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-e2e-repl-nightly-logs
run: python3 scripts/e2e_binary.py run -- cargo nextest run --profile e2e-repl-nightly -p e2e_test
RUSTFS_S3_STACK: ${{ matrix.stack }}
run: python3 scripts/e2e_binary.py run --features "$RUSTFS_BUILD_FEATURES" -- cargo nextest run --profile e2e-repl-nightly -p e2e_test
- name: Prove the stack and upload the leg
if: always()
uses: ./.github/actions/dual-stack-leg
with:
profile: e2e-repl-nightly
stack: ${{ matrix.stack }}
logs: ${{ runner.temp }}/rustfs-e2e-repl-nightly-logs
- name: Upload nextest junit report
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-replication-nightly-junit-${{ github.run_number }}
name: e2e-replication-nightly-junit-${{ github.run_number }}-${{ matrix.stack }}
path: |
target/nextest/e2e-repl-nightly/junit.xml
${{ runner.temp }}/rustfs-e2e-repl-nightly-list.json
@@ -121,9 +139,12 @@ jobs:
cluster-nightly:
if: github.repository == 'rustfs/rustfs'
name: Cluster fault e2e (nightly)
name: Cluster fault e2e (nightly, ${{ matrix.stack }}${{ matrix.stack == 'gateway' && ', report-only until T2.10' || '' }})
runs-on: sm-standard-4
timeout-minutes: 90
# The gateway leg is report-only until T2.10.
continue-on-error: ${{ matrix.stack == 'gateway' }}
strategy: { fail-fast: false, matrix: { stack: [legacy, gateway] } }
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
@@ -144,6 +165,11 @@ jobs:
run: |
python3 scripts/e2e_binary.py build --bins --features e2e-test-hooks
# A gateway binary that refuses to start skips the suite; the summary says why.
- name: Gateway preflight
if: matrix.stack == 'gateway'
run: python3 scripts/dual_stack.py preflight --binary target/debug/rustfs --out "$RUNNER_TEMP/leg/stack.json"
- name: Verify cluster fault e2e membership
env:
NEXTEST_LISTING: ${{ runner.temp }}/rustfs-e2e-nightly-list.json
@@ -155,13 +181,22 @@ jobs:
env:
CARGO_BIN_EXE_rustfs: ${{ github.workspace }}/target/debug/rustfs
RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-e2e-nightly-logs
RUSTFS_S3_STACK: ${{ matrix.stack }}
run: python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run --profile e2e-nightly -p e2e_test
- name: Prove the stack and upload the leg
if: always()
uses: ./.github/actions/dual-stack-leg
with:
profile: e2e-nightly
stack: ${{ matrix.stack }}
logs: ${{ runner.temp }}/rustfs-e2e-nightly-logs
- name: Upload cluster fault diagnostics
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-cluster-nightly-${{ github.run_number }}
name: e2e-cluster-nightly-${{ github.run_number }}-${{ matrix.stack }}
path: |
target/nextest/e2e-nightly/junit.xml
${{ runner.temp }}/rustfs-e2e-nightly-list.json
@@ -171,12 +206,15 @@ jobs:
protocols-nightly:
if: github.repository == 'rustfs/rustfs'
name: Protocol e2e (nightly)
name: Protocol e2e (nightly, ${{ matrix.stack }}${{ matrix.stack == 'gateway' && ', report-only until T2.10' || '' }})
runs-on: sm-standard-4
timeout-minutes: 90
# The gateway leg is report-only until T2.10.
continue-on-error: ${{ matrix.stack == 'gateway' }}
strategy: { fail-fast: false, matrix: { stack: [legacy, gateway] } }
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
RUSTFS_BUILD_FEATURES: ftps,webdav,sftp
RUSTFS_BUILD_FEATURES: ${{ matrix.stack == 'gateway' && 'ftps,webdav,sftp,gateway-stack' || 'ftps,webdav,sftp' }}
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
@@ -200,6 +238,11 @@ jobs:
- name: Build protocol server
run: python3 scripts/e2e_binary.py build --features "$RUSTFS_BUILD_FEATURES"
# A gateway binary that refuses to start skips the suite; the summary says why.
- name: Gateway preflight
if: matrix.stack == 'gateway'
run: python3 scripts/dual_stack.py preflight --binary target/debug/rustfs --out "$RUNNER_TEMP/leg/stack.json"
# The suite owns fixed protocol ports and serializes its internal cases.
- name: Verify protocol e2e membership
env:
@@ -211,14 +254,28 @@ jobs:
- name: Run protocol e2e nightly suite
env:
RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-protocol-e2e-logs
run: >-
python3 scripts/e2e_binary.py run --features "$RUSTFS_BUILD_FEATURES" -- cargo nextest run -j 1 --profile e2e-protocols -p e2e_test --no-capture
RUSTFS_S3_STACK: ${{ matrix.stack }}
# These servers set no log level of their own; the stack proof reads their start-up line.
RUST_LOG: rustfs=info
# The servers inherit nextest's stdout (--no-capture): keep it where the stack proof reads it.
run: |
set -o pipefail
mkdir -p "$RUSTFS_E2E_LOG_DIR"
python3 scripts/e2e_binary.py run --features "$RUSTFS_BUILD_FEATURES" -- cargo nextest run -j 1 --profile e2e-protocols -p e2e_test --no-capture 2>&1 | tee "$RUSTFS_E2E_LOG_DIR/nextest.log"
- name: Prove the stack and upload the leg
if: always()
uses: ./.github/actions/dual-stack-leg
with:
profile: e2e-protocols
stack: ${{ matrix.stack }}
logs: ${{ runner.temp }}/rustfs-protocol-e2e-logs
- name: Upload protocol diagnostics
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-protocol-nightly-${{ github.run_number }}
name: e2e-protocol-nightly-${{ github.run_number }}-${{ matrix.stack }}
path: |
target/nextest/e2e-protocols/junit.xml
${{ runner.temp }}/rustfs-e2e-protocols-list.json
@@ -226,6 +283,20 @@ jobs:
retention-days: 7
if-no-files-found: warn
dual-stack-summary:
name: Both-stack diff summary
needs: [repl-nightly, cluster-nightly, protocols-nightly]
if: github.repository == 'rustfs/rustfs' && always()
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- uses: ./.github/actions/dual-stack-summary
with:
profiles: e2e-repl-nightly,e2e-nightly,e2e-protocols
alert-on-failure:
name: Alert on scheduled failure
needs: [repl-nightly, cluster-nightly, protocols-nightly]
+395
View File
@@ -0,0 +1,395 @@
# Copyright 2026 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Gateway black-box acceptance lanes (rustfs/backlog#2775 T3.4, #2776 T3.5).
#
# Runs two suites of the rustfs/gateway repository against the candidate binary,
# once per S3 stack (RUSTFS_S3_STACK=legacy and =gateway), one server per stack:
#
# - the conformance corpus, `--profile rustfs`. That profile describes the
# gateway edge, so scripts/check_conformance_rulings.py judges the gateway
# stack against .config/conformance-rulings.toml (red: an unruled failure, a
# ruled case failing differently, a stale ruling, a skip on the server's own
# answer) and publishes the legacy run as a reference that is never judged:
# it only has to serve, stay up and complete;
# - the client matrix, ci/compat/run_matrix.sh --external, judged per stack by
# scripts/compat_matrix_report.py (no `fail`, no `sut-unregistered`, every
# client ran; a client that could not be installed is reported as skipped).
# The legacy matrix is the baseline: every cell that passes on legacy must
# pass on gateway (`compare`).
#
# The suites are checked out at the rustfs/gateway rev Cargo.toml pins, so the
# suite and the framework under test never drift apart. The verdict of each
# stack is computed from what the run left on disk (startup record, the stack
# the server's own log names, runner exit code, liveness after the corpus,
# report), never from the exit status of an earlier step. Needs
# scripts/dual_stack.py and the `s3_stack` startup field of rustfs/rustfs#9367.
#
# The gateway stack is report-only until T2.10: its job may fail without failing
# the workflow, and its summary says what it observed (did not start, or how many
# cases ran, passed, failed and were answered 501). The stack comparison is
# report-only for the same reason; its conformance half is information only.
#
# Not a pull-request gate. Scheduled failures are owned by the scheduled-validation
# watchdog (.github/scheduled-validations.json).
name: Gateway Acceptance
on:
schedule:
- cron: "29 3 * * *"
workflow_dispatch:
# Matches ci.yml: these variables are hashed into the rust-cache key, so the
# candidate build reads the same ci-dev cache as build-rustfs-debug-binary.
env:
CARGO_TERM_COLOR: always
RUST_BACKTRACE: 1
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
permissions:
contents: read
defaults:
run:
shell: bash
jobs:
plan:
name: Read the gateway pin
runs-on: ubuntu-latest
timeout-minutes: 5
outputs:
gateway-rev: ${{ steps.pin.outputs.rev }}
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Read the rustfs/gateway rev from Cargo.toml
id: pin
run: |
set -euo pipefail
python3 - <<'PY' >> "$GITHUB_OUTPUT"
import re, sys, tomllib
dependencies = tomllib.load(open("Cargo.toml", "rb"))["workspace"]["dependencies"]
revs = {spec.get("rev") for spec in dependencies.values()
if isinstance(spec, dict) and spec.get("git") == "https://github.com/rustfs/gateway.git"}
if len(revs) != 1 or not re.fullmatch(r"[0-9a-f]{40}", str(next(iter(revs)))):
sys.exit(f"Cargo.toml must pin rustfs/gateway at exactly one full rev, found {sorted(map(str, revs))}")
print(f"rev={revs.pop()}")
PY
cat "$GITHUB_OUTPUT"
build-candidate:
name: Build candidate binary
# The same debug build as ci.yml's build-rustfs-debug-binary, reading the
# ci-dev cache written on sm-standard-4.
runs-on: sm-standard-4
timeout-minutes: 60
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-dev
cache-save-if: 'false'
install-build-packaging-tools: 'false'
- name: Build debug binary
run: python3 scripts/e2e_binary.py build --bins --features e2e-test-hooks
- name: Upload candidate binary
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: gateway-acceptance-candidate-binary
overwrite: true
path: |
target/debug/rustfs
target/debug/rustfs.e2e.json
if-no-files-found: error
retention-days: 1
build-gateway-tools:
name: Build gateway conformance runner and compat-sut
needs: [ plan ]
# The gateway workspace builds with its own rust-toolchain.toml, which rustup
# installs on first use; nothing here names a toolchain.
runs-on: ubuntu-latest
timeout-minutes: 60
steps:
- name: Checkout rustfs/gateway at the pinned rev
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/gateway
ref: ${{ needs.plan.outputs.gateway-rev }}
path: gateway
persist-credentials: false
- name: Build
working-directory: gateway
run: |
cargo build --locked --release -p rustfs-gateway-conformance -p rustfs-gateway-compat-sut \
--bin rustfs-gateway-conformance --bin compat-sut
- name: Upload tools
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: gateway-acceptance-tools
overwrite: true
path: |
gateway/target/release/rustfs-gateway-conformance
gateway/target/release/compat-sut
if-no-files-found: error
retention-days: 1
acceptance:
name: ${{ matrix.stack == 'gateway' && 'Acceptance (gateway, report-only until T2.10)' || 'Acceptance (legacy)' }}
needs: [ plan, build-candidate, build-gateway-tools ]
# GitHub-hosted like the s3-tests targets of reference-parity.yml: the job
# only runs prebuilt binaries and the pinned clients, none of which needs Docker.
runs-on: ubuntu-latest
# About 3x the 15 minutes the legacy job took in run 43325101234 (client install about
# 8, matrix about 3, corpus about 3), so a hung server costs 45 runner minutes, not more.
timeout-minutes: 45
continue-on-error: ${{ matrix.stack == 'gateway' }}
strategy:
fail-fast: false
matrix:
stack: [ legacy, gateway ]
env:
STACK: ${{ matrix.stack }}
ENDPOINT: http://127.0.0.1:9000
# The conformance suite signs with these fixed fixture credentials (the AWS
# documentation example pair), so the candidate is started with them and the
# client matrix is handed the same pair.
ACCESS_KEY: AKIAIOSFODNN7EXAMPLE
SECRET_KEY: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Checkout rustfs/gateway at the pinned rev
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/gateway
ref: ${{ needs.plan.outputs.gateway-rev }}
path: gateway
persist-credentials: false
- name: Download candidate binary
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: gateway-acceptance-candidate-binary
path: target/debug
- name: Download gateway tools
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: gateway-acceptance-tools
path: tools
# Records `ready`, or what the process said when it exited, in lane/startup.txt. `ready`
# also needs the server's own `server_starting` log event to name the requested stack
# (its `s3_stack` field comes from rustfs/rustfs#9367; scripts/dual_stack.py reads it).
- name: Start candidate (${{ matrix.stack }} stack)
run: |
set -uo pipefail
chmod +x target/debug/rustfs tools/*
mkdir -p lane "$RUNNER_TEMP"/data/rustfs{0..3}
target/debug/rustfs --version | tee lane/candidate-version.txt
# The four directories share the runner's one disk; the bypass is the CI case the
# physical-disk guard sanctions (as in mint.yml).
RUSTFS_S3_STACK="$STACK" RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true RUST_LOG=info target/debug/rustfs --address 127.0.0.1:9000 \
--access-key "$ACCESS_KEY" --secret-key "$SECRET_KEY" "$RUNNER_TEMP"/data/rustfs{0..3} \
> lane/rustfs.log 2>&1 &
pid=$!
echo "$pid" > lane/rustfs.pid
# Liveness first: a ready answer counts only while this process is still running.
for _ in $(seq 120); do
if ! kill -0 "$pid" 2>/dev/null; then
status=0
wait "$pid" || status=$?
echo "did not start: exit ${status}: $(grep -v '^[[:space:]]*$' lane/rustfs.log | tail -n 1)" > lane/startup.txt
cat lane/startup.txt
exit 1
fi
if curl -sf "$ENDPOINT/health/ready" >/dev/null; then
if python3 scripts/dual_stack.py stack --expect "$STACK" --out lane/stack.json lane/rustfs.log > /dev/null; then
echo ready > lane/startup.txt
exit 0
fi
echo "did not start: stack not observed" > lane/startup.txt
cat lane/startup.txt
exit 1
fi
sleep 1
done
echo "did not start: /health/ready did not answer within 120 seconds" > lane/startup.txt
exit 1
# The exit code is data for the judge step, not this step's result.
- name: Run gateway conformance (rustfs profile)
run: |
set +e
tools/rustfs-gateway-conformance run --root gateway/conformance --profile rustfs \
--endpoint "$ENDPOINT" --allow-external-fixtures \
--json lane/report.json > lane/conformance.log 2>&1
echo "$?" > lane/runner-exit.txt
if kill -0 "$(cat lane/rustfs.pid)" 2>/dev/null; then echo alive; else echo exited; fi > lane/alive-after-conformance.txt
tail -n 20 lane/conformance.log
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
# Toolchains for the matrix's SDK driver programs, as the gateway's own
# client-matrix.yml sets them up; the client versions come from
# gateway/compat/versions.toml.
- uses: actions/setup-go@41dfa10bad2bb2ae585af6ee5bb4d7d973ad74ed # v5.1.0
with:
go-version: "1.26"
cache: false
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: "22"
- uses: actions/setup-dotnet@67a3573c9a986a3f9c594539f4ab511d57bb3ce9 # v4.3.1
with:
dotnet-version: "8.0.x"
- uses: actions/setup-java@cf277c60eb25467037889841efdb72551f06f6c3 # v4.9.1
with:
distribution: temurin
java-version: "21"
- name: Install Maven
env:
MAVEN_VERSION: "3.9.16"
MAVEN_SHA512: "831a8591fe20c8243b1dbe7d71e3244f31d1665b0804b2e825e38cbbe5ce0cafb8338851f90780735568773e0a6cd07bbec107cda0b896b008b861075358b6f6"
run: |
set -euo pipefail
archive="$RUNNER_TEMP/apache-maven.tar.gz"
curl -sSfL -o "$archive" "https://archive.apache.org/dist/maven/maven-3/${MAVEN_VERSION}/binaries/apache-maven-${MAVEN_VERSION}-bin.tar.gz"
echo "${MAVEN_SHA512} ${archive}" | sha512sum -c -
tar -xzf "$archive" -C "$RUNNER_TEMP"
echo "$RUNNER_TEMP/apache-maven-${MAVEN_VERSION}/bin" >> "$GITHUB_PATH"
# One client at a time, so a client that cannot be installed is reported
# as skipped with its reason instead of stopping the others.
- name: Install pinned clients
working-directory: gateway
run: |
set -uo pipefail
python3 -m pip install --disable-pip-version-check "pyyaml==6.0.2"
installed=()
for client in $(python3 -c 'import tomllib; print(" ".join(sorted(tomllib.load(open("compat/versions.toml", "rb"))["clients"])))'); do
if bash ci/compat/install_clients.sh --clients "$client" > "../lane/install-${client}.log" 2>&1; then
installed+=("$client")
else
echo "${client}: skipped: $(tail -n 1 "../lane/install-${client}.log")" >> ../lane/skipped-clients.txt
fi
done
echo "COMPAT_CLIENTS=$(IFS=,; echo "${installed[*]}")" >> "$GITHUB_ENV"
cat ../lane/skipped-clients.txt 2>/dev/null || echo "every pinned client installed"
# The exit code is data for the judge step, not this step's result.
- name: Run client matrix (external)
working-directory: gateway
env:
GATEWAY_COMPAT_SUT_BIN: ${{ github.workspace }}/tools/compat-sut
GATEWAY_COMPAT_RUN_DIR: ${{ github.workspace }}/lane/compat
GATEWAY_COMPAT_MATRIX_OUT: ${{ github.workspace }}/lane/compat/matrix.json
run: |
set +e
export GATEWAY_COMPAT_ACCESS_KEY="$ACCESS_KEY" GATEWAY_COMPAT_SECRET_KEY="$SECRET_KEY"
PATH="$(go env GOPATH)/bin:$PATH"
if [[ -f ../lane/skipped-clients.txt ]]; then
bash ci/compat/run_matrix.sh --external "$ENDPOINT" --product rustfs --clients "$COMPAT_CLIENTS"
else
bash ci/compat/run_matrix.sh --external "$ENDPOINT" --product rustfs
fi
echo "$?" > ../lane/matrix-exit.txt
- name: Judge and summarize
if: always()
run: |
set -uo pipefail
status=0
python3 scripts/check_conformance_rulings.py --stack "$STACK" --run-dir lane | tee -a "$GITHUB_STEP_SUMMARY" || status=1
python3 scripts/compat_matrix_report.py check --stack "$STACK" --run-dir lane | tee -a "$GITHUB_STEP_SUMMARY" || status=1
exit "$status"
- name: Upload lane results
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: gateway-acceptance-lane-${{ matrix.stack }}
overwrite: true
path: |
lane/*.txt
lane/*.log
lane/report.json
lane/stack.json
lane/compat/matrix.json
lane/compat/probe.jsonl
lane/compat/sut.log
lane/compat/results/**
if-no-files-found: error
retention-days: 14
compare:
name: Compare stacks (report-only until T2.10)
needs: [ acceptance ]
if: always()
runs-on: ubuntu-latest
timeout-minutes: 10
continue-on-error: true
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download lane results
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
pattern: gateway-acceptance-lane-*
path: lanes
- name: Diff legacy and gateway
run: |
set -uo pipefail
status=0
{
python3 scripts/check_conformance_rulings.py \
--diff lanes/gateway-acceptance-lane-legacy lanes/gateway-acceptance-lane-gateway || status=1
echo
python3 scripts/compat_matrix_report.py diff \
--legacy lanes/gateway-acceptance-lane-legacy --gateway lanes/gateway-acceptance-lane-gateway || status=1
} > diff.md
cat diff.md
cat diff.md >> "$GITHUB_STEP_SUMMARY"
exit "$status"
+51 -184
View File
@@ -1,136 +1,26 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name: Publish helm chart to artifacthub
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Build candidate Helm chart
on:
workflow_run:
workflows: [ "Build and Release" ]
types: [ completed ]
workflow_dispatch:
workflow_call:
inputs:
version:
description: "Release version to publish, e.g. 1.0.0-beta.1 or v1.0.0-beta.1"
required: true
default: "1.0.0-beta.1"
candidate_manifest:
type: string
required: true
permissions:
contents: read
actions: read
jobs:
build-helm-package:
package:
runs-on: ubuntu-latest
timeout-minutes: 30
if: |
(github.event_name == 'workflow_dispatch' && !contains(github.event.inputs.version, '-preview')) ||
(
github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push' &&
contains(github.event.workflow_run.head_branch, '.') &&
!contains(github.event.workflow_run.head_branch, '-preview')
)
outputs:
raw_tag: ${{ steps.version.outputs.raw_tag }}
app_version: ${{ steps.version.outputs.app_version }}
chart_version: ${{ steps.version.outputs.chart_version }}
timeout-minutes: 15
steps:
- name: Checkout release source
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: ${{ github.event.workflow_run.head_sha || (startsWith(inputs.version, 'refs/tags/') && inputs.version || format('refs/tags/{0}', inputs.version)) }}
ref: ${{ fromJson(inputs.candidate_manifest).source_sha }}
persist-credentials: false
# Both inputs reach the shell through env rather than `${{ }}`
# interpolation. A git ref name may contain `$(...)` — anything without a
# space is a legal tag — and interpolation pastes it into the script
# verbatim, where bash would run it. Reading "$RAW_INPUT" instead makes it
# data.
- name: Normalize release version
id: version
env:
RAW_INPUT: ${{ github.event.inputs.version }}
RAW_BRANCH: ${{ github.event.workflow_run.head_branch }}
run: |
set -eux
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
RAW="$RAW_INPUT"
else
RAW="$RAW_BRANCH"
fi
case "$RAW" in
refs/tags/*)
RAW_TAG="${RAW#refs/tags/}"
;;
*)
RAW_TAG="$RAW"
;;
esac
./scripts/helm_chart_version.sh "$RAW_TAG"
# Docker and Helm share the binary-build trigger. Do not publish a chart
# until the image selected by its default appVersion is pullable.
- name: Wait for release image
timeout-minutes: 20
env:
APP_VERSION: ${{ steps.version.outputs.app_version }}
run: |
set -euo pipefail
if [[ ! "$APP_VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.-]+)?$ ]] ||
[[ "$APP_VERSION" == *-preview* ]]; then
echo "Invalid deliverable image version: $APP_VERSION" >&2
exit 1
fi
for attempt in {1..60}; do
if token=$(curl --fail --silent --show-error --connect-timeout 10 --max-time 20 \
'https://auth.docker.io/token?service=registry.docker.io&scope=repository:rustfs/rustfs:pull' \
| jq -er '.token // empty') &&
curl --fail --silent --show-error --head --connect-timeout 10 --max-time 20 \
--header "Authorization: Bearer $token" \
--header 'Accept: application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json' \
"https://registry-1.docker.io/v2/rustfs/rustfs/manifests/$APP_VERSION" > /dev/null; then
echo "Release image is available: rustfs/rustfs:$APP_VERSION"
exit 0
fi
echo "Waiting for rustfs/rustfs:$APP_VERSION ($attempt/60)"
if (( attempt < 60 )); then
sleep 15
fi
done
echo "Release image is unavailable; refusing to publish Helm chart" >&2
exit 1
- name: Replace chart version and app version
env:
CHART_VERSION: ${{ steps.version.outputs.chart_version }}
APP_VERSION: ${{ steps.version.outputs.app_version }}
run: |
set -eux
sed -i -E "s/^version:.*/version: \"${CHART_VERSION}\"/" helm/rustfs/Chart.yaml
sed -i -E "s/^appVersion:.*/appVersion: \"${APP_VERSION}\"/" helm/rustfs/Chart.yaml
- name: Set up Helm
uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
- name: Install yq
- uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
- name: Install pinned yq
run: |
set -euo pipefail
yq_version="4.54.1"
@@ -145,66 +35,43 @@ jobs:
echo "$tools_dir" >> "$GITHUB_PATH"
"$tools_dir/yq" --version
- name: Test Helm Chart Templates
run: ./scripts/test_helm_templates.sh
- name: Package Helm Chart
run: |
set -eux
cp helm/README.md helm/rustfs/
helm package ./helm/rustfs \
--destination helm/rustfs/ \
--version "${{ steps.version.outputs.chart_version }}"
- name: Upload helm package as artifact
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: helm-package
path: |
helm/rustfs/*.tgz
helm/rustfs/README.md
retention-days: 1
publish-helm-package:
runs-on: ubuntu-latest
timeout-minutes: 30
needs: [ build-helm-package ]
if: needs.build-helm-package.result == 'success'
steps:
- name: Checkout helm package repo
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
# persist-credentials-exempt: this checkout's token IS the push credential —
# the job git-pushes to rustfs/helm below. Clearing it breaks chart publishing.
repository: rustfs/helm
token: ${{ secrets.RUSTFS_HELM_PACKAGE }}
- name: Download helm package
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: helm-package
path: ./
- name: Set up helm
uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
- name: Generate index
run: helm repo index . --url https://charts.rustfs.com
# app_version is derived from the triggering tag name, and this job holds
# the cross-repository push token with rustfs/helm already checked out —
# the worst place in the repo to paste an attacker-influenced string into
# a shell line. Passed through env so bash treats it as data.
- name: Push helm package and index file
- name: Test and package the exact application version
id: chart
env:
GIT_USERNAME: ${{ secrets.USERNAME }}
GIT_EMAIL: ${{ secrets.EMAIL_ADDRESS }}
APP_VERSION: ${{ needs.build-helm-package.outputs.app_version }}
GH_TOKEN: ${{ github.token }}
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
run: |
set -eux
git config --global user.name "${GIT_USERNAME}"
git config --global user.email "${GIT_EMAIL}"
git add .
git commit -m "Update rustfs helm package with ${APP_VERSION}." || echo "No changes to commit"
git push origin main
python3 - <<'PYCHART'
import sys, subprocess, shutil
from pathlib import Path
sys.path.insert(0, 'scripts/release')
import candidate as c, pipeline
inputs = pipeline.manifest_env()
current = pipeline.current_run('release-complete.yml')
if any(a['name'] == 'helm-package' for a in c.artifact_list(current['id'])):
c.download_artifact(current, 'helm-package', 'chart')
record = c.read_json('chart/chart.json')
c.require(record['inputs_sha256'] == c.digest(inputs), 'chart belongs to another candidate')
c.emit_output('reused', 'true')
else:
chart = Path('helm/rustfs/Chart.yaml')
import re
value = chart.read_text()
for key in ('version', 'appVersion'):
value, count = re.subn(rf'(?m)^{key}:.*$', f'{key}: "{inputs["version"]}"', value)
c.require(count == 1, 'ambiguous chart metadata')
chart.write_text(value)
subprocess.run(['bash', 'scripts/test_helm_templates.sh'], check=True)
shutil.copyfile('helm/README.md', 'helm/rustfs/README.md')
subprocess.run(['helm', 'package', 'helm/rustfs', '--destination', 'chart'], check=True)
path = Path('chart') / f"rustfs-{inputs['version']}.tgz"
c.write_json('chart/chart.json', {'version': inputs['version'], 'inputs_sha256': c.digest(inputs),
'name': path.name, 'sha256': c.file_digest(path), 'size': path.stat().st_size})
PYCHART
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.chart.outputs.reused != 'true'
with:
name: helm-package
path: chart/*
if-no-files-found: error
retention-days: 30
+463
View File
@@ -0,0 +1,463 @@
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
# Manual Linux zip/deb/rpm packaging. Workflow artifacts only.
# Dispatch uses this file from the selected branch. An empty ref packages
# the tip of refs/heads/main. A non-empty ref must already exist; the
# resolver fetches that tag, branch, or commit so checkout is not limited
# to the default branch. Nothing is published.
name: Manual Linux packages
on:
workflow_dispatch:
inputs:
architecture:
description: "Linux targets: x86 (x86_64 musl+gnu), arm (aarch64 musl+gnu), or all"
required: true
type: choice
options:
- x86
- arm
- all
default: all
ref:
description: "Git tag to package. Leave empty to package the tip of origin/main. A branch name or full commit SHA is also accepted."
required: false
type: string
default: ""
version:
description: "Package version: MAJOR.MINOR.PATCH, or MAJOR.MINOR.PATCH-(alpha|beta|rc).N. No leading v. Required for origin/main. For a tag, leave empty to derive it from the tag; an explicit value is used as entered."
required: false
type: string
default: ""
permissions:
contents: read
env:
CARGO_TERM_COLOR: always
RUST_BACKTRACE: 1
CARGO_INCREMENTAL: 0
jobs:
prepare:
name: Resolve source and version
if: github.repository == 'rustfs/rustfs'
runs-on: ubuntu-latest
timeout-minutes: 15
outputs:
source_sha: ${{ steps.source.outputs.source_sha }}
source_kind: ${{ steps.source.outputs.source_kind }}
source_name: ${{ steps.source.outputs.source_name }}
version: ${{ steps.source.outputs.version }}
packaging: ${{ steps.source.outputs.packaging }}
tool_sha: ${{ steps.source.outputs.tool_sha }}
confirm_from: ${{ steps.source.outputs.confirm_from }}
toolchain: ${{ steps.inputs.outputs.toolchain }}
console: ${{ steps.inputs.outputs.console }}
matrix: ${{ steps.matrix.outputs.matrix }}
steps:
# Tooling checkout. An empty ref then detaches this tree to the tip of
# origin/main. A tag or other ref is fetched and pinned without
# replacing this tree, because console resolution needs candidate.py.
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: main
fetch-depth: 0
persist-credentials: false
- name: Resolve source and version
id: source
shell: bash
env:
SOURCE_REF: ${{ inputs.ref }}
PACKAGE_VERSION: ${{ inputs.version }}
run: |
set -euo pipefail
bash scripts/release/package_manual_artifacts.sh resolve-source \
--source "$SOURCE_REF" \
--version "$PACKAGE_VERSION" | tee -a "$GITHUB_OUTPUT"
- name: Install stable Rust
uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable
with:
toolchain: stable
- name: Resolve console assets and toolchain
id: inputs
shell: bash
env:
GH_TOKEN: ${{ github.token }}
run: |
set -euo pipefail
toolchain=$(rustc --version | awk '{print $2}')
[[ "$toolchain" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]
gh api repos/rustfs/console/releases/latest > console-release.json
TOOLCHAIN="$toolchain" python3 - <<'PY'
import json, os, sys
sys.path.insert(0, "scripts/release")
import candidate
console = candidate.console_asset(candidate.read_json("console-release.json"))
candidate.emit_output("console", json.dumps(console, separators=(",", ":")))
candidate.emit_output("toolchain", os.environ["TOOLCHAIN"])
print(f"toolchain={os.environ['TOOLCHAIN']}")
print(f"console={console['tag']} {console['name']}")
PY
- name: Select Linux targets
id: matrix
shell: bash
env:
ARCHITECTURE: ${{ inputs.architecture }}
run: |
set -euo pipefail
matrix=$(bash scripts/release/package_manual_artifacts.sh matrix "$ARCHITECTURE")
count=$(python3 -c 'import json,sys; print(len(json.loads(sys.argv[1])["include"]))' "$matrix")
[[ "$count" -gt 0 ]]
printf 'matrix=%s\n' "$matrix" >> "$GITHUB_OUTPUT"
printf 'Selected %s target(s) for %s\n' "$count" "$ARCHITECTURE"
package:
name: Package RustFS (${{ matrix.target_id }})
needs: prepare
if: github.repository == 'rustfs/rustfs'
# x86_64 uses sm-standard-4, the official release compile pool: the workspace
# release build needs that disk (placement rule: disk) and reads build.yml's
# cache without writing it. aarch64 uses ubicloud-standard-16-arm so the
# packaged binary and console smoke execute natively (placement rule: hardware).
runs-on: ${{ matrix.os }}
timeout-minutes: 210
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
RUSTUP_TOOLCHAIN: ${{ needs.prepare.outputs.toolchain }}
RUSTFLAGS: "${{ matrix.rustflags }}"
strategy:
fail-fast: false
matrix: ${{ fromJson(needs.prepare.outputs.matrix) }}
steps:
- name: Checkout the pinned source commit
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: ${{ needs.prepare.outputs.source_sha }}
fetch-depth: 0
persist-credentials: false
# Tags cut before the manual packaging script still compile from their
# own tree. The official zip/deb/rpm recipe, and confirm-source when the
# tagged script does not implement it, come from the prepare checkout.
- name: Checkout packaging recipe
if: ${{ needs.prepare.outputs.packaging == 'tool' || needs.prepare.outputs.confirm_from == 'tool' }}
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: ${{ needs.prepare.outputs.tool_sha }}
path: packaging-tools
fetch-depth: 1
persist-credentials: false
- name: Confirm the pinned source
shell: bash
env:
SOURCE_KIND: ${{ needs.prepare.outputs.source_kind }}
SOURCE_NAME: ${{ needs.prepare.outputs.source_name }}
SOURCE_SHA: ${{ needs.prepare.outputs.source_sha }}
run: |
set -euo pipefail
helper=scripts/release/package_manual_artifacts.sh
if [[ ! -f "$helper" ]] || ! grep -q 'confirm-source)' "$helper"; then
helper=packaging-tools/scripts/release/package_manual_artifacts.sh
fi
if [[ ! -f "$helper" ]] || ! grep -q 'confirm-source)' "$helper"; then
printf 'no packaging script implements confirm-source\n' >&2
exit 1
fi
bash "$helper" confirm-source \
--kind "$SOURCE_KIND" \
--name "$SOURCE_NAME" \
--sha "$SOURCE_SHA"
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: ${{ needs.prepare.outputs.toolchain }}
target: ${{ matrix.target }}
cache-shared-key: build-${{ matrix.target }}${{ matrix.glibc && format('-glibc-{0}', matrix.glibc) || '' }}${{ matrix.os == 'ubicloud-standard-16-arm' && '-ubicloud-arm' || '' }}
# build.yml owns this key. This job only restores it.
cache-save-if: 'false'
install-cross-tools: ${{ matrix.cross }}
install-build-packaging-tools: 'true'
install-test-tools: 'false'
- name: Download static console assets
shell: bash
env:
GH_TOKEN: ${{ github.token }}
CONSOLE_IDENTITY: ${{ needs.prepare.outputs.console }}
run: |
set -euo pipefail
python3 - <<'PYCONSOLE'
import json, os, pathlib, subprocess, zipfile
value = json.loads(os.environ["CONSOLE_IDENTITY"])
subprocess.run(["python3", ".github/actions/setup/download_release_asset.py",
"--repository", "rustfs/console", "--tag", value["tag"], "--asset", value["name"],
"--sha256", value["sha256"], "--output", "console.zip"], check=True)
destination = pathlib.Path("rustfs/static")
destination.mkdir(parents=True, exist_ok=True)
with zipfile.ZipFile("console.zip") as archive:
for member in archive.infolist():
path = pathlib.PurePosixPath(member.filename)
if path.is_absolute() or ".." in path.parts or "\\" in member.filename:
raise SystemExit("unsafe Console archive member")
archive.extractall(destination)
if not (destination / "index.html").is_file():
raise SystemExit("Console archive has no index.html")
PYCONSOLE
rm -f console.zip
- name: Build RustFS
shell: bash
env:
RUSTFS_BUILD_VERSION: ${{ needs.prepare.outputs.version }}
BUILD_TARGET: ${{ matrix.target }}
BUILD_OS: ${{ matrix.build_os }}
BUILD_FEATURES: ${{ matrix.features }}
BUILD_GLIBC: ${{ matrix.glibc }}
BUILD_CROSS: ${{ matrix.cross }}
run: |
set -euo pipefail
touch rustfs/build.rs
feature_args=()
if [[ -n "$BUILD_FEATURES" ]]; then
feature_args=(--features "$BUILD_FEATURES")
fi
# gnu targets pin glibc 2.31 through cargo-zigbuild (<target>.<glibc>).
target_arg="$BUILD_TARGET"
if [[ -n "$BUILD_GLIBC" ]]; then
target_arg="${target_arg}.${BUILD_GLIBC}"
fi
profile_args=()
if [[ "$BUILD_TARGET" == *-linux-gnu && "$BUILD_FEATURES" == pyroscope ]]; then
profile_args=(--config 'profile.release.strip="none"')
fi
case "$BUILD_CROSS" in
true)
cargo zigbuild --release --locked --target "$target_arg" "${feature_args[@]}" "${profile_args[@]}" -p rustfs --bin rustfs
;;
false)
cargo build --release --locked --target "$target_arg" "${feature_args[@]}" "${profile_args[@]}" -p rustfs --bin rustfs
;;
*)
printf 'unexpected cross flag: %q\n' "$BUILD_CROSS" >&2
exit 1
;;
esac
bin="target/${BUILD_TARGET}/release/rustfs"
version_output=$("$bin" --version)
first=${version_output%%$'\n'*}
if [[ "$first" != "rustfs ${RUSTFS_BUILD_VERSION}" ]]; then
printf 'binary version line is %q, expected %q\n' "$first" "rustfs ${RUSTFS_BUILD_VERSION}" >&2
printf '%s\n' "$version_output" >&2
exit 1
fi
if ! grep -Fxq "build target : ${BUILD_TARGET}" <<< "$version_output" || \
! grep -Fxq "build os : ${BUILD_OS}" <<< "$version_output" || \
! grep -Fxq "rust channel : stable-${BUILD_TARGET}" <<< "$version_output"; then
printf 'release binary reports an unexpected compile target\n' >&2
printf '%s\n' "$version_output" >&2
exit 1
fi
if [[ -n "$BUILD_GLIBC" ]]; then
glibc_check=scripts/release/check_glibc.py
if [[ ! -f "$glibc_check" ]]; then
glibc_check=packaging-tools/scripts/release/check_glibc.py
fi
if [[ ! -f "$glibc_check" ]]; then
printf 'glibc check script not found\n' >&2
exit 1
fi
python3 "$glibc_check" "$BUILD_GLIBC" "$bin"
fi
- name: Generate the CPU symbol catalog
if: endsWith(matrix.target, '-linux-gnu') && matrix.features == 'pyroscope'
shell: bash
env:
BUILD_TARGET: ${{ matrix.target }}
run: |
set -euo pipefail
cargo metadata --locked --format-version 1 --filter-platform "$BUILD_TARGET" \
--features rustfs/pyroscope > "$RUNNER_TEMP/cpu-catalog-metadata.json"
features=$(jq -ce '
(.packages[] | select(.name == "rustfs") | .id) as $id |
.resolve.nodes[] | select(.id == $id) | .features | map(select(. != "default")) | sort
' "$RUNNER_TEMP/cpu-catalog-metadata.json")
release_dir="target/${BUILD_TARGET}/release"
rm -f "$release_dir/rustfs.cpu-symbol-catalog.json" "$release_dir/rustfs.cpu-symbol-catalog.sha256"
cargo run --locked -p rustfs-connect-symbol-catalog -- \
"$release_dir/rustfs" "$(git rev-parse HEAD)" "$BUILD_TARGET" "$features" \
"$release_dir/rustfs.cpu-symbol-catalog.json" > "$release_dir/rustfs.cpu-symbol-catalog.sha256"
- name: Install DEB and RPM tools
shell: bash
run: |
set -euo pipefail
sudo apt-get update
sudo apt-get install -y ruby ruby-dev build-essential rpm fakeroot cpio
sudo gem install fpm --version 1.17.0 --no-document
- name: Package zip, deb, and rpm
shell: bash
env:
PACKAGE_VERSION: ${{ needs.prepare.outputs.version }}
ARCH: ${{ matrix.arch }}
LIBC: ${{ matrix.libc }}
BUILD_TARGET: ${{ matrix.target }}
BUILD_FEATURES: ${{ matrix.features }}
SOURCE_KIND: ${{ needs.prepare.outputs.source_kind }}
SOURCE_SHA: ${{ needs.prepare.outputs.source_sha }}
PACKAGING: ${{ needs.prepare.outputs.packaging }}
run: |
set -euo pipefail
case "$PACKAGING" in
source) script=scripts/release/package_manual_artifacts.sh ;;
tool) script=packaging-tools/scripts/release/package_manual_artifacts.sh ;;
*)
printf 'unexpected packaging recipe: %q\n' "$PACKAGING" >&2
exit 1
;;
esac
if [[ ! -f "$script" ]]; then
printf 'packaging script not found: %s\n' "$script" >&2
exit 1
fi
binary="target/${BUILD_TARGET}/release/rustfs"
extra=()
if [[ "$BUILD_TARGET" == *-linux-gnu && "$BUILD_FEATURES" == pyroscope ]]; then
extra=(
--zip-extra "target/${BUILD_TARGET}/release/rustfs.cpu-symbol-catalog.json"
--zip-extra "target/${BUILD_TARGET}/release/rustfs.cpu-symbol-catalog.sha256"
)
fi
bash "$script" package \
--version "$PACKAGE_VERSION" \
--arch "$ARCH" \
--libc "$LIBC" \
--binary "$binary" \
--output dist \
"${extra[@]}"
printf 'Packaged %s from %s %s\n' "$PACKAGE_VERSION" "$SOURCE_KIND" "$SOURCE_SHA"
- name: Verify the packaged binary
shell: bash
env:
PACKAGE_VERSION: ${{ needs.prepare.outputs.version }}
ARCH: ${{ matrix.arch }}
LIBC: ${{ matrix.libc }}
BUILD_TARGET: ${{ matrix.target }}
BUILD_OS: ${{ matrix.build_os }}
BUILD_FEATURES: ${{ matrix.features }}
run: |
set -euo pipefail
stem="rustfs-linux-${ARCH}-${LIBC}-v${PACKAGE_VERSION}"
package_file="dist/${stem}.zip"
[[ -f "$package_file" && -f "dist/${stem}.deb" && -f "dist/${stem}.rpm" ]]
[[ "$(find dist -maxdepth 1 -type f | wc -l)" -eq 3 ]]
archive_members=$(unzip -Z1 "$package_file" | sort)
if [[ "$BUILD_TARGET" == *-linux-gnu && "$BUILD_FEATURES" == pyroscope ]]; then
expected_members=$(printf '%s\n' rustfs rustfs.cpu-symbol-catalog.json rustfs.cpu-symbol-catalog.sha256 | sort)
else
expected_members=$(printf '%s\n' rustfs)
fi
if [[ "$archive_members" != "$expected_members" ]]; then
printf 'Linux package does not contain the expected files\n' >&2
printf '%s\n' "$archive_members" >&2
exit 1
fi
binary_dir=$(mktemp -d)
trap 'rm -rf "$binary_dir"' EXIT
unzip -q "$package_file" rustfs -d "$binary_dir"
version_output=$("$binary_dir/rustfs" --version)
first=${version_output%%$'\n'*}
if [[ "$first" != "rustfs ${PACKAGE_VERSION}" ]]; then
printf 'packaged binary version line is %q, expected %q\n' "$first" "rustfs ${PACKAGE_VERSION}" >&2
exit 1
fi
if ! grep -Fxq "build target : ${BUILD_TARGET}" <<< "$version_output" || \
! grep -Fxq "build os : ${BUILD_OS}" <<< "$version_output"; then
printf 'packaged binary reports an unexpected compile target\n' >&2
exit 1
fi
- name: Verify packaged console
if: matrix.target == 'x86_64-unknown-linux-gnu' || matrix.target == 'aarch64-unknown-linux-gnu'
shell: bash
env:
PACKAGE_VERSION: ${{ needs.prepare.outputs.version }}
ARCH: ${{ matrix.arch }}
LIBC: ${{ matrix.libc }}
run: |
set -euo pipefail
package_file="dist/rustfs-linux-${ARCH}-${LIBC}-v${PACKAGE_VERSION}.zip"
find_free_port() {
python3 - <<'PY'
import socket
with socket.socket() as sock:
sock.bind(("127.0.0.1", 0))
print(sock.getsockname()[1])
PY
}
package_dir=$(mktemp -d)
data_dir=$(mktemp -d)
log_file="${RUNNER_TEMP}/rustfs-console-smoke.log"
server_pid=""
trap '
if [[ -n "${server_pid:-}" ]]; then
kill "$server_pid" 2>/dev/null || true
wait "$server_pid" 2>/dev/null || true
fi
rm -rf "$package_dir" "$data_dir"
' EXIT
unzip -q "$package_file" -d "$package_dir"
api_port=$(find_free_port)
console_port=$(find_free_port)
console_url="http://127.0.0.1:${console_port}/rustfs/console/"
"$package_dir/rustfs" server \
--address "127.0.0.1:${api_port}" \
--console-enable \
--console-address "127.0.0.1:${console_port}" \
--access-key console-smoke \
--secret-key console-smoke-secret \
"$data_dir" >"$log_file" 2>&1 &
server_pid=$!
for _ in {1..40}; do
response=$(curl --silent --output /dev/null --write-out '%{http_code} %{content_type}' "$console_url" || true)
if [[ "$response" == 200\ text/html* ]]; then
exit 0
fi
sleep 0.25
done
printf 'Console endpoint did not return 200 text/html: %s\n' "$response" >&2
cat "$log_file"
exit 1
- name: Upload packages
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: packages-${{ matrix.target_id }}-v${{ needs.prepare.outputs.version }}
path: dist/*
if-no-files-found: error
compression-level: 0
retention-days: 30
+90 -9
View File
@@ -44,6 +44,13 @@
# well inside the 120-minute budget and buys deterministic infra. The
# docker-capable self-hosted `dind-sm-standard-2` label was the alternative but
# has fewer cores and reintroduces fleet-state risk for no reliability gain.
#
# Stack dimension (rustfs/backlog#2774, T3.3): one image, run with RUSTFS_S3_STACK=legacy and
# =gateway. `compare` (scripts/mint_stack_diff.py) holds the gate that every case legacy passes
# passes on gateway with no new failure. The gateway leg and `compare` are report-only until
# T2.10. A gateway server that refuses to start stops its leg at "Wait for RustFS ready", before
# mint runs; the summary says `skipped: gateway stack refuses startup: <why>`. The stack is read
# from the server's own start-up line (scripts/dual_stack.py stack).
name: mint
@@ -108,13 +115,9 @@ defaults:
shell: bash
jobs:
mint:
# GitHub-hosted: reliably provides a running Docker daemon + buildx +
# python3. See the header note (ci-2) for why the self-hosted sm-standard-4
# label was abandoned (no docker.sock on the pod its only run landed on).
# TODO(ci-8): scheduled-failure alerting (auto-open issue on a red weekly
# run) is added by the ci-8 composite action; do not implement it here.
runs-on: sm-standard-2
image:
name: Build the RustFS image once
runs-on: ubuntu-latest
timeout-minutes: 120
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
@@ -132,6 +135,48 @@ jobs:
--cache-to "type=gha,mode=max,scope=${BUILDX_CACHE_SCOPE}" \
-t rustfs-ci \
-f Dockerfile.source .
docker save rustfs-ci --output "${RUNNER_TEMP}/rustfs-ci.tar"
- name: Upload the image
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: mint-image
path: ${{ runner.temp }}/rustfs-ci.tar
retention-days: 1
mint:
name: mint (${{ matrix.stack }}${{ matrix.stack == 'gateway' && ', report-only until T2.10' || '' }})
needs: [image]
# Report-only until T2.10: the gateway stack cannot serve the suites yet.
continue-on-error: ${{ matrix.stack == 'gateway' }}
strategy:
fail-fast: false
matrix:
stack: [legacy, gateway]
env:
STACK: ${{ matrix.stack }}
# GitHub-hosted: reliably provides a running Docker daemon + buildx +
# python3. See the header note (ci-2) for why the self-hosted sm-standard-4
# label was abandoned (no docker.sock on the pod its only run landed on).
# sm-standard-2 has no docker.sock either (weekly runs failed at "Enable
# buildx" from 2026-09-27 to 2026-11-08), so keep this job on ubuntu-latest.
# TODO(ci-8): scheduled-failure alerting (auto-open issue on a red weekly
# run) is added by the ci-8 composite action; do not implement it here.
runs-on: ubuntu-latest
timeout-minutes: 120
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download the image
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: mint-image
path: ${{ runner.temp }}
- name: Load the image
run: docker load --input "${RUNNER_TEMP}/rustfs-ci.tar"
- name: Start RustFS
run: |
@@ -149,6 +194,8 @@ jobs:
-e RUSTFS_SECRET_KEY="${S3_SECRET_KEY}" \
-e RUSTFS_VOLUMES="/data/rustfs{0...3}" \
-e RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true \
-e RUSTFS_S3_STACK="${STACK}" \
-e RUST_LOG=rustfs=info \
-v /tmp/rustfs-mint:/data \
rustfs-ci
@@ -244,16 +291,50 @@ jobs:
mkdir -p artifacts/rustfs-mint
docker logs rustfs-mint > artifacts/rustfs-mint/rustfs.log 2>&1 || true
- name: Prove which stack the server selected
if: always()
run: python3 scripts/dual_stack.py stack --expect "${STACK}" --out artifacts/stack.json artifacts/rustfs-mint
- name: Upload artifacts
if: always() && env.ACT != 'true'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: mint
name: mint-${{ matrix.stack }}
path: artifacts/**
compare:
name: Compare stacks (report-only until T2.10)
needs: [mint]
if: always()
continue-on-error: true
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download both legs
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
pattern: mint-{legacy,gateway}
path: legs
# A leg that never uploaded leaves no directory: create both so that the
# comparison reports "no result" for it instead of failing to find it.
- name: Compare the stacks
run: |
# The shell runs with -e; the exit status is reported after the summary is written.
set +e -u -o pipefail
mkdir -p legs/mint-legacy legs/mint-gateway
python3 scripts/mint_stack_diff.py legs/mint-legacy legs/mint-gateway | tee report.txt
status="${PIPESTATUS[0]}"
{ echo '## Mint: gateway versus legacy'; echo; echo '```'; cat report.txt; echo '```'; } >> "$GITHUB_STEP_SUMMARY"
exit "$status"
alert-on-failure:
name: Alert on scheduled failure
needs: [mint]
needs: [image, mint]
# `always()` is required: without it this job is skipped when a needed
# job fails. Alerts only for scheduled runs (backlog#1149 ci-8) — manual
# dispatch failures are already watched by a human.
+2 -2
View File
@@ -47,8 +47,8 @@ permissions:
contents: read
concurrency:
group: oidc-keycloak-live-${{ github.ref }}
cancel-in-progress: true
group: oidc-keycloak-live-${{ github.event_name }}-${{ github.ref }}
cancel-in-progress: ${{ github.event_name != 'schedule' }}
jobs:
oidc-keycloak-live:
@@ -68,11 +68,14 @@ env:
jobs:
minio-source:
name: MinIO source (read-through, list-through, backfill)
name: MinIO source (read-through, list-through, backfill, ${{ matrix.stack }}${{ matrix.stack == 'gateway' && ', report-only until T2.10' || '' }})
# Keep this scheduled interoperability lane scoped to the upstream repo.
if: github.repository == 'rustfs/rustfs'
runs-on: ubuntu-latest
timeout-minutes: 90
# The gateway leg is report-only until T2.10.
continue-on-error: ${{ matrix.stack == 'gateway' }}
strategy: { fail-fast: false, matrix: { stack: [legacy, gateway] } }
env:
NO_PROXY: 127.0.0.1,localhost
# Fixed credentials of the container this job starts and throws away;
@@ -137,6 +140,11 @@ jobs:
- name: Build the RustFS binary under test
run: python3 scripts/e2e_binary.py build --bins
# A gateway binary that refuses to start skips the suite; the summary says why.
- name: Gateway preflight
if: matrix.stack == 'gateway'
run: python3 scripts/dual_stack.py preflight --binary target/debug/rustfs --out "$RUNNER_TEMP/leg/stack.json"
# The lane selects tests by module, so a rename would quietly shrink it.
# The committed digest in .config/e2e-odm-interop-selection.txt fails
# closed on that.
@@ -146,8 +154,19 @@ jobs:
python3 ./scripts/check_test_wiring.py --check-profile e2e-odm-interop "${NEXTEST_LISTING}"
- name: Run the interop cases against MinIO
env:
RUSTFS_S3_STACK: ${{ matrix.stack }}
RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-e2e-logs
run: python3 scripts/e2e_binary.py run -- cargo nextest run --profile e2e-odm-interop -p e2e_test --no-tests=fail
- name: Prove the stack and upload the leg
if: always()
uses: ./.github/actions/dual-stack-leg
with:
profile: e2e-odm-interop
stack: ${{ matrix.stack }}
logs: ${{ runner.temp }}/rustfs-e2e-logs
- name: Build the MinIO interop report
if: always()
uses: ./.github/actions/odm-interop-report
@@ -171,7 +190,7 @@ jobs:
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: odm-interop-minio-${{ github.run_number }}-${{ github.run_attempt }}
name: odm-interop-minio-${{ github.run_number }}-${{ github.run_attempt }}-${{ matrix.stack }}
path: |
artifacts/odm-interop/minio
target/nextest/e2e-odm-interop/junit.xml
@@ -297,6 +316,20 @@ jobs:
target/nextest/e2e-odm-interop/junit.xml
retention-days: 14
dual-stack-summary:
name: Both-stack diff summary
needs: [minio-source]
if: github.repository == 'rustfs/rustfs' && always()
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- uses: ./.github/actions/dual-stack-summary
with:
profiles: e2e-odm-interop
alert-on-failure:
name: Alert on scheduled failure
needs: [minio-source, cloud-source]
+38 -629
View File
@@ -1,644 +1,53 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Package Workflow - Build DEB/RPM packages
#
# This workflow builds DEB and RPM packages from pre-built Linux binaries
# and uploads them to Cloudflare R2 and the GitHub release.
#
# Trigger:
# - workflow_run: automatically package after "Build and Release" completes
# for a release tag (the mac/windows/linux binaries are already uploaded
# to the GitHub release before packaging starts)
# - workflow_dispatch: manual fallback with a release tag and/or exact build run ID
#
# Flow:
# 1. Resolve and validate the selected Build workflow run and source identity
# 2. Download Linux binaries (x86_64-gnu, aarch64-gnu, x86_64-musl,
# aarch64-musl) from build artifacts
# 3. Build DEB packages for amd64 and arm64, gnu and musl variants
# 4. Build RPM packages for x86_64 and aarch64, gnu and musl variants
# 5. Upload all packages to Cloudflare R2 and the GitHub release
name: Package DEB/RPM
permissions:
# contents: write is required to upload packages to the GitHub release
contents: write
actions: read
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Build candidate packages
on:
# Follows the same pattern as docker.yml: run after the release build
# workflow completes, so packaging is triggered only by release tags
# (e.g. 1.0.0-rc.2, 1.0.0-rc.3), never by development builds.
workflow_run:
workflows: [ "Build and Release" ]
types: [ completed ]
workflow_dispatch:
workflow_call:
inputs:
tag:
description: "Release tag to package (e.g. 1.0.0-beta.12). Leave empty for latest main build."
required: false
candidate_manifest:
type: string
build_run_id:
description: "Build workflow run ID (when combined with tag, both must identify the same release commit)"
required: false
type: string
concurrency:
group: ${{ github.workflow }}-${{ github.event.workflow_run.head_branch || github.event.inputs.tag || github.run_id }}
cancel-in-progress: true
env:
HEAD_BRANCH: ${{ github.event.workflow_run.head_branch }}
WORKFLOW_RUN_ID: ${{ github.event.workflow_run.id }}
required: true
permissions:
contents: read
actions: read
jobs:
# Resolve which build run to use and extract version info
resolve:
name: Resolve Build
# Auto-trigger only from successful tag builds of "Build and Release".
# Tag pushes arrive as event == push with head_branch != main (a
# non-main push head_branch is the release tag name). Manual dispatch
# stays available as a fallback for backfills and re-runs.
if: >-
github.event_name == 'workflow_dispatch' ||
(github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push' &&
github.event.workflow_run.head_branch != 'main')
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 10
outputs:
version: ${{ steps.resolve.outputs.version }}
build_type: ${{ steps.resolve.outputs.build_type }}
build_run_id: ${{ steps.resolve.outputs.build_run_id }}
build_run_number: ${{ steps.resolve.outputs.build_run_number }}
head_sha: ${{ steps.resolve.outputs.head_sha }}
dev_sequence: ${{ steps.resolve.outputs.dev_sequence }}
tag: ${{ steps.resolve.outputs.tag }}
steps:
- name: Resolve build run
id: resolve
shell: bash
env:
GH_TOKEN: ${{ github.token }}
EVENT_NAME: ${{ github.event_name }}
REPOSITORY: ${{ github.repository }}
INPUT_TAG: ${{ github.event.inputs.tag }}
INPUT_RUN_ID: ${{ github.event.inputs.build_run_id }}
run: |
set -euo pipefail
fail() {
echo "❌ $1" >&2
exit 1
}
TAG=""
BUILD_RUN_ID=""
case "$EVENT_NAME" in
workflow_run)
TAG="$HEAD_BRANCH"
BUILD_RUN_ID="$WORKFLOW_RUN_ID"
;;
workflow_dispatch)
TAG="$INPUT_TAG"
BUILD_RUN_ID="$INPUT_RUN_ID"
;;
*) fail "unsupported event: $EVENT_NAME" ;;
esac
# Validate and classify tags before using them in API paths or logs.
semver_core='(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)'
prerelease_id='(alpha|beta|rc)\.(0|[1-9][0-9]*)'
if [[ -n "$TAG" ]]; then
if [[ "$TAG" =~ ^${semver_core}(-${prerelease_id})?-preview\.(0|[1-9][0-9]*)$ ]]; then
BUILD_TYPE=preview
elif [[ "$TAG" =~ ^${semver_core}-${prerelease_id}$ ]]; then
BUILD_TYPE=prerelease
elif [[ "$TAG" =~ ^${semver_core}$ ]]; then
BUILD_TYPE=release
else
fail "tag is not a supported strict package version"
fi
else
BUILD_TYPE=development
fi
if [[ -n "$BUILD_RUN_ID" ]]; then
[[ "$BUILD_RUN_ID" =~ ^[1-9][0-9]*$ ]] || fail "build run ID must be a positive decimal integer"
echo "Using selected build run: $BUILD_RUN_ID"
elif [[ -n "$TAG" ]]; then
echo "Looking for build run for tag: $TAG"
BUILD_RUN_ID=$(gh api --method GET \
"repos/${REPOSITORY}/actions/workflows/build.yml/runs" \
-f branch="$TAG" -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' 2>/dev/null || true)
if [[ -z "$BUILD_RUN_ID" ]]; then
BUILD_RUN_ID=$(gh api --method GET \
"repos/${REPOSITORY}/actions/workflows/build.yml/runs" \
-f event=push -f status=success -F per_page=100 2>/dev/null |
jq -r --arg tag "$TAG" \
'[.workflow_runs[] | select(.head_branch == $tag)][0].id // empty' || true)
fi
[[ "$BUILD_RUN_ID" =~ ^[1-9][0-9]*$ ]] || fail "no successful build run found for tag"
echo "Found build run: $BUILD_RUN_ID"
else
echo "No tag specified, looking for latest main build"
BUILD_RUN_ID=$(gh api --method GET \
"repos/${REPOSITORY}/actions/workflows/build.yml/runs" \
-f branch=main -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' 2>/dev/null || true)
[[ "$BUILD_RUN_ID" =~ ^[1-9][0-9]*$ ]] || fail "no successful main build found"
echo "Latest main build: $BUILD_RUN_ID"
fi
# Fetch once and use the same immutable run metadata for identity,
# ordering, workflow provenance, and release-channel validation.
RUN_JSON=$(gh api "repos/${REPOSITORY}/actions/runs/${BUILD_RUN_ID}") ||
fail "cannot read selected build run"
RUN_ID=$(jq -r '.id // empty' <<<"$RUN_JSON")
RUN_NUMBER=$(jq -r '.run_number // empty' <<<"$RUN_JSON")
RUN_STATUS=$(jq -r '.status // empty' <<<"$RUN_JSON")
RUN_CONCLUSION=$(jq -r '.conclusion // empty' <<<"$RUN_JSON")
RUN_PATH=$(jq -r '.path // empty' <<<"$RUN_JSON")
HEAD_SHA=$(jq -r '.head_sha // empty' <<<"$RUN_JSON")
RUN_HEAD_BRANCH=$(jq -r '.head_branch // empty' <<<"$RUN_JSON")
[[ "$RUN_ID" == "$BUILD_RUN_ID" ]] || fail "run metadata ID mismatch"
[[ "$RUN_NUMBER" =~ ^[1-9][0-9]*$ ]] || fail "build run number must be a positive decimal integer"
[[ "$RUN_STATUS" == completed && "$RUN_CONCLUSION" == success ]] || fail "selected build run is not successful"
[[ "$RUN_PATH" == .github/workflows/build.yml ]] || fail "selected run is not Build and Release"
[[ "$HEAD_SHA" =~ ^[0-9a-f]{40}$ ]] || fail "selected build run has an invalid head SHA"
[[ "$RUN_HEAD_BRANCH" != *$'\n'* && -n "$RUN_HEAD_BRANCH" ]] || fail "selected build run has an invalid head branch"
if [[ -n "$TAG" ]]; then
[[ "$RUN_HEAD_BRANCH" == "$TAG" ]] || fail "tag and build run head branch do not match"
TAG_REF_JSON=$(gh api "repos/${REPOSITORY}/git/ref/tags/${TAG}") ||
fail "cannot resolve release tag ref"
TAG_OBJECT_TYPE=$(jq -r '.object.type // empty' <<<"$TAG_REF_JSON")
TAG_OBJECT_SHA=$(jq -r '.object.sha // empty' <<<"$TAG_REF_JSON")
depth=0
while [[ "$TAG_OBJECT_TYPE" == tag && $depth -lt 5 ]]; do
TAG_OBJECT_JSON=$(gh api "repos/${REPOSITORY}/git/tags/${TAG_OBJECT_SHA}") ||
fail "cannot peel annotated release tag"
TAG_OBJECT_TYPE=$(jq -r '.object.type // empty' <<<"$TAG_OBJECT_JSON")
TAG_OBJECT_SHA=$(jq -r '.object.sha // empty' <<<"$TAG_OBJECT_JSON")
depth=$((depth + 1))
done
[[ "$TAG_OBJECT_TYPE" == commit && "$TAG_OBJECT_SHA" =~ ^[0-9a-f]{40}$ ]] ||
fail "release tag does not resolve to a commit"
[[ "$TAG_OBJECT_SHA" == "$HEAD_SHA" ]] || fail "release tag commit and build run head SHA do not match"
VERSION="$TAG"
DEV_SEQUENCE=""
else
VERSION="dev-${HEAD_SHA}"
DEV_SEQUENCE="$RUN_NUMBER"
fi
{
echo "version=$VERSION"
echo "build_type=$BUILD_TYPE"
echo "build_run_id=$BUILD_RUN_ID"
echo "build_run_number=$RUN_NUMBER"
echo "head_sha=$HEAD_SHA"
echo "dev_sequence=$DEV_SEQUENCE"
echo "tag=${TAG}"
} >> "$GITHUB_OUTPUT"
echo "📊 Resolved:"
echo " Version: $VERSION"
echo " Build type: $BUILD_TYPE"
echo " Build run ID: $BUILD_RUN_ID"
echo " Build run number: $RUN_NUMBER"
# Build DEB and RPM packages for each architecture and libc variant
package:
name: Package (${{ matrix.arch }}-${{ matrix.libc }})
needs: resolve
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
runs-on: ubuntu-latest
timeout-minutes: 45
strategy:
fail-fast: false
max-parallel: 1
matrix:
include:
- arch: x86_64
libc: gnu
deb_arch: amd64
rpm_arch: x86_64
artifact_name: "rustfs-linux-x86_64-gnu"
# gnu binaries dynamically link glibc; musl binaries are static
# and run without a libc dependency.
deb_depends: "libc6 (>= 2.31)"
rpm_depends: "glibc >= 2.31"
- arch: aarch64
libc: gnu
deb_arch: arm64
rpm_arch: aarch64
artifact_name: "rustfs-linux-aarch64-gnu"
deb_depends: "libc6 (>= 2.31)"
rpm_depends: "glibc >= 2.31"
- arch: x86_64
libc: musl
deb_arch: amd64
rpm_arch: x86_64
artifact_name: "rustfs-linux-x86_64-musl"
deb_depends: ""
rpm_depends: ""
- arch: aarch64
libc: musl
deb_arch: arm64
rpm_arch: aarch64
artifact_name: "rustfs-linux-aarch64-musl"
deb_depends: ""
rpm_depends: ""
target: [linux-x86_64-musl, linux-aarch64-musl, linux-x86_64-gnu, linux-aarch64-gnu]
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: ${{ fromJson(inputs.candidate_manifest).source_sha }}
persist-credentials: false
- name: Normalize package metadata
id: versions
shell: bash
env:
BUILD_TYPE: ${{ needs.resolve.outputs.build_type }}
SOURCE_VERSION: ${{ needs.resolve.outputs.version }}
DEV_SEQUENCE: ${{ needs.resolve.outputs.dev_sequence }}
ARCH: ${{ matrix.rpm_arch }}
LIBC: ${{ matrix.libc }}
run: |
set -euo pipefail
normalized=$(./scripts/release/package_versions.sh \
"$BUILD_TYPE" "$SOURCE_VERSION" "$DEV_SEQUENCE" "$ARCH" "$LIBC")
printf '%s\n' "$normalized" >> "$GITHUB_OUTPUT"
- name: Download binary artifact from build run
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
pattern: ${{ matrix.artifact_name }}*
path: ./binary-artifact
run-id: ${{ needs.resolve.outputs.build_run_id }}
github-token: ${{ github.token }}
merge-multiple: true
- name: Extract binary
id: binary
shell: bash
run: |
set -euo pipefail
ZIP_FILE=$(find ./binary-artifact -name "*.zip" -type f | head -1)
if [[ -z "$ZIP_FILE" ]]; then
echo "❌ No binary artifact found"
find ./binary-artifact -mindepth 1 -maxdepth 1 -print 2>/dev/null || true
exit 1
fi
echo "Found artifact: $ZIP_FILE"
mkdir -p ./bin
unzip -o "$ZIP_FILE" -d ./bin
if [[ ! -f ./bin/rustfs ]]; then
echo "❌ rustfs binary not found in archive"
exit 1
fi
chmod +x ./bin/rustfs
stat --printf='%n %s bytes\n' ./bin/rustfs
echo "✅ Binary extracted"
- name: Build DEB package
id: deb
shell: bash
env:
DEB_VERSION: ${{ steps.versions.outputs.deb_version }}
DEB_ARCH: ${{ matrix.deb_arch }}
DEB_FILE: ${{ steps.versions.outputs.deb_file }}
DEB_DEPENDS: ${{ matrix.deb_depends }}
LIBC: ${{ matrix.libc }}
run: |
set -euo pipefail
PKG_DIR="${DEB_FILE%.deb}"
echo "Building DEB: ${DEB_FILE}"
mkdir -p "${PKG_DIR}/DEBIAN"
mkdir -p "${PKG_DIR}/usr/bin"
mkdir -p "${PKG_DIR}/etc/default"
mkdir -p "${PKG_DIR}/lib/systemd/system"
mkdir -p "${PKG_DIR}/usr/share/doc/rustfs"
cp ./bin/rustfs "${PKG_DIR}/usr/bin/"
chmod 755 "${PKG_DIR}/usr/bin/rustfs"
cp deploy/build/rustfs.service "${PKG_DIR}/lib/systemd/system/"
cat > "${PKG_DIR}/etc/default/rustfs" << 'ENVEOF'
# RustFS Environment Configuration
# See https://docs.rustfs.com for more information
# RUSTFS_ACCESS_KEY=<changeme>
# RUSTFS_SECRET_KEY=<changeme>
# RUSTFS_VOLUMES=""
# RUSTFS_ADDRESS=":9000"
# RUSTFS_CONSOLE_ADDRESS=":9001"
# RUSTFS_CONSOLE_ENABLE=true
# RUSTFS_OBS_LOGGER_LEVEL=error
# RUSTFS_OBS_LOG_DIRECTORY="/var/log/rustfs/"
ENVEOF
cat > "${PKG_DIR}/DEBIAN/control" << EOF
Package: rustfs
Version: ${DEB_VERSION}
Section: utils
Priority: optional
Architecture: ${DEB_ARCH}
EOF
# gnu binaries dynamically link glibc and must declare it; musl
# binaries are statically linked and need no libc dependency.
if [[ -n "$DEB_DEPENDS" ]]; then
echo "Depends: ${DEB_DEPENDS}" >> "${PKG_DIR}/DEBIAN/control"
fi
cat >> "${PKG_DIR}/DEBIAN/control" << EOF
Maintainer: RustFS Team <support@rustfs.com>
Description: High-performance distributed object storage (${LIBC} build)
RustFS is a high-performance distributed object storage software
built using Rust. It is compatible with MinIO and S3 API.
Homepage: https://rustfs.com
EOF
# Declare /etc/default/rustfs as a conffile so dpkg preserves user
# modifications on upgrade instead of silently overwriting them.
cat > "${PKG_DIR}/DEBIAN/conffiles" << 'CONFFILES'
/etc/default/rustfs
CONFFILES
./scripts/release/package_service_scripts.sh after-install > "${PKG_DIR}/DEBIAN/postinst"
chmod 755 "${PKG_DIR}/DEBIAN/postinst"
./scripts/release/package_service_scripts.sh before-remove > "${PKG_DIR}/DEBIAN/prerm"
chmod 755 "${PKG_DIR}/DEBIAN/prerm"
./scripts/release/package_service_scripts.sh after-remove > "${PKG_DIR}/DEBIAN/postrm"
chmod 755 "${PKG_DIR}/DEBIAN/postrm"
cp LICENSE "${PKG_DIR}/usr/share/doc/rustfs/"
cp README.md "${PKG_DIR}/usr/share/doc/rustfs/"
fakeroot dpkg-deb --build "${PKG_DIR}" "$DEB_FILE"
[[ $(dpkg-deb -f "$DEB_FILE" Package) == rustfs ]]
[[ $(dpkg-deb -f "$DEB_FILE" Version) == "$DEB_VERSION" ]]
[[ $(dpkg-deb -f "$DEB_FILE" Architecture) == "$DEB_ARCH" ]]
dpkg-deb --fsys-tarfile "$DEB_FILE" | tar -tf - | grep -Fx './usr/bin/rustfs' >/dev/null
stat --printf='%n %s bytes\n' "$DEB_FILE"
echo "deb_file=$DEB_FILE" >> "$GITHUB_OUTPUT"
echo "✅ DEB built: $DEB_FILE"
- name: Build RPM package
id: rpm
shell: bash
env:
RPM_VERSION: ${{ steps.versions.outputs.rpm_version }}
RPM_RELEASE: ${{ steps.versions.outputs.rpm_release }}
RPM_ARCH: ${{ matrix.rpm_arch }}
RPM_FILE: ${{ steps.versions.outputs.rpm_file }}
RPM_DEPENDS: ${{ matrix.rpm_depends }}
LIBC: ${{ matrix.libc }}
run: |
set -euo pipefail
echo "Building RPM for ${RPM_ARCH} (${LIBC})"
sudo apt-get update && sudo apt-get install -y ruby ruby-dev build-essential rpm
sudo gem install fpm
./scripts/test_package_versions.sh --require-package-managers
# gnu binaries dynamically link glibc and must declare it; musl
# binaries are statically linked and need no libc dependency.
depends_args=()
if [[ -n "$RPM_DEPENDS" ]]; then
depends_args+=(--depends "$RPM_DEPENDS")
fi
# Create config file for fpm (DEB build creates it in its package dir structure,
# but fpm needs the file to exist before packaging)
mkdir -p ./tmp-pkg/etc/default
cat > ./tmp-pkg/etc/default/rustfs << 'ENVEOF'
# RustFS Environment Configuration
# See https://docs.rustfs.com for more information
# RUSTFS_ACCESS_KEY=<changeme>
# RUSTFS_SECRET_KEY=<changeme>
# RUSTFS_VOLUMES=""
# RUSTFS_ADDRESS=":9000"
# RUSTFS_CONSOLE_ADDRESS=":9001"
# RUSTFS_CONSOLE_ENABLE=true
# RUSTFS_OBS_LOGGER_LEVEL=error
# RUSTFS_OBS_LOG_DIRECTORY="/var/log/rustfs/"
ENVEOF
fpm -s dir -t rpm \
--name rustfs \
--version "$RPM_VERSION" \
--iteration "$RPM_RELEASE" \
--architecture "$RPM_ARCH" \
--package "$RPM_FILE" \
"${depends_args[@]}" \
--description "High-performance distributed object storage (${LIBC} build)" \
--maintainer "RustFS Team <support@rustfs.com>" \
--url "https://rustfs.com" \
--license "Apache-2.0" \
--before-install <(./scripts/release/package_service_scripts.sh rpm-before-install) \
--before-upgrade <(./scripts/release/package_service_scripts.sh rpm-before-upgrade) \
--after-install <(./scripts/release/package_service_scripts.sh after-install) \
--after-upgrade <(./scripts/release/package_service_scripts.sh after-install) \
--before-remove <(./scripts/release/package_service_scripts.sh before-remove) \
--after-remove <(./scripts/release/package_service_scripts.sh after-remove) \
--rpm-posttrans <(./scripts/release/package_service_scripts.sh rpm-posttrans) \
--config-files /etc/default/rustfs \
./bin/rustfs=/usr/bin/rustfs \
./tmp-pkg/etc/default/rustfs=/etc/default/rustfs \
deploy/build/rustfs.service=/lib/systemd/system/rustfs.service \
LICENSE=/usr/share/doc/rustfs/LICENSE \
README.md=/usr/share/doc/rustfs/README.md
if [[ ! -f "$RPM_FILE" ]]; then
echo "❌ RPM build failed"
exit 1
fi
RPM_METADATA=$(rpm -qp --qf '%{NAME}\n%{VERSION}\n%{RELEASE}\n%{ARCH}\n' "$RPM_FILE")
EXPECTED_METADATA=$(printf 'rustfs\n%s\n%s\n%s' "$RPM_VERSION" "$RPM_RELEASE" "$RPM_ARCH")
[[ "$RPM_METADATA" == "$EXPECTED_METADATA" ]]
rpm -qpl "$RPM_FILE" | grep -Fx '/usr/bin/rustfs' >/dev/null
stat --printf='%n %s bytes\n' "$RPM_FILE"
echo "rpm_file=$RPM_FILE" >> "$GITHUB_OUTPUT"
echo "✅ RPM built: $RPM_FILE"
- name: Upload packages to artifacts
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: packages-${{ matrix.arch }}-${{ matrix.libc }}
path: |
*.deb
*.rpm
retention-days: 30
- name: Upload packages to Cloudflare R2
if: needs.resolve.outputs.build_type != 'preview' && env.R2_ACCESS_KEY_ID != ''
env:
R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
R2_BUCKET: ${{ secrets.R2_BUCKET }}
AWS_EC2_METADATA_DISABLED: true
BUILD_TYPE: ${{ needs.resolve.outputs.build_type }}
DEB_FILE: ${{ steps.deb.outputs.deb_file }}
RPM_FILE: ${{ steps.rpm.outputs.rpm_file }}
shell: bash
run: |
set -euo pipefail
if [[ -z "$R2_ACCESS_KEY_ID" || -z "$R2_SECRET_ACCESS_KEY" || -z "$R2_ENDPOINT" || -z "$R2_BUCKET" ]]; then
echo "⚠️ R2 credentials missing, skipping upload"
exit 0
fi
if ! command -v aws >/dev/null 2>&1; then
sudo apt-get update && sudo apt-get install -y awscli
fi
export AWS_ACCESS_KEY_ID="$R2_ACCESS_KEY_ID"
export AWS_SECRET_ACCESS_KEY="$R2_SECRET_ACCESS_KEY"
export AWS_DEFAULT_REGION="auto"
if [[ "$BUILD_TYPE" == "development" ]]; then
R2_PREFIX="artifacts/rustfs/packages/dev"
else
R2_PREFIX="artifacts/rustfs/packages/release"
fi
R2_PATH="s3://${R2_BUCKET}/${R2_PREFIX}/"
echo "📤 Uploading to $R2_PATH"
for f in "$DEB_FILE" "$RPM_FILE"; do
if [[ -n "$f" && -f "$f" ]]; then
echo "Uploading: $f"
aws s3 cp "$f" "$R2_PATH" --endpoint-url "$R2_ENDPOINT" --only-show-errors
fi
done
echo "✅ Upload complete"
# Also upload as latest for release/prerelease
if [[ "$BUILD_TYPE" == "release" || "$BUILD_TYPE" == "prerelease" ]]; then
LATEST_PATH="s3://${R2_BUCKET}/artifacts/rustfs/packages/latest/"
for f in "$DEB_FILE" "$RPM_FILE"; do
if [[ -n "$f" && -f "$f" ]]; then
echo "Uploading latest: $(basename "$f")"
aws s3 cp "$f" "$LATEST_PATH" --endpoint-url "$R2_ENDPOINT" --only-show-errors
fi
done
echo "✅ Latest packages updated"
fi
- name: Upload packages to GitHub Release
if: needs.resolve.outputs.tag != ''
- name: Reuse completed package set
id: reuse
env:
GH_TOKEN: ${{ github.token }}
TAG: ${{ needs.resolve.outputs.tag }}
DEB_FILE: ${{ steps.deb.outputs.deb_file }}
RPM_FILE: ${{ steps.rpm.outputs.rpm_file }}
shell: bash
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
TARGET: ${{ matrix.target }}
run: python3 scripts/release/deliver.py reuse --kind packages --target "$TARGET"
- name: Install packaging tools
if: steps.reuse.outputs.reused != 'true'
run: |
set -euo pipefail
# Upload the packages, then refresh the release checksums so the new
# assets are covered, matching the binary release flow.
for f in "$DEB_FILE" "$RPM_FILE"; do
if [[ -n "$f" && -f "$f" ]]; then
echo "📤 Uploading $(basename "$f") to GitHub release ${TAG}..."
gh release upload "$TAG" "$f" --clobber
fi
done
CHECKSUM_DIR="$(mktemp -d)"
gh release download "$TAG" -p 'SHA256SUMS' -p 'SHA512SUMS' \
-D "$CHECKSUM_DIR" --clobber
for spec in "SHA256SUMS:sha256sum" "SHA512SUMS:sha512sum"; do
asset="${spec%%:*}"
checksum_cmd="${spec##*:}"
checksum_file="${CHECKSUM_DIR}/${asset}"
for f in "$DEB_FILE" "$RPM_FILE"; do
if [[ -n "$f" && -f "$f" ]]; then
base="$(basename "$f")"
# GitHub stores release asset names with '~' normalized to '.'.
# Current package file names embed the raw tag and contain no
# '~', but older releases may still ship such assets, so the
# checksum entries must reference the name as stored on the
# release.
github_base="${base//\~/.}"
# Remove any stale entry (both naming variants), then append
grep -Fv -- "$base" "$checksum_file" > "${checksum_file}.tmp" || true
grep -Fv -- "$github_base" "${checksum_file}.tmp" > "${checksum_file}.tmp2" || true
mv "${checksum_file}.tmp2" "$checksum_file"
digest=$("$checksum_cmd" -- "$f" | awk '{print $1}')
printf '%s %s\n' "$digest" "$github_base" >> "$checksum_file"
fi
done
echo "📤 Updating ${asset} for release ${TAG}..."
gh release upload "$TAG" "$checksum_file" --clobber
done
echo "✅ GitHub release assets updated"
# Summary
summary:
name: Summary
needs: [ resolve, package ]
# Every completed Build and Release run triggers this workflow; when
# resolve declined it (not a release tag), there is nothing to summarise.
if: always() && needs.resolve.result != 'skipped'
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- name: Print summary
shell: bash
sudo apt-get update
sudo apt-get install -y ruby ruby-dev build-essential rpm fakeroot cpio
sudo gem install fpm --version 1.17.0 --no-document
- name: Package accepted executable and verify contents
if: steps.reuse.outputs.reused != 'true'
env:
SUMMARY_VERSION: ${{ needs.resolve.outputs.version }}
SUMMARY_BUILD_TYPE: ${{ needs.resolve.outputs.build_type }}
SUMMARY_BUILD_RUN_ID: ${{ needs.resolve.outputs.build_run_id }}
SUMMARY_PACKAGE_STATUS: ${{ needs.package.result }}
run: |
{
echo "## 📦 Package Summary"
echo ""
echo "| Item | Value |"
echo "|------|-------|"
echo "| Version | \`${SUMMARY_VERSION}\` |"
echo "| Build Type | ${SUMMARY_BUILD_TYPE} |"
echo "| Build Run | #${SUMMARY_BUILD_RUN_ID} |"
echo "| Package Status | ${SUMMARY_PACKAGE_STATUS} |"
} >> "$GITHUB_STEP_SUMMARY"
GH_TOKEN: ${{ github.token }}
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
TARGET: ${{ matrix.target }}
run: python3 scripts/release/deliver.py package --target "$TARGET" --output delivery
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.reuse.outputs.reused != 'true'
with:
name: packages-${{ matrix.target }}
path: delivery/*
if-no-files-found: error
compression-level: 0
retention-days: 30
+91
View File
@@ -0,0 +1,91 @@
name: S3-over-RDMA feature check
# The `nixl` and `cuobject` features are off in every other build, so the code
# behind them is compiled nowhere else. This job compiles the `nixl` half on
# Linux and runs rustfs-nixl's unit tests; nixl-sys builds through its stub
# fallback, so no NIXL SDK is needed.
# The cuObject half needs the NVIDIA cuObject SDK and is not compiled here.
# Not a required check and not part of the `Test and Lint` aggregate.
on:
pull_request:
paths:
- "crates/nixl/**"
- "crates/cuobject/**"
- "rustfs/src/storage/rpc/nixl*"
- "rustfs/src/app/object/cuobject.rs"
- "rustfs/src/app/object/get.rs"
- "rustfs/src/app/object/put.rs"
- "rustfs/src/app/multipart_usecase/upload_part.rs"
- "Cargo.toml"
- "rustfs/Cargo.toml"
- ".github/workflows/rdma-feature-check.yml"
schedule:
- cron: "37 5 * * 1"
workflow_dispatch:
permissions:
contents: read
concurrency:
group: rdma-feature-check-${{ github.ref }}
cancel-in-progress: true
jobs:
nixl-check:
name: cargo check and unit tests (nixl)
runs-on: sm-standard-4
timeout-minutes: 60
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-feat-nixl
cache-save-if: 'false'
install-build-packaging-tools: 'false'
- name: Install only missing native build tools
run: |
packages=()
command -v g++ >/dev/null || packages+=(g++)
ldconfig -p | grep -q 'libclang' || packages+=(libclang-dev)
if (( ${#packages[@]} )); then
sudo apt-get update
sudo apt-get install --no-install-recommends -y "${packages[@]}"
fi
- name: Check the nixl feature build
run: cargo check --locked -p rustfs --features nixl
# These unit tests use a fake sysfs and plain host memory; none needs an
# RDMA device or the NIXL libraries.
- name: Run the rustfs-nixl unit tests
run: cargo test --locked -p rustfs-nixl --features sys
alert-on-failure:
name: Alert on scheduled failure
needs: [nixl-check]
# Only scheduled runs open or update the tracking issue; pull-request and
# manual runs stay quiet.
if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
uses: ./.github/actions/schedule-failure-issue
with:
github-token: ${{ secrets.BACKLOG_ISSUE_TOKEN }}
+444
View File
@@ -0,0 +1,444 @@
# Copyright 2026 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# s3-tests reference parity lane (rustfs/backlog#2773).
#
# Runs the ceph/s3-tests implemented list (scripts/s3-tests/implemented_tests.txt)
# against every target in .config/reference-parity-targets.toml and fails on
# any per-node-id difference between a candidate and the reference:
#
# - reference: the release pinned in .config/reference-binary.toml, downloaded
# and sha256-verified by .github/actions/reference-binary;
# - candidate targets: the debug binary built from this checkout exactly as
# ci.yml's build-rustfs-debug-binary job builds it for S3 Implemented Tests.
#
# Every target runs the same harness (scripts/s3-tests/run.sh from this
# checkout), test list and server configuration on its own hosted runner. A
# target's failures get one rerun against a fresh server; a case that passes it
# is flaky, not a difference. scripts/s3-tests/reference_parity.py judges the
# result and honours .config/reference-parity-tolerated.txt, whose entries must
# each match a real difference.
#
# The `differential` job adds the 73-operation dual-stack differential
# (crates/e2e_test/src/dual_stack_differential, rustfs/backlog#2740) against the
# same pinned reference: side A is the released binary (RUSTFS_E2E_REFERENCE_BINARY,
# started without RUSTFS_S3_STACK), side B is the candidate built above on its
# legacy stack. It reuses the candidate build artifact, never builds the candidate
# twice, and prints reference_side, version_a and version_b from the report the test
# wrote; it fails when side A was not the reference or both sides report the same
# version. Every (op, field, a, b) difference goes to the job summary. Report-only
# until the first result has been reviewed.
#
# Report-only: not part of the Test and Lint aggregate. Scheduled failures are
# owned by the scheduled-validation watchdog (.github/scheduled-validations.json).
name: Reference Parity
on:
pull_request:
paths:
- ".github/workflows/reference-parity.yml"
- ".github/actions/reference-binary/**"
- ".config/reference-binary.toml"
- ".config/reference-parity-targets.toml"
- ".config/reference-parity-tolerated.txt"
- "scripts/s3-tests/reference_parity.py"
- "scripts/junit_diff.py"
- "crates/e2e_test/src/dual_stack_differential/**"
schedule:
- cron: "37 1 * * *"
workflow_dispatch:
# Matches ci.yml: these variables are hashed into the rust-cache key, so the
# candidate build reads the same ci-dev cache as build-rustfs-debug-binary.
env:
CARGO_TERM_COLOR: always
RUST_BACKTRACE: 1
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
permissions:
contents: read
defaults:
run:
shell: bash
jobs:
plan:
name: Plan parity targets
runs-on: ubuntu-latest
timeout-minutes: 5
outputs:
targets: ${{ steps.targets.outputs.targets }}
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Validate target list
id: targets
run: |
set -euo pipefail
targets="$(python3 scripts/s3-tests/reference_parity.py targets --config .config/reference-parity-targets.toml)"
echo "targets=${targets}" >> "$GITHUB_OUTPUT"
echo "Parity targets: ${targets}"
build-candidate:
name: Build candidate binary
# Critical-path CPU and cache ownership: the same debug build as ci.yml's
# build-rustfs-debug-binary, reading the ci-dev cache written on sm-standard-4.
runs-on: sm-standard-4
timeout-minutes: 60
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-dev-ea7eec74fe14
cache-save-if: 'false'
install-build-packaging-tools: 'false'
- name: Build debug binary
run: python3 scripts/e2e_binary.py build --bins --features e2e-test-hooks
- name: Upload candidate binary
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: reference-parity-candidate-binary
overwrite: true
path: |
target/debug/rustfs
target/debug/rustfs.e2e.json
if-no-files-found: error
retention-days: 1
s3tests:
name: s3-tests (${{ matrix.target.name }})
needs: [ plan, build-candidate ]
runs-on: ubuntu-latest
timeout-minutes: 75
strategy:
fail-fast: false
matrix:
target: ${{ fromJSON(needs.plan.outputs.targets) }}
env:
TARGET_NAME: ${{ matrix.target.name }}
TARGET_KIND: ${{ matrix.target.binary }}
# Above this many first-run failures the target is systemically broken;
# a rerun would only repeat it, so the first run is judged as is.
RERUN_CAP: "50"
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Fetch pinned reference binary
id: reference
if: matrix.target.binary == 'reference'
uses: ./.github/actions/reference-binary
- name: Download candidate binary
if: matrix.target.binary == 'candidate'
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: reference-parity-candidate-binary
path: target/debug
- name: Select target binary
env:
REFERENCE_PATH: ${{ steps.reference.outputs.path }}
run: |
set -euo pipefail
case "$TARGET_KIND" in
reference) binary="$REFERENCE_PATH" ;;
candidate) chmod +x target/debug/rustfs; binary="$PWD/target/debug/rustfs" ;;
*) echo "unknown target binary kind: $TARGET_KIND" >&2; exit 1 ;;
esac
test -x "$binary"
echo "TARGET_BINARY=$binary" >> "$GITHUB_ENV"
mkdir -p "parity/$TARGET_NAME"
{
echo "target=$TARGET_NAME"
echo "binary=$TARGET_KIND"
echo "sha256=$(sha256sum "$binary" | cut -d' ' -f1)"
echo "version=$("$binary" --version | tr '\n' ' ')"
} | tee "parity/$TARGET_NAME/target.txt"
- name: Apply target environment
env:
TARGET_JSON: ${{ toJSON(matrix.target) }}
run: |
set -euo pipefail
python3 scripts/s3-tests/reference_parity.py export-env --target-json "$TARGET_JSON" \
| tee -a "$GITHUB_ENV" "parity/$TARGET_NAME/target-env.txt"
- name: Run implemented s3-tests
run: |
set -uo pipefail
report="parity/$TARGET_NAME"
harness="artifacts/s3tests-single"
# Same invocation as ci.yml's S3 Implemented Tests, with a fresh data
# root and port per run. $1 optionally replaces the test list.
run_suite() {
local list="$1" label="$2" root port
root="${RUNNER_TEMP}/rustfs-s3tests-${label}"
rm -rf "$harness" "$root"
mkdir -p "$root"
port="$(python3 -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1", 0)); print(s.getsockname()[1]); s.close()')"
env DEPLOY_MODE=binary \
RUSTFS_BINARY="$TARGET_BINARY" \
TEST_MODE=single \
MAXFAIL=0 \
XDIST=4 \
S3_PORT="$port" \
DATA_ROOT="$root" \
S3TESTS_CONF="$harness/s3tests.conf" \
${list:+IMPLEMENTED_TESTS_FILE="$list"} \
./scripts/s3-tests/run.sh
}
# A run is usable only if it produced a JUnit report; test failures
# are data for the comparison, a missing report is a lane failure.
keep() {
local prefix="$1"
for file in pytest.log compat-report.md; do
if [[ -f "$harness/$file" ]]; then cp "$harness/$file" "$report/${prefix}${file}"; fi
done
if [[ -f "$harness/rustfs-single/rustfs.log" ]]; then cp "$harness/rustfs-single/rustfs.log" "$report/${prefix}rustfs.log"; fi
test -s "$harness/junit.xml" || { echo "::error::$TARGET_NAME produced no JUnit report (${prefix:-first} run)"; return 1; }
cp "$harness/junit.xml" "$report/${prefix}junit.xml"
}
run_suite "" first
echo "first_run_exit=$?" >> "$report/target.txt"
keep "" || exit 1
cp "$harness/selected-nodeids.txt" "$report/selected-nodeids.txt" || exit 1
failures="$(python3 scripts/s3-tests/reference_parity.py failed --report "$report")" || exit 1
count="$(printf '%s' "$failures" | grep -c . || true)"
echo "first_run_failures=$count" >> "$report/target.txt"
if (( count == 0 )); then
exit 0
fi
if (( count > RERUN_CAP )); then
echo "::warning::$TARGET_NAME has $count failures (cap $RERUN_CAP); judging the first run without a rerun"
echo "rerun=skipped-over-cap" >> "$report/target.txt"
exit 0
fi
printf '%s\n' "$failures" > "$RUNNER_TEMP/rerun-tests.txt"
run_suite "$RUNNER_TEMP/rerun-tests.txt" rerun
echo "rerun_exit=$?" >> "$report/target.txt"
keep rerun- || exit 1
- name: Upload target report
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: reference-parity-report-${{ matrix.target.name }}
overwrite: true
path: parity/${{ matrix.target.name }}/
if-no-files-found: error
retention-days: 14
differential:
name: Dual-stack differential (reference vs candidate)
needs: [ build-candidate ]
# Compiles the e2e_test harness against the ci-dev cache like the other
# sm-standard-4 jobs; the candidate server itself is the downloaded artifact.
runs-on: sm-standard-4
timeout-minutes: 90
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
NO_PROXY: 127.0.0.1,localhost,::1
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-dev-ea7eec74fe14
cache-save-if: 'false'
install-build-packaging-tools: 'false'
- name: Fetch pinned reference binary
id: reference
uses: ./.github/actions/reference-binary
- name: Download candidate binary
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: reference-parity-candidate-binary
path: target/debug
- name: Run dual-stack differential
id: differential
continue-on-error: true
env:
REFERENCE_PATH: ${{ steps.reference.outputs.path }}
RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-differential-logs
run: |
set -euo pipefail
export no_proxy="$NO_PROXY"
chmod +x target/debug/rustfs
test -x "$REFERENCE_PATH"
mkdir -p "$RUSTFS_E2E_LOG_DIR"
# Side A: the released binary. Side B: the candidate on its legacy stack.
export RUSTFS_E2E_REFERENCE_BINARY="$REFERENCE_PATH"
test_name='dual_stack_differential::tests::both_stacks_answer_every_operation_identically'
python3 scripts/e2e_binary.py run --features e2e-test-hooks -- \
cargo test --locked -p e2e_test --lib "$test_name" -- --exact --nocapture \
2>&1 | tee "$RUNNER_TEMP/differential-test.log"
- name: Summarize report and verify the reference side
if: always()
run: |
set -euo pipefail
python3 - <<'PY'
import glob, json, os, sys
summary = os.environ["GITHUB_STEP_SUMMARY"]
reports = sorted(glob.glob("target/dual-stack/self-comparison-*.json"))
lines = ["## Dual-stack differential: reference vs candidate", ""]
failures = []
if len(reports) != 1:
failures.append(f"expected exactly one self-comparison report, found {len(reports)}: {reports}")
else:
report = json.load(open(reports[0]))
side = report.get("reference_side")
version_a = report.get("version_a")
version_b = report.get("version_b")
lines += [
f"- report: `{reports[0]}`",
f"- reference_side: `{side}`",
f"- version_a: `{(version_a or '').splitlines()[0] if version_a else version_a}`",
f"- version_b: `{(version_b or '').splitlines()[0] if version_b else version_b}`",
f"- stack_a: `{report.get('stack_a')}`, stack_b: `{report.get('stack_b')}`",
f"- compared_ops: {report.get('compared_ops')}, compared_cases: {report.get('compared_cases')}",
"",
]
print(f"reference_side={side}\nversion_a={version_a}\nversion_b={version_b}")
lines += ["```", "version_a:", version_a or "", "", "version_b:", version_b or "", "```", ""]
if side != "a":
failures.append(f"reference_side is {side!r}, expected 'a'")
if not version_a or not version_b:
failures.append("version_a or version_b is missing")
elif version_a == version_b:
failures.append("version_a equals version_b: the reference was replaced by the candidate")
diffs = report.get("differences") or []
lines.append(f"### Differences: {len(diffs)}")
by_op = {}
for d in diffs:
by_op.setdefault(d["op"], []).append(d)
for op in sorted(by_op):
lines += ["", f"#### {op}", "", "| case | field | a (reference) | b (candidate) |", "| --- | --- | --- | --- |"]
for d in by_op[op]:
cell = lambda v: "`" + str(v).replace("|", "\\|").replace("`", "'") + "`"
lines.append(f"| {d['case']} | {d['field']} | {cell(d['a'])} | {cell(d['b'])} |")
if failures:
lines += ["", "### Observation check failed", ""] + [f"- {f}" for f in failures]
open(summary, "a").write("\n".join(lines) + "\n")
print("\n".join(lines))
for f in failures:
print(f"::error::{f}")
sys.exit(1 if failures else 0)
PY
- name: Upload differential reports
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: reference-differential-report
overwrite: true
path: |
target/dual-stack/*.json
${{ runner.temp }}/differential-test.log
if-no-files-found: warn
retention-days: 14
- name: Fail when the differential found differences
if: steps.differential.outcome == 'failure'
run: |
echo "::error::the dual-stack differential failed; see the job summary for every (op, field, a, b)"
exit 1
compare:
name: Compare targets
needs: [ plan, s3tests ]
if: always() && needs.plan.result == 'success'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download target reports
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
pattern: reference-parity-report-*
path: parity-download
- name: Compare per node id
env:
TARGETS: ${{ needs.plan.outputs.targets }}
S3TESTS_RESULT: ${{ needs.s3tests.result }}
run: |
set -uo pipefail
mkdir -p parity
for dir in parity-download/reference-parity-report-*; do
if [[ -d "$dir" ]]; then mv "$dir" "parity/${dir#parity-download/reference-parity-report-}"; fi
done
python3 scripts/s3-tests/reference_parity.py compare \
--targets-json "$TARGETS" \
--reports parity \
--tolerated .config/reference-parity-tolerated.txt \
--markdown "$GITHUB_STEP_SUMMARY" \
--json parity/diff.json > parity/diff.md
status=$?
cat parity/diff.md
if [[ "$S3TESTS_RESULT" != "success" ]]; then
echo "::error::s3-tests target jobs finished with '$S3TESTS_RESULT'"
status=1
fi
exit "$status"
- name: Upload parity diff
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: reference-parity-diff
overwrite: true
path: |
parity/diff.md
parity/diff.json
if-no-files-found: warn
retention-days: 14
+133
View File
@@ -0,0 +1,133 @@
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Release candidate (Linux)
on:
workflow_dispatch:
inputs:
version:
description: Final product version (no preview suffix)
type: string
required: true
source_sha:
description: Full reviewed source commit
type: string
required: true
permissions:
contents: read
actions: read
concurrency:
group: release-candidate-${{ inputs.version }}
cancel-in-progress: false
jobs:
freeze:
if: github.repository == 'rustfs/rustfs' && github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
timeout-minutes: 15
outputs:
manifest: ${{ steps.freeze.outputs.manifest }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Freeze or recover original inputs
id: freeze
env:
GH_TOKEN: ${{ github.token }}
VERSION: ${{ inputs.version }}
SOURCE_SHA: ${{ inputs.source_sha }}
run: python3 scripts/release/pipeline.py prepare --version "$VERSION" --source "$SOURCE_SHA" --directory frozen
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.freeze.outputs.reused != 'true'
with:
name: release-inputs
path: frozen/inputs.json
if-no-files-found: error
retention-days: 30
linux:
needs: freeze
uses: ./.github/workflows/build.yml
with:
candidate_manifest: ${{ needs.freeze.outputs.manifest }}
platforms: linux
upgrade:
needs: [freeze, linux]
runs-on: ubuntu-latest
timeout-minutes: 150
env:
RUSTUP_TOOLCHAIN: ${{ fromJson(needs.freeze.outputs.manifest).toolchain }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: ${{ fromJson(needs.freeze.outputs.manifest).workflow_sha }}
persist-credentials: false
- uses: ./.github/actions/setup
with:
rust-version: ${{ fromJson(needs.freeze.outputs.manifest).toolchain }}
cache-shared-key: release-upgrade-harness-hosted
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
install-build-packaging-tools: "false"
install-test-tools: "false"
- name: Verify archived server against previous releases
env:
GH_TOKEN: ${{ github.token }}
CANDIDATE_MANIFEST: ${{ needs.freeze.outputs.manifest }}
run: |
python3 - <<'PYVERIFY'
import sys
from pathlib import Path
sys.path.insert(0, 'scripts/release')
import candidate as c, pipeline, verify
inputs = pipeline.manifest_env()
run = pipeline.current_run('release-candidate.yml')
if any(a['name'] == 'release-upgrade' for a in c.artifact_list(run['id'])):
c.download_artifact(run, 'release-upgrade', 'receipts')
pipeline.verify_evidence(inputs, Path('receipts'), {run['id']: run})
c.emit_output('reused', 'true')
else:
target = 'linux-x86_64-gnu'
c.download_artifact(run, f"candidate-{inputs['candidate_id']}-{target}", 'binary')
verify.upgrade(inputs, Path('binary') / f"rustfs-{target}-v{inputs['version']}.zip", Path('receipts'))
PYVERIFY
id: verify
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.verify.outputs.reused != 'true'
with:
name: release-upgrade
path: receipts/*
if-no-files-found: error
retention-days: 30
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: failure()
with:
name: release-upgrade-failure-${{ github.run_attempt }}
path: receipts/*
retention-days: 7
seal-linux:
needs: [freeze, linux, upgrade]
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Collect trusted receipts and seal Linux bytes
id: seal
env:
GH_TOKEN: ${{ github.token }}
run: |
python3 scripts/release/pipeline.py linux-seal --directory sealed
{
echo '## Linux acceptance'
echo 'Use this run ID and manifest SHA-256 when starting Release candidate (complete).'
sha256sum sealed/linux-manifest.json
echo 'Record cluster, performance, integrity and upgrade acceptance in backlog before continuing.'
} >> "$GITHUB_STEP_SUMMARY"
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.seal.outputs.reused != 'true'
with:
name: release-linux
path: sealed/*
if-no-files-found: error
compression-level: 0
retention-days: 30
+97
View File
@@ -0,0 +1,97 @@
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Release candidate (complete)
on:
workflow_dispatch:
inputs:
candidate_run_id:
description: Successful Linux candidate run ID
type: string
required: true
linux_manifest_sha256:
description: Exact accepted Linux manifest SHA-256
type: string
required: true
acceptance_report:
description: Backlog report accepting cluster, performance, integrity and upgrade results
type: string
required: true
permissions:
contents: read
actions: read
concurrency:
group: release-complete-${{ inputs.candidate_run_id }}
cancel-in-progress: false
jobs:
acceptance:
if: github.repository == 'rustfs/rustfs' && github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
timeout-minutes: 30
environment: release-linux-acceptance
outputs:
manifest: ${{ steps.accept.outputs.manifest }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Bind acceptance to the exact Linux manifest
id: accept
env:
GH_TOKEN: ${{ github.token }}
CANDIDATE_RUN: ${{ inputs.candidate_run_id }}
MANIFEST_SHA: ${{ inputs.linux_manifest_sha256 }}
REPORT: ${{ inputs.acceptance_report }}
run: python3 scripts/release/pipeline.py complete-prepare --run-id "$CANDIDATE_RUN" --digest "$MANIFEST_SHA" --report "$REPORT" --directory linux
desktop:
needs: acceptance
uses: ./.github/workflows/build.yml
with:
candidate_manifest: ${{ needs.acceptance.outputs.manifest }}
platforms: desktop
packages:
needs: acceptance
uses: ./.github/workflows/package.yml
with:
candidate_manifest: ${{ needs.acceptance.outputs.manifest }}
images:
needs: acceptance
uses: ./.github/workflows/docker.yml
with:
candidate_manifest: ${{ needs.acceptance.outputs.manifest }}
chart:
needs: acceptance
uses: ./.github/workflows/helm-package.yml
with:
candidate_manifest: ${{ needs.acceptance.outputs.manifest }}
seal:
needs: [acceptance, desktop, packages, images, chart]
runs-on: ubuntu-latest
timeout-minutes: 45
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Verify every deliverable and seal the candidate
id: seal
env:
GH_TOKEN: ${{ github.token }}
CANDIDATE_RUN: ${{ inputs.candidate_run_id }}
MANIFEST_SHA: ${{ inputs.linux_manifest_sha256 }}
REPORT: ${{ inputs.acceptance_report }}
run: |
python3 scripts/release/pipeline.py complete-seal --run-id "$CANDIDATE_RUN" --digest "$MANIFEST_SHA" --report "$REPORT" --directory sealed
{
echo '## Final publication approval'
echo 'Review this manifest, then explicitly dispatch Publish accepted release with its SHA-256.'
sha256sum sealed/release-manifest.json
} >> "$GITHUB_STEP_SUMMARY"
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.seal.outputs.reused != 'true'
with:
name: release-seal
path: sealed/*
if-no-files-found: error
compression-level: 0
retention-days: 30
+193
View File
@@ -0,0 +1,193 @@
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Publish accepted release
on:
workflow_dispatch:
inputs:
complete_run_id:
description: Successful completed candidate run ID
type: string
required: true
manifest_sha256:
description: Exact approved final manifest SHA-256 (authorizes publication)
type: string
required: true
permissions:
contents: read
actions: read
concurrency:
group: release-publish-${{ inputs.complete_run_id }}
cancel-in-progress: false
jobs:
verify:
if: github.repository == 'rustfs/rustfs' && github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
timeout-minutes: 45
outputs:
version: ${{ steps.verify.outputs.version }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Recheck approved bytes and trusted certification
id: verify
env:
GH_TOKEN: ${{ github.token }}
COMPLETE_RUN: ${{ inputs.complete_run_id }}
MANIFEST_SHA: ${{ inputs.manifest_sha256 }}
run: python3 scripts/release/publish.py verify --run-id "$COMPLETE_RUN" --digest "$MANIFEST_SHA" --directory sealed
version:
needs: verify
runs-on: ubuntu-latest
timeout-minutes: 90
concurrency:
group: release-version-${{ needs.verify.outputs.version }}
cancel-in-progress: false
environment: release-publication
permissions:
contents: write
actions: read
packages: write
security-events: write
env:
GH_TOKEN: ${{ github.token }}
COMPLETE_RUN: ${{ inputs.complete_run_id }}
MANIFEST_SHA: ${{ inputs.manifest_sha256 }}
AWS_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
AWS_DEFAULT_REGION: auto
AWS_REQUEST_CHECKSUM_CALCULATION: when_required
AWS_RESPONSE_CHECKSUM_VALIDATION: when_required
AWS_EC2_METADATA_DISABLED: "true"
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
R2_BUCKET: ${{ secrets.R2_BUCKET }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Install promotion tools
run: |
sudo apt-get update
sudo apt-get install -y skopeo
aws --version
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
username: rustfs
password: ${{ secrets.DOCKERHUB_TOKEN }}
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
password: ${{ secrets.GHCR_PASSWORD }}
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
registry: quay.io
username: ${{ secrets.QUAY_USERNAME }}
password: ${{ secrets.QUAY_PASSWORD }}
- name: Share registry credentials with the OCI copier
run: |
install -m 600 "$HOME/.docker/config.json" "$RUNNER_TEMP/registry-auth.json"
echo "REGISTRY_AUTH_FILE=$RUNNER_TEMP/registry-auth.json" >> "$GITHUB_ENV"
- name: Promote verified version
run: python3 scripts/release/publish.py version --run-id "$COMPLETE_RUN" --digest "$MANIFEST_SHA" --directory sealed
- name: Publish accepted musl amd64 scan
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4
with:
sarif_file: sealed/scan-musl-amd64.sarif
category: container-image-musl-amd64
- name: Publish accepted musl arm64 scan
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4
with:
sarif_file: sealed/scan-musl-arm64.sarif
category: container-image-musl-arm64
- name: Publish accepted GNU amd64 scan
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4
with:
sarif_file: sealed/scan-gnu-amd64.sarif
category: container-image-gnu-amd64
- name: Publish accepted GNU arm64 scan
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4
with:
sarif_file: sealed/scan-gnu-arm64.sarif
category: container-image-gnu-arm64
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: release-version-receipt-${{ github.run_attempt }}
path: publication.json
if-no-files-found: error
retention-days: 90
channels:
needs: [verify, version]
runs-on: ubuntu-latest
timeout-minutes: 90
concurrency:
group: rustfs-release-channels
cancel-in-progress: false
permissions:
contents: write
actions: read
packages: write
env:
GH_TOKEN: ${{ github.token }}
COMPLETE_RUN: ${{ inputs.complete_run_id }}
MANIFEST_SHA: ${{ inputs.manifest_sha256 }}
AWS_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
AWS_DEFAULT_REGION: auto
AWS_REQUEST_CHECKSUM_CALCULATION: when_required
AWS_RESPONSE_CHECKSUM_VALIDATION: when_required
AWS_EC2_METADATA_DISABLED: "true"
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
R2_BUCKET: ${{ secrets.R2_BUCKET }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Install promotion tools
run: |
sudo apt-get update
sudo apt-get install -y skopeo
aws --version
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
username: rustfs
password: ${{ secrets.DOCKERHUB_TOKEN }}
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
password: ${{ secrets.GHCR_PASSWORD }}
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
registry: quay.io
username: ${{ secrets.QUAY_USERNAME }}
password: ${{ secrets.QUAY_PASSWORD }}
- name: Share registry credentials with the OCI copier
run: |
install -m 600 "$HOME/.docker/config.json" "$RUNNER_TEMP/registry-auth.json"
echo "REGISTRY_AUTH_FILE=$RUNNER_TEMP/registry-auth.json" >> "$GITHUB_ENV"
- uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
- name: Install pinned OSS client
run: |
curl --fail --location --retry 3 --output "$RUNNER_TEMP/ossutil.zip" https://gosspublic.alicdn.com/ossutil/v2/2.1.1/ossutil-2.1.1-linux-amd64.zip
printf '%s %s\n' 60f3a808cbbdfd4b5269b8f967c6a66f564c9dacff52d93a5d21a588976ab5a2 "$RUNNER_TEMP/ossutil.zip" | sha256sum --check -
unzip -q "$RUNNER_TEMP/ossutil.zip" -d "$RUNNER_TEMP/ossutil-extract"
install -m 755 "$RUNNER_TEMP/ossutil-extract/ossutil-2.1.1-linux-amd64/ossutil" "$RUNNER_TEMP/ossutil"
echo "OSSUTIL_BIN=$RUNNER_TEMP/ossutil" >> "$GITHUB_ENV"
- name: Promote verified channels
env:
HELM_TOKEN: ${{ secrets.RUSTFS_HELM_PACKAGE }}
OSS_ACCESS_KEY_ID: ${{ secrets.ALICLOUDOSS_KEY_ID }}
OSS_ACCESS_KEY_SECRET: ${{ secrets.ALICLOUDOSS_KEY_SECRET }}
OSS_REGION: cn-beijing
OSS_ENDPOINT: https://oss-cn-beijing.aliyuncs.com
run: python3 scripts/release/publish.py channels --run-id "$COMPLETE_RUN" --digest "$MANIFEST_SHA" --directory sealed
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: release-channels-receipt-${{ github.run_attempt }}
path: publication.json
if-no-files-found: error
retention-days: 90
@@ -26,7 +26,13 @@ permissions:
jobs:
alert-on-incomplete-run:
name: Alert on incomplete scheduled run
if: github.repository == 'rustfs/rustfs'
# The registry still owns responsibility and cadence selection. Successful
# and PR runs can never produce an incomplete scheduled-run alert.
if: >-
github.repository == 'rustfs/rustfs' &&
github.event.workflow_run.conclusion != 'success' &&
(github.event.workflow_run.event == 'schedule' ||
github.event.workflow_run.event == 'workflow_run')
# Stay on GitHub-hosted runners: this monitor must keep running when the
# self-hosted sm-standard-* pool is down, which is exactly when the
# scheduled validations it watches stall.
+1 -1
View File
@@ -155,7 +155,7 @@ default build (lifecycle:
- ⚠️ PARTIALLY VIOLATED: serving-side `s3s` references remain in ecstore
(bucket metadata/replication/lifecycle DTOs and error mapping). The
count is ratcheted shrink-only by `scripts/check_s3s_footprint.sh`
(`S3S_ECSTORE_FILES_BASELINE`; the `object_lock` module was converted to
(the `ecstore_files` line of `.config/s3s-footprint-baseline.txt`; the `object_lock` module was converted to
storage-level types as the first ratchet step). Target state: the
baseline reaches zero and ecstore's `Cargo.toml` drops `s3s`.
+18
View File
@@ -7,17 +7,27 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Notifications and audit
- Notify and audit targets that keep the default `queue_dir` now work on native Windows: the defaults `/opt/rustfs/events` and `/opt/rustfs/audit` resolve per node, to `%ProgramData%\RustFS\events` and `%ProgramData%\RustFS\audit` on Windows, instead of failing validation as non-absolute paths. The new `RUSTFS_TARGET_STORE_DIR` (Helm `config.rustfs.target_store_dir`) relocates both default queues on any platform. Linux and macOS behavior is unchanged when it is unset, persisted configuration is unchanged, and pending entries are not migrated when the root changes. See `docs/operations/target-queue-storage.md`.
### Deployment
- Before upgrading, verify that the configured root access key is at least 3 bytes and the root secret key is at least 8 bytes. RustFS now rejects shorter values at startup and identifies the setting to update. Version 1.0.5 accepted these settings at startup and returned HTTP 500 when root requested STS credentials. Coordinate root credential changes across all nodes and clients. See [root credential rotation considerations](docs/operations/two-factor-auth.md#root-credentials-cannot-be-changed-at-runtime).
- Existing volumes created by older root-running container images must be writable by the non-root runtime identity before upgrading. The default container UID/GID is `10001:10001`; stop the affected instances, verify host mount paths, and for ordinary bind mounts run `sudo chown -R 10001:10001 /actual/data /actual/logs` on each RustFS data/log mount. Account for ACLs, NFS, Kubernetes, and rootless UID mappings. Image-build ownership changes do not migrate mounted data. See the Docker Quick Start in [README.md](README.md).
- Local disk initialization now checks file creation, writing, and deletion in the data root and fixed internal metadata directories. Unwritable disks are rejected using existing startup/quorum rules, with filesystem errors and Unix runtime/owner IDs logged for permission failures, instead of admitting those disks and discovering the mismatch on a later write. Existing object trees are not scanned and volume ownership is never changed automatically.
### Security
- **Key-scoped KMS policy statements now hold for every spelling of an AWS key**: a statement such as `Deny kms:Decrypt` on `arn:aws:kms:::key/<key-id>` was matched against the key identifier exactly as the request spelled it, so naming the same key by its AWS ARN (`arn:aws:kms:<region>:<account>:key/<key-id>`) was evaluated as a different key and the `Deny` did not apply. With `RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY=true` and the AWS KMS backend, an identity allowed on all keys but denied one could still write under that key, and read objects stored under its ARN spelling; the per-key admin endpoints were affected the same way. A key ARN is now evaluated as its key id, on the SSE-KMS data path and the admin endpoints alike. A request that names a key by alias, which only the backend can resolve, now fails closed against key-scoped statements: every key-scoped `Deny` applies and only an `Allow` on every key covers it. The KMS deletion impact report recognizes the default key and bucket default-encryption references across the id and ARN spellings as well. Deployments without key-scoped KMS statements, and the other KMS backends, whose key ids have a single spelling, are unaffected.
- **Header-signed SigV4 requests honour only signed headers** (GHSA-xm99-m3gq-83g8): a request authenticated with a SigV4 `Authorization` header that carries an `x-amz-*` request header not listed in its `SignedHeaders` is now rejected with `403 AccessDenied` ("There were headers present in the request which were not signed"), matching AWS S3 and the presigned rule from GHSA-g8w9-qw9q-fghr. Previously anyone holding one header-signed `PutObject` request could add an unsigned `x-amz-copy-source` and turn it into a `CopyObject` that ran with the signer's permissions, copying any object the signer could read into the target. An `Authorization` header whose algorithm token is not `AWS4-HMAC-SHA256` is now rejected instead of being verified as SigV4. The request-envelope headers `x-amz-content-sha256`, `x-amz-decoded-content-length`, `x-amz-trailer` and `x-amz-checksum-algorithm` (the same set the upstream `s3s` fix exempts) and `x-amz-cf-id` (CloudFront) remain tolerated unsigned; AWS SDKs and RustFS's own signers already sign every other `x-amz-*` header. SigV2, JWT and anonymous requests are unchanged.
### Replication
- Site replication now seals an SSE-KMS replica under the peer's key of the same name as the source object's key. The replica write used to carry only the `aws:kms` intent, so an object written under a key named in the request was sealed under the peer's bucket default or site default instead, and with `RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY=true` a caller scoped to the named key could read the object on the site that took the write but got `AccessDenied` for its replica. This changes the default outbound request to site-replication peers: every key name in use, including each site's default key name, must exist on every site, or the affected SSE-KMS objects stay `FAILED` until the key is created. Existing replicas are not rewritten. Bucket replication through a remote target still sends the intent only.
- Site replication now replicates SSE-KMS objects when the receiving site runs with `RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY=true`. The receiving site authorizes each replica write against the KMS key it resolves to, and the managed `site-replicator-0` account held no `kms` grant, so every SSE-KMS object stayed `FAILED` with `AccessDenied` while SSE-S3 and SSE-C objects replicated. The account's embedded policy now grants `kms:GenerateDataKey` on every key; existing accounts are refreshed by the reconcile tick and the failed objects catch up afterwards. It is deliberately not granted `kms:Decrypt`, so reads proxied to a peer for an SSE-KMS object are still refused under enforcement. Bucket replication through a remote target needs the same grant on the identity whose credentials the target uses. Sites that do not enable enforcement are unaffected.
- Object Lock replication PUTs now carry a required integrity header, fixing target rejection introduced by the plain-payload default ([#7097](https://github.com/rustfs/rustfs/pull/7097)). This changes the default outbound request for locked objects but adds no persisted format.
- Multipart source objects stay on the multipart transport even when their checksum record is a whole-object checksum, so objects above the single-PUT limit remain replicable ([#7047](https://github.com/rustfs/rustfs/pull/7047)).
- Targets that mint their own version IDs now use a per-target version ledger for tag, retention, legal-hold, and permanent-delete mutations; ambiguous pre-ledger matches fail with backoff instead of guessing ([#7368](https://github.com/rustfs/rustfs/pull/7368)). This adds dual-prefixed internal metadata keys that older readers ignore.
@@ -33,6 +43,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- **Presigned URLs honour only signed headers** (GHSA-g8w9-qw9q-fghr): a SigV4 presigned request that carries an `x-amz-*` request header not listed in `X-Amz-SignedHeaders` is now rejected with `403 AccessDenied` ("There were headers present in the request which were not signed"), matching AWS S3. Previously the holder of a presigned `PutObject` URL could add unsigned `x-amz-tagging`, `x-amz-storage-class`, `x-amz-website-redirect-location`, ACL, metadata, Object Lock or SSE headers and have them applied. Presigners that intend a property must set it before signing so the SDK lists the header in `SignedHeaders`; `x-amz-cf-id` (CloudFront) remains tolerated unsigned. Header-signed SigV4 and SigV2 requests are unchanged.
### Fixed
- **Vault KMS backends no longer fail when the server answers reads from a replica**: a Vault or OpenBao node that serves reads locally (an OpenBao standby, a Vault Enterprise performance standby) can return the state from before a write for a short while after acknowledging it. Against such a node the KV2 backend failed every key creation with `404 Key not found` while leaving the key stored without its description and tags, failed operations issued right after a create or update with `Key not found` or `Concurrent modification`, and the Transit backend failed a disable followed quickly by an enable. Key creation on the KV2 backend is now a single create-only write that carries the description and tags. Both backends now re-read a key record when the answer is older than a write the same RustFS node made in the previous five seconds, pause before re-reading after a lost check-and-set write and allow five attempts instead of three, and retry a Transit key delete that is refused because the deletion switch is not visible yet. A different RustFS node reading the same replica can still observe the earlier state until the replica catches up; see [OpenBao as the Vault KMS backend](docs/operations/kms-openbao-compatibility.md).
- **KMS keys could not be described by ARN through the path route**: `GET /rustfs/admin/v3/kms/keys/{key_id}` read the path segment verbatim, so a key id that has to be percent-encoded to fit in one segment — an AWS key ARN, with its `:` and `/` — was looked up, authorized and audited under its encoded spelling and answered `404`. The segment is now percent-decoded first; a segment that is not valid UTF-8 after decoding is refused with `400 InvalidArgument`. The `describe-key?keyId=` form was unaffected.
- **SSE-KMS objects written through the AWS KMS backend could not be read**: `GetObject`, `CopyObject` and every other read of an `aws:kms` object failed with `400 InvalidRequest` ("SSE-S3 requires RUSTFS_SSE_S3_MASTER_KEY to be set ...") on a deployment using `RUSTFS_KMS_BACKEND=aws`, although the write and `HeadObject` succeeded. The AWS backend stored the `GenerateDataKey` blob as it came from AWS, the read path took those unrecognized bytes for a locally wrapped key, and AWS was never asked to decrypt. The backend now stores the blob inside the same data-key envelope the other backends use, and a blob stored without one is routed to AWS when the AWS backend is active. Objects written by 1.0.4 and 1.0.5 are intact and become readable after the upgrade, with no rewrite. Setting `RUSTFS_SSE_S3_MASTER_KEY` was never a workaround. An object written by this release is not readable by an older one; the same was already true of every object the older release wrote itself.
- **Immediate KMS key deletion no longer reports success without destroying the key** (behavior change): with `RUSTFS_KMS_ALLOW_IMMEDIATE_DELETION=true`, a `DELETE /rustfs/admin/v3/kms/keys/delete` request with `force_immediate` against an `Enabled` or `Disabled` key on the Vault KV2 or Vault Transit backend returned success but only moved the key to `PendingDeletion`; the key material stayed in Vault and objects encrypted under it remained readable, while the Local backend destroyed the key in that one request. Key destruction is now the same two explicit steps on every backend: schedule the deletion, then either let the waiting window elapse or send `force_immediate` to destroy the pending key early. `force_immediate` against a key that is not `PendingDeletion` is refused with `400` and changes nothing, so a success response always means the key is gone and no single request can destroy a key in use. Automation that destroyed Local keys with one `force_immediate` request must schedule the deletion first. On Vault Transit, a read under a key that no longer exists in Vault now fails with `KMS.NotFoundException` instead of `503 ServiceUnavailable`. No persisted format changes. See `docs/operations/kms-key-destruction.md`.
- **`mc admin info` and `mc admin replicate resync` work again**: since the cluster snapshot view (#3803), `GET /v3/info` and `GET /v3/storageinfo` returned the madmin `InfoMessage` / `StorageInfo` nested under an `info` key next to `admin_discovery` (and, for `/info`, `bitrotSelftest`). madmin decodes those bodies directly, so `mc admin info` printed empty values and `mc admin replicate resync start/status` rejected every alias with "alias provided is not part of cluster replication" because the local `deploymentID` read as empty. Requests on the MinIO-compatible `/minio/admin/v3` prefix now receive the bare madmin body again; `/rustfs/admin/v3`, which rc and the console use, keeps the envelope unchanged.
- **Replication status writebacks no longer invalidate scanner cycles**: every replication status writeback (`persist_replication_state_if_current` in `Update` mode, including the heals the scanner queues during its own walk) marked the bucket dirty on the namespace path, advanced the dirty-usage generation and woke the scanner, so a site-replication resync or any steadily replicated bucket ended every cycle `superseded`, the authoritative usage baseline was never certified, and the console reported usage freshness as stale for as long as replication ran (#8375). Those writebacks are now status class: they do not touch the dirty-usage generation, the activity snapshot, the dirty-usage wake or the durable dirty journal, and only the process-local producer coverage is recorded. They need no rescan queue, because a deployment with an active replication rule already walks every bucket on every cycle and that walk refreshes `replication_stats`. Tier-expiration cleanup, which removes data, stays on the namespace path. No persisted format, protocol, admin API or configuration changes. Design: `docs/architecture/scanner-per-bucket-convergence.md`.
- **A bucket written during a scanner walk no longer supersedes the whole cycle**: the post-walk cluster activity proof required the entire activity snapshot and the dirty-usage generation to be unchanged, so a single-node deployment with one bucket written every few seconds never reached a `Complete` cycle, `usage_snapshot_converged` never became true and the authoritative usage baseline was never certified (#8494). Dirty usage is now certified per bucket: a structurally complete walk that the guardrail did not veto is `Complete`, buckets whose dirty generation still matches the start snapshot are cleared, and buckets written during the walk stay dirty and are published as the pending set, so `usage_snapshot_converged` means "no bucket pending" and the next cycle is scoped to the pending buckets. Such a cycle writes the authoritative usage object; the observed object is now selected by an explicit observational mark that vetoed cycles carry. The guardrail itself vetoes only on the fields that change which namespace the walk described (peer set, process instance, protocol version, topology, maintenance generation, data movement, movement generation, publication blocked) and reports the field and host in the scanner log and in the new `usage_freshness` fields `guardrail_veto_total`, `last_guardrail_veto_unix_secs` and `last_guardrail_veto_reason`. In a distributed deployment a remote peer whose dirty usage moved during the walk still supersedes the cycle until scoped acknowledgements can name its buckets. `.usage.v2.json` gains the additive, default-when-absent fields `usage_snapshot_pending_buckets`, `usage_snapshot_pending_bucket_count` and `usage_snapshot_observational`; older readers ignore them and treat the snapshot as an authoritative but unconverged publication. Quota is unchanged: an uncertified bucket keeps its last usage value as the floor and live writes do not advance it. No protocol or configuration changes. Design: `docs/architecture/scanner-per-bucket-convergence.md`.
@@ -60,6 +77,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- **Per-pool erasure parity**: Erasure parity (STANDARD and reduced-redundancy) is now resolved independently for every pool instead of reusing the first pool's value. A heterogeneous topology — for example a 4-drive pool plus a 2-drive pool created during expansion — previously inherited the first pool's parity and could resolve to zero data shards in the smaller pool, panicking Reed-Solomon construction on write. Automatic parity now resolves per pool (for example `2+2` in the 4-drive pool and `1+1` in the 2-drive pool). Fixes #4801.
### Added
- **Multiple bucket regions**: `RUSTFS_REGION` / `--region` accepts a comma-separated list. The first entry stays the server's own region (request signing and `GetBucketLocation` are unchanged); every entry is accepted as a `CreateBucket` `LocationConstraint`, so one deployment can stand in for buckets that live in several AWS regions (local development and test rigs) now that location constraints are validated (#8739). Regions outside the list are still rejected with `InvalidLocationConstraint`. A single region behaves exactly as before; a comma was never a valid region, so no existing value changes meaning.
- **On-Demand Migration**: Lazy, pull-style migration of an existing S3-compatible bucket into RustFS. A local bucket is attached to an external source bucket; a GET for a key that does not exist locally fetches it from the source, streams it to the client, and stores it locally in the same pass, so every later read is served locally. The module is on by default; set `RUSTFS_ON_DEMAND_MIGRATION_ENABLED=false` on every node to turn it off. A bucket with no source configured behaves exactly as before — the runtime never intervenes on its reads and makes no outbound call. Operator guide at `docs/operations/on-demand-migration.md`.
- Per-bucket configuration persisted as `on-demand-migration.json` in the bucket metadata: source provider (`s3`, `aws`, `minio`, `rustfs`, `r2`, `gcs`), endpoint, region, addressing style, credentials and TLS material, an optional key-prefix filter and source-prefix rewrite, and a policy block covering the inline size threshold, multipart part size, concurrency, queue capacity, timeouts, bandwidth limit and negative-cache TTL
- Admin routes under `/rustfs/admin/v3/on-demand-migration/{bucket}`: `PUT` (with `?dry-run=true` to validate and probe the source without saving), `GET`, `DELETE`, `GET .../status`, plus `POST .../backfill?op=start|cancel` and `GET .../backfill` for the background full-backfill job with its resumable checkpoint. Authorized by the new `admin:GetBucketOnDemandMigration` and `admin:SetBucketOnDemandMigration` actions; every response redacts `secret_key` and `session_token`
Generated
+867 -378
View File
File diff suppressed because it is too large Load Diff
+110 -85
View File
@@ -23,9 +23,12 @@ members = [
"crates/connect-symbol-catalog", # Release-only CPU profile symbol catalog generator
"crates/credentials", # Credential management system
"crates/crypto", # Cryptography and security features
"crates/cuobject", # NVIDIA cuObject S3-over-RDMA staging and transfers (optional)
"crates/nixl", # NIXL/UCX S3-over-RDMA server side (optional)
"crates/ecstore", # Erasure coding storage implementation
"crates/e2e_test", # End-to-end test suite
"crates/filemeta", # File metadata management
"crates/gateway-adapter", # RustFS implementations of the gateway extension traits
"crates/heal", # Erasure set and object healing
"crates/heal-contracts", # Heal request/response channel contracts
"crates/iam", # Identity and Access Management
@@ -48,7 +51,7 @@ members = [
"crates/replication", # Replication contracts and wire formats
"crates/concurrency", # Concurrency management for RustFS - timeout, locking, backpressure, and I/O scheduling
"crates/s3-client", # S3 client for engine-side consumption of remote S3 endpoints (tiering, transition targets)
"crates/s3-types", # S3 event type definitions
"crates/s3-types", # S3 event and error type definitions
"crates/s3-ops", # S3 operation definitions and mapping
"crates/s3select-api", # S3 Select API interface
"crates/s3select-query", # S3 Select query engine
@@ -75,9 +78,8 @@ edition = "2024"
license = "Apache-2.0"
repository = "https://github.com/rustfs/rustfs"
rust-version = "1.99.0"
version = "1.0.6"
version = "1.0.5"
homepage = "https://rustfs.com"
description = "RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. "
keywords = ["RustFS", "Minio", "object-storage", "filesystem", "s3"]
categories = ["web-programming", "development-tools", "filesystem", "network-programming"]
authors = ["RustFS Team"]
@@ -92,56 +94,71 @@ redundant_clone = "warn"
[workspace.dependencies]
# RustFS Internal Crates
rustfs = { path = "./rustfs", version = "1.0.6" }
rustfs-heal = { path = "crates/heal", version = "1.0.6" }
rustfs-heal-contracts = { path = "crates/heal-contracts", version = "1.0.6" }
rustfs-scanner-metrics = { path = "crates/scanner-metrics", version = "1.0.6" }
rustfs-audit = { path = "crates/audit", version = "1.0.6" }
rustfs-checksums = { path = "crates/checksums", version = "1.0.6" }
rustfs-common = { path = "crates/common", version = "1.0.6" }
rustfs-data-usage = { path = "crates/data-usage", version = "1.0.6" }
rustfs-config = { path = "./crates/config", version = "1.0.6" }
rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.6" }
rustfs-credentials = { path = "crates/credentials", version = "1.0.6" }
rustfs-crypto = { path = "crates/crypto", version = "1.0.6" }
rustfs-ecstore = { path = "crates/ecstore", version = "1.0.6" }
rustfs-filemeta = { path = "crates/filemeta", version = "1.0.6" }
rustfs-iam = { path = "crates/iam", version = "1.0.6" }
rustfs-keystone = { path = "crates/keystone", version = "1.0.6" }
rustfs-license = { path = "crates/license", version = "1.0.6" }
rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.6" }
rustfs-kms = { path = "crates/kms", version = "1.0.6" }
rustfs-lock = { path = "crates/lock", version = "1.0.6" }
rustfs-madmin = { path = "crates/madmin", version = "1.0.6" }
rustfs-notify = { path = "crates/notify", version = "1.0.6" }
rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.6" }
rustfs-io-core = { path = "crates/io-core", version = "1.0.6" }
rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.6" }
rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.6", default-features = false }
rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.6" }
rustfs-obs = { path = "crates/obs", version = "1.0.6" }
rustfs-policy = { path = "crates/policy", version = "1.0.6" }
rustfs-protos = { path = "crates/protos", version = "1.0.6" }
rustfs-protocols = { path = "crates/protocols", version = "1.0.6" }
rustfs-replication = { path = "crates/replication", version = "1.0.6" }
rustfs-rio = { path = "crates/rio", version = "1.0.6" }
rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.6" }
rustfs-s3-client = { path = "crates/s3-client", version = "1.0.6" }
rustfs-s3-types = { path = "crates/s3-types", version = "1.0.6" }
rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.6" }
rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.6" }
rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.6" }
rustfs-scanner = { path = "crates/scanner", version = "1.0.6" }
rustfs-security-governance = { path = "crates/security-governance", version = "1.0.6" }
rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.6" }
rustfs-signer = { path = "crates/signer", version = "1.0.6" }
rustfs-storage-api = { path = "crates/storage-api", version = "1.0.6" }
rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.6" }
rustfs-targets = { path = "crates/targets", version = "1.0.6" }
rustfs-test-utils = { path = "crates/test-utils", version = "1.0.6" }
rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.6" }
rustfs-utils = { path = "crates/utils", version = "1.0.6" }
rustfs-zip = { path = "./crates/zip", version = "1.0.6" }
rustfs-gateway-adapter = { path = "crates/gateway-adapter", version = "1.0.5" }
rustfs-heal = { path = "crates/heal", version = "1.0.5" }
rustfs-heal-contracts = { path = "crates/heal-contracts", version = "1.0.5" }
rustfs-scanner-metrics = { path = "crates/scanner-metrics", version = "1.0.5" }
rustfs-audit = { path = "crates/audit", version = "1.0.5" }
rustfs-checksums = { path = "crates/checksums", version = "1.0.5" }
rustfs-common = { path = "crates/common", version = "1.0.5" }
rustfs-data-usage = { path = "crates/data-usage", version = "1.0.5" }
rustfs-config = { path = "./crates/config", version = "1.0.5" }
rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.5" }
rustfs-credentials = { path = "crates/credentials", version = "1.0.5" }
rustfs-crypto = { path = "crates/crypto", version = "1.0.5" }
rustfs-cuobject = { path = "crates/cuobject", version = "1.0.5" }
rustfs-nixl = { path = "crates/nixl", version = "1.0.5" }
rustfs-ecstore = { path = "crates/ecstore", version = "1.0.5" }
rustfs-filemeta = { path = "crates/filemeta", version = "1.0.5" }
rustfs-iam = { path = "crates/iam", version = "1.0.5" }
rustfs-keystone = { path = "crates/keystone", version = "1.0.5" }
rustfs-license = { path = "crates/license", version = "1.0.5" }
rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.5" }
rustfs-kms = { path = "crates/kms", version = "1.0.5" }
rustfs-lock = { path = "crates/lock", version = "1.0.5" }
rustfs-madmin = { path = "crates/madmin", version = "1.0.5" }
rustfs-notify = { path = "crates/notify", version = "1.0.5" }
rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.5" }
rustfs-io-core = { path = "crates/io-core", version = "1.0.5" }
rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.5" }
rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.5", default-features = false }
rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.5" }
rustfs-obs = { path = "crates/obs", version = "1.0.5" }
rustfs-policy = { path = "crates/policy", version = "1.0.5" }
rustfs-protos = { path = "crates/protos", version = "1.0.5" }
rustfs-protocols = { path = "crates/protocols", version = "1.0.5" }
rustfs-replication = { path = "crates/replication", version = "1.0.5" }
rustfs-rio = { path = "crates/rio", version = "1.0.5" }
rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.5" }
rustfs-s3-client = { path = "crates/s3-client", version = "1.0.5" }
rustfs-s3-types = { path = "crates/s3-types", version = "1.0.5" }
rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.5" }
rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.5" }
rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.5" }
rustfs-scanner = { path = "crates/scanner", version = "1.0.5" }
rustfs-security-governance = { path = "crates/security-governance", version = "1.0.5" }
rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.5" }
rustfs-signer = { path = "crates/signer", version = "1.0.5" }
rustfs-storage-api = { path = "crates/storage-api", version = "1.0.5" }
rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.5" }
rustfs-targets = { path = "crates/targets", version = "1.0.5" }
rustfs-test-utils = { path = "crates/test-utils", version = "1.0.5" }
rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.5" }
rustfs-utils = { path = "crates/utils", version = "1.0.5" }
rustfs-zip = { path = "./crates/zip", version = "1.0.5" }
# The gateway's S3 types: the DTOs and persisted bucket-configuration codecs the
# engine crates move onto from s3s (rustfs/backlog#2745). The one pin of the
# gateway in this workspace; no crate may enable a `compat-s3s*` feature except
# the s3s-facing edge of the rustfs binary.
rustfs-gateway-types = { git = "https://github.com/rustfs/gateway.git", rev = "2a6b8d1f7663db978d8784ab3ea331cade19237b", version = "0.40.0", default-features = false }
rustfs-gateway-stream = { git = "https://github.com/rustfs/gateway.git", rev = "2a6b8d1f7663db978d8784ab3ea331cade19237b", version = "0.2.0", default-features = false }
# The gateway framework itself, consumed for its extension traits by the ring-2
# adapter. `default-features = false` keeps the `server` feature, and with it the
# server crate, out of the build.
rustfs-gateway = { git = "https://github.com/rustfs/gateway.git", rev = "2a6b8d1f7663db978d8784ab3ea331cade19237b", version = "0.63.0", default-features = false }
# Test-only: builds the `EffectiveHost` a host-resolution test queries with.
rustfs-gateway-http = { git = "https://github.com/rustfs/gateway.git", rev = "2a6b8d1f7663db978d8784ab3ea331cade19237b", default-features = false }
# Async Runtime and Networking
async-channel = "2.5.0"
@@ -149,7 +166,7 @@ async_zip = { default-features = false, version = "0.0.19" }
mysql_async = { default-features = false, version = "0.37.1" }
async-compression = { version = "0.4.52" }
async-trait = "0.1.92"
async-nats = { version = "0.51.0", default-features = false }
async-nats = { version = "0.52.0", default-features = false }
axum = "0.8.9"
futures = "0.3.34"
futures-core = "0.3.34"
@@ -169,6 +186,7 @@ memchr = "2.8.3"
minlz = "1.2.3"
reqwest = "0.13.5"
rustfs-kafka-async = { version = "1.4.1" }
slab = { version = "0.4.12" }
socket2 = { version = "0.6.5" }
tokio = { version = "1.54.1" }
tokio-rustls = { default-features = false, version = "0.26.6" }
@@ -196,7 +214,7 @@ bytesize = "2.7.0"
byteorder = "1.5.0"
flatbuffers = "25.12.19"
form_urlencoded = "1.2.2"
prost = "0.14.4"
prost = "0.14.6"
quick-xml = "0.42.0"
rmp = { version = "0.8.15" }
rmp-serde = { version = "1.3.1" }
@@ -218,14 +236,14 @@ blake2 = "0.11.0"
chacha20poly1305 = { version = "0.11.0" }
crc-fast = "1.10.0"
hmac = { version = "0.13.0" }
jsonwebtoken = { version = "11.1.0" }
jsonwebtoken = { version = "12.0.0" }
openidconnect = { default-features = false, version = "4.1.0" }
pbkdf2 = "0.13.0"
p256 = { version = "0.14.0", features = ["ecdsa", "pkcs8"] }
rsa = { version = "=0.10.0-rc.18" }
rustls = { default-features = false, version = "0.23.46" }
rustls-native-certs = "0.8"
rustls-pki-types = "1.15.1"
rustls-pki-types = "1.16.0"
x509-parser = "0.18.1"
sha1 = "0.11.0"
sha2 = "0.11.1"
@@ -236,7 +254,7 @@ proptest = "1"
# Time and Date
chrono = { version = "0.4.45" }
humantime = "2.4.0"
jiff = { version = "0.2.39" }
jiff = { version = "0.2.43" }
time = { version = "0.3.55" }
# Database
@@ -246,22 +264,22 @@ tokio-postgres-rustls = "0.14.0"
# Utilities and Tools
anyhow = "1.0.104"
arc-swap = "1.9.3"
arc-swap = "1.9.5"
rustfs-tokio-tar = { version = "0.7.1" }
# Candidate Snowball parser versions exercised by rustfs-zip compatibility fixtures.
tar-codec = "0.0.15"
tar-framing = "0.0.15"
tar-codec = "0.0.16"
tar-framing = "0.0.16"
atoi = "3.1.1"
atomic_enum = "0.3.0"
aws-config = { version = "1.13.0" }
aws-config = { version = "1.13.2" }
aws-credential-types = { version = "1.3.1" }
aws-sdk-kms = { default-features = false, version = "1.129.0" }
aws-sdk-s3 = { default-features = false, version = "1.157.0" }
aws-sdk-sts = { default-features = false, version = "1.124.0" }
aws-sdk-kms = { default-features = false, version = "1.131.0" }
aws-sdk-s3 = { default-features = false, version = "1.159.0" }
aws-sdk-sts = { default-features = false, version = "1.126.0" }
aws-smithy-async = { version = "1.3.1" }
aws-smithy-http-client = { default-features = false, version = "1.5.2" }
aws-smithy-runtime-api = { version = "1.19.2" }
aws-smithy-types = { version = "1.9.1" }
aws-smithy-runtime-api = { version = "1.21.0" }
aws-smithy-types = { version = "1.9.4" }
base64-simd = "0.8.0"
brotli = "9.1.1"
clap = { version = "4.6.7" }
@@ -277,10 +295,10 @@ derive_builder = "0.20.2"
ed25519-dalek = "3.0.0"
enumset = "1.1.15"
faster-hex = "1.0.0"
flate2 = "1.1.10"
flate2 = "1.1.11"
glob = "0.3.4"
google-cloud-storage = "1.21.0"
google-cloud-auth = "1.18.0"
google-cloud-storage = "1.22.0"
google-cloud-auth = "1.19.0"
hashbrown = { version = "0.17.1" }
# Base32 for RFC 6238 TOTP shared secrets (RFC 4648 unpadded, the alphabet
# every authenticator app expects). Already in the graph transitively.
@@ -291,10 +309,12 @@ hostname = "0.4.2"
ipnetwork = { version = "0.21.1" }
lazy_static = "1.5.1"
libc = "0.2.191"
cuobject = "0.1.0"
libsystemd = "0.7.2"
local-ip-address = "0.6.13"
log = "0.4"
memmap2 = "0.9.12"
nixl-sys = { git = "https://github.com/houseme/nixl", rev = "cc636c83f13d692e85b67e55aed092413d16fb41" }
lz4 = "1.28.1"
lz4_flex = { version = "=0.14.0", default-features = false, features = ["std", "frame"] }
matchit = "0.9.2"
@@ -304,7 +324,7 @@ moka = { version = "0.12.16" }
netif = "0.1.6"
num_cpus = { version = "1.17.0" }
nvml-wrapper = "0.13.0"
object = { version = "=0.40.0", default-features = false }
object = { version = "=0.41.0", default-features = false }
parking_lot = "0.13.0"
same-file = "1.0.6"
path-absolutize = "4.0.1"
@@ -318,26 +338,31 @@ pretty_assertions = "1.4.1"
rand = { version = "0.10.3" }
ratelimit = "2.0.0"
rayon = "1.12.0"
rustfs-erasure-codec = { version = "9.0.0" }
rustfs-erasure-codec = { version = "9.1.0" }
reed-solomon-simd = "3.1.0"
regex = { version = "1.13.1" }
rumqttc = { package = "rumqttc-next", version = "0.34.0" }
redis = { version = "1.8.1" }
rustify = { version = "0.7", default-features = false }
rustix = { version = "1.1.6" }
rust-embed = { version = "8.13.0" }
rust-embed = { version = "8.14.0" }
rustc-hash = { version = "2.1.3" }
s3s = { git = "https://github.com/s3s-project/s3s", rev = "5761ddfe4c509fcac87e3b99fd5bd9e176b442c0", features = ["minio"] }
s3s-sigv4 = { git = "https://github.com/s3s-project/s3s", rev = "5761ddfe4c509fcac87e3b99fd5bd9e176b442c0" }
# The same s3s pin under the name the legacy edge of the gateway migration uses
# (crates/s3-types, feature `compat-s3s`). Keep `rev` identical to `s3s` above.
# DELETE BY T4.2 (rustfs/backlog#2784).
legacy-s3s = { package = "s3s", git = "https://github.com/s3s-project/s3s", rev = "5761ddfe4c509fcac87e3b99fd5bd9e176b442c0", features = ["minio"] }
serial_test = "4.0.1"
shadow-rs = { default-features = false, version = "2.0.0" }
siphasher = "1.0.4"
smallvec = { version = "1.16.4" }
smallvec = { version = "1.16.5" }
compact_str = "0.10.0"
snap = "1.1.2"
starshard = { version = "2.4.0" }
starshard = { version = "2.4.1" }
strum = { version = "0.28.0" }
sysinfo = "0.40.1"
sync_wrapper = "1.0.2"
sysinfo = "0.40.3"
temp-env = "0.3.6"
tempfile = "3.27.0"
test-case = "3.4.1"
@@ -351,7 +376,7 @@ tracing-subscriber = { version = "0.3.23" }
transform-stream = "0.3.1"
url = "2.5.8"
urlencoding = "2.1.3"
uuid = { version = "1.29.0" }
uuid = { version = "1.29.2" }
vaultrs = { version = "0.8.0" }
tar = "0.4.46"
walkdir = "2.5.0"
@@ -359,13 +384,13 @@ winapi-util = "0.1.11"
windows = { version = "0.62.2" }
windows-sys = "0.61.2"
xxhash-rust = { version = "0.8.19" }
zip = "9.3.0"
zstd = "0.14.3"
zip = "9.5.1"
zstd = "0.14.4"
# Observability and Metrics
metrics = "0.24.6"
metrics-util = "0.20"
dial9 = { version = "0.5.7", default-features = false }
dial9 = { version = "0.5.8", default-features = false }
opentelemetry = { version = "0.33.1" }
opentelemetry-appender-tracing = { version = "0.33.1" }
opentelemetry-otlp = { version = "0.33.1" }
@@ -376,20 +401,20 @@ opentelemetry-stdout = { version = "0.33.1" }
pyroscope = { version = "2.1.5" }
# FTP and SFTP
libunftp = { version = "0.24.0" }
unftp-core = "0.1.0"
libunftp = { version = "0.25.0" }
unftp-core = "0.1.1"
suppaftp = { version = "12.2.1" }
rcgen = { version = "0.14.10", default-features = false, features = ["aws_lc_rs", "crypto", "pem"] }
russh = { version = "0.65.1" }
russh-sftp = "3.0.1"
# WebDAV
dav-server = "0.11.0"
dav-server = "0.12.0"
# Performance Analysis and Memory Profiling
rustfs-mimalloc = { version = "0.7.3" }
rustfs-mimalloc = { version = "0.7.4" }
# Preserve Unicode focus filters until rustfs/backlog#2302 is resolved.
hotpath = { version = "0.30.3", default-features = false }
hotpath = { version = "0.31.2", default-features = false }
# Snapshot testing for output format regression detection
insta = { version = "1.49" }
+11 -2
View File
@@ -12,7 +12,9 @@
# See the License for the specific language governing permissions and
# limitations under the License.
FROM alpine:3.24.2 AS build
ARG BINARY_SOURCE=download
FROM alpine:3.24.2 AS download
ARG TARGETARCH
ARG RELEASE=latest
@@ -74,6 +76,14 @@ COPY scripts/download_rc.sh /usr/local/bin/download_rc.sh
RUN sh /usr/local/bin/download_rc.sh "$TARGETARCH" musl "$RC_RELEASE" /build/rc
# Candidate packaging supplies the accepted executables from verified archives.
# The default download stage remains available for local image builds.
FROM scratch AS supplied
ARG TARGETARCH
COPY .release-inputs/${TARGETARCH}/ /build/
FROM ${BINARY_SOURCE} AS build
FROM alpine:3.24.2
ARG RELEASE=latest
@@ -103,7 +113,6 @@ RUN apk upgrade --no-cache && \
tzdata \
&& test "$(TZ=Asia/Kolkata date +%z)" = "+0530"
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=build /build/rustfs /usr/bin/rustfs
COPY --from=build /build/rc /usr/bin/rc
COPY entrypoint.sh /etc/rustfs/entrypoint.sh
+11 -1
View File
@@ -12,7 +12,9 @@
# See the License for the specific language governing permissions and
# limitations under the License.
FROM ubuntu:26.04 AS build
ARG BINARY_SOURCE=download
FROM ubuntu:26.04 AS download
ARG TARGETARCH
ARG RELEASE=latest
@@ -80,6 +82,14 @@ RUN set -eux; \
COPY scripts/download_rc.sh /usr/local/bin/download_rc.sh
RUN sh /usr/local/bin/download_rc.sh "$TARGETARCH" gnu "$RC_RELEASE" /build/rc
# Candidate packaging supplies the accepted executables from verified archives.
# The default download stage remains available for local image builds.
FROM scratch AS supplied
ARG TARGETARCH
COPY .release-inputs/${TARGETARCH}/ /build/
FROM ${BINARY_SOURCE} AS build
FROM ubuntu:26.04
ARG RELEASE=latest
+29 -4
View File
@@ -24,9 +24,10 @@
#
# Supports cross-compilation for amd64 and arm64 via TARGETPLATFORM.
ARG TARGETPLATFORM
ARG BUILDPLATFORM
ARG TARGETARCH
# Do not declare TARGETPLATFORM, BUILDPLATFORM or TARGETARCH here: a global
# declaration without a value shadows BuildKit's automatic values, so the
# stage-level re-declarations below see empty strings. They are only
# declared inside the stage that uses them.
ARG RUSTFS_BUILD_FEATURES=""
ARG RC_RELEASE=latest
@@ -169,7 +170,31 @@ RUN --mount=type=cache,target=/usr/local/cargo/registry \
;; \
esac
RUN sh /usr/src/rustfs/scripts/download_rc.sh "$TARGETARCH" musl "$RC_RELEASE" /usr/local/bin/rc
# `RC_RELEASE=disabled` is reserved for local server-only validation when the
# GitHub release asset is unavailable. The placeholder fails if invoked, so a
# production image that needs the CLI must keep the default release download.
# TARGETARCH is only provided by BuildKit. Fall back to TARGETPLATFORM (for
# example set by docker-compose) and then to the build host, like the cargo
# step above, so every path that builds this file passes a real architecture.
RUN set -eu; \
if [ "${RC_RELEASE}" = "disabled" ]; then \
install -m 0755 /bin/false /usr/local/bin/rc; \
exit 0; \
fi; \
rc_arch="${TARGETARCH:-}"; \
if [ -z "${rc_arch}" ]; then \
case "${TARGETPLATFORM:-}" in \
linux/amd64) rc_arch=amd64 ;; \
linux/arm64) rc_arch=arm64 ;; \
*) \
case "$(uname -m)" in \
x86_64) rc_arch=amd64 ;; \
aarch64|arm64) rc_arch=arm64 ;; \
*) echo "TARGETARCH is empty and TARGETPLATFORM='${TARGETPLATFORM:-}' is not linux/amd64 or linux/arm64; cannot choose an rc architecture on $(uname -m)" >&2; exit 1 ;; \
esac ;; \
esac; \
fi; \
sh /usr/src/rustfs/scripts/download_rc.sh "${rc_arch}" musl "${RC_RELEASE}" /usr/local/bin/rc
# -----------------------------
# Development stage (keeps toolchain)
+8 -4
View File
@@ -145,7 +145,7 @@ chown -R 10001:10001 data logs
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
# Using specific version
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.4
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.5
```
If you use [podman](https://github.com/containers/podman) instead of docker, you can install the RustFS with the below command
@@ -199,7 +199,7 @@ docker run -d --name rustfs -p 9000:9000 \
Notes:
- `RUSTFS_NOTIFY_ENABLE=true` enables the global notify module switch.
- For ARN `arn:rustfs:sqs::primary:webhook`, use instance-scoped env vars with `_PRIMARY`.
- If queue dir is omitted, the official image maps `/opt/rustfs/events` into the persistent `/data` volume so it is writable by the runtime user and pending events survive container recreation. Override `queue_dir` when using another deployment layout.
- If queue dir is omitted, the official image maps `/opt/rustfs/events` into the persistent `/data` volume so it is writable by the runtime user and pending events survive container recreation. Override `queue_dir` when using another deployment layout, or set `RUSTFS_TARGET_STORE_DIR` to move every default queue; native Windows defaults to `%ProgramData%\RustFS`. See [Target queue storage](docs/operations/target-queue-storage.md).
- `RUSTFS_NOTIFY_WEBHOOK_SKIP_TLS_VERIFY_PRIMARY` defaults to `false`; enabling it skips webhook TLS certificate verification, allows MITM attacks, and emits a startup warning. Prefer `RUSTFS_NOTIFY_WEBHOOK_CLIENT_CA_PRIMARY` for private CAs.
- Since `1.0.0-beta.11`, webhook endpoints on private or container networks
(`Docker Compose service names`, `host.docker.internal`, RFC 1918 addresses) are
@@ -224,7 +224,7 @@ For developers who want to build RustFS Docker images from source with multi-arc
./docker-buildx.sh --push
# Build specific version
./docker-buildx.sh --release 1.0.4 --push
./docker-buildx.sh --release 1.0.5 --push
# Build for custom registry
./docker-buildx.sh --registry your-registry.com --namespace yourname --push
@@ -242,7 +242,7 @@ You can also use Make targets for convenience:
```bash
make docker-buildx # Build locally
make docker-buildx-push # Build and push
make docker-buildx-version VERSION=1.0.4 # Build specific version
make docker-buildx-version VERSION=1.0.5 # Build specific version
make help-docker # Show all Docker-related commands
```
@@ -311,6 +311,10 @@ x env use rustfs
rustfs --help
```
### 7\. Docker Swarm with the SwarmCLI chart (Option 7)
Follow the [SwarmCLI chart guide](https://docs.rustfs.com/en/installation/cloud-native/swarmcli-chart) to run RustFS on Docker Swarm, as a single-node service or a distributed, erasure-coded cluster with one peer per node, with the open-source [SwarmCLI `rustfs` chart](https://github.com/Eldara-Tech/swarmcli-charts/tree/main/charts/rustfs). The chart keeps the root credentials in Docker secrets and refuses the default `rustfsadmin` keys, upgrades and rolls back as versioned releases, and can serve the S3 API over HTTPS through Traefik. Its Console stays off until you enable it. The chart and [SwarmCLI](https://github.com/Eldara-Tech/swarmcli) are maintained by Eldara Tech, not by the RustFS project.
---
### Accessing RustFS
+7 -3
View File
@@ -138,7 +138,7 @@ chown -R 10001:10001 data logs
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
# 使用指定版本运行
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.4
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.5
```
如果您通过绑定挂载启用 TLS 证书目录,也请用同样方式准备该目录:
@@ -176,7 +176,7 @@ docker compose -f docker-compose-simple.yml up -d
./docker-buildx.sh --push
# 构建指定版本
./docker-buildx.sh --release 1.0.4 --push
./docker-buildx.sh --release 1.0.5 --push
# 构建并推送到自定义仓库
./docker-buildx.sh --registry your-registry.com --namespace yourname --push
@@ -193,7 +193,7 @@ docker compose -f docker-compose-simple.yml up -d
```bash
make docker-buildx # 本地构建
make docker-buildx-push # 构建并推送
make docker-buildx-version VERSION=1.0.4 # 构建指定版本
make docker-buildx-version VERSION=1.0.5 # 构建指定版本
make help-docker # 显示所有 Docker 相关命令
```
@@ -239,6 +239,10 @@ x env use rustfs
rustfs --help
```
### 7\. 使用 SwarmCLI Chart 部署到 Docker Swarm (Option 7)
请按照 [SwarmCLI Chart 指南](https://docs.rustfs.com/zh/installation/cloud-native/swarmcli-chart),使用开源的 [SwarmCLI `rustfs` Chart](https://github.com/Eldara-Tech/swarmcli-charts/tree/main/charts/rustfs) 在 Docker Swarm 上运行 RustFS,可部署为单节点服务,或每个节点一个对等节点、采用纠删码的分布式集群。该 Chart 将 root 凭证保存在 Docker Secret 中并拒绝默认的 `rustfsadmin` 密钥,以带版本的发布进行升级和回滚,并可通过 Traefik 以 HTTPS 提供 S3 API。控制台默认关闭,需要时再启用。该 Chart 和 [SwarmCLI](https://github.com/Eldara-Tech/swarmcli) 由 Eldara Tech 维护,并非 RustFS 项目的一部分。
---
### 访问 RustFS
+52 -72
View File
@@ -34,9 +34,6 @@ async fn test_audit_system_startup_performance() {
println!("Audit system startup took: {elapsed:?}");
// Should complete within 5 seconds
assert!(elapsed < Duration::from_secs(5), "Startup took too long: {elapsed:?}");
// Clean up
let _ = system.close().await;
}
@@ -68,9 +65,6 @@ async fn test_concurrent_target_creation() {
println!("Concurrent target creation took: {elapsed:?}");
// Should complete quickly even with multiple targets
assert!(elapsed < Duration::from_secs(10), "Target creation took too long: {elapsed:?}");
// Verify it fails with expected error (server not initialized)
match result {
Err(AuditError::StorageNotAvailable(_)) => {
@@ -166,9 +160,6 @@ async fn test_audit_log_dispatch_performance() {
println!("Audit log dispatch took: {elapsed:?}");
// Should be very fast (sub-millisecond for no targets)
assert!(elapsed < Duration::from_millis(100), "Dispatch took too long: {elapsed:?}");
assert!(
matches!(result, Err(AuditError::NotInitialized(_))),
"Dispatch on a stopped system should return NotInitialized, got: {result:?}"
@@ -207,11 +198,12 @@ async fn test_system_state_transitions() {
}
#[test]
fn test_event_name_mask_performance() {
fn test_event_name_mask_matches_expansion() {
use rustfs_targets::EventName;
// Test that event name mask calculation is efficient
let events = vec![
// A compound event's mask is exactly the union of its leaves' masks, and a
// leaf owns exactly one bit. Timing is printed for information only.
let events = [
EventName::ObjectCreatedPut,
EventName::ObjectAccessedGet,
EventName::ObjectRemovedDelete,
@@ -220,47 +212,59 @@ fn test_event_name_mask_performance() {
];
let start = Instant::now();
// Calculate masks for many events
for _ in 0..1000 {
for event in &events {
let _mask = event.mask();
}
for event in events {
let union = event.expand().iter().fold(0u64, |acc, leaf| acc | leaf.mask());
assert_eq!(event.mask(), union, "{event:?} mask must equal the union of its expansion");
}
println!("Event mask check took: {:?}", start.elapsed());
let elapsed = start.elapsed();
println!("Event mask calculation (5000 ops) took: {elapsed:?}");
// Should be very fast
assert!(elapsed < Duration::from_millis(100), "Mask calculation too slow: {elapsed:?}");
assert_eq!(EventName::ObjectCreatedPut.mask().count_ones(), 1);
assert_eq!(EventName::ObjectCreatedAll.mask().count_ones(), 4);
}
#[test]
fn test_event_name_expansion_performance() {
use rustfs_targets::EventName;
use std::collections::HashSet;
// Test that event name expansion is efficient
let compound_events = vec![
EventName::ObjectCreatedAll,
EventName::ObjectAccessedAll,
EventName::ObjectRemovedAll,
EventName::Everything,
// Expansion is a pure lookup. Wall-clock bounds flake on shared runners, so
// this asserts what the time was a proxy for: each compound event yields a
// fixed-size set of distinct leaf events and nothing else (no duplicates,
// no compound left to expand again). Timing is printed for information only.
let cases = [
(EventName::ObjectCreatedAll, 4),
(EventName::ObjectAccessedAll, 5),
(EventName::ObjectRemovedAll, 3),
];
let start = Instant::now();
// Expand events many times
for _ in 0..1000 {
for event in &compound_events {
let _expanded = event.expand();
for (event, expected_len) in cases {
let expanded = event.expand();
assert_eq!(expanded.len(), expected_len, "{event:?} expansion size");
assert_eq!(
expanded.iter().collect::<HashSet<_>>().len(),
expanded.len(),
"{event:?} expansion has duplicates"
);
assert_eq!(expanded, event.expand(), "{event:?} expansion must be deterministic");
for leaf in &expanded {
assert_eq!(leaf.expand(), vec![*leaf], "{event:?} expanded to non-leaf {leaf:?}");
}
}
let elapsed = start.elapsed();
println!("Event expansion (4000 ops) took: {elapsed:?}");
// Should be very fast
assert!(elapsed < Duration::from_millis(100), "Expansion too slow: {elapsed:?}");
let everything = EventName::Everything.expand();
assert!(everything.len() > 12, "Everything must cover at least every compound's leaves");
assert_eq!(
everything.iter().collect::<HashSet<_>>().len(),
everything.len(),
"Everything has duplicates"
);
for (event, _) in cases {
for leaf in event.expand() {
assert!(everything.contains(&leaf), "Everything is missing {leaf:?}");
}
}
println!("Event expansion check took: {:?}", start.elapsed());
}
#[tokio::test]
@@ -278,46 +282,22 @@ async fn test_registry_operations_performance() {
let elapsed = start.elapsed();
println!("Registry operations (2000 ops) took: {elapsed:?}");
// Should be very fast for empty registry
assert!(elapsed < Duration::from_millis(100), "Registry ops too slow: {elapsed:?}");
}
// Performance requirements validation
// The former 3k EPS / latency assertions measured the CI host, not the code
// (the loop only formats strings and reads the clock), so they are replaced by
// a count of what the loop produced. Throughput is printed for information.
#[test]
fn test_performance_requirements() {
// According to requirements: ≥ 3k EPS/node; P99 < 30ms (default)
// These are synthetic tests since we can't actually achieve 3k EPS
// without real server storage and network targets, but we can validate
// that our core algorithms are efficient enough
use std::collections::HashSet;
let start = Instant::now();
// Simulate processing 3000 events worth of operations
for i in 0..3000 {
// Simulate event name parsing and processing
let _event_id = format!("s3:ObjectCreated:Put_{i}");
let _timestamp = jiff::Timestamp::now().to_string();
// Simulate basic audit entry creation overhead
let _entry_size = 512; // bytes
let _processing_time = std::time::Duration::from_nanos(100); // simulated
}
let ids: HashSet<String> = (0..3000).map(|i| format!("s3:ObjectCreated:Put_{i}")).collect();
let timestamps = (0..3000).filter(|_| !jiff::Timestamp::now().to_string().is_empty()).count();
let elapsed = start.elapsed();
let eps = 3000.0 / elapsed.as_secs_f64();
println!("Simulated 3000 events in {elapsed:?} ({eps:.0} EPS)");
println!("Simulated 3000 events in {elapsed:?}");
// Our core processing should easily handle 3k EPS worth of CPU overhead
// The actual EPS limit will be determined by network I/O to targets
assert!(eps > 10000.0, "Core processing too slow for 3k EPS target: {eps} EPS");
// P99 latency requirement: < 30ms
// For core processing, we should be much faster than this
let avg_latency = elapsed / 3000;
println!("Average processing latency: {avg_latency:?}");
assert!(avg_latency < Duration::from_millis(1), "Processing latency too high: {avg_latency:?}");
assert_eq!(ids.len(), 3000, "every simulated event id must be distinct");
assert_eq!(timestamps, 3000, "every simulated event must get a timestamp");
}
+3 -4
View File
@@ -30,12 +30,11 @@ workspace = true
[features]
default = []
hotpath = ["hotpath/hotpath"]
hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu"]
hotpath = []
hotpath-alloc = ["hotpath"]
hotpath-cpu = ["hotpath"]
[dependencies]
hotpath.workspace = true
bytes = { workspace = true, features = ["serde"] }
crc-fast = { workspace = true }
http = { workspace = true }
+100
View File
@@ -902,4 +902,104 @@ mod tests {
assert_eq!(raw.len(), 8);
assert_eq!(&raw[..], reference.digest().to_be_bytes().as_slice());
}
#[test]
fn checksums_streaming_chunks_and_header_encoding_match_oneshot_across_unaligned_boundaries() {
use crate::http::HttpChecksum;
let all = [
ChecksumAlgorithm::Crc32,
ChecksumAlgorithm::Crc32c,
ChecksumAlgorithm::Crc64Nvme,
ChecksumAlgorithm::Sha1,
ChecksumAlgorithm::Sha256,
ChecksumAlgorithm::Sha512,
ChecksumAlgorithm::Xxhash3,
ChecksumAlgorithm::Xxhash64,
ChecksumAlgorithm::Xxhash128,
];
for &payload_len in &[0usize, 1, 31, 33, 1024, 4096 + 37, 64 * 1024 + 19] {
let payload: Vec<u8> = (0..payload_len).map(|i| ((i * 19 + 5) % 253) as u8).collect();
for algorithm in all {
let mut one_shot = algorithm.into_impl();
one_shot.update(&payload);
let expected_raw = one_shot.finalize();
assert_eq!(expected_raw.len(), algorithm.raw_len());
let mut one_shot_hdr = algorithm.into_impl();
one_shot_hdr.update(&payload);
let expected_headers = one_shot_hdr.headers();
for &chunk_size in &[1usize, 31, 1021, 4096] {
if payload_len > 4096 && chunk_size == 1 {
continue;
}
let mut streaming: Box<dyn HttpChecksum> = algorithm.into_impl();
for chunk in payload.chunks(chunk_size) {
streaming.update(chunk);
}
let actual_headers = streaming.headers();
assert_eq!(
actual_headers.get(algorithm.http_header_name()),
expected_headers.get(algorithm.http_header_name()),
"{algorithm:?} streaming mismatch at payload_len={payload_len}, chunk_size={chunk_size}"
);
}
}
}
}
#[test]
#[ignore = "utility throughput microbenchmark; run via cargo test -p rustfs-checksums --lib --release -- --ignored --nocapture"]
fn microbench_checksums_streaming_chunks_and_header_encoding() {
use crate::http::HttpChecksum;
use std::hint::black_box;
use std::time::Instant;
let all = [
ChecksumAlgorithm::Crc32,
ChecksumAlgorithm::Crc32c,
ChecksumAlgorithm::Crc64Nvme,
ChecksumAlgorithm::Sha1,
ChecksumAlgorithm::Sha256,
ChecksumAlgorithm::Sha512,
ChecksumAlgorithm::Xxhash3,
ChecksumAlgorithm::Xxhash64,
ChecksumAlgorithm::Xxhash128,
];
let payload: Vec<u8> = (0..(256 * 1024)).map(|i| ((i * 19 + 5) % 253) as u8).collect();
for &chunk_size in &[4 * 1024usize, 64 * 1024, 256 * 1024] {
for algorithm in all {
let mut one_shot = algorithm.into_impl();
one_shot.update(&payload);
let expected_headers = one_shot.headers();
let iters = 8usize;
let t0 = Instant::now();
let mut last_headers = http::HeaderMap::new();
for _ in 0..iters {
let mut streaming: Box<dyn HttpChecksum> = algorithm.into_impl();
for chunk in payload.chunks(chunk_size) {
streaming.update(black_box(chunk));
}
last_headers = streaming.headers();
black_box(&last_headers);
}
let elapsed = t0.elapsed();
assert_eq!(
last_headers.get(algorithm.http_header_name()),
expected_headers.get(algorithm.http_header_name()),
"{algorithm:?} streaming chunk_size={chunk_size} header mismatch"
);
let total_mib = (payload.len() * iters) as f64 / (1024.0 * 1024.0);
let mibs = total_mib / elapsed.as_secs_f64().max(1e-9);
println!(
"[microbench][checksums] algo={algorithm:?} chunk_size={chunk_size}B throughput={mibs:.1} MiB/s (size={}B)",
algorithm.raw_len()
);
}
}
}
}
+3 -4
View File
@@ -30,13 +30,12 @@ workspace = true
[features]
default = []
test-util = []
hotpath = ["hotpath/hotpath", "hotpath/tokio"]
hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu"]
hotpath = []
hotpath-alloc = ["hotpath"]
hotpath-cpu = ["hotpath"]
[dependencies]
arc-swap.workspace = true
hotpath.workspace = true
tokio = { workspace = true, features = ["fs", "rt-multi-thread"] }
tonic = { workspace = true, features = ["gzip", "deflate"] }
uuid = { workspace = true, features = ["v4", "fast-rng", "macro-diagnostics"] }
+157
View File
@@ -23,7 +23,80 @@ pub static GLOBAL_LOCAL_NODE_NAME: LazyLock<RwLock<String>> = LazyLock::new(|| R
pub static GLOBAL_RUSTFS_HOST: LazyLock<RwLock<String>> = LazyLock::new(|| RwLock::new("".to_string()));
pub static GLOBAL_RUSTFS_PORT: LazyLock<RwLock<String>> = LazyLock::new(|| RwLock::new("9000".to_string()));
pub static GLOBAL_RUSTFS_ADDR: LazyLock<RwLock<String>> = LazyLock::new(|| RwLock::new("".to_string()));
static NEXT_GRPC_CHANNEL_ID: AtomicU64 = AtomicU64::new(1);
static NEXT_GRPC_CONNECTION_GENERATION: AtomicU64 = AtomicU64::new(1);
#[derive(Debug)]
struct GrpcConnectionStateInner {
channel_id: u64,
generation: AtomicU64,
}
/// Lock-free identity and physical-connection generation shared by every clone of one gRPC channel.
#[derive(Clone, Debug)]
pub struct GrpcConnectionState(Arc<GrpcConnectionStateInner>);
/// A point-in-time identity for the physical connection currently used by a gRPC channel.
#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq)]
pub struct GrpcConnectionSnapshot {
pub channel_id: u64,
pub generation: u64,
}
impl GrpcConnectionState {
pub fn new() -> Self {
Self(Arc::new(GrpcConnectionStateInner {
channel_id: next_nonzero_id(&NEXT_GRPC_CHANNEL_ID),
generation: AtomicU64::new(0),
}))
}
/// Publish a successfully opened physical transport before tonic makes it request-ready.
pub fn record_connected(&self) -> GrpcConnectionSnapshot {
let generation = next_nonzero_id(&NEXT_GRPC_CONNECTION_GENERATION);
self.0.generation.store(generation, Ordering::Release);
GrpcConnectionSnapshot {
channel_id: self.0.channel_id,
generation,
}
}
/// Return `None` until the first physical transport has connected successfully.
pub fn snapshot(&self) -> Option<GrpcConnectionSnapshot> {
let generation = self.0.generation.load(Ordering::Acquire);
(generation != 0).then_some(GrpcConnectionSnapshot {
channel_id: self.0.channel_id,
generation,
})
}
}
impl Default for GrpcConnectionState {
fn default() -> Self {
Self::new()
}
}
fn next_nonzero_id(counter: &AtomicU64) -> u64 {
loop {
let id = counter.fetch_add(1, Ordering::Relaxed);
if id != 0 {
return id;
}
}
}
pub static GLOBAL_CONN_MAP: LazyLock<RwLock<HashMap<String, Channel>>> = LazyLock::new(|| RwLock::new(HashMap::new()));
#[derive(Clone)]
struct TrackedGrpcConnection {
channel: Channel,
connection_state: GrpcConnectionState,
}
static GLOBAL_TRACKED_CONN_MAP: LazyLock<RwLock<HashMap<String, TrackedGrpcConnection>>> =
LazyLock::new(|| RwLock::new(HashMap::new()));
static GLOBAL_CONN_CACHE_WRITER: tokio::sync::Mutex<()> = tokio::sync::Mutex::const_new(());
pub static GLOBAL_ROOT_CERT: LazyLock<RwLock<Option<Vec<u8>>>> = LazyLock::new(|| RwLock::new(None));
pub static GLOBAL_MTLS_IDENTITY: LazyLock<RwLock<Option<MtlsIdentityPem>>> = LazyLock::new(|| RwLock::new(None));
pub static GLOBAL_OUTBOUND_TLS_GENERATION: LazyLock<AtomicU64> = LazyLock::new(|| AtomicU64::new(0));
@@ -231,6 +304,16 @@ pub async fn evict_connection(addr: &str) {
/// Evict a stale/dead connection from the global connection cache with an explicit log level.
pub async fn evict_connection_with_log_level(addr: &str, log_level: ConnectionEvictionLogLevel) {
let _ = remove_connection_with_log_level(addr, log_level).await;
}
/// Remove a cached connection and report whether it existed. Transport owners use this to
/// attach a rebuild reason only to real cache transitions.
pub async fn remove_connection_with_log_level(addr: &str, log_level: ConnectionEvictionLogLevel) -> bool {
// Writes are rare and serialized so the optimized tracked cache and public compatibility map
// cannot publish different channel clones for one key.
let _writer = GLOBAL_CONN_CACHE_WRITER.lock().await;
GLOBAL_TRACKED_CONN_MAP.write().await.remove(addr);
let removed = GLOBAL_CONN_MAP.write().await.remove(addr);
if removed.is_some() {
match log_level {
@@ -254,6 +337,7 @@ pub async fn evict_connection_with_log_level(addr: &str, log_level: ConnectionEv
}
}
}
removed.is_some()
}
/// Get a cached gRPC connection for the given address.
@@ -261,8 +345,37 @@ pub async fn cached_connection(addr: &str) -> Option<Channel> {
GLOBAL_CONN_MAP.read().await.get(addr).cloned()
}
/// Get a cached channel together with its shared physical-connection state, when tracked.
pub async fn cached_connection_with_state(addr: &str) -> Option<(Channel, Option<GrpcConnectionState>)> {
if let Some(cached) = GLOBAL_TRACKED_CONN_MAP.read().await.get(addr).cloned() {
return Some((cached.channel, Some(cached.connection_state)));
}
// A miss may race a low-frequency publish/evict. Serialize only this fallback, recheck the
// tracked map, then consult legacy entries injected through the compatibility API.
let _writer = GLOBAL_CONN_CACHE_WRITER.lock().await;
if let Some(cached) = GLOBAL_TRACKED_CONN_MAP.read().await.get(addr).cloned() {
return Some((cached.channel, Some(cached.connection_state)));
}
cached_connection(addr).await.map(|channel| (channel, None))
}
/// Cache a gRPC connection for the given address.
pub async fn cache_connection(addr: String, channel: Channel) {
let _writer = GLOBAL_CONN_CACHE_WRITER.lock().await;
GLOBAL_TRACKED_CONN_MAP.write().await.remove(&addr);
GLOBAL_CONN_MAP.write().await.insert(addr, channel);
}
/// Cache a gRPC channel and its physical-connection state as one coherent entry.
pub async fn cache_connection_with_state(addr: String, channel: Channel, connection_state: GrpcConnectionState) {
let _writer = GLOBAL_CONN_CACHE_WRITER.lock().await;
GLOBAL_TRACKED_CONN_MAP.write().await.insert(
addr.clone(),
TrackedGrpcConnection {
channel: channel.clone(),
connection_state,
},
);
GLOBAL_CONN_MAP.write().await.insert(addr, channel);
}
@@ -279,6 +392,8 @@ pub async fn has_cached_connection(addr: &str) -> bool {
/// Clear all cached connections. Useful for full cluster reset/recovery.
pub async fn clear_all_connections() {
let _writer = GLOBAL_CONN_CACHE_WRITER.lock().await;
GLOBAL_TRACKED_CONN_MAP.write().await.clear();
let mut map = GLOBAL_CONN_MAP.write().await;
let count = map.len();
map.clear();
@@ -363,4 +478,46 @@ mod tls_snapshot_tests {
assert!(!debug.contains("PRIVATE-KEY-MARKER"));
assert!(debug.contains("[REDACTED]"));
}
#[tokio::test]
async fn conditional_connection_removal_preserves_legacy_evict_api() {
let key = "http://conditional-eviction-test.invalid";
cache_connection(
key.to_owned(),
tonic::transport::Endpoint::from_static("http://127.0.0.1:1").connect_lazy(),
)
.await;
assert!(remove_connection_with_log_level(key, ConnectionEvictionLogLevel::Debug).await);
assert!(!remove_connection_with_log_level(key, ConnectionEvictionLogLevel::Debug).await);
let _: () = evict_connection_with_log_level(key, ConnectionEvictionLogLevel::Debug).await;
}
#[tokio::test]
async fn cached_connection_keeps_channel_and_transport_state_coherent() {
let key = "http://tracked-connection-test.invalid";
let channel = tonic::transport::Endpoint::from_static("http://127.0.0.1:1").connect_lazy();
let state = GrpcConnectionState::new();
assert!(state.snapshot().is_none());
let first = state.record_connected();
cache_connection_with_state(key.to_owned(), channel, state.clone()).await;
let (_, cached_state) = cached_connection_with_state(key).await.expect("tracked cache entry");
let cached_state = cached_state.expect("tracked cache state");
assert_eq!(cached_state.snapshot(), Some(first));
let second = state.record_connected();
assert_eq!(cached_state.snapshot(), Some(second));
assert_eq!(first.channel_id, second.channel_id);
assert_ne!(first.generation, second.generation);
cache_connection(
key.to_owned(),
tonic::transport::Endpoint::from_static("http://127.0.0.1:2").connect_lazy(),
)
.await;
let (_, replacement_state) = cached_connection_with_state(key).await.expect("legacy replacement entry");
assert!(replacement_state.is_none(), "legacy replacement must remove stale tracked state");
assert!(remove_connection_with_log_level(key, ConnectionEvictionLogLevel::Debug).await);
assert!(cached_connection_with_state(key).await.is_none());
}
}
+3 -4
View File
@@ -25,7 +25,6 @@ keywords = ["configuration", "settings", "management", "rustfs", "Minio"]
categories = ["web-programming", "development-tools", "config"]
[dependencies]
hotpath.workspace = true
const-str = { workspace = true, optional = true, features = ["std", "proc"] }
serde = { workspace = true, optional = true, features = ["derive"] }
serde_json = { workspace = true, optional = true, features = ["raw_value"] }
@@ -35,9 +34,9 @@ workspace = true
[features]
default = ["constants"]
hotpath = ["hotpath/hotpath"]
hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu"]
hotpath = []
hotpath-alloc = ["hotpath"]
hotpath-cpu = ["hotpath"]
audit = ["dep:const-str", "constants"]
constants = ["dep:const-str"]
notify = ["dep:const-str", "constants"]
+9
View File
@@ -287,6 +287,13 @@ Drive timeout profile preset:
- values must be positive; `0` or an invalid value falls back to the default, and values above `60` are clamped to `60`.
- the setting is read by the aggregating node only; it does not change the internode RPC wire contract. Any retry shares one round deadline rather than receiving a fresh timeout.
## Target queue directory
- `RUSTFS_TARGET_STORE_DIR`
- absolute root for notify and audit target queues whose `queue_dir` is the default `/opt/rustfs/events` or `/opt/rustfs/audit`; they open `<root>/events` and `<root>/audit` on this node.
- unset keeps those paths on Linux and macOS; native Windows uses `%ProgramData%\RustFS\events` and `%ProgramData%\RustFS\audit`.
- explicit `queue_dir` values are never relocated, and the persisted target configuration is unchanged. Changing the root does not migrate pending entries. See `docs/operations/target-queue-storage.md`.
## Startup filesystem boundary policy
- `RUSTFS_UNSUPPORTED_FS_POLICY` controls startup behavior when RustFS detects local endpoint filesystems that are outside the supported production boundary.
@@ -295,6 +302,8 @@ Drive timeout profile preset:
RustFS production guidance remains direct-attached local POSIX filesystems. Network-mounted filesystems (for example `nfs`, `cifs`, `smb2`, and `fuse.*`) are treated as unsupported by this startup guard. On Windows, endpoints on network drives (UNC shares such as `\\server\share` and mapped drive letters) are treated the same way, even though Windows reports the server's filesystem (usually `NTFS`) for them.
A local POSIX filesystem bind-mounted into a gVisor sandbox with directfs is a supported storage path. gVisor presents that bind mount as `9p`/`V9FS`, so the host filesystem type is not visible; the startup guard recognizes mountinfo options `trans=fd` and `directfs` and does not treat the mount as unsupported. Genuine 9p, virtio, and NFS shares remain unsupported.
## 📄 License
This project is licensed under the Apache License 2.0 - see the [LICENSE](../../LICENSE) file for details.
+2 -2
View File
@@ -35,8 +35,8 @@ pub const DEFAULT_HEALTH_OBJECT_PROGRESS_TIMEOUT_MS: u64 = 30_000;
/// Additional time beyond the configured object-lock acquisition deadline.
pub const HEALTH_OBJECT_PROGRESS_LOCK_MARGIN_MS: u64 = 5_000;
/// Timeout for cluster health readiness collectors (milliseconds).
/// This bounds expensive storage and lock quorum checks used by cluster probes.
/// End-to-end budget for cluster health readiness probes (milliseconds).
/// Includes cache/singleflight waiting, storage observations, and lock checks.
pub const ENV_HEALTH_CLUSTER_TIMEOUT_MS: &str = "RUSTFS_HEALTH_CLUSTER_TIMEOUT_MS";
pub const DEFAULT_HEALTH_CLUSTER_TIMEOUT_MS: u64 = 2000;
+152 -8
View File
@@ -50,10 +50,8 @@ const _: () = assert!(DEFAULT_ADMIN_PEER_PROBE_TIMEOUT_SECS <= MAX_ADMIN_PEER_PR
// ── Client-side internode gRPC channel tuning (P0) ──
// These mirror the server-side HTTP/2 transport tuning in `rustfs/src/server/http.rs`
// on the *client* `tonic` `Endpoint` used for internode control-plane RPCs. Prior to
// this, the client channel only set timeouts/keepalive, leaving Nagle enabled and the
// default 64KiB HTTP/2 window in place — hurting small lock-RPC latency and large
// metadata-response throughput respectively.
// on the *client* `tonic` `Endpoint` used for internode RPCs. Explicit values make
// the operator's receive-credit policy independent of changes to library defaults.
/// Disable Nagle's algorithm on internode gRPC client sockets.
///
@@ -69,8 +67,8 @@ const _: () = assert!(DEFAULT_INTERNODE_RPC_TCP_NODELAY);
/// HTTP/2 initial stream window size (bytes) for internode gRPC client channels.
///
/// The library default (64KiB) throttles larger unary responses (e.g. `ReadMultiple`,
/// `BatchReadVersion`) by the bandwidth-delay product. Set to 0 to fall back to the
/// Size this for the bandwidth-delay product and concurrent response budget of
/// unary responses such as `ReadMultiple` and `BatchReadVersion`. Set to 0 to use the
/// `tonic`/`hyper` default. Defaults to 1MiB.
pub const ENV_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE: &str = "RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE";
pub const DEFAULT_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE: u32 = 1024 * 1024;
@@ -284,6 +282,26 @@ const _: () = assert!(!DEFAULT_INTERNODE_CHANNEL_ISOLATION);
pub const ENV_INTERNODE_BULK_CHANNELS: &str = "RUSTFS_INTERNODE_BULK_CHANNELS";
pub const DEFAULT_INTERNODE_BULK_CHANNELS: usize = 2;
/// Optional receive-window overrides for isolated bulk channels. When unset, inherit the
/// corresponding `RUSTFS_INTERNODE_RPC_HTTP2_*` setting. Zero selects the library default.
pub const ENV_INTERNODE_BULK_HTTP2_STREAM_WINDOW_SIZE: &str = "RUSTFS_INTERNODE_BULK_HTTP2_STREAM_WINDOW_SIZE";
pub const ENV_INTERNODE_BULK_HTTP2_CONN_WINDOW_SIZE: &str = "RUSTFS_INTERNODE_BULK_HTTP2_CONN_WINDOW_SIZE";
/// Transport admission and pending-buffer limits for isolated bulk channels. These limits
/// apply to the tonic service; complete unary bodies are observed separately by RPC metrics.
pub const ENV_INTERNODE_BULK_CONCURRENCY_LIMIT: &str = "RUSTFS_INTERNODE_BULK_CONCURRENCY_LIMIT";
pub const DEFAULT_INTERNODE_BULK_CONCURRENCY_LIMIT: usize = 64;
pub const ENV_INTERNODE_BULK_BUFFER_SIZE: &str = "RUSTFS_INTERNODE_BULK_BUFFER_SIZE";
pub const DEFAULT_INTERNODE_BULK_BUFFER_SIZE: usize = 1024;
/// Per-peer limit held through complete bulk-unary response decoding and validation. Zero keeps
/// the compatibility behavior with no additional admission gate.
pub const ENV_INTERNODE_BULK_BODY_CONCURRENCY_LIMIT: &str = "RUSTFS_INTERNODE_BULK_BODY_CONCURRENCY_LIMIT";
pub const DEFAULT_INTERNODE_BULK_BODY_CONCURRENCY_LIMIT: usize = 0;
pub const ENV_INTERNODE_BULK_BODY_WAIT_TIMEOUT_MS: &str = "RUSTFS_INTERNODE_BULK_BODY_WAIT_TIMEOUT_MS";
pub const DEFAULT_INTERNODE_BULK_BODY_WAIT_TIMEOUT_MS: u64 = 1_000;
const _: () = assert!(DEFAULT_INTERNODE_BULK_BODY_CONCURRENCY_LIMIT == 0);
/// Profile selector for conservative internode HTTP data-plane client tuning.
pub const ENV_INTERNODE_HTTP_TUNING_PROFILE: &str = "RUSTFS_INTERNODE_HTTP_TUNING_PROFILE";
pub const DEFAULT_INTERNODE_HTTP_TUNING_PROFILE: &str = "legacy";
@@ -323,8 +341,130 @@ pub const DEFAULT_INTERNODE_DATA_TRANSPORT: &str = "tcp-http";
/// Legacy alias for "tcp-http". Both values select the TCP/HTTP transport backend.
pub const INTERNODE_DATA_TRANSPORT_TCP: &str = "tcp";
/// Internode reads whose data moves over NIXL/UCX (InfiniBand rails or RoCE by
/// LID/GID, no IP needed); requests, writes and walks stay on TCP/HTTP. Requires
/// a binary built with the `nixl` feature and `RUSTFS_NIXL_ENABLE`; until the
/// NIXL agent is up, and for any read it cannot serve, it uses TCP/HTTP.
pub const INTERNODE_DATA_TRANSPORT_NIXL: &str = "nixl";
/// Known internode transport backend names accepted by the config parser.
pub const KNOWN_INTERNODE_DATA_TRANSPORT_BACKENDS: &[&str] = &[DEFAULT_INTERNODE_DATA_TRANSPORT, INTERNODE_DATA_TRANSPORT_TCP];
pub const KNOWN_INTERNODE_DATA_TRANSPORT_BACKENDS: &[&str] = &[
DEFAULT_INTERNODE_DATA_TRANSPORT,
INTERNODE_DATA_TRANSPORT_TCP,
INTERNODE_DATA_TRANSPORT_NIXL,
];
/// Check the configured RDMA devices at startup and leave out any that is down,
/// slower than its peers, or loses its link during a short watch.
pub const ENV_RUSTFS_NIXL_DEVICE_CHECK: &str = "RUSTFS_NIXL_DEVICE_CHECK";
pub const DEFAULT_RUSTFS_NIXL_DEVICE_CHECK: bool = true;
/// Leave out a device whose link dropped this many times more since boot than
/// the quietest device of its link layer; 0 skips this test.
pub const ENV_RUSTFS_NIXL_DEVICE_MAX_LINK_DROPS: &str = "RUSTFS_NIXL_DEVICE_MAX_LINK_DROPS";
pub const DEFAULT_RUSTFS_NIXL_DEVICE_MAX_LINK_DROPS: u64 = 16;
/// How long the startup device check watches the link counters; 0 skips the watch.
pub const ENV_RUSTFS_NIXL_DEVICE_FLAP_WINDOW_MS: &str = "RUSTFS_NIXL_DEVICE_FLAP_WINDOW_MS";
pub const DEFAULT_RUSTFS_NIXL_DEVICE_FLAP_WINDOW_MS: u64 = 3_000;
/// NIXL internode reads in flight per remote shard stream.
pub const ENV_RUSTFS_NIXL_INTERNODE_DEPTH: &str = "RUSTFS_NIXL_INTERNODE_DEPTH";
pub const DEFAULT_RUSTFS_NIXL_INTERNODE_DEPTH: usize = 4;
/// Whether S3 GETs and PUTs carrying the NIXL headers are served over NIXL/UCX
/// S3-over-RDMA (requires a build with the `nixl` feature and libnixl). The
/// pipeline settings below (`RUSTFS_CUOBJECT_GET_FANOUT*`, `_INFLIGHT_WRITES`,
/// `_SHARD_ARENA_BYTES`) apply to this transport as well.
pub const ENV_RUSTFS_NIXL_ENABLE: &str = "RUSTFS_NIXL_ENABLE";
pub const DEFAULT_RUSTFS_NIXL_ENABLE: bool = false;
/// This node's NIXL agent name; defaults to `rustfs-<first local endpoint host>`.
pub const ENV_RUSTFS_NIXL_AGENT_NAME: &str = "RUSTFS_NIXL_AGENT_NAME";
/// Comma-separated UCX device names (`mlx5_9,mlx5_10`; NIXL adds the port); empty lets UCX use every device.
pub const ENV_RUSTFS_NIXL_UCX_DEVICES: &str = "RUSTFS_NIXL_UCX_DEVICES";
/// UCX progress workers of the agent.
pub const ENV_RUSTFS_NIXL_NUM_WORKERS: &str = "RUSTFS_NIXL_NUM_WORKERS";
pub const DEFAULT_RUSTFS_NIXL_NUM_WORKERS: u32 = 1;
/// Staging chunk size per transfer descriptor to a client.
pub const ENV_RUSTFS_NIXL_CHUNK_BYTES: &str = "RUSTFS_NIXL_CHUNK_BYTES";
pub const DEFAULT_RUSTFS_NIXL_CHUNK_BYTES: usize = 16 * 1024 * 1024;
/// Registered staging memory of the agent.
pub const ENV_RUSTFS_NIXL_POOL_BYTES: &str = "RUSTFS_NIXL_POOL_BYTES";
pub const DEFAULT_RUSTFS_NIXL_POOL_BYTES: usize = 1024 * 1024 * 1024;
/// Transfers in flight per node at most.
pub const ENV_RUSTFS_NIXL_MAX_INFLIGHT: &str = "RUSTFS_NIXL_MAX_INFLIGHT";
pub const DEFAULT_RUSTFS_NIXL_MAX_INFLIGHT: usize = 256;
/// A transfer that has not completed after this many seconds fails the request.
pub const ENV_RUSTFS_NIXL_TRANSFER_TIMEOUT_SECS: &str = "RUSTFS_NIXL_TRANSFER_TIMEOUT_SECS";
pub const DEFAULT_RUSTFS_NIXL_TRANSFER_TIMEOUT_SECS: u64 = 60;
/// Whether S3 GETs carrying an `x-amz-rdma-token` are served over cuObject S3-over-RDMA
/// (requires a build with the `cuobject` feature and libcuobjserver).
pub const ENV_RUSTFS_CUOBJECT_ENABLE: &str = "RUSTFS_CUOBJECT_ENABLE";
pub const DEFAULT_RUSTFS_CUOBJECT_ENABLE: bool = false;
/// Comma-separated RDMA-capable IP addresses that get a cuObject server each.
/// Defaults to the hosts of this node's own endpoints.
pub const ENV_RUSTFS_CUOBJECT_ADDRS: &str = "RUSTFS_CUOBJECT_ADDRS";
/// UDP/RDMA control port of the cuObject servers.
pub const ENV_RUSTFS_CUOBJECT_PORT: &str = "RUSTFS_CUOBJECT_PORT";
pub const DEFAULT_RUSTFS_CUOBJECT_PORT: u16 = 18515;
/// Staging chunk size per RDMA write to a client.
pub const ENV_RUSTFS_CUOBJECT_CHUNK_BYTES: &str = "RUSTFS_CUOBJECT_CHUNK_BYTES";
pub const DEFAULT_RUSTFS_CUOBJECT_CHUNK_BYTES: usize = 16 * 1024 * 1024;
/// Registered staging memory per cuObject server.
pub const ENV_RUSTFS_CUOBJECT_POOL_BYTES: &str = "RUSTFS_CUOBJECT_POOL_BYTES";
pub const DEFAULT_RUSTFS_CUOBJECT_POOL_BYTES: usize = 1024 * 1024 * 1024;
/// Dynamically connected initiators per cuObject server (concurrent client connections).
/// Staging chunks (RDMA writes) kept in flight per GET, so decoding never waits
/// for the write of the previous chunk; each costs one chunk of the pool.
pub const ENV_RUSTFS_CUOBJECT_INFLIGHT_WRITES: &str = "RUSTFS_CUOBJECT_INFLIGHT_WRITES";
pub const DEFAULT_RUSTFS_CUOBJECT_INFLIGHT_WRITES: usize = 3;
/// Staging chunks (RDMA reads from the client buffer) kept in flight per
/// RDMA PUT body, so ingest never waits on one chunk's round trip at a time;
/// each costs one chunk of the pool.
pub const ENV_RUSTFS_CUOBJECT_PUT_READ_AHEAD: &str = "RUSTFS_CUOBJECT_PUT_READ_AHEAD";
pub const DEFAULT_RUSTFS_CUOBJECT_PUT_READ_AHEAD: usize = 4;
/// Concurrent erasure-block-aligned segments one cuObject GET is split into
/// on the server (each segment decodes and RDMA-writes its own stripes), so a
/// single stream keeps several stripes in flight. 1 disables the fan-out.
pub const ENV_RUSTFS_CUOBJECT_GET_FANOUT: &str = "RUSTFS_CUOBJECT_GET_FANOUT";
pub const DEFAULT_RUSTFS_CUOBJECT_GET_FANOUT: usize = 8;
/// Smallest segment worth its own producer (rounded up to whole erasure
/// blocks): each segment pays a reader setup per part it touches, so short
/// requests stay sequential.
pub const ENV_RUSTFS_CUOBJECT_GET_FANOUT_MIN_BYTES: &str = "RUSTFS_CUOBJECT_GET_FANOUT_MIN_BYTES";
pub const DEFAULT_RUSTFS_CUOBJECT_GET_FANOUT_MIN_BYTES: u64 = 192 * 1024 * 1024;
/// Segment producers a node runs across all its cuObject GETs: each new GET
/// gets a fair share (budget / open GETs, at least 1, at most the fan-out)
/// and never waits for one, so many-stream load degrades to sequential GETs
/// instead of a reader-setup storm.
pub const ENV_RUSTFS_CUOBJECT_GET_FANOUT_BUDGET: &str = "RUSTFS_CUOBJECT_GET_FANOUT_BUDGET";
pub const DEFAULT_RUSTFS_CUOBJECT_GET_FANOUT_BUDGET: usize = 16;
/// Registered host memory the erasure decoder reads shards into so a
/// cuObject GET can RDMA-write data shards straight from those buffers
/// (zero-copy emit); 0 keeps the staging copy. Blocks are
/// `RUSTFS_ERASURE_SHARD_BYTES` (4 MiB when unset); objects with larger
/// shards fall back to heap buffers and the staging copy.
pub const ENV_RUSTFS_CUOBJECT_SHARD_ARENA_BYTES: &str = "RUSTFS_CUOBJECT_SHARD_ARENA_BYTES";
pub const DEFAULT_RUSTFS_CUOBJECT_SHARD_ARENA_BYTES: usize = 0;
pub const ENV_RUSTFS_CUOBJECT_NUM_DCIS: &str = "RUSTFS_CUOBJECT_NUM_DCIS";
pub const DEFAULT_RUSTFS_CUOBJECT_NUM_DCIS: u32 = 128;
/// Complete cuObject transfers through the library's asynchronous mode: the
/// write is submitted with a handle and the task waits on the channel's
/// completion descriptor instead of holding a blocking thread for the whole
/// transfer. `false` keeps one blocking library call per transfer.
pub const ENV_RUSTFS_CUOBJECT_ASYNC_TRANSFERS: &str = "RUSTFS_CUOBJECT_ASYNC_TRANSFERS";
pub const DEFAULT_RUSTFS_CUOBJECT_ASYNC_TRANSFERS: bool = true;
#[cfg(test)]
mod tests {
@@ -369,8 +509,12 @@ mod tests {
fn internode_channel_isolation_defaults_and_env_names() {
// The isolation-off default is asserted at compile time next to the constant definition.
assert_eq!(DEFAULT_INTERNODE_BULK_CHANNELS, 2);
assert_eq!(DEFAULT_INTERNODE_BULK_BODY_CONCURRENCY_LIMIT, 0);
assert_eq!(DEFAULT_INTERNODE_BULK_BODY_WAIT_TIMEOUT_MS, 1_000);
assert_eq!(ENV_INTERNODE_CHANNEL_ISOLATION, "RUSTFS_INTERNODE_CHANNEL_ISOLATION");
assert_eq!(ENV_INTERNODE_BULK_CHANNELS, "RUSTFS_INTERNODE_BULK_CHANNELS");
assert_eq!(ENV_INTERNODE_BULK_BODY_CONCURRENCY_LIMIT, "RUSTFS_INTERNODE_BULK_BODY_CONCURRENCY_LIMIT");
assert_eq!(ENV_INTERNODE_BULK_BODY_WAIT_TIMEOUT_MS, "RUSTFS_INTERNODE_BULK_BODY_WAIT_TIMEOUT_MS");
}
#[test]
@@ -455,6 +599,6 @@ mod tests {
assert_eq!(ENV_RUSTFS_INTERNODE_DATA_TRANSPORT, "RUSTFS_INTERNODE_DATA_TRANSPORT");
assert_eq!(DEFAULT_INTERNODE_DATA_TRANSPORT, "tcp-http");
assert_eq!(INTERNODE_DATA_TRANSPORT_TCP, "tcp");
assert_eq!(KNOWN_INTERNODE_DATA_TRANSPORT_BACKENDS, &["tcp-http", "tcp"]);
assert_eq!(KNOWN_INTERNODE_DATA_TRANSPORT_BACKENDS, &["tcp-http", "tcp", "nixl"]);
}
}
+55
View File
@@ -117,6 +117,30 @@ pub const ENV_OBJECT_GET_SKIP_BITROT_VERIFY: &str = "RUSTFS_OBJECT_GET_SKIP_BITR
/// Default: bitrot verification is enabled on GetObject reads (do not skip).
pub const DEFAULT_OBJECT_GET_SKIP_BITROT_VERIFY: bool = false;
/// Comma-separated buckets that hold a content-addressed cache (an LLM KV
/// cache) and serve PutObject, GetObject and HeadObject without distributed
/// locks or authoritative bucket-metadata reads. See the ecstore
/// `bucket::lockless_cache` module for what such a bucket gives up.
///
/// Default is empty: every bucket keeps the locked paths.
pub const ENV_LOCKLESS_CACHE_BUCKETS: &str = "RUSTFS_LOCKLESS_CACHE_BUCKETS";
/// Smallest declared body, in bytes, whose ETag MD5 is computed on a
/// blocking-pool thread while the body streams into the erasure encoder,
/// instead of inline on the request's task. The ETag and Content-MD5
/// checks are unchanged; `0` keeps every MD5 inline.
///
/// Each offloaded upload holds one blocking-pool thread until its body ends,
/// and that pool also serves disk I/O, so size the threshold against the
/// expected number of concurrent large uploads before enabling it.
pub const ENV_OBJECT_ETAG_MD5_OFFLOAD_MIN_BYTES: &str = "RUSTFS_OBJECT_ETAG_MD5_OFFLOAD_MIN_BYTES";
/// Default: disabled; every ETag MD5 is computed inline. Set
/// `RUSTFS_OBJECT_ETAG_MD5_OFFLOAD_MIN_BYTES` (for example `8388608`) to opt in.
pub const DEFAULT_OBJECT_ETAG_MD5_OFFLOAD_MIN_BYTES: u64 = 0;
const _: () = assert!(DEFAULT_OBJECT_ETAG_MD5_OFFLOAD_MIN_BYTES == 0);
/// Create independent shard commitments for new writes after the fleet is upgraded.
/// Existing protected objects and multipart uploads retain their protection.
pub const ENV_SHARD_INTEGRITY_WRITE: &str = "RUSTFS_SHARD_INTEGRITY_WRITE";
@@ -298,6 +322,37 @@ pub const ENV_OBJECT_DISK_READ_TIMEOUT: &str = "RUSTFS_OBJECT_DISK_READ_TIMEOUT"
/// Default disk read timeout in seconds.
pub const DEFAULT_OBJECT_DISK_READ_TIMEOUT: u64 = 10;
/// Environment variable for the smallest bitrot frame (hash + shard bytes)
/// that is verified on a blocking thread instead of inline on the decode task
/// (`0` disables off-thread verification).
///
/// The shard reads of one stripe complete on one task, so with large shards
/// their hashes were computed one after another on one core: at 4 MiB shards
/// a 12+4 stripe spent about 10 ms hashing serially. Frames at or above this
/// size are hashed on the blocking pool so the shards of a stripe verify in
/// parallel; small frames stay inline because a thread hop would cost more
/// than the hash.
pub const ENV_BITROT_VERIFY_OFFLOAD_MIN_BYTES: &str = "RUSTFS_BITROT_VERIFY_OFFLOAD_MIN_BYTES";
/// Default smallest off-thread bitrot frame: 2 MiB (frames of the legacy 1 MiB
/// block, and 1 MiB shards, stay inline).
pub const DEFAULT_BITROT_VERIFY_OFFLOAD_MIN_BYTES: usize = 2 * 1024 * 1024;
/// Environment variable for the erasure shard size, in bytes, of newly written
/// objects (`0`, the default, keeps the historical 1 MiB erasure block).
///
/// When set, the erasure block of a new object is `data shards x shard bytes`,
/// so every shard read and write is exactly this large and aligned: 4 MiB on a
/// 12+4 layout gives a 48 MiB block, 48x fewer stripes per object than today.
/// Existing objects keep the block recorded in their `xl.meta`. System
/// metadata under `.rustfs.sys` always uses the 1 MiB block. The value must be
/// a multiple of 4 KiB and at most `MAX_ERASURE_SHARD_BYTES`; invalid values
/// are ignored with a warning.
pub const ENV_ERASURE_SHARD_BYTES: &str = "RUSTFS_ERASURE_SHARD_BYTES";
/// Default erasure shard size override (`0` = derive from the 1 MiB block).
pub const DEFAULT_ERASURE_SHARD_BYTES: usize = 0;
/// Largest accepted erasure shard size override (64 MiB).
pub const MAX_ERASURE_SHARD_BYTES: usize = 64 * 1024 * 1024;
/// Environment variable for the per-shard erasure write stall timeout (seconds).
///
/// A single shard write (or shard-writer shutdown) that makes no forward
+15
View File
@@ -114,6 +114,14 @@ pub const ENV_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE: &str = "RUSTFS_OBJECT_FILE_
pub const ENV_OBJECT_FILE_CACHE_RECLAIM_THRESHOLD: &str = "RUSTFS_OBJECT_FILE_CACHE_RECLAIM_THRESHOLD";
pub const DEFAULT_OBJECT_FILE_CACHE_RECLAIM_WRITE_ENABLE: bool = false;
pub const DEFAULT_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE: bool = true;
/// Bytes of page-cache read-ahead a ranged file stream keeps queued in front
/// of its read position (`posix_fadvise(WILLNEED)`, Linux only; `0` disables).
/// A shard stream served to a peer reads 4 MiB chunks one after another; the
/// kernel's own read-ahead window ramps up slowly for large reads, so without
/// this each chunk waits for its own device round trips.
pub const ENV_OBJECT_STREAM_READ_AHEAD_BYTES: &str = "RUSTFS_OBJECT_STREAM_READ_AHEAD_BYTES";
pub const DEFAULT_OBJECT_STREAM_READ_AHEAD_BYTES: usize = 16 * 1024 * 1024;
pub const DEFAULT_OBJECT_FILE_CACHE_RECLAIM_THRESHOLD: usize = 4 * 1024 * 1024;
/// Threshold for small object seek support in bytes.
@@ -158,3 +166,10 @@ pub const ENV_OBJECT_DATA_CACHE_MIN_FREE_MEMORY_PERCENT: &str = "RUSTFS_OBJECT_D
pub const ENV_OBJECT_DATA_CACHE_FILL_CONCURRENCY_PER_CPU: &str = "RUSTFS_OBJECT_DATA_CACHE_FILL_CONCURRENCY_PER_CPU";
pub const ENV_OBJECT_DATA_CACHE_FILL_CONCURRENCY_MAX: &str = "RUSTFS_OBJECT_DATA_CACHE_FILL_CONCURRENCY_MAX";
pub const ENV_OBJECT_DATA_CACHE_IDENTITY_KEYS_MAX: &str = "RUSTFS_OBJECT_DATA_CACHE_IDENTITY_KEYS_MAX";
/// Fill the object data cache from the PUT body as it streams in, so the first
/// GET after a write is a hit (write-once KV-cache chunks). Only objects that
/// fit the cache entry limit and take the plain streaming PUT path (no
/// compression, no SSE) are filled; the body is kept in memory once more
/// during the PUT.
pub const ENV_OBJECT_DATA_CACHE_FILL_ON_PUT: &str = "RUSTFS_OBJECT_DATA_CACHE_FILL_ON_PUT";
pub const DEFAULT_OBJECT_DATA_CACHE_FILL_ON_PUT: bool = false;
+6
View File
@@ -144,6 +144,12 @@ pub const POSTGRES_QUEUE_LIMIT: &str = "queue_limit";
/// Applies to both notify and audit target queue stores.
pub const ENV_TARGET_STORE_COMPRESS: &str = "RUSTFS_TARGET_STORE_COMPRESS";
/// Environment variable that relocates the root of the default notify and audit queue directories.
///
/// It applies only to targets whose `queue_dir` is the logical default (`EVENT_DEFAULT_DIR` or
/// `AUDIT_DEFAULT_DIR`); their queues move to `<root>/events` and `<root>/audit` on this node.
pub const ENV_TARGET_STORE_DIR: &str = "RUSTFS_TARGET_STORE_DIR";
/// Queue-store compression is enabled by default to reduce disk footprint.
pub const DEFAULT_TARGET_STORE_COMPRESS: bool = true;
+10 -1
View File
@@ -55,6 +55,9 @@ pub const ENV_OBS_PUT_STAGE_METRICS_ENABLED: &str = "RUSTFS_OBS_PUT_STAGE_METRIC
pub const ENV_OBS_GET_STAGE_LOCAL_SUMMARY_ENABLED: &str = "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_ENABLED";
pub const ENV_OBS_GET_STAGE_LOCAL_SUMMARY_SAMPLE_RATE: &str = "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_SAMPLE_RATE";
pub const ENV_OBS_GET_STAGE_LOCAL_SUMMARY_INTERVAL_SECS: &str = "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_INTERVAL_SECS";
/// Path of the sampled GetObject stage summary file (defaults to `/var/log/rustfs/get-stage-summary.log`).
pub const ENV_OBS_GET_STAGE_LOCAL_SUMMARY_FILE: &str = "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_FILE";
pub const DEFAULT_OBS_GET_STAGE_LOCAL_SUMMARY_FILE: &str = "/var/log/rustfs/get-stage-summary.log";
pub const ENV_OBS_LOGGER_LEVEL: &str = "RUSTFS_OBS_LOGGER_LEVEL";
pub const ENV_OBS_LOG_STDOUT_ENABLED: &str = "RUSTFS_OBS_LOG_STDOUT_ENABLED";
@@ -91,7 +94,12 @@ pub const ENV_OBS_LOG_MATCH_MODE: &str = "RUSTFS_OBS_LOG_MATCH_MODE";
/// Default values for log cleanup
pub const DEFAULT_OBS_LOG_MAX_TOTAL_SIZE_BYTES: u64 = 2 * 1024 * 1024 * 1024; // 2 GiB
pub const DEFAULT_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES: u64 = 0; // No single file limit
pub const DEFAULT_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES: u64 = 0; // No per-file trim by the cleaner
/// Size at which the active log file is rolled when
/// `RUSTFS_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES` is unset. Without a size roll
/// the active file is bounded only by the rotation period, and the cleaner
/// never touches the active file. An explicit `0` keeps size rolling off.
pub const DEFAULT_OBS_LOG_ACTIVE_FILE_ROLL_SIZE_BYTES: u64 = 256 * 1024 * 1024; // 256 MiB
pub const DEFAULT_OBS_LOG_COMPRESS_OLD_FILES: bool = true;
pub const DEFAULT_OBS_LOG_GZIP_COMPRESSION_LEVEL: u32 = 6;
pub const DEFAULT_OBS_LOG_COMPRESSION_ALGORITHM_GZIP: &str = "gzip";
@@ -173,6 +181,7 @@ mod tests {
ENV_OBS_GET_STAGE_LOCAL_SUMMARY_INTERVAL_SECS,
"RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_INTERVAL_SECS"
);
assert_eq!(ENV_OBS_GET_STAGE_LOCAL_SUMMARY_FILE, "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_FILE");
// Test log cleanup related env keys
assert_eq!(ENV_OBS_LOG_MAX_TOTAL_SIZE_BYTES, "RUSTFS_OBS_LOG_MAX_TOTAL_SIZE_BYTES");
assert_eq!(ENV_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES, "RUSTFS_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES");
+3 -4
View File
@@ -26,12 +26,11 @@ categories = ["web-programming", "development-tools", "data-structures", "securi
[features]
default = []
hotpath = ["hotpath/hotpath"]
hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu"]
hotpath = []
hotpath-alloc = ["hotpath"]
hotpath-cpu = ["hotpath"]
[dependencies]
hotpath.workspace = true
base64-simd = { workspace = true }
hmac = { workspace = true }
rand = { workspace = true, features = ["serde"] }
+3 -4
View File
@@ -29,7 +29,6 @@ documentation = "https://docs.rs/rustfs-crypto/latest/rustfs_crypto/"
workspace = true
[dependencies]
hotpath.workspace = true
aes-gcm = { workspace = true, optional = true, features = ["rand_core"] }
argon2 = { workspace = true, optional = true }
chacha20poly1305 = { workspace = true, optional = true }
@@ -50,9 +49,9 @@ time = { workspace = true, features = ["parsing", "formatting", "macros", "serde
[features]
default = ["crypto", "fips"]
hotpath = ["hotpath/hotpath"]
hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu"]
hotpath = []
hotpath-alloc = ["hotpath"]
hotpath-cpu = ["hotpath"]
fips = []
crypto = [
"dep:aes-gcm",
+2 -5
View File
@@ -18,9 +18,6 @@ use crate::Error;
use crate::jwt::Claims;
pub fn decode(token: &str, token_secret: &[u8]) -> Result<TokenData<Claims>, Error> {
Ok(jsonwebtoken::decode(
token,
&DecodingKey::from_secret(token_secret),
&Validation::new(Algorithm::HS512),
)?)
let key = DecodingKey::from_secret(token_secret)?;
Ok(jsonwebtoken::decode(token, &key, &Validation::new(Algorithm::HS512))?)
}
+2 -5
View File
@@ -18,9 +18,6 @@ use crate::Error;
use crate::jwt::Claims;
pub fn encode(token_secret: &[u8], claims: &Claims) -> Result<String, Error> {
Ok(jsonwebtoken::encode(
&Header::new(Algorithm::HS512),
claims,
&EncodingKey::from_secret(token_secret),
)?)
let key = EncodingKey::from_secret(token_secret)?;
Ok(jsonwebtoken::encode(&Header::new(Algorithm::HS512), claims, &key)?)
}
+26
View File
@@ -0,0 +1,26 @@
[package]
name = "rustfs-cuobject"
edition.workspace = true
license.workspace = true
repository.workspace = true
rust-version.workspace = true
version.workspace = true
homepage.workspace = true
description = "RustFS staging pool and transfers for NVIDIA cuObject S3-over-RDMA (GPUDirect object loading)"
keywords = ["rdma", "cuobject", "gpudirect", "s3"]
categories = ["network-programming"]
documentation = "https://docs.rs/rustfs-cuobject/latest/rustfs_cuobject/"
[lints]
workspace = true
[dependencies]
bytes = { workspace = true, optional = true }
cuobject = { workspace = true, default-features = false, optional = true }
parking_lot = { workspace = true, optional = true }
thiserror = { workspace = true, optional = true }
tokio = { workspace = true, features = ["net", "rt-multi-thread", "sync", "time"], optional = true }
[features]
# Compiles the cuObject server bindings (needs the NVIDIA cuObject SDK).
server = ["dep:bytes", "dep:cuobject", "cuobject/server", "dep:parking_lot", "dep:thiserror", "dep:tokio"]
+775
View File
@@ -0,0 +1,775 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Server side of NVIDIA cuObject S3-over-RDMA.
//!
//! A client that supports the protocol (NVIDIA cuObject, NIXL's object plugin,
//! …) sends an ordinary S3 GET with an `x-amz-rdma-token` header describing a
//! registered client buffer (host or GPU memory). Instead of streaming the
//! body over HTTP, the server RDMA-writes the object bytes into that buffer
//! through NVIDIA's `libcuobjserver` and answers with an empty body plus
//! `x-amz-rdma-reply` and `x-amz-rdma-bytes-transferred`.
//!
//! The library bindings come from the [`cuobject`] crate. This crate owns
//! RustFS's pool of registered, page-aligned host staging buffers: the regular
//! read path fills a staging chunk and
//! [`CuObjectServer::write_to_client_owned`] pushes it.
//!
//! Everything below needs the `server` feature, which compiles the
//! NVIDIA cuObject server bindings; without it the crate is empty, so a
//! workspace build does not need the cuObject SDK.
#![cfg(feature = "server")]
use bytes::Bytes;
use cuobject::{Channel, Completion, RegisteredBuffer, Server, ServerOptions, SgBuffer, SgEntry, Source};
use std::collections::VecDeque;
use std::net::IpAddr;
use std::num::NonZeroUsize;
use std::sync::Arc;
use std::time::Duration;
use thiserror::Error;
use tokio::io::Interest;
use tokio::io::unix::AsyncFd;
use tokio::sync::Semaphore;
/// Header carrying the client's RDMA descriptor on a request.
pub const RDMA_TOKEN_HEADER: &str = cuobject::TOKEN_HEADER;
/// Header carrying the transfer status on the response (HTTP-style code).
pub const RDMA_REPLY_HEADER: &str = cuobject::REPLY_HEADER;
/// Header carrying the payload byte count moved over RDMA.
pub const RDMA_BYTES_TRANSFERRED_HEADER: &str = cuobject::BYTES_TRANSFERRED_HEADER;
/// Reply code for a declined transfer (the client then errors or falls back).
pub const RDMA_REPLY_NOT_IMPLEMENTED: u16 = cuobject::REPLY_NOT_IMPLEMENTED;
/// Largest single `handleGetObject` the library accepts.
pub const MAX_RDMA_OP_BYTES: usize = cuobject::MAX_TRANSFER_BYTES;
/// Default RDMA control port of the cuObject server.
pub const DEFAULT_PORT: u16 = cuobject::DEFAULT_PORT;
/// Most ranges one scatter-gather transfer may carry (the library's `max_sge`).
pub const MAX_SG_RANGES: usize = cuobject::DEFAULT_MAX_SG_ENTRIES;
/// Version of the cuObject server headers the bindings were built against, as
/// (major, minor).
pub fn library_version() -> (u32, u32) {
cuobject::header_version()
}
#[derive(Debug, Error)]
pub enum CuObjectError {
#[error("cuobject {op} failed: {source}")]
Os {
op: &'static str,
#[source]
source: std::io::Error,
},
#[error("cuobject rdma token is malformed: {0}")]
Token(String),
#[error("cuobject rdma token buffer ({buffer} bytes) is smaller than the response ({needed} bytes)")]
TokenTooSmall { buffer: u64, needed: u64 },
#[error("cuobject rejected the request: {0}")]
Invalid(&'static str),
#[error("cuobject transfer failed with verbs status {verbs_status}: {source}")]
Transfer {
verbs_status: i32,
#[source]
source: std::io::Error,
},
#[error("cuobject transfer moved {moved} of {expected} bytes")]
ShortTransfer { moved: u64, expected: u64 },
#[error("cuobject staging pool exhausted")]
PoolExhausted,
#[error("cuobject library offers no completion descriptor for channel {0}")]
AsyncUnavailable(u16),
#[error("cuobject read source failed: {0}")]
Source(#[source] std::io::Error),
}
impl From<cuobject::Error> for CuObjectError {
fn from(error: cuobject::Error) -> Self {
match error {
cuobject::Error::Library { op, source } => Self::Os { op, source },
cuobject::Error::Transfer { verbs_status, source } => Self::Transfer { verbs_status, source },
cuobject::Error::InvalidArgument(reason) => Self::Invalid(reason),
cuobject::Error::InvalidToken(reason) => Self::Token(reason.to_owned()),
cuobject::Error::TokenTooSmall { buffer, needed } => Self::TokenTooSmall { buffer, needed },
cuobject::Error::AsyncUnavailable(channel) => Self::AsyncUnavailable(channel),
other => Self::Os {
op: "cuobject",
source: std::io::Error::other(other),
},
}
}
}
/// Parsed `x-amz-rdma-token`.
///
/// Wire format (cuObject RDMA descriptor protocol): colon-separated hex
/// fields `addr:size:rkey:lid:dctn:gid_present:gid`; only the leading
/// address and size are interpreted here, the library consumes the rest.
#[derive(Debug, Clone)]
pub struct RdmaToken {
inner: cuobject::RdmaToken,
/// Start of the client buffer (a copy of the parsed field).
pub remote_addr: u64,
/// Length of the client buffer (a copy of the parsed field).
pub buffer_len: u64,
}
impl RdmaToken {
pub fn parse(raw: &str) -> Result<Self, CuObjectError> {
let inner = cuobject::RdmaToken::parse(raw)?;
Ok(Self {
remote_addr: inner.remote_addr(),
buffer_len: inner.buffer_len(),
inner,
})
}
pub fn as_str(&self) -> &str {
self.inner.as_str()
}
}
/// Tuning for one server instance.
#[derive(Debug, Clone)]
pub struct CuObjectConfig {
pub port: u16,
pub num_dcis: i32,
pub cq_depth: u32,
/// Size of one staging buffer; one RDMA write moves at most this much.
pub chunk_bytes: usize,
/// Total registered staging memory per server instance.
pub pool_bytes: usize,
/// Complete transfers through the library's asynchronous mode (submit,
/// then wait on the channel's completion descriptor) instead of one
/// blocking library call per transfer.
pub async_transfers: bool,
}
impl Default for CuObjectConfig {
fn default() -> Self {
Self {
port: DEFAULT_PORT,
num_dcis: 128,
cq_depth: 640,
chunk_bytes: 64 * 1024 * 1024,
pool_bytes: 1024 * 1024 * 1024,
async_transfers: true,
}
}
}
struct Staging {
buffer: RegisteredBuffer<Arc<Server>>,
free: parking_lot::Mutex<VecDeque<usize>>,
permits: Semaphore,
chunk_bytes: usize,
}
/// One cuObject server bound to the RDMA device that owns an IP address.
pub struct CuObjectServer {
server: Arc<Server>,
staging: Staging,
addr: IpAddr,
port: u16,
/// One permit per library channel (DC initiator): a transfer waits here
/// instead of failing with `EBUSY` when every channel is in use.
channels: Arc<Semaphore>,
async_transfers: bool,
}
impl std::fmt::Debug for CuObjectServer {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("CuObjectServer")
.field("addr", &self.addr)
.field("port", &self.port)
.field("chunk_bytes", &self.staging.chunk_bytes)
.finish()
}
}
/// Several byte ranges of one [`RegisteredArena`], sent in order as one
/// RDMA operation (at most [`MAX_SG_RANGES`]).
pub struct ArenaRanges {
pub arena: Arc<RegisteredArena>,
/// `(local_offset, len)` pairs, in stream order.
pub ranges: Vec<(usize, usize)>,
}
impl ArenaRanges {
pub fn total_len(&self) -> usize {
self.ranges.iter().map(|(_, len)| *len).sum()
}
}
/// A byte range of a [`RegisteredArena`] to send from.
pub struct ArenaRange {
pub arena: Arc<RegisteredArena>,
pub local_offset: usize,
pub len: usize,
}
/// Host memory registered with one server, owned by the caller (see
/// [`CuObjectServer::register_arena`]).
pub struct RegisteredArena {
buffer: RegisteredBuffer<Arc<Server>>,
server: Arc<CuObjectServer>,
id: u64,
}
impl RegisteredArena {
/// Length of the registered memory.
pub fn len(&self) -> usize {
self.buffer.len()
}
pub fn is_empty(&self) -> bool {
self.buffer.is_empty()
}
/// Process-unique id of this registration.
pub fn id(&self) -> u64 {
self.id
}
/// The server this memory is registered with.
pub fn server(&self) -> &Arc<CuObjectServer> {
&self.server
}
/// Raw pointer to the start of the registered memory. The memory lives as
/// long as this arena; callers hand out disjoint ranges of it and never
/// touch a range while a transfer sends it.
pub fn as_mut_ptr(&self) -> *mut u8 {
self.buffer.as_mut_ptr()
}
}
impl std::fmt::Debug for RegisteredArena {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("RegisteredArena")
.field("id", &self.id)
.field("len", &self.len())
.finish()
}
}
/// A reserved staging chunk; returned to the pool on drop.
///
/// The chunk's index is unique while it is reserved, so its range of the
/// staging buffer has one owner. Transfers take the chunk by value, so no
/// Rust borrow of the range overlaps an RDMA operation on it.
pub struct StagingChunk {
server: Arc<CuObjectServer>,
index: usize,
len: usize,
}
impl StagingChunk {
pub fn capacity(&self) -> usize {
self.server.staging.chunk_bytes
}
/// Bytes currently staged.
pub fn len(&self) -> usize {
self.len
}
pub fn is_empty(&self) -> bool {
self.len == 0
}
/// Copies `data` at `self.len()` and advances the staged length.
pub fn append(&mut self, data: &[u8]) -> Result<(), CuObjectError> {
if self.len + data.len() > self.capacity() {
return Err(CuObjectError::Invalid("staging append exceeds chunk"));
}
self.spare_mut()?[..data.len()].copy_from_slice(data);
self.len += data.len();
Ok(())
}
pub fn clear(&mut self) {
self.len = 0;
}
pub fn is_full(&self) -> bool {
self.len >= self.capacity()
}
/// The bytes staged so far.
#[allow(unsafe_code)]
pub fn staged(&self) -> Result<&[u8], CuObjectError> {
// SAFETY: the range lies in this chunk (see the type docs); `&self`
// excludes `spare_mut`, and transfers own the chunk they move.
Ok(unsafe { self.server.staging.buffer.slice(self.start(), self.len)? })
}
/// The unfilled remainder of the chunk, for a reader to fill directly.
#[allow(unsafe_code)]
pub fn spare_mut(&mut self) -> Result<&mut [u8], CuObjectError> {
let len = self.capacity() - self.len;
// SAFETY: the range lies in this chunk (see the type docs) and
// `&mut self` excludes every other borrow of it.
Ok(unsafe { self.server.staging.buffer.slice_mut(self.start() + self.len, len)? })
}
/// Marks `n` more bytes of the spare area as staged.
pub fn advance(&mut self, n: usize) -> Result<(), CuObjectError> {
if self.len + n > self.capacity() {
return Err(CuObjectError::Invalid("staging advance exceeds chunk"));
}
self.len += n;
Ok(())
}
fn start(&self) -> usize {
self.index * self.capacity()
}
}
impl Drop for StagingChunk {
fn drop(&mut self) {
self.server.staging.free.lock().push_back(self.index);
self.server.staging.permits.add_permits(1);
}
}
/// The local side of one GET, owned by the transfer until the DMA finished.
enum Local {
/// A range of the staging buffer, inside a chunk the transfer also owns.
Staging { offset: usize, len: usize },
/// A range of a caller-owned arena.
Arena {
arena: Arc<RegisteredArena>,
offset: usize,
len: usize,
},
/// Several ranges of a caller-owned arena, sent as one operation.
ArenaList {
arena: Arc<RegisteredArena>,
entries: Vec<SgEntry>,
},
}
impl Local {
fn len(&self) -> usize {
match self {
Self::Staging { len, .. } | Self::Arena { len, .. } => *len,
Self::ArenaList { entries, .. } => entries.iter().map(|entry| entry.size).sum(),
}
}
}
/// Why an asynchronous GET did not complete.
enum AsyncFailure {
/// The library has no completion descriptor; nothing was submitted.
Unavailable,
/// The GET failed, or was never submitted; nothing is in flight.
Failed(CuObjectError),
/// Waiting failed after submission: the NIC may still read the memory.
InFlight(CuObjectError),
}
impl From<cuobject::Error> for AsyncFailure {
fn from(error: cuobject::Error) -> Self {
Self::Failed(error.into())
}
}
/// The handle of the one GET in flight on an asynchronous channel.
const ASYNC_HANDLE: NonZeroUsize = NonZeroUsize::MIN;
impl CuObjectServer {
/// Creates the server and its registered staging pool.
pub fn new(addr: IpAddr, config: &CuObjectConfig) -> Result<Arc<Self>, CuObjectError> {
if config.chunk_bytes == 0 || config.chunk_bytes > MAX_RDMA_OP_BYTES || config.pool_bytes < config.chunk_bytes {
return Err(CuObjectError::Invalid("invalid staging configuration"));
}
let mut options = ServerOptions::default();
// A non-positive count keeps the library default.
options.num_dcis = u32::try_from(config.num_dcis).unwrap_or(0);
options.cq_depth = config.cq_depth;
let server = Arc::new(Server::new(addr, config.port, &options)?);
let chunks = config.pool_bytes / config.chunk_bytes;
let buffer = RegisteredBuffer::new(Arc::clone(&server), chunks * config.chunk_bytes)?;
Ok(Arc::new(Self {
channels: Arc::new(Semaphore::new(usize::try_from(config.num_dcis).unwrap_or(1).max(1))),
async_transfers: config.async_transfers,
server,
staging: Staging {
buffer,
free: parking_lot::Mutex::new((0..chunks).collect()),
permits: Semaphore::new(chunks),
chunk_bytes: config.chunk_bytes,
},
addr,
port: config.port,
}))
}
pub fn addr(&self) -> IpAddr {
self.addr
}
pub fn port(&self) -> u16 {
self.port
}
pub fn chunk_bytes(&self) -> usize {
self.staging.chunk_bytes
}
/// Reserves one staging chunk, waiting up to `timeout` for a free one.
pub async fn reserve(self: &Arc<Self>, timeout: Duration) -> Result<StagingChunk, CuObjectError> {
let permit = tokio::time::timeout(timeout, self.staging.permits.acquire())
.await
.map_err(|_| CuObjectError::PoolExhausted)?
.map_err(|_| CuObjectError::PoolExhausted)?;
permit.forget();
let index = self.staging.free.lock().pop_front().ok_or(CuObjectError::PoolExhausted)?;
Ok(StagingChunk {
server: Arc::clone(self),
index,
len: 0,
})
}
/// Reserves one staging chunk without waiting; `None` when the pool is empty.
pub fn try_reserve(self: &Arc<Self>) -> Option<StagingChunk> {
let permit = self.staging.permits.try_acquire().ok()?;
permit.forget();
let index = self.staging.free.lock().pop_front()?;
Some(StagingChunk {
server: Arc::clone(self),
index,
len: 0,
})
}
/// Staging chunks currently free in the pool (a hint; another reservation may win).
pub fn available_chunks(&self) -> usize {
self.staging.permits.available_permits()
}
/// Registers `len` bytes of zero-filled, page-aligned host memory with this
/// server for callers that manage the memory themselves (a shard arena the
/// decoder reads into and the server RDMA-writes from without staging).
pub fn register_arena(self: &Arc<Self>, len: usize) -> Result<Arc<RegisteredArena>, CuObjectError> {
let buffer = RegisteredBuffer::new(Arc::clone(&self.server), len)?;
static NEXT_ID: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(1);
Ok(Arc::new(RegisteredArena {
buffer,
server: Arc::clone(self),
id: NEXT_ID.fetch_add(1, std::sync::atomic::Ordering::Relaxed),
}))
}
/// RDMA-writes the staged bytes of `chunk` to `token.remote_addr +
/// remote_offset`. The transfer owns the chunk until the DMA finished, so
/// cancelling the awaiting future can never release the staging slot under
/// an in-flight write. Returns the chunk with the byte count for reuse.
pub async fn write_to_client_owned(
self: &Arc<Self>,
key: &str,
token: &RdmaToken,
remote_offset: u64,
chunk: StagingChunk,
) -> Result<(u64, StagingChunk), CuObjectError> {
if chunk.is_empty() {
return Ok((0, chunk));
}
let local = Local::Staging {
offset: chunk.start(),
len: chunk.len(),
};
self.get_owned(key, token, remote_offset, local, chunk).await
}
/// RDMA-writes `source.len` bytes at `source.local_offset` of
/// `source.arena` to `token.remote_addr + remote_offset`, keeping `keep`
/// alive until the DMA finished (the caller passes whatever owns the arena
/// bytes). The arena must have been registered with this server.
pub async fn write_to_client_from_arena_owned<T: Send + 'static>(
self: &Arc<Self>,
key: &str,
token: &RdmaToken,
remote_offset: u64,
source: ArenaRange,
keep: T,
) -> Result<(u64, T), CuObjectError> {
if source.len == 0 {
return Ok((0, keep));
}
let local = Local::Arena {
arena: source.arena,
offset: source.local_offset,
len: source.len,
};
self.get_owned(key, token, remote_offset, local, keep).await
}
/// RDMA-writes all of `source.ranges` (in order, as one operation) to
/// `token.remote_addr + remote_offset`, keeping `keep` alive until the DMA
/// finished. The arena must have been registered with this server.
pub async fn write_to_client_sg_owned<T: Send + 'static>(
self: &Arc<Self>,
key: &str,
token: &RdmaToken,
remote_offset: u64,
source: ArenaRanges,
keep: T,
) -> Result<(u64, T), CuObjectError> {
if source.ranges.len() > MAX_SG_RANGES {
return Err(CuObjectError::Invalid("too many scatter-gather ranges"));
}
if source.total_len() == 0 {
return Ok((0, keep));
}
let entries = source
.ranges
.iter()
.map(|&(offset, size)| SgEntry {
offset: offset as u64,
size,
})
.collect();
let local = Local::ArenaList {
arena: source.arena,
entries,
};
self.get_owned(key, token, remote_offset, local, keep).await
}
/// RDMA-reads `len` bytes from `token.remote_addr + remote_offset` into the
/// (empty) `chunk`, owning it for the whole DMA like
/// [`Self::write_to_client_owned`]. On success the chunk holds `len` staged
/// bytes.
pub async fn read_from_client_owned(
self: &Arc<Self>,
key: &str,
token: &RdmaToken,
remote_offset: u64,
len: usize,
mut chunk: StagingChunk,
) -> Result<StagingChunk, CuObjectError> {
chunk.clear();
if len == 0 {
return Ok(chunk);
}
if len > chunk.capacity() {
return Err(CuObjectError::Invalid("read exceeds the staging chunk"));
}
let server = Arc::clone(self);
let key = key.to_owned();
let token = token.inner.clone();
let channel_permit = self.channel_permit().await?;
tokio::task::spawn_blocking(move || {
let _channel_permit = channel_permit;
let mut channel = Channel::lease(Arc::clone(&server.server))?;
let moved = channel.put(&key, &token, remote_offset, &server.staging.buffer, chunk.start(), len)?;
check_moved(moved as u64, len as u64)?;
chunk.advance(len)?;
Ok(chunk)
})
.await
.map_err(|error| join_error("read task", error))?
}
async fn channel_permit(&self) -> Result<tokio::sync::OwnedSemaphorePermit, CuObjectError> {
Arc::clone(&self.channels)
.acquire_owned()
.await
.map_err(|_| CuObjectError::Invalid("server closed"))
}
/// RDMA-writes `local` to the client and hands `keep` back once the DMA
/// finished. The transfer runs in a task of its own that owns `local`,
/// `keep` and the channel, so dropping this future never releases memory
/// the NIC may still read.
async fn get_owned<T: Send + 'static>(
self: &Arc<Self>,
key: &str,
token: &RdmaToken,
remote_offset: u64,
local: Local,
keep: T,
) -> Result<(u64, T), CuObjectError> {
let expected = local.len() as u64;
let server = Arc::clone(self);
let key = key.to_owned();
let token = token.inner.clone();
let channel_permit = self.channel_permit().await?;
tokio::spawn(async move {
if server.async_transfers {
match server.get_async(&key, &token, remote_offset, &local).await {
Ok(()) => {
drop(channel_permit);
return Ok((expected, keep));
}
Err(AsyncFailure::Unavailable) => {}
Err(AsyncFailure::Failed(error)) => return Err(error),
Err(AsyncFailure::InFlight(error)) => {
// The NIC may still read this memory: never release it.
std::mem::forget(local);
std::mem::forget(keep);
return Err(error);
}
}
}
tokio::task::spawn_blocking(move || {
let _channel_permit = channel_permit;
let mut channel = Channel::lease(Arc::clone(&server.server))?;
let mut list = None;
let source = server.source(&local, &mut list)?;
let moved = channel.get(&key, &token, remote_offset, source)?;
check_moved(moved as u64, expected)?;
// `keep` (the chunk or the arena owner) is released only here.
Ok((expected, keep))
})
.await
.map_err(|error| join_error("write task", error))?
})
.await
.map_err(|error| join_error("write task", error))?
}
/// Moves `local` with the library's asynchronous mode: the GET is
/// submitted on a channel of its own and the task waits on that channel's
/// completion descriptor instead of holding a blocking thread.
// SAFETY: the allowance covers only this function; its two unsafe calls
// each state the invariant they rely on.
#[allow(unsafe_code)]
async fn get_async(
&self,
key: &str,
token: &cuobject::RdmaToken,
remote_offset: u64,
local: &Local,
) -> Result<(), AsyncFailure> {
let mut channel = Channel::lease(Arc::clone(&self.server))?;
let fd = match channel.completion_fd() {
Ok(fd) => fd,
Err(cuobject::Error::AsyncUnavailable(_)) => return Err(AsyncFailure::Unavailable),
Err(error) => return Err(error.into()),
};
// SAFETY: the library owns the descriptor and keeps it open while the
// server lives; `fd` holds the server, and nothing here closes or
// replaces the descriptor while it is registered.
let completions = unsafe { AsyncFd::register_with_interest(fd, Interest::READABLE) }.map_err(|error| {
AsyncFailure::Failed(CuObjectError::Os {
op: "completion fd",
source: error.into_parts().1,
})
})?;
let mut list = None;
let source = self.source(local, &mut list).map_err(AsyncFailure::Failed)?;
// SAFETY: the memory behind `source` belongs to `local`, owned by the
// calling task together with the transfer's `keep`; staging chunks and
// arena ranges given to a transfer are not borrowed by anyone else.
// This function returns only after the completion carrying the handle
// was drained, except through `InFlight`, where the channel and the
// list are leaked here and the caller leaks `local` and `keep`.
unsafe { channel.get_async(key, token, remote_offset, source, ASYNC_HANDLE) }?;
let mut events = [Completion::default(); 8];
loop {
let mut ready = match completions.readable().await {
Ok(ready) => ready,
Err(source) => {
std::mem::forget(list);
std::mem::forget(channel);
return Err(AsyncFailure::InFlight(CuObjectError::Os {
op: "completion wait",
source,
}));
}
};
loop {
let landed = match channel.process_completions(&mut events) {
Ok(landed) => landed,
Err(error) => {
std::mem::forget(list);
std::mem::forget(channel);
return Err(AsyncFailure::InFlight(error.into()));
}
};
if let Some(done) = events[..landed].iter().find(|event| event.handle() == ASYNC_HANDLE.get()) {
// A completion carries no byte count: success moved the whole range.
return done
.result()
.map_err(|source| AsyncFailure::Failed(CuObjectError::Transfer { verbs_status: 0, source }));
}
if landed < events.len() {
break;
}
}
ready.clear_ready();
}
}
/// The library view of `local`; a scatter-gather list is built into `list`.
fn source<'a: 'l, 'l>(
&'a self,
local: &'a Local,
list: &'l mut Option<SgBuffer<'a, Arc<Server>>>,
) -> Result<Source<'l, Arc<Server>>, CuObjectError> {
Ok(match local {
Local::Staging { offset, len } => Source::Range {
buffer: &self.staging.buffer,
offset: *offset,
len: *len,
},
Local::Arena { arena, offset, len } => Source::Range {
buffer: &arena.buffer,
offset: *offset,
len: *len,
},
Local::ArenaList { arena, entries } => Source::List(list.insert(arena.buffer.sg_buffer(entries)?)),
})
}
}
fn check_moved(moved: u64, expected: u64) -> Result<u64, CuObjectError> {
if moved != expected {
return Err(CuObjectError::ShortTransfer { moved, expected });
}
Ok(moved)
}
fn join_error(op: &'static str, error: tokio::task::JoinError) -> CuObjectError {
CuObjectError::Os {
op,
source: std::io::Error::other(error),
}
}
/// Copies owned bytes into staging; convenience for callers holding `Bytes`.
pub fn fill_chunk(chunk: &mut StagingChunk, data: &Bytes) -> Result<(), CuObjectError> {
chunk.append(data)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn token_parses_leading_fields_and_rejects_garbage() {
let token =
RdmaToken::parse("0000f52eddc00000:00100000:001824d2:0000:002873:1:00000000000000000000ffff0a0d5410").expect("token");
assert_eq!(token.remote_addr, 0x0000_f52e_ddc0_0000);
assert_eq!(token.buffer_len, 0x0010_0000);
assert!(matches!(RdmaToken::parse("nothex:00100000:1:2:3:4:5"), Err(CuObjectError::Token(_))));
assert!(RdmaToken::parse("0000f52eddc00000:00000000:1:2:3:4:5").is_err());
assert!(RdmaToken::parse("0000f52eddc00000:00100000").is_err());
}
}
@@ -171,6 +171,70 @@ impl DataUsageBucketObservation {
}
impl DataUsageInfo {
/// Hide values that cannot be bound to the current storage-owned identity.
///
/// `incarnations` contains canonical non-nil identities read from storage;
/// a missing entry means identity was unavailable. Neither legacy values
/// without identity nor metadata read failures establish confirmed zero.
/// Changed responses are display-only and cannot authorize quota or writes.
/// If retained counters overflow, all values remain unknown.
/// Returns the number of canonical values removed.
pub fn retain_current_bucket_usage_for_admin(&mut self, incarnations: &HashMap<String, String>) -> usize {
let original_count = self.buckets_usage.len();
let original_observation_count = self.usage_snapshot_bucket_observations.len();
let matches_current = |bucket: &String, usage: &BucketUsageInfo| {
incarnations.get(bucket).is_some_and(|current| {
!current.is_empty()
&& current != "00000000-0000-0000-0000-000000000000"
&& usage.bucket_incarnation.as_ref() == Some(current)
})
};
self.buckets_usage.retain(|bucket, usage| matches_current(bucket, usage));
self.usage_snapshot_bucket_observations.retain(|bucket, observation| {
matches_current(bucket, &observation.usage)
&& observation
.sources
.iter()
.all(|source| incarnations.get(bucket) == Some(&source.bucket_incarnation))
});
if original_count == self.buckets_usage.len()
&& original_observation_count == self.usage_snapshot_bucket_observations.len()
{
return 0;
}
let totals = self.buckets_usage.values().try_fold([0_u64; 4], |total, usage| {
Some([
total[0].checked_add(usage.size)?,
total[1].checked_add(usage.objects_count)?,
total[2].checked_add(usage.versions_count)?,
total[3].checked_add(usage.delete_markers_count)?,
])
});
let [size, objects, versions, markers] = totals.unwrap_or_else(|| {
self.buckets_usage.clear();
self.usage_snapshot_bucket_observations.clear();
[0; 4]
});
self.bucket_sizes = self
.buckets_usage
.iter()
.map(|(bucket, usage)| (bucket.clone(), usage.size))
.collect();
self.buckets_count = u64::try_from(self.buckets_usage.len()).unwrap_or(u64::MAX);
self.objects_total_size = size;
self.objects_total_count = objects;
self.versions_total_count = versions;
self.delete_markers_total_count = markers;
self.replication_info.clear();
self.tier_stats = None;
self.unknown_tier_stats = None;
self.usage_snapshot_complete = false;
self.usage_snapshot_partial = true;
self.usage_snapshot_converged = Some(false);
self.usage_snapshot_admin_projection = true;
original_count.saturating_sub(self.buckets_usage.len())
}
/// Overlay validated companions on an owned admin response only.
///
/// `incarnations` must come from current storage metadata, not the companion.
@@ -321,3 +321,110 @@ fn provenance_and_companions_change_candidate_bytes_and_round_trip_as_maps() {
let restored: DataUsageInfo = rmp_serde::from_slice(&named).expect("read named map with additive fields");
assert_eq!(restored, canonical);
}
#[test]
fn current_identity_filter_preserves_an_unchanged_complete_snapshot() {
let mut candidate = snapshot();
let original = candidate.clone();
assert_eq!(candidate.retain_current_bucket_usage_for_admin(&incarnations()), 0);
assert_eq!(candidate, original);
assert!(candidate.is_complete_bucket_usage_snapshot());
}
#[test]
fn current_identity_filter_hides_unbound_values_without_confirming_zero() {
for identity in [
None,
Some(""),
Some("corrupt"),
Some("00000000-0000-0000-0000-000000000000"),
Some("22222222-2222-4222-8222-222222222222"),
] {
let mut candidate = snapshot();
candidate.usage_snapshot_bucket_observations.clear();
candidate
.buckets_usage
.get_mut("bucket")
.expect("canonical")
.bucket_incarnation = identity.map(str::to_string);
assert_eq!(
candidate.retain_current_bucket_usage_for_admin(&incarnations()),
1,
"identity {identity:?}"
);
assert!(!candidate.buckets_usage.contains_key("bucket"));
assert!(!candidate.bucket_sizes.contains_key("bucket"));
assert_eq!(candidate.usage_snapshot_converged, Some(false));
assert!(candidate.usage_snapshot_admin_projection);
assert!(!candidate.is_complete_bucket_usage_snapshot());
assert!(!candidate.is_valid_partial_snapshot());
}
for current in ["", "00000000-0000-0000-0000-000000000000"] {
let mut candidate = snapshot();
candidate
.buckets_usage
.get_mut("bucket")
.expect("canonical")
.bucket_incarnation = Some(current.to_string());
assert_eq!(
candidate.retain_current_bucket_usage_for_admin(&HashMap::from([("bucket".to_string(), current.to_string())])),
1
);
assert!(candidate.usage_snapshot_admin_projection);
}
let mut candidate = snapshot();
assert_eq!(candidate.retain_current_bucket_usage_for_admin(&HashMap::new()), 1);
assert!(candidate.usage_snapshot_bucket_observations.is_empty());
assert_eq!(candidate.usage_snapshot_converged, Some(false));
}
#[test]
fn current_identity_filter_keeps_healthy_buckets_and_recalculates_partial_totals() {
let mut candidate = snapshot();
candidate.buckets_usage.insert("cold".to_string(), usage(7));
candidate.buckets_count = 2;
candidate.bucket_sizes.insert("cold".to_string(), 7);
candidate.calculate_totals();
candidate.replication_info.insert("target".to_string(), Default::default());
candidate.tier_stats = Some(Default::default());
candidate.unknown_tier_stats = Some(Default::default());
assert_eq!(
candidate.retain_current_bucket_usage_for_admin(&HashMap::from([("cold".to_string(), INCARNATION.to_string())])),
1
);
assert_eq!(candidate.buckets_usage.len(), 1);
assert_eq!(candidate.buckets_usage["cold"].size, 7);
assert_eq!(candidate.bucket_sizes, HashMap::from([("cold".to_string(), 7)]));
assert_eq!(
(
candidate.buckets_count,
candidate.objects_total_size,
candidate.objects_total_count,
candidate.versions_total_count
),
(1, 7, 7, 7)
);
assert!(candidate.replication_info.is_empty());
assert!(candidate.tier_stats.is_none());
assert!(candidate.unknown_tier_stats.is_none());
assert!(candidate.usage_snapshot_admin_projection);
assert_eq!(candidate.usage_snapshot_converged, Some(false));
}
#[test]
fn current_identity_filter_overflow_leaves_totals_unknown() {
let mut candidate = snapshot();
candidate.buckets_usage.insert("large".to_string(), usage(u64::MAX));
candidate.buckets_usage.insert("small".to_string(), usage(1));
candidate.buckets_count = 3;
let current = HashMap::from([
("large".to_string(), INCARNATION.to_string()),
("small".to_string(), INCARNATION.to_string()),
]);
assert_eq!(candidate.retain_current_bucket_usage_for_admin(&current), 3);
assert!(candidate.buckets_usage.is_empty());
assert!(candidate.bucket_sizes.is_empty());
assert_eq!(candidate.usage_snapshot_converged, Some(false));
assert!(candidate.usage_snapshot_admin_projection);
assert!(!candidate.is_complete_bucket_usage_snapshot());
}
+8 -8
View File
@@ -99,10 +99,8 @@ rustfs-filemeta.workspace = true
bytes = { workspace = true, features = ["serde"] }
aws-sdk-s3 = { workspace = true, default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
aws-sdk-sts = { workspace = true, default-features = false, features = ["default-https-client", "rt-tokio"] }
aws-config = { workspace = true }
aws-smithy-http-client = { workspace = true, default-features = false, features = ["rustls-aws-lc"] }
aws-smithy-types.workspace = true
async-compression = { workspace = true, features = ["tokio", "bzip2", "lz4", "xz"] }
async-trait = { workspace = true }
flate2.workspace = true
http.workspace = true
@@ -129,24 +127,26 @@ rand = { workspace = true, features = ["serde"] }
chrono = { workspace = true, features = ["serde"] }
hex-simd = { workspace = true }
md-5 = { workspace = true }
opentelemetry-proto = { workspace = true }
prost.workspace = true
sha2 = { workspace = true }
rustfs-tokio-tar = { workspace = true }
s3s = { workspace = true, features = ["minio"] }
zstd.workspace = true
time = { workspace = true, features = ["parsing", "formatting", "macros", "serde"] }
suppaftp = { workspace = true, features = ["rustls-aws-lc-rs", "tokio-rustls-aws-lc-rs"] }
rcgen.workspace = true
local-ip-address.workspace = true
anyhow.workspace = true
rustls = { workspace = true, default-features = false, features = ["aws-lc-rs", "logging", "tls12", "prefer-post-quantum", "std"] }
russh = { workspace = true, features = ["serde"] }
russh-sftp = { workspace = true }
zip.workspace = true
clap = { workspace = true, features = ["derive", "env"] }
[dev-dependencies]
async-compression = { workspace = true, features = ["tokio", "bzip2", "lz4", "xz"] }
aws-config = { workspace = true }
opentelemetry-proto = { workspace = true }
russh = { workspace = true, features = ["serde"] }
russh-sftp = { workspace = true }
suppaftp = { workspace = true, features = ["rustls-aws-lc-rs", "tokio-rustls-aws-lc-rs"] }
rustfs-tokio-tar = { workspace = true }
zip.workspace = true
same-file.workspace = true
aws-smithy-http-client = { workspace = true, default-features = false, features = ["test-util"] }
rustfs-heal.workspace = true

Some files were not shown because too many files have changed in this diff Show More