Since #9059 the shared test archives and debug binaries are named
with the run attempt, and the archive identity binds the attempt. A
"Re-run failed jobs" attempt does not rerun the successful producer,
so its consumers look for <name>-2, fail with "Artifact not found",
and every flaky test forces a full rerun.
Give those artifacts one name per run, overwrite them when a producer
does rerun, and bind the archive identity to the run instead of the
attempt. Source, tree, lock, toolchain and environment stay bound, and
an archive from another run is still rejected.
The planned package list is passed to cargo as bare -p names. The zstd
compat crate is named zstd so it can patch crates.io, and with zstd
0.14.3 also locked, cargo rejects -p zstd as ambiguous, failing
Workspace Lint and Build on main since #9059.
Qualify such names with the local version from Cargo.lock, and fail
loudly if even that cannot disambiguate.
Sync main 84e24e767 into gateway/integration. Ports main-side additions to
the integration boundaries: the tier deletion diagnostics admin handlers move
to the owned admin request/response types, and the new lifecycle rule checks
in ecstore use the persisted lifecycle types instead of s3s DTOs. Keeps the
gateway git source in deny.toml while dropping the smithy-rs pin main retired,
and refreshes the test-count baseline for the merged tree.
Sync main 9601a7c4b into gateway/integration. Ports main-side additions to the
owned admin boundary (plugin delegation/attempt handlers, notification target
edit tests), drops a new ecstore s3s::dto reference in favour of the persisted
lifecycle filter, and refreshes the e2e selection digests and test-count
baseline for the merged tree.
* test(decommission): cover held target lock and partial-copy resume
Port two tests from the closed lock-recovery branch onto the deferral
that defers any typed namespace lock timeout on a version copy.
- A version commits, then every attempt at the next one times out on
the target lock: the entry is deferred as a lock-timeout deferral, no
failed item is counted, both source versions stay, and the resumed
entry moves both versions with their exact bytes.
- A real write lock held on the target key makes target selection
return a typed object-lock timeout, the entry defers instead of
failing, and once the lock is released selection recovers and the
resumed entry completes the move.
* chore(ci): shrink the error(format!) ratchet baseline for rebalance
The rebalance control and meta modules on main now have fewer
Error::other(format!) call sites than the baseline records, which
fails the shrink-only ratchet in Quick Checks. Record the smaller
counts.
Report selected integer offsets and widths with strict member and payload validation. Cross-check compiler fixtures with bpftool and independent C offsetof/sizeof results.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): add memory acceptance runner and budget probes
Add owned S3 workload phases with process, endpoint, cgroup and binary
identity checks for scanner memory and scanner-disabled request bursts.
Keep scale, restart, heap-attribution and overall-acceptance limits explicit.
Characterize healthy and seeded coverage-gap cache retention through the
real folder walker and production cache codec without changing runtime code.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): bind acceptance capacity to effective storage
Validate the serving process's complete local disk inventory before load
writes and during runtime sampling. Resolve each disk inside the owned
storage root, reject nested mounts and filesystem mismatches, and bind
storage observations to phase identity for container and native runs.
Cover the unused-mount bypass, CLI-overridden environment, native paths,
symlink escapes, named volumes and stable multi-disk inventory ordering.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): fence status observations before workload writes
Close each status API window with a second storage-bound process sample
before Phase entry can authorize bucket creation or PUT requests.
Add a fail-before regression for a process change during the initial status
request, asserting no ownership or workload writes occur.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(tools): add read-only BTF prototype inspection
Expose bounded declared prototype headers through a standard-library CLI, preserve explicit void and failure semantics, and wire local contracts plus an independent compiler/bpftool reference into CI.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(tools): retain native reference failure diagnostics
Clear checkout credentials and capture bounded native stderr alongside stdout, preserving exact exit status and owned child cleanup. Keep the first reference failure unresolved rather than guessing its missing diagnostics.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* ci(tools): select a concrete bpftool file parser
The hosted PATH wrapper can exist without tools for its Azure kernel. Select an executable ELF by availability/version before comparing the fixed fixture, installing generic tools only when needed. Data comparison errors remain fatal and never trigger tool switching.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Source: main 4e48c390e7
Target: gateway/integration 1000860cb8
Conflicts resolved, keeping both sides:
- crates/ecstore/src/disk/{fs,local}.rs: main's batched rollback writes,
unified blocking write path and relocated open_file, with integration's
native publication records restored at each publication point and the
publication_history field kept beside main's multipart proof authority.
- crates/ecstore/src/set_disk/ops/multipart.rs: union of the imports.
- rustfs/src/storage/rpc/node_service.rs: main's MRF lifecycle reader
capability branch; the heal capability keeps integration's restore
copy-back probe.
- rustfs/src/admin/handlers/{audit,event,plugins_instances}.rs: main's
target_read inventory snapshot on integration's owned request boundary;
the new event.rs inventory test now builds owned requests.
- scripts/check_s3s_footprint.sh: integration's baseline-file version.
- .config/e2e-{full,smoke}-selection.txt: digests recomputed over the
union of both heads' CI listings (full 700 darwin / 708 linux, smoke 232).
Also migrated main's new rebalance delete-all expiry test to the persisted
lifecycle types integration uses in ecstore, and raised the test-count
baseline to the merged counts (every decrease comes from main).
* refactor(admin): complete the owned admin request boundary
Every type implementing the owned admin `Operation` now serves the legacy
router through one blanket impl that converts the s3s request and response
at that single edge, so handlers register with `AdminOperation(&Handler {})`
and need no adapter statics; `LegacyAdminAdapter` is removed.
`AdminRequest` gains the accessors handlers read today (`remote_addr`,
`client_info`, `request_context`, `req_info`, `req_info_mut`,
`install_req_info`, `envelope_mut`, `take_body`), the request helpers gain
envelope forms (`authenticate_envelope_credentials`,
`object_store_from_envelope`, the table-catalog `*_from_envelope` helpers,
`KmsAdminAudit::from_envelope`, `AccountAuditContext::from_envelope`,
`CallerIdentity::resolve_envelope`), `read_compatible_admin_body` accepts
either body type, and the owned `Body` gains `store_all_limited` and
`Default`. `admin::operation::test_support::TestRequest` builds owned
requests for handler tests. Unused accessors carry a per-method dead-code
expectation that each migrating handler batch removes.
No handler behavior changes; the remaining migrations follow in batches.
* chore(admin): add the T1.12 admin handler codemod
The script rewrites an admin handler file from the s3s request and response
types to the owned boundary (imports, signatures, request field and helper
calls, body access, response construction, and test request literals) and
leaves a TODO comment at every site it cannot rewrite mechanically. It is
idempotent and is removed once T1.12 completes.
* refactor(admin): migrate replication and KMS handlers to owned requests (#8951)
Move the T1.12 batch 2 admin handlers (site_replication, kms_keys, event,
replication, diagnostics, on_demand_migration, audit, account, kms_rekey,
kms_key_lifecycle, object_data_cache, tls_debug, integrity) from the s3s
request/response types to the owned AdminRequest/AdminResponse boundary.
Handler logic, authorization order, error codes and response bytes are
unchanged; the request/response plumbing now goes through the envelope
accessors and helpers introduced by the owned boundary.
The shared response helpers (utils::json_response, utils::empty_response,
handlers::admin_json_response, target_descriptor::build_json_response,
profile::authorize_profile_request) and the site replication repair
executor still answer with the s3s types, so the migrated files carry
owned-boundary forms of those helpers and convert the repair response at
its call edge. The streaming console-log body and the two devnull drains
move from the s3s byte stream to the owned Body stream/frame API.
Drop the dead_code expectations for the owned accessors and envelope
helpers this batch starts using, and lower the s3s footprint baseline to
the measured 99 files.
Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).
* refactor(admin): add owned forms of the shared admin response helpers
Handler batches were copying `json_response`, `empty_response`,
`admin_json_response`, and `build_json_response` into each migrated file
because the shared helpers still return the s3s response. Provide owned
forms next to each legacy helper (`owned_json_response`,
`owned_empty_response`, `owned_admin_json_response`,
`owned_build_json_response`, `authorize_profile_envelope`) with parity
tests against the legacy shape, derive `Debug` for `AdminResponse` so
tests can use `expect_err`, and fix three codemod rules: strip owned
string conversions in test credentials, annotate non-literal URI parses,
and leave a rebound request parameter immutable.
* refactor(admin): use the shared owned response helpers in batch 2 (#8954)
Replace the per-file owned-boundary copies of json_response,
empty_response, admin_json_response, build_json_response and
authorize_profile_request in the batch 2 admin handlers with the shared
owned_* helpers and authorize_profile_envelope that the boundary branch
now provides, and restore expect_err in the handler tests now that
AdminResponse implements Debug; the `.err().expect(..)` stand-ins were
failing the base branch's clippy step (clippy::err_expect under
-D warnings). The site replication repair call edge
keeps its conversion because the repair executor still answers with the
s3s response type.
Drop the dead_code expectations on the shared owned helpers this batch
now uses in non-test code.
Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).
* fix(admin): preserve legacy body limit error responses
* refactor(admin): move policy, ILM and batch 3 handlers to owned requests (#8956)
Batch 3 of T1.12 (rustfs/backlog#2788, parent rustfs/backlog#2734) moves
14 admin handler files to AdminRequest/AdminResponse and the owned
Operation trait: ilm_transition, policies, idp_compat, config_admin,
plugins_instances, profile_admin, group, heal, object_zip_download,
kms_key_metadata, durability, inspect_archive, sts and
gateway_key_inventory. Handler logic, authorization order, error codes
and response bytes are unchanged; test request literals use
test_support::TestRequest.
The handlers use the shared owned helpers (owned_json_response,
owned_admin_json_response, owned_build_json_response,
authorize_profile_envelope). The dead_code expectations on the
AdminRequest accessors and RequestEnvelope::uri_mut they now exercise
are removed. The legacy target_descriptor::build_json_response keeps
only its equivalence test caller and carries the matching expectation
until it retires.
object_zip_download builds the s3s request the S3 gate still reads from
the owned request facts; heal authenticates through
authenticate_envelope_credentials; the trace, inspect-archive and zip
bodies use Body::from_stream. extract_plugin_instance_filters takes the
URI so extensions.rs keeps calling it, and the two idp_compat
delegations stay on the legacy trait until their delegates migrate.
The s3s footprint baseline drops from 99 to 90 files.
* test(admin): anchor the replication diff body contract on the call
The source-contract test looked for `let body = read_compatible_admin_body(`
on one line; rustfmt wrapped that binding after the handler moved to the
owned request boundary, so the marker was no longer found. Anchor on the
call itself so formatting cannot change what the contract checks.
* refactor(admin): migrate table catalog and batch 1 to owned requests (#8955)
* fix(admin): preserve catalog body limit response bytes
* refactor(listing): keep paginated listing on ECStore only
`Sets` and `SetDisks` carried their own copies of `inner_list_objects_v2`,
`list_objects_generic`, `inner_list_object_versions`, `list_path` and
`list_merged`, plus a third copy of the task that converts merged raw entries
into the walk result stream. Only `ECStore` can answer a paginated listing,
since a page merges every set of every pool; the copies drifted (the `Sets`
copy still rejected `//` prefixes that #8605 made literal on `ECStore`) and
were reachable only from tests and one decommission helper.
Remove the `ListOperations` impls for `Sets` and `SetDisks` together with the
duplicated pagination code. `SetDisks::walk` stays as an inherent method for
set-scoped callers and now honors `versions_sort` like the store-wide walk;
`SetDisks::list_path_result` stays for the recursive delete lister. The walk
entry conversion moves into one `spawn_walk_entry_forwarder` shared by both
walks. The decommission receipt lister walks its pool's sets instead of
paginating through the removed `Sets` listing.
Tests that exercised the removed layers now exercise `ECStore` for pagination
or `SetDisks::walk` for per-pool version inspection. The architecture and
logging guard rows that pinned the removed `SetDisks`/`Sets` listing entry
points are dropped, and the error-other ratchet baseline shrinks by one.
* fix(listing): exclude synthetic directories from receipt walks
* refactor(listing): remove env-gated ListObjects index provider
Delete the experimental, default-off key-only / verified-page /
metadata-fast index provider from `ECStore::list_objects_generic` together
with everything only it drove: the `RUSTFS_LIST_OBJECTS_INDEX_*`,
`RUSTFS_LIST_OBJECTS_NAMESPACE_JOURNAL_*` and
`RUSTFS_LIST_OBJECTS_METADATA_FAST_*` readers, the persistent key-only
index and its on-disk format, the namespace mutation journal (state,
chaos injector, system/local backends, degraded set, global lock), the
index lifecycle/health/source-selection types, the verified-index
candidate pager, the `list-chaos` cargo feature, the
`rustfs_s3_list_objects_index_*` metrics and the archived bench script.
`list_objects_generic` now always walks. The write-path hooks
`observe_list_objects_mutation(s)` are gone; their callers bump the scanner
namespace generation directly through `observe_scanner_namespace_mutations`,
which is the sub-step the old hook performed before taking the journal
lock, so the generation observed by the scanner, the node RPC service and
the data-movement lease checks is unchanged while PUT/DELETE no longer take
a global tokio RwLock or read environment variables.
`ListSourceMode::{metadata_authority, requires_live_metadata_verification,
can_satisfy_strong_listing}` and `ListMetadataAuthority` were only reachable
from the provider and are removed as well; the enum, its variants, the
cursor encoding and `ListContinuationV2` are left untouched.
* docs(listing): define index retirement upgrade prerequisites
* fix(ci): retire the removed listing index test requirement
* refactor(listing): keep paginated listing on ECStore only
`Sets` and `SetDisks` carried their own copies of `inner_list_objects_v2`,
`list_objects_generic`, `inner_list_object_versions`, `list_path` and
`list_merged`, plus a third copy of the task that converts merged raw entries
into the walk result stream. Only `ECStore` can answer a paginated listing,
since a page merges every set of every pool; the copies drifted (the `Sets`
copy still rejected `//` prefixes that #8605 made literal on `ECStore`) and
were reachable only from tests and one decommission helper.
Remove the `ListOperations` impls for `Sets` and `SetDisks` together with the
duplicated pagination code. `SetDisks::walk` stays as an inherent method for
set-scoped callers and now honors `versions_sort` like the store-wide walk;
`SetDisks::list_path_result` stays for the recursive delete lister. The walk
entry conversion moves into one `spawn_walk_entry_forwarder` shared by both
walks. The decommission receipt lister walks its pool's sets instead of
paginating through the removed `Sets` listing.
Tests that exercised the removed layers now exercise `ECStore` for pagination
or `SetDisks::walk` for per-pool version inspection. The architecture and
logging guard rows that pinned the removed `SetDisks`/`Sets` listing entry
points are dropped, and the error-other ratchet baseline shrinks by one.
* fix(listing): exclude synthetic directories from receipt walks
* fix(listing): stop abandoned walks and filter directory entries
Source: main 0f24511535
Target: gateway/integration 8cf6187ff4
Resolve the e2e-full selection digest conflict by recomputing both
platform digests over the merged nextest listing (685 darwin / 693
linux): main adds two rpc_fault_proxy unit tests, integration keeps its
dual-stack differential and anonymous-access additions. Raise the rustfs
test-count baseline row to the merged count (5932).
* fix(scripts): collect scanner host telemetry on macOS
Use platform-supported iostat arguments and expose optional collector
failures without interrupting scanner-status collection.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(ecstore): reuse validated metadata read path buffers
Share the checked bucket/object tuple across metadata read consumers and borrow clean normalization inputs without skipping filesystem validation. Keep Windows reconstruction and the second bucket check intact.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(scripts): finalize scanner telemetry for failed runs
Initialize host telemetry artifacts before external captures so reused output directories cannot retain prior successes. Share normal and error-path finalization, reap collectors on request failure, and preserve the primary exit status even if status recording fails.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): distinguish unreadable recovery inventories
Treat only the exact system-prefix read-quorum result as incomplete during bounded polling. Preserve strict final reads and the shutdown/drain assertions, record failed observations, and cover interrupted empty streaks and unrelated errors.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* perf(ecstore): avoid head-of-line blocking in delete recovery
Recover independent keys through a bounded rolling window while preserving per-key hint generations, admission budgets, cancellation retry, and the existing transaction decision engine.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(ecstore): bound durable delete preparation and bootstrap recovery
Prepare independent Linux delete transactions with bounded concurrency while retaining per-key result order and serial fallback for overlapping, rollback, and legacy requests. Preserve startup discovery until bucket metadata is initialized, and add release performance and process-crash acceptance runners.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): verify clean exits in recovery acceptance
Reject forced or OOM exits when measuring clean startup and verify original ETags across mixed-layout crash recovery and a second restart.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): distinguish prepared and decided batch crashes
Observe real preparation and receipt states separately, retain non-vacuous acknowledged-delete sentinels, and require read-only physical convergence before S3 reads can trigger cleanup.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): move final delete recovery options
Move the options value into the batch-ordering regression's final case
instead of cloning it. Keep the earlier clone needed for reuse.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): replay delete receipts after interrupted proof GC
Retire only the exact empty transaction leaf before journal activation. Preserve occupied directories and prefix children, and cover partial cross-disk proof GC without a fresh delete. Tighten crash evidence to distinguish directory-only remnants from retained content.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): purge every explicit delete-marker version
Keep explicit marker removals on the canonical version-delete path so
same-key batches are not truncated and pending replication does not retain
source markers. Preserve marker identity and timestamps in the response.
Add real four-disk regressions for mixed version cleanup, duplicate marker
IDs, pending replication, and null/UUID marker removal.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(filemeta): index explicit delete-marker batches
Reuse the batch index and final compaction for ordinary marker removals.
Keep small marker-only histories on the scalar path and preserve fallback
for free versions and complex mutations.
Add scalar-equivalence, boundary, corruption and deterministic scan-count
coverage, a real-disk mixed-history regression, and reproducible marker
benchmark cases. Preserve existing benchmark defaults while honoring CLI
sampling overrides.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(filemeta): remove redundant marker fixture clone
Move the original metadata into its final test fixture after the last
borrow, avoiding the redundant clone reported by strict Clippy checks.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(storage): validate explicit marker deletion and recovery
Exercise UUID/null and duplicate explicit marker removals over real HTTP
with retries disabled, including immediate four-node visibility and exact
retained payloads. Cover all-drive PREPARED/APPLIED/DECISION process
termination, rollback/commit recovery, and old-ID retries across a second
restart without deleting a fresh marker.
Use flat sampling and fixed per-case fixture batches in the metadata
benchmark. Update profile selection digests for the added regressions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(heal): synchronize crash fixture startup and flush readiness
Give disk setup and the durable flush boundary separate readiness gates,
keeping each phase bounded to ten seconds. Preserve the real SIGKILL and
restart assertions while reporting early child exits and retaining output.
Reap fixture children on assertion failure and cover early-exit diagnostics.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(e2e): use the resumed decommission completion budget
* test(heal): exercise checkpoints written by rc.5
* test(heal): route upgrade fixtures through the storage facade
* fix(deps): update ctutils to address constant-time advisory
* test(lifecycle): gate recovery I/O while advancing test time
* feat(scanner): configure startup recovery backoff cap
* test(scanner): compare recovery retry count without narrowing
* docs(scanner): document recovery backoff cap and expose it in Helm
List scanner.recovery_backoff_cap in the runtime controls table, drop the
fixed ten-minute wording from the follower recovery section, and map
config.rustfs.scanner.recovery_backoff_cap_secs to
RUSTFS_SCANNER_RECOVERY_BACKOFF_CAP_SECS in the chart.
* fix(s3): report payload SHA256 mismatches accurately
Distinguish signed payload SHA256 failures from MD5 and additional
checksum failures at the shared API error boundary. Preserve the
payload error through archive extraction and use typed s3s errors.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* refactor(s3): classify body failures in one bounded pass
Share typed body-error classification across direct and storage I/O.
Retain marker precedence while replacing repeated source-chain scans,
and map storage-wrapped short HashReader bodies to IncompleteBody.
Cover nested priorities, real short reads, and one traversal budget.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(s3): route upload stream errors through the app facade
Keep the real stream-error fixtures behind the existing test-only S3
boundary without increasing the direct s3s import footprint.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(s3): preserve typed body failures across archive reads
Route TAR entry failures through the shared archive error adapter and
preserve every classified body error. Keep generic archive parsing as
InvalidArgument without formatting discarded transport diagnostics.
Use the existing timeout marker for PUT and Snowball body guards. Add
regressions for real stream verification after sniffing and timeout
responses, and tighten the reduced S3 macro footprint baseline.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
The scheduled sync has stopped on conflicts since 2026-10-19. Resolved by keeping both sides: main's restructured delete-rollback path in disk/local.rs with the integration-only native-publication record, main's per-instance region with the rustfs-s3-types Region, main's new lifecycle, rebalance and admin code ported to the persisted lifecycle types and rustfs-s3-types errors, both sides' new scanner tests, and the union of the access_consumer imports. The e2e smoke, full (darwin) and repl-nightly selections and the test-count baseline are regenerated from the merged tree.
* refactor(protocols): move error types to rustfs-s3-types by codemod
T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and
the error items of protocols move off s3s by scripts/codemods, run with
--hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it
downcasts the backend error, whose runtime type is still s3s::S3Error.
- rustfs/src/protocols/client.rs: the two webdav session methods now
return rustfs_s3_types::S3Result and convert the s3s error with From.
- codemod: a child module that calls s3_error! through `use super::*`
gets its own `use rustfs_s3_types::s3_error;` once its parent imports
that macro by name; otherwise it is listed as GLOB.
- s3-types: a table test pins code, status and the s3s error the legacy
edge renders for every variant a moved call site names.
- s3_error_lines 1585 -> 1584 (a lifecycle comment reworded).
* refactor(admin): move error types to rustfs-s3-types (WIP, no build)
T1.2 batch 4 (rustfs/backlog#2743), work in progress: rustfs check
still fails with 16 errors (listed in the PR body).
- codemod: --facade PATH treats a crate-local re-export of s3s's error
items like s3s for the four names (9 new self-tests, 9 mutants red).
- codemod output for rustfs/src/admin and rustfs/src/site_replication
(--facade crate::storage_api::site_replication::s3); 6 glob children
get an explicit rustfs_s3_types::s3_error import.
- admin/router.rs: the Operation trait returns rustfs_s3_types::S3Result.
- error.rs: From<ApiError> builds the rustfs-s3-types error; the s3s
conversion is derived from it.
* fix(admin): convert error types at the s3s boundary
Batch 4 of T1.2 left the admin and site_replication modules returning
rustfs_s3_types::S3Error while their callers and callees on the storage
and auth side still speak s3s::S3Error. Convert at each edge with the
existing From impls in rustfs-s3-types (compat_s3s.rs) instead of
changing any code, message or status:
- admin/auth.rs, admin/handlers/object_zip_download.rs,
admin/handlers/table_catalog/mod.rs: map the s3s result returned by
check_key_valid / authorize_request / authorize_internal_object_request
into the admin error type with .map_err(Into::into).
- storage/ecfs.rs: the s3s-facing side maps the admin site_replication
gate results the other way.
- admin/handlers/pools.rs: ApiError.code stays s3s::S3ErrorCode, so the
rustfs code is converted with .into() when the struct is built, in
production and in the two tests that build or compare it.
- admin/handlers/heal.rs, admin/runtime_sources.rs: the router returns
s3s errors (S3Route impl); the expected rustfs code is converted with
.into() before the comparison.
- admin/handlers/table_catalog/mod.rs: iceberg_rest_error takes
&'static str so the Cow<'static, str> custom code borrows it; every
caller already passes an ICEBERG_ERROR_* constant or a &'static str.
- admin/handlers/quota.rs: drop a now same-type .into() after s3_error!.
- storage_api.rs, site_replication/mod.rs: drop re-exports and an import
that no longer have users (clippy -D warnings).
cargo check -p rustfs --all-targets and cargo clippy -p rustfs
--all-targets -- -D warnings are clean; the s3s footprint baseline is
unchanged at files=172, s3_error_lines=303, ecstore_files=28.
Refs rustfs/backlog#2743, rustfs/backlog#2734.
* fix(decommission): defer source cleanup quorum failures
A quorum failure while deleting the source of an already copied entry
was fatal and marked the whole pool failed, although the copied source
is still in place and an uncommitted delete transaction is settled by
delete recovery. Carry the cleanup error typed and resolve a quorum
failure as Deferred: the entry is replayed at the end of the round, and
a deferral that persists takes the existing capacity-blocked pause path
instead of a terminal failure. Other cleanup errors and quorum errors
from other stages stay terminal.
Add set-level regressions for parent and child keys (`a/b`, `a/b/c`):
parent cleanup leaves every child file untouched, and parent cleanup
reaches quorum while the child is cleaned up or rewritten concurrently.
* test(ecstore): keep source cleanup errors typed in prefix cleanup tests
Return SourceCleanupError from the test helper instead of formatting it
into Error::other, and shrink the error(format!) ratchet baseline for
core/pools.rs, which lost one call site in the cleanup failure change.
* fix(decommission): retry cleanup quorum failures and count replays once
Retry a source cleanup quorum failure in place, up to three attempts
with the existing source-cleanup backoff, before deferring the entry.
Carry the versions an entry already counted on its deferral, so the
end-of-round replay does not add a fully counted entry to
items_decommissioned and bytes_done again.
A deferral that persists pauses the pool through the existing deferred
drain path. When source cleanup caused it, persist a reason led by
"source cleanup blocked" without the carrier error's free-drive wording,
and report waitingReason "source_cleanup" instead of "capacity".
Add a source_cleanup test fault stage and a routine test that drives a
cleanup quorum fault through deferral and replay to completion.
* fix(decommission): wait seconds between cleanup quorum retries
Give the in-place source cleanup quorum retry a fixed floor of one and
then two seconds, with the existing jitter on top, so it can ride out a
drive reconnect or health-check blip instead of retrying within a few
hundred milliseconds. The wait stays bounded and cancel-safe.
Find a source cleanup pause reason through stage wrappers, as the
capacity-blocked check does, and add a routine test where cleanup keeps
losing quorum: the decommission pauses with a source cleanup reason, the
bucket stays queued and the source stays in place.
Document the additive waitingReason value source_cleanup and its
compatibility with older nodes reading the same pool.bin.
* fix(decommission): never retry a lost source cleanup fence
The cleanup retry classifier used is_quorum_error(), which also matches
NamespaceLockQuorumUnavailable. The delete commit fence reports a lost
namespace lock or mutation fence with that variant, so a lost fence was
retried in place and only stopped later at the pre-cleanup fence check.
A lost fence means this worker no longer owns the key; it must stop
without retrying. Retry and defer only drive quorum failures:
ErasureReadQuorum, ErasureWriteQuorum, InsufficientReadQuorum and
InsufficientWriteQuorum.
---------
Co-authored-by: Hauser <housemecn@gmail.com>
* fix(decommission): run the leader-side start to completion
A forwarded decommission start awaits `ECStore::decommission` inline in
the RPC handler. When the forwarding node's RPC times out, the stream is
reset and the handler future is dropped wherever it is. Dropped after
`pool.bin` records the pool as active but before the peers reload and
the worker is spawned, the start left the pool active with no worker and
peers on the old metadata, and a retry was rejected as already active.
An admin client that disconnects can do the same.
Run the start in its own task and let the caller await only its result.
The start then always reaches its end: it fails before persisting
anything, or persists the pool, has the peers reload and spawns the
local workers, rolling back if the reload fails. A retry still cannot
start twice: it waits on the start gate held by the running start and
either reports the busy gate or is rejected because the pool is active.
`decommission` now takes `self: &Arc<Self>` and spawns workers on that
store instead of looking up the global object store, which is the same
store in production and makes the path testable; the unused
`require_decommission_store` helper is removed.
Refs rustfs/backlog#2873
* fix(decommission): detach the leader-side start from its caller
9ac4c10de carried only the reproduction (the testable entry point, the
peer-reload pause point and the proof test). This adds the fix it
describes: `ECStore::decommission` spawns the start as its own task and
awaits only its result, so a caller that stops waiting, such as a
forwarded start whose internode RPC timed out, can no longer drop the
start between persisting the pool as active and having the peers reload
and the local worker start.
Also prove a retry while the first start runs cannot start the pool
twice, update the operator and architecture docs for the new behavior,
and shrink the error(format!) ratchet baseline for the removed helper.
Refs rustfs/backlog#2873
* fix(decommission): single-flight start, recheck pool state under gate
Review of 3a5276c5b:
- A start re-checks the pool state under the start gate before it
cancels routines, so a start racing another one can no longer cancel
the worker the other start just spawned.
- Only one start runs per instance at a time: another request, including
the retry of an operator whose request timed out, is rejected as
already running instead of stacking a second detached start and its
preflight heals. The start runs through `run_detached_mutation`, and a
failure nobody waits for any more is logged.
- Tests: the proof test pins its drop after the save and requires a
worker that persists a cancellation; the retry test really drops the
first caller; new tests cover the under-gate recheck and a peer reload
failure after the caller is gone. That last one shows the start
rollback cannot clear a fresh start because the pool metadata
stale-write guard refuses it; the gap is asserted and documented and
tracked in rustfs/backlog#2873.
- The start tests run on their own large-stack thread: the io_uring lane
(`uring_`, which also selects `during_`) runs them with libtest's 2 MiB
default stack, where the proof test overflowed.
- Docs: single-flight starts, status and cancel during preflight, the
rollback gap and its recovery, and the release requirement.
Refs rustfs/backlog#2873
* fix(decommission): turn away local start retries at the start slot
Second review of 29253430a:
- The leader-local admin pre-check refreshed rebalance metadata under
the start gate before the start slot was checked, so a local retry
waited out its 20 s gate budget behind a running start (which holds
the gate through its peer reload) and then got a 503. The pre-check
now checks the slot first and returns `DecommissionAlreadyRunning`,
which the handler reports as a 400 that keeps the reason.
- Prove the reload-failure path really ran in the detached start (the
injection is consumed), add a test that a fresh start's failed
rollback reports both the reload error and the stale-write refusal,
and assert an abandoned start stays tracked for shutdown.
Refs rustfs/backlog#2873
* fix(decommission): adopt target intents left by a killed leader
A decommission leader killed after persisting a target capacity intent and
before finalizing it left that intent in the exclusive pending slot of the
still-active reservation. The resumed routine derives the same reservation
identity, but every other mutation on that target was rejected as "another
mutation" until the interrupted object happened to be retried on the same
target, so the resumed decommission exhausted its retries and stalled.
At routine start, before any worker runs, the resumed routine now adopts the
reservation's pending intents: under the target fence (or the legacy pool
metadata fence) it moves each one into a mutation-scoped inflight marker that
keeps its identity and bytes. The interrupted mutation's retry settles the
marker through the existing commit, temporary-release and equivalent-target
paths, and an equivalent committed target now settles a mutation's own marker
when no pending intent remains. Adoption is pinned to the persisted
decommission start time and reservation identity, never runs mid-incarnation,
and a failure pauses the routine instead of marking the decommission failed.
* fix(decommission): keep adopted intents within budget and settle them once
Address the adversarial review of the leftover intent adoption.
- Merge an adopted intent into its mutation's staged marker by summing only
while the target's temporary-copy budget allows, never below the larger of
the two. A multipart completion moves its staged parts into the object, so
summing them could exceed the budget and pause every later routine start;
the adopted marker now exceeds the budget only where the completion's own
finalize would have. Reuse resolve and record_inflight for the move.
- Distinguish a temporary release that proved the exact committed target from
one that could not prove the target identity. When the committed mutation no
longer has a pending intent, release only the bytes the cleanup freed and
keep the adopted completion for the equivalent-target reconciliation.
- Let an equivalent target settle a mutation's own marker while another
mutation holds the pending slot, instead of failing on that unrelated intent.
- Cover real leader kills before and after the target commit at routine level
for both lock models, an injected adoption failure that pauses and then
completes, an adopted multipart completion surviving a background abort, and
a live legacy intent. Routine tests reuse the large-stack test helper.
* fix(decommission): pause instead of failing on target slot contention
The S00 leader-kill run with the null-version fix merged ended the resumed
decommission as failed. A re-run of a versioned key reconciled its delete
marker against an already charged target copy while another worker of the
same incarnation held the target's pending slot between a failed write and
its retry. The reconciliation rejected that unrelated intent, the entry
context wrapper flattened the error to a string the intent-conflict retry
could not recognize, and the bucket runner reported a sibling bucket's
OperationCanceled instead of the capacity block, so the routine marked the
decommission failed.
- An equivalent-target reconciliation, including the temporary checkpoint
variant, no longer depends on another mutation's pending intent: a target
holds one pending intent at a time, so this mutation has none to settle.
- The entry context keeps the typed cause, and the capacity-block and
intent-conflict classifiers also recognize the rendered message that an
error clone keeps.
- Concurrent bucket and listing failures keep the root cause over sibling
cancellations, so a capacity block pauses the routine.
* test(decommission): delete marker copy waits for a busy target gate
The Linux run of the S00 matrix showed the delete marker copy meeting a busy
target capacity gate without any leader kill. The copy already waits on that
gate like an ordinary object copy; pin it with an entry-level test that holds
the gate, checks the entry keeps waiting, and expects the key to migrate once
the gate is released. Share the versioned delete marker fixture with the
foreign pending slot test.
* fix(decommission): keep an adopted completion unless the target is absent
Close the remaining path where an adopted completion could lose its record
before being charged. A multipart cleanup that found a target it could not
prove as the exact owned copy still dropped the whole marker of a mutation
without a pending intent, after which the equivalent-target reconciliation
would treat the commit as already charged.
Any cleanup that did not prove the exact target absent now releases only the
bytes it freed when the mutation has no pending intent, and leaves the rest
for the reconciliation. This makes the three-state release unnecessary, so
the release proof goes back to the existing clear-pending flag.
* chore(ci): shrink the error format ratchet baseline for pools.rs
9eae47ce8 removed one Error::other(format!) call site from crates/ecstore/src/core/pools.rs, so the ratchet requires the baseline to drop from 178 to 177.
* fix(decommission): keep adopted completions in their own capacity marker
An adopted completion shared the mutation's staged marker, so whether
it survived a cleanup depended on the pool-wide free-space delta, and a
cleanup that freed little space left staged bytes behind and repeated
forever.
Record each adopted intent under a mutation id derived from the
interrupted mutation (decommission_adopted_intent_mutation_id) as a
separate temporaryMutations entry. A commit or an equivalent-target
reconciliation settles both markers, a temporary cleanup releases the
staged marker as before, and only a cleanup that proved the exact
target absent drops the adopted one. While the target holds only that
adopted completion, a cleanup keeps the delayed-release exemption of
the published pending intent it replaced. Older nodes see an ordinary
temporary mutation they never release, which only over-reserves.
Also log capacity_intents_adopted as soon as the adopted ledger is
durable, merge the root-cause helpers into decommission_root_cause,
drop the decommission_capacity_blocked: prefix fallback, and log the
already-charged equivalent-target branch at debug level.
Refs rustfs/backlog#2873, rustfs/backlog#2872
* fix(decommission): return uncommitted retries to their adopted target
A retry that had not committed yet went to the reserved target with the
most remaining allocation. When that was not the target holding the
mutation's adopted completion, the commit settled only its own target
and the adopted marker stayed inflight until the reservation was
released.
Prefer a reserved target that holds the mutation's adopted marker while
it still fits the object (decommission_reserved_target_choice).
Also cover the remaining settlement paths of the derived marker
(non-growing replacement, equivalent temporary reconciliation and its
discovery gate, scanner backlog retirement), repeated adoption of one
mutation and the zero-byte staging record in both lock models, drop two
conditions that always equal another one, and correct the adopted-id
doc comment about nodes without adoption.
Refs rustfs/backlog#2873, rustfs/backlog#2872
* fix(decommission): count a completion adopted twice only once
A mutation killed twice leaves an adopted completion and then a new
intent for the same object version. Adoption added the two, so an
11-byte object counted 22 inflight bytes and could trip the capacity
check once later commits shrank the budget. Both reserve the same
completion, so merge them by their maximum before adding staged parts;
a single adoption is unchanged.
Also assert in the three-pool sticky test that no entry failed, the
retry is charged on its adopted target and the source is cleaned, run
it for the legacy lock model too, stop saying operators rely on the
adoption log before rollback, and document that the selection fallback
cannot help an adopted completion.
Refs rustfs/backlog#2873, rustfs/backlog#2872
* fix(decommission): log where an adopted retry is placed
An e2e run could not show whether a resumed retry went back to the
target holding its adopted completion. Target selection now logs
adopted_target_reused when it does, and adopted_target_bypassed with
the adopted target pools when the object no longer fits there and the
adopted marker stays behind. Both carry the mutation id and the source
and target pools.
Refs rustfs/backlog#2873, rustfs/backlog#2872
Clippy ran only after merge, so lint regressions merged freely and then
failed every main run until fixed (10 of 12 main runs on 2026-10-17/18).
Run the workspace Clippy step on every event, warm its check-mode
artifacts in cache-warm with the identical invocation so pull requests
re-check only what they touch, and pin the unconditional step in the
ci_gate contract and its self-test.
The first pull-request run at CARGO_BUILD_JOBS=3 that met a contended
node (39193827281) hit the 75-minute nextest timeout before any test
started. Its sampler shows cargo spinning for 70 minutes with the pod's
io.pressure "full" at 86-88% and cpu.pressure at zero, with node-level
IO pressure already at 36% when the step began: three concurrent
workspace-scale compile units saturate the pod's shared storage, the
mechanism the original #5394 mitigation named. Keep 3 on main pushes and
dispatches, where three runs succeeded, and 2 on pull requests, in both
the nextest and the serial ILM step; pin the ILM budget guard back to
that expression and record the evidence in the step comment.
overtrue