Keep native publication recording in main's native_io helpers and resource
writers, and adapt merged tests to the owned Body, Bytes write input, and
fallible find_disk.
Record native publications on the paths main's commit-fusion work opened
directly: the lazy truncate writer and the fused delete-recovery activation
and receipt renames. Keep extract_params_header in the shared audit builder.
* feat(scanner): share node admission with background heal
* fix(scanner): drain candidates after listing completion
* fix(scanner): yield cache leases for shared node admission
* test(scanner): attest adaptive admission in process receipts
* fix(concurrency): declare Tokio time dependency
* feat(concurrency): retain bounded physical resource subsets
* fix(scanner): retain resource ownership in native filesystem jobs
* feat(concurrency): acquire physical groups before node work
* feat(scanner): gate background reads by physical device
* feat(concurrency): combine preallocated resource selections
* feat(scanner): share physical topology across disk plans
* fix(storage): retain only native disk resource subsets
* feat(storage): admit auxiliary physical resources without waiting
* feat(heal): admit physical resources before object repair
* fix(storage): retain resources during native format probes
* fix(storage): retain resources through direct writer I/O
* fix(storage): retain resources through buffered writer I/O
* fix(storage): retain physical resources through streamed reads
* fix(storage): retain resources through native filesystem jobs
* fix(storage): retain physical ownership during bitrot reads
* fix(ecstore): retain resources during positioned reads
* fix(ecstore): retain staged metadata write ownership
* fix(ecstore): retain native io_uring read ownership
* fix(ecstore): enforce receiver-side background admission
* fix(ecstore): retain resources through whole-file native I/O
* fix(ecstore): retain physical ownership during durable rename sync
* fix(storage): account for retained background stream buffers
* fix(storage): retain memory charges for background read results
* fix(storage): account for native io_uring read allocations
* fix(storage): retain memory charges for direct-write buffers
* fix(ecstore): account for heal and RPC reader buffers
* test(scanner): verify samples and clean concurrent collectors
* fix(storage): retain inline and HTTP writer memory reservations
* perf(rio): reuse reserved HTTP writer storage
* fix(rpc): reserve background scanner pipe storage
* perf(rpc): reuse reserved response reader storage
* feat(scanner): measure cache copy allocation requests
* fix(rpc): budget namespace scanner responses
* fix(scanner): retain the enabled supervisor task handle
* fix(scanner): budget namespace frame read staging
* fix(scanner): account namespace envelope decode storage
* fix(heal): retain and join the MRF consumer on shutdown
* test(heal): isolate process-global MRF consumer fixture
* fix(heal): join manager workers during shutdown
* test(heal): verify shutdown at native read boundaries
* fix(heal): drain channel commands before stopping MRF
* fix(heal): admit resume scans and retain replacement workers
* build: pin profiling sources and trust fixed ownership dependencies
* fix(heal): yield node admission while accepted native work drains
* fix(startup): retain shutdown work and preserve deadline failures
* fix(heal): admit inventory probes and retain replacement work
* fix(shutdown): preserve maintenance and frontend join failures
* fix(shutdown): retain HTTP child tasks through real retirement
* fix(shutdown): join actual TLS reload work before retirement
* test(multipart): preserve failures before fault barriers
* fix: keep physical admission inside repository guards
Route the multipart lock diagnostics filter through the e2e storage
compatibility module, reuse the parent module's s3s body aliases in the
namespace memory test, and share one Reed-Solomon verify error mapping
across the reserved-buffer and default paths.
* fix: reconcile physical admission with main's new call sites
Main added a source-admission metadata write layer, a CAS write fence,
fixture writers and scanner handoff tests against the old borrowed-slice
and node-permit shapes. Pass owned Bytes through the new metadata layer,
answer the scanner handoff with ScannerNodePermit, gate the test-only
disk lookup re-export, and clear the lint findings that stable 1.99
reports for the shutdown and admission code without suppressions.
* test(concurrency): name reversed attachment groups clearly
* fix(ecstore): box disk operation futures below admission layers
Every LocalDiskWrapper operation reaches the LocalDisk future through the
health, admission and timeout layers (track_cleanup_mutation, the
track_disk_health family, the io-permit and native-resource admission
split, and track_admitted_disk_operation). Each layer embedded that future,
and unoptimized builds copy it into every awaiting frame, so one local
delete cost about 1.3 MiB of stack and 16 tests overflowed libtest's
default 2 MiB stack.
Box the operation future once where track_admitted_disk_operation creates
it, through a non-async helper so the temporary never stays in a polled
frame. The layers above now hold only a pointer; the cost is one
allocation per local disk operation.
* fix(heal): box object heal futures below admission
The admission wrappers took the object heal future by value, so unoptimized
builds copied it into three awaiting frames, about 650 KiB per heal. Create
it through a non-async helper instead, leaving only a pointer in those frames.
* test(ecstore): count the I/O backend's root FD owner
* ci(ft): update the lane gate to the 54-case manifest with scenario F
auto-testing #118/#119 added FT scenario F and case B5, so the lane now
runs 54 cases over scenarios A,B,C,C2,D,E,F and passes 54/54 — but the
workflow gate still pinned the 44-case A-E manifest and failed the
report step. Align the gate and the contract-test fixture with the
54-case manifest.
* ci(heal): give heal a 90-minute run budget inside a 120-minute job
The heal task was still progressing (scanned 494 -> 571+, failed=0)
when the 45-minute step cap killed it, so the verification steps never
ran and the lane produced invalid evidence (chain 40796474356). The
step budget is now 90 minutes, and the job-level budget rises from 60
to 120 minutes so the enclosing job cannot cancel the raised step
budget. The timeout contract now records heal as the exception to the
shared 60/45 baseline.
* ci(chain): force the performance probe offline while pf-testing is broken
The pf-testing environment is temporarily broken, so every nightly
chain would end red in the performance lane. performance is
required=false in scheduled-validations.json, so forcing the probe's
performance_ready output to offline takes the chain through its
designed skip path: the lane job is skipped, aggregate receives
--allow-skipped performance, and the report records the optional-lane
skip with its reason. Marked TEMPORARY; restore the probe once the
environment is healthy again.
* test(table): drop a stale workflow-comment pin left by #9059#9059 rewrote the table suite step comments but kept the assertion for
the old '# Red only for harness/environment breakdowns' line, so
test_security_workflow.py fails against current main. The semantics
stay pinned by the continue-on-error and HARNESS_OK assertions.
Since #9059 the shared test archives and debug binaries are named
with the run attempt, and the archive identity binds the attempt. A
"Re-run failed jobs" attempt does not rerun the successful producer,
so its consumers look for <name>-2, fail with "Artifact not found",
and every flaky test forces a full rerun.
Give those artifacts one name per run, overwrite them when a producer
does rerun, and bind the archive identity to the run instead of the
attempt. Source, tree, lock, toolchain and environment stay bound, and
an archive from another run is still rejected.
The planned package list is passed to cargo as bare -p names. The zstd
compat crate is named zstd so it can patch crates.io, and with zstd
0.14.3 also locked, cargo rejects -p zstd as ambiguous, failing
Workspace Lint and Build on main since #9059.
Qualify such names with the local version from Cargo.lock, and fail
loudly if even that cannot disambiguate.
Sync main 84e24e767 into gateway/integration. Ports main-side additions to
the integration boundaries: the tier deletion diagnostics admin handlers move
to the owned admin request/response types, and the new lifecycle rule checks
in ecstore use the persisted lifecycle types instead of s3s DTOs. Keeps the
gateway git source in deny.toml while dropping the smithy-rs pin main retired,
and refreshes the test-count baseline for the merged tree.
Sync main 9601a7c4b into gateway/integration. Ports main-side additions to the
owned admin boundary (plugin delegation/attempt handlers, notification target
edit tests), drops a new ecstore s3s::dto reference in favour of the persisted
lifecycle filter, and refreshes the e2e selection digests and test-count
baseline for the merged tree.
* test(decommission): cover held target lock and partial-copy resume
Port two tests from the closed lock-recovery branch onto the deferral
that defers any typed namespace lock timeout on a version copy.
- A version commits, then every attempt at the next one times out on
the target lock: the entry is deferred as a lock-timeout deferral, no
failed item is counted, both source versions stay, and the resumed
entry moves both versions with their exact bytes.
- A real write lock held on the target key makes target selection
return a typed object-lock timeout, the entry defers instead of
failing, and once the lock is released selection recovers and the
resumed entry completes the move.
* chore(ci): shrink the error(format!) ratchet baseline for rebalance
The rebalance control and meta modules on main now have fewer
Error::other(format!) call sites than the baseline records, which
fails the shrink-only ratchet in Quick Checks. Record the smaller
counts.
Report selected integer offsets and widths with strict member and payload validation. Cross-check compiler fixtures with bpftool and independent C offsetof/sizeof results.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): add memory acceptance runner and budget probes
Add owned S3 workload phases with process, endpoint, cgroup and binary
identity checks for scanner memory and scanner-disabled request bursts.
Keep scale, restart, heap-attribution and overall-acceptance limits explicit.
Characterize healthy and seeded coverage-gap cache retention through the
real folder walker and production cache codec without changing runtime code.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): bind acceptance capacity to effective storage
Validate the serving process's complete local disk inventory before load
writes and during runtime sampling. Resolve each disk inside the owned
storage root, reject nested mounts and filesystem mismatches, and bind
storage observations to phase identity for container and native runs.
Cover the unused-mount bypass, CLI-overridden environment, native paths,
symlink escapes, named volumes and stable multi-disk inventory ordering.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): fence status observations before workload writes
Close each status API window with a second storage-bound process sample
before Phase entry can authorize bucket creation or PUT requests.
Add a fail-before regression for a process change during the initial status
request, asserting no ownership or workload writes occur.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(tools): add read-only BTF prototype inspection
Expose bounded declared prototype headers through a standard-library CLI, preserve explicit void and failure semantics, and wire local contracts plus an independent compiler/bpftool reference into CI.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(tools): retain native reference failure diagnostics
Clear checkout credentials and capture bounded native stderr alongside stdout, preserving exact exit status and owned child cleanup. Keep the first reference failure unresolved rather than guessing its missing diagnostics.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* ci(tools): select a concrete bpftool file parser
The hosted PATH wrapper can exist without tools for its Azure kernel. Select an executable ELF by availability/version before comparing the fixed fixture, installing generic tools only when needed. Data comparison errors remain fatal and never trigger tool switching.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Source: main 4e48c390e7
Target: gateway/integration 1000860cb8
Conflicts resolved, keeping both sides:
- crates/ecstore/src/disk/{fs,local}.rs: main's batched rollback writes,
unified blocking write path and relocated open_file, with integration's
native publication records restored at each publication point and the
publication_history field kept beside main's multipart proof authority.
- crates/ecstore/src/set_disk/ops/multipart.rs: union of the imports.
- rustfs/src/storage/rpc/node_service.rs: main's MRF lifecycle reader
capability branch; the heal capability keeps integration's restore
copy-back probe.
- rustfs/src/admin/handlers/{audit,event,plugins_instances}.rs: main's
target_read inventory snapshot on integration's owned request boundary;
the new event.rs inventory test now builds owned requests.
- scripts/check_s3s_footprint.sh: integration's baseline-file version.
- .config/e2e-{full,smoke}-selection.txt: digests recomputed over the
union of both heads' CI listings (full 700 darwin / 708 linux, smoke 232).
Also migrated main's new rebalance delete-all expiry test to the persisted
lifecycle types integration uses in ecstore, and raised the test-count
baseline to the merged counts (every decrease comes from main).
* refactor(admin): complete the owned admin request boundary
Every type implementing the owned admin `Operation` now serves the legacy
router through one blanket impl that converts the s3s request and response
at that single edge, so handlers register with `AdminOperation(&Handler {})`
and need no adapter statics; `LegacyAdminAdapter` is removed.
`AdminRequest` gains the accessors handlers read today (`remote_addr`,
`client_info`, `request_context`, `req_info`, `req_info_mut`,
`install_req_info`, `envelope_mut`, `take_body`), the request helpers gain
envelope forms (`authenticate_envelope_credentials`,
`object_store_from_envelope`, the table-catalog `*_from_envelope` helpers,
`KmsAdminAudit::from_envelope`, `AccountAuditContext::from_envelope`,
`CallerIdentity::resolve_envelope`), `read_compatible_admin_body` accepts
either body type, and the owned `Body` gains `store_all_limited` and
`Default`. `admin::operation::test_support::TestRequest` builds owned
requests for handler tests. Unused accessors carry a per-method dead-code
expectation that each migrating handler batch removes.
No handler behavior changes; the remaining migrations follow in batches.
* chore(admin): add the T1.12 admin handler codemod
The script rewrites an admin handler file from the s3s request and response
types to the owned boundary (imports, signatures, request field and helper
calls, body access, response construction, and test request literals) and
leaves a TODO comment at every site it cannot rewrite mechanically. It is
idempotent and is removed once T1.12 completes.
* refactor(admin): migrate replication and KMS handlers to owned requests (#8951)
Move the T1.12 batch 2 admin handlers (site_replication, kms_keys, event,
replication, diagnostics, on_demand_migration, audit, account, kms_rekey,
kms_key_lifecycle, object_data_cache, tls_debug, integrity) from the s3s
request/response types to the owned AdminRequest/AdminResponse boundary.
Handler logic, authorization order, error codes and response bytes are
unchanged; the request/response plumbing now goes through the envelope
accessors and helpers introduced by the owned boundary.
The shared response helpers (utils::json_response, utils::empty_response,
handlers::admin_json_response, target_descriptor::build_json_response,
profile::authorize_profile_request) and the site replication repair
executor still answer with the s3s types, so the migrated files carry
owned-boundary forms of those helpers and convert the repair response at
its call edge. The streaming console-log body and the two devnull drains
move from the s3s byte stream to the owned Body stream/frame API.
Drop the dead_code expectations for the owned accessors and envelope
helpers this batch starts using, and lower the s3s footprint baseline to
the measured 99 files.
Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).
* refactor(admin): add owned forms of the shared admin response helpers
Handler batches were copying `json_response`, `empty_response`,
`admin_json_response`, and `build_json_response` into each migrated file
because the shared helpers still return the s3s response. Provide owned
forms next to each legacy helper (`owned_json_response`,
`owned_empty_response`, `owned_admin_json_response`,
`owned_build_json_response`, `authorize_profile_envelope`) with parity
tests against the legacy shape, derive `Debug` for `AdminResponse` so
tests can use `expect_err`, and fix three codemod rules: strip owned
string conversions in test credentials, annotate non-literal URI parses,
and leave a rebound request parameter immutable.
* refactor(admin): use the shared owned response helpers in batch 2 (#8954)
Replace the per-file owned-boundary copies of json_response,
empty_response, admin_json_response, build_json_response and
authorize_profile_request in the batch 2 admin handlers with the shared
owned_* helpers and authorize_profile_envelope that the boundary branch
now provides, and restore expect_err in the handler tests now that
AdminResponse implements Debug; the `.err().expect(..)` stand-ins were
failing the base branch's clippy step (clippy::err_expect under
-D warnings). The site replication repair call edge
keeps its conversion because the repair executor still answers with the
s3s response type.
Drop the dead_code expectations on the shared owned helpers this batch
now uses in non-test code.
Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).
* fix(admin): preserve legacy body limit error responses
* refactor(admin): move policy, ILM and batch 3 handlers to owned requests (#8956)
Batch 3 of T1.12 (rustfs/backlog#2788, parent rustfs/backlog#2734) moves
14 admin handler files to AdminRequest/AdminResponse and the owned
Operation trait: ilm_transition, policies, idp_compat, config_admin,
plugins_instances, profile_admin, group, heal, object_zip_download,
kms_key_metadata, durability, inspect_archive, sts and
gateway_key_inventory. Handler logic, authorization order, error codes
and response bytes are unchanged; test request literals use
test_support::TestRequest.
The handlers use the shared owned helpers (owned_json_response,
owned_admin_json_response, owned_build_json_response,
authorize_profile_envelope). The dead_code expectations on the
AdminRequest accessors and RequestEnvelope::uri_mut they now exercise
are removed. The legacy target_descriptor::build_json_response keeps
only its equivalence test caller and carries the matching expectation
until it retires.
object_zip_download builds the s3s request the S3 gate still reads from
the owned request facts; heal authenticates through
authenticate_envelope_credentials; the trace, inspect-archive and zip
bodies use Body::from_stream. extract_plugin_instance_filters takes the
URI so extensions.rs keeps calling it, and the two idp_compat
delegations stay on the legacy trait until their delegates migrate.
The s3s footprint baseline drops from 99 to 90 files.
* test(admin): anchor the replication diff body contract on the call
The source-contract test looked for `let body = read_compatible_admin_body(`
on one line; rustfmt wrapped that binding after the handler moved to the
owned request boundary, so the marker was no longer found. Anchor on the
call itself so formatting cannot change what the contract checks.
* refactor(admin): migrate table catalog and batch 1 to owned requests (#8955)
* fix(admin): preserve catalog body limit response bytes
* refactor(listing): keep paginated listing on ECStore only
`Sets` and `SetDisks` carried their own copies of `inner_list_objects_v2`,
`list_objects_generic`, `inner_list_object_versions`, `list_path` and
`list_merged`, plus a third copy of the task that converts merged raw entries
into the walk result stream. Only `ECStore` can answer a paginated listing,
since a page merges every set of every pool; the copies drifted (the `Sets`
copy still rejected `//` prefixes that #8605 made literal on `ECStore`) and
were reachable only from tests and one decommission helper.
Remove the `ListOperations` impls for `Sets` and `SetDisks` together with the
duplicated pagination code. `SetDisks::walk` stays as an inherent method for
set-scoped callers and now honors `versions_sort` like the store-wide walk;
`SetDisks::list_path_result` stays for the recursive delete lister. The walk
entry conversion moves into one `spawn_walk_entry_forwarder` shared by both
walks. The decommission receipt lister walks its pool's sets instead of
paginating through the removed `Sets` listing.
Tests that exercised the removed layers now exercise `ECStore` for pagination
or `SetDisks::walk` for per-pool version inspection. The architecture and
logging guard rows that pinned the removed `SetDisks`/`Sets` listing entry
points are dropped, and the error-other ratchet baseline shrinks by one.
* fix(listing): exclude synthetic directories from receipt walks
* refactor(listing): remove env-gated ListObjects index provider
Delete the experimental, default-off key-only / verified-page /
metadata-fast index provider from `ECStore::list_objects_generic` together
with everything only it drove: the `RUSTFS_LIST_OBJECTS_INDEX_*`,
`RUSTFS_LIST_OBJECTS_NAMESPACE_JOURNAL_*` and
`RUSTFS_LIST_OBJECTS_METADATA_FAST_*` readers, the persistent key-only
index and its on-disk format, the namespace mutation journal (state,
chaos injector, system/local backends, degraded set, global lock), the
index lifecycle/health/source-selection types, the verified-index
candidate pager, the `list-chaos` cargo feature, the
`rustfs_s3_list_objects_index_*` metrics and the archived bench script.
`list_objects_generic` now always walks. The write-path hooks
`observe_list_objects_mutation(s)` are gone; their callers bump the scanner
namespace generation directly through `observe_scanner_namespace_mutations`,
which is the sub-step the old hook performed before taking the journal
lock, so the generation observed by the scanner, the node RPC service and
the data-movement lease checks is unchanged while PUT/DELETE no longer take
a global tokio RwLock or read environment variables.
`ListSourceMode::{metadata_authority, requires_live_metadata_verification,
can_satisfy_strong_listing}` and `ListMetadataAuthority` were only reachable
from the provider and are removed as well; the enum, its variants, the
cursor encoding and `ListContinuationV2` are left untouched.
* docs(listing): define index retirement upgrade prerequisites
* fix(ci): retire the removed listing index test requirement
* refactor(listing): keep paginated listing on ECStore only
`Sets` and `SetDisks` carried their own copies of `inner_list_objects_v2`,
`list_objects_generic`, `inner_list_object_versions`, `list_path` and
`list_merged`, plus a third copy of the task that converts merged raw entries
into the walk result stream. Only `ECStore` can answer a paginated listing,
since a page merges every set of every pool; the copies drifted (the `Sets`
copy still rejected `//` prefixes that #8605 made literal on `ECStore`) and
were reachable only from tests and one decommission helper.
Remove the `ListOperations` impls for `Sets` and `SetDisks` together with the
duplicated pagination code. `SetDisks::walk` stays as an inherent method for
set-scoped callers and now honors `versions_sort` like the store-wide walk;
`SetDisks::list_path_result` stays for the recursive delete lister. The walk
entry conversion moves into one `spawn_walk_entry_forwarder` shared by both
walks. The decommission receipt lister walks its pool's sets instead of
paginating through the removed `Sets` listing.
Tests that exercised the removed layers now exercise `ECStore` for pagination
or `SetDisks::walk` for per-pool version inspection. The architecture and
logging guard rows that pinned the removed `SetDisks`/`Sets` listing entry
points are dropped, and the error-other ratchet baseline shrinks by one.
* fix(listing): exclude synthetic directories from receipt walks
* fix(listing): stop abandoned walks and filter directory entries
Source: main 0f24511535
Target: gateway/integration 8cf6187ff4
Resolve the e2e-full selection digest conflict by recomputing both
platform digests over the merged nextest listing (685 darwin / 693
linux): main adds two rpc_fault_proxy unit tests, integration keeps its
dual-stack differential and anonymous-access additions. Raise the rustfs
test-count baseline row to the merged count (5932).
* fix(scripts): collect scanner host telemetry on macOS
Use platform-supported iostat arguments and expose optional collector
failures without interrupting scanner-status collection.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(ecstore): reuse validated metadata read path buffers
Share the checked bucket/object tuple across metadata read consumers and borrow clean normalization inputs without skipping filesystem validation. Keep Windows reconstruction and the second bucket check intact.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(scripts): finalize scanner telemetry for failed runs
Initialize host telemetry artifacts before external captures so reused output directories cannot retain prior successes. Share normal and error-path finalization, reap collectors on request failure, and preserve the primary exit status even if status recording fails.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): distinguish unreadable recovery inventories
Treat only the exact system-prefix read-quorum result as incomplete during bounded polling. Preserve strict final reads and the shutdown/drain assertions, record failed observations, and cover interrupted empty streaks and unrelated errors.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* perf(ecstore): avoid head-of-line blocking in delete recovery
Recover independent keys through a bounded rolling window while preserving per-key hint generations, admission budgets, cancellation retry, and the existing transaction decision engine.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(ecstore): bound durable delete preparation and bootstrap recovery
Prepare independent Linux delete transactions with bounded concurrency while retaining per-key result order and serial fallback for overlapping, rollback, and legacy requests. Preserve startup discovery until bucket metadata is initialized, and add release performance and process-crash acceptance runners.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): verify clean exits in recovery acceptance
Reject forced or OOM exits when measuring clean startup and verify original ETags across mixed-layout crash recovery and a second restart.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): distinguish prepared and decided batch crashes
Observe real preparation and receipt states separately, retain non-vacuous acknowledged-delete sentinels, and require read-only physical convergence before S3 reads can trigger cleanup.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): move final delete recovery options
Move the options value into the batch-ordering regression's final case
instead of cloning it. Keep the earlier clone needed for reuse.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): replay delete receipts after interrupted proof GC
Retire only the exact empty transaction leaf before journal activation. Preserve occupied directories and prefix children, and cover partial cross-disk proof GC without a fresh delete. Tighten crash evidence to distinguish directory-only remnants from retained content.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): purge every explicit delete-marker version
Keep explicit marker removals on the canonical version-delete path so
same-key batches are not truncated and pending replication does not retain
source markers. Preserve marker identity and timestamps in the response.
Add real four-disk regressions for mixed version cleanup, duplicate marker
IDs, pending replication, and null/UUID marker removal.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(filemeta): index explicit delete-marker batches
Reuse the batch index and final compaction for ordinary marker removals.
Keep small marker-only histories on the scalar path and preserve fallback
for free versions and complex mutations.
Add scalar-equivalence, boundary, corruption and deterministic scan-count
coverage, a real-disk mixed-history regression, and reproducible marker
benchmark cases. Preserve existing benchmark defaults while honoring CLI
sampling overrides.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(filemeta): remove redundant marker fixture clone
Move the original metadata into its final test fixture after the last
borrow, avoiding the redundant clone reported by strict Clippy checks.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(storage): validate explicit marker deletion and recovery
Exercise UUID/null and duplicate explicit marker removals over real HTTP
with retries disabled, including immediate four-node visibility and exact
retained payloads. Cover all-drive PREPARED/APPLIED/DECISION process
termination, rollback/commit recovery, and old-ID retries across a second
restart without deleting a fresh marker.
Use flat sampling and fixed per-case fixture batches in the metadata
benchmark. Update profile selection digests for the added regressions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(heal): synchronize crash fixture startup and flush readiness
Give disk setup and the durable flush boundary separate readiness gates,
keeping each phase bounded to ten seconds. Preserve the real SIGKILL and
restart assertions while reporting early child exits and retaining output.
Reap fixture children on assertion failure and cover early-exit diagnostics.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(e2e): use the resumed decommission completion budget
* test(heal): exercise checkpoints written by rc.5
* test(heal): route upgrade fixtures through the storage facade
* fix(deps): update ctutils to address constant-time advisory
* test(lifecycle): gate recovery I/O while advancing test time
* feat(scanner): configure startup recovery backoff cap
* test(scanner): compare recovery retry count without narrowing
* docs(scanner): document recovery backoff cap and expose it in Helm
List scanner.recovery_backoff_cap in the runtime controls table, drop the
fixed ten-minute wording from the follower recovery section, and map
config.rustfs.scanner.recovery_backoff_cap_secs to
RUSTFS_SCANNER_RECOVERY_BACKOFF_CAP_SECS in the chart.
overtrue