* fix(scanner): treat replication status writebacks as status-class hits
A replication status writeback (persist_replication_state_if_current in
Update mode, including the heals the scanner queues during its own walk)
marked the bucket dirty on the namespace path: it advanced the dirty-usage
generation, woke the scanner, reached the durable dirty journal and
entered the activity snapshot. Any replicated bucket therefore made every
scanner cycle Superseded, the authoritative usage baseline was never
certified and usage freshness stayed stale for as long as replication ran
(#8375).
The writeback is now status class: record_status_class_mutation records
only the process-local producer coverage bit and nothing else. There is
no rescan queue for it, because a deployment with an active replication
rule already walks every bucket on every cycle
(ScannerMaintenanceFeatures::needs_regular_cycle) and that walk refreshes
replication_stats. The routing lives in a named, unit-tested function in
rustfs/src/storage/storage_api.rs; tier-expiration cleanup removes data
and stays on the namespace path.
No persisted format, protocol, admin API or configuration changes.
This is phase 1 of docs/architecture/scanner-per-bucket-convergence.md,
which this change adds together with its index entry.
Refs #8375, #8494, rustfs/backlog#2816.
* fix(scanner): refresh replication statistics during maintenance
Refresh compacted prefixes and completed bucket caches during replication maintenance. Bind set execution reuse to the observed replication statistics so a same-cycle retry cannot restore older counters.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(scanner): avoid expanding the legacy S3 DTO surface
Decode the existing replication configuration type from a JSON fixture instead of adding direct S3 DTO references to a new file. Preserve the compacted-prefix and same-cycle assertions and the footprint ratchet.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(scanner): keep raw enumeration bookkeeping linear per entry
The page writer inserted into its building page with a push-and-sort on
every entry and committed each page through a clone. Insert in sorted
position and move the committed page into the index instead.
Add a regression test that records 20,000 entries in one prefix and pins
the cost to one hash per entry and well under a second, so the quadratic
re-normalization removed by #8114 (rustfs#8113) cannot return once pacing
builds on this walker.
* feat(scanner): decouple bucket walkers from disk count
Add the scanner key walkers_per_disk (RUSTFS_SCANNER_WALKERS_PER_DISK,
1 through 16, default 1). Each local coordinator disk of a set spawns that
many bucket walkers; remote disks keep one. The walker count is what
max_concurrent_disk_scans caps and the disk count stays the floor.
The walkers of one disk share its scanner admission through
LocalDiskAdmissionShare: the first walker to start a bucket takes the
per-disk permit and the last to finish releases it, so a peer's remote
request still sees one scan session per physical disk. Without the share
the second walker was rejected as busy and retired.
The disk_concurrency_budget log line reports walkers_per_disk, walkers and
the reason for the effective limit. The default stays 1 because no local
disk reports rotational media today; the design keeps a higher flash
default as the intent.
* docs(scanner): document walkers_per_disk in the Helm chart and configuration manuals
RUSTFS_SCANNER_WALKERS_PER_DISK was only described in the scanner runtime
controls table. Add it to the Helm chart (values, configmap rendering,
README table, template test), the configuration crate's environment
reference with operator guidance, and a short section in the operations
manual on when to raise it and how to confirm it took effect.
---------
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Enable preserve_order for the Arrow JSON schema reference in tests and verify first-seen field ordering across nested records.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Since #8535 DeleteObjects honours Quiet=true the way S3 does and omits the
successful entries from the response, so the quota batch-delete test has
been asserting two Deleted entries against an empty list on every main run.
The test only needs the batch to succeed and free quota; make it verbose so
the existing assertion holds and check that no entry failed.
* test(e2e): gate the rc.5 baseline restarts on drive re-admission
The mixed-version rolling upgrade failed three times on unrelated branches
with `MultipartAcknowledgementDeadline { node: 2, key: "one-current-node/..." }`.
The baseline kills the next rc.5 node about 10 ms after the previous one
reports ListBuckets, while that node is still replaying its startup traffic.
Two failed operations drop the peer's drive to Offline on the pinned release,
and only the 2 s recovery probe brings it back three successes later. In both
failing runs node 2 held node 3's drive as `faulty` when the multipart load
started three seconds later; it then committed every upload at three of four
drives with a heal per object, and on a two-vCPU runner one upload exceeded
the 30 s acknowledgement budget. The green run on the same branch only
reached `suspect`. An all-rc.5 cluster reproduces the same degradation after
the same restart sequence and an all-current cluster shows none of it.
Write a slash-free probe object before the first restart and, after each
baseline restart, block until every node reports all four drives of that
object as `ok` through a dry-run object heal. `storageinfo` and `info` only
carry each node's local drives; the object heal runs on the receiving node and
classifies a short-circuited remote drive as `faulty`. The gate stays in the
all-rc.5 baseline: admin heal first probes every peer's heal-control
capability, and a mixed-version cluster fails that probe on both binaries.
Record every node's lock table in `mixed-multipart-failure.json`; the stalled
writer is a previous-release node whose table the current node cannot show.
The 30 s budget, layout, load and assertions are unchanged.
* test(e2e): probe drive re-admission with a parity-0 write per node
The dry-run heal gate observed only node 0: the pinned rc.5 admin heal
forwards start and status requests to the heal-control coordinator whatever
node receives them, so the loop checked node 0's drive handles four times
and never the restarted peers' handles.
Run the cluster with `RUSTFS_STORAGE_CLASS_RRS=EC:0` and probe each node
with a reduced-redundancy PUT instead. A parity-0 write on the four-drive
set needs all four drives (`write_quorum = data_drives`), so the PUT fails
with 503 `ServiceUnavailable` until the serving node writes through every
one of its own drive handles. The probe is plain S3 and executes on the
receiving node on both binaries, so the gate now also runs before each
mixed-version load and after the last upgrade. Standard-class objects keep
the default layout.
`parity_zero_write_needs_every_drive` pins the premise on the workspace
build: with one node down a standard write still commits, the
reduced-redundancy write fails with 503, and the gate converges once the
node returns. The diagnostic snapshot test now checks the per-node lock
probe array.
* fix(rebalance): avoid contention when peers join the same run
* fix(rebalance): wait for fleet proof during initialization
* test(rebalance): check contextualized competing run errors
* fix(rebalance): retain typed worker admission lock errors
* test(rebalance): retry the init readiness 503 and pin the retry error shape (#8586)
The e2e harness only retried the pre-fix 500 carrying the fleet-proof
marker; the init stage now answers 503 + Retry-After after its own
readiness budget, so start_rebalance treats that as the same wait. The
missing-proof regression also asserts the typed error shape that the admin
handler's retry predicate depends on, not only the rendered message.
---------
Co-authored-by: Chris <anzhengchao@gmail.com>
* fix(multipart): retain commit guards across request cancellation
* fix(multipart): preserve commit task error context
* fix(multipart): reclaim verified orphan upload staging (#8572)
* fix(multipart): reclaim verified orphan upload staging
* fix(multipart): share orphan guards across matching lock domains
* fix(multipart): preserve orphan cleanup task error context
* fix(ecstore): box guarded delete transition identity
* fix(multipart): keep NoSuchUpload when orphan recovery cannot prove absence (#8587)
Recovery is a side path of an idempotent Abort. An offline disk, lock
contention, partially overlapping lock domains or unreadable bucket
metadata all stop the staging-absence proof before any deletion; those
used to propagate as 503/500, where the base answered NoSuchUpload. They
now keep that answer and delete nothing; only failures inside the
deletion task remain request failures, because a retry can finish them.
---------
Co-authored-by: Chris <anzhengchao@gmail.com>
* feat(diagnostics): attribute storage reads to request origins
Port the diagnostics instrumentation from its experiment base onto main
without carrying the base branch's unrelated changes.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* refactor(diagnostics): consolidate read context scopes
Keep caller and entry provenance in one task-local context, combine the object-read scope, and classify source paths without allocating.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
The outbound target matrix only modeled targets that adopt the forwarded
source ETag, so the post-PUT replica ETag audit was never exercised against
a target that computes its own ETag (AWS S3 and most compatible stores),
and no matrix shape was encrypted. That gap let the SSE-S3 "replica etag
mismatch" regression fixed in #8573 ship unnoticed.
Add an ignore_source_etags mode to the fake S3 target, a ComputesOwnEtags
matrix row that uses it, and an SseS3 object shape (the source runs with the
local KMS). The shape asserts its source ETag is not the plaintext MD5 so
the cell cannot pass vacuously. Against a build without #8573 the new row
fails exactly computes-own-etags/SseS3 with "replica etag mismatch"; with it
the whole matrix passes.
Also correct the #8573 docs and comment that called the replication PUT
checksum the integrity check for encrypted sources: managed-SSE sources
forward no checksum, so their replicas have no post-PUT content audit.
* fix(scanner): persist partial progress after budget cancellation
Separate progress-only cycles from observational and authoritative usage
updates. Keep peer movement leases alive under the parent cycle context
for a bounded drain after the scan budget stops the walk, while preserving
the existing strict publication fences for emitted usage updates.
Skip root rename fences and the usage persister when no update was emitted.
Cover real lease admission, cancellation, bounded retries, and durable
partial-cycle advancement without usage publication or dirty acknowledgement.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* refactor(scanner): consolidate publication policy and ownership
Classify emitted updates directly in the publication plan constructor and
exercise that single policy from the namespace-admission tests.
Share the parent cancellation state for read-only progress proof checks
and move the one-use root fence into the persister, avoiding a child token
allocation and a JSON string copy without changing lease or commit rules.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
`cargo clippy --all-targets -- -D warnings` fails on main since #8535:
`CurrentDeleteGeneration::Present` carries a `CurrentDeleteTransition`
(three Strings, an Option<String>, a 32-byte identity and two small
fields, ~211 bytes) while `Absent` carries nothing, which trips
clippy::large_enum_variant. The workflow only runs clippy on pushes to
main (`if: github.event_name != 'pull_request'`), so the PR CI stayed
green and main's CI has been red since 606d8a3a.
Store the transition behind a Box so the enum is pointer-sized plus the
small fields. The value is transient ("never persisted in object
metadata") and its serde form is unchanged, because serde treats Box<T>
transparently; PartialEq/Clone/Debug derive through it. The two tests
that matched the struct through the Option now bind the Box and check
the same fields in a guard.
Claude-Session: https://claude.ai/code/session_01CezCdjE9AdShFUi9tNk797
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
New encrypted writes store a keyed HMAC fingerprint as the object ETag
(docs/operations/encrypted-etags.md). Replication forwards that ETag on
x-rustfs-source-etag, but a target that computes its own ETag (AWS S3 and
most compatible stores) answers the plaintext PUT with the MD5 of what it
received. Both values are 32 hex characters, so verify_single_part_replica
treated the fingerprint as a comparable MD5 and failed every single-part
SSE-S3 / SSE-KMS replica with "replica etag mismatch"; MRF and resync then
retried the object forever. RustFS and MinIO targets adopt the source ETag
and were unaffected.
Exempt sources whose stored ETag cannot describe the wire bytes: any
managed-SSE or SSE-C source, decided by the existing replication
source-encryption classifier. The checksum header on the replication PUT
remains the integrity check for those objects; plaintext sources keep the
audit unchanged.
Claude-Session: https://claude.ai/code/session_01CezCdjE9AdShFUi9tNk797
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* fix(ecstore): retain system metadata repair ownership
Return system metadata convergence failures to the fenced writer after
committed cleanup instead of submitting an unexecutable MRF handoff.
Drain system rename tails before acknowledging and preserve ordinary
incarnation-bound object repair.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* refactor(ecstore): separate PUT commit and completion outcomes
Keep committed metadata writes on their retained cleanup owner when the
writer must handle repair. Propagate the completion error at the API
boundary without a duplicate temporary-data deletion or worker wait.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): scope repair completion to pool metadata owners
Keep ordinary system metadata quorum-compatible and explicitly select
owner-observed repair completion for fenced pool metadata CAS writes.
Seed format-heal metadata before injecting the missing-format fault.
Preserve degraded bucket recreation and test all completion modes.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): treat replication status writebacks as status-class hits
A replication status writeback (persist_replication_state_if_current in
Update mode, including the heals the scanner queues during its own walk)
marked the bucket dirty on the namespace path: it advanced the dirty-usage
generation, woke the scanner, reached the durable dirty journal and
entered the activity snapshot. Any replicated bucket therefore made every
scanner cycle Superseded, the authoritative usage baseline was never
certified and usage freshness stayed stale for as long as replication ran
(#8375).
The writeback is now status class: record_status_class_mutation records
only the process-local producer coverage bit and nothing else. There is
no rescan queue for it, because a deployment with an active replication
rule already walks every bucket on every cycle
(ScannerMaintenanceFeatures::needs_regular_cycle) and that walk refreshes
replication_stats. The routing lives in a named, unit-tested function in
rustfs/src/storage/storage_api.rs; tier-expiration cleanup removes data
and stays on the namespace path.
No persisted format, protocol, admin API or configuration changes.
This is phase 1 of docs/architecture/scanner-per-bucket-convergence.md,
which this change adds together with its index entry.
Refs #8375, #8494, rustfs/backlog#2816.
* fix(scanner): refresh replication statistics during maintenance
Refresh compacted prefixes and completed bucket caches during replication maintenance. Bind set execution reuse to the observed replication statistics so a same-cycle retry cannot restore older counters.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(scanner): avoid expanding the legacy S3 DTO surface
Decode the existing replication configuration type from a JSON fixture instead of adding direct S3 DTO references to a new file. Preserve the compacted-prefix and same-cycle assertions and the footprint ratchet.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): seed the ordinary-timeout peer fixture under the default drive deadline
The one-second ENV_DRIVE_MAX_TIMEOUT_DURATION in
target_ordinary_timeout_keeps_its_physical_namespace_owner wrapped the
whole test, so the fixture's write_metadata and the final read_version
were bounded by it too. On a loaded CI runner the seed took longer than a
second and the test failed with "seed real metadata: Timeout" before the
paused rename under test ran. Only the rename now runs under the
one-second deadline; the fixture and the tail read use the default.
---------
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
* fix(ci): remove needless returns from Linux disk helpers
* fix(test): verify encrypted purge replicas by version body
* fix(test): expire namespace timeout after fixture preparation
* fix(ci): remove remaining needless collection and borrows
* fix(test): wait for active admin heal at interruption checkpoint
* fix(heal): unblock automatic replacement recovery
Repair bucket publication directories before adopting survivor delete proofs.
Keep explicit empty-volume removal after proof settlement and preserve the
admitted bucket incarnation for administrator replay.
Discover unfinished intents across every local set and resume the durable
owner with its exact target scope before admitting new sibling replacements.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(heal): route scanner fixtures through the storage boundary
Expose the existing disk-layout bootstrap symbols as test-only aliases in
heal::storage_api and use that boundary from scanner recovery regressions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* refactor(heal): index replacement discovery within each scan
Keep a single endpoint-keyed disk inventory with first-snapshot precedence
and deterministic discovery order. Index local target lookups and move
resume state into generation candidates instead of cloning it.
Retain fresh durable reads and the existing conflict and mutation fences.
Cover duplicate disk handles, endpoint uniqueness and discovery ordering.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(heal): retain committed journals in recovery fixtures
Build one certified survivor delete decision for both replacement and
empty-volume removal tests. Clear target contents without moving its root
inode so Linux disk handles observe the same empty target. Drain PUT tails
and assert the proof remains before the real Heal entry.
Expose the existing recovery state through the ECStore disk API for typed
fixture checks.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(heal): use server stack for replacement regression
The replacement regression polls the composed Heal future on libtest's
smaller calling-thread stack and aborts before reporting an assertion.
Run it on a bounded 8 MiB test thread and Tokio workers, matching the
debug server's stack budget. Keep all storage and object assertions intact.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(replication): skip version-addressed marker pre-check on targets that mint their own version ids
* fix(replication): keep bodiless 403 fail-closed and verify resynced markers on targets that mint their own ids
---------
Co-authored-by: Chris <anzhengchao@gmail.com>
* fix(deps): patch profiling dependency for RUSTSEC-2026-0344
Vendor the published Pyroscope package with an object 0.40 dependency and
lock ruzstd 0.9.1. Preserve runtime sources and profiling features, add
bounded decoder regressions, and run the patched package in audit CI.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
Signed-off-by: Hauser <housemecn@gmail.com>
* fix(deps): correct vendored pyroscope documentation typos
Correct copied comments and changelog spelling for the repository typo
check, and describe these comment-only changes in the vendor provenance.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
Signed-off-by: Hauser <housemecn@gmail.com>
* fix(ecstore): refresh compact transition proof per test case
Install the short-lived homogeneous fleet proof immediately before each
kill-point transaction. Earlier cancellation and recovery work must not
age the next case's writer capability fixture.
Report an early transition result instead of waiting for an unreachable
kill point. Preserve cancellation, remote-owner and payload assertions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
Signed-off-by: Hauser <housemecn@gmail.com>
---------
Signed-off-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
fix(ecstore): suppress clippy too_many_arguments on rename_data_commit_from_observations
The function takes 9 slice parameters for the renamedata commit phase.
Annotate with #[allow(clippy::too_many_arguments)] to match the existing
codebase pattern for internal multi-slice commit helpers.
Group disk observations into a borrowed struct to keep the commit helper
within Clippy's argument limit without changing quorum snapshot logic.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Keep same-name buckets in independent clusters separate while deduplicating
replicated usage observations. Document the existing scanner-backed exporter
and cover its gauge contract and cluster-scoped queries.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(e2e): bound scanner witnesses by the reported schedule
Require a fresh completed observed walk and account once for planned
superseded waiting, preserving the active-cycle execution budget.
Cover invalid schedules and early wakeup, and refresh exact platform
selection digests without removing existing tests.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): refresh nightly scanner budget membership
Record the three scanner budget regressions in both platform nightly
selection digests, preserving every previously selected test.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Preserve canonical claim encoding for the signature tamper test and pause the admin cancellation fixture at its existing run fence instead of durable deletion.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): reuse observations and expose complete usage
Allow superseded retries to reach the existing validated cache planner while
bootstrap remains pending. Serve complete observations in server info without
changing quota authority or dirty acknowledgements.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(scanner): route fixture setup through storage boundary
Keep the single-disk scanner fixture on the existing storage_api boundary
instead of importing the ECStore API facade directly.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(upgrade): separate rc.5 data and candidate query checks
Verify historical fixture bytes before upgrading without requiring the
pinned rc.5 reader to satisfy the candidate's cross-part range contract.
Reuse the complete-body oracle and retain all upgraded range, partNumber,
attributes and replication assertions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): replace timing races with recovery evidence
Prove no_lock batch progress with the existing cleanup barrier while the
caller retains its guard. Check namespace counters and every drive's
metadata absence rather than using a one-second disk I/O deadline.
Keep durable ILM's manual recovery as the sole tier journal consumer and
verify exact migrated input before retaining the original recovery stats,
receipts and remote-version assertions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
fix(ecstore): preserve retry evidence during delete cleanup
Bind normal cleanup to the coordinator transaction after quorum succeeds, and remove the rollback backup before marked data. Preserve data and markers if backup cleanup fails.
Add deterministic LocalDisk crash and retry regressions plus real four-disk single and batch coordinator coverage. This prevents one post-quorum cleanup window; it does not resolve ambiguous inline or historical residue in #8464.
Co-authored-by: Zhengchao An <overtrue.us@gmail.com>
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(s3): evaluate conditional reads before opening object bodies
Reuse locked metadata snapshots for conditional GETs and unify HEAD
preconditions and cache headers. Preserve body-cache hooks and validate
encryption metadata and key authorization before returning bodyless errors.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* refactor(s3): unify conditional reads at metadata quorum
Apply scoped metadata conditions to the existing reader pipeline instead
of maintaining parallel prepared-read branches. Preserve quorum checks,
early-stop, coalescing, and cache-hook behavior on condition misses.
Track metadata-only termination through selection and reject body access
or a changed decision. Keep concrete reader futures off request stacks.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(s3): strengthen conditional read acceptance evidence
Assert zero shard, decode, cache-fill, and remote-body calls on matching
conditions with live positive controls. Exercise GET and HEAD through
s3s and Hyper with conditional-header and precedence matrices.
Add reproducible paired application-read benchmarks with isolated
metadata controls and strict drift, artifact, and resource gates.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* refactor(s3): remove obsolete conditional comparison helper
Keep ETag comparison tests on the canonical read-condition evaluator,
remove unused date-trait imports, and name the HTTP case type.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(s3): validate full conditional etag fields
Reject adjacent tags and malformed trailing list members before accepting a
match. Keep the existing request-scoped metadata read and authorization order,
while removing redundant validation wrappers and unused benchmark overrides.
Replace simulated phase coverage with real 304/412 cache-fill probes and extend
HTTP/1.1 and HTTP/2 ingress regressions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): cache terminal conditional metadata safely
Preserve quorum-validated conditional metadata when early termination
cancels the remaining fanout. Restrict those entries to terminal GET
readers; body and full-layout consumers still obtain their own metadata.
Serialize cache publication per key so conditional entries cannot replace
complete layouts or renew their TTL. Retain generation fencing and the
final selected-metadata predicate check before any body access.
Cover real four-disk hedging, nonterminal body and range reads, consumer
isolation, generation races, atomic publication and actual cache expiry.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(log-analyzer): align heal writer failure rule with current logs
Match the current structured all-targets failure, retain legacy log support, and reject partial or unrelated failures.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Reject causal cycles before source repair and share conservative backup
classification between discovery and inspection. Fence public raw deletion
and derive exact-key removals from successful settlement without an extra
metadata read. Preserve the existing physical phases and strict source checks.
Make outage and renewal fixtures deterministic without weakening their
convergence assertions, and complete strict TLS/RPC lint cleanup.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): bound sentinel tests by the lock wait, not recreation IO
Both bucket incarnation sentinel tests gave the bucket recreation that
follows the paused delete (force delete_bucket, make_bucket, put_object
on four fsync'ing disks) 10 s in total. The delete releases the sentinel
milliseconds after the barrier, but on a runner starved for IO the
recreation alone can exceed 10 s:
delete_objects_holds_bucket_incarnation_sentinel_through_commit failed
that way on main in run 37760321449 (job 113259251979).
That bound is not what catches a kept sentinel: delete_bucket waits for
it on its lifecycle write lock, which fails after the lock acquire
timeout, and the recreation's expect reports the lock error. Keep that
check and give the recreation 120 s only as a hang guard.
The recreation starts that lock wait before the barrier is released, so
the wait also has to cover the delete's own commit, and the 5 s default
leaves a starved runner little room for it. Run both tests with
RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=30 and bound the delete, which was
not bounded, by the same 30 s. A kept sentinel still fails the tests,
after 30 s instead of 5 s.
Signed-off-by: Petr Slavyagin <disident47@gmail.com>
* test(ecstore): keep background recovery out of dispatch rollback tests
dispatch_manifest_detached_workers_are_store_bounded_and_shutdown_cancellable
seeds six aborting dispatch manifests, installs a pause hook and expects
its own recovery passes to park four detached workers on it. Its first
store also ran the background tier-delete journal recovery loop, which
runs a pass every TIER_DELETE_JOURNAL_RECOVERY_INTERVAL (60 s). When the
setup took longer than that, a background pass rolled the manifests
back before the hook was installed, no worker paused, and the 30 s wait
failed. This happened on main on a runner starved for IO, in run
37760321449 (job 113259251979).
Build the first store with background recovery suppressed, as the
neighbouring dispatch tests do. The restarted store keeps it: the second
half of the test checks that it picks up the retained manifests.
dispatch_manifest_rollback_bounded_concurrency_reaches_tail_behind_slow_member
and dispatch_manifest_page_timeout_detaches_one_deduplicated_worker_until_convergence
have the same exposure: each seeds an aborting manifest on a store with
background recovery and only then installs its slow-delete hook. They
have not failed in CI, but with the setup stretched past 60 s they fail
the same way. Suppress background recovery for them too.
Signed-off-by: Petr Slavyagin <disident47@gmail.com>
---------
Signed-off-by: Petr Slavyagin <disident47@gmail.com>
* fix(ecstore): recover crash-interrupted delete retries
Retain set-quorum cleanup proof and restore metadata before dropping rollback
markers. Reclaim eligible exact-key residue during explicit delete retries,
with bucket identity, version, retention and namespace fencing guards.
Preserve prefix children and snapshot ownership through batched physical
cleanup, and cover interrupted transactions and failure boundaries.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): route recovery test imports through contracts
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): preserve missing-key deletes at quorum
* feat(ecstore): add certified set-wide delete recovery
Atomically prepare deletion journals and exact targets, publish durable set
receipts, and recover through one fenced disk/RPC core used by startup and
scoped administrator healing. Preserve returning generations, ordinary
missing-key quorum semantics, and legacy rollback wire compatibility.
Retain namespace and volume ownership through late physical operations,
order related transactions causally, and keep proof until full convergence.
Remove the superseded local-only recovery path and add storage, protocol,
RPC, durability and isolated Docker regression coverage.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): reclaim completed multipart delete sidecars
* fix(ecstore): recover multipart sidecars with verified proofs
Recognize native multipart sidecars at the shared recovery boundary and
restore them without treating metadata or proof files as shard payloads.
Validate required proof generations and bind every present live/staged
shard copy to its metadata-bound Merkle proof before restoring metadata.
Cover protected native MPU rollback, split source layouts, missing or
corrupt proofs, and payload corruption with recomputed bitrot checksums.
Add real multipart Docker regressions in both integrity modes.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): settle superseded committed version deletes
Allow a validated committed decision to finish its own data cleanup after
a later decision removes current metadata. Prepared copies need an
inspected same-transaction receipt matching immutable local intent before
promotion, and retain their local journal without importing peer hashes.
Reuse the authority matcher for dry-run and settlement. Cover chained
explicit-version deletions, partial receipt publication, mismatch rejection
and no-resurrection, while retaining the no-decision absence protection.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): clarify delete recovery case label
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): restore recovery CI guard coverage
Keep production error sites visible to the ratchet by placing test modules last. Preserve native encoding errors, group preparation identity and ownership, and reduce RPC enum size without changing the wire payload.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(admin): preserve owned OIDC validation errors
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* refactor(ecstore): unify recovery authority and snapshot work
Use one user-bucket legacy settlement boundary and one adoption source predicate for preview and execution. Reuse immutable snapshot digests, consume owned generation metadata, index version membership, and prepare empty witnesses through the canonical journal path.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): preserve delete recovery invariants
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
唐小鸭