Retain the reviewed diagnostic dependency while starting request-stage work
from current main. This integration is not a performance qualification.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): select the latest version at read quorum like MinIO
GET and HEAD escalated latest-version selection to write quorum when
enough metadata answered and skipped a newer version below write quorum
on degraded reads. LIST and heal already pick the newer side of a split
write at read quorum, so such keys listed but answered 503 until heal.
Select with MinIO's getObjectFileInfo rule: the most common mod time
that reaches the object's data blocks wins, ties go to the newer time,
and one content identity at that time must reach the same quorum. Two
identities tied at the selected time fail instead of resolving by hash
order, and the selected identity's version-stack shape follows disk
order.
* fix(ecstore): resolve selection ties by disk order and guard writes
Review of the read-quorum selection found three gaps:
- Rejecting a same-time half split made explicit-version reads of a
partially applied tagging update fail and blocked the retry that
repairs it. Resolve ties to the identity on the lowest-index disk,
for reads and heal alike, instead of rejecting or using map order.
- Delete markers and empty versions can lower the common read quorum
below the selected version's data blocks, so HEAD could report a
version GET cannot decode. Require that many copies, as MinIO's
getLatestFileInfo does.
- A selection can now return exactly read-quorum disks, and partial
metadata writes are not rolled back. Refuse an update before writing
when its target disks cannot reach write quorum.
* fix(ecstore): count identity copies in the data-blocks guard
The first-candidate shortcut counts byte-identical copies, so copies
that differ only in identity-neutral fields such as replication status
were missing from the data-blocks check. When the shortcut's count
falls short, count every copy of the selected content identity before
refusing the version.
* fix(ecstore): keep commit-proof selection for system-bucket reads
Pool metadata's PREPARE/COMMIT recovery and identity epochs treat an
ordinary read of pool.bin as proof that the record committed. Under the
read-quorum rule, a COMMIT that reached only half of a 2+2 set was served
as committed, and the pool_meta_native_physical_* integration tests
failed.
Route reads through select_fileinfo_for_read: S3 buckets keep MinIO's
read-quorum rule, while is_meta_bucketname reads use the previous
algorithm unchanged as select_committed_fileinfo, so a split stays
unreadable until native recovery resolves it.
* fix(ecstore): serve the latest readable version past dangling versions
A failed versioned PUT whose rollback also failed can leave the new
version on top of fewer drives than its data blocks. Reading only each
drive's top version then left the committed version below it short of
read quorum, so GET and HEAD of the latest version answered 503 although
that version is intact in every stack. MinIO resolves the stacks in
pickLatestQuorumFilesInfo.
When every drive answered and an S3 latest read still cannot select a
version, walk the newest version headers of each stack through
exact-version reads: serve the readable version with the newest selected
modification time, skip versions whose exact read proves them dangling
(the NoSuchVersion an explicit GET reports) and queue heal for them, and
keep the read-quorum error on any other outcome. Candidates are ordered
by each version's newest modification time across drives, and at most
eight exact reads are spent.
Test PUTs now wait for every drive so tests that edit one drive's
metadata no longer race the PUT's background tail writes.
* fix(ecstore): resolve dangling latest versions in metadata updates
A failed versioned PUT whose rollback also failed can leave the new
version on fewer drives than its data blocks. GET and HEAD of the
latest version already look past such versions, but read-before-write
callers without a version id still answered 503: object tagging read
each drive's top version with the quorum-error read, and
put_object_metadata and the CopyObject source read selected from
read_all_xl, whose merged latest version is missing on the other
drives, so object_quorum_from_meta failed before selection.
Object tagging now reads through a variant that runs the same walk past
dangling versions as a latest GET while keeping the quorum error for a
dangling key. put_object_metadata and the CopyObject source try the walk
when their quorum or selection step fails with read quorum, before the
existing dangling cleanup could report a key with a committed version
absent. A version-only copy takes the served version's own copies and
write quorum. The walk still applies only to S3 buckets, latest reads,
and sets where every drive answered, and the update still targets only
the served version's disks, so a version below write quorum is refused
before writing.
* fix(ecstore): refuse version-only copies below write quorum up front
Review of the dangling-latest metadata updates found three gaps:
- A version-only CopyObject wrote the new version and reverted on a
write-quorum miss; refuse it before writing when the served version's
disks cannot reach its write quorum, like update_object_meta_with_opts.
- Replication status write-back, including the zero-write ValidateOnly
mode, now keeps its read-quorum error and never walks past dangling
versions, since the walk queues heal for the versions it skips.
- The erasure-coding doc overstated that these callers never report a
dangling key as absent; put_object_metadata and CopyObject keep their
existing dangling cleanup when the walk finds no readable version.
Omit span metadata and inferred parent IDs for diagnostic root events. Verify per-shard index order and document existing production metric timing.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
Record local batch eligibility, blocking queue and execution time, shard
outcomes, and actual read bytes behind the existing GET-stage metrics gate.
Keep captured metric handles with the blocking worker after cancellation
and use scoped low-latency histogram buckets for the new duration series.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): select the latest version at read quorum like MinIO
GET and HEAD escalated latest-version selection to write quorum when
enough metadata answered and skipped a newer version below write quorum
on degraded reads. LIST and heal already pick the newer side of a split
write at read quorum, so such keys listed but answered 503 until heal.
Select with MinIO's getObjectFileInfo rule: the most common mod time
that reaches the object's data blocks wins, ties go to the newer time,
and one content identity at that time must reach the same quorum. Two
identities tied at the selected time fail instead of resolving by hash
order, and the selected identity's version-stack shape follows disk
order.
* fix(ecstore): resolve selection ties by disk order and guard writes
Review of the read-quorum selection found three gaps:
- Rejecting a same-time half split made explicit-version reads of a
partially applied tagging update fail and blocked the retry that
repairs it. Resolve ties to the identity on the lowest-index disk,
for reads and heal alike, instead of rejecting or using map order.
- Delete markers and empty versions can lower the common read quorum
below the selected version's data blocks, so HEAD could report a
version GET cannot decode. Require that many copies, as MinIO's
getLatestFileInfo does.
- A selection can now return exactly read-quorum disks, and partial
metadata writes are not rolled back. Refuse an update before writing
when its target disks cannot reach write quorum.
* fix(ecstore): count identity copies in the data-blocks guard
The first-candidate shortcut counts byte-identical copies, so copies
that differ only in identity-neutral fields such as replication status
were missing from the data-blocks check. When the shortcut's count
falls short, count every copy of the selected content identity before
refusing the version.
* fix(ecstore): keep commit-proof selection for system-bucket reads
Pool metadata's PREPARE/COMMIT recovery and identity epochs treat an
ordinary read of pool.bin as proof that the record committed. Under the
read-quorum rule, a COMMIT that reached only half of a 2+2 set was served
as committed, and the pool_meta_native_physical_* integration tests
failed.
Route reads through select_fileinfo_for_read: S3 buckets keep MinIO's
read-quorum rule, while is_meta_bucketname reads use the previous
algorithm unchanged as select_committed_fileinfo, so a split stays
unreadable until native recovery resolves it.
* fix(ecstore): serve the latest readable version past dangling versions
A failed versioned PUT whose rollback also failed can leave the new
version on top of fewer drives than its data blocks. Reading only each
drive's top version then left the committed version below it short of
read quorum, so GET and HEAD of the latest version answered 503 although
that version is intact in every stack. MinIO resolves the stacks in
pickLatestQuorumFilesInfo.
When every drive answered and an S3 latest read still cannot select a
version, walk the newest version headers of each stack through
exact-version reads: serve the readable version with the newest selected
modification time, skip versions whose exact read proves them dangling
(the NoSuchVersion an explicit GET reports) and queue heal for them, and
keep the read-quorum error on any other outcome. Candidates are ordered
by each version's newest modification time across drives, and at most
eight exact reads are spent.
Test PUTs now wait for every drive so tests that edit one drive's
metadata no longer race the PUT's background tail writes.
* chore(ci): shrink the error(format!) ratchet baseline for rebalance
The rebalance control and meta modules now have fewer
Error::other(format!) call sites than the baseline records, which
fails the shrink-only ratchet in Quick Checks. Record the smaller
counts.
* fix(ecstore): preserve canonical order in latest-version fallback
Keep the diagnostic implementation private to its owner and expose explicit feature-gated helpers through the existing storage boundary.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
Keep timing and request correlation in a separate default-off feature. Preserve the existing positioned-read backend, byte and error contracts.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
Report selected integer offsets and widths with strict member and payload validation. Cross-check compiler fixtures with bpftool and independent C offsetof/sizeof results.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): select the latest version at read quorum like MinIO
GET and HEAD escalated latest-version selection to write quorum when
enough metadata answered and skipped a newer version below write quorum
on degraded reads. LIST and heal already pick the newer side of a split
write at read quorum, so such keys listed but answered 503 until heal.
Select with MinIO's getObjectFileInfo rule: the most common mod time
that reaches the object's data blocks wins, ties go to the newer time,
and one content identity at that time must reach the same quorum. Two
identities tied at the selected time fail instead of resolving by hash
order, and the selected identity's version-stack shape follows disk
order.
* fix(ecstore): resolve selection ties by disk order and guard writes
Review of the read-quorum selection found three gaps:
- Rejecting a same-time half split made explicit-version reads of a
partially applied tagging update fail and blocked the retry that
repairs it. Resolve ties to the identity on the lowest-index disk,
for reads and heal alike, instead of rejecting or using map order.
- Delete markers and empty versions can lower the common read quorum
below the selected version's data blocks, so HEAD could report a
version GET cannot decode. Require that many copies, as MinIO's
getLatestFileInfo does.
- A selection can now return exactly read-quorum disks, and partial
metadata writes are not rolled back. Refuse an update before writing
when its target disks cannot reach write quorum.
* fix(ecstore): count identity copies in the data-blocks guard
The first-candidate shortcut counts byte-identical copies, so copies
that differ only in identity-neutral fields such as replication status
were missing from the data-blocks check. When the shortcut's count
falls short, count every copy of the selected content identity before
refusing the version.
* fix(ecstore): keep commit-proof selection for system-bucket reads
Pool metadata's PREPARE/COMMIT recovery and identity epochs treat an
ordinary read of pool.bin as proof that the record committed. Under the
read-quorum rule, a COMMIT that reached only half of a 2+2 set was served
as committed, and the pool_meta_native_physical_* integration tests
failed.
Route reads through select_fileinfo_for_read: S3 buckets keep MinIO's
read-quorum rule, while is_meta_bucketname reads use the previous
algorithm unchanged as select_committed_fileinfo, so a split stays
unreadable until native recovery resolves it.
* test(scanner): add memory acceptance runner and budget probes
Add owned S3 workload phases with process, endpoint, cgroup and binary
identity checks for scanner memory and scanner-disabled request bursts.
Keep scale, restart, heap-attribution and overall-acceptance limits explicit.
Characterize healthy and seeded coverage-gap cache retention through the
real folder walker and production cache codec without changing runtime code.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): bind acceptance capacity to effective storage
Validate the serving process's complete local disk inventory before load
writes and during runtime sampling. Resolve each disk inside the owned
storage root, reject nested mounts and filesystem mismatches, and bind
storage observations to phase identity for container and native runs.
Cover the unused-mount bypass, CLI-overridden environment, native paths,
symlink escapes, named volumes and stable multi-disk inventory ordering.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): fence status observations before workload writes
Close each status API window with a second storage-bound process sample
before Phase entry can authorize bucket creation or PUT requests.
Add a fail-before regression for a process change during the initial status
request, asserting no ownership or workload writes occur.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Retain original prepare and commit-settlement RPCs beyond failed replacement and Abort cleanup, with and without bucket recreation. Require typed refusal or a physically verified idempotent result, unchanged fresh-upload bytes and complete object reads.
Extend scoped proxy observations without rewriting signed frames or trailers, and preserve registered test membership and existing budgets.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(tools): add read-only BTF prototype inspection
Expose bounded declared prototype headers through a standard-library CLI, preserve explicit void and failure semantics, and wire local contracts plus an independent compiler/bpftool reference into CI.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(tools): retain native reference failure diagnostics
Clear checkout credentials and capture bounded native stderr alongside stdout, preserving exact exit status and owned child cleanup. Keep the first reference failure unresolved rather than guessing its missing diagnostics.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* ci(tools): select a concrete bpftool file parser
The hosted PATH wrapper can exist without tools for its Azure kernel. Select an executable ELF by availability/version before comparing the fixed fixture, installing generic tools only when needed. Data comparison errors remain fatal and never trigger tool switching.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Verify a sixteen-drive set rejects reads and writes after two nodes stop,
then resumes complete S3 reads and new writes when only one node returns.
Keep the fourth node stopped throughout checks from all three live
coordinators and verify the original 12+4 shard geometry on disk.
Let the existing GET recovery probe retry the compatible SlowDownRead
response only with HTTP 503, retaining fatal status and byte checks.
Register the case in both platform selections and document the boundary
between process recovery and physical disk reattachment acceptance.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(observability): correlate physical namespace lock releases
Map logical namespace guards to their actual granted peer lock IDs and local physical guard identities. Record physical completion only after successful manager release and registry removal, and record validated positive remote unlock responses without changing release behavior.
Exercise real distributed ownership, asynchronous cleanup, duplicate and disabled guards, stale-owner retirement, and explicit release followed by Drop. Document the required diagnostic filters and the remaining filesystem-tail boundaries.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): remove unused lock manager test import
The diagnostic regression uses inherent manager methods and does not need the LockManager trait import. Remove it to pass strict warning checks.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
A cursor for a key that literally ends in a legacy-shaped tag is issued
wrapped in one extra `[rustfs_cache:v2,return:]`, because the listing layer
still strips one legacy tag from incoming markers. ListObjects (V1)
unwrapped it in `calculate_next_marker`; the client then sent the bare key
back as `marker`, the listing layer stripped the key's own tail, and the
next page restarted at the stem and returned the same key forever. Keep the
wrapper in V1 NextMarker and cover the V1 resume path (marker sent back as
StartAfter) on a real store.
Document that sampled list quorum modes can also omit committed objects
whose copies mostly sit outside the sample, and recommend `strict` for
backup, sync, replication and lifecycle tooling.
* chore(deps): upgrade russh to 0.65.0 and refresh crates
Run two cargo update and cargo upgrade rounds from origin/main.
Adapt SFTP public-key callbacks to PublicKeyOffer and VerifiedPublicKey,
preserving explicit rejection and password-only authentication.
Cover the rejection policy with a pinned-host-key SSH handshake test.
Refresh Arrow, Parquet, Brotli, ZIP, and compatible dependencies.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(deps): route legacy zstd consumers to fixed codecs
Replace the old registry implementation with a private version-constraint
adapter backed only by zstd 0.14.3 and zstd-safe 8.2.0. Preserve legacy
features and Avro, Arrow IPC, Parquet, and Kafka formats.
Cover abandoned contexts, legacy decode errors, seekable bounds, stored
frames, consumer interoperability, and resolved implementation versions.
Record the adapter removal condition in the compatibility register.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* refactor(listing): list latest versions with MinIO's listing quorum
Latest-version SetDisks::list_path now follows MinIO's er.listPath:
ceil(asked/2) is the listing quorum for min_disks, directories and
objects, agreed entries are emitted as read, and a split sample goes
through MetaCacheEntries::resolve. One object can no longer fail a
page. A set with fewer than half of its drives online still fails the
page with ErasureReadQuorum.
Remove the write-quorum layer this replaces: ListingSupplement,
ListingReconciler and the LIST object read lock, the latest_listing_*
quorum helpers, resolve_agreed_entry, resolve_with_write_quorum and the
enforce_write_quorum / write_quorum_slack merge branches, and the
per-version listing quorum helpers in filemeta.
Make merge_file_meta_borrowed_streams pick a version id's most common
header regardless of reader order, so an interrupted null-version
overwrite no longer drops the committed generation from the merge.
* test(listing): do not assert sampled optimal listings are deterministic
optimal asks a random half of the drives, so an EC:8 object committed on nine
of sixteen drives can miss the four-copy listing quorum in roughly one sample
in six, exactly as in MinIO. Keep the deterministic assertion for strict and
for optimal when every online drive is asked.
* fix(storage): 404 for dangling objects and strict list quorum default
Stage F.0 of the ListObjects quorum plan. Once listing returns to MinIO's listing quorum, a PUT that reached half of the drives but not read quorum becomes visible in LIST, so GET and HEAD must not answer it with a retryable 503.
GET, HEAD and prepared GET metadata now report a metadata read-quorum failure as ObjectNotFound or VersionNotFound when every drive answered with valid metadata of a single generation or not-found and is_object_dangling proves the object or version dangling, matching MinIO's getObjectFileInfo. Offline or unanswered drives, timeouts, corrupt xl.meta, two generations, remote versions and the meta bucket keep the read-quorum error. Nothing is deleted on the read path; heal keeps owning cleanup and its grace window. Read-before-write callers of get_object_fileinfo keep the quorum error.
RUSTFS_LIST_OBJECTS_QUORUM now defaults to strict, matching MinIO, and the operations doc explains that optimal samples drives and can briefly show deleted objects.
A new test pins the write-path precondition: a PUT that misses write quorum leaves no new version or data directory on any drive once its rollback completes.
* test(listing): prove the list quorum env var is read with a non-default value
Version listings bypassed listing quorum: SetDisks::list_path returned
early into list_versions_authoritatively, whose resolver seeds unread
slots as DiskNotFound. With any one drive of the set offline, every
version held by fewer than ceil(n/2) drives (for example a deleted
version kept by drives that were offline during the delete) failed the
whole page with ErasureReadQuorum, surfaced as S3 503.
Route merged version pages through the listing-quorum branch of
SetDisks::list_path, as MinIO's listPath does with requestedVersions =
0, so such versions are omitted instead of failing the page. Keep the
set-level floor: fewer than ceil(n/2) online drives still fails the
page.
Set walks (tier reference proofs, tier-delete journals, pool
maintenance) and the recursive prefix-delete scan act on absence, so
they keep the authoritative per-version scan and fail closed.
* refactor(listing): make cursors plain keys and opaque V2 tokens
RustFS inherited MinIO's metacache cursor tag and appended
`[rustfs_cache:v2,id:<uuid>,src:walker,gen:live]` to every internal
continuation marker, although nothing ever consumed the id: four parsers
had drifted apart, V1 NextMarker had to be stripped on the way out, and
ListObjectVersions leaked the tag onto the wire as NextKeyMarker.
ecstore stops assigning `list_id` in `ECStore::list_path` and
`SetDisks::list_path_result`, drops `append_list_cache_id_to_marker` and
the `cache_id` parameters of the pagination helpers, and removes the
unused `ListPathOptions::encode_marker`; internal markers are the plain
last key, last common prefix or last raw scanned key.
The API layer builds the V2 NextContinuationToken as base64 of the
server-side cursor in one place and decodes it in one place; V1
NextMarker is the decoded cursor and the `public_list_marker` tag
stripper is gone; NextKeyMarker is the plain key. Base64 of the plain
key is what nodes from earlier releases already read, so a rolling
upgrade needs no switch.
Compatibility, registered as `listing-cursor-legacy-tag`: `parse_marker`
and `list_marker_key` still strip a trailing legacy tag from incoming
markers so a client that paginates across the upgrade keeps its position.
* fix(listing): preserve literal cache-tag cursor boundaries
* perf(ilm): back off idle tier-delete journal recovery
Use bounded idle backoff only after complete empty journal and dispatch scans, while preserving immediate notifications and the retry cadence for observed work.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): reconcile interrupted pool metadata publication
Bind pool metadata CAS and Undo to exact native predecessors. Recover an interrupted physical publication only from an ordinary committed authority and complete verified identity, current and backup evidence, then require normal V3 CAS and quorum readback.
Preserve known-conflict gates, startup and heal lock ordering, targeted admission and the original kill-recovery deadline. Wake proof-ready recovery immediately and bound parallel capacity sampling without removing mutation fences.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(ecstore): reuse proven pool metadata durability
Publish native rollback backups directly in their owned UUID directories. Reuse a one-shot data-durability receipt only for an exact hardlink to the pinned, fully synchronized head and root.
Preserve new-head and directory synchronization, native predecessor CAS, conditional Undo and uncertainty gates. Keep unknown, copied, raw-parent, cancelled and failed operations cold. Add bounded diagnostics, sync-call regressions and path-isolated test hooks.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(multipart): exercise selective RPC delay and peer loss
Add authenticated method-entry barriers for metadata and lifecycle locks under e2e-test-hooks. Verify real dispatch, write quorum, prior part payloads, lock drain and incarnation isolation without changing production budgets.
Record cancellation separately from normal exits so late filesystem completion cannot overwrite Drop evidence.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* docs: define multipart delay gates and receipt invariants
Document RPC entry evidence, peer-loss and cancellation oracles, remaining incident attribution limits, and immutable head inodes required for durability receipts.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(readiness): expose missing runtime drive slots
Report configured slots without a runtime handle separately from unavailable inventoried drives. Preserve quorum and probe status semantics. Wait for every coordinator to regain its complete live inventory before multipart RPC fault windows and single-attempt recovery oracles.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): retry committed promotion convergence safely
Require peer reload convergence on every worker attempt, including unchanged retries. Recheck durable identity, replicas, namespace fence, generation and cancellation before allowing any data movement.
Keep known committed promotions active on a verified propagation pause. Release precise owners, retain queued successors and reuse the existing watchdog. Preserve uncertainty gates and terminal operation behavior.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): guard native metadata mutation aliases
Reject protected RenamePart source and destination paths before I/O. Classify the protected namespace through the native path resolver, and require canonical conditional RenameData and Undo keys so aliases cannot create separate lease domains.
Preserve ordinary multipart and canonical CAS behavior, warm durability receipts and zero-allocation canonical fast paths. Cover real alias, byte identity, receipt and normal mutation behavior.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): register Darwin multipart RPC delay gates
Update distributed membership from the actual Darwin listing. Preserve the full-profile digest and leave Linux unchanged until its own listing is produced, as required by the contributor guide.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(multipart): fence native publication and recovery
Publish multipart data through private hardlinks and atomically consume the
prepared source head. Bind exact compensation capabilities and native recovery
to the physical cohort, source generation and durable publication state. Require
fresh terminal confirmation for native replay and preserve indeterminate data
when no safe recovery witness exists.
Release logical guards only after validated irreversible quorum success while
retaining physical mutation leases, accounting and resource ownership through
actual callbacks. Preserve local instance proof authority and historical Copy
metadata without granting healed donors fresh publication authority.
Keep general movement identity strict while preserving directory capacity
compatibility in a separate checked comparison.
Cover native fault and cancellation boundaries, zero-byte objects, real Copy,
heal and migration, exact physical readback and distributed RPC delay. Retain
all quorum and durability requirements and document rollout limitations.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): fix ratchet scan and Windows fault hooks
Move native pool metadata inventory tests after production items so the
error-format ratchet sees the existing call site. Preserve the baseline
instead of accepting a false count decrease.
Make the directory-barrier failure hook available on every test platform
and exercise the parent-barrier failure/retry cases in Windows CI. Keep
production synchronization behavior and all existing assertions intact.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(protos): remove redundant recovery request clone
Move the baseline into its final struct update while retaining the earlier
clone needed to construct the first request variant.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): align RPC delay gates with native abort
* fix(storage): validate native source recovery and CI gates
Preserve wire layouts while bounding native response sizes and fix the
last-use recovery request clone. Avoid destination replay reads only for
fully observed initial native sources; keep upgraded and recovered sources
conservative. Cover real legacy publication after a cold restart.
Align upgrade acceptance with the native mixed-version mutation pause and
use exact upload-scoped test completion evidence before advancing time.
Keep all original quorum, ownership, byte and deadline assertions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): verify native abort under partial admission
Delay native cleanup at the exact upload root after all-participant closure.
Verify quorum-only admissions retain every source byte on refused Abort and
complete the same upload without creating a missing original participant.
Keep sub-quorum orphans and their discoverable identities isolated while a
fresh all-participant upload proves service recovery. Route shared constants
through the existing E2E storage facade.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): avoid formatting a fixed invalid upload path
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): align native MPU fault recovery assertions
Verify real source participants, preserved orphan bytes and same-upload completion after quorum-only admission. Require full physical inventories so an expected read-quorum error cannot hide unrelated leakage. Preserve ordinary quorum and recovery budgets, and document listing availability through the affected set.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): keep MPU tail deadlines on the wall clock
Global Tokio time advancement can expire drive-health probes while real
filesystem callbacks are pending. Use a short real drive deadline with the
existing physical-entry and cleanup barriers, retaining the byte, lock,
resource-permit and final-shard assertions.
Fixes#8937
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(storage): align native fixtures with instance ownership
Open independent disks for the simulated capacity-loss instance, scope four coordinator journal disks without rebinding source authority, and probe configured namespace keys while retaining physical I/O hooks.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): isolate native multipart future construction
Construct optional native publication futures behind synchronous heap boundaries so ordinary rename callers do not carry their state machines. Preserve native guards and ownership without adding allocations to ordinary PUT, copy or heal branches.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(heal): assert coordinator journal scope after rejoin
Check the exact original four-member journal map after every rejoin and manager stop while preserving all sixteen native data slots. Reconnect expansion fails the test rather than being filtered away.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(heal): bound object repair future construction
Construct the normal and FileCorrupt deep-retry state machines behind a synchronous heap boundary. Preserve guards and cancellation while bounding the shared rename and deletion call stacks without increasing test stacks.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): qualify fixture disk identity checks
Call the same DiskAPI method explicitly so the independently reopened disk identity assertion compiles under every target and feature combination.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): isolate native inspection from mutation state
Select a dedicated read-only inspection future behind the existing single Box allocation. Preserve validation, source ownership and the canonical inspection body while keeping mutation state out of inline read callers. Reject misrouted operations with typed errors and retain source evidence.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): pause abort time after native cleanup
Wait for durable source deletion on the wall clock before advancing the
capacity lease clock at the post-delete barrier. Paused time can otherwise
expire the barrier while native filesystem cleanup is still pending.
Retain lease-loss injection, durable inflight-byte accounting and restart
reconciliation assertions. Document the test-util feature required to
select the native decommission fixtures in package-scoped regressions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): reclaim data from owned multipart undo
Retain exact runtime-only Undo ownership through deferred snapshot cleanup. Revalidate every target metadata version and the retirement anchor, keep the delete claim through the physical callback, and remove the anchor after the payload barrier. Preserve generic cleanup refusal when metadata is missing.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): preserve unformatted bootstrap routes
Keep explicitly unformatted local paths reachable by an in-flight elected formatter while quarantining their service slots and disk-ID membership. Own bounded legacy-read handles so the actual asynchronous creator future remains Send.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(s3): bound CopyObject tracing poll frames
Construct the business future behind an outlined single-Box boundary and lazily instrument its handle. Preserve Send, Unpin, span timing and cancellation without embedding concrete business-state construction in the tracing poll.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(mpu): observe recovery after direct legacy bootstrap
Seed all sixteen real previous-release sources before attaching the same-port method observers. Require exact per-peer disk-route Unimplemented responses with zero publication calls, retain byte and cleanup assertions, and remove superseded raw transport. Register the two new regressions from actual Linux and macOS listings.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(mpu): prepare the homogeneous previous-release fleet
Reuse the existing release baseline before creating the genuine legacy multipart sources. Restart old processes with their normal budgets and require every drive through the reduced-redundancy probes before installing the mixed-version observers and current fault budgets.
Keep standard EC:4, single-attempt calls, all mixed-version byte and reclamation assertions, and their original deadlines unchanged.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(mpu): isolate release compatibility from fault profiles
Keep normal component configuration for the observed mixed-release rollout, which injects no RPC delay or loss. Retain the short component profile in actual method-fault scenarios and preserve the single-attempt 45-second probe and every typed error, routing, source byte and reclamation assertion.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(mpu): use normal shutdown for planned release replacements
Use the existing bounded SIGTERM path during homogeneous release preparation and the observed mixed-version rollout. Preserve every multipart inventory, routing, quorum and byte assertion, and keep actual crash recovery on its original hard-kill path.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* revert: retain the verified release compatibility fixture
Restore the published fixture after the local configuration and lifecycle experiments failed to resolve the macOS startup failures. Keep their diagnostic records separate from the delivered implementation.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(copy): bound inherent handler polling stack
Build the inherent CopyObject future through a synchronous noinline
factory at the existing trait await. This keeps its large state out of
the trait wrapper's active polling frame during historical self-copy.
Preserve cancellation, tracing and post-write notifications, with one
additional allocation restricted to COPY.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): validate authoritative quota and native pool heals
Require actual authoritative quota usage in the existing readiness loop
instead of treating HTTP success or unknown usage as a numeric baseline.
Keep the original quota assertions and convergence budgets.
Allow the owning pool metadata CAS to republish into a fresh valid data
directory while retaining exact version, layout, completeness and shard
fingerprints. Require every node's real metadata/read/write readiness
before accepting recovery. Keep ordinary object manifests and the original
180-second recovery deadline unchanged, with negative oracle regressions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): distinguish certified native Abort recovery
Require native participants with a persisted full closure certificate to retire their heads and reclaim every surviving source on retry. Keep uncertified legacy authority below identity quorum conservative, while retaining actual partial-delete and committed-object byte checks.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
* refactor(listing): keep paginated listing on ECStore only
`Sets` and `SetDisks` carried their own copies of `inner_list_objects_v2`,
`list_objects_generic`, `inner_list_object_versions`, `list_path` and
`list_merged`, plus a third copy of the task that converts merged raw entries
into the walk result stream. Only `ECStore` can answer a paginated listing,
since a page merges every set of every pool; the copies drifted (the `Sets`
copy still rejected `//` prefixes that #8605 made literal on `ECStore`) and
were reachable only from tests and one decommission helper.
Remove the `ListOperations` impls for `Sets` and `SetDisks` together with the
duplicated pagination code. `SetDisks::walk` stays as an inherent method for
set-scoped callers and now honors `versions_sort` like the store-wide walk;
`SetDisks::list_path_result` stays for the recursive delete lister. The walk
entry conversion moves into one `spawn_walk_entry_forwarder` shared by both
walks. The decommission receipt lister walks its pool's sets instead of
paginating through the removed `Sets` listing.
Tests that exercised the removed layers now exercise `ECStore` for pagination
or `SetDisks::walk` for per-pool version inspection. The architecture and
logging guard rows that pinned the removed `SetDisks`/`Sets` listing entry
points are dropped, and the error-other ratchet baseline shrinks by one.
* fix(listing): exclude synthetic directories from receipt walks
* refactor(listing): remove env-gated ListObjects index provider
Delete the experimental, default-off key-only / verified-page /
metadata-fast index provider from `ECStore::list_objects_generic` together
with everything only it drove: the `RUSTFS_LIST_OBJECTS_INDEX_*`,
`RUSTFS_LIST_OBJECTS_NAMESPACE_JOURNAL_*` and
`RUSTFS_LIST_OBJECTS_METADATA_FAST_*` readers, the persistent key-only
index and its on-disk format, the namespace mutation journal (state,
chaos injector, system/local backends, degraded set, global lock), the
index lifecycle/health/source-selection types, the verified-index
candidate pager, the `list-chaos` cargo feature, the
`rustfs_s3_list_objects_index_*` metrics and the archived bench script.
`list_objects_generic` now always walks. The write-path hooks
`observe_list_objects_mutation(s)` are gone; their callers bump the scanner
namespace generation directly through `observe_scanner_namespace_mutations`,
which is the sub-step the old hook performed before taking the journal
lock, so the generation observed by the scanner, the node RPC service and
the data-movement lease checks is unchanged while PUT/DELETE no longer take
a global tokio RwLock or read environment variables.
`ListSourceMode::{metadata_authority, requires_live_metadata_verification,
can_satisfy_strong_listing}` and `ListMetadataAuthority` were only reachable
from the provider and are removed as well; the enum, its variants, the
cursor encoding and `ListContinuationV2` are left untouched.
* docs(listing): define index retirement upgrade prerequisites
* fix(ci): retire the removed listing index test requirement
* test(e2e): assert Abort refusal below identity quorum
Keep exactly-quorum success and recoverable cleanup assertions. For
minority staging authority, require typed refusal, zero further Delete
RPCs, unchanged per-disk metadata and the previous object body. This
aligns the phase matrix with the existing conservative recovery contract.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): exercise simultaneous multipart terminal contenders
Retain the four pairwise resend cases and add live/cancelled three-way Complete, Abort and resend races. Observe distinct forwarded lock owners and verify terminal bytes and physical cleanup.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): freeze contender evidence before multipart cleanup
Retire the passive lock observer before terminal ListParts checks so the reported count identifies only the two contenders at the preparation barrier.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): require both multipart terminal race outcomes
Retain the free races and select both terminal winners with independently released, owner-scoped lock gates. Verify real contender admission, unchanged signed packets and exact terminal bytes for live and cancelled part waiters.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): isolate losing coordinator lock votes
Hold every remote vote for the losing coordinator while the same gate observes the winner. Prevent partial lock grants from delaying the requested terminal order and retain ordinary RPC caller lifetime.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): preserve uncertainty after partial multipart abort
Keep recovery callers from reporting NoSuchUpload while minority primary metadata remains. Retain all quorum and deletion checks. Exercise actual partial Abort and explicitly retained late RPC packets after caller retirement, staging cleanup and bucket recreation.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): prevent retired multipart staging resurrection
Require the upload root to exist before preparing recovery publication. Preserve rooted directory walks for nested bases on non-Linux Unix and cover missing-source native work, configured aliases and pinned mixed-version coordinator recovery.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): use IPv4 peer aliases for pinned mixed recovery
Keep direct IPv6 S3 listeners behind IPv4 RPC proxies for the pinned previous release, whose peer health resolver rejects bracketed IPv6 hosts. Preserve signed authorities, ports and strict recovery assertions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): separate pinned startup from MPU fault budgets
Use the old fleet defaults during bootstrap, then apply the original short fault budgets to the current replacement. Keep direct S3 URLs aligned with IPv6 listeners and preserve all quorum and physical cleanup assertions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): seed legacy MPU through transparent transport
Retire raw proxy connections before enabling method faults so legacy streaming setup stays byte-transparent and every later scoped mutation must traverse the fault barrier. Verify raw and adapted protocol authority, bodies and trailers.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): retain direct listeners for legacy MPU peers
Advertise same-port localhost peers for the pinned resolver while retaining direct IPv4 S3 listeners. Exact scoped RPC hit counts still reject bypassed injection and all physical authority assertions remain unchanged.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(server): keep startup gRPC on the authenticated router
Use the hybrid protocol predicate for startup RPC admission so early control responses retain production authentication and owner validation. Keep REST FullReady gating and pinned replay capability rejection intact. Add deterministic regressions, scoped diagnostics and bounded legacy fixture resources.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ci): verify MPU recovery across upgrade source versions
Run the exact sixteen-drive mixed MPU case for every checksum-verified upgrade source, independently of generic compatibility failures. Require one listed test and retain per-version logs while preserving the historical distributed source pin and every recovery assertion.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ci): trigger upgrade MPU checks for fixture changes
Include distributed fixtures and the RPC fault proxy in the upgrade
workflow paths so test-only changes schedule the strict recovery gate.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(storage): verify resend owners across MPU finalization
Hold real part-transaction RPCs while another coordinator completes or
aborts the upload, with both live and cancelled client response waiters.
Require full object bytes, terminal upload state, and all staging roots
to remain consistent after the part owner drains.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(scanner): expose startup phases and attempt deadlines
Share one node-local startup snapshot between native status and metrics.
Track actual waits and first attempts without changing admission timing,
coverage acknowledgement, or publication semantics.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): observe startup recovery before initialization
Create the observation owner at enabled startup entry and move it through
the existing initializer, preserving admission timing and disabled cleanup.
Cover blocked recovery discovery and cancellation through the public entry.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(e2e): tighten scoped multipart fault acceptance
Target Abort root deletion instead of child transaction cleanup, require
actual metadata peer hits, and keep faulted APIs single-attempt. Validate
bounded ordinary recovery and safe quorum refusal for partial mutations,
with the recovery enhancement tracked in backlog#2940.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(e2e): drive concurrent TLS reads independently
Give each coordinator its own supervised task and cooperative polling
budget while retaining the existing workload, deadline and byte/hash
assertions. Preserve node logs after fixture teardown and report the
operation, response phase and received bytes when a transfer times out.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scripts): collect scanner host telemetry on macOS
Use platform-supported iostat arguments and expose optional collector
failures without interrupting scanner-status collection.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(ecstore): reuse validated metadata read path buffers
Share the checked bucket/object tuple across metadata read consumers and borrow clean normalization inputs without skipping filesystem validation. Keep Windows reconstruction and the second bucket check intact.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(scripts): finalize scanner telemetry for failed runs
Initialize host telemetry artifacts before external captures so reused output directories cannot retain prior successes. Share normal and error-path finalization, reap collectors on request failure, and preserve the primary exit status even if status recording fails.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): distinguish unreadable recovery inventories
Treat only the exact system-prefix read-quorum result as incomplete during bounded polling. Preserve strict final reads and the shutdown/drain assertions, record failed observations, and cover interrupted empty streaks and unrelated errors.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* perf(ecstore): avoid head-of-line blocking in delete recovery
Recover independent keys through a bounded rolling window while preserving per-key hint generations, admission budgets, cancellation retry, and the existing transaction decision engine.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(ecstore): bound durable delete preparation and bootstrap recovery
Prepare independent Linux delete transactions with bounded concurrency while retaining per-key result order and serial fallback for overlapping, rollback, and legacy requests. Preserve startup discovery until bucket metadata is initialized, and add release performance and process-crash acceptance runners.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): verify clean exits in recovery acceptance
Reject forced or OOM exits when measuring clean startup and verify original ETags across mixed-layout crash recovery and a second restart.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): distinguish prepared and decided batch crashes
Observe real preparation and receipt states separately, retain non-vacuous acknowledged-delete sentinels, and require read-only physical convergence before S3 reads can trigger cleanup.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): move final delete recovery options
Move the options value into the batch-ordering regression's final case
instead of cloning it. Keep the earlier clone needed for reuse.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): replay delete receipts after interrupted proof GC
Retire only the exact empty transaction leaf before journal activation. Preserve occupied directories and prefix children, and cover partial cross-disk proof GC without a fresh delete. Tighten crash evidence to distinguish directory-only remnants from retained content.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): bound implicit startup delay and publish config early
Limit warm-cache startup to the smaller of the cycle interval and one
minute before jitter, while preserving explicit startup settings.
Publish configuration gauges when applying runtime settings and retain
timeout diagnostics until an actual new cycle begins.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): unify startup waits and skip unchanged configuration
Bound read-only startup inspection and make startup and pre-leadership
retries respond to cancellation and configuration changes. Preserve one
startup deadline and discover replication behind lifecycle-only buckets.
Avoid irrelevant bucket listings and repeated per-disk reconfiguration.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): retain effective startup policy and fallback metrics
Keep the sampled startup deadline when configuration changes leave its
normalized policy unchanged. Initialize retained runtime metrics after
an invalid first refresh without suppressing the validation error or
replacing a newer accepted configuration.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* perf(rpc): remove redundant boxing from read streams
Keep the capped read stream concrete until StreamingBlob owns its type erasure. Add verified four-node TLS coverage for full, unaligned range and concurrent reads, with bounded client memory and explicit integrity checks.
Document protocol comparison commands and the old-node workaround during rolling upgrades.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): preserve TLS trust for alternate credentials
Derive credential-specific clients from the cluster configuration so they retain the generated CA. Exercise successful trusted reads and reject an invalid replacement identity instead of inheriting root credentials.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): collapse acceptance report path guard
Combine the optional report path and existing-file checks into a let chain
without changing the report or PID sidecar collision behavior.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(e2e): add MPU RPC quorum and recovery acceptance
Exercise separate storage and lock RPC delays across multipart phases,
including quorum boundaries, cancellation, bucket recreation, and bounded
eight-node load. Honor TMPDIR for isolated cluster drive directories.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(lock): probe healthy voters behind silent peers
Spread additional probes across the existing acquisition attempt budget
when the initial quorum-plus-spare wave stalls. Preserve quorum thresholds,
fast-path fan-out, and existing ownership and late-response cleanup.
Cover every four-node read-voter ordering, a stalled eight-node write
quorum, and normal fan-out. Extend MPU acceptance to EC12+4 at twelve and
eleven metadata acknowledgements.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Hauser