Retain the reviewed diagnostic dependency while starting request-stage work
from current main. This integration is not a performance qualification.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): identify stale multipart cleanup failures
Preserve quorum failure metrics while avoiding an UploadPart warning for background cleanup. Keep the existing write quorum and source recovery checks intact.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(ecstore): skip redundant stale upload part scans
Once the default deadline is due, lifecycle rules cannot make an upload ineligible. Avoid querying lifecycle and listing parts in that case while preserving the locked recheck.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): validate multipart lifecycle size
Skip part scans when no abort rule uses size. Treat unavailable, incomplete or overflowing sizes as unknown and keep size-independent lifecycle rules eligible.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): select the latest version at read quorum like MinIO
GET and HEAD escalated latest-version selection to write quorum when
enough metadata answered and skipped a newer version below write quorum
on degraded reads. LIST and heal already pick the newer side of a split
write at read quorum, so such keys listed but answered 503 until heal.
Select with MinIO's getObjectFileInfo rule: the most common mod time
that reaches the object's data blocks wins, ties go to the newer time,
and one content identity at that time must reach the same quorum. Two
identities tied at the selected time fail instead of resolving by hash
order, and the selected identity's version-stack shape follows disk
order.
* fix(ecstore): resolve selection ties by disk order and guard writes
Review of the read-quorum selection found three gaps:
- Rejecting a same-time half split made explicit-version reads of a
partially applied tagging update fail and blocked the retry that
repairs it. Resolve ties to the identity on the lowest-index disk,
for reads and heal alike, instead of rejecting or using map order.
- Delete markers and empty versions can lower the common read quorum
below the selected version's data blocks, so HEAD could report a
version GET cannot decode. Require that many copies, as MinIO's
getLatestFileInfo does.
- A selection can now return exactly read-quorum disks, and partial
metadata writes are not rolled back. Refuse an update before writing
when its target disks cannot reach write quorum.
* fix(ecstore): count identity copies in the data-blocks guard
The first-candidate shortcut counts byte-identical copies, so copies
that differ only in identity-neutral fields such as replication status
were missing from the data-blocks check. When the shortcut's count
falls short, count every copy of the selected content identity before
refusing the version.
* fix(ecstore): keep commit-proof selection for system-bucket reads
Pool metadata's PREPARE/COMMIT recovery and identity epochs treat an
ordinary read of pool.bin as proof that the record committed. Under the
read-quorum rule, a COMMIT that reached only half of a 2+2 set was served
as committed, and the pool_meta_native_physical_* integration tests
failed.
Route reads through select_fileinfo_for_read: S3 buckets keep MinIO's
read-quorum rule, while is_meta_bucketname reads use the previous
algorithm unchanged as select_committed_fileinfo, so a split stays
unreadable until native recovery resolves it.
* fix(ecstore): serve the latest readable version past dangling versions
A failed versioned PUT whose rollback also failed can leave the new
version on top of fewer drives than its data blocks. Reading only each
drive's top version then left the committed version below it short of
read quorum, so GET and HEAD of the latest version answered 503 although
that version is intact in every stack. MinIO resolves the stacks in
pickLatestQuorumFilesInfo.
When every drive answered and an S3 latest read still cannot select a
version, walk the newest version headers of each stack through
exact-version reads: serve the readable version with the newest selected
modification time, skip versions whose exact read proves them dangling
(the NoSuchVersion an explicit GET reports) and queue heal for them, and
keep the read-quorum error on any other outcome. Candidates are ordered
by each version's newest modification time across drives, and at most
eight exact reads are spent.
Test PUTs now wait for every drive so tests that edit one drive's
metadata no longer race the PUT's background tail writes.
* fix(ecstore): resolve dangling latest versions in metadata updates
A failed versioned PUT whose rollback also failed can leave the new
version on fewer drives than its data blocks. GET and HEAD of the
latest version already look past such versions, but read-before-write
callers without a version id still answered 503: object tagging read
each drive's top version with the quorum-error read, and
put_object_metadata and the CopyObject source read selected from
read_all_xl, whose merged latest version is missing on the other
drives, so object_quorum_from_meta failed before selection.
Object tagging now reads through a variant that runs the same walk past
dangling versions as a latest GET while keeping the quorum error for a
dangling key. put_object_metadata and the CopyObject source try the walk
when their quorum or selection step fails with read quorum, before the
existing dangling cleanup could report a key with a committed version
absent. A version-only copy takes the served version's own copies and
write quorum. The walk still applies only to S3 buckets, latest reads,
and sets where every drive answered, and the update still targets only
the served version's disks, so a version below write quorum is refused
before writing.
* fix(ecstore): refuse version-only copies below write quorum up front
Review of the dangling-latest metadata updates found three gaps:
- A version-only CopyObject wrote the new version and reverted on a
write-quorum miss; refuse it before writing when the served version's
disks cannot reach its write quorum, like update_object_meta_with_opts.
- Replication status write-back, including the zero-write ValidateOnly
mode, now keeps its read-quorum error and never walks past dangling
versions, since the walk queues heal for the versions it skips.
- The erasure-coding doc overstated that these callers never report a
dangling key as absent; put_object_metadata and CopyObject keep their
existing dangling cleanup when the walk finds no readable version.
Omit span metadata and inferred parent IDs for diagnostic root events. Verify per-shard index order and document existing production metric timing.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
Record local batch eligibility, blocking queue and execution time, shard
outcomes, and actual read bytes behind the existing GET-stage metrics gate.
Keep captured metric handles with the blocking worker after cancellation
and use scoped low-latency histogram buckets for the new duration series.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(rebalance): reuse the target pool holding a dangling copy
Rebalance places a version with a fresh free-space choice whenever the
exact-version pool lookup finds no target copy. An attempt that loses
write quorum can leave the version on fewer drives than read quorum,
and object reads report such a copy as absent (#8963), so the retry
could commit the version on a different pool and leave it in two. The
distributed kill-before-source-cleanup case saw this: a delete marker
whose first copy failed with InsufficientWriteQuorum ended on pools 1
and 3 after the source was cleaned up.
Before the free-space fallback, data movement without a decommission
capacity owner now targets an eligible pool where any drive already
holds the exact version, so the retry rewrites that copy to full
quorum instead of creating a second one. This covers both the object
copy path and the delete-marker path.
* fix(rebalance): check free space before reusing a holding pool
The pool holding an earlier copy was returned before any space check,
so a full holder pinned every retry and could stall a deferred entry.
The holder now passes through the same filters as the free-space
choice (suspended, rebalancing and source pools, has_space_for for the
expected size, and the max-used filter), computed once per selection;
placement falls back to that weighted choice when the holder does not
qualify. The two free-space helpers share one implementation.
The tests no longer rely on random placement. A test-only override
pins each target pool's reported capacity so the free-space choice
always picks the non-holder pool, the earlier copies keep the version
ids but carry different bytes and modification times, and the moved
versions are checked for marker state, modification time, data on
every drive and readable bytes. A second test covers a full holder.
* chore(ci): shrink the error(format!) ratchet baseline for rebalance
The rebalance control and meta modules on main now have fewer
Error::other(format!) call sites than the baseline records, which
fails the shrink-only ratchet in Quick Checks. Record the smaller
counts.
* fix(ecstore): select the latest version at read quorum like MinIO
GET and HEAD escalated latest-version selection to write quorum when
enough metadata answered and skipped a newer version below write quorum
on degraded reads. LIST and heal already pick the newer side of a split
write at read quorum, so such keys listed but answered 503 until heal.
Select with MinIO's getObjectFileInfo rule: the most common mod time
that reaches the object's data blocks wins, ties go to the newer time,
and one content identity at that time must reach the same quorum. Two
identities tied at the selected time fail instead of resolving by hash
order, and the selected identity's version-stack shape follows disk
order.
* fix(ecstore): resolve selection ties by disk order and guard writes
Review of the read-quorum selection found three gaps:
- Rejecting a same-time half split made explicit-version reads of a
partially applied tagging update fail and blocked the retry that
repairs it. Resolve ties to the identity on the lowest-index disk,
for reads and heal alike, instead of rejecting or using map order.
- Delete markers and empty versions can lower the common read quorum
below the selected version's data blocks, so HEAD could report a
version GET cannot decode. Require that many copies, as MinIO's
getLatestFileInfo does.
- A selection can now return exactly read-quorum disks, and partial
metadata writes are not rolled back. Refuse an update before writing
when its target disks cannot reach write quorum.
* fix(ecstore): count identity copies in the data-blocks guard
The first-candidate shortcut counts byte-identical copies, so copies
that differ only in identity-neutral fields such as replication status
were missing from the data-blocks check. When the shortcut's count
falls short, count every copy of the selected content identity before
refusing the version.
* fix(ecstore): keep commit-proof selection for system-bucket reads
Pool metadata's PREPARE/COMMIT recovery and identity epochs treat an
ordinary read of pool.bin as proof that the record committed. Under the
read-quorum rule, a COMMIT that reached only half of a 2+2 set was served
as committed, and the pool_meta_native_physical_* integration tests
failed.
Route reads through select_fileinfo_for_read: S3 buckets keep MinIO's
read-quorum rule, while is_meta_bucketname reads use the previous
algorithm unchanged as select_committed_fileinfo, so a split stays
unreadable until native recovery resolves it.
* fix(ecstore): serve the latest readable version past dangling versions
A failed versioned PUT whose rollback also failed can leave the new
version on top of fewer drives than its data blocks. Reading only each
drive's top version then left the committed version below it short of
read quorum, so GET and HEAD of the latest version answered 503 although
that version is intact in every stack. MinIO resolves the stacks in
pickLatestQuorumFilesInfo.
When every drive answered and an S3 latest read still cannot select a
version, walk the newest version headers of each stack through
exact-version reads: serve the readable version with the newest selected
modification time, skip versions whose exact read proves them dangling
(the NoSuchVersion an explicit GET reports) and queue heal for them, and
keep the read-quorum error on any other outcome. Candidates are ordered
by each version's newest modification time across drives, and at most
eight exact reads are spent.
Test PUTs now wait for every drive so tests that edit one drive's
metadata no longer race the PUT's background tail writes.
* chore(ci): shrink the error(format!) ratchet baseline for rebalance
The rebalance control and meta modules now have fewer
Error::other(format!) call sites than the baseline records, which
fails the shrink-only ratchet in Quick Checks. Record the smaller
counts.
* fix(ecstore): preserve canonical order in latest-version fallback
Keep the diagnostic implementation private to its owner and expose explicit feature-gated helpers through the existing storage boundary.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(decommission): cover held target lock and partial-copy resume
Port two tests from the closed lock-recovery branch onto the deferral
that defers any typed namespace lock timeout on a version copy.
- A version commits, then every attempt at the next one times out on
the target lock: the entry is deferred as a lock-timeout deferral, no
failed item is counted, both source versions stay, and the resumed
entry moves both versions with their exact bytes.
- A real write lock held on the target key makes target selection
return a typed object-lock timeout, the entry defers instead of
failing, and once the lock is released selection recovers and the
resumed entry completes the move.
* chore(ci): shrink the error(format!) ratchet baseline for rebalance
The rebalance control and meta modules on main now have fewer
Error::other(format!) call sites than the baseline records, which
fails the shrink-only ratchet in Quick Checks. Record the smaller
counts.
Keep timing and request correlation in a separate default-off feature. Preserve the existing positioned-read backend, byte and error contracts.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(sftp): allow bucket-scoped root listings
Listing the SFTP root required s3:ListAllMyBuckets. Without it a
bucket-scoped user got PermissionDenied on `ls /`; with it they saw every
bucket on the instance. S3 ListBuckets and the WebDAV root (#6298)
already fall back to a per-bucket filtered listing.
When the global ListAllMyBuckets check is denied, list buckets through
the session-aware ListBuckets path instead, so IAM filters the result to
buckets the principal may ListBucket or GetBucketLocation. Principals
with ListAllMyBuckets keep the unfiltered listing, and IAM outages and
timeouts still fail as before.
list_buckets_for_session and its request builder were WebDAV-only; they
are now also compiled with the sftp feature.
* fix(protocols): re-check the session key for bucket-scoped listings
list_buckets_for_session built its request from the identity captured at
login. Through the SFTP root-listing fallback, a service account disabled
or expired after login could keep listing bucket names, unfiltered when
its parent may list every bucket.
Validate the session key with check_key_valid_with_context, as
authorize_protocol_request does for every other protocol operation, build
the request from the fresh credentials and is_owner, map an invalid key
to AccessDenied, and attach the server context.
The SFTP fallback now passes the session's request headers and transport
security, like every other SFTP authorization. DummyBackend fails closed
when no session listing is queued, matching the trait default, and gains
a stall_authorization hook. New tests cover a service account disabled
mid-session, IAM outage and timeout on the global check (no fallback),
non-AccessDenied backend errors, truncation of fallback output and the
fail-closed default.
* test(protocols): compare error code without referencing s3s
The session-listing test compared against s3s::S3ErrorCode, which made
policy_test.rs a new s3s-referencing file and tripped the s3s footprint
ratchet (210 files against a baseline of 209). Compare the code's string
form instead.
* fix(protocols): refresh session bucket discovery without stale copies
Build bucket listing requests directly from freshly validated credentials
and server-scoped ownership instead of cloning and replacing login state.
Reuse the same authorization path for WebDAV capacity discovery.
Preserve typed access-denied checks through the protocol facade, and cover
request conditions, session-policy updates, revocation, listing bounds,
and fallback deadlines with regression tests.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
A node killed mid-copy leaves its namespace lock on the peers until the
lock lease expires. When the restarted leader resumes the decommission
and reaches that key inside the lease, every in-place copy attempt times
out acquiring the lock, and the version was counted as a failed object,
so the pool decommission could no longer complete.
A lock acquisition timeout leaves the source in place, like a version
copy that lost drive quorum, so defer the entry to the end of the round
instead of counting it as failed. A replay that still times out pauses
the decommission through the existing durable-pause path, which is
relaunched automatically.
This is the intermittent failure of the distributed e2e case
scanner_heal_topology_decommission_kill_before_source_cleanup_resumes_without_loss.
* fix(ecstore): select the latest version at read quorum like MinIO
GET and HEAD escalated latest-version selection to write quorum when
enough metadata answered and skipped a newer version below write quorum
on degraded reads. LIST and heal already pick the newer side of a split
write at read quorum, so such keys listed but answered 503 until heal.
Select with MinIO's getObjectFileInfo rule: the most common mod time
that reaches the object's data blocks wins, ties go to the newer time,
and one content identity at that time must reach the same quorum. Two
identities tied at the selected time fail instead of resolving by hash
order, and the selected identity's version-stack shape follows disk
order.
* fix(ecstore): resolve selection ties by disk order and guard writes
Review of the read-quorum selection found three gaps:
- Rejecting a same-time half split made explicit-version reads of a
partially applied tagging update fail and blocked the retry that
repairs it. Resolve ties to the identity on the lowest-index disk,
for reads and heal alike, instead of rejecting or using map order.
- Delete markers and empty versions can lower the common read quorum
below the selected version's data blocks, so HEAD could report a
version GET cannot decode. Require that many copies, as MinIO's
getLatestFileInfo does.
- A selection can now return exactly read-quorum disks, and partial
metadata writes are not rolled back. Refuse an update before writing
when its target disks cannot reach write quorum.
* fix(ecstore): count identity copies in the data-blocks guard
The first-candidate shortcut counts byte-identical copies, so copies
that differ only in identity-neutral fields such as replication status
were missing from the data-blocks check. When the shortcut's count
falls short, count every copy of the selected content identity before
refusing the version.
* fix(ecstore): keep commit-proof selection for system-bucket reads
Pool metadata's PREPARE/COMMIT recovery and identity epochs treat an
ordinary read of pool.bin as proof that the record committed. Under the
read-quorum rule, a COMMIT that reached only half of a 2+2 set was served
as committed, and the pool_meta_native_physical_* integration tests
failed.
Route reads through select_fileinfo_for_read: S3 buckets keep MinIO's
read-quorum rule, while is_meta_bucketname reads use the previous
algorithm unchanged as select_committed_fileinfo, so a split stays
unreadable until native recovery resolves it.
- Resolve existing directories below base on Linux in a single openat2(BENEATH | NO_SYMLINKS) syscall in open_existing_directory_below_base_std instead of opening and closing every ancestor component.
- Reuse the pinned source directory guard across sibling metadata reads and control probes in validate_source_mutation, load_source, OrdinaryMultipartDeleteCheck::validate, and prepare_multipart_commit_staging.
- Read normal multipart source heads (<= 64 KiB) in a single pass in load_source while preserving constant-memory streaming observation for large prepared histories.
- Record root_missing during load_source, reuse loaded.original in restore_original, and coalesce finish source directory fsync into persist_with_source_sync.
- Reuse the opened commit directory in prepare_rename_commit_guard and LinuxCommitDirectories::validate when source_parent == destination_parent.
* test(scanner): add memory acceptance runner and budget probes
Add owned S3 workload phases with process, endpoint, cgroup and binary
identity checks for scanner memory and scanner-disabled request bursts.
Keep scale, restart, heap-attribution and overall-acceptance limits explicit.
Characterize healthy and seeded coverage-gap cache retention through the
real folder walker and production cache codec without changing runtime code.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): bind acceptance capacity to effective storage
Validate the serving process's complete local disk inventory before load
writes and during runtime sampling. Resolve each disk inside the owned
storage root, reject nested mounts and filesystem mismatches, and bind
storage observations to phase identity for container and native runs.
Cover the unused-mount bypass, CLI-overridden environment, native paths,
symlink escapes, named volumes and stable multi-disk inventory ordering.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): fence status observations before workload writes
Close each status API window with a second storage-bound process sample
before Phase entry can authorize bucket creation or PUT requests.
Add a fail-before regression for a process change during the initial status
request, asserting no ownership or workload writes occur.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Retain original prepare and commit-settlement RPCs beyond failed replacement and Abort cleanup, with and without bucket recreation. Require typed refusal or a physically verified idempotent result, unchanged fresh-upload bytes and complete object reads.
Extend scoped proxy observations without rewriting signed frames or trailers, and preserve registered test membership and existing budgets.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Verify a sixteen-drive set rejects reads and writes after two nodes stop,
then resumes complete S3 reads and new writes when only one node returns.
Keep the fourth node stopped throughout checks from all three live
coordinators and verify the original 12+4 shard geometry on disk.
Let the existing GET recovery probe retry the compatible SlowDownRead
response only with HTTP 503, retaining fatal status and byte checks.
Register the case in both platform selections and document the boundary
between process recovery and physical disk reattachment acceptance.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
- Document rules in crates/ecstore/AGENTS.md and the adversarial performance lens against per-disk spawn_blocking hopping, parent-chain openat amplification, un-sized read_to_end buffers, and redundant state reloads under the same lock.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d85aaf47-2a4b-4c2c-97a3-dbe8c7717e2a
* fix(observability): correlate physical namespace lock releases
Map logical namespace guards to their actual granted peer lock IDs and local physical guard identities. Record physical completion only after successful manager release and registry removal, and record validated positive remote unlock responses without changing release behavior.
Exercise real distributed ownership, asynchronous cleanup, duplicate and disabled guards, stale-owner retirement, and explicit release followed by Drop. Document the required diagnostic filters and the remaining filesystem-tail boundaries.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): remove unused lock manager test import
The diagnostic regression uses inherent manager methods and does not need the LockManager trait import. Remove it to pass strict warning checks.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): compact completed subtrees during checkpoint sweeps
Release completed object entries while preserving resumable frontier nodes,
including heal rewalks and coverage gaps. Borrow child links during usage
aggregation and remove duplicate update-cache subtree copies.
Add real-disk restart and allocation-retention regressions for #8961.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(scanner): avoid unused progress trees and traversal copies
Bind progress caches to actual subscribers and keep durable scanner metadata
in one authoritative cache. Remove mirror-only heal synchronization, use one
tagged folder queue, and consume child ownership when deleting subtrees.
Preserve observer output, restart coverage and failed-object retry semantics.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(scanner): compact completed leaf entries in place
Keep completed-folder eligibility and frontier handling in one compaction
boundary. Borrow leaf statistics and update the cached sampling flag without
cloning, deleting, or reinserting the entry; aggregate only non-leaf nodes.
Pin allocation identity, complete metadata, failure guards and caller state
with a regression that fails under the previous compaction implementation.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
A cursor for a key that literally ends in a legacy-shaped tag is issued
wrapped in one extra `[rustfs_cache:v2,return:]`, because the listing layer
still strips one legacy tag from incoming markers. ListObjects (V1)
unwrapped it in `calculate_next_marker`; the client then sent the bare key
back as `marker`, the listing layer stripped the key's own tail, and the
next page restarted at the stem and returned the same key forever. Keep the
wrapper in V1 NextMarker and cover the V1 resume path (marker sent back as
StartAfter) on a real store.
Document that sampled list quorum modes can also omit committed objects
whose copies mostly sit outside the sample, and recommend `strict` for
backup, sync, replication and lifecycle tooling.
- Coalesce InternalBuf::Ref and InternalBuf::Owned in write_all_internal
onto a single blocking_queue operation, avoiding 3-5 extra spawn_blocking
round-trips per disk on metadata writes.
- Short-circuit recover_part_transaction when no disk reports an active
part transaction (new.meta), skipping redundant rollback and part.N.meta
reads on normal UploadPart calls.
- Pre-allocate exact buffer capacity from file.metadata().len() in
read_file_all and read_multipart_metadata, reducing per-file read()
syscalls to 1 and avoiding Vec growth reallocations.
- Collapse os_mkdir_all and make_dir_all to a single fast-path blocking
call via create_dir_all_std.
- Accept &Path in BucketExistenceCache and use cached_access in
rename_file, create_file, and read_version.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d85aaf47-2a4b-4c2c-97a3-dbe8c7717e2a
* chore(deps): upgrade russh to 0.65.0 and refresh crates
Run two cargo update and cargo upgrade rounds from origin/main.
Adapt SFTP public-key callbacks to PublicKeyOffer and VerifiedPublicKey,
preserving explicit rejection and password-only authentication.
Cover the rejection policy with a pinned-host-key SSH handshake test.
Refresh Arrow, Parquet, Brotli, ZIP, and compatible dependencies.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(deps): route legacy zstd consumers to fixed codecs
Replace the old registry implementation with a private version-constraint
adapter backed only by zstd 0.14.3 and zstd-safe 8.2.0. Preserve legacy
features and Avro, Arrow IPC, Parquet, and Kafka formats.
Cover abandoned contexts, legacy decode errors, seekable bounds, stored
frames, consumer interoperability, and resolved implementation versions.
Record the adapter removal condition in the compatibility register.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): renew faulty local drives after partial recovery
Make local online detection honor the same faulty gate as foreground I/O
and remote drives, allowing the existing reconnect path to replace a
stranded handle once its original format is readable again.
Cover partial EC:4 recovery, health-state admission, and preserve explicit
test faults across handle renewal.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(ecstore): reconnect set endpoints concurrently
Poll independent endpoint recovery futures within the existing maximum
erasure-set width so a stalled format probe cannot delay a ready drive.
Keep outer-set traversal and per-slot validation and publication unchanged.
Remove repetitive per-drive reconnect-loop info logs.
Add an authenticated two-peer regression that holds the first format read
while the second slot is admitted and serves data with its validated ID.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* refactor(listing): list latest versions with MinIO's listing quorum
Latest-version SetDisks::list_path now follows MinIO's er.listPath:
ceil(asked/2) is the listing quorum for min_disks, directories and
objects, agreed entries are emitted as read, and a split sample goes
through MetaCacheEntries::resolve. One object can no longer fail a
page. A set with fewer than half of its drives online still fails the
page with ErasureReadQuorum.
Remove the write-quorum layer this replaces: ListingSupplement,
ListingReconciler and the LIST object read lock, the latest_listing_*
quorum helpers, resolve_agreed_entry, resolve_with_write_quorum and the
enforce_write_quorum / write_quorum_slack merge branches, and the
per-version listing quorum helpers in filemeta.
Make merge_file_meta_borrowed_streams pick a version id's most common
header regardless of reader order, so an interrupted null-version
overwrite no longer drops the committed generation from the merge.
* test(listing): do not assert sampled optimal listings are deterministic
optimal asks a random half of the drives, so an EC:8 object committed on nine
of sixteen drives can miss the four-copy listing quorum in roughly one sample
in six, exactly as in MinIO. Keep the deterministic assertion for strict and
for optimal when every online drive is asked.
* fix(storage): 404 for dangling objects and strict list quorum default
Stage F.0 of the ListObjects quorum plan. Once listing returns to MinIO's listing quorum, a PUT that reached half of the drives but not read quorum becomes visible in LIST, so GET and HEAD must not answer it with a retryable 503.
GET, HEAD and prepared GET metadata now report a metadata read-quorum failure as ObjectNotFound or VersionNotFound when every drive answered with valid metadata of a single generation or not-found and is_object_dangling proves the object or version dangling, matching MinIO's getObjectFileInfo. Offline or unanswered drives, timeouts, corrupt xl.meta, two generations, remote versions and the meta bucket keep the read-quorum error. Nothing is deleted on the read path; heal keeps owning cleanup and its grace window. Read-before-write callers of get_object_fileinfo keep the quorum error.
RUSTFS_LIST_OBJECTS_QUORUM now defaults to strict, matching MinIO, and the operations doc explains that optimal samples drives and can briefly show deleted objects.
A new test pins the write-path precondition: a PUT that misses write quorum leaves no new version or data directory on any drive once its rollback completes.
* test(listing): prove the list quorum env var is read with a non-default value
* perf(ecstore): optimize cold HEAD and GET metadata read paths
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d85aaf47-2a4b-4c2c-97a3-dbe8c7717e2a
* perf(ecstore): cache verified inline body and avoid early-stop allocations
- Store verified_inline_body on GetObjectMetadataCacheEntry so warm GET
cache hits serve the pre-verified inline Bytes payload in O(1) without
re-running HighwayHash256S bitrot verification across shards.
- Verify inline shard slices directly in verify_inline_body_from_observations
without acquiring MODERN_REED_SOLOMON_CACHE or allocating InlineBitrotReaders.
- Gate MetadataFanoutDiagnostics timers and latency sorting on
get_stage_metrics_enabled().
- Avoid pinning full xl.meta buffers in GetObjectMetadataCacheEntry by
using stack Uuid::encode_buffer() lookup without Bytes::slice_ref.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d85aaf47-2a4b-4c2c-97a3-dbe8c7717e2a
* fix(ecstore): restore inline verification test context
---------
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
Copilot-Session: d85aaf47-2a4b-4c2c-97a3-dbe8c7717e2a
Version listings bypassed listing quorum: SetDisks::list_path returned
early into list_versions_authoritatively, whose resolver seeds unread
slots as DiskNotFound. With any one drive of the set offline, every
version held by fewer than ceil(n/2) drives (for example a deleted
version kept by drives that were offline during the delete) failed the
whole page with ErasureReadQuorum, surfaced as S3 503.
Route merged version pages through the listing-quorum branch of
SetDisks::list_path, as MinIO's listPath does with requestedVersions =
0, so such versions are omitted instead of failing the page. Keep the
set-level floor: fewer than ceil(n/2) online drives still fails the
page.
Set walks (tier reference proofs, tier-delete journals, pool
maintenance) and the recursive prefix-delete scan act on absence, so
they keep the authoritative per-version scan and fail closed.
* refactor(listing): make cursors plain keys and opaque V2 tokens
RustFS inherited MinIO's metacache cursor tag and appended
`[rustfs_cache:v2,id:<uuid>,src:walker,gen:live]` to every internal
continuation marker, although nothing ever consumed the id: four parsers
had drifted apart, V1 NextMarker had to be stripped on the way out, and
ListObjectVersions leaked the tag onto the wire as NextKeyMarker.
ecstore stops assigning `list_id` in `ECStore::list_path` and
`SetDisks::list_path_result`, drops `append_list_cache_id_to_marker` and
the `cache_id` parameters of the pagination helpers, and removes the
unused `ListPathOptions::encode_marker`; internal markers are the plain
last key, last common prefix or last raw scanned key.
The API layer builds the V2 NextContinuationToken as base64 of the
server-side cursor in one place and decodes it in one place; V1
NextMarker is the decoded cursor and the `public_list_marker` tag
stripper is gone; NextKeyMarker is the plain key. Base64 of the plain
key is what nodes from earlier releases already read, so a rolling
upgrade needs no switch.
Compatibility, registered as `listing-cursor-legacy-tag`: `parse_marker`
and `list_marker_key` still strip a trailing legacy tag from incoming
markers so a client that paginates across the upgrade keeps its position.
* fix(listing): preserve literal cache-tag cursor boundaries
* test(e2e): wait for known quota reporting snapshots
Wait for the expected numeric usage before asserting quota statistics or
projected checks. Preserve null as unknown and keep the existing quota
assertions, authorization failures, and bounded convergence deadline.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): refresh full-suite membership locks
Regenerate the Darwin and Linux e2e-full digests from compiled nextest
listings after merging main. Both selections retain all base members
and add exactly the five quota reporting regressions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(s3-client): make range preflight assertion deterministic
Assert that an overflowing range returns its specific InvalidInput error
on the first poll instead of waiting for an idle shared TCP port.
This avoids unrelated connections affecting the local-validation oracle
and removes the listener-permission skip from this test.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): make lock error metric fixture deterministic
Pass a typed quorum failure through the existing admission result resolver
so the lock-error metric test does not race a real 250 ms acquire deadline.
Preserve the exact metric assertions and verify the typed storage error.
Existing acquisition tests retain distributed quorum and timeout coverage.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): isolate durable I/O regression fixtures
Reserve nextest capacity for eight native storage regressions without
changing deadlines, retries, or internal concurrency. Observe physical
publication independently of expired async wrappers, and stop endpoint
reconnection before the fenced format-heal fixture takes ownership.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test: isolate drive provenance and receipt fixtures
Use the existing Linux shared-memory fixture pattern for the bounded
drive CLI provenance probe, retaining its one-second request and exact
binary/signature checks. Seed the full 1001-receipt pagination fixture
one durable source/target pair at a time instead of issuing 32 at once.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(e2e): read publication logs incrementally
Retain the complete matching lock history while reading only appended bytes after an initial scan before remote DELETE is held. Keep the original field projection, file ordering, polling cadence, lock deadlines and same-attempt assertions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): remove redundant publication fixture clone
Serialize the existing root by reference through json! without cloning the Value. Keep all compatibility rows and assertions unchanged.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): avoid per-set max_entries truncation before cross-set prefix usage merge
compute_prefix_usage applies the caller's max_entries cap to each individual erasure set's loaded.prefix_usage(bucket, prefix, max_entries) before merging sub_prefix_map across sets, silently dropping and under-counting sub-prefix usage in multi-set deployments.
Root Cause:
In crates/scanner/src/prefix_usage.rs, compute_prefix_usage queries each erasure set with loaded.prefix_usage(bucket, prefix, max_entries) (line 178) and then merges the returned query.sub_prefixes across sets into sub_prefix_map (lines 216-218). When sets_total > 1 and the number of distinct child prefixes exceeds max_entries, any sub-prefix that ranks within the top max_entries on one set and outside the top max_entries on another set has its non-top set contributions discarded prior to merging, producing silently under-counted size, objects, versions, and delete_markers.
Fix:
Pass usize::MAX as the per-set limit when sets_total > 1 in compute_prefix_usage so each erasure set returns its full sub-prefix breakdown, and apply max_entries truncation only after merging and sorting sub_prefix_map across all sets.
Signed-off-by: czcbangkai <czcbangkai@google.com>
* test(scanner): use snake_case in prefix_usage regression test name
---------
Signed-off-by: czcbangkai <czcbangkai@google.com>
Co-authored-by: Hauser <housemecn@gmail.com>
* perf(ilm): back off idle tier-delete journal recovery
Use bounded idle backoff only after complete empty journal and dispatch scans, while preserving immediate notifications and the retry cadence for observed work.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): reconcile interrupted pool metadata publication
Bind pool metadata CAS and Undo to exact native predecessors. Recover an interrupted physical publication only from an ordinary committed authority and complete verified identity, current and backup evidence, then require normal V3 CAS and quorum readback.
Preserve known-conflict gates, startup and heal lock ordering, targeted admission and the original kill-recovery deadline. Wake proof-ready recovery immediately and bound parallel capacity sampling without removing mutation fences.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(ecstore): reuse proven pool metadata durability
Publish native rollback backups directly in their owned UUID directories. Reuse a one-shot data-durability receipt only for an exact hardlink to the pinned, fully synchronized head and root.
Preserve new-head and directory synchronization, native predecessor CAS, conditional Undo and uncertainty gates. Keep unknown, copied, raw-parent, cancelled and failed operations cold. Add bounded diagnostics, sync-call regressions and path-isolated test hooks.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(multipart): exercise selective RPC delay and peer loss
Add authenticated method-entry barriers for metadata and lifecycle locks under e2e-test-hooks. Verify real dispatch, write quorum, prior part payloads, lock drain and incarnation isolation without changing production budgets.
Record cancellation separately from normal exits so late filesystem completion cannot overwrite Drop evidence.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* docs: define multipart delay gates and receipt invariants
Document RPC entry evidence, peer-loss and cancellation oracles, remaining incident attribution limits, and immutable head inodes required for durability receipts.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(readiness): expose missing runtime drive slots
Report configured slots without a runtime handle separately from unavailable inventoried drives. Preserve quorum and probe status semantics. Wait for every coordinator to regain its complete live inventory before multipart RPC fault windows and single-attempt recovery oracles.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): retry committed promotion convergence safely
Require peer reload convergence on every worker attempt, including unchanged retries. Recheck durable identity, replicas, namespace fence, generation and cancellation before allowing any data movement.
Keep known committed promotions active on a verified propagation pause. Release precise owners, retain queued successors and reuse the existing watchdog. Preserve uncertainty gates and terminal operation behavior.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): guard native metadata mutation aliases
Reject protected RenamePart source and destination paths before I/O. Classify the protected namespace through the native path resolver, and require canonical conditional RenameData and Undo keys so aliases cannot create separate lease domains.
Preserve ordinary multipart and canonical CAS behavior, warm durability receipts and zero-allocation canonical fast paths. Cover real alias, byte identity, receipt and normal mutation behavior.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): register Darwin multipart RPC delay gates
Update distributed membership from the actual Darwin listing. Preserve the full-profile digest and leave Linux unchanged until its own listing is produced, as required by the contributor guide.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(multipart): fence native publication and recovery
Publish multipart data through private hardlinks and atomically consume the
prepared source head. Bind exact compensation capabilities and native recovery
to the physical cohort, source generation and durable publication state. Require
fresh terminal confirmation for native replay and preserve indeterminate data
when no safe recovery witness exists.
Release logical guards only after validated irreversible quorum success while
retaining physical mutation leases, accounting and resource ownership through
actual callbacks. Preserve local instance proof authority and historical Copy
metadata without granting healed donors fresh publication authority.
Keep general movement identity strict while preserving directory capacity
compatibility in a separate checked comparison.
Cover native fault and cancellation boundaries, zero-byte objects, real Copy,
heal and migration, exact physical readback and distributed RPC delay. Retain
all quorum and durability requirements and document rollout limitations.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): fix ratchet scan and Windows fault hooks
Move native pool metadata inventory tests after production items so the
error-format ratchet sees the existing call site. Preserve the baseline
instead of accepting a false count decrease.
Make the directory-barrier failure hook available on every test platform
and exercise the parent-barrier failure/retry cases in Windows CI. Keep
production synchronization behavior and all existing assertions intact.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(protos): remove redundant recovery request clone
Move the baseline into its final struct update while retaining the earlier
clone needed to construct the first request variant.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): align RPC delay gates with native abort
* fix(storage): validate native source recovery and CI gates
Preserve wire layouts while bounding native response sizes and fix the
last-use recovery request clone. Avoid destination replay reads only for
fully observed initial native sources; keep upgraded and recovered sources
conservative. Cover real legacy publication after a cold restart.
Align upgrade acceptance with the native mixed-version mutation pause and
use exact upload-scoped test completion evidence before advancing time.
Keep all original quorum, ownership, byte and deadline assertions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): verify native abort under partial admission
Delay native cleanup at the exact upload root after all-participant closure.
Verify quorum-only admissions retain every source byte on refused Abort and
complete the same upload without creating a missing original participant.
Keep sub-quorum orphans and their discoverable identities isolated while a
fresh all-participant upload proves service recovery. Route shared constants
through the existing E2E storage facade.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): avoid formatting a fixed invalid upload path
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): align native MPU fault recovery assertions
Verify real source participants, preserved orphan bytes and same-upload completion after quorum-only admission. Require full physical inventories so an expected read-quorum error cannot hide unrelated leakage. Preserve ordinary quorum and recovery budgets, and document listing availability through the affected set.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): keep MPU tail deadlines on the wall clock
Global Tokio time advancement can expire drive-health probes while real
filesystem callbacks are pending. Use a short real drive deadline with the
existing physical-entry and cleanup barriers, retaining the byte, lock,
resource-permit and final-shard assertions.
Fixes#8937
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(storage): align native fixtures with instance ownership
Open independent disks for the simulated capacity-loss instance, scope four coordinator journal disks without rebinding source authority, and probe configured namespace keys while retaining physical I/O hooks.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): isolate native multipart future construction
Construct optional native publication futures behind synchronous heap boundaries so ordinary rename callers do not carry their state machines. Preserve native guards and ownership without adding allocations to ordinary PUT, copy or heal branches.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(heal): assert coordinator journal scope after rejoin
Check the exact original four-member journal map after every rejoin and manager stop while preserving all sixteen native data slots. Reconnect expansion fails the test rather than being filtered away.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(heal): bound object repair future construction
Construct the normal and FileCorrupt deep-retry state machines behind a synchronous heap boundary. Preserve guards and cancellation while bounding the shared rename and deletion call stacks without increasing test stacks.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): qualify fixture disk identity checks
Call the same DiskAPI method explicitly so the independently reopened disk identity assertion compiles under every target and feature combination.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): isolate native inspection from mutation state
Select a dedicated read-only inspection future behind the existing single Box allocation. Preserve validation, source ownership and the canonical inspection body while keeping mutation state out of inline read callers. Reject misrouted operations with typed errors and retain source evidence.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): pause abort time after native cleanup
Wait for durable source deletion on the wall clock before advancing the
capacity lease clock at the post-delete barrier. Paused time can otherwise
expire the barrier while native filesystem cleanup is still pending.
Retain lease-loss injection, durable inflight-byte accounting and restart
reconciliation assertions. Document the test-util feature required to
select the native decommission fixtures in package-scoped regressions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): reclaim data from owned multipart undo
Retain exact runtime-only Undo ownership through deferred snapshot cleanup. Revalidate every target metadata version and the retirement anchor, keep the delete claim through the physical callback, and remove the anchor after the payload barrier. Preserve generic cleanup refusal when metadata is missing.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): preserve unformatted bootstrap routes
Keep explicitly unformatted local paths reachable by an in-flight elected formatter while quarantining their service slots and disk-ID membership. Own bounded legacy-read handles so the actual asynchronous creator future remains Send.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(s3): bound CopyObject tracing poll frames
Construct the business future behind an outlined single-Box boundary and lazily instrument its handle. Preserve Send, Unpin, span timing and cancellation without embedding concrete business-state construction in the tracing poll.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(mpu): observe recovery after direct legacy bootstrap
Seed all sixteen real previous-release sources before attaching the same-port method observers. Require exact per-peer disk-route Unimplemented responses with zero publication calls, retain byte and cleanup assertions, and remove superseded raw transport. Register the two new regressions from actual Linux and macOS listings.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(mpu): prepare the homogeneous previous-release fleet
Reuse the existing release baseline before creating the genuine legacy multipart sources. Restart old processes with their normal budgets and require every drive through the reduced-redundancy probes before installing the mixed-version observers and current fault budgets.
Keep standard EC:4, single-attempt calls, all mixed-version byte and reclamation assertions, and their original deadlines unchanged.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(mpu): isolate release compatibility from fault profiles
Keep normal component configuration for the observed mixed-release rollout, which injects no RPC delay or loss. Retain the short component profile in actual method-fault scenarios and preserve the single-attempt 45-second probe and every typed error, routing, source byte and reclamation assertion.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(mpu): use normal shutdown for planned release replacements
Use the existing bounded SIGTERM path during homogeneous release preparation and the observed mixed-version rollout. Preserve every multipart inventory, routing, quorum and byte assertion, and keep actual crash recovery on its original hard-kill path.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* revert: retain the verified release compatibility fixture
Restore the published fixture after the local configuration and lifecycle experiments failed to resolve the macOS startup failures. Keep their diagnostic records separate from the delivered implementation.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(copy): bound inherent handler polling stack
Build the inherent CopyObject future through a synchronous noinline
factory at the existing trait await. This keeps its large state out of
the trait wrapper's active polling frame during historical self-copy.
Preserve cancellation, tracing and post-write notifications, with one
additional allocation restricted to COPY.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): validate authoritative quota and native pool heals
Require actual authoritative quota usage in the existing readiness loop
instead of treating HTTP success or unknown usage as a numeric baseline.
Keep the original quota assertions and convergence budgets.
Allow the owning pool metadata CAS to republish into a fresh valid data
directory while retaining exact version, layout, completeness and shard
fingerprints. Require every node's real metadata/read/write readiness
before accepting recovery. Keep ordinary object manifests and the original
180-second recovery deadline unchanged, with negative oracle regressions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): distinguish certified native Abort recovery
Require native participants with a persisted full closure certificate to retire their heads and reclaim every surviving source on retry. Keep uncertified legacy authority below identity quorum conservative, while retaining actual partial-delete and committed-object byte checks.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
Hauser