Connect Feature Boundaries, real io_uring and the rio-v2, Swift, SFTP and
HTTP/3 feature lanes no longer gate a merge. They run only in the scheduled
and manual full-validation runs, which report a regression within a day; on
the pull-request path they bought no earlier signal worth the runner
occupancy (Connect alone measured 53.5 median minutes and occasionally set
the wall). Workspace lint and tests, the short core contract, the debug
build, E2E smoke and the S3 lanes stay required on PRs and merge groups.
* ci: retain CI01 timing sample and fix the OIDC live gate tool
Record the measured PR feedback and runner occupancy for the CI plan's CI01
criterion, including the honest gap between the experimental targets and the
sampled medians, and replace the awscurl dependency in the live Keycloak gate
with the AWS CLI so the security suite runs on PEP 668 managed fleets.
* docs(testing): attach live run evidence to the event-drill index
* feat(scanner): bound first-build enumeration with the cycle budget
Charge raw directory entries and their name bytes, and let one bucket scan hold a child budget that stops its own token while charging the cycle. A count budget also bounds one directory listing, so a wide first build can no longer finish a full read_dir before the first object is charged.
The new stop reason maps onto the persisted Directories variant so older readers keep decoding checkpoints. A directory with queued children keeps its index until they end, and every index on the active walk path is saved as an optional map field: losing either one would recharge its replay and stop a fixed budget from advancing.
* feat(scanner): keep bucket service order across restarts
Save each set's first-in-first-out bucket order as an optional map field of its usage cache, including after an incomplete cycle, which otherwise writes nothing. Durable age now outranks the dirty-first and leader-local cohort orders, which only break its ties, so a restart cannot hand the budget back to the same first buckets. Failed turns add visible debt without moving a bucket forward; a budget stop or a resource veto adds none. A bucket that spends its quantum and saves forward motion returns to the queue tail within the cycle.
The order authorizes nothing: another movement epoch or a corrupt value starts a new one, and a recreated bucket joins as a new arrival.
* test(scanner): cover quantum handoff and saved service order
Run the real worker queue with injected bucket shares and verify a spent bucket yields to waiters, a share without durable motion ends its turn, and the saved order keeps certified coverage while rejecting older owners.
* docs(scanner): describe budgeted first build and service order
* test(scanner): pin index retention and requeue motion
Bound the nested restart fixture tightly enough that dropping a directory index fails it, and cover a repeated share whose frontier never moves.
* fix(scanner): keep service generations and owner stops intact
Queue a new bucket behind waiters but ahead of buckets that already had a turn, matching the leader-local cohort, and keep the set cache untouched when the owner was cancelled so an interrupted cycle cannot advance its revision. The order holds one record per inventory bucket, like the set cache's entries, so a truncating bound that could starve the long-waiting buckets is removed. Publish the incomplete scope before saving so a contended lock cannot delay it.
* ci(e2e): refresh the full selection digests
* fix(scanner): correct a comment spelling
* fix(scanner): separate service debt from vetoes and stale owners
Keep a resource veto and a remote count-budget stop out of failure debt,
settle the claim before a requeue exposes the bucket to another worker, and
stamp the service order with its leader so an older owner cannot replace it
after an incomplete save. The checkpoint fixture's budget now covers both
service orders.
* fix(ecstore): create parents before resource-scoped writes
#9064 moved parent creation into the lazy writer, so a write that holds a
resource unit reached the native open without its parent directory and
failed with FileNotFound on nested paths.
* fix(e2e): retry SlowDown codes during upgrade write readiness
The rolling-upgrade readiness probe only retried 503 ServiceUnavailable,
so a transient 503 SlowDown from a still-recovering node failed the
four_node_rolling_upgrade test immediately. Accept SlowDown and
SlowDownRead alongside ServiceUnavailable, matching the existing chaos
test transient-error set.
* feat(scanner): share node admission with background heal
* fix(scanner): drain candidates after listing completion
* fix(scanner): yield cache leases for shared node admission
* test(scanner): attest adaptive admission in process receipts
* fix(concurrency): declare Tokio time dependency
* feat(concurrency): retain bounded physical resource subsets
* fix(scanner): retain resource ownership in native filesystem jobs
* feat(concurrency): acquire physical groups before node work
* feat(scanner): gate background reads by physical device
* feat(concurrency): combine preallocated resource selections
* feat(scanner): share physical topology across disk plans
* fix(storage): retain only native disk resource subsets
* feat(storage): admit auxiliary physical resources without waiting
* feat(heal): admit physical resources before object repair
* fix(storage): retain resources during native format probes
* fix(storage): retain resources through direct writer I/O
* fix(storage): retain resources through buffered writer I/O
* fix(storage): retain physical resources through streamed reads
* fix(storage): retain resources through native filesystem jobs
* fix(storage): retain physical ownership during bitrot reads
* fix(ecstore): retain resources during positioned reads
* fix(ecstore): retain staged metadata write ownership
* fix(ecstore): retain native io_uring read ownership
* fix(ecstore): enforce receiver-side background admission
* fix(ecstore): retain resources through whole-file native I/O
* fix(ecstore): retain physical ownership during durable rename sync
* fix(storage): account for retained background stream buffers
* fix(storage): retain memory charges for background read results
* fix(storage): account for native io_uring read allocations
* fix(storage): retain memory charges for direct-write buffers
* fix(ecstore): account for heal and RPC reader buffers
* test(scanner): verify samples and clean concurrent collectors
* fix(storage): retain inline and HTTP writer memory reservations
* perf(rio): reuse reserved HTTP writer storage
* fix(rpc): reserve background scanner pipe storage
* perf(rpc): reuse reserved response reader storage
* feat(scanner): measure cache copy allocation requests
* fix(rpc): budget namespace scanner responses
* fix(scanner): retain the enabled supervisor task handle
* fix(scanner): budget namespace frame read staging
* fix(scanner): account namespace envelope decode storage
* fix(heal): retain and join the MRF consumer on shutdown
* test(heal): isolate process-global MRF consumer fixture
* fix(heal): join manager workers during shutdown
* test(heal): verify shutdown at native read boundaries
* fix(heal): drain channel commands before stopping MRF
* fix(heal): admit resume scans and retain replacement workers
* build: pin profiling sources and trust fixed ownership dependencies
* fix(heal): yield node admission while accepted native work drains
* fix(startup): retain shutdown work and preserve deadline failures
* fix(heal): admit inventory probes and retain replacement work
* fix(shutdown): preserve maintenance and frontend join failures
* fix(shutdown): retain HTTP child tasks through real retirement
* fix(shutdown): join actual TLS reload work before retirement
* test(multipart): preserve failures before fault barriers
* fix: keep physical admission inside repository guards
Route the multipart lock diagnostics filter through the e2e storage
compatibility module, reuse the parent module's s3s body aliases in the
namespace memory test, and share one Reed-Solomon verify error mapping
across the reserved-buffer and default paths.
* fix: reconcile physical admission with main's new call sites
Main added a source-admission metadata write layer, a CAS write fence,
fixture writers and scanner handoff tests against the old borrowed-slice
and node-permit shapes. Pass owned Bytes through the new metadata layer,
answer the scanner handoff with ScannerNodePermit, gate the test-only
disk lookup re-export, and clear the lint findings that stable 1.99
reports for the shutdown and admission code without suppressions.
* test(concurrency): name reversed attachment groups clearly
* fix(ecstore): box disk operation futures below admission layers
Every LocalDiskWrapper operation reaches the LocalDisk future through the
health, admission and timeout layers (track_cleanup_mutation, the
track_disk_health family, the io-permit and native-resource admission
split, and track_admitted_disk_operation). Each layer embedded that future,
and unoptimized builds copy it into every awaiting frame, so one local
delete cost about 1.3 MiB of stack and 16 tests overflowed libtest's
default 2 MiB stack.
Box the operation future once where track_admitted_disk_operation creates
it, through a non-async helper so the temporary never stays in a polled
frame. The layers above now hold only a pointer; the cost is one
allocation per local disk operation.
* fix(heal): box object heal futures below admission
The admission wrappers took the object heal future by value, so unoptimized
builds copied it into three awaiting frames, about 650 KiB per heal. Create
it through a non-async helper instead, leaving only a pointer in those frames.
* test(ecstore): count the I/O backend's root FD owner
A request refused by the authorization layer was logged as
s3_authorization_denied but never reached the audit stream. Emit an audit
entry from DenialContext::deny with the account, bucket, object, denied
action and denial reason. Entries are bounded by a per-class token bucket
(lower for anonymous callers) with a suppressed count carried on the next
entry, quiet probe denials are skipped, DeleteObjects audits its refused
keys as one aggregate entry, and the S3 response is unchanged.
A replicated force-delete that a target rejected (for example a peer whose
credentials lack s3:ForceDeleteObject) was only logged; the per-target
replication counters never saw it, so `replication status` reported zero
failures while the sites had already diverged.
Build a ReplicatedTargetInfo for every target outcome of
replicate_force_delete_to_targets (missing client, offline target,
rejected remove, success) with a Pending previous status, and feed each
through ReplicationStats::update so rejections count as failures and
accepted force-deletes count as replicated.
Document that the managed site-replicator-0 policy must name
s3:ForceDeleteObject explicitly and that a bucket-replication target with
its own credentials needs the same grant.
Since #9059 the shared test archives and debug binaries are named
with the run attempt, and the archive identity binds the attempt. A
"Re-run failed jobs" attempt does not rerun the successful producer,
so its consumers look for <name>-2, fail with "Artifact not found",
and every flaky test forces a full rerun.
Give those artifacts one name per run, overwrite them when a producer
does rerun, and bind the archive identity to the run instead of the
attempt. Source, tree, lock, toolchain and environment stay bound, and
an archive from another run is still rejected.
* fix(ecstore): select the latest version at read quorum like MinIO
GET and HEAD escalated latest-version selection to write quorum when
enough metadata answered and skipped a newer version below write quorum
on degraded reads. LIST and heal already pick the newer side of a split
write at read quorum, so such keys listed but answered 503 until heal.
Select with MinIO's getObjectFileInfo rule: the most common mod time
that reaches the object's data blocks wins, ties go to the newer time,
and one content identity at that time must reach the same quorum. Two
identities tied at the selected time fail instead of resolving by hash
order, and the selected identity's version-stack shape follows disk
order.
* fix(ecstore): resolve selection ties by disk order and guard writes
Review of the read-quorum selection found three gaps:
- Rejecting a same-time half split made explicit-version reads of a
partially applied tagging update fail and blocked the retry that
repairs it. Resolve ties to the identity on the lowest-index disk,
for reads and heal alike, instead of rejecting or using map order.
- Delete markers and empty versions can lower the common read quorum
below the selected version's data blocks, so HEAD could report a
version GET cannot decode. Require that many copies, as MinIO's
getLatestFileInfo does.
- A selection can now return exactly read-quorum disks, and partial
metadata writes are not rolled back. Refuse an update before writing
when its target disks cannot reach write quorum.
* fix(ecstore): count identity copies in the data-blocks guard
The first-candidate shortcut counts byte-identical copies, so copies
that differ only in identity-neutral fields such as replication status
were missing from the data-blocks check. When the shortcut's count
falls short, count every copy of the selected content identity before
refusing the version.
* fix(ecstore): keep commit-proof selection for system-bucket reads
Pool metadata's PREPARE/COMMIT recovery and identity epochs treat an
ordinary read of pool.bin as proof that the record committed. Under the
read-quorum rule, a COMMIT that reached only half of a 2+2 set was served
as committed, and the pool_meta_native_physical_* integration tests
failed.
Route reads through select_fileinfo_for_read: S3 buckets keep MinIO's
read-quorum rule, while is_meta_bucketname reads use the previous
algorithm unchanged as select_committed_fileinfo, so a split stays
unreadable until native recovery resolves it.
* fix(ecstore): serve the latest readable version past dangling versions
A failed versioned PUT whose rollback also failed can leave the new
version on top of fewer drives than its data blocks. Reading only each
drive's top version then left the committed version below it short of
read quorum, so GET and HEAD of the latest version answered 503 although
that version is intact in every stack. MinIO resolves the stacks in
pickLatestQuorumFilesInfo.
When every drive answered and an S3 latest read still cannot select a
version, walk the newest version headers of each stack through
exact-version reads: serve the readable version with the newest selected
modification time, skip versions whose exact read proves them dangling
(the NoSuchVersion an explicit GET reports) and queue heal for them, and
keep the read-quorum error on any other outcome. Candidates are ordered
by each version's newest modification time across drives, and at most
eight exact reads are spent.
Test PUTs now wait for every drive so tests that edit one drive's
metadata no longer race the PUT's background tail writes.
* fix(ecstore): resolve dangling latest versions in metadata updates
A failed versioned PUT whose rollback also failed can leave the new
version on fewer drives than its data blocks. GET and HEAD of the
latest version already look past such versions, but read-before-write
callers without a version id still answered 503: object tagging read
each drive's top version with the quorum-error read, and
put_object_metadata and the CopyObject source read selected from
read_all_xl, whose merged latest version is missing on the other
drives, so object_quorum_from_meta failed before selection.
Object tagging now reads through a variant that runs the same walk past
dangling versions as a latest GET while keeping the quorum error for a
dangling key. put_object_metadata and the CopyObject source try the walk
when their quorum or selection step fails with read quorum, before the
existing dangling cleanup could report a key with a committed version
absent. A version-only copy takes the served version's own copies and
write quorum. The walk still applies only to S3 buckets, latest reads,
and sets where every drive answered, and the update still targets only
the served version's disks, so a version below write quorum is refused
before writing.
* fix(ecstore): refuse version-only copies below write quorum up front
Review of the dangling-latest metadata updates found three gaps:
- A version-only CopyObject wrote the new version and reverted on a
write-quorum miss; refuse it before writing when the served version's
disks cannot reach its write quorum, like update_object_meta_with_opts.
- Replication status write-back, including the zero-write ValidateOnly
mode, now keeps its read-quorum error and never walks past dangling
versions, since the walk queues heal for the versions it skips.
- The erasure-coding doc overstated that these callers never report a
dangling key as absent; put_object_metadata and CopyObject keep their
existing dangling cleanup when the walk finds no readable version.
Record local batch eligibility, blocking queue and execution time, shard
outcomes, and actual read bytes behind the existing GET-stage metrics gate.
Keep captured metric handles with the blocking worker after cancellation
and use scoped low-latency histogram buckets for the new duration series.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): select the latest version at read quorum like MinIO
GET and HEAD escalated latest-version selection to write quorum when
enough metadata answered and skipped a newer version below write quorum
on degraded reads. LIST and heal already pick the newer side of a split
write at read quorum, so such keys listed but answered 503 until heal.
Select with MinIO's getObjectFileInfo rule: the most common mod time
that reaches the object's data blocks wins, ties go to the newer time,
and one content identity at that time must reach the same quorum. Two
identities tied at the selected time fail instead of resolving by hash
order, and the selected identity's version-stack shape follows disk
order.
* fix(ecstore): resolve selection ties by disk order and guard writes
Review of the read-quorum selection found three gaps:
- Rejecting a same-time half split made explicit-version reads of a
partially applied tagging update fail and blocked the retry that
repairs it. Resolve ties to the identity on the lowest-index disk,
for reads and heal alike, instead of rejecting or using map order.
- Delete markers and empty versions can lower the common read quorum
below the selected version's data blocks, so HEAD could report a
version GET cannot decode. Require that many copies, as MinIO's
getLatestFileInfo does.
- A selection can now return exactly read-quorum disks, and partial
metadata writes are not rolled back. Refuse an update before writing
when its target disks cannot reach write quorum.
* fix(ecstore): count identity copies in the data-blocks guard
The first-candidate shortcut counts byte-identical copies, so copies
that differ only in identity-neutral fields such as replication status
were missing from the data-blocks check. When the shortcut's count
falls short, count every copy of the selected content identity before
refusing the version.
* fix(ecstore): keep commit-proof selection for system-bucket reads
Pool metadata's PREPARE/COMMIT recovery and identity epochs treat an
ordinary read of pool.bin as proof that the record committed. Under the
read-quorum rule, a COMMIT that reached only half of a 2+2 set was served
as committed, and the pool_meta_native_physical_* integration tests
failed.
Route reads through select_fileinfo_for_read: S3 buckets keep MinIO's
read-quorum rule, while is_meta_bucketname reads use the previous
algorithm unchanged as select_committed_fileinfo, so a split stays
unreadable until native recovery resolves it.
* fix(ecstore): serve the latest readable version past dangling versions
A failed versioned PUT whose rollback also failed can leave the new
version on top of fewer drives than its data blocks. Reading only each
drive's top version then left the committed version below it short of
read quorum, so GET and HEAD of the latest version answered 503 although
that version is intact in every stack. MinIO resolves the stacks in
pickLatestQuorumFilesInfo.
When every drive answered and an S3 latest read still cannot select a
version, walk the newest version headers of each stack through
exact-version reads: serve the readable version with the newest selected
modification time, skip versions whose exact read proves them dangling
(the NoSuchVersion an explicit GET reports) and queue heal for them, and
keep the read-quorum error on any other outcome. Candidates are ordered
by each version's newest modification time across drives, and at most
eight exact reads are spent.
Test PUTs now wait for every drive so tests that edit one drive's
metadata no longer race the PUT's background tail writes.
* chore(ci): shrink the error(format!) ratchet baseline for rebalance
The rebalance control and meta modules now have fewer
Error::other(format!) call sites than the baseline records, which
fails the shrink-only ratchet in Quick Checks. Record the smaller
counts.
* fix(ecstore): preserve canonical order in latest-version fallback
Report selected integer offsets and widths with strict member and payload validation. Cross-check compiler fixtures with bpftool and independent C offsetof/sizeof results.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): select the latest version at read quorum like MinIO
GET and HEAD escalated latest-version selection to write quorum when
enough metadata answered and skipped a newer version below write quorum
on degraded reads. LIST and heal already pick the newer side of a split
write at read quorum, so such keys listed but answered 503 until heal.
Select with MinIO's getObjectFileInfo rule: the most common mod time
that reaches the object's data blocks wins, ties go to the newer time,
and one content identity at that time must reach the same quorum. Two
identities tied at the selected time fail instead of resolving by hash
order, and the selected identity's version-stack shape follows disk
order.
* fix(ecstore): resolve selection ties by disk order and guard writes
Review of the read-quorum selection found three gaps:
- Rejecting a same-time half split made explicit-version reads of a
partially applied tagging update fail and blocked the retry that
repairs it. Resolve ties to the identity on the lowest-index disk,
for reads and heal alike, instead of rejecting or using map order.
- Delete markers and empty versions can lower the common read quorum
below the selected version's data blocks, so HEAD could report a
version GET cannot decode. Require that many copies, as MinIO's
getLatestFileInfo does.
- A selection can now return exactly read-quorum disks, and partial
metadata writes are not rolled back. Refuse an update before writing
when its target disks cannot reach write quorum.
* fix(ecstore): count identity copies in the data-blocks guard
The first-candidate shortcut counts byte-identical copies, so copies
that differ only in identity-neutral fields such as replication status
were missing from the data-blocks check. When the shortcut's count
falls short, count every copy of the selected content identity before
refusing the version.
* fix(ecstore): keep commit-proof selection for system-bucket reads
Pool metadata's PREPARE/COMMIT recovery and identity epochs treat an
ordinary read of pool.bin as proof that the record committed. Under the
read-quorum rule, a COMMIT that reached only half of a 2+2 set was served
as committed, and the pool_meta_native_physical_* integration tests
failed.
Route reads through select_fileinfo_for_read: S3 buckets keep MinIO's
read-quorum rule, while is_meta_bucketname reads use the previous
algorithm unchanged as select_committed_fileinfo, so a split stays
unreadable until native recovery resolves it.
* test(scanner): add memory acceptance runner and budget probes
Add owned S3 workload phases with process, endpoint, cgroup and binary
identity checks for scanner memory and scanner-disabled request bursts.
Keep scale, restart, heap-attribution and overall-acceptance limits explicit.
Characterize healthy and seeded coverage-gap cache retention through the
real folder walker and production cache codec without changing runtime code.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): bind acceptance capacity to effective storage
Validate the serving process's complete local disk inventory before load
writes and during runtime sampling. Resolve each disk inside the owned
storage root, reject nested mounts and filesystem mismatches, and bind
storage observations to phase identity for container and native runs.
Cover the unused-mount bypass, CLI-overridden environment, native paths,
symlink escapes, named volumes and stable multi-disk inventory ordering.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): fence status observations before workload writes
Close each status API window with a second storage-bound process sample
before Phase entry can authorize bucket creation or PUT requests.
Add a fail-before regression for a process change during the initial status
request, asserting no ownership or workload writes occur.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Retain original prepare and commit-settlement RPCs beyond failed replacement and Abort cleanup, with and without bucket recreation. Require typed refusal or a physically verified idempotent result, unchanged fresh-upload bytes and complete object reads.
Extend scoped proxy observations without rewriting signed frames or trailers, and preserve registered test membership and existing budgets.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(tools): add read-only BTF prototype inspection
Expose bounded declared prototype headers through a standard-library CLI, preserve explicit void and failure semantics, and wire local contracts plus an independent compiler/bpftool reference into CI.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(tools): retain native reference failure diagnostics
Clear checkout credentials and capture bounded native stderr alongside stdout, preserving exact exit status and owned child cleanup. Keep the first reference failure unresolved rather than guessing its missing diagnostics.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* ci(tools): select a concrete bpftool file parser
The hosted PATH wrapper can exist without tools for its Azure kernel. Select an executable ELF by availability/version before comparing the fixed fixture, installing generic tools only when needed. Data comparison errors remain fatal and never trigger tool switching.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Verify a sixteen-drive set rejects reads and writes after two nodes stop,
then resumes complete S3 reads and new writes when only one node returns.
Keep the fourth node stopped throughout checks from all three live
coordinators and verify the original 12+4 shard geometry on disk.
Let the existing GET recovery probe retry the compatible SlowDownRead
response only with HTTP 503, retaining fatal status and byte checks.
Register the case in both platform selections and document the boundary
between process recovery and physical disk reattachment acceptance.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(observability): correlate physical namespace lock releases
Map logical namespace guards to their actual granted peer lock IDs and local physical guard identities. Record physical completion only after successful manager release and registry removal, and record validated positive remote unlock responses without changing release behavior.
Exercise real distributed ownership, asynchronous cleanup, duplicate and disabled guards, stale-owner retirement, and explicit release followed by Drop. Document the required diagnostic filters and the remaining filesystem-tail boundaries.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): remove unused lock manager test import
The diagnostic regression uses inherent manager methods and does not need the LockManager trait import. Remove it to pass strict warning checks.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
A cursor for a key that literally ends in a legacy-shaped tag is issued
wrapped in one extra `[rustfs_cache:v2,return:]`, because the listing layer
still strips one legacy tag from incoming markers. ListObjects (V1)
unwrapped it in `calculate_next_marker`; the client then sent the bare key
back as `marker`, the listing layer stripped the key's own tail, and the
next page restarted at the stem and returned the same key forever. Keep the
wrapper in V1 NextMarker and cover the V1 resume path (marker sent back as
StartAfter) on a real store.
Document that sampled list quorum modes can also omit committed objects
whose copies mostly sit outside the sample, and recommend `strict` for
backup, sync, replication and lifecycle tooling.
* chore(deps): upgrade russh to 0.65.0 and refresh crates
Run two cargo update and cargo upgrade rounds from origin/main.
Adapt SFTP public-key callbacks to PublicKeyOffer and VerifiedPublicKey,
preserving explicit rejection and password-only authentication.
Cover the rejection policy with a pinned-host-key SSH handshake test.
Refresh Arrow, Parquet, Brotli, ZIP, and compatible dependencies.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(deps): route legacy zstd consumers to fixed codecs
Replace the old registry implementation with a private version-constraint
adapter backed only by zstd 0.14.3 and zstd-safe 8.2.0. Preserve legacy
features and Avro, Arrow IPC, Parquet, and Kafka formats.
Cover abandoned contexts, legacy decode errors, seekable bounds, stored
frames, consumer interoperability, and resolved implementation versions.
Record the adapter removal condition in the compatibility register.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* refactor(listing): list latest versions with MinIO's listing quorum
Latest-version SetDisks::list_path now follows MinIO's er.listPath:
ceil(asked/2) is the listing quorum for min_disks, directories and
objects, agreed entries are emitted as read, and a split sample goes
through MetaCacheEntries::resolve. One object can no longer fail a
page. A set with fewer than half of its drives online still fails the
page with ErasureReadQuorum.
Remove the write-quorum layer this replaces: ListingSupplement,
ListingReconciler and the LIST object read lock, the latest_listing_*
quorum helpers, resolve_agreed_entry, resolve_with_write_quorum and the
enforce_write_quorum / write_quorum_slack merge branches, and the
per-version listing quorum helpers in filemeta.
Make merge_file_meta_borrowed_streams pick a version id's most common
header regardless of reader order, so an interrupted null-version
overwrite no longer drops the committed generation from the merge.
* test(listing): do not assert sampled optimal listings are deterministic
optimal asks a random half of the drives, so an EC:8 object committed on nine
of sixteen drives can miss the four-copy listing quorum in roughly one sample
in six, exactly as in MinIO. Keep the deterministic assertion for strict and
for optimal when every online drive is asked.
* fix(storage): 404 for dangling objects and strict list quorum default
Stage F.0 of the ListObjects quorum plan. Once listing returns to MinIO's listing quorum, a PUT that reached half of the drives but not read quorum becomes visible in LIST, so GET and HEAD must not answer it with a retryable 503.
GET, HEAD and prepared GET metadata now report a metadata read-quorum failure as ObjectNotFound or VersionNotFound when every drive answered with valid metadata of a single generation or not-found and is_object_dangling proves the object or version dangling, matching MinIO's getObjectFileInfo. Offline or unanswered drives, timeouts, corrupt xl.meta, two generations, remote versions and the meta bucket keep the read-quorum error. Nothing is deleted on the read path; heal keeps owning cleanup and its grace window. Read-before-write callers of get_object_fileinfo keep the quorum error.
RUSTFS_LIST_OBJECTS_QUORUM now defaults to strict, matching MinIO, and the operations doc explains that optimal samples drives and can briefly show deleted objects.
A new test pins the write-path precondition: a PUT that misses write quorum leaves no new version or data directory on any drive once its rollback completes.
* test(listing): prove the list quorum env var is read with a non-default value
Version listings bypassed listing quorum: SetDisks::list_path returned
early into list_versions_authoritatively, whose resolver seeds unread
slots as DiskNotFound. With any one drive of the set offline, every
version held by fewer than ceil(n/2) drives (for example a deleted
version kept by drives that were offline during the delete) failed the
whole page with ErasureReadQuorum, surfaced as S3 503.
Route merged version pages through the listing-quorum branch of
SetDisks::list_path, as MinIO's listPath does with requestedVersions =
0, so such versions are omitted instead of failing the page. Keep the
set-level floor: fewer than ceil(n/2) online drives still fails the
page.
Set walks (tier reference proofs, tier-delete journals, pool
maintenance) and the recursive prefix-delete scan act on absence, so
they keep the authoritative per-version scan and fail closed.
* refactor(listing): make cursors plain keys and opaque V2 tokens
RustFS inherited MinIO's metacache cursor tag and appended
`[rustfs_cache:v2,id:<uuid>,src:walker,gen:live]` to every internal
continuation marker, although nothing ever consumed the id: four parsers
had drifted apart, V1 NextMarker had to be stripped on the way out, and
ListObjectVersions leaked the tag onto the wire as NextKeyMarker.
ecstore stops assigning `list_id` in `ECStore::list_path` and
`SetDisks::list_path_result`, drops `append_list_cache_id_to_marker` and
the `cache_id` parameters of the pagination helpers, and removes the
unused `ListPathOptions::encode_marker`; internal markers are the plain
last key, last common prefix or last raw scanned key.
The API layer builds the V2 NextContinuationToken as base64 of the
server-side cursor in one place and decodes it in one place; V1
NextMarker is the decoded cursor and the `public_list_marker` tag
stripper is gone; NextKeyMarker is the plain key. Base64 of the plain
key is what nodes from earlier releases already read, so a rolling
upgrade needs no switch.
Compatibility, registered as `listing-cursor-legacy-tag`: `parse_marker`
and `list_marker_key` still strip a trailing legacy tag from incoming
markers so a client that paginates across the upgrade keeps its position.
* fix(listing): preserve literal cache-tag cursor boundaries
* perf(ilm): back off idle tier-delete journal recovery
Use bounded idle backoff only after complete empty journal and dispatch scans, while preserving immediate notifications and the retry cadence for observed work.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): reconcile interrupted pool metadata publication
Bind pool metadata CAS and Undo to exact native predecessors. Recover an interrupted physical publication only from an ordinary committed authority and complete verified identity, current and backup evidence, then require normal V3 CAS and quorum readback.
Preserve known-conflict gates, startup and heal lock ordering, targeted admission and the original kill-recovery deadline. Wake proof-ready recovery immediately and bound parallel capacity sampling without removing mutation fences.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(ecstore): reuse proven pool metadata durability
Publish native rollback backups directly in their owned UUID directories. Reuse a one-shot data-durability receipt only for an exact hardlink to the pinned, fully synchronized head and root.
Preserve new-head and directory synchronization, native predecessor CAS, conditional Undo and uncertainty gates. Keep unknown, copied, raw-parent, cancelled and failed operations cold. Add bounded diagnostics, sync-call regressions and path-isolated test hooks.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(multipart): exercise selective RPC delay and peer loss
Add authenticated method-entry barriers for metadata and lifecycle locks under e2e-test-hooks. Verify real dispatch, write quorum, prior part payloads, lock drain and incarnation isolation without changing production budgets.
Record cancellation separately from normal exits so late filesystem completion cannot overwrite Drop evidence.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* docs: define multipart delay gates and receipt invariants
Document RPC entry evidence, peer-loss and cancellation oracles, remaining incident attribution limits, and immutable head inodes required for durability receipts.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(readiness): expose missing runtime drive slots
Report configured slots without a runtime handle separately from unavailable inventoried drives. Preserve quorum and probe status semantics. Wait for every coordinator to regain its complete live inventory before multipart RPC fault windows and single-attempt recovery oracles.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): retry committed promotion convergence safely
Require peer reload convergence on every worker attempt, including unchanged retries. Recheck durable identity, replicas, namespace fence, generation and cancellation before allowing any data movement.
Keep known committed promotions active on a verified propagation pause. Release precise owners, retain queued successors and reuse the existing watchdog. Preserve uncertainty gates and terminal operation behavior.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): guard native metadata mutation aliases
Reject protected RenamePart source and destination paths before I/O. Classify the protected namespace through the native path resolver, and require canonical conditional RenameData and Undo keys so aliases cannot create separate lease domains.
Preserve ordinary multipart and canonical CAS behavior, warm durability receipts and zero-allocation canonical fast paths. Cover real alias, byte identity, receipt and normal mutation behavior.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): register Darwin multipart RPC delay gates
Update distributed membership from the actual Darwin listing. Preserve the full-profile digest and leave Linux unchanged until its own listing is produced, as required by the contributor guide.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(multipart): fence native publication and recovery
Publish multipart data through private hardlinks and atomically consume the
prepared source head. Bind exact compensation capabilities and native recovery
to the physical cohort, source generation and durable publication state. Require
fresh terminal confirmation for native replay and preserve indeterminate data
when no safe recovery witness exists.
Release logical guards only after validated irreversible quorum success while
retaining physical mutation leases, accounting and resource ownership through
actual callbacks. Preserve local instance proof authority and historical Copy
metadata without granting healed donors fresh publication authority.
Keep general movement identity strict while preserving directory capacity
compatibility in a separate checked comparison.
Cover native fault and cancellation boundaries, zero-byte objects, real Copy,
heal and migration, exact physical readback and distributed RPC delay. Retain
all quorum and durability requirements and document rollout limitations.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): fix ratchet scan and Windows fault hooks
Move native pool metadata inventory tests after production items so the
error-format ratchet sees the existing call site. Preserve the baseline
instead of accepting a false count decrease.
Make the directory-barrier failure hook available on every test platform
and exercise the parent-barrier failure/retry cases in Windows CI. Keep
production synchronization behavior and all existing assertions intact.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(protos): remove redundant recovery request clone
Move the baseline into its final struct update while retaining the earlier
clone needed to construct the first request variant.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): align RPC delay gates with native abort
* fix(storage): validate native source recovery and CI gates
Preserve wire layouts while bounding native response sizes and fix the
last-use recovery request clone. Avoid destination replay reads only for
fully observed initial native sources; keep upgraded and recovered sources
conservative. Cover real legacy publication after a cold restart.
Align upgrade acceptance with the native mixed-version mutation pause and
use exact upload-scoped test completion evidence before advancing time.
Keep all original quorum, ownership, byte and deadline assertions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): verify native abort under partial admission
Delay native cleanup at the exact upload root after all-participant closure.
Verify quorum-only admissions retain every source byte on refused Abort and
complete the same upload without creating a missing original participant.
Keep sub-quorum orphans and their discoverable identities isolated while a
fresh all-participant upload proves service recovery. Route shared constants
through the existing E2E storage facade.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): avoid formatting a fixed invalid upload path
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): align native MPU fault recovery assertions
Verify real source participants, preserved orphan bytes and same-upload completion after quorum-only admission. Require full physical inventories so an expected read-quorum error cannot hide unrelated leakage. Preserve ordinary quorum and recovery budgets, and document listing availability through the affected set.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): keep MPU tail deadlines on the wall clock
Global Tokio time advancement can expire drive-health probes while real
filesystem callbacks are pending. Use a short real drive deadline with the
existing physical-entry and cleanup barriers, retaining the byte, lock,
resource-permit and final-shard assertions.
Fixes#8937
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(storage): align native fixtures with instance ownership
Open independent disks for the simulated capacity-loss instance, scope four coordinator journal disks without rebinding source authority, and probe configured namespace keys while retaining physical I/O hooks.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): isolate native multipart future construction
Construct optional native publication futures behind synchronous heap boundaries so ordinary rename callers do not carry their state machines. Preserve native guards and ownership without adding allocations to ordinary PUT, copy or heal branches.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(heal): assert coordinator journal scope after rejoin
Check the exact original four-member journal map after every rejoin and manager stop while preserving all sixteen native data slots. Reconnect expansion fails the test rather than being filtered away.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(heal): bound object repair future construction
Construct the normal and FileCorrupt deep-retry state machines behind a synchronous heap boundary. Preserve guards and cancellation while bounding the shared rename and deletion call stacks without increasing test stacks.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): qualify fixture disk identity checks
Call the same DiskAPI method explicitly so the independently reopened disk identity assertion compiles under every target and feature combination.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): isolate native inspection from mutation state
Select a dedicated read-only inspection future behind the existing single Box allocation. Preserve validation, source ownership and the canonical inspection body while keeping mutation state out of inline read callers. Reject misrouted operations with typed errors and retain source evidence.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): pause abort time after native cleanup
Wait for durable source deletion on the wall clock before advancing the
capacity lease clock at the post-delete barrier. Paused time can otherwise
expire the barrier while native filesystem cleanup is still pending.
Retain lease-loss injection, durable inflight-byte accounting and restart
reconciliation assertions. Document the test-util feature required to
select the native decommission fixtures in package-scoped regressions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): reclaim data from owned multipart undo
Retain exact runtime-only Undo ownership through deferred snapshot cleanup. Revalidate every target metadata version and the retirement anchor, keep the delete claim through the physical callback, and remove the anchor after the payload barrier. Preserve generic cleanup refusal when metadata is missing.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): preserve unformatted bootstrap routes
Keep explicitly unformatted local paths reachable by an in-flight elected formatter while quarantining their service slots and disk-ID membership. Own bounded legacy-read handles so the actual asynchronous creator future remains Send.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(s3): bound CopyObject tracing poll frames
Construct the business future behind an outlined single-Box boundary and lazily instrument its handle. Preserve Send, Unpin, span timing and cancellation without embedding concrete business-state construction in the tracing poll.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(mpu): observe recovery after direct legacy bootstrap
Seed all sixteen real previous-release sources before attaching the same-port method observers. Require exact per-peer disk-route Unimplemented responses with zero publication calls, retain byte and cleanup assertions, and remove superseded raw transport. Register the two new regressions from actual Linux and macOS listings.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(mpu): prepare the homogeneous previous-release fleet
Reuse the existing release baseline before creating the genuine legacy multipart sources. Restart old processes with their normal budgets and require every drive through the reduced-redundancy probes before installing the mixed-version observers and current fault budgets.
Keep standard EC:4, single-attempt calls, all mixed-version byte and reclamation assertions, and their original deadlines unchanged.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(mpu): isolate release compatibility from fault profiles
Keep normal component configuration for the observed mixed-release rollout, which injects no RPC delay or loss. Retain the short component profile in actual method-fault scenarios and preserve the single-attempt 45-second probe and every typed error, routing, source byte and reclamation assertion.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(mpu): use normal shutdown for planned release replacements
Use the existing bounded SIGTERM path during homogeneous release preparation and the observed mixed-version rollout. Preserve every multipart inventory, routing, quorum and byte assertion, and keep actual crash recovery on its original hard-kill path.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* revert: retain the verified release compatibility fixture
Restore the published fixture after the local configuration and lifecycle experiments failed to resolve the macOS startup failures. Keep their diagnostic records separate from the delivered implementation.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(copy): bound inherent handler polling stack
Build the inherent CopyObject future through a synchronous noinline
factory at the existing trait await. This keeps its large state out of
the trait wrapper's active polling frame during historical self-copy.
Preserve cancellation, tracing and post-write notifications, with one
additional allocation restricted to COPY.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): validate authoritative quota and native pool heals
Require actual authoritative quota usage in the existing readiness loop
instead of treating HTTP success or unknown usage as a numeric baseline.
Keep the original quota assertions and convergence budgets.
Allow the owning pool metadata CAS to republish into a fresh valid data
directory while retaining exact version, layout, completeness and shard
fingerprints. Require every node's real metadata/read/write readiness
before accepting recovery. Keep ordinary object manifests and the original
180-second recovery deadline unchanged, with negative oracle regressions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): distinguish certified native Abort recovery
Require native participants with a persisted full closure certificate to retire their heads and reclaim every surviving source on retry. Keep uncertified legacy authority below identity quorum conservative, while retaining actual partial-delete and committed-object byte checks.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
* refactor(listing): keep paginated listing on ECStore only
`Sets` and `SetDisks` carried their own copies of `inner_list_objects_v2`,
`list_objects_generic`, `inner_list_object_versions`, `list_path` and
`list_merged`, plus a third copy of the task that converts merged raw entries
into the walk result stream. Only `ECStore` can answer a paginated listing,
since a page merges every set of every pool; the copies drifted (the `Sets`
copy still rejected `//` prefixes that #8605 made literal on `ECStore`) and
were reachable only from tests and one decommission helper.
Remove the `ListOperations` impls for `Sets` and `SetDisks` together with the
duplicated pagination code. `SetDisks::walk` stays as an inherent method for
set-scoped callers and now honors `versions_sort` like the store-wide walk;
`SetDisks::list_path_result` stays for the recursive delete lister. The walk
entry conversion moves into one `spawn_walk_entry_forwarder` shared by both
walks. The decommission receipt lister walks its pool's sets instead of
paginating through the removed `Sets` listing.
Tests that exercised the removed layers now exercise `ECStore` for pagination
or `SetDisks::walk` for per-pool version inspection. The architecture and
logging guard rows that pinned the removed `SetDisks`/`Sets` listing entry
points are dropped, and the error-other ratchet baseline shrinks by one.
* fix(listing): exclude synthetic directories from receipt walks
* refactor(listing): remove env-gated ListObjects index provider
Delete the experimental, default-off key-only / verified-page /
metadata-fast index provider from `ECStore::list_objects_generic` together
with everything only it drove: the `RUSTFS_LIST_OBJECTS_INDEX_*`,
`RUSTFS_LIST_OBJECTS_NAMESPACE_JOURNAL_*` and
`RUSTFS_LIST_OBJECTS_METADATA_FAST_*` readers, the persistent key-only
index and its on-disk format, the namespace mutation journal (state,
chaos injector, system/local backends, degraded set, global lock), the
index lifecycle/health/source-selection types, the verified-index
candidate pager, the `list-chaos` cargo feature, the
`rustfs_s3_list_objects_index_*` metrics and the archived bench script.
`list_objects_generic` now always walks. The write-path hooks
`observe_list_objects_mutation(s)` are gone; their callers bump the scanner
namespace generation directly through `observe_scanner_namespace_mutations`,
which is the sub-step the old hook performed before taking the journal
lock, so the generation observed by the scanner, the node RPC service and
the data-movement lease checks is unchanged while PUT/DELETE no longer take
a global tokio RwLock or read environment variables.
`ListSourceMode::{metadata_authority, requires_live_metadata_verification,
can_satisfy_strong_listing}` and `ListMetadataAuthority` were only reachable
from the provider and are removed as well; the enum, its variants, the
cursor encoding and `ListContinuationV2` are left untouched.
* docs(listing): define index retirement upgrade prerequisites
* fix(ci): retire the removed listing index test requirement
* test(e2e): assert Abort refusal below identity quorum
Keep exactly-quorum success and recoverable cleanup assertions. For
minority staging authority, require typed refusal, zero further Delete
RPCs, unchanged per-disk metadata and the previous object body. This
aligns the phase matrix with the existing conservative recovery contract.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): exercise simultaneous multipart terminal contenders
Retain the four pairwise resend cases and add live/cancelled three-way Complete, Abort and resend races. Observe distinct forwarded lock owners and verify terminal bytes and physical cleanup.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): freeze contender evidence before multipart cleanup
Retire the passive lock observer before terminal ListParts checks so the reported count identifies only the two contenders at the preparation barrier.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): require both multipart terminal race outcomes
Retain the free races and select both terminal winners with independently released, owner-scoped lock gates. Verify real contender admission, unchanged signed packets and exact terminal bytes for live and cancelled part waiters.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): isolate losing coordinator lock votes
Hold every remote vote for the losing coordinator while the same gate observes the winner. Prevent partial lock grants from delaying the requested terminal order and retain ordinary RPC caller lifetime.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Chris An