Retain the reviewed diagnostic dependency while starting request-stage work
from current main. This integration is not a performance qualification.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* ci: move lightweight jobs to GitHub-hosted Ubuntu
Move API-only control jobs and prebuilt Docker image packaging to ubuntu-latest. Keep lab suites and heavy compilation on their existing runners, and remove redundant GitHub CLI installation.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* ci: retain functional health on the automation runner
Restore smoke-testing and the GitHub CLI installation fallback for functional-chain health, as requested by the runner operator.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix(ecstore): identify stale multipart cleanup failures
Preserve quorum failure metrics while avoiding an UploadPart warning for background cleanup. Keep the existing write quorum and source recovery checks intact.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(ecstore): skip redundant stale upload part scans
Once the default deadline is due, lifecycle rules cannot make an upload ineligible. Avoid querying lifecycle and listing parts in that case while preserving the locked recheck.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): validate multipart lifecycle size
Skip part scans when no abort rule uses size. Treat unavailable, incomplete or overflowing sizes as unknown and keep size-independent lifecycle rules eligible.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): select the latest version at read quorum like MinIO
GET and HEAD escalated latest-version selection to write quorum when
enough metadata answered and skipped a newer version below write quorum
on degraded reads. LIST and heal already pick the newer side of a split
write at read quorum, so such keys listed but answered 503 until heal.
Select with MinIO's getObjectFileInfo rule: the most common mod time
that reaches the object's data blocks wins, ties go to the newer time,
and one content identity at that time must reach the same quorum. Two
identities tied at the selected time fail instead of resolving by hash
order, and the selected identity's version-stack shape follows disk
order.
* fix(ecstore): resolve selection ties by disk order and guard writes
Review of the read-quorum selection found three gaps:
- Rejecting a same-time half split made explicit-version reads of a
partially applied tagging update fail and blocked the retry that
repairs it. Resolve ties to the identity on the lowest-index disk,
for reads and heal alike, instead of rejecting or using map order.
- Delete markers and empty versions can lower the common read quorum
below the selected version's data blocks, so HEAD could report a
version GET cannot decode. Require that many copies, as MinIO's
getLatestFileInfo does.
- A selection can now return exactly read-quorum disks, and partial
metadata writes are not rolled back. Refuse an update before writing
when its target disks cannot reach write quorum.
* fix(ecstore): count identity copies in the data-blocks guard
The first-candidate shortcut counts byte-identical copies, so copies
that differ only in identity-neutral fields such as replication status
were missing from the data-blocks check. When the shortcut's count
falls short, count every copy of the selected content identity before
refusing the version.
* fix(ecstore): keep commit-proof selection for system-bucket reads
Pool metadata's PREPARE/COMMIT recovery and identity epochs treat an
ordinary read of pool.bin as proof that the record committed. Under the
read-quorum rule, a COMMIT that reached only half of a 2+2 set was served
as committed, and the pool_meta_native_physical_* integration tests
failed.
Route reads through select_fileinfo_for_read: S3 buckets keep MinIO's
read-quorum rule, while is_meta_bucketname reads use the previous
algorithm unchanged as select_committed_fileinfo, so a split stays
unreadable until native recovery resolves it.
* fix(ecstore): serve the latest readable version past dangling versions
A failed versioned PUT whose rollback also failed can leave the new
version on top of fewer drives than its data blocks. Reading only each
drive's top version then left the committed version below it short of
read quorum, so GET and HEAD of the latest version answered 503 although
that version is intact in every stack. MinIO resolves the stacks in
pickLatestQuorumFilesInfo.
When every drive answered and an S3 latest read still cannot select a
version, walk the newest version headers of each stack through
exact-version reads: serve the readable version with the newest selected
modification time, skip versions whose exact read proves them dangling
(the NoSuchVersion an explicit GET reports) and queue heal for them, and
keep the read-quorum error on any other outcome. Candidates are ordered
by each version's newest modification time across drives, and at most
eight exact reads are spent.
Test PUTs now wait for every drive so tests that edit one drive's
metadata no longer race the PUT's background tail writes.
* fix(ecstore): resolve dangling latest versions in metadata updates
A failed versioned PUT whose rollback also failed can leave the new
version on fewer drives than its data blocks. GET and HEAD of the
latest version already look past such versions, but read-before-write
callers without a version id still answered 503: object tagging read
each drive's top version with the quorum-error read, and
put_object_metadata and the CopyObject source read selected from
read_all_xl, whose merged latest version is missing on the other
drives, so object_quorum_from_meta failed before selection.
Object tagging now reads through a variant that runs the same walk past
dangling versions as a latest GET while keeping the quorum error for a
dangling key. put_object_metadata and the CopyObject source try the walk
when their quorum or selection step fails with read quorum, before the
existing dangling cleanup could report a key with a committed version
absent. A version-only copy takes the served version's own copies and
write quorum. The walk still applies only to S3 buckets, latest reads,
and sets where every drive answered, and the update still targets only
the served version's disks, so a version below write quorum is refused
before writing.
* fix(ecstore): refuse version-only copies below write quorum up front
Review of the dangling-latest metadata updates found three gaps:
- A version-only CopyObject wrote the new version and reverted on a
write-quorum miss; refuse it before writing when the served version's
disks cannot reach its write quorum, like update_object_meta_with_opts.
- Replication status write-back, including the zero-write ValidateOnly
mode, now keeps its read-quorum error and never walks past dangling
versions, since the walk queues heal for the versions it skips.
- The erasure-coding doc overstated that these callers never report a
dangling key as absent; put_object_metadata and CopyObject keep their
existing dangling cleanup when the walk finds no readable version.
Omit span metadata and inferred parent IDs for diagnostic root events. Verify per-shard index order and document existing production metric timing.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
Run cargo update --verbose and cargo upgrade --verbose twice in the
standalone fuzz workspace. Refresh its lockfile, including the fixed
rustls-webpki, webpki-root-certs, zstd, and zstd-safe releases.
Repeat main's pinned SigV4a fix in the standalone workspace and redirect
its shared Smithy types to registry crates, removing crypto-bigint 0.5.5.
Keep pkcs1 at 0.8.0-rc.4 to remain compatible with the workspace-pinned
rsa 0.10.0-rc.18; rc.5 changes the PKCS#1 key types and fails to compile.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Record local batch eligibility, blocking queue and execution time, shard
outcomes, and actual read bytes behind the existing GET-stage metrics gate.
Keep captured metric handles with the blocking worker after cancellation
and use scoped low-latency histogram buckets for the new duration series.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(rebalance): reuse the target pool holding a dangling copy
Rebalance places a version with a fresh free-space choice whenever the
exact-version pool lookup finds no target copy. An attempt that loses
write quorum can leave the version on fewer drives than read quorum,
and object reads report such a copy as absent (#8963), so the retry
could commit the version on a different pool and leave it in two. The
distributed kill-before-source-cleanup case saw this: a delete marker
whose first copy failed with InsufficientWriteQuorum ended on pools 1
and 3 after the source was cleaned up.
Before the free-space fallback, data movement without a decommission
capacity owner now targets an eligible pool where any drive already
holds the exact version, so the retry rewrites that copy to full
quorum instead of creating a second one. This covers both the object
copy path and the delete-marker path.
* fix(rebalance): check free space before reusing a holding pool
The pool holding an earlier copy was returned before any space check,
so a full holder pinned every retry and could stall a deferred entry.
The holder now passes through the same filters as the free-space
choice (suspended, rebalancing and source pools, has_space_for for the
expected size, and the max-used filter), computed once per selection;
placement falls back to that weighted choice when the holder does not
qualify. The two free-space helpers share one implementation.
The tests no longer rely on random placement. A test-only override
pins each target pool's reported capacity so the free-space choice
always picks the non-holder pool, the earlier copies keep the version
ids but carry different bytes and modification times, and the moved
versions are checked for marker state, modification time, data on
every drive and readable bytes. A second test covers a full holder.
* chore(ci): shrink the error(format!) ratchet baseline for rebalance
The rebalance control and meta modules on main now have fewer
Error::other(format!) call sites than the baseline records, which
fails the shrink-only ratchet in Quick Checks. Record the smaller
counts.
* fix(ecstore): select the latest version at read quorum like MinIO
GET and HEAD escalated latest-version selection to write quorum when
enough metadata answered and skipped a newer version below write quorum
on degraded reads. LIST and heal already pick the newer side of a split
write at read quorum, so such keys listed but answered 503 until heal.
Select with MinIO's getObjectFileInfo rule: the most common mod time
that reaches the object's data blocks wins, ties go to the newer time,
and one content identity at that time must reach the same quorum. Two
identities tied at the selected time fail instead of resolving by hash
order, and the selected identity's version-stack shape follows disk
order.
* fix(ecstore): resolve selection ties by disk order and guard writes
Review of the read-quorum selection found three gaps:
- Rejecting a same-time half split made explicit-version reads of a
partially applied tagging update fail and blocked the retry that
repairs it. Resolve ties to the identity on the lowest-index disk,
for reads and heal alike, instead of rejecting or using map order.
- Delete markers and empty versions can lower the common read quorum
below the selected version's data blocks, so HEAD could report a
version GET cannot decode. Require that many copies, as MinIO's
getLatestFileInfo does.
- A selection can now return exactly read-quorum disks, and partial
metadata writes are not rolled back. Refuse an update before writing
when its target disks cannot reach write quorum.
* fix(ecstore): count identity copies in the data-blocks guard
The first-candidate shortcut counts byte-identical copies, so copies
that differ only in identity-neutral fields such as replication status
were missing from the data-blocks check. When the shortcut's count
falls short, count every copy of the selected content identity before
refusing the version.
* fix(ecstore): keep commit-proof selection for system-bucket reads
Pool metadata's PREPARE/COMMIT recovery and identity epochs treat an
ordinary read of pool.bin as proof that the record committed. Under the
read-quorum rule, a COMMIT that reached only half of a 2+2 set was served
as committed, and the pool_meta_native_physical_* integration tests
failed.
Route reads through select_fileinfo_for_read: S3 buckets keep MinIO's
read-quorum rule, while is_meta_bucketname reads use the previous
algorithm unchanged as select_committed_fileinfo, so a split stays
unreadable until native recovery resolves it.
* fix(ecstore): serve the latest readable version past dangling versions
A failed versioned PUT whose rollback also failed can leave the new
version on top of fewer drives than its data blocks. Reading only each
drive's top version then left the committed version below it short of
read quorum, so GET and HEAD of the latest version answered 503 although
that version is intact in every stack. MinIO resolves the stacks in
pickLatestQuorumFilesInfo.
When every drive answered and an S3 latest read still cannot select a
version, walk the newest version headers of each stack through
exact-version reads: serve the readable version with the newest selected
modification time, skip versions whose exact read proves them dangling
(the NoSuchVersion an explicit GET reports) and queue heal for them, and
keep the read-quorum error on any other outcome. Candidates are ordered
by each version's newest modification time across drives, and at most
eight exact reads are spent.
Test PUTs now wait for every drive so tests that edit one drive's
metadata no longer race the PUT's background tail writes.
* chore(ci): shrink the error(format!) ratchet baseline for rebalance
The rebalance control and meta modules now have fewer
Error::other(format!) call sites than the baseline records, which
fails the shrink-only ratchet in Quick Checks. Record the smaller
counts.
* fix(ecstore): preserve canonical order in latest-version fallback
Keep the diagnostic implementation private to its owner and expose explicit feature-gated helpers through the existing storage boundary.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(decommission): cover held target lock and partial-copy resume
Port two tests from the closed lock-recovery branch onto the deferral
that defers any typed namespace lock timeout on a version copy.
- A version commits, then every attempt at the next one times out on
the target lock: the entry is deferred as a lock-timeout deferral, no
failed item is counted, both source versions stay, and the resumed
entry moves both versions with their exact bytes.
- A real write lock held on the target key makes target selection
return a typed object-lock timeout, the entry defers instead of
failing, and once the lock is released selection recovers and the
resumed entry completes the move.
* chore(ci): shrink the error(format!) ratchet baseline for rebalance
The rebalance control and meta modules on main now have fewer
Error::other(format!) call sites than the baseline records, which
fails the shrink-only ratchet in Quick Checks. Record the smaller
counts.
Keep timing and request correlation in a separate default-off feature. Preserve the existing positioned-read backend, byte and error contracts.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(sftp): allow bucket-scoped root listings
Listing the SFTP root required s3:ListAllMyBuckets. Without it a
bucket-scoped user got PermissionDenied on `ls /`; with it they saw every
bucket on the instance. S3 ListBuckets and the WebDAV root (#6298)
already fall back to a per-bucket filtered listing.
When the global ListAllMyBuckets check is denied, list buckets through
the session-aware ListBuckets path instead, so IAM filters the result to
buckets the principal may ListBucket or GetBucketLocation. Principals
with ListAllMyBuckets keep the unfiltered listing, and IAM outages and
timeouts still fail as before.
list_buckets_for_session and its request builder were WebDAV-only; they
are now also compiled with the sftp feature.
* fix(protocols): re-check the session key for bucket-scoped listings
list_buckets_for_session built its request from the identity captured at
login. Through the SFTP root-listing fallback, a service account disabled
or expired after login could keep listing bucket names, unfiltered when
its parent may list every bucket.
Validate the session key with check_key_valid_with_context, as
authorize_protocol_request does for every other protocol operation, build
the request from the fresh credentials and is_owner, map an invalid key
to AccessDenied, and attach the server context.
The SFTP fallback now passes the session's request headers and transport
security, like every other SFTP authorization. DummyBackend fails closed
when no session listing is queued, matching the trait default, and gains
a stall_authorization hook. New tests cover a service account disabled
mid-session, IAM outage and timeout on the global check (no fallback),
non-AccessDenied backend errors, truncation of fallback output and the
fail-closed default.
* test(protocols): compare error code without referencing s3s
The session-listing test compared against s3s::S3ErrorCode, which made
policy_test.rs a new s3s-referencing file and tripped the s3s footprint
ratchet (210 files against a baseline of 209). Compare the code's string
form instead.
* fix(protocols): refresh session bucket discovery without stale copies
Build bucket listing requests directly from freshly validated credentials
and server-scoped ownership instead of cloning and replacing login state.
Reuse the same authorization path for WebDAV capacity discovery.
Preserve typed access-denied checks through the protocol facade, and cover
request conditions, session-policy updates, revocation, listing bounds,
and fallback deadlines with regression tests.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
A node killed mid-copy leaves its namespace lock on the peers until the
lock lease expires. When the restarted leader resumes the decommission
and reaches that key inside the lease, every in-place copy attempt times
out acquiring the lock, and the version was counted as a failed object,
so the pool decommission could no longer complete.
A lock acquisition timeout leaves the source in place, like a version
copy that lost drive quorum, so defer the entry to the end of the round
instead of counting it as failed. A replay that still times out pauses
the decommission through the existing durable-pause path, which is
relaunched automatically.
This is the intermittent failure of the distributed e2e case
scanner_heal_topology_decommission_kill_before_source_cleanup_resumes_without_loss.
Report selected integer offsets and widths with strict member and payload validation. Cross-check compiler fixtures with bpftool and independent C offsetof/sizeof results.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): select the latest version at read quorum like MinIO
GET and HEAD escalated latest-version selection to write quorum when
enough metadata answered and skipped a newer version below write quorum
on degraded reads. LIST and heal already pick the newer side of a split
write at read quorum, so such keys listed but answered 503 until heal.
Select with MinIO's getObjectFileInfo rule: the most common mod time
that reaches the object's data blocks wins, ties go to the newer time,
and one content identity at that time must reach the same quorum. Two
identities tied at the selected time fail instead of resolving by hash
order, and the selected identity's version-stack shape follows disk
order.
* fix(ecstore): resolve selection ties by disk order and guard writes
Review of the read-quorum selection found three gaps:
- Rejecting a same-time half split made explicit-version reads of a
partially applied tagging update fail and blocked the retry that
repairs it. Resolve ties to the identity on the lowest-index disk,
for reads and heal alike, instead of rejecting or using map order.
- Delete markers and empty versions can lower the common read quorum
below the selected version's data blocks, so HEAD could report a
version GET cannot decode. Require that many copies, as MinIO's
getLatestFileInfo does.
- A selection can now return exactly read-quorum disks, and partial
metadata writes are not rolled back. Refuse an update before writing
when its target disks cannot reach write quorum.
* fix(ecstore): count identity copies in the data-blocks guard
The first-candidate shortcut counts byte-identical copies, so copies
that differ only in identity-neutral fields such as replication status
were missing from the data-blocks check. When the shortcut's count
falls short, count every copy of the selected content identity before
refusing the version.
* fix(ecstore): keep commit-proof selection for system-bucket reads
Pool metadata's PREPARE/COMMIT recovery and identity epochs treat an
ordinary read of pool.bin as proof that the record committed. Under the
read-quorum rule, a COMMIT that reached only half of a 2+2 set was served
as committed, and the pool_meta_native_physical_* integration tests
failed.
Route reads through select_fileinfo_for_read: S3 buckets keep MinIO's
read-quorum rule, while is_meta_bucketname reads use the previous
algorithm unchanged as select_committed_fileinfo, so a split stays
unreadable until native recovery resolves it.
- Resolve existing directories below base on Linux in a single openat2(BENEATH | NO_SYMLINKS) syscall in open_existing_directory_below_base_std instead of opening and closing every ancestor component.
- Reuse the pinned source directory guard across sibling metadata reads and control probes in validate_source_mutation, load_source, OrdinaryMultipartDeleteCheck::validate, and prepare_multipart_commit_staging.
- Read normal multipart source heads (<= 64 KiB) in a single pass in load_source while preserving constant-memory streaming observation for large prepared histories.
- Record root_missing during load_source, reuse loaded.original in restore_original, and coalesce finish source directory fsync into persist_with_source_sync.
- Reuse the opened commit directory in prepare_rename_commit_guard and LinuxCommitDirectories::validate when source_parent == destination_parent.
* test(scanner): add memory acceptance runner and budget probes
Add owned S3 workload phases with process, endpoint, cgroup and binary
identity checks for scanner memory and scanner-disabled request bursts.
Keep scale, restart, heap-attribution and overall-acceptance limits explicit.
Characterize healthy and seeded coverage-gap cache retention through the
real folder walker and production cache codec without changing runtime code.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): bind acceptance capacity to effective storage
Validate the serving process's complete local disk inventory before load
writes and during runtime sampling. Resolve each disk inside the owned
storage root, reject nested mounts and filesystem mismatches, and bind
storage observations to phase identity for container and native runs.
Cover the unused-mount bypass, CLI-overridden environment, native paths,
symlink escapes, named volumes and stable multi-disk inventory ordering.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): fence status observations before workload writes
Close each status API window with a second storage-bound process sample
before Phase entry can authorize bucket creation or PUT requests.
Add a fail-before regression for a process change during the initial status
request, asserting no ownership or workload writes occur.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Retain original prepare and commit-settlement RPCs beyond failed replacement and Abort cleanup, with and without bucket recreation. Require typed refusal or a physically verified idempotent result, unchanged fresh-upload bytes and complete object reads.
Extend scoped proxy observations without rewriting signed frames or trailers, and preserve registered test membership and existing budgets.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(tools): add read-only BTF prototype inspection
Expose bounded declared prototype headers through a standard-library CLI, preserve explicit void and failure semantics, and wire local contracts plus an independent compiler/bpftool reference into CI.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(tools): retain native reference failure diagnostics
Clear checkout credentials and capture bounded native stderr alongside stdout, preserving exact exit status and owned child cleanup. Keep the first reference failure unresolved rather than guessing its missing diagnostics.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* ci(tools): select a concrete bpftool file parser
The hosted PATH wrapper can exist without tools for its Azure kernel. Select an executable ELF by availability/version before comparing the fixed fixture, installing generic tools only when needed. Data comparison errors remain fatal and never trigger tool switching.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Verify a sixteen-drive set rejects reads and writes after two nodes stop,
then resumes complete S3 reads and new writes when only one node returns.
Keep the fourth node stopped throughout checks from all three live
coordinators and verify the original 12+4 shard geometry on disk.
Let the existing GET recovery probe retry the compatible SlowDownRead
response only with HTTP 503, retaining fatal status and byte checks.
Register the case in both platform selections and document the boundary
between process recovery and physical disk reattachment acceptance.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
- Document rules in crates/ecstore/AGENTS.md and the adversarial performance lens against per-disk spawn_blocking hopping, parent-chain openat amplification, un-sized read_to_end buffers, and redundant state reloads under the same lock.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d85aaf47-2a4b-4c2c-97a3-dbe8c7717e2a
* fix(observability): correlate physical namespace lock releases
Map logical namespace guards to their actual granted peer lock IDs and local physical guard identities. Record physical completion only after successful manager release and registry removal, and record validated positive remote unlock responses without changing release behavior.
Exercise real distributed ownership, asynchronous cleanup, duplicate and disabled guards, stale-owner retirement, and explicit release followed by Drop. Document the required diagnostic filters and the remaining filesystem-tail boundaries.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): remove unused lock manager test import
The diagnostic regression uses inherent manager methods and does not need the LockManager trait import. Remove it to pass strict warning checks.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): compact completed subtrees during checkpoint sweeps
Release completed object entries while preserving resumable frontier nodes,
including heal rewalks and coverage gaps. Borrow child links during usage
aggregation and remove duplicate update-cache subtree copies.
Add real-disk restart and allocation-retention regressions for #8961.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(scanner): avoid unused progress trees and traversal copies
Bind progress caches to actual subscribers and keep durable scanner metadata
in one authoritative cache. Remove mirror-only heal synchronization, use one
tagged folder queue, and consume child ownership when deleting subtrees.
Preserve observer output, restart coverage and failed-object retry semantics.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(scanner): compact completed leaf entries in place
Keep completed-folder eligibility and frontier handling in one compaction
boundary. Borrow leaf statistics and update the cached sampling flag without
cloning, deleting, or reinserting the entry; aggregate only non-leaf nodes.
Pin allocation identity, complete metadata, failure guards and caller state
with a regression that fails under the previous compaction implementation.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
A cursor for a key that literally ends in a legacy-shaped tag is issued
wrapped in one extra `[rustfs_cache:v2,return:]`, because the listing layer
still strips one legacy tag from incoming markers. ListObjects (V1)
unwrapped it in `calculate_next_marker`; the client then sent the bare key
back as `marker`, the listing layer stripped the key's own tail, and the
next page restarted at the stem and returned the same key forever. Keep the
wrapper in V1 NextMarker and cover the V1 resume path (marker sent back as
StartAfter) on a real store.
Document that sampled list quorum modes can also omit committed objects
whose copies mostly sit outside the sample, and recommend `strict` for
backup, sync, replication and lifecycle tooling.
- Coalesce InternalBuf::Ref and InternalBuf::Owned in write_all_internal
onto a single blocking_queue operation, avoiding 3-5 extra spawn_blocking
round-trips per disk on metadata writes.
- Short-circuit recover_part_transaction when no disk reports an active
part transaction (new.meta), skipping redundant rollback and part.N.meta
reads on normal UploadPart calls.
- Pre-allocate exact buffer capacity from file.metadata().len() in
read_file_all and read_multipart_metadata, reducing per-file read()
syscalls to 1 and avoiding Vec growth reallocations.
- Collapse os_mkdir_all and make_dir_all to a single fast-path blocking
call via create_dir_all_std.
- Accept &Path in BucketExistenceCache and use cached_access in
rename_file, create_file, and read_version.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d85aaf47-2a4b-4c2c-97a3-dbe8c7717e2a
* chore(deps): upgrade russh to 0.65.0 and refresh crates
Run two cargo update and cargo upgrade rounds from origin/main.
Adapt SFTP public-key callbacks to PublicKeyOffer and VerifiedPublicKey,
preserving explicit rejection and password-only authentication.
Cover the rejection policy with a pinned-host-key SSH handshake test.
Refresh Arrow, Parquet, Brotli, ZIP, and compatible dependencies.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(deps): route legacy zstd consumers to fixed codecs
Replace the old registry implementation with a private version-constraint
adapter backed only by zstd 0.14.3 and zstd-safe 8.2.0. Preserve legacy
features and Avro, Arrow IPC, Parquet, and Kafka formats.
Cover abandoned contexts, legacy decode errors, seekable bounds, stored
frames, consumer interoperability, and resolved implementation versions.
Record the adapter removal condition in the compatibility register.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Hauser