Separate completed traversal from persisted object health debt so leader
handoffs reuse healthy subtrees and retry failures without losing progress.
Fence old readers with a new receipt digest domain and preserve exact
accounting under diagnostic eviction, compaction and punctuation ordering.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(filemeta): read large metacache records incrementally
Replace the aggregate 16 MiB guard with fallible chunked growth so
versioned xl.meta records remain listable without trusting length prefixes.
Discard skipped payloads with a bounded buffer and distinguish truncated
records from compatible empty or between-entry EOF.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(filemeta): remove redundant metacache reader copies
Track the returned entry with a boolean instead of cloning its metadata,
decode object names directly from the borrowed slice, and derive the read
position from buffer length. Preserve scratch reuse, exact output capacity,
wire framing, and sticky corruption errors.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(utils): NUL-terminate and extend paths passed to Win32 volume APIs
`get_volume_name` returned the volume root truncated at its NUL and the
result was then passed to `GetVolumeInformationW` as a PCWSTR, an
out-of-bounds read. The disk-space and volume APIs also received the
caller's raw path, so they failed beyond MAX_PATH, and a UNC share root
lacked the trailing backslash these APIs require.
- Wrap every Win32 path in a NUL-terminated `WidePath` that rejects
interior NULs; volume roots returned by `GetVolumePathNameW` keep
their terminator.
- Convert inputs to the extended-length `\\?\` / `\\?\UNC\` form after
`GetFullPathNameW` normalization, and give directory queries a
trailing backslash.
- Query `GetDiskFreeSpaceW` and `GetVolumeInformationW` on the volume
root as documented; size buffers to the documented MAX_PATH + 1 or
the input length.
- Report Win32 error codes instead of raw HRESULTs so `ErrorKind`
matches `std::fs`.
- Keep volume ids in their conventional `C:\` / `\\server\share\` form.
Add Windows-only `os::` tests for long directories, UNC share roots and
non-empty fstype, and run them in the Windows filesystem workflow.
* fix(utils): report Windows network volumes to the fs guard
Windows names the server's filesystem (usually NTFS) for SMB shares, so
the startup unsupported-filesystem guard, which matches fstype names
such as cifs or smb2, never fired for UNC paths or mapped drives.
- Add `DiskInfo::remote`. On Windows it is set when the directory handle
answers `FileRemoteProtocolInfo` (only network redirectors do, and the
handle follows symlinks and junctions to their target) or the volume
root's `GetDriveTypeW` is `DRIVE_REMOTE`. Unix keeps it false because
the client fstype already names network mounts.
- Treat remote volumes as unsupported in the startup guard and label
them "<fstype> on network share".
- Cover UNC shares, paths beyond MAX_PATH below a share, a directory
symlink to a share and a `net use` mapped drive with Windows-only
`os::` tests that skip when `\\localhost\<drive>$` is unreachable;
the Windows filesystem workflow maps the drive before running them.
* chore(deps): replace yanked inout 0.2.3 with 0.2.4
cargo-deny's advisories check rejects the yanked inout 0.2.3 that main's lockfile pins, failing every open PR. Lockfile-only patch bump.
* fix(utils): detect network targets from the final NT path
On the Windows runner, a directory symlink to \\localhost\C$ was reported as local: FileRemoteProtocolInfo did not identify it, and the volume root of the link is a fixed drive.
Also treat a handle whose final NT path (GetFinalPathNameByHandleW with VOLUME_NAME_NT) lies under \Device\Mup\ as remote; the Multiple UNC Provider fronts every UNC and mapped-drive target. The tests now assert the handle probe on its own for local and share paths and report both signals when it disagrees.
---------
Co-authored-by: overtrue <anzhengchao@gmail.com>
* fix(ecstore): share metadata cache identity across Windows key aliases
Windows disks treat `\` as a path separator and `path_join_buf` drops the
empty segments that leaves, so `a\b`, `a\\b`, `\a\b` and `a\b\` all
address the xl.meta of `a/b`. The GetObject metadata cache keyed and
hashed the raw spelling, so DELETE `a\b` retired nothing cached for
`a/b`: GET/HEAD `a/b` kept returning the deleted object for up to the
2s cache TTL.
Fold the object name into the same segment form the disk resolves to
before computing the cache hash, key and lookup key. Off Windows the
name stays verbatim and borrowed, so `\` remains an ordinary key byte.
* fix(ecstore): serialize Windows backslash key aliases on one namespace lock
Windows disks treat `\` as a path separator and drop the empty segments
it leaves, so `a\b`, `a\\b`, `\a\b` and `a\b\` all address the xl.meta
of `a/b`. Namespace locks keyed the raw spelling, so writes, deletes and
heals through different spellings ran concurrently on one xl.meta and a
read-modify-write in rename_data could drop a version.
Fold the object name into its disk identity (rustfs_utils::path::
object_name_disk_identity, a no-op off Windows) wherever a namespace
lock key is formed:
- SetDisks::new_ns_lock, which every object lock path goes through;
- the batch delete lock request and its dedup/lookups, which build
ObjectKeys directly.
Two consumers compared lock identities by spelling and follow along:
- WriteCommitGuard records the folded key from the lock resource, so
matching a borrowed guard now folds the target name too;
- CopyObject holds the source read guard until EOF on streaming reads.
`a\b\` -> `a/b` is not a same-object copy by path_join_buf, but shares
one lock, so it now takes the self-copy lock instead of waiting on
itself until the lock timeout.
Distributed lock keys are folded by the requesting node only; lock
servers compare them verbatim.
* refactor(ecstore): key the metadata cache by the shared disk identity
The metadata cache fix carried its own copy of the Windows backslash
folding rule. Use rustfs_utils::path::object_name_disk_identity, which
namespace locks already use, so cache and lock identities cannot drift.
* ci(windows): run filesystem tests when key identity rules change
The Windows backslash folding for object names lives in utils path.rs
and is consumed by the namespace lock and write commit guard; run the
windows_ tests when any of them changes.
* fix(ecstore): keep Windows disk roots out of verbatim form
The disk root comes from GetFinalPathNameByHandleW, which returns a
verbatim path. `simplified` only strips `\\?\C:` below MAX_PATH, so a
data directory past 260 characters (or one reached through a UNC share)
kept the prefix. Disk IO extends the root by joining `/`-separated
relative paths as strings (`read_raw` builds `<object>/xl.meta` with
`path_join`), and inside a verbatim path `/` is not a separator: the
first metadata read, pool.bin.identity, failed with ERROR_INVALID_NAME
until startup gave up.
Add `rustfs_utils::non_verbatim`, which strips the disk and UNC verbatim
prefixes without a length limit while still keeping paths whose
components Win32 would rewrite, and use it for the publication root.
std re-adds the extended-length prefix for long paths.
Cover a root past MAX_PATH with a disk-level read/write test and run all
`dunce::` tests in the Windows filesystem workflow.
* fix(ecstore): query Win32 volume APIs with extended long disk roots
A disk root past the legacy limit used to reach `rustfs_utils::os` in
verbatim form, which the Win32 volume APIs accept. Now that the root
stays non-verbatim, GetDiskFreeSpaceExW rejected it with
ERROR_PATH_NOT_FOUND and LocalDisk::new failed in the Windows filesystem
workflow. Pass long roots to `get_info` and `get_physical_device_ids` in
extended-length form; shorter roots stay unchanged so reported volume
identities keep their `C:\` form.
On Windows, URL endpoints such as http://host:9000/C:/data{1...4} (the only
form multi-node and pool-expansion layouts accept) produced the disk path
`/C:/data1`: `try_from` stripped the synthetic slash, but `Url::set_path`
put it back, and `get_file_path` only removed it for `file` endpoints.
Windows resolves `/C:/data1` as an invalid `C:` component under the current
drive, so disk validation and `resolve_local_disk_root` failed.
- `get_file_path` now restores `/X:/...` to `X:/...` for every scheme on
Windows; URL paths without a drive (`/export1`, `/C:foo`) are unchanged.
- UNC shares (`\\nas\share\d1`, `//nas/share/d1`, `\\?\UNC\...`) are rejected
with an explicit error instead of becoming a file URL whose host is the
share server and whose path drops the share name.
- A drive root (`http://host:9000/D:/`) is accepted in URL form, matching
the path-style `D:\` endpoint that was already accepted.
- The "distinct physical disks" error explains on Windows that directories
on one volume share a device identity, alongside the existing
RUSTFS_UNSAFE_BYPASS_DISK_CHECK hint.
- The Windows filesystem lane now runs on layout changes and runs the
layout::endpoint:: tests; new Windows tests carry the windows_ prefix.
- Document Windows endpoint forms, the same-volume check, and the SNSD
expansion limit in cluster-lifecycle-operations.md.
`run_process` called `block_on(async_main())` on the native main thread,
so the whole startup chain ran on whatever stack the platform hands the
main thread: 8 MiB from `ulimit -s` on Linux and macOS, but the 1 MiB PE
default on Windows. A Windows debug build overflowed before it listened.
`RUST_MIN_STACK` does not reach the native main thread.
Run the runtime on a `rustfs-main` thread with an explicit 8 MiB stack,
keeping the budget Linux and macOS already had and making it independent
of the linker, toolchain (msvc/gnu) and `ulimit`. Panics are re-raised
on the main thread so the exit status is unchanged.
Measured on macOS arm64 by lowering the main-thread stack with
`ulimit -s` before this change: debug startup overflows at 1792 KiB and
reaches /health/ready at 2048 KiB; release overflows at 512 KiB and is
ready at 768 KiB. After it, debug starts with a 256 KiB main thread.
Add a Windows `Startup Smoke` job that builds the debug server, runs
`--version`, and waits for /health/ready on a single volume.
* fix(ecstore): admit CopyObject destinations like PutObject
Object-key platform checks were split across call sites with gaps:
CopyObject only ran check_bucket_and_object_names on its destination and
then reached the pools' put_object directly, so it admitted keys PutObject
rejects (over 1024 bytes, leading slash, and on Windows reserved characters
or unaddressable segments, the "writes but cannot be read" case of #3449).
- rustfs-utils: expose check_windows_file_name as the single per-segment
Win32 rule set; dunce's simplified() reuses it. Reserved device names now
include COM/LPT with superscript digits and CONIN$/CONOUT$.
- ecstore: replace the duplicated reserved-name table with that function;
Windows also rejects control characters 0x01-0x1F (NUL keeps its
platform-independent ObjectNameInvalid answer).
- ecstore: add check_new_object_args as the shared admission entry for
PutObject, cross-key CopyObject destinations and CreateMultipartUpload.
In-place copies (metadata replace, restore) keep the copy-only check so
existing keys stay rewritable. CreateMultipartUpload now also rejects
over-NAME_MAX segments up front instead of failing at completion.
- CI: run bucket::utils and dunce tests in the Windows filesystem job.
* fix(ecstore): reject new backslash keys on Windows hosts
A Windows disk treats '\' as a path separator, so a new key `a\b` lands on
the same xl.meta as `a/b` and the two keys overwrite each other.
Reject '\' only where a key enters the namespace (check_new_object_args:
PutObject, cross-key CopyObject destinations, CreateMultipartUpload) and
only on Windows. Reads, deletes, in-flight multipart operations, in-place
copies, data movement and internal meta buckets keep accepting it, so keys
already stored with '\' stay readable, deletable and movable.
Pending owner ruling: whether new '\' keys should be rejected at all, and
whether reads/deletes should keep accepting them.
* ci(windows): quote filtered test runs and drop duplicate path
fix(data-usage): retain dirty counts over observations
Preserve unreconciled local mutations when an observed snapshot completes later. Add focused selection-to-overlay regressions for writes, deletes and authority boundaries.
Validation checkpoint: the pre-fix regression failed as expected. Post-fix tests, formatting and runtime verification were not run before the user stopped the task. This local commit is not ready for publication.
tempfile's Windows `persist` passes the raw paths to MoveFileExW and
SetFileAttributesW, bypassing std's extended-length handling. Once
`<object>/<rollback>/xl.meta.bkp` reached 260 UTF-16 units every delete
of that key failed with 500, while PUT and GET kept working.
Add `rustfs_utils::extended_length_path`, which normalizes a path with
GetFullPathNameW and prefixes it with `\\?\` (or `\\?\UNC\`), and use it
for the staging directory and destination handed to tempfile. Cover the
single and batch delete paths with a deep-key regression test and run the
helper's Windows assertions in the Windows filesystem workflow.
Fixes#8545
Since 1.0.2 an encrypted object's ETag is a keyed fingerprint, not the
plaintext MD5. A target that computes its own ETag (AWS S3, Wasabi) never
reports it, so every "find the existing replica by exact ETag" step missed
encrypted objects on targets that mint their own version ids:
- a heal / MRF re-drive after a lost PUT response wrote a second version;
- an existing-object resync re-PUT a replica its ledger already named
(the ledger-located HEAD was confirmed by ETag only), and on the AWS
400 shape the ledger was not consulted at all;
- resync verification counted such replicas as failed;
- a ledger-less version purge reported the replica as already gone.
Encrypted sources now recognise their replica by shape: the byte count the
replication PUT sent (plaintext for managed SSE, ciphertext for SSE-C) and
a target write time no earlier than the source version. An ETag match
still wins (RustFS/MinIO targets). The ledger is consulted first on both
the 404 and the AWS 400 shape. Without a ledger, only a unique shape match
among the unclaimed ListObjectVersions candidates is adopted and recorded;
several shape matches mean re-PUT for convergence and a refused purge. The
AWS current-version fallback stays ETag-only, and a listing the
target refuses on the 400/403 shape keeps that fallback (re-PUT) rather
than stalling. An unversioned target's `null` version is never matched by
shape: it holds whichever generation was written last. Plaintext sources keep exact
ETag identity.
Builds on the ETag-computing fake target mode and the ComputesOwnEtags
matrix row from rustfs#8583. Adds an SSE-S3 variant of the mint-own-ids
re-drive test (red before this change: the heal left two target versions),
and both variants now also assert the purge.
* fix(scanner): treat replication status writebacks as status-class hits
A replication status writeback (persist_replication_state_if_current in
Update mode, including the heals the scanner queues during its own walk)
marked the bucket dirty on the namespace path: it advanced the dirty-usage
generation, woke the scanner, reached the durable dirty journal and
entered the activity snapshot. Any replicated bucket therefore made every
scanner cycle Superseded, the authoritative usage baseline was never
certified and usage freshness stayed stale for as long as replication ran
(#8375).
The writeback is now status class: record_status_class_mutation records
only the process-local producer coverage bit and nothing else. There is
no rescan queue for it, because a deployment with an active replication
rule already walks every bucket on every cycle
(ScannerMaintenanceFeatures::needs_regular_cycle) and that walk refreshes
replication_stats. The routing lives in a named, unit-tested function in
rustfs/src/storage/storage_api.rs; tier-expiration cleanup removes data
and stays on the namespace path.
No persisted format, protocol, admin API or configuration changes.
This is phase 1 of docs/architecture/scanner-per-bucket-convergence.md,
which this change adds together with its index entry.
Refs #8375, #8494, rustfs/backlog#2816.
* fix(scanner): refresh replication statistics during maintenance
Refresh compacted prefixes and completed bucket caches during replication maintenance. Bind set execution reuse to the observed replication statistics so a same-cycle retry cannot restore older counters.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(scanner): avoid expanding the legacy S3 DTO surface
Decode the existing replication configuration type from a JSON fixture instead of adding direct S3 DTO references to a new file. Preserve the compacted-prefix and same-cycle assertions and the footprint ratchet.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(scanner): keep raw enumeration bookkeeping linear per entry
The page writer inserted into its building page with a push-and-sort on
every entry and committed each page through a clone. Insert in sorted
position and move the committed page into the index instead.
Add a regression test that records 20,000 entries in one prefix and pins
the cost to one hash per entry and well under a second, so the quadratic
re-normalization removed by #8114 (rustfs#8113) cannot return once pacing
builds on this walker.
* feat(scanner): decouple bucket walkers from disk count
Add the scanner key walkers_per_disk (RUSTFS_SCANNER_WALKERS_PER_DISK,
1 through 16, default 1). Each local coordinator disk of a set spawns that
many bucket walkers; remote disks keep one. The walker count is what
max_concurrent_disk_scans caps and the disk count stays the floor.
The walkers of one disk share its scanner admission through
LocalDiskAdmissionShare: the first walker to start a bucket takes the
per-disk permit and the last to finish releases it, so a peer's remote
request still sees one scan session per physical disk. Without the share
the second walker was rejected as busy and retired.
The disk_concurrency_budget log line reports walkers_per_disk, walkers and
the reason for the effective limit. The default stays 1 because no local
disk reports rotational media today; the design keeps a higher flash
default as the intent.
* docs(scanner): document walkers_per_disk in the Helm chart and configuration manuals
RUSTFS_SCANNER_WALKERS_PER_DISK was only described in the scanner runtime
controls table. Add it to the Helm chart (values, configmap rendering,
README table, template test), the configuration crate's environment
reference with operator guidance, and a short section in the operations
manual on when to raise it and how to confirm it took effect.
---------
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* chore(deps): refresh workspace dependencies
Run cargo update and cargo upgrade twice from origin/main.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(deps): pin Pulsar 6.9.0 until recovery is bounded
Keep Pulsar on 6.9.0 after reproducing background recovery and lookup activity after delivery timeout and target destruction in 6.9.1. Use an exact requirement to prevent a later cargo update from selecting the affected release.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Enable preserve_order for the Arrow JSON schema reference in tests and verify first-seen field ordering across nested records.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Since #8535 DeleteObjects honours Quiet=true the way S3 does and omits the
successful entries from the response, so the quota batch-delete test has
been asserting two Deleted entries against an empty list on every main run.
The test only needs the batch to succeed and free quota; make it verbose so
the existing assertion holds and check that no entry failed.
* test(e2e): gate the rc.5 baseline restarts on drive re-admission
The mixed-version rolling upgrade failed three times on unrelated branches
with `MultipartAcknowledgementDeadline { node: 2, key: "one-current-node/..." }`.
The baseline kills the next rc.5 node about 10 ms after the previous one
reports ListBuckets, while that node is still replaying its startup traffic.
Two failed operations drop the peer's drive to Offline on the pinned release,
and only the 2 s recovery probe brings it back three successes later. In both
failing runs node 2 held node 3's drive as `faulty` when the multipart load
started three seconds later; it then committed every upload at three of four
drives with a heal per object, and on a two-vCPU runner one upload exceeded
the 30 s acknowledgement budget. The green run on the same branch only
reached `suspect`. An all-rc.5 cluster reproduces the same degradation after
the same restart sequence and an all-current cluster shows none of it.
Write a slash-free probe object before the first restart and, after each
baseline restart, block until every node reports all four drives of that
object as `ok` through a dry-run object heal. `storageinfo` and `info` only
carry each node's local drives; the object heal runs on the receiving node and
classifies a short-circuited remote drive as `faulty`. The gate stays in the
all-rc.5 baseline: admin heal first probes every peer's heal-control
capability, and a mixed-version cluster fails that probe on both binaries.
Record every node's lock table in `mixed-multipart-failure.json`; the stalled
writer is a previous-release node whose table the current node cannot show.
The 30 s budget, layout, load and assertions are unchanged.
* test(e2e): probe drive re-admission with a parity-0 write per node
The dry-run heal gate observed only node 0: the pinned rc.5 admin heal
forwards start and status requests to the heal-control coordinator whatever
node receives them, so the loop checked node 0's drive handles four times
and never the restarted peers' handles.
Run the cluster with `RUSTFS_STORAGE_CLASS_RRS=EC:0` and probe each node
with a reduced-redundancy PUT instead. A parity-0 write on the four-drive
set needs all four drives (`write_quorum = data_drives`), so the PUT fails
with 503 `ServiceUnavailable` until the serving node writes through every
one of its own drive handles. The probe is plain S3 and executes on the
receiving node on both binaries, so the gate now also runs before each
mixed-version load and after the last upgrade. Standard-class objects keep
the default layout.
`parity_zero_write_needs_every_drive` pins the premise on the workspace
build: with one node down a standard write still commits, the
reduced-redundancy write fails with 503, and the gate converges once the
node returns. The diagnostic snapshot test now checks the per-node lock
probe array.
* fix(rebalance): avoid contention when peers join the same run
* fix(rebalance): wait for fleet proof during initialization
* test(rebalance): check contextualized competing run errors
* fix(rebalance): retain typed worker admission lock errors
* test(rebalance): retry the init readiness 503 and pin the retry error shape (#8586)
The e2e harness only retried the pre-fix 500 carrying the fleet-proof
marker; the init stage now answers 503 + Retry-After after its own
readiness budget, so start_rebalance treats that as the same wait. The
missing-proof regression also asserts the typed error shape that the admin
handler's retry predicate depends on, not only the rendered message.
---------
Co-authored-by: Chris <anzhengchao@gmail.com>
* fix(multipart): retain commit guards across request cancellation
* fix(multipart): preserve commit task error context
* fix(multipart): reclaim verified orphan upload staging (#8572)
* fix(multipart): reclaim verified orphan upload staging
* fix(multipart): share orphan guards across matching lock domains
* fix(multipart): preserve orphan cleanup task error context
* fix(ecstore): box guarded delete transition identity
* fix(multipart): keep NoSuchUpload when orphan recovery cannot prove absence (#8587)
Recovery is a side path of an idempotent Abort. An offline disk, lock
contention, partially overlapping lock domains or unreadable bucket
metadata all stop the staging-absence proof before any deletion; those
used to propagate as 503/500, where the base answered NoSuchUpload. They
now keep that answer and delete nothing; only failures inside the
deletion task remain request failures, because a retry can finish them.
---------
Co-authored-by: Chris <anzhengchao@gmail.com>
* feat(diagnostics): attribute storage reads to request origins
Port the diagnostics instrumentation from its experiment base onto main
without carrying the base branch's unrelated changes.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* refactor(diagnostics): consolidate read context scopes
Keep caller and entry provenance in one task-local context, combine the object-read scope, and classify source paths without allocating.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Read the connection peer address using the stored extension type and distinguish retryable stale epochs from terminal signature errors. Cover diagnostics and authenticated ReadAll recovery without changing replay protection.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
The outbound target matrix only modeled targets that adopt the forwarded
source ETag, so the post-PUT replica ETag audit was never exercised against
a target that computes its own ETag (AWS S3 and most compatible stores),
and no matrix shape was encrypted. That gap let the SSE-S3 "replica etag
mismatch" regression fixed in #8573 ship unnoticed.
Add an ignore_source_etags mode to the fake S3 target, a ComputesOwnEtags
matrix row that uses it, and an SseS3 object shape (the source runs with the
local KMS). The shape asserts its source ETag is not the plaintext MD5 so
the cell cannot pass vacuously. Against a build without #8573 the new row
fails exactly computes-own-etags/SseS3 with "replica etag mismatch"; with it
the whole matrix passes.
Also correct the #8573 docs and comment that called the replication PUT
checksum the integrity check for encrypted sources: managed-SSE sources
forward no checksum, so their replicas have no post-PUT content audit.
* fix(scanner): persist partial progress after budget cancellation
Separate progress-only cycles from observational and authoritative usage
updates. Keep peer movement leases alive under the parent cycle context
for a bounded drain after the scan budget stops the walk, while preserving
the existing strict publication fences for emitted usage updates.
Skip root rename fences and the usage persister when no update was emitted.
Cover real lease admission, cancellation, bounded retries, and durable
partial-cycle advancement without usage publication or dirty acknowledgement.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* refactor(scanner): consolidate publication policy and ownership
Classify emitted updates directly in the publication plan constructor and
exercise that single policy from the namespace-admission tests.
Share the parent cancellation state for read-only progress proof checks
and move the one-use root fence into the persister, avoiding a child token
allocation and a JSON string copy without changing lease or commit rules.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
`cargo clippy --all-targets -- -D warnings` fails on main since #8535:
`CurrentDeleteGeneration::Present` carries a `CurrentDeleteTransition`
(three Strings, an Option<String>, a 32-byte identity and two small
fields, ~211 bytes) while `Absent` carries nothing, which trips
clippy::large_enum_variant. The workflow only runs clippy on pushes to
main (`if: github.event_name != 'pull_request'`), so the PR CI stayed
green and main's CI has been red since 606d8a3a.
Store the transition behind a Box so the enum is pointer-sized plus the
small fields. The value is transient ("never persisted in object
metadata") and its serde form is unchanged, because serde treats Box<T>
transparently; PartialEq/Clone/Debug derive through it. The two tests
that matched the struct through the Option now bind the Box and check
the same fields in a guard.
Claude-Session: https://claude.ai/code/session_01CezCdjE9AdShFUi9tNk797
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
New encrypted writes store a keyed HMAC fingerprint as the object ETag
(docs/operations/encrypted-etags.md). Replication forwards that ETag on
x-rustfs-source-etag, but a target that computes its own ETag (AWS S3 and
most compatible stores) answers the plaintext PUT with the MD5 of what it
received. Both values are 32 hex characters, so verify_single_part_replica
treated the fingerprint as a comparable MD5 and failed every single-part
SSE-S3 / SSE-KMS replica with "replica etag mismatch"; MRF and resync then
retried the object forever. RustFS and MinIO targets adopt the source ETag
and were unaffected.
Exempt sources whose stored ETag cannot describe the wire bytes: any
managed-SSE or SSE-C source, decided by the existing replication
source-encryption classifier. The checksum header on the replication PUT
remains the integrity check for those objects; plaintext sources keep the
audit unchanged.
Claude-Session: https://claude.ai/code/session_01CezCdjE9AdShFUi9tNk797
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
A user-bucket delete fans out through the delete-transaction RPC. When a
participant answers Unimplemented (a peer still running a release without
that RPC, or a protocol-version mismatch) the coordinator reports
StorageError::UnsupportedDisk, which had no S3 mapping: clients received a
500 InternalError whose message was the variant's Display text,
"drive does not support O_DIRECT". That text describes the startup drive
probe and sends an operator mid-upgrade after the wrong cause.
Map UnsupportedDisk to 503 ServiceUnavailable with a constant message that
names the protocol-version cause and the remedy (finish upgrading every
node, then retry). The typed source is retained for diagnostics and no
Retry-After is added, since retrying the same node does not help until the
cluster upgrade completes. No request path other than delete recovery
produces this variant; the startup drive probe still fails startup as
before.
Claude-Session: https://claude.ai/code/session_01CezCdjE9AdShFUi9tNk797
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Hauser