Add a default-on multi-role adversarial validation policy to the root
AGENTS.md (risk tiers, six reviewer roles, findings protocol, exit
criteria), a shared adversarial-validation skill with repo-specific
attack probes grounded in shipped bugs, seven audit fixes to the root
instruction files, and an actionlint gate for workflow changes.
In the default (non-rio-v2) build, SSE-C encrypted with AES-256-GCM using a deterministic nonce derived from MD5("{bucket}-{key}"), and decrypt recomputed it. Overwriting the same object under the same SSE-C key reused an identical (key, nonce) pair, breaking AES-GCM confidentiality and integrity. The base nonce was never persisted.
Generate a fresh random 12-byte nonce for every single-PUT and multipart session, persist it under both x-rustfs-encryption-iv and the MinIO interop key, and resolve it from stored metadata on decrypt and multipart part encryption. Objects written before this change carry no stored IV and fall back to generate_ssec_nonce, so existing objects still decrypt.
Refs rustfs/backlog#1039
fix(object-capacity): harden clock handling in write window, background intervals and scope registry
Three low-severity robustness gaps (S32+S33+#35):
- WriteRecord keyed its 60-bucket write window by wall-clock unix
seconds while the debounce used Instant. An NTP step backwards marked
recent buckets as future and silently suppressed write-triggered
refreshes until the wall clock caught up; a forward step aged the
whole window at once. Bucket keys now derive from a monotonic
per-record epoch, immune to clock steps.
- Background refresh/metrics intervals were only clamped at zero;
RUSTFS_CAPACITY_SCHEDULED_INTERVAL=u64::MAX overflowed
Instant + Duration and panicked the spawned task, silently killing
scheduled refreshes. Intervals now clamp into [1s, 30 days] via one
helper with a structured warn.
- record_capacity_scope merged new disks into an expired entry and
refreshed its recorded_at, resurrecting a scope take_capacity_scope
would have discarded (and the merge path never pruned). Expired
entries are now replaced, not merged into.
Ref: rustfs/backlog#1022 (S32+S33+#35 from audit rustfs/backlog#1010)
- build.yml: update latest.json only for stable release tags
(alpha/beta/rc tags previously overwrote the stable pointer with
release_type "stable"); drop the placeholder .asc files; build the
Linux targets only on main pushes (tags/schedule/dispatch keep the
full matrix); remove the unreachable --build clause and a needless
full-history clone
- ci.yml: event-scoped concurrency so merges stop cancelling the
weekly scheduled run; drop the redundant skip-duplicate-actions
gate job; run clippy before tests; trim the unused toolchain from
the typos job
- ci-docs-only.yml (new): satisfy the required "Test and Lint" check
on docs-only PRs that ci.yml skips via paths-ignore
- audit.yml: drop cargo-audit (cargo-deny advisories covers the same
RustSec database); same event-scoped concurrency fix
- docker.yml: job-level short-circuit for per-merge dev builds; send
the Trivy SARIF to code scanning; unify the upload-artifact pin
- performance-ab.yml: add concurrency; only re-run on labeled events
when the added label is perf-ab
- stagger the Sunday crons (build 01:00, audit 03:00,
nix-flake-update 05:00, mint 06:00)
- delete performance.yml: disabled since 2025-07; idle-server
profiling, no benchmark baseline, stale ecstore package name
fix(metrics): normalize peer-health key so servers_offline_total counts each peer once
rustfs_cluster_servers_offline_total is the count of distinct keys in the
addr-keyed CLUSTER_PEER_HEALTH map. The data path keys a peer by
endpoint.grid_host() (scheme://host:port) while the lock path keys it by
url::Url::to_string() (scheme://host:port/, trailing slash), so one physical
peer becomes two health entries and a single downed node reports 2 (2N for N).
Normalize the address (trim trailing slashes) at the map boundary in
record_peer_reachable/record_peer_unreachable/cluster_peer_is_offline/
cluster_peer_should_bypass so both forms collapse to one canonical key.
Pure observability fix; peer selection and quorum are unchanged.
Fixesrustfs/backlog#1043
Co-authored-by: heihutu <heihutu@gmail.com>
The tracker never influenced traversal: walkdir's follow behavior is
fixed up front by follow_links(), and should_follow() only gated the
tracker's own bookkeeping. Its 'depth limit' compared tree depth (not
symlink chain depth), so RUSTFS_CAPACITY_MAX_SYMLINK_DEPTH was a
complete no-op while its telemetry claimed symlinks were skipped that
walkdir had in fact followed and counted; record_symlink was always
called with size 0, so tracked_bytes never left zero (S12).
Remove the tracker, its skipped/summary events, the symlink metric and
the depth env knob end to end (the env's 'as u8' truncation goes with
it), and document the real semantics at the walker: follow_links(true)
counts targets with walkdir's ancestor-loop detection breaking cycles,
follow_links(false) — the default — counts no symlink targets. The scan
root itself is pre-resolved since backlog#1015.
Ref: rustfs/backlog#1018 (S12 from audit rustfs/backlog#1010)
PR #4560 added a 15th parameter (allow_inplace_legacy_fallback) to
build_codec_streaming_part_reader while the negative arity tests from a
concurrently developed branch still call it with 14 arguments — the two
changes merged cleanly textually but broke the workspace test build on
main (E0061 in set_disk/read.rs), failing CI for every open PR.
Pass false at both call sites: these tests assert Err outcomes
(oversized part, missing quorum) that are independent of the fallback
mode, and false preserves the historical whole-request fallback
semantics used by eager first-part reads.
walkdir defaults to follow_root_links=true precisely so a root that is
itself a symlink still descends, but the scanner overrode it with the
general follow_symlinks flag (default false). For the common indirection
layout (/data/rustfs0 -> /mnt/vol0) the walk yielded a single skipped
symlink entry and returned used_bytes=0, file_count=0, is_estimated=false
with no error and no log — an exact 0 committed straight into the
per-disk baseline (S05).
Canonicalize the scan root before walking (also covering nested
indirection and keeping the dedicated-mount statvfs check on the real
path); resolution failures surface as scan errors instead of a silent
zero. Inner-symlink policy is unchanged.
Ref: rustfs/backlog#1015 (S05 from audit rustfs/backlog#1010)
RUSTFS_CAPACITY_MAX_FILES_THRESHOLD=0 made every file count as
overflow with an empty exact prefix, so disks with fewer files than the
sample rate committed 0 bytes as the cluster total; STAT_TIMEOUT=0 with
dynamic timeout disabled made every scan fail at the first entry. Both
were accepted unvalidated, unlike the already-clamped sample_rate and
interval values.
Zero values for the threshold and the stat/min/max timeouts now fall
back to their defaults with a single structured warn at config load
(S11). The symlink-depth knob is left untouched here: it is removed
entirely by the backlog#1018 fix.
Ref: rustfs/backlog#1019 (S11 from audit rustfs/backlog#1010)
Two ecstore test groups pass in isolation but flake under the full parallel
nextest suite, producing CI noise on every in-flight PR (backlog #937):
- store::bucket::tests::bucket_delete_* race make_bucket into
InsufficientWriteQuorum because they share global disk-registry/lock-client
state with other concurrently running tests.
- bucket_lifecycle_ops::tests::concurrent_resend_same_part_commits_one_generation
exceeds its already-maxed 60s lock-acquire deadline only when the suite
saturates disk I/O.
serial_test's #[serial] does not serialize these across nextest's per-test
process boundary. Add a nextest test-group (max-threads = 1) that matches both
groups so they run serialized, removing the load-induced flake. No test or
production code changes; long-term global-singleton isolation is tracked
separately.
Co-authored-by: heihutu <heihutu@gmail.com>
Progress/timeout checks only ran when file_count advanced, so trees of
directory-only or error-dense entries (dirs, traversal errors, symlinks
never increment file_count) bypassed the entire cooperative time budget
and held the blocking thread for unbounded time, while each error entry
logged its own warn — permission-dense trees flooded the log (S13).
The stall detector was structurally unreachable: it fired on 'no file
progress between checks', but checks themselves only ran when file
progress happened, so RUSTFS_CAPACITY_STALL_TIMEOUT never did anything
since its introduction (S09).
- Progress checks (timeout + early-sampling entry) are now driven by a
visited-entry counter that also advances on directories and errors, so
every tree shape reaches the budget checks.
- Remove the unreachable stall detector and its plumbing end to end
(ProgressMonitor fields, ScanLimits, config getter, env const, stall
metric). Genuine walker wedges are handled by the hard outer
wall-clock budget from backlog#1017; no decorative protection is left.
- Cap per-entry error warns at 10 per scan with an explicit suppression
notice; had_partial_errors still records the condition.
Ref: rustfs/backlog#1016 (S09+S13 from audit rustfs/backlog#1010)
perf(ecstore): wire legacy decode stripe prefetch / bitrot-decode overlap (backlog#930 HP-9 step 2)
The legacy GET decode duplex loop (default GET path + every Range request)
was strictly serial: read a stripe, reconstruct it, emit it, and only then
begin reading the next stripe. Two switches introduced by PR#3972 to overlap
this work — RUSTFS_GET_DECODE_STRIPE_PREFETCH_COUNT and
RUSTFS_GET_BITROT_DECODE_OVERLAP_ENABLE — were config-only with zero call
sites since introduction.
Wire both into the loop as a depth-1 stripe prefetch: while the current
stripe is reconstructed and emitted, the next stripe's shard reads (including
bitrot verification) run concurrently, hiding read latency under the emit /
duplex-backpressure stage. ParallelReader::read is inherently serial (it takes
&mut self and advances a shared stripe cursor), so at most one read can be in
flight; a prefetch count above 1 therefore collapses to the same
single-stripe-ahead pipeline rather than reading several stripes ahead. Both
switches feed one gate, legacy_stripe_prefetch_enabled().
Default (count == 1, overlap == false) keeps the loop on the pre-existing
strictly-serial path, byte for byte: the prefetch pipeline is a separate branch
and the serial branch is unchanged. The reconstruct/emit body is factored into
a shared emit_decoded_stripe helper used by both branches so error attribution,
read-quorum handling, reconstruction verification and stage metrics cannot
drift between them.
Correctness guarantees preserved under prefetch:
- A speculatively prefetched read for stripe N+1 is only consumed when the loop
reaches N+1; if stripe N stops the loop, the in-flight read is awaited and
dropped, so its errors never surface against stripe N.
- Bitrot (HighwayHash) verification runs inside each stripe read and is not
bypassed or reordered; a corrupt shard is still rejected and, when
unrecoverable, the read fails without emitting garbage.
- Shard buffers are recycled only after the overlapping next read has claimed
its own — one extra stripe of memory (double buffering) with buffer reuse
preserved at a one-stripe lag.
- Per-stripe exact length advance (backlog#799 B2), lockstep reconstruction
verification (backlog#832) and the hash_size == 0 pass-through are unchanged.
Adds regression tests exercising serial-default, count>1 and overlap configs:
full/range reads byte-exact, degraded (missing-shard) reconstruction, corrupt
shard rejected-but-recovered, unrecoverable corruption erroring with no output,
late-stripe failure attributed correctly with stripe 0 still emitted,
hash_size == 0 pass-through, and the gate defaulting off.
Co-authored-by: heihutu <heihutu@gmail.com>
Two report-only defects on the heal result-reporting surface (they do not
affect the data path or heal decisions):
- default_heal_result (set_disk/ops/heal.rs): the offline-disk branch pushed
an Offline record but fell through into the unconditional push, emitting a
second (Corrupt) record for the same disk. This grew before/after.drives to
disk_count + offline_count and misaligned every entry after the first offline
slot. Add `continue` after the offline push, drive `disk_len` and the loop
from a single `self.disks` snapshot, and assert `errs.len() == disk_len`.
- Sets::heal_format (core/sets.rs): the before/after drive lists were
pre-filled with N default placeholders and then N real entries were pushed,
yielding a 2N list whose healed status updates (indexed 0..N) landed on the
blank placeholder half. Assign the lists directly from formats_to_drives_info
(mirroring the set-level heal_format) so the healed updates hit the real
entries.
Add regression tests covering offline/online record alignment and the
NoHealRequired and heal paths of the pool-level heal_format.
Co-authored-by: heihutu <heihutu@gmail.com>
Sub-change A only: the two local-disk metadata read paths
(`read_metadata_with_dmtime`, `read_all_data_with_dmtime` in
crates/ecstore/src/disk/local.rs) previously dispatched open, fstat and the
xl.meta read as three separate async fs hops (three spawn_blocking round-trips
under tokio::fs). Each is now folded into a single tokio::task::spawn_blocking
closure using std::fs, cutting per-metadata-read dispatch from 3 to 1.
This does NOT touch sub-change B (removing the entry-point access() call).
Correctness first: this is the hottest metadata read path, so the refactor
preserves byte-for-byte-equivalent Results. Every error mapping is kept
identical:
- open failure -> to_file_error
- is_dir -> Error::FileNotFound (not to_file_error(EISDIR))
- metadata failure -> to_file_error
- xl.meta parse -> propagated verbatim from the parser
- try_reserve -> Error::other
- read_to_end -> to_file_error
For read_all_data_with_dmtime the async NotFound -> access(volume_dir) ->
VolumeNotFound fallback (and its warn! event) is preserved on the async side:
the closure returns the raw open error unmapped; only open() can yield ENOENT
once the fd is valid, so gating the fallback on the open error is equivalent to
the original open-arm-only fallback.
To run the parser inside a blocking closure, filemeta gains a synchronous twin
`read_xl_meta_no_data_sync` (+ `read_more_sync`) in
crates/filemeta/src/filemeta/version.rs that line-for-line mirrors the async
version, differing only in std vs tokio read_exact. A new equivalence test
(`read_xl_meta_sync_equivalence_tests`) feeds identical buffers to both the
async and sync readers and asserts equal Ok bytes / equal Err variants across:
v1.0; v1.1/v1.2/v1.3; large meta triggering read_more; header truncation ->
UnexpectedEof; CRC-trailer truncation -> FileCorrupt; unknown major/minor ->
InvalidData; and want boundaries (exact fit, inline-data drop, read_more EOF).
Co-authored-by: heihutu <heihutu@gmail.com>
Two disk-layer correctness fixes in crates/ecstore/src/disk/local.rs.
move_to_trash (#948, ECA-07) previously handled only DiskFull in its
tail error block and let every other rename failure (EIO/EACCES/ENOTDIR/
cross-device) fall through to `return Ok(())`, so a delete that never
landed on a faulty disk was reported as success and the drive fault was
hidden from heal/offline logic. It now keeps the DiskFull in-place-remove
fallback, treats a missing source (FileNotFound) as benign, and
propagates every other error (already mapped by to_file_error, e.g.
I/O -> FaultyDisk, permission -> FileAccessDenied), matching MinIO's
deleteFile.
rename_file and rename_part (#960, ECA-19) directory (trailing-slash)
branch unconditionally removed the destination before rename and
propagated the resulting NotFound, so renaming a directory to a new
location always failed. Both now tolerate ErrorKind::NotFound on that
pre-rename remove and continue, matching MinIO's RenameFile.
Adds regression tests covering benign-missing-source, real-error
propagation, the unchanged move happy path, and directory rename to a
missing destination for both rename_file and rename_part.
Co-authored-by: heihutu <heihutu@gmail.com>
An object transitioned to an unversioned remote tier legally records an
empty remote version_id (per CLAUDE.md: a tier version of `None`/`""`
means the tier bucket is unversioned, so remote GET/DELETE must omit the
versionId). Two recovery paths wrongly treated that empty sentinel as an
incomplete/unrecoverable record while the persist/encode and worker
paths accept it, producing permanent leaks in the crash-recovery window.
- tier_delete_journal::into_jentry rejected entries with an empty
version_id as "incomplete", so unversioned-tier WAL entries could never
be decoded during recovery: the remote object was orphaned and the
journal file leaked forever. Drop the version_id emptiness check; keep
the obj_name/tier_name checks. Truncated payloads are still rejected at
JSON deserialization (all fields are required, non-Option, no default,
with deny_unknown_fields).
- tier_free_version_recovery::is_recoverable_tier_free_version required a
non-empty version_id, silently filtering out free versions of
unversioned-tier objects so a first enqueue failure leaked the remote
object and local free version permanently. Drop the version_id check;
keep free_version + name + tier checks.
Both downstream paths already issue a versionless remote delete for an
empty version_id, so no further changes are needed. Adds regression
tests covering the empty-version_id case and the preserved guards.
Co-authored-by: heihutu <heihutu@gmail.com>
bitrot_shard_file_size only counts per-block checksum bytes for the two
streaming Highway variants, while BitrotWriter::write interleaves a hash
for any hash_algo.size() > 0 and bitrot_verify's read loop assumes an
interleaved hash per block. The three disagree for non-streaming
algorithms (SHA256/HighwayHash256/BLAKE2b512/Md5), but the divergence is
unreachable in production: every write path hardcodes HighwayHash256S and
ErasureInfo::get_checksum_info defaults to HighwayHash256S.
Per the audit decision (backlog#959), do NOT change the size formula:
it is a byte-for-byte port of MinIO's bitrotShardFileSize and its bare
return for non-streaming algorithms is correct for MinIO whole-file
bitrot; changing it would break legacy interop. Instead, document the
per-algorithm layout contract at bitrot_shard_file_size, BitrotWriter,
and bitrot_verify, and add regression tests that pin the invariants:
get_checksum_info defaults to HighwayHash256S, and the size formula
counts per-block hash bytes for streaming variants only while returning
the bare size for non-streaming ones. No disk layout or formula change.
Co-authored-by: heihutu <heihutu@gmail.com>
reduce_errs seeds best_err with a synthetic Error::other("nil") placeholder
via unwrap_or. When the error slice is empty or every error is ignored,
err_counts is empty and nil_count is 0, so both nil tie-break conditions are
false and the else branch returned (0, Some(Error::other("nil"))). That
placeholder leaked to build_write_quorum_failure_summary, where dominant_error
became Some("nil"), skipping the nil_dominated early return and falling back to
the "other_error" metric label. Quorum decisions were unaffected (max_count 0
still fails any positive quorum), but write-quorum failure metric labels and
tracing fields were misclassified.
Add an explicit short-circuit: when best_count == 0 and nil_count == 0, return
the (0, None) sentinel so the placeholder never escapes. Add regression tests
for the all-ignored slice, empty slice, and the summary label path, which the
existing test_build_write_quorum_failure_summary (nil_count > 0) did not cover.
Fixes#957
Co-authored-by: heihutu <heihutu@gmail.com>
Two dirty-mark lifecycle gaps (S19+S30):
- A commit cleared every dirty mark for the disks it scanned, even marks
recorded while the walker was already past the written prefix, so the
next dirty-subset refresh served stale cached bytes as exact until the
next full scan. Dirty marks now carry the instant they were last
recorded and a commit only clears marks that predate its scan start
(CapacityUpdate::scan_started_at, set by refresh_capacity_with_scope).
- The write side marks every disk of an EC set dirty, including remote
peers, while scans and clearing only cover local disks — remote or
removed disks stayed marked forever, keeping the dirty-disk gauge
permanently non-zero with bounded memory stuck behind it. The refresh
disk selection now prunes dirty entries outside the current local
topology before consulting them.
Ref: rustfs/backlog#1020 (S19+S30 from audit rustfs/backlog#1010)
complete_multipart_upload deleted every part.N.meta via
cleanup_multipart_path *before* the authoritative rename_data commit.
If rename_data then failed write quorum, the upload directory was left
with data files but no part metadata, so a retried CompleteMultipartUpload
could never read the parts again and the upload became permanently
uncompletable (client must abort and re-upload all parts).
Move cleanup_multipart_path to run only after rename_data returns Ok,
matching the existing "clean up only after commit" pattern already used
for the old data-dir GC and the upload-dir delete_all. On a failed commit
the staging part.N.meta now survive so the retry can complete.
Add a hermetic regression test that forces rename_data to fail on every
disk (destination bucket dir made read-only) and asserts the staging
part.N.meta are preserved for retry.
Fixes#946.
Co-authored-by: heihutu <heihutu@gmail.com>
fix(io-core): decrement available_buffers gauge on pooled-buffer reuse (backlog#806)
return_buffer does a fetch_add(1) on push but take_or_allocate_buffer's
available.pop() reuse path had no matching fetch_sub, so the available_buffers
gauge only ever grew and never reflected the real pool size. Decrement it on
reuse; + regression test.
load_all fetched policies/users/user-policies by passing the FULL remaining
list to load_*_concurrent every iteration and then split_off(32), so each item
was re-fetched once per preceding chunk — O(n^2) redundant loads on startup for
>32 items. Replace the split_off loops with chunks(32) so each item is fetched
exactly once. Behavior-preserving (cache inserts were already idempotent).
Erasure heal set write_quorum to the count of available target writers,
so every replacement disk had to succeed writing every block or the whole
object heal aborted. A single flapping replacement disk failing one block
made MultiWriter::write return Err, heal propagate Err, and the ops layer
delete the entire tmp staging dir — leaving the other healthy replacement
disks unhealed and blocking redundancy recovery indefinitely.
Set the heal write_quorum to 1, matching upstream MinIO's writeQuorum=1
for heal. MultiWriter already marks a failed writer as None, and the ops
layer (set_disk/ops/heal.rs) already drops the failed writer while
committing the survivors; the read-side quorum check and parity
cross-checks that guarantee reconstruction correctness are unchanged.
Add regression tests: one healthy-and-one-failing target still heals the
healthy targets and drops the failing one; all-targets-failing still
returns Err so nothing is falsely committed.
Fixes#947
Co-authored-by: heihutu <heihutu@gmail.com>
Two defensive gaps left the refresh singleflight vulnerable to a
permanent wedge (running=true forever: scheduled refreshes silently
stop, admin capacity joiners hang until restart):
- spawn_refresh_if_needed's spawned task had no leader guard, unlike
refresh_or_join: a panic after the refresh future completes (commit,
metrics) killed the task before the reset block. The task now holds
the same RAII RefreshLeaderGuard, disarmed only after the state is
reset and the result published (S20).
- refresh_fn() was evaluated before AssertUnwindSafe wrapping in both
paths, so a panic while constructing the future escaped catch_unwind;
construction now happens inside the wrapped future.
- RefreshLeaderGuard::drop silently skipped the reset when try_lock was
contended and no tokio runtime was current; it now falls back to a
blocking reset, which is safe precisely because there is no executor
to stall in that context (S31).
Ref: rustfs/backlog#1021 (S20+S31 from audit rustfs/backlog#1010)
Close the ~10ms instrumentation residual on the PUT success path that the
existing coarse writer_setup/encode/rename stage metrics do not attribute.
Adds `hp_guard!` measurement scopes to the previously uninstrumented
sub-stages called out in backlog#935 item 2:
- SetDisks::acquire_read_lock / acquire_write_lock (namespace lock acquire)
- S3::put_object_prelookup (pre-write get_object_info lookup)
- MultiWriter::shutdown (bitrot writer flush/close)
- SetDisks::commit_rename_data_dir (old data-dir reclaim)
- S3Access::put_object (S3 authorization segment)
Instrumentation only: `hp_guard!` expands to nothing without the `hotpath`
feature, so this is a pure-observation change with zero behavior impact and
zero cost in default builds. The pre-lookup site wraps only the lookup call
in a scoped block so the guard measures that slice exactly while preserving
the existing match and control flow.
Verified: cargo check -p rustfs-ecstore (default and --features hotpath) and
cargo check -p rustfs (default and --features hotpath) all pass.
Co-authored-by: heihutu <heihutu@gmail.com>
fix(ecstore): route rebalance delete markers to store and count expiries
Rebalance migrated delete markers via SetDisks::delete_object (single set,
no cross-pool routing), which silently rewrote the marker back onto the
source set. The subsequent source cleanup then deleted the whole entry,
losing the delete marker and reviving logically-deleted objects (#942, P0).
Route delete-marker migration through ECStore::delete_object via a store-
level closure that mirrors the existing transfer closure, so the marker
lands on the cross-pool target honouring data_movement/src_pool_idx/
delete_marker; on failure it is not counted as moved, so the source entry
is not cleaned up. The unused MigrationBackend::delete_object_for_migration
footgun is removed.
Rebalance source cleanup also ignored lifecycle-expired versions: the gate
used rebalanced == total_versions and passed an empty allowed_missing, so
any entry with an expired version could never be cleaned up, leaking the
migrated versions in the source pool (#950). Mirror decommission: gate on
rebalanced + expired == total_versions and feed expired version identities
into the cleanup preflight allowed_missing, plus a source_retained warning
for parity with decommission diagnostics.
Adds regression tests for delete-marker store routing and the expiry-aware
cleanup gate.
Co-authored-by: heihutu <heihutu@gmail.com>
Non-slash delimiter ListObjects/ListObjectVersions force a recursive walk and
defer common-prefix folding to from_meta_cache_entries_sorted_infos. When a page
of max_keys+1 raw candidates collapses into fewer than max_keys common prefixes,
list_objects_paginate left is_truncated=false with no next_marker even though the
walker had filled its raw candidate limit (disk_has_more=true). Clients were told
listing was complete and silently lost every key beyond the scan window, a
data-loss bug for backup/sync/mirror/reconcile consumers (backlog ECA-03 / #944).
Add a delimiter re-fold guard: when disk_has_more is true but the folded visible
entries are fewer than max_keys, set is_truncated=true and continue from the last
RAW scanned key (captured before folding via last_scanned_entry_name). Using the
last raw key rather than a folded prefix is required for correctness: forward_past
advances on name > marker, and a folded prefix such as "data-" is lexicographically
smaller than its own keys ("data-0001"), so it would re-list and re-fold the same
keys forever. A real scanned key strictly advances past the scanned window and
keeps pagination finite. The slash-delimiter and no-delimiter paths fold during
the walk and are unaffected.
Regression tests cover the direct re-fold case, the disk-exhausted no-marker case,
the no-delimiter path guard, and an end-to-end 5000 data-* + 100 other-* paging
loop asserting bounded termination and full common-prefix coverage.
Co-authored-by: heihutu <heihutu@gmail.com>
is_all_not_found and is_all_volume_not_found delegated to
is_err_bucket_not_found, which matches StorageError::DiskNotFound. When
every erasure set in every pool was unreachable, the per-set aggregate
DiskNotFound slice was classified as "all not found", so list_merged
returned Ok(empty) and walk_result_from_set_errors was fed an
availability failure disguised as an empty listing. Clients saw a
successful empty ListObjects and mistook a full outage for an empty
bucket.
Introduce is_err_strict_not_found (FileNotFound / VolumeNotFound /
FileVersionNotFound / ObjectNotFound / VersionNotFound, excluding
DiskNotFound) for is_all_not_found, mirroring MinIO's isAllNotFound and
DiskError::is_all_not_found. Give is_all_volume_not_found its own
is_err_strict_volume_not_found (VolumeNotFound / BucketNotFound, minus
DiskNotFound) so genuine volume/bucket absence still surfaces while a
missing object under an existing volume is not escalated. Add
is_all_disk_not_found and a pre-check in walk_result_from_set_errors so
an all-offline slice surfaces DiskNotFound instead of being swallowed as
an empty listing, while partial outages (a healthy set present) stay
tolerated as before.
Regression tests cover all-DiskNotFound rejection, genuine not-found
acceptance, volume-not-found semantics, partial/empty short-circuit, and
the walk full-offline vs partial-offline paths.
Co-authored-by: heihutu <heihutu@gmail.com>
Two correctness defects on the opt-in codec-streaming GET path.
ECA-02 (#943): ErasureDecodeReader only decremented `remaining` for the
main fill buffer. Under the default DualInFlight policy each fill also
produces a queued stripe that is delivered to the client via
`prefetched_bufs.pop_front()` without touching `remaining`, so any object
larger than one erasure block finished with `remaining > 0` and the GET
terminated with LessData despite delivering all bytes. The inflated
`remaining` was also fed back into the fill worker, which used it to trim
the final stripe and to decide whether to read past EOF. Account for the
queued-stripe bytes when they enter the prefetch queue; queued buffers
come only from `Ok(true)` decodes so they are non-empty and bounded by
`remaining - main_buf.len()`, ruling out underflow.
ECA-04 (#945): the codec-streaming gate did not inspect `opts.part_number`.
A partNumber GET carries `range == None`, so it was not classified as a
Range request and reached the full-object codec-streaming reader, which
drops the storage offset/length returned by GetObjectReader::new. A
partNumber >= 2 request would then stream the whole object. Mirror the
direct-memory part_number fallback and route any partNumber request back
to the legacy duplex path, which applies the offset/length correctly.
Regression tests: DualInFlight read_to_end on a multi-block object and on
a non-block-aligned object; SingleInFlight vs DualInFlight byte-identical
output; gate fallback on partNumber requests.
Co-authored-by: heihutu <heihutu@gmail.com>
The whole disk traversal runs inside spawn_blocking with no timeout
anywhere on the async side; the in-scan ProgressMonitor checks are
cooperative and only run between walker entries. A stat/readdir blocked
on a dying disk or hung NFS mount therefore never returns: the blocking
thread leaks, the refresh singleflight stays running forever, every
subsequent scheduled refresh is skipped as inflight, and every admin
capacity query joins an unbounded wait until process restart (S02).
- Wrap each disk scan in tokio::time::timeout with a hard wall-clock
ceiling of 2x the cooperative budget (min 5s). On expiry the caller
fails the disk scan (releasing the singleflight through the normal
error path, where the degraded/partial machinery from backlog#1014
keeps the failed disk's last-known value) and a shared AtomicBool asks
the blocking walker to exit at its next entry, bounding the thread
leak to the single wedged syscall.
- Bound refresh_or_join joiner waits at 5 minutes so admin queries
degrade into a clear error instead of hanging if the leader wedges in
a way the drop/panic guards don't cover.
Ref: rustfs/backlog#1017 (S02 from audit rustfs/backlog#1010)
refactor(concurrency): remove zero-caller facade modules and fix no-default-features build (backlog#1025)
The audit in rustfs/backlog#1010 (consistent with #805) established that most of crates/concurrency was a decorative facade with zero production callers; the real runtime concurrency control lives in rustfs/src/storage/*. This deletes the dead facades and keeps only what the workspace actually consumes.
Deleted (zero callers verified by workspace-wide grep):
- manager.rs: ConcurrencyManager, lifecycle start/stop, misleading 'started' lifecycle logs
- config.rs: ConcurrencyConfig, ConcurrencyFeatures, from_env
- timeout.rs: TimeoutManager, TimeoutGuard, TimeoutManagerPolicy
- lock.rs: LockManager, LockScopeGuard, OptimizedLockGuard
- scheduler.rs: SchedulerManager, SchedulerPolicy, IoStrategy
- deadlock.rs facade: DeadlockManager, RequestTracker
- backpressure.rs facade: BackpressureManager, BackpressurePipe
- the prelude module, unused io-core re-exports, and all feature flags
Kept (real callers in ecstore/heal/rustfs):
- workload.rs admission contract types (unchanged)
- workers.rs Workers pool (unchanged, retained per #4498)
- GetObjectQueueSnapshot (moved from manager.rs to new queue.rs)
- PipeBackpressurePolicy (used by rustfs/src/storage/backpressure.rs)
- DeadlockMonitorPolicy (used by rustfs/src/storage/deadlock_detector.rs)
- OperationProgress re-export (used by rustfs/src/storage/timeout_wrapper.rs)
Removing the feature flags fixes the previously broken cargo check -p rustfs-concurrency --no-default-features (E0432). Docs and the logging guardrail file list are updated to match.
Ref: rustfs/backlog#1025
B5 switched heal enumeration to list_object_versions, which only reflects the
read-quorum metadata view: a version present on fewer than read-quorum disks was
never enumerated, so it was never healed. Add a per-erasure-set disk-walk UNION
enumerator (mirrors MinIO global-heal.go objQuorum=1 listPathRaw +
mergeXLV2Versions) that surfaces every (object, version) present on ANY disk and
feeds each to the existing per-version heal_object.
- filemeta: MetaCacheEntries::resolve_union (dir_quorum=1/obj_quorum=1) yields the
cross-disk version union at one tested seam.
- ecstore: SetDisks::heal_walk_versions_page (list_path_raw fan-out, min_disks=1,
dual object/version page bound, inclusive-forward de-overlap) + ECStore delegator
+ HealWalkVersion.
- ecstore data-safety guard: before dangling-delete, try_regenerate_recoverable_meta
physically probes part files via check_parts; when >= data_blocks data shards
survive (meta lost but data recoverable) it regenerates xl.meta from a surviving
FileInfo with the correct per-disk shard index instead of dangling-deleting.
Genuine torn writes (< data_blocks) keep the current behavior — no resurrection.
- heal: dw1: forward-marker cursor codec (reuses ResumeState.resume_cursor,
idempotent restart on foreign tokens); list_versions_for_heal_page_disk_walk
trait method (default falls back to the B5 read-quorum path); heal_bucket_with_resume
selects the disk-walk enumerator when scan_mode==Deep || source==AutoHeal, else
the unchanged B5 path; anti-loop guard aborts on (empty && truncated).
Closesrustfs/backlog#920
Zhengchao An