`MultiWriter` recorded per-writer failures only in a private `errs` vector and
nulled the failed writer locally, but `put_object` never used that to prune the
disk set: `rename_data` ran over the full `shuffle_disks`, so a disk that took a
short/failed write still had its truncated shard renamed into place and counted
as an online disk. The object then claimed N good shards while one was
short/corrupt, so a single later disk failure could drop it below reconstructable
quorum — silent data loss.
- `write_shard`: null the writer on a generic write error too (not only on
`ShortWrite`), so a failed writer is uniformly represented as `None` — matching
`shutdown_writer`, which already does this.
- `put_object`: after encode, drop every disk whose writer failed
(`drop_failed_writer_disks`) from `shuffle_disks` before `rename_data`, and
re-check write quorum over the survivors. `rename_data` already re-checks
quorum and rolls back if too few disks remain, so excluded disks are neither
committed nor counted (MinIO sets failed writers to nil before `renameData`).
Adds unit tests for the exclusion/quorum accounting.
Refs backlog#799 (B3).
PR #4220 added a purge for orphan empty-directory trees (folder keys with
no xl.meta) on the delete path, but the guard only accepted
object-not-found. Over the real HTTP DELETE path the guard is never
reached: `del_opts` pins `version_id = Uuid::nil()` for directory keys, so
the missing dir object fails the specific-version lookup with
version-not-found (FileVersionNotFound), not object-not-found. The guard
short-circuits, the store returns the error, and the API layer turns it
into a fake 204 — the ghost folder survives, exactly the #4189 symptom.
The existing unit tests passed because they call
`purge_orphan_dir_object` directly, bypassing the nil-version lookup.
Accept both misses in the guard (extracted as
`should_purge_orphan_dir_on_missing`) so the folder delete actually takes
effect. Non-directory keys and non-miss errors (e.g. quorum failures) are
unaffected; the cross-disk data-safety refusal in
`purge_orphan_dir_object` is unchanged.
Verified end-to-end against a running 4-disk erasure server: DELETE of a
planted orphan `ghost/` tree now purges it on all drives and clears the
listing, a real object under the same prefix is untouched, and a prefix
holding real data on any drive is refused (all drives preserved).
Adds predicate regression tests covering version-not-found /
object-not-found on directory keys, and the negative cases.
P6 of the SetDisks God-Object split (tracking backlog#815, issue backlog#821;
depends on P5 #4288). Relocate the core object read/write hot-path contract
impls out of the set_disk/mod.rs God-Object into their own module home:
- impl ObjectIO for SetDisks (~1,010 lines) -> set_disk/ops/object.rs
- impl ObjectOperations for SetDisks (~1,137 lines) -> set_disk/ops/object.rs
- Registered pub(crate) mod object; under set_disk/ops.
Pure move — zero logic change. Both contracts stay implemented for SetDisks, so
their EcstoreObjectIO / EcstoreObjectOperations associated-type bounds are
unchanged and the contract-compat tests still guard them. Method bodies are
moved verbatim: a whitespace-insensitive token-stream diff of the two impl
blocks (with their #[async_trait::async_trait] attributes) against the pre-move
mod.rs source is byte-identical (5,754 tokens each) — NO visibility widening was
required, because the impls reach SetDisks helpers and the P5 io_primitives
through inherent self./Self:: calls that resolve across modules unchanged. The
two inherent impl SetDisks blocks that sat between the trait impls (lock batch
helpers) remain in mod.rs, verified present exactly once and uncorrupted.
The issue's borrow/Arc-clone-avoidance optimization is intentionally deferred:
it is a perf-sensitive change requiring the #738 benchmark and would risk the
'byte-level behavior unchanged' acceptance; this PR delivers the relocation.
Verification:
- cargo check / clippy -D warnings -p rustfs-ecstore --all-targets: clean
- cargo test -p rustfs-ecstore --lib: 1841 passed, 0 failed
- all five arch guard scripts: pass
- token-stream diff of moved impls vs original: identical (mod.rs diff is a pure
relocation; ObjectIO/ObjectOperations each defined exactly once post-move)
fix(ecstore): lockstep stripe read to stop EC GET desync truncation
On the reconstruction-verifying GET path, ParallelReader::read used a
data-first schedule: it read only `data_shards` readers per stripe and
pulled in a parity reader as a substitute on demand. Because the shard
readers are streaming (advanced only by being read, no seek), a parity
reader first used mid-object was still positioned at its stream start
(block 0) and returned an earlier stripe than the surviving data shards.
Every shard passed its own bitrot hash, yet the set was mutually
misaligned, so decode_data_with_reconstruction_verification correctly
rejected it with "inconsistent read source shards" and the large-object
GET truncated mid-stream (client "unexpected EOF").
Add read_lockstep(): on the verify_reconstruction path, read every live
shard reader once per stripe and wait for all of them, so all readers
advance one block per stripe and stay mutually aligned; any reader that
errors is retired for the rest of the object (a stream that failed
mid-block can no longer be trusted to be aligned). The adaptive
data-first path is unchanged for non-verifying callers (e.g. heal).
Adds a regression test reproducing a data shard that dies partway through
a multi-stripe object; it must still reconstruct byte-exact output.
Refs backlog#832.
* refactor(ecstore): sink write/rename/delete primitives into set_disk::core::io_primitives (backlog#820)
P5 step 2 of the SetDisks God-Object split (tracking backlog#815, issue
backlog#820; follows step 1 #4285). Relocate the entire set_disk/write.rs
primitive family into the core/io_primitives.rs module home established by
step 1:
- Module items: dangling_delete_grace() + its env consts, and the
OrphanDirScan scan-result enum.
- impl SetDisks write/rename/delete primitives shared across mod.rs and the
ops/ operation families: rename_data, commit_rename_data_dir,
cleanup_multipart_path, rename_part, eval_disks, write_unique_file_info,
update_object_meta(_with_opts), delete_if_dangling, delete_prefix,
scan_orphan_dir, purge_orphan_dir_object, check_write_precondition,
default_read_quorum, default_write_quorum.
- The dangling_delete_grace unit tests move with their subject.
set_disk/write.rs is removed and 'mod write;' dropped from mod.rs.
Pure move + visibility adjustment, zero logic change. Method bodies are moved
verbatim; pub(super) items are widened to pub(in crate::set_disk) so mod.rs /
ops still reach them (write.rs's super was set_disk; the deeper module needs the
explicit path to preserve identical reach). Callers use inherent self./Self::
calls, so no call sites change.
Verification:
- cargo check / clippy -D warnings -p rustfs-ecstore --all-targets: clean
- cargo test -p rustfs-ecstore --lib: 1841 passed, 0 failed (moved
dangling_delete_grace tests run as set_disk::core::io_primitives::tests::*)
- all five arch guard scripts: pass
- token-stream diff of moved block vs original write.rs: identical modulo
visibility tokens (2713 tokens each)
* refactor(ecstore): sink shared read primitives into set_disk::core::io_primitives (backlog#820)
P5 step 3 (final) of the SetDisks God-Object split (tracking backlog#815, issue
backlog#820; follows steps 1 #4285 and 2). Relocate the shared, low-level
metadata/erasure READ PRIMITIVE methods out of set_disk/read.rs into the
core/io_primitives.rs module home, leaving the object-read operation itself in
read.rs.
Moved into a new impl SetDisks block in io_primitives.rs (verbatim bodies):
- read_parts, read_all_fileinfo (+ _observed / _inner / _full_wait /
_early_stop variants), read_all_xl, load_file_info_versions_exact,
read_all_raw_file_info, pick_latest_quorum_files_info, read_multiple_files.
- The should_allow_metadata_early_stop free helper (called by the moved
read_all_fileinfo_observed).
Kept in read.rs: the object-read operation and its private helpers
(read_version_optimized, get_object_fileinfo, get_object_info_and_quorum,
try_get_object_direct_data_shards_with_fileinfo, get_object_with_fileinfo,
get_object_decode_reader_with_fileinfo, build_codec_streaming_part_reader) and
the metadata-cache helpers/tests. read.rs reaches the moved primitives through
the SetDisks core (inherent self./Self:: calls; the sole cross-boundary edge,
get_object_fileinfo -> read_all_fileinfo_observed, is handled by widening that
one method to pub(in crate::set_disk)).
Pure move + visibility adjustment, zero logic change. pub(super) primitives are
widened to pub(in crate::set_disk); the three internal-only fanout variants
(_inner/_full_wait/_early_stop) stay private; load_file_info_versions_exact
stays pub(crate). Method bodies are moved verbatim.
Verification:
- cargo check / clippy -D warnings -p rustfs-ecstore --all-targets: clean
- cargo test -p rustfs-ecstore --lib: 1841 passed, 0 failed
- all five arch guard scripts: pass
- token-stream diff of the three moved regions vs original read.rs: identical
modulo visibility tokens, the new impl-block scaffolding, and rustfmt
signature re-wrapping (read.rs diff is pure deletion, zero added lines)
* refactor(ecstore): extract read IO primitives to set_disk::core::io_primitives (backlog#820)
P5 step 1 of the SetDisks God-Object split (tracking backlog#815, issue
backlog#820). Relocate the module-level low-level read/erasure primitives out of
the 5,898-line set_disk/read.rs into a new core/io_primitives.rs module home:
- Metadata-fanout quorum machinery (MetadataQuorumAccumulator,
MetadataFanoutDiagnostics/Observation, MetadataEarlyStopDecision,
MetadataCacheLookup).
- Bitrot reader scheduling/creation (BitrotReaderSetup and the
create_bitrot_readers_* / schedule / fill helpers).
- Shard-cost classification, read-repair heal dedup, and codec-streaming
reader helpers.
Pure move + visibility adjustment, zero logic change. The moved block is
byte-identical to the pre-move read.rs source modulo the module header swap
(use super::* -> use super::super::*) and item visibility widening
(module-private / pub(super) -> pub(in crate::set_disk) so read.rs still
reaches them). read.rs's impl SetDisks body and imports are byte-identical to
before; mod.rs only gains 'mod core;' and repoints two
GetCodecStreamingReaderBuildOutcome references to the new path.
Verification:
- cargo check / clippy -D warnings -p rustfs-ecstore --all-targets: clean
- cargo test -p rustfs-ecstore --lib: 1840 passed, 0 failed
- normalized + token-stream diff of moved block vs original: identical modulo
visibility tokens and rustfmt signature re-wrapping
* fix(arch): allow io_primitives as READ_REPAIR_HEAL_CACHE owner (backlog#820)
The READ_REPAIR_HEAL_CACHE owner-path guard in
check_architecture_migration_rules.sh pinned the cache to
set_disk/read.rs. P5 step 1 relocated the cache (and its accessor
helpers) to set_disk/core/io_primitives.rs, so allow that path too.
Guard intent is unchanged: the cache stays behind the ECStore set-disk
read-owner helpers.
---------
Co-authored-by: houseme <housemecn@gmail.com>
P1 of the SetDisks split (tracking backlog#815, depends on P0 backlog#816).
Give the Heal operation family its own module home: relocate set_disk/heal.rs
to set_disk/ops/heal.rs and move the HealOperations storage-api contract impl
beside its inherent helpers. The contract stays implemented for SetDisks so its
associated-type bounds are unchanged (ecstore_contract_compat_test still
covers it).
Method bodies are moved unchanged. The four inherent helpers widen from
pub(super) to pub(in crate::set_disk) to preserve their exact prior visibility
from the deeper module. get_pool_and_set now reads topology through
SetDisksCtx to keep the Heal family aligned with the P0 borrow pattern; the
read is provably identical (ctx.format()/pool_index() alias the core fields).
Runtime behavior is unchanged.
P0 of the SetDisks split (tracking backlog#815). Introduce SetDisksCtx<'a>,
a Copy borrow handle over the shared SetDisks core state (topology/config,
disks, locker trio), and validate the pattern by routing the List family's
delete_all through a ListOperations<'a> service unit.
No trait impl is moved and runtime behavior is byte-for-byte unchanged; the
public SetDisks::delete_all entry point is preserved and delegates to the
borrow-based unit.
* fix(rio): reject corrupted short compressed/encrypted blocks instead of panicking
DecompressReader::poll_read and DecryptReader::poll_read sliced the block
body with a fixed `[0..16]` index to read the length varint. The body length
comes from an untrusted 24-bit header field, so a corrupted/truncated block
shorter than 16 bytes made the slice panic and crash the request task — a
read-path DoS on GET of tiered/corrupted data.
Pass the whole (arbitrary-length-safe) slice to uvarint and reject a
non-positive or out-of-range length prefix with InvalidData. Adds a repro
test for each reader; all existing round-trip tests still pass.
Refs rustfs/backlog#812
* fix(utils): close SSRF bypass via IPv4-mapped IPv6 addresses
validate_outbound_ip branched on the IpAddr variant, and the V6 branch's
is_loopback/is_unicast_link_local/is_unique_local checks never inspect the
embedded IPv4 of an IPv4-mapped address (::ffff:a.b.c.d). The metadata guard
also only matched the plain V4 169.254.169.254. So ::ffff:127.0.0.1,
::ffff:10.0.0.5 and ::ffff:169.254.169.254 all passed the outbound guard,
letting an attacker reach loopback/private/metadata endpoints.
Normalize IPv4-mapped IPv6 to its embedded IPv4 (via to_ipv4_mapped, which
matches only the true mapped form) before classification. Adds reject tests
for mapped loopback/private/metadata and an allow test for public IPv6.
Refs rustfs/backlog#813
* fix(ecstore): streaming last-part loss, GCS tier Range/remove, stat_all_dirs alignment
Four confirmed data-reliability defects:
- put_object_multipart_stream: the CompleteMultipartUpload part-collection loop
used exclusive `1..total_parts_count`, dropping the final part (and collecting
zero parts for a single-part object) — silently truncating the completed object.
Extracted collect_complete_parts (1..=total_parts_count) with unit tests.
- GCS warm backend get() ignored the requested byte range, returning the whole
object for a Range GET; now applies ReadRange::segment like the other backends.
- GCS warm backend remove() was an empty stub, so deleting a tiered object left
it on GCS forever; now deletes via StorageControl (added a control-plane client),
and in_use() actually lists (prefix-scoped) instead of always returning false.
- stat_all_dirs skipped None disk slots and dropped JoinErrors, returning a
compressed, misaligned error vector; heal_object_dir then zipped it against the
full disks array and could make_volume on the WRONG disk. Now returns one
index-aligned entry per slot (None -> DiskNotFound), and heal no longer
pre-fills the drive report (which would double it). Added an alignment test.
Refs rustfs/backlog#807
* fix(kms): stop Vault backend from destroying/reviving keys on failure
Two confirmed key-safety defects in the Vault KV2 backend:
- get_key_material() 'self-healed' a decrypt or wrong-length failure by minting a
fresh random master key and overwriting the stored value. That destroys the
original key material, making every DEK ever wrapped by it permanently
undecryptable. Decryption must never mutate the stored key: both branches now
return a cryptographic_error instead. (The empty-material bootstrap path, which
only fills a never-initialized key, is intentionally left intact.)
- cancel_key_deletion() reset key_state to Enabled only in the returned response
and never persisted it, so the key stayed PendingDeletion in storage and would
still be reaped. It now writes the state back via update_key_metadata_in_storage
and fails the request if the write fails.
Adds ignored (Vault-requiring) integration tests documenting both behaviours.
The third item (VaultTransit key state only in memory -> revived as Enabled after
restart) is deferred: a fail-closed guard would break restart availability for all
transit keys; the correct fix needs a persistent metadata store + Vault integration
testing. Tracked in rustfs/backlog#808.
Refs rustfs/backlog#808
* fix(admin): clamp STS AssumeRole duration; persist ImportBucketMetadata to disk
Two confirmed admin-API defects:
- Standard AssumeRole used the raw client-supplied DurationSeconds with no upper
bound, so a caller could mint near-permanent temporary credentials. Clamp it to
the AWS/MinIO STS window [900, 43200] (with 0 -> default 3600) via a shared
clamp_assume_role_duration helper, and build the exp claim with saturating_add.
This matches the existing AssumeRoleWithWebIdentity path.
- ImportBucketMetadata only mutated an in-memory map and returned 200, silently
dropping every imported config. It now persists each non-empty config via
metadata_sys::update (which merges onto existing on-disk metadata) and returns
InternalError if a write fails. Mapping extracted to imported_configs_to_persist
with unit tests.
Refs rustfs/backlog#809
* fix(heal): enqueue displacing request in release builds
push_displacing_lower_priority folded the real enqueue call into
debug_assert_eq!(self.push(request), Accepted). In release builds
(debug_assertions off) the whole macro — including its argument — is compiled
out, so after evicting a lower-priority queued item the new high-priority
request was silently dropped and never healed. Hoist self.push(request) out of
the assertion so the side effect runs in all builds. Adds a --release regression
test.
Refs rustfs/backlog#811
* fix(iam): propagate real delete_policy backend errors instead of swallowing them
delete_policy's is_from_notify path had its error handling inverted: a real
backend failure (disk IO / insufficient quorum) evicted the cache and returned
Ok(()), reporting a phantom success while policy.json survived on disk (to be
reloaded on the next full IAM reload); NoSuchPolicy — which should be idempotent
success — returned Err. Propagate real errors and let NoSuchPolicy fall through
to the idempotent cache-evict + Ok, matching delete_user / the notification
handler in the same file. Adds a backend-error-injection regression test.
Refs rustfs/backlog#810
* fix(utils): also normalize IPv4-compatible IPv6 in the SSRF guard
The initial fix only unwrapped IPv4-mapped (::ffff:a.b.c.d) addresses; the
deprecated IPv4-compatible form (::a.b.c.d, e.g. ::127.0.0.1 / ::169.254.169.254)
still bypassed the guard. Reject pure-IPv6 specials (::, ::1, fe80::, fc00::)
first, then normalize BOTH embedded-IPv4 forms before the IPv4 rules. Adds tests
for compatible-form loopback/metadata and confirms ::1 / :: stay rejected.
Found by adversarial review of the initial fix. Refs rustfs/backlog#813
* fix(ecstore): fix the same last-part loss in the parallel streaming path
put_object_multipart_stream_parallel had the identical off-by-one
(1..total_parts_count) that truncated the last part / produced zero parts for a
single-part upload — reachable when concurrent stream parts are enabled. Reuse
collect_complete_parts, which now returns an error instead of panicking on a gap
in the parts map. Adds a missing-part error test.
Found by adversarial review of the initial fix. Refs rustfs/backlog#807
* fix(kms): local backend must preserve key material on status change
LocalKmsClient (the default KMS backend) regenerated the master key material on
enable_key/disable_key/schedule_key_deletion/cancel_key_deletion — a pure status
change. A single disable+enable cycle therefore destroyed the original key,
making every DEK ever wrapped by it permanently undecryptable (silent data loss,
no network needed). Preserve the existing material via get_key_material and
re-save with only the status changed. Adds a hermetic regression test that wraps
a DEK, cycles all four status methods, and asserts the DEK still decrypts.
Found by adversarial review of the Vault fix. Refs rustfs/backlog#808
* test(rio): cover the length-prefix guard; correct its comment
Add a DecompressReader test that feeds an unterminated length varint so uvarint
returns 0 and the new guard (not the downstream codec) produces the InvalidData
error, and reword the guard comment which overclaimed that the > len bound
prevents a reachable panic (it is belt-and-suspenders). No behavior change.
Found by adversarial review. Refs rustfs/backlog#812
* test(rio): build test block headers via vec! to satisfy clippy
The new corrupted-block tests built the header with Vec::new() + repeated push,
tripping clippy::vec_init_then_push (-D warnings in CI). Construct the fixed
header bytes with vec![] instead. No behavior change.
---------
Co-authored-by: houseme <housemecn@gmail.com>
perf(get,heal): land verified fixes for backlog #800-#804
Five fixes from the GET-path performance audit and scanner/heal
completeness audit (rustfs/backlog#800..#804), each verified locally:
- backlog#800 (heal checkpoint O(N^2)): ResumeCheckpoint object sets are
now HashSet (Vec::contains was O(n) per healed object; 1.5ms at n=1M
vs 11ns measured), and per-object checkpoint/resume-state persistence
is batched (1000 mutations / 5s) instead of rewriting the whole file
per object. complete_page() prunes the sets at page boundaries so
memory stays bounded; positions still persist unconditionally and
legacy Vec-format checkpoints still deserialize.
- backlog#801 (DiskInfo.healing never set): erasure-set heal now writes
a healing marker (.rustfs.sys/healing.bin) on the disks it rebuilds
(endpoints plumbed via HealRequest/HealTask.heal_endpoints from the
auto disk scanner) and clears it on success. LocalDisk::disk_info
surfaces the marker, so scanner heal coordination, lock selection and
admin/metrics healing counts see the rebuild.
- backlog#802 (cache probe after data read): new GetObjectBodyCacheHook
in ecstore lets the app-layer object data cache serve the body inside
get_object_reader, after metadata quorum resolution (etag known) but
before the erasure shard read/decode. Previously a hit still paid the
full disk read. Hook is None/no-op when the cache is disabled.
- backlog#803 (GET hot-path redundant work): ObjectInfo is cloned for
event notification only when an event will actually be built (GET and
HEAD paths; events are currently suppressed so the clone was pure
waste); get_opts/put_opts/del_opts resolve bucket versioning with one
metadata-sys lookup instead of two; skip_verify_bitrot and
get_lock_acquire_timeout env reads are cached via OnceLock; the
io-priority metric is no longer double-counted; GetObject input fields
are cloned selectively instead of cloning the whole input.
- backlog#804 (disk permit starvation): the disk-read permit wait is now
bounded (RUSTFS_OBJECT_DISK_PERMIT_WAIT_TIMEOUT, default 5s, 0 =
previous unbounded behavior); on timeout the GET proceeds without a
permit and the bypass is counted. DiskReadPermitReader also releases
the permit at body EOF instead of holding it until the client drops
the stream.
Verification: make pre-commit; cargo clippy -D warnings on the four
changed crates; full rustfs lib suite (2096 tests) green; rustfs-heal
lib suite green with new unit tests for checkpoint pruning/legacy
format/throttle, permit EOF release, and the cache hook (hit + SSE
skip). The heal_integration_test and one set_disk listing test fail
identically on unmodified main (pre-existing global-state ordering
flakes, verified via git stash A/B).
Co-authored-by: houseme <housemecn@gmail.com>
* fix(rio): propagate http writer shutdown errors
* fix(ecstore): unify remote lock rpc deadlines
* fix(storage): reject corrupt read multiple payloads
* feat(rio): add internode http tuning profiles
* feat(metrics): add internode baseline signals
* feat(ecstore): observe shard locality topology
* feat(ecstore): gate shard locality scheduling
* feat(ecstore): gate batch read version rpc
* feat(ecstore): observe batch processor adaptation
* feat(ecstore): gate batch processor observation
* docs: add get benchmark regression analysis
* docs: add issue 797 execution plan status
* fix(ecstore): require explicit batch rpc support
* fix(ecstore): honor documented batch read gate
* fix(ecstore): keep batch read gate stable per call
* chore: update workspace dependencies
* feat(ecstore): log batch read gate decisions
* feat(ecstore): count batch read gate decisions
* test(issue-797): add local internode A/B runner
* test(rio): fix tuning profile spelling fixture
* fix(protocols): adapt sftp channel open callbacks
* fix(metrics): wrap batch processor observation args
* chore(docs): keep issue notes local only
* fix(storage): address internode review feedback
* fix(storage): address internode data-path review findings
- Run the BatchReadVersion auto-mode unary fallback outside the batch
RPC deadline so each read_version keeps its own per-op timeout and
health accounting instead of racing the whole batch against one
drive timeout.
- Cap adaptive batch-processor concurrency growth at a hard multiple
of the configured baseline so sustained fast batches cannot ratchet
past the configured limit.
- Parse RUSTFS_INTERNODE_HTTP_* tuning, RUSTFS_BATCH_PROCESSOR_ADAPTIVE,
and RUSTFS_METADATA_BATCH_READ once per process instead of re-reading
the environment on hot paths.
- Skip shard read-cost collection in observe mode when stage metrics
are disabled, and cache the local endpoint host list instead of
rebuilding it on every read.
- Allow --warp-extra-args values starting with -- and drop the unused
warp_hosts_csv helper in the issue-797 A/B runner.
* fix(storage): address internode data-path review findings
- Run the BatchReadVersion auto-mode unary fallback outside the batch
RPC deadline so each read_version keeps its own per-op timeout and
health accounting instead of racing the whole batch against one
drive timeout.
- Cap adaptive batch-processor concurrency growth at a hard multiple
of the configured baseline so sustained fast batches cannot ratchet
past the configured limit.
- Parse RUSTFS_INTERNODE_HTTP_* tuning, RUSTFS_BATCH_PROCESSOR_ADAPTIVE,
and RUSTFS_METADATA_BATCH_READ once per process instead of re-reading
the environment on hot paths.
- Skip shard read-cost collection in observe mode when stage metrics
are disabled, and cache the local endpoint host list instead of
rebuilding it on every read.
- Allow --warp-extra-args values starting with -- and drop the unused
warp_hosts_csv helper in the issue-797 A/B runner.
Co-Authored-By: heihutu<heihutu@gmail.com>
* fix(storage): align buffer clamp test with media cap
* fix(ecstore): release optimized read locks before streaming
---------
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
* fix(core-storage): fix critical correctness defects from core-storage audit
Fixes verified defects found in a deep audit of the core storage path
(erasure coding, disk persistence, quorum, heal, replication resync):
- ecstore/disk: rewrite live xl.meta atomically (temp+rename) in
delete_versions_internal and write_metadata instead of in-place
truncate, which exposed torn metadata to concurrent readers and
crashes on the DeleteObjects hot path
- ecstore/erasure: allow heal to reconstruct from exactly data_shards
bitrot-verified sources; requiring data_shards+1 made objects
permanently unhealable after losing parity_shards disks
- ecstore/set_disk: direct-memory inline GET applied the erasure
distribution permutation twice (shuffled inputs re-indexed through
distribution), concatenating wrong shards into the response body in
degraded reads; collect from canonical disk-ordered inputs
- ecstore/set_disk: heal now preserves the committed inline layout
instead of recomputing it with a hardcoded unversioned threshold,
which split quorum identity of healed replicas and caused endless
re-heal churn
- ecstore/replication: resync results channel switched from
broadcast(1) to mpsc; a lagged broadcast receiver ended the stats
collector and every subsequent failure went uncounted, letting
failed resyncs be marked completed
- ecstore/replication: ignore an empty persisted resync checkpoint;
resuming with one skipped every object and marked the resync
completed without replicating anything
- ecstore/replication: fix inverted not-found error classification in
replicate_object/replicate_delete logging paths
- ecstore/erasure: guard decode paths against zero block_size or
data_shards from corrupt on-disk metadata (divide-by-zero panic)
- ecstore/disk: os::read_dir no longer consumes the entry limit on
entries it does not return (is_empty_dir misjudgment); create_file
opens with O_TRUNC to avoid stale trailing bytes
- filemeta: treat Some(nil) version id as a null version in
matches_not_strict; disk-loaded headers never store None, so the
mod_time quorum guard for unversioned overwrites never fired and an
interrupted overwrite could displace the committed version in merge
- filemeta: fix msgpack skip lengths for fixext (missed the ext type
byte) and ext16/32 (over-skipped) unknown fields
- filemeta: return FileCorrupt instead of usize underflow when
xl.meta is truncated inside the CRC trailer
- filemeta: surface delete-marker insertion failure in delete_version
instead of reporting success when the data dir is shared
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(replication): drop duplicate cfg(test) etag import from boundary module
The test module already imports content_matches_by_etag locally, so the
top-level cfg(test) import is unused under -D warnings and fails clippy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
fix(ecstore): purge orphan empty-directory trees on folder delete (#4189)
Empty "phantom" folders — on-disk directory trees with no xl.meta
anywhere — show up in listings as common prefixes but cannot be removed
by any S3 call: DeleteObject on the folder key encodes it to __XLDIR__,
finds no metadata, and returns NotFound, which the API layer masks as a
fake 204 while the tree survives on disk.
Make handle_delete_object attempt a guarded purge when a dir-object key
resolves to NotFound:
- sweep every erasure set of every pool (orphan fragments can sit on
any set, left behind by whichever sets stored the deleted children)
- per set, refuse if ANY disk holds a regular file under the prefix,
so degraded/healable objects are never destroyed
- remove empty directories children-first with non-recursive deletes;
a racing PutObject makes the rmdir fail with DirectoryNotEmpty and
the entry is skipped
This state is unrepresentable on AWS S3 (CommonPrefixes derive only
from real keys; DeleteObject is an idempotent 204), so the purge moves
visible behavior closer to S3. It deliberately goes beyond MinIO, whose
DeleteObject leaves these trees stranded and whose heal only removes
dangling (minority-presence) dirs — a long-standing complaint
(minio/minio#15257, #19516, #18444).
Also move the test-only content_matches_by_etag import in
replication_target_boundary.rs into the cfg(test) module; it failed
cargo clippy -D warnings and blocked the pre-PR gate.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
content_matches_by_etag is only used inside the #[cfg(test)] module, so
the lib-scope import from #4211 fails cargo clippy -D warnings on every
non-test build.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(ecstore): remove reachable panics in tiering, replication, and heal paths
- Parse x-amz-expiration leniently in tier PUT responses; any lifecycle
rule on the remote tier bucket returns an RFC1123 date that the previous
ISO8601 unwrap turned into a panic of the ILM transition worker
- Skip invalid user-metadata header values (with a warning) when building
tier and replication PUT headers instead of panicking on non-ASCII input
- Heal: tolerate absent data_dir for delete markers and remote objects
- transition_object: don't unwrap version_id on unversioned buckets when
recording partial writes for offline disks
- Admin server info: use port_or_known_default() so default-port (80/443)
endpoints don't panic is_server_resolvable
- Tier ListObjectsV2 client: decode response body with from_utf8_lossy
- walk_internal: log merge_entry_channels errors instead of dropping them
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(ecstore): fail heal explicitly when data_dir is missing
Address review feedback: unwrap_or_default() silently substituted a nil
UUID when latest metadata lacked data_dir. Delete markers and remote
objects legitimately have no data_dir and skip the data-heal block, but
for a regular object a missing data_dir means corrupt metadata — return
FileCorrupt with a descriptive log instead of building part paths under
a nil UUID directory.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Zhengchao An