fix(ecstore): only evict remote lock channel on transport failures (#4567)
The remote lock RPC path evicted the cached gRPC channel on *any*
`tonic::Status`. Genuine transport failures (connect refused/reset,
keepalive/deadline) surface that way, but so do server-produced
application statuses on a perfectly healthy channel: `Unauthenticated`
/`PermissionDenied` from the signature interceptor, `Internal`
/`FailedPrecondition` when the peer lock service is not ready yet,
`InvalidArgument`, `ResourceExhausted`, `Unimplemented`, and so on.
Evicting the channel for those cannot help — the channel is fine — and
each spurious eviction re-dials the connection and advances the peer
toward the offline threshold via `record_peer_unreachable`, producing
background channel churn on an otherwise healthy cluster.
Classify the tonic status with `is_transport_failure` (underlying
transport source present, or one of `Unavailable`/`DeadlineExceeded`
/`Unknown`/`Cancelled`) and only evict the cached channel when it is a
real transport failure. The client-side timeout arm still evicts. This
mirrors the transport-vs-application distinction already used on the
remote-disk RPC path.
* refactor(ecstore): migrate the background-services cancel token into InstanceContext (Phase 5 Slice 13)
Phase 5 Slice 13 (backlog#939): move the background-services cancellation token
out of the process static into the per-instance InstanceContext, so cancelling
one instance's background workers (scanner/heal/tier/lifecycle) no longer
touches another instance.
- InstanceContext gains `background_cancel_token: OnceLock<CancellationToken>`
with `init_background_cancel_token` (set-once) and `background_cancel_token()`
returning an owned clone.
- global.rs `init_/get_/create_/shutdown_` helpers keep their signatures and
route through the current instance's context; the static is removed. The
getter now returns an owned `Option<CancellationToken>` instead of a
`Option<&'static _>`, which is what lets the token live in the context.
- Callers adapt to the owned token: the metadata-refresh loop drops `.cloned()`;
the lifecycle worker/loops take the owned token (a shared fallback is cloned
when the token is somehow uninitialized). No Arc cycle is introduced —
workers hold a token clone, not the instance context.
Single-instance behavior is unchanged: startup creates one token in the
bootstrap context the ECStore adopts, and shutdown cancels that same token.
Tests: the token is set-once and cancelling one instance's token leaves a
distinct instance uninitialized.
Verification: cargo test -p rustfs-ecstore (23 instance-context tests green),
cargo clippy -p rustfs-ecstore --all-targets (clean), make pre-commit (pass).
Refs: backlog#939 (Phase 5, Slice 13)
* test(ecstore): prove multi-instance isolation; document embedded guard retention (Phase 5 Slice 14) (#4588)
Phase 5 (backlog#939) capstone. The prior 13 slices moved every piece of
per-instance runtime state out of process globals into ECStore's
InstanceContext. This slice proves the result and records the remaining work.
- Add `two_instances_isolate_all_migrated_state`: an end-to-end acceptance test
that constructs two independent InstanceContexts and verifies NONE of the
migrated state is shared — erasure setup, lock manager, region, deployment id,
the four service handles (tier/notifier/expiry/transition), the local disk
registry, the bucket monitor, and the background cancel token. This is the
object-graph isolation carrier working end to end.
- Document why the embedded single-instance guard (EMBEDDED_SERVER_STARTED) is
intentionally retained: storage startup still publishes into the process-level
bootstrap context (write-once region/endpoints/deployment id) and the single
GLOBAL_OBJECT_API handle, so a second startup would fail-fast on that shared
state. Lifting the guard requires threading a per-instance context through
startup — a follow-up beyond migrating the globals. The guard is NOT removed:
rejecting the second start is safer than the panic it would otherwise become.
Verification: cargo test -p rustfs-ecstore (acceptance test + all instance-context
tests green), cargo clippy -p rustfs-ecstore --all-targets (clean), make
pre-commit (pass).
Refs: backlog#939 (Phase 5, Slice 14). Stacked on Slice 13 (#4586).
The internode HTTP/2 window/keepalive tuning (apply_http_client_tuning +
http2_keep_alive_* in build_http_client) only takes effect when the connection
negotiates HTTP/2, which happens via TLS-ALPN. Over plaintext internode
transport the connection is HTTP/1.1 and all the tuning is silently inert,
with no signal to the operator.
Make this honest without changing protocol behavior (no h2c, no
prior_knowledge, no opt-in flag):
- InternodeHttpClientTuning gains h2_tuning_explicit, true when the operator
set a window size, a non-default tuning profile, or the keepalive env.
- Add pure predicate should_warn_h2_inert(negotiated_is_http2,
h2_tuning_explicit, already_warned) and maybe_warn_h2_inert, gated by a
process-lifetime AtomicBool so the warn fires at most once.
- Call maybe_warn_h2_inert at both record_internode_http_version sites using
the actual negotiated resp.version().
- Document the TLS-ALPN requirement at apply_http_client_tuning and near the
ENV_INTERNODE_HTTP2_* constants.
Tests cover the should_warn_h2_inert truth table and that h2_tuning_explicit
reflects explicit windows and a non-default profile.
fix(server): count in-flight HTTP requests with an RAII guard (backlog#806-35)
The active-requests gauge was maintained with tower-http TraceLayer hooks:
on_request (+1), on_response (-1) and on_failure (-1). With the default
ServerErrorsAsFailures classifier a 5xx response fires BOTH on_response and
on_failure, so every 5xx decremented the gauge twice (net -1); a streaming
200 that failed mid-body did the same. The gauge drifted downward and
saturated at zero, corrupting the readiness busy-protection signal
(alias_busy_threshold_exceeded).
Replace the hook arithmetic with an InFlightGuard (RAII): InFlightLayer wraps
each request future, incrementing on entry and decrementing exactly once when
the future resolves (any status) or errors/cancels. Removed the +1 from both
on_request hooks and the -1 from trace_on_response and both on_failure hooks,
keeping their tracing and failure-metric work intact. Applied to both the
external and internode stacks.
Added a test driving the layer for 2xx, 5xx and a no-response service error,
asserting the gauge nets to zero in every case (the 5xx path is the fix).
fix(ecstore): age LastMinuteLatency samples individually (backlog#806-16)
The rolling one-minute latency window stored bare Duration samples plus a
single, never-updated start_time. Its retain predicate ignored the element
and evaluated the constant now.duration_since(start_time) < 60s, so once the
window had existed for 60s every add() dropped ALL samples and the average
degenerated to the latest single sample.
Each sample now carries its own Instant and is retained by its individual
age. The type backs only in-memory EpHealth + admin display (the wire shape
is target::LatencyStat with curr/avg/max), so Serialize/Deserialize is kept
only to satisfy the enclosing LatencyStat derive; the sample Instant is
serde(skip) and restarts aging on reload.
fix(io-metrics): drop client-controlled bucket label from rustfs_s3_operations_total (backlog#806-22)
The bucket dimension on the rustfs_s3_operations_total counter is
client-controlled and unbounded, letting any client explode metric
cardinality (a Prometheus/OTEL cardinality DoS that grows the in-process
OTEL aggregation store even without a scrape). Drop the bucket label so the
series is keyed by the bounded op dimension only (<=122 variants), matching
MinIO which never labels its default operation counters with bucket.
BREAKING: rustfs_s3_operations_total no longer carries a "bucket" label.
record_s3_op(op, bucket) -> record_s3_op(op); all call sites in
rustfs/src/storage/{ecfs.rs,helper.rs} updated (bucket bindings retained
where still used by audit/notify paths). Add metrics-util debugging recorder
dev-dep and tests asserting the series carries only the op label and that
series count equals distinct-op count.
DeleteServiceAccount looked up the target service account and, on any
get_service_account error OTHER than not-found (e.g. a stored-credential decode
error), set svc_account = None and continued. The non-admin owner guard then
used svc_account.is_some_and(|v| v.parent_user != user), which is FALSE for a
None target, so the guard was skipped and the code fell through to the delete —
a non-owner could delete a service account whose stored credential failed to
decode (authz bypass, fail-open).
Extract a fail-closed helper non_admin_may_delete_service_account(caller, target)
that returns true only when the target loaded AND is owned by the caller; a None
target (unverifiable ownership) or an owner mismatch now denies. Admins (holding
RemoveServiceAccount) are unaffected. + unit test.
Refs rustfs/backlog#806
fix(rio): typed-first internode HTTP error classification (backlog#805)
classify_reqwest_error matched reqwest/hyper error text by English
substrings, brittle to library/OS-locale wording. Refactor to a pure,
testable classify_transport_error(err, is_timeout, is_connect, is_body)
that trusts structured signals first: caller timeout, then the io::Error
kind found anywhere in the source chain (typed wins over any string/body
signal so a real ConnectionRefused is never mislabeled DnsResolutionFailed),
then a DNS-only string heuristic gated behind is_connect, then body, then
Unknown.
Flip DnsResolutionFailed to retryable, mirroring MinIO IsNetworkOrHostDown
(*net.DNSError => network-or-host-down => retryable); RustFS already retries
transient DNS at endpoints.rs, and bounded retries cost at most one extra
attempt on permanent NXDOMAIN.
Swap the ecstore remote_disk non-retryable exemplar from DnsResolutionFailed
to Unknown accordingly.
Add a default-on multi-role adversarial validation policy to the root
AGENTS.md (risk tiers, six reviewer roles, findings protocol, exit
criteria), a shared adversarial-validation skill with repo-specific
attack probes grounded in shipped bugs, seven audit fixes to the root
instruction files, and an actionlint gate for workflow changes.
In the default (non-rio-v2) build, SSE-C encrypted with AES-256-GCM using a deterministic nonce derived from MD5("{bucket}-{key}"), and decrypt recomputed it. Overwriting the same object under the same SSE-C key reused an identical (key, nonce) pair, breaking AES-GCM confidentiality and integrity. The base nonce was never persisted.
Generate a fresh random 12-byte nonce for every single-PUT and multipart session, persist it under both x-rustfs-encryption-iv and the MinIO interop key, and resolve it from stored metadata on decrypt and multipart part encryption. Objects written before this change carry no stored IV and fall back to generate_ssec_nonce, so existing objects still decrypt.
Refs rustfs/backlog#1039
fix(object-capacity): harden clock handling in write window, background intervals and scope registry
Three low-severity robustness gaps (S32+S33+#35):
- WriteRecord keyed its 60-bucket write window by wall-clock unix
seconds while the debounce used Instant. An NTP step backwards marked
recent buckets as future and silently suppressed write-triggered
refreshes until the wall clock caught up; a forward step aged the
whole window at once. Bucket keys now derive from a monotonic
per-record epoch, immune to clock steps.
- Background refresh/metrics intervals were only clamped at zero;
RUSTFS_CAPACITY_SCHEDULED_INTERVAL=u64::MAX overflowed
Instant + Duration and panicked the spawned task, silently killing
scheduled refreshes. Intervals now clamp into [1s, 30 days] via one
helper with a structured warn.
- record_capacity_scope merged new disks into an expired entry and
refreshed its recorded_at, resurrecting a scope take_capacity_scope
would have discarded (and the merge path never pruned). Expired
entries are now replaced, not merged into.
Ref: rustfs/backlog#1022 (S32+S33+#35 from audit rustfs/backlog#1010)
- build.yml: update latest.json only for stable release tags
(alpha/beta/rc tags previously overwrote the stable pointer with
release_type "stable"); drop the placeholder .asc files; build the
Linux targets only on main pushes (tags/schedule/dispatch keep the
full matrix); remove the unreachable --build clause and a needless
full-history clone
- ci.yml: event-scoped concurrency so merges stop cancelling the
weekly scheduled run; drop the redundant skip-duplicate-actions
gate job; run clippy before tests; trim the unused toolchain from
the typos job
- ci-docs-only.yml (new): satisfy the required "Test and Lint" check
on docs-only PRs that ci.yml skips via paths-ignore
- audit.yml: drop cargo-audit (cargo-deny advisories covers the same
RustSec database); same event-scoped concurrency fix
- docker.yml: job-level short-circuit for per-merge dev builds; send
the Trivy SARIF to code scanning; unify the upload-artifact pin
- performance-ab.yml: add concurrency; only re-run on labeled events
when the added label is perf-ab
- stagger the Sunday crons (build 01:00, audit 03:00,
nix-flake-update 05:00, mint 06:00)
- delete performance.yml: disabled since 2025-07; idle-server
profiling, no benchmark baseline, stale ecstore package name
fix(metrics): normalize peer-health key so servers_offline_total counts each peer once
rustfs_cluster_servers_offline_total is the count of distinct keys in the
addr-keyed CLUSTER_PEER_HEALTH map. The data path keys a peer by
endpoint.grid_host() (scheme://host:port) while the lock path keys it by
url::Url::to_string() (scheme://host:port/, trailing slash), so one physical
peer becomes two health entries and a single downed node reports 2 (2N for N).
Normalize the address (trim trailing slashes) at the map boundary in
record_peer_reachable/record_peer_unreachable/cluster_peer_is_offline/
cluster_peer_should_bypass so both forms collapse to one canonical key.
Pure observability fix; peer selection and quorum are unchanged.
Fixesrustfs/backlog#1043
Co-authored-by: heihutu <heihutu@gmail.com>
The tracker never influenced traversal: walkdir's follow behavior is
fixed up front by follow_links(), and should_follow() only gated the
tracker's own bookkeeping. Its 'depth limit' compared tree depth (not
symlink chain depth), so RUSTFS_CAPACITY_MAX_SYMLINK_DEPTH was a
complete no-op while its telemetry claimed symlinks were skipped that
walkdir had in fact followed and counted; record_symlink was always
called with size 0, so tracked_bytes never left zero (S12).
Remove the tracker, its skipped/summary events, the symlink metric and
the depth env knob end to end (the env's 'as u8' truncation goes with
it), and document the real semantics at the walker: follow_links(true)
counts targets with walkdir's ancestor-loop detection breaking cycles,
follow_links(false) — the default — counts no symlink targets. The scan
root itself is pre-resolved since backlog#1015.
Ref: rustfs/backlog#1018 (S12 from audit rustfs/backlog#1010)
PR #4560 added a 15th parameter (allow_inplace_legacy_fallback) to
build_codec_streaming_part_reader while the negative arity tests from a
concurrently developed branch still call it with 14 arguments — the two
changes merged cleanly textually but broke the workspace test build on
main (E0061 in set_disk/read.rs), failing CI for every open PR.
Pass false at both call sites: these tests assert Err outcomes
(oversized part, missing quorum) that are independent of the fallback
mode, and false preserves the historical whole-request fallback
semantics used by eager first-part reads.
walkdir defaults to follow_root_links=true precisely so a root that is
itself a symlink still descends, but the scanner overrode it with the
general follow_symlinks flag (default false). For the common indirection
layout (/data/rustfs0 -> /mnt/vol0) the walk yielded a single skipped
symlink entry and returned used_bytes=0, file_count=0, is_estimated=false
with no error and no log — an exact 0 committed straight into the
per-disk baseline (S05).
Canonicalize the scan root before walking (also covering nested
indirection and keeping the dedicated-mount statvfs check on the real
path); resolution failures surface as scan errors instead of a silent
zero. Inner-symlink policy is unchanged.
Ref: rustfs/backlog#1015 (S05 from audit rustfs/backlog#1010)
RUSTFS_CAPACITY_MAX_FILES_THRESHOLD=0 made every file count as
overflow with an empty exact prefix, so disks with fewer files than the
sample rate committed 0 bytes as the cluster total; STAT_TIMEOUT=0 with
dynamic timeout disabled made every scan fail at the first entry. Both
were accepted unvalidated, unlike the already-clamped sample_rate and
interval values.
Zero values for the threshold and the stat/min/max timeouts now fall
back to their defaults with a single structured warn at config load
(S11). The symlink-depth knob is left untouched here: it is removed
entirely by the backlog#1018 fix.
Ref: rustfs/backlog#1019 (S11 from audit rustfs/backlog#1010)
Two ecstore test groups pass in isolation but flake under the full parallel
nextest suite, producing CI noise on every in-flight PR (backlog #937):
- store::bucket::tests::bucket_delete_* race make_bucket into
InsufficientWriteQuorum because they share global disk-registry/lock-client
state with other concurrently running tests.
- bucket_lifecycle_ops::tests::concurrent_resend_same_part_commits_one_generation
exceeds its already-maxed 60s lock-acquire deadline only when the suite
saturates disk I/O.
serial_test's #[serial] does not serialize these across nextest's per-test
process boundary. Add a nextest test-group (max-threads = 1) that matches both
groups so they run serialized, removing the load-induced flake. No test or
production code changes; long-term global-singleton isolation is tracked
separately.
Co-authored-by: heihutu <heihutu@gmail.com>
Progress/timeout checks only ran when file_count advanced, so trees of
directory-only or error-dense entries (dirs, traversal errors, symlinks
never increment file_count) bypassed the entire cooperative time budget
and held the blocking thread for unbounded time, while each error entry
logged its own warn — permission-dense trees flooded the log (S13).
The stall detector was structurally unreachable: it fired on 'no file
progress between checks', but checks themselves only ran when file
progress happened, so RUSTFS_CAPACITY_STALL_TIMEOUT never did anything
since its introduction (S09).
- Progress checks (timeout + early-sampling entry) are now driven by a
visited-entry counter that also advances on directories and errors, so
every tree shape reaches the budget checks.
- Remove the unreachable stall detector and its plumbing end to end
(ProgressMonitor fields, ScanLimits, config getter, env const, stall
metric). Genuine walker wedges are handled by the hard outer
wall-clock budget from backlog#1017; no decorative protection is left.
- Cap per-entry error warns at 10 per scan with an explicit suppression
notice; had_partial_errors still records the condition.
Ref: rustfs/backlog#1016 (S09+S13 from audit rustfs/backlog#1010)
perf(ecstore): wire legacy decode stripe prefetch / bitrot-decode overlap (backlog#930 HP-9 step 2)
The legacy GET decode duplex loop (default GET path + every Range request)
was strictly serial: read a stripe, reconstruct it, emit it, and only then
begin reading the next stripe. Two switches introduced by PR#3972 to overlap
this work — RUSTFS_GET_DECODE_STRIPE_PREFETCH_COUNT and
RUSTFS_GET_BITROT_DECODE_OVERLAP_ENABLE — were config-only with zero call
sites since introduction.
Wire both into the loop as a depth-1 stripe prefetch: while the current
stripe is reconstructed and emitted, the next stripe's shard reads (including
bitrot verification) run concurrently, hiding read latency under the emit /
duplex-backpressure stage. ParallelReader::read is inherently serial (it takes
&mut self and advances a shared stripe cursor), so at most one read can be in
flight; a prefetch count above 1 therefore collapses to the same
single-stripe-ahead pipeline rather than reading several stripes ahead. Both
switches feed one gate, legacy_stripe_prefetch_enabled().
Default (count == 1, overlap == false) keeps the loop on the pre-existing
strictly-serial path, byte for byte: the prefetch pipeline is a separate branch
and the serial branch is unchanged. The reconstruct/emit body is factored into
a shared emit_decoded_stripe helper used by both branches so error attribution,
read-quorum handling, reconstruction verification and stage metrics cannot
drift between them.
Correctness guarantees preserved under prefetch:
- A speculatively prefetched read for stripe N+1 is only consumed when the loop
reaches N+1; if stripe N stops the loop, the in-flight read is awaited and
dropped, so its errors never surface against stripe N.
- Bitrot (HighwayHash) verification runs inside each stripe read and is not
bypassed or reordered; a corrupt shard is still rejected and, when
unrecoverable, the read fails without emitting garbage.
- Shard buffers are recycled only after the overlapping next read has claimed
its own — one extra stripe of memory (double buffering) with buffer reuse
preserved at a one-stripe lag.
- Per-stripe exact length advance (backlog#799 B2), lockstep reconstruction
verification (backlog#832) and the hash_size == 0 pass-through are unchanged.
Adds regression tests exercising serial-default, count>1 and overlap configs:
full/range reads byte-exact, degraded (missing-shard) reconstruction, corrupt
shard rejected-but-recovered, unrecoverable corruption erroring with no output,
late-stripe failure attributed correctly with stripe 0 still emitted,
hash_size == 0 pass-through, and the gate defaulting off.
Co-authored-by: heihutu <heihutu@gmail.com>
Two report-only defects on the heal result-reporting surface (they do not
affect the data path or heal decisions):
- default_heal_result (set_disk/ops/heal.rs): the offline-disk branch pushed
an Offline record but fell through into the unconditional push, emitting a
second (Corrupt) record for the same disk. This grew before/after.drives to
disk_count + offline_count and misaligned every entry after the first offline
slot. Add `continue` after the offline push, drive `disk_len` and the loop
from a single `self.disks` snapshot, and assert `errs.len() == disk_len`.
- Sets::heal_format (core/sets.rs): the before/after drive lists were
pre-filled with N default placeholders and then N real entries were pushed,
yielding a 2N list whose healed status updates (indexed 0..N) landed on the
blank placeholder half. Assign the lists directly from formats_to_drives_info
(mirroring the set-level heal_format) so the healed updates hit the real
entries.
Add regression tests covering offline/online record alignment and the
NoHealRequired and heal paths of the pool-level heal_format.
Co-authored-by: heihutu <heihutu@gmail.com>
Sub-change A only: the two local-disk metadata read paths
(`read_metadata_with_dmtime`, `read_all_data_with_dmtime` in
crates/ecstore/src/disk/local.rs) previously dispatched open, fstat and the
xl.meta read as three separate async fs hops (three spawn_blocking round-trips
under tokio::fs). Each is now folded into a single tokio::task::spawn_blocking
closure using std::fs, cutting per-metadata-read dispatch from 3 to 1.
This does NOT touch sub-change B (removing the entry-point access() call).
Correctness first: this is the hottest metadata read path, so the refactor
preserves byte-for-byte-equivalent Results. Every error mapping is kept
identical:
- open failure -> to_file_error
- is_dir -> Error::FileNotFound (not to_file_error(EISDIR))
- metadata failure -> to_file_error
- xl.meta parse -> propagated verbatim from the parser
- try_reserve -> Error::other
- read_to_end -> to_file_error
For read_all_data_with_dmtime the async NotFound -> access(volume_dir) ->
VolumeNotFound fallback (and its warn! event) is preserved on the async side:
the closure returns the raw open error unmapped; only open() can yield ENOENT
once the fd is valid, so gating the fallback on the open error is equivalent to
the original open-arm-only fallback.
To run the parser inside a blocking closure, filemeta gains a synchronous twin
`read_xl_meta_no_data_sync` (+ `read_more_sync`) in
crates/filemeta/src/filemeta/version.rs that line-for-line mirrors the async
version, differing only in std vs tokio read_exact. A new equivalence test
(`read_xl_meta_sync_equivalence_tests`) feeds identical buffers to both the
async and sync readers and asserts equal Ok bytes / equal Err variants across:
v1.0; v1.1/v1.2/v1.3; large meta triggering read_more; header truncation ->
UnexpectedEof; CRC-trailer truncation -> FileCorrupt; unknown major/minor ->
InvalidData; and want boundaries (exact fit, inline-data drop, read_more EOF).
Co-authored-by: heihutu <heihutu@gmail.com>
Two disk-layer correctness fixes in crates/ecstore/src/disk/local.rs.
move_to_trash (#948, ECA-07) previously handled only DiskFull in its
tail error block and let every other rename failure (EIO/EACCES/ENOTDIR/
cross-device) fall through to `return Ok(())`, so a delete that never
landed on a faulty disk was reported as success and the drive fault was
hidden from heal/offline logic. It now keeps the DiskFull in-place-remove
fallback, treats a missing source (FileNotFound) as benign, and
propagates every other error (already mapped by to_file_error, e.g.
I/O -> FaultyDisk, permission -> FileAccessDenied), matching MinIO's
deleteFile.
rename_file and rename_part (#960, ECA-19) directory (trailing-slash)
branch unconditionally removed the destination before rename and
propagated the resulting NotFound, so renaming a directory to a new
location always failed. Both now tolerate ErrorKind::NotFound on that
pre-rename remove and continue, matching MinIO's RenameFile.
Adds regression tests covering benign-missing-source, real-error
propagation, the unchanged move happy path, and directory rename to a
missing destination for both rename_file and rename_part.
Co-authored-by: heihutu <heihutu@gmail.com>
An object transitioned to an unversioned remote tier legally records an
empty remote version_id (per CLAUDE.md: a tier version of `None`/`""`
means the tier bucket is unversioned, so remote GET/DELETE must omit the
versionId). Two recovery paths wrongly treated that empty sentinel as an
incomplete/unrecoverable record while the persist/encode and worker
paths accept it, producing permanent leaks in the crash-recovery window.
- tier_delete_journal::into_jentry rejected entries with an empty
version_id as "incomplete", so unversioned-tier WAL entries could never
be decoded during recovery: the remote object was orphaned and the
journal file leaked forever. Drop the version_id emptiness check; keep
the obj_name/tier_name checks. Truncated payloads are still rejected at
JSON deserialization (all fields are required, non-Option, no default,
with deny_unknown_fields).
- tier_free_version_recovery::is_recoverable_tier_free_version required a
non-empty version_id, silently filtering out free versions of
unversioned-tier objects so a first enqueue failure leaked the remote
object and local free version permanently. Drop the version_id check;
keep free_version + name + tier checks.
Both downstream paths already issue a versionless remote delete for an
empty version_id, so no further changes are needed. Adds regression
tests covering the empty-version_id case and the preserved guards.
Co-authored-by: heihutu <heihutu@gmail.com>
bitrot_shard_file_size only counts per-block checksum bytes for the two
streaming Highway variants, while BitrotWriter::write interleaves a hash
for any hash_algo.size() > 0 and bitrot_verify's read loop assumes an
interleaved hash per block. The three disagree for non-streaming
algorithms (SHA256/HighwayHash256/BLAKE2b512/Md5), but the divergence is
unreachable in production: every write path hardcodes HighwayHash256S and
ErasureInfo::get_checksum_info defaults to HighwayHash256S.
Per the audit decision (backlog#959), do NOT change the size formula:
it is a byte-for-byte port of MinIO's bitrotShardFileSize and its bare
return for non-streaming algorithms is correct for MinIO whole-file
bitrot; changing it would break legacy interop. Instead, document the
per-algorithm layout contract at bitrot_shard_file_size, BitrotWriter,
and bitrot_verify, and add regression tests that pin the invariants:
get_checksum_info defaults to HighwayHash256S, and the size formula
counts per-block hash bytes for streaming variants only while returning
the bare size for non-streaming ones. No disk layout or formula change.
Co-authored-by: heihutu <heihutu@gmail.com>
reduce_errs seeds best_err with a synthetic Error::other("nil") placeholder
via unwrap_or. When the error slice is empty or every error is ignored,
err_counts is empty and nil_count is 0, so both nil tie-break conditions are
false and the else branch returned (0, Some(Error::other("nil"))). That
placeholder leaked to build_write_quorum_failure_summary, where dominant_error
became Some("nil"), skipping the nil_dominated early return and falling back to
the "other_error" metric label. Quorum decisions were unaffected (max_count 0
still fails any positive quorum), but write-quorum failure metric labels and
tracing fields were misclassified.
Add an explicit short-circuit: when best_count == 0 and nil_count == 0, return
the (0, None) sentinel so the placeholder never escapes. Add regression tests
for the all-ignored slice, empty slice, and the summary label path, which the
existing test_build_write_quorum_failure_summary (nil_count > 0) did not cover.
Fixes#957
Co-authored-by: heihutu <heihutu@gmail.com>
Two dirty-mark lifecycle gaps (S19+S30):
- A commit cleared every dirty mark for the disks it scanned, even marks
recorded while the walker was already past the written prefix, so the
next dirty-subset refresh served stale cached bytes as exact until the
next full scan. Dirty marks now carry the instant they were last
recorded and a commit only clears marks that predate its scan start
(CapacityUpdate::scan_started_at, set by refresh_capacity_with_scope).
- The write side marks every disk of an EC set dirty, including remote
peers, while scans and clearing only cover local disks — remote or
removed disks stayed marked forever, keeping the dirty-disk gauge
permanently non-zero with bounded memory stuck behind it. The refresh
disk selection now prunes dirty entries outside the current local
topology before consulting them.
Ref: rustfs/backlog#1020 (S19+S30 from audit rustfs/backlog#1010)
complete_multipart_upload deleted every part.N.meta via
cleanup_multipart_path *before* the authoritative rename_data commit.
If rename_data then failed write quorum, the upload directory was left
with data files but no part metadata, so a retried CompleteMultipartUpload
could never read the parts again and the upload became permanently
uncompletable (client must abort and re-upload all parts).
Move cleanup_multipart_path to run only after rename_data returns Ok,
matching the existing "clean up only after commit" pattern already used
for the old data-dir GC and the upload-dir delete_all. On a failed commit
the staging part.N.meta now survive so the retry can complete.
Add a hermetic regression test that forces rename_data to fail on every
disk (destination bucket dir made read-only) and asserts the staging
part.N.meta are preserved for retry.
Fixes#946.
Co-authored-by: heihutu <heihutu@gmail.com>
fix(io-core): decrement available_buffers gauge on pooled-buffer reuse (backlog#806)
return_buffer does a fetch_add(1) on push but take_or_allocate_buffer's
available.pop() reuse path had no matching fetch_sub, so the available_buffers
gauge only ever grew and never reflected the real pool size. Decrement it on
reuse; + regression test.
load_all fetched policies/users/user-policies by passing the FULL remaining
list to load_*_concurrent every iteration and then split_off(32), so each item
was re-fetched once per preceding chunk — O(n^2) redundant loads on startup for
>32 items. Replace the split_off loops with chunks(32) so each item is fetched
exactly once. Behavior-preserving (cache inserts were already idempotent).
Erasure heal set write_quorum to the count of available target writers,
so every replacement disk had to succeed writing every block or the whole
object heal aborted. A single flapping replacement disk failing one block
made MultiWriter::write return Err, heal propagate Err, and the ops layer
delete the entire tmp staging dir — leaving the other healthy replacement
disks unhealed and blocking redundancy recovery indefinitely.
Set the heal write_quorum to 1, matching upstream MinIO's writeQuorum=1
for heal. MultiWriter already marks a failed writer as None, and the ops
layer (set_disk/ops/heal.rs) already drops the failed writer while
committing the survivors; the read-side quorum check and parity
cross-checks that guarantee reconstruction correctness are unchanged.
Add regression tests: one healthy-and-one-failing target still heals the
healthy targets and drops the failing one; all-targets-failing still
returns Err so nothing is falsely committed.
Fixes#947
Co-authored-by: heihutu <heihutu@gmail.com>
Two defensive gaps left the refresh singleflight vulnerable to a
permanent wedge (running=true forever: scheduled refreshes silently
stop, admin capacity joiners hang until restart):
- spawn_refresh_if_needed's spawned task had no leader guard, unlike
refresh_or_join: a panic after the refresh future completes (commit,
metrics) killed the task before the reset block. The task now holds
the same RAII RefreshLeaderGuard, disarmed only after the state is
reset and the result published (S20).
- refresh_fn() was evaluated before AssertUnwindSafe wrapping in both
paths, so a panic while constructing the future escaped catch_unwind;
construction now happens inside the wrapped future.
- RefreshLeaderGuard::drop silently skipped the reset when try_lock was
contended and no tokio runtime was current; it now falls back to a
blocking reset, which is safe precisely because there is no executor
to stall in that context (S31).
Ref: rustfs/backlog#1021 (S20+S31 from audit rustfs/backlog#1010)
Close the ~10ms instrumentation residual on the PUT success path that the
existing coarse writer_setup/encode/rename stage metrics do not attribute.
Adds `hp_guard!` measurement scopes to the previously uninstrumented
sub-stages called out in backlog#935 item 2:
- SetDisks::acquire_read_lock / acquire_write_lock (namespace lock acquire)
- S3::put_object_prelookup (pre-write get_object_info lookup)
- MultiWriter::shutdown (bitrot writer flush/close)
- SetDisks::commit_rename_data_dir (old data-dir reclaim)
- S3Access::put_object (S3 authorization segment)
Instrumentation only: `hp_guard!` expands to nothing without the `hotpath`
feature, so this is a pure-observation change with zero behavior impact and
zero cost in default builds. The pre-lookup site wraps only the lookup call
in a scoped block so the guard measures that slice exactly while preserving
the existing match and control flow.
Verified: cargo check -p rustfs-ecstore (default and --features hotpath) and
cargo check -p rustfs (default and --features hotpath) all pass.
Co-authored-by: heihutu <heihutu@gmail.com>
Zhengchao An