* perf(ecstore): read bitrot hash+data in one pass on the shard path
BitrotReader::read issued two reads per block: one read_exact for the
32-byte hash, then a separate loop for the shard data. On the streaming
disk reader (a raw tokio File whose every read is a spawn_blocking
round-trip) that is two dispatches per block. Since the on-disk layout is
a contiguous [hash][data] run, pull both in a single pass into a reused
scratch buffer and split afterwards, halving the per-block dispatch count
on the streaming path. The no-hash path still reads straight into the
caller buffer with no extra copy, and an in-memory Cursor (inline/mmap)
just does a slice copy.
All existing invariants are preserved: a short read of either the hash or
the data maps to UnexpectedEof before and independent of the hash check
(backlog#799 B2), and the hash-mismatch / InvalidData semantics are
unchanged.
Correctness-only change; the per-dispatch latency win is platform
dependent and its default reliance is left to the warp size-bucket
benchmark gate tracked by backlog#935, per the backlog#933 acceptance
note.
Refs: rustfs/backlog#933 (HP-12 item 2), rustfs/backlog#936
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs: reword bitrot test comment to satisfy typos check
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
A dirty-subset refresh scans only the dirty disks, so its raw
`CapacityUpdate` carries just that subset's `total_used`/`file_count`.
`update_capacity` recomputed the correct cluster-wide total into a local
variable and wrote it to the cache, but never wrote it back to the
`CapacityUpdate` that `refresh_or_join`/`spawn_refresh_if_needed` return
and publish to joiners. The admin blocking path (`resolve_admin_used_capacity`
-> `refresh_or_join_admin_disks(allow_dirty_subset=true)`) consumes
`update.total_used`, so a single dirty disk in an N-disk cluster made
admin StorageInfo report only the scanned subset's bytes (a large
transient undercount for that request and same-cycle joiners).
`CachedDiskCapacity` also dropped each disk's `file_count`/`is_estimated`,
so subset refreshes could not recompute a correct cluster file count and
would launder an estimated per-disk value into an exact cluster total.
Fix:
- `update_capacity` now reconciles `total_used`, `file_count`, and
`is_estimated` from the full per-disk cache and returns the corrected
`CapacityUpdate`.
- `CachedDiskCapacity` stores `file_count`/`is_estimated` per disk;
`file_count` sums the cache and `is_estimated` is the OR across disks.
- `refresh_or_join` and `spawn_refresh_if_needed` rebind their result to
the reconciled update so the leader return value and the value
published to joiners both carry cluster totals.
Tests: extend the subset-refresh test to assert the returned update's
reconciled `total_used`/`file_count`/`is_estimated`, and add a
`refresh_or_join` dirty-subset test asserting the leader returns the
merged cluster total (not the subset sum) and matches the cache.
Refs: https://github.com/rustfs/backlog/issues/1011
Phase 5 Slice 4 (backlog#939): move the S3 region — a write-once identity
scalar — out of the GLOBAL_REGION process static into the per-instance
InstanceContext, so two instances can serve different regions.
- InstanceContext gains `region: OnceLock<Region>` with `set_region()` /
`region()`. `set_region` keeps the write-once fail-fast contract: a second
write panics, exactly as the process global did (not downgraded to a warn).
- global.rs `set_global_region` / `get_global_region` keep their signatures and
forward to the current instance's context; the GLOBAL_REGION static is
removed. Single-instance: startup writes the bootstrap context (which the
ECStore adopts), so reads are unchanged.
Tests: set/get round-trip, two contexts hold distinct regions, and a second
set_region panics (fail-fast preserved).
Verification: cargo test -p rustfs-ecstore (9 instance-context tests green),
cargo clippy -p rustfs-ecstore --all-targets (clean), make pre-commit (pass).
Refs: backlog#939 (Phase 5, Slice 4). Stacked on Slice 3 (#4417).
The erasure-set finalize block only gated completion on failed_objects, so a pass with transient skips (unmet quorum, DiskNotFound, SlowDown, OperationCanceled) but zero hard failures was marked completed, its resume/checkpoint state cleaned up and the per-disk healing marker cleared. That violated the Transient invariant: the skipped versions were never re-healed on a later pass.
Broaden the finalize gate to failed_objects > 0 || skipped_objects > 0 and reuse the existing bounded-retry path (schedule_retry + checkpoint reset_for_retry, returning Err so the caller preserves state and keeps the healing markers). Transient conditions are deferred to the next heal cycle, never hot-retried in place.
Also fix the object/EC-decode heal success paths, which passed object_size as the failed positional arg to update_progress, corrupting objects_failed and the admin-visible success rate; pass 0 instead.
Add heal tests for the transient-skip finalize behavior and a progress test asserting a successful heal reports zero failures.
Refs rustfs/backlog#1033
extract_params_header copied every request/response header verbatim into
the maps that feed audit entries (requestQuery/responseHeader) and
notification events (req_params). Sensitive headers such as
Authorization and X-Amz-Security-Token were serialized in plaintext and
forwarded to external sinks (webhook/kafka/file), leaking long-lived
credentials.
Redact credential-bearing headers at this single chokepoint: the header
name is kept for correlation, but its value is replaced with the shared
REDACTED_SECRET placeholder. A case-insensitive sensitive-header list
covers authorization, x-amz-security-token, x-amz-content-sha256,
cookie, and set-cookie. Non-sensitive headers keep their existing
behavior.
Refs: https://github.com/rustfs/backlog/issues/963
Moka evicts entries by TTL, time-to-idle, and capacity-LRU without going through invalidate_object, so the per-object identity index (by_object) never dropped the evicted keys and grew without bound, bypassing the memory cap. Register an async eviction listener that removes truly evicted keys from the identity index, holding only a Weak reference to avoid an Arc cycle.
Refs rustfs/backlog#1031
* fix(scanner): scope long walk timeouts
* fix(scanner): bound IAM config walks
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
`LiveEventHistory::snapshot_since` resumed from the oldest retained event
whenever a consumer's cursor pointed before it, without signaling that the
in-between events had been evicted from the ring buffer. Cursor-based
consumers therefore assumed the returned batch was contiguous with their
cursor and silently lost events.
Add a `gap` flag to `LiveEventBatch`, set when the consumer's cursor is
older than the oldest retained sequence, so consumers can detect the loss
and trigger a full re-sync or alert. Add a regression test that fills and
evicts past the ring-buffer capacity and asserts a lagging cursor reports
`gap = true` (and that contiguous/caught-up cursors do not).
Refs rustfs/backlog#969
fix(ecstore): stop ListMultipartUploads from returning one upload past max-uploads (backlog#954)
The result-collection loop in `SetDisks::list_multipart_uploads` pushed an
upload into the page and only then checked `ret_uploads.len() > max_uploads`,
so each page returned up to max_uploads + 1 uploads and pointed
`next_upload_id_marker` at that surplus entry, violating the S3
ListMultipartUploads max-uploads contract.
Move the cap check before the push (MinIO-style) so a page never exceeds
max_uploads, and derive `is_truncated` from the post-marker cursor position
(`upload_idx < uploads.len()`) rather than comparing the page length against
the full listing length. The old length comparison mis-reported truncation on
the final page whenever a marker had skipped earlier entries, which prevented
marker-based pagination from terminating.
Add a regression test that starts more in-progress uploads than a page holds
and asserts a single page returns exactly max_uploads with a correct
next-marker, that the exact-boundary case is not marked truncated, and that
paginating one upload at a time enumerates every upload once with no loss,
duplication, or non-termination.
Incidental: add the missing `ctx` field to the `new_multipart_lock_test_store`
cfg(test) helper so the ecstore test build compiles after the #4413/#4437
merge collision.
Refs: https://github.com/rustfs/backlog/issues/954
Statement equality drives merge/dedup of policy statements. Omitting NotResource let semantically-distinct statements be treated as duplicates and dropped, which can shrink Deny coverage and escalate privileges. Compare not_resources too and add regression tests.
Refs rustfs/backlog#1028
`create_file` writes erasure shard / multipart part data through the page
cache, so the commit-point `sync_dir_files` fdatasync in `rename_data` has to
flush the whole shard's dirty pages (profiling: create_file avg 67.52ms vs
1.51ms nosync; BitrotWriter::write p95 40.86ms), and concurrent PUTs on one
device stall each other's writeback inside the rename critical section.
Add an opt-in Linux O_DIRECT streaming writer that reuses the direct-io read
infrastructure (statx DIOALIGN probe, aligned bounce buffer, supported latch,
one-time fallback log). Incoming bytes are staged into an aligned buffer;
full aligned batches and the shutdown tail are flushed on the blocking pool
(same offloading posture as the buffered writer it replaces). The
sub-alignment tail is written buffered after clearing O_DIRECT via fcntl
(MinIO's recipe). Shard bytes reach the device during the write phase, so the
unchanged commit-point `sync_dir_files` fdatasync degrades to a cheap
metadata/device FLUSH instead of flushing ~2MiB of dirty pages.
Gated by `RUSTFS_OBJECT_DIRECT_IO_WRITE_ENABLE` (default false), Linux-only,
and durability-neutral: `sync_dir_files` still fdatasyncs every shard at the
commit point, so the strict/relaxed/none durability tiers keep their exact
guarantees. Filesystems that reject O_DIRECT (tmpfs, overlayfs, 9p) latch the
path off and fall back to buffered writes; an O_DIRECT open/write EINVAL is
never surfaced as `InvalidInput` (which `to_file_error` maps to `FileNotFound`
and would masquerade as a missing shard, triggering a spurious EC rebuild).
Unit tests cover the env gate, the alignment/staging-capacity and tail-split
math, an end-to-end `create_file` round trip across sizes crossing the
alignment and multi-batch boundaries (buffered fallback on macOS / O_DIRECT on
a block device), and the writer state machine over a plain file on Linux. Real
O_DIRECT throughput on Linux NVMe (ext4/xfs) is deferred to the azure 4-node
bench per the issue's GO/NO-GO gate; the development host is macOS and cannot
exercise O_DIRECT.
Refs: rustfs/backlog#927, rustfs/backlog#936
Co-authored-by: heihutu <heihutu@gmail.com>
perf(ecstore): per-bucket durability tier overrides (HP-5 phase 2)
Let a bucket override the process-wide RUSTFS_DURABILITY_MODE with its own
strict/relaxed/none tier, stored as a durability.json extension entry in the
bucket metadata file and resolved at commit points via effective_durability.
System-critical buckets (.rustfs.sys, .minio.sys) can never carry an override
and stay pinned to strict; the legacy full-off switch keeps its historical
semantics and per-bucket overrides do not apply under it. Overrides are
published and cleared through the existing bucket metadata cache-invalidation
path, and an admin GET/PUT handler exposes the configuration. Default behavior
is unchanged: with no override a bucket follows the global mode, which defaults
to strict and stays byte-for-byte identical to before.
Refs: https://github.com/rustfs/backlog/issues/938, https://github.com/rustfs/backlog/issues/936
Co-authored-by: heihutu <heihutu@gmail.com>
fix(object-capacity): use u128 intermediate to stop sampling estimate overflow (backlog#1012)
The three sampling extrapolation sites in scan.rs computed
`overflow_sampled_bytes.saturating_mul(overflow_count) / sampled_count`,
multiplying in u64 before dividing. On very large disks the product exceeds
u64::MAX and saturates to a constant, so dividing by a sampled_count that grows
with disk size yields an estimate that monotonically shrinks — bigger disks
report less capacity. It only sets is_estimated, with no alert.
Extract a shared `estimate_overflow_bytes` helper that performs the
multiplication in u128, divides, then clamps to u64::MAX, so the intermediate
product can never overflow. It also guards division by zero with `.max(1)` even
though callers only enter the branch when sampled_count > 0.
Adds unit tests covering the realistic ~105 TB regression scenario (asserting
the inputs actually overflow u64 and the fixed estimate dwarfs the saturated
value), monotonic scaling across disk sizes, extreme-value clamping, small
non-overflowing inputs (unchanged), and the zero-sampled-count guard, all
compared against a u128 reference implementation.
Refs: https://github.com/rustfs/backlog/issues/1012
The MQTT target spawned its rumqttc event loop exactly once through a
`OnceCell`. When the loop exited on a fatal protocol error, the finished
`JoinHandle` stayed in the cell, so `init()` never respawned it. The target
then went permanently silent: `publish` calls kept enqueuing but nothing was
ever delivered, while the target still looked healthy.
Replace the one-shot spawn with a supervisor task. The `OnceCell` now guards
a single supervisor that runs one event-loop session at a time and, when a
session exits, rebuilds the client and event loop from the latest
`MqttOptions` after an exponential backoff (1s..30s). A session that connected
resets the backoff so a transient drop reconnects promptly; repeated immediate
failures back off to avoid a reconnect storm. Cancellation from `close()` is
handled by the supervisor dropping the in-flight session future, so no
per-session cancel channel is needed.
Rebuilding options per session also keeps TLS hot-reload working across
reconnects. The backoff policy is a pure function and the supervisor loop is
generic over the session runner, so both are unit tested without a broker:
one test asserts the session restarts repeatedly until cancelled, another that
a live session stops promptly on cancel.
Refs: https://github.com/rustfs/backlog/issues/972
fix(targets): disable HTTP redirects on webhook delivery client (backlog#974)
The webhook reqwest client used the default redirect policy (follows up to
10 redirects). Even though the configured endpoint is validated against
outbound-egress rules, a malicious or compromised endpoint could return a
3xx response to bounce the outbound request to an internal address (e.g. the
cloud metadata service at 169.254.169.254), bypassing that validation (SSRF).
- Set `.redirect(reqwest::redirect::Policy::none())` on the delivery client
so redirects are never followed.
- Treat a 3xx response as a delivery failure (error carries the status code)
instead of silently succeeding.
- Add a regression test that stands up a loopback server returning a 302 to
an internal metadata address and asserts the client surfaces the 3xx rather
than following it.
Refs: https://github.com/rustfs/backlog/issues/974
* fix(swift): replace assert!(false) with panic! in expiration_worker test
clippy::assertions_on_constants fails the swift clippy CI job under -D warnings.
* fix(ecstore): add missing ctx field to multipart lock test store
The multipart list-parts lock test (#4437) constructs ECStore without the
ctx field added by the Phase 5 InstanceContext work (backlog#939). Both
landed on main independently, leaving a semantic conflict that breaks the
ecstore lib-test build (E0063). Adopt the process bootstrap context,
matching the existing bootstrap_ctx() test in store/mod.rs.
fix(ecstore): validate erasure distribution values to avoid shuffle index panic (backlog#949)
The element values of `erasure.distribution` read from `xl.meta` were never
range-checked. `FileInfo::is_valid()` and `MetaObjectV1::valid()` only verified
`distribution.len()` and the `erasure.index` bound, not that each distribution
value is a valid 1-based slot in `[1, N]`. The metadata shuffle helpers then use
these values directly as `distribution[k] - 1` indices, so a corrupt or
adversarial `xl.meta` carrying a `0` (usize underflow) or a value greater than N
(out-of-bounds) triggers a panic in the shuffle path, turning bad-disk metadata
that erasure coding is meant to tolerate into a request/task crash.
Fix, two layers:
- Validate distribution values at metadata acceptance: `is_valid_distribution`
now requires the distribution to be a permutation of `1..=N` (correct length,
every value in range, no duplicates). `FileInfo::is_valid()` and
`MetaObjectV1::valid()` use it, so `find_file_info_in_quorum` rejects corrupt
metadata and it surfaces as a clean `ErasureReadQuorum` error instead of an
index path.
- Defensive indexing in the shuffle helpers (`shuffle_disks_and_parts_metadata`,
`_by_index`, `_by_index_owned`, `shuffle_parts_metadata`, `shuffle_disks`,
`shuffle_check_parts`): out-of-range distribution values are skipped via
`checked_sub(1)` + bounds-checked slot access instead of a bare `idx - 1`
index, matching the existing pattern in
`collect_inline_data_shard_fileinfos_by_index`.
Regression tests: `is_valid_distribution`/`is_valid`/`valid` reject distributions
containing `0`, values greater than N, duplicates, and wrong length while
accepting valid permutations; the shuffle helpers no longer panic on corrupt
distributions and preserve output length.
Refs: https://github.com/rustfs/backlog/issues/949
fix(notify): serialize persisted config read-modify-write to prevent lost updates (backlog#968)
The notify config write path performed a read -> modify -> write over the
persisted server config with no serialization: two concurrent updates could
both read the same base config, apply disjoint changes, and race their
full-config writes. The later write silently overwrote the earlier one,
losing updates (e.g. adding two targets concurrently could drop one) with no
error surfaced.
Guard the whole read -> modify -> write with a process-global tokio Mutex
(NOTIFY_CONFIG_RMW_LOCK). The persisted config is a single process-global
resource, so serializing the RMW makes concurrent updates apply in sequence
and every change is preserved. The lock is only acquired inside
update_server_config and is released before the in-memory reload, so it never
nests with the per-manager config RwLock and introduces no lock-ordering risk.
The RMW sequence is extracted into serialized_read_modify_write with injected
read/save operations so the exact production serialization path is exercised
by a regression test that concurrently adds 32 distinct targets and asserts
all of them survive.
Refs: https://github.com/rustfs/backlog/issues/968
fix(audit): propagate dispatch delivery failures instead of swallowing them (backlog#962)
AuditPipeline::dispatch and dispatch_batch accumulated per-target save()
errors, logged them, and then unconditionally returned Ok(()). When every
configured target failed the audit entry was lost outright (for store-backed
targets a failed save() means the event was neither delivered nor persisted
for replay), yet callers such as dispatch_audit_log saw success and assumed
the audit trail was intact. Audit is a compliance-critical path, so a total
delivery failure that reports success is a silent data-loss bug.
Both methods now distinguish three outcomes: all targets succeeded (Ok),
partial failure where at least one target accepted the event (log a warning
and return Ok, since the entry is not lost), and total failure where no
delivery succeeded while errors were recorded (record the failure metric,
log an error, and return Err(AuditError::Target) carrying the first target
error). This lets the caller react instead of assuming success.
Adds regression tests with a FailingTarget mock asserting that dispatch and
dispatch_batch return Err on total failure and Ok on partial failure.
fix(targets): surface truncated store batches instead of silently returning partial entries (backlog#967)
get_multiple deserializes a persisted batch by pulling item_count items
from a concatenated JSON stream. When the stream ended early (a truncated
or corrupted batch file), the partial-read branch only logged a warn! and
broke out of the loop, returning Ok(partial). Callers then treated the
batch as fully delivered, deleted the store entry, and permanently lost
the remaining events with no error.
Return StoreError::Deserialization as soon as the stream ends before
item_count items are read, so the caller keeps the entry on disk and can
retry rather than silently dropping events. The empty-file case already
returned an error and remains covered by this path.
Add a regression test that writes a 3-item batch, truncates the file at a
clean boundary after two items, and asserts get_multiple returns Err and
leaves the batch file in place.
Refs: https://github.com/rustfs/backlog/issues/967
* fix(audit): resolve AB-BA lock-order deadlock between registry and stream_cancellers (backlog#961)
`AuditSystem::runtime_status_snapshot` acquired `stream_cancellers.read()`
before `registry.lock()`, while every other path that holds both locks
(`clear_runtime_targets`, `AuditRuntimeFacade::replace_targets`) acquires
`registry` first. Under concurrency (e.g. admin status polling racing a
config reload) this AB-BA ordering could deadlock the whole audit control
plane with no self-recovery.
Reorder `runtime_status_snapshot` to acquire `registry` before
`stream_cancellers`, unifying the global lock order to
`registry -> stream_cancellers` across all double-lock paths, and add a
multi-threaded regression test that hammers both critical sections with a
timeout guard to fail fast if the ordering regresses.
Refs: https://github.com/rustfs/backlog/issues/961
* docs(audit): reword AB-BA to ABBA to satisfy typos check (backlog#961)
fix(notify): strip rustfs/minio internal metadata from event userMetadata (backlog#964)
Notification event construction only stripped keys prefixed with
`x-amz-meta-internal-`, which is not a prefix RustFS actually uses. Real
internal metadata lives under `x-rustfs-internal-*` / `x-minio-internal-*`
(xl.meta compat, incl. `x-minio-internal-server-side-encryption-*`) and
server-side-encryption details under `x-rustfs-encryption-*` /
`x-minio-encryption-*`. None of these were filtered, so they leaked into
`s3.object.userMetadata` sent to downstream notification targets
(webhook/MQ) — an information disclosure of SSE/replication/healing
internal state.
Filter via the shared classifiers `rustfs_utils::http::is_internal_key`
and `is_encryption_metadata_key` (case-insensitive, covering both key
flavors), while retaining the legacy `x-amz-meta-internal-*` strip for
backward compat. Genuine user metadata (`x-amz-meta-*`, content-type, ...)
is preserved unchanged.
Adds a regression test asserting both internal prefixes, the SSE internal
key, both encryption prefixes (incl. mixed-case) and the legacy prefix are
stripped while user keys survive.
Refs: https://github.com/rustfs/backlog/issues/964
houseme