Phase 5 Slice 9 (backlog#939): move the lifecycle expiry state and transition
state — two stateful service handles defined in bucket_lifecycle_ops — into the
per-instance InstanceContext, following the lazy-materialization pattern from
Slices 7-8.
- InstanceContext gains `expiry_state: OnceLock<Arc<RwLock<ExpiryState>>>` and
`transition_state: OnceLock<Arc<TransitionState>>`, each materialized lazily
on first access (both constructors are cheap — empty worker vecs/channels,
workers started later — so lazy init reproduces the eager static's
create-once-then-share).
- The lifecycle owner helpers `get_global_expiry_state` / `get_global_transition_state`
and the runtime-source handles `expiry_state_handle` / `transition_state_handle`
keep their signatures and route through the current instance's context; the
two statics (and the now-unused lazy_static import) are removed. The scanner
boundary and lifecycle-state arch guards still pass.
Tests: expiry/transition state are stable within an instance (same Arc) and
distinct across instances.
Verification: cargo test -p rustfs-ecstore (20 instance-context tests green),
cargo clippy -p rustfs-ecstore --all-targets (clean), make pre-commit (pass).
Refs: backlog#939 (Phase 5, Slice 9).
Each erasure block runs its Reed-Solomon encode through
tokio::task::block_in_place on the multi-threaded runtime. That parks the
worker and asks the scheduler to relocate other tasks, but the encode itself
is only ~110µs per 1MiB block (p99 ~542µs) — the profiling in backlog#932
flagged the scheduling disturbance as comparable to the compute it guards.
Call the encode closure inline on the multi-threaded runtime instead. The
CurrentThread (and any other) flavor keeps spawn_blocking so the sole executor
thread is never blocked and block_in_place's multi-thread-only requirement is
respected. Applied to both ingest paths (encode_block / Vec and
encode_block_bytes_mut / BytesMut); no change to encode output, quorum,
shutdown, or error handling.
Adds encode_works_on_multi_thread_runtime to cover the previously-untested
multi-threaded arm for both ingest paths, asserting streaming and batched
encode produce identical shard bytes.
This is the low-risk, correctness-neutral item that backlog#932's adversarial
verification recommended splitting out and doing first; the larger per-writer
pipeline restructure it belongs to stays gated on a Linux multi-disk baseline.
Refs: rustfs/backlog#932 (HP-11), rustfs/backlog#936
Co-authored-by: heihutu <heihutu@gmail.com>
* ci(perf): warp A/B relative-budget gate for the hotpath series
performance.yml only uploads a samply profile and a cargo-bench artifact with
no pass/fail, so a 26x-class write-path regression like #4221 or the GET perf
churn would sail through CI and only surface in customer load tests. This adds
the missing gate — the acceptance surface every queued HP change (#922/#923/
#925/#927/#930/#932) needs before its default can flip.
- scripts/hotpath_warp_ab_gate.sh: applies the relative budget to the
baseline_compare.csv that run_object_batch_bench_enhanced.sh already emits
(it computes deltas but never gates). A metric regressing past --fail-pct
fails, past --warn-pct warns; reqps/throughput are higher-is-better, latency
lower-is-better. --allow-regression downgrades a FAIL to an exempted WARN so
a deliberate correctness cost (e.g. #4221) is recorded, not blocked
(rustfs/backlog#935 correction 1). Unit-checked across pass/warn/fail/exempt.
- scripts/run_hotpath_warp_ab.sh: single-host baseline-binary vs candidate-
binary A/B over {put-4mib, get-4mib, mixed-256k} x {drive-sync on, off},
reusing the enhanced bench as the warp driver and the single-node local-disk
lifecycle. Has --dry-run; warp is assumed pre-installed as elsewhere in
scripts/. Drive-sync on/off keeps a sync-semantics change from being masked
by nosync numbers.
- .github/workflows/performance-ab.yml: nightly on main (post-merge detection)
plus opt-in pre-merge via the `perf-ab` label; `perf-deliberate-tradeoff`
runs the gate with --allow-regression; posts the gate table as a PR comment
and uploads the run. A Linux runner answers "do the macOS conclusions hold".
The gate logic is unit-validated with synthetic compare CSVs; the orchestrator
and workflow are shellcheck- and --dry-run-validated. The first real warp
measurement belongs on the Linux runner (no warp/multi-disk rig locally).
Refs: rustfs/backlog#935 (HP-14 warp A/B gate, item 4), rustfs/backlog#725
(cooled A/B harness precedent), rustfs/backlog#936
Co-Authored-By: heihutu <heihutu@gmail.com>
* ci(perf): pin upload-artifact, add external-cluster A/B mode + runbook
- Pin actions/upload-artifact to the repo-standard full-length SHA (# v6);
the previous @v4 float tripped the workflow-pin guard.
- Add an external deployment mode to run_hotpath_warp_ab.sh so warp can target
an already-running cluster instead of a throwaway single-node server:
--endpoint selects the cluster and --deploy-hook runs between phases to swap
in the phase's binary and drive-sync config (context passed via
HOTPATH_AB_PHASE / HOTPATH_AB_BINARY / HOTPATH_AB_DRIVE_SYNC). This maps onto
the team's ansible harness (cargo zigbuild -> ansible rustfs-manage --tags
stop,config,binary-copy,start -> warp).
- Restructure the matrix loop to bring a deployment up once per (phase,
drive-sync) and run all three workloads against it, instead of restarting per
workload — fewer server starts / cluster redeploys. Baseline medians are read
from a deterministic path, dropping the bash-4-only associative array so the
script (and its --dry-run self-check) runs on macOS bash 3.2 too.
- Add docs/operations/hotpath-warp-ab-runbook.md tying local mode, the ansible
external mode, and the budget/exemption together.
Verification: shellcheck clean; --dry-run in both modes prints the expected
4 deployments x 3 workloads and passes exactly 6 compare CSVs to the gate;
check_workflow_pins.sh, check_doc_paths.sh, and make pre-commit all pass.
Refs: rustfs/backlog#935, rustfs/backlog#936
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Refs rustfs/backlog#1006
- aggregate request traffic samples per type to avoid future counter collisions
- keep request schema and collector crate-internal until a production stats source exists
- preserve focused regression coverage for the internal request collector logic
Co-authored-by: heihutu <heihutu@gmail.com>
The rename_data commit sequence is the highest-risk durability path in the
store, and the HP-1/HP-4/HP-5 work (fsync coalescing, group commit, relaxed
durability tiers) all need a standing gate that proves a power loss mid-commit
can never leave a mixed or corrupt object. There was one graceful-rollback
failpoint but no crash-consistency coverage.
Add a deterministic harness that models a hard power loss — the commit
sequence stops dead at the armed step with no in-process cleanup — and then
reopens the disk to assert the raw on-disk state is coherent:
- Two pre-commit injection points, RenameDataCrashPoint::{AfterDataRename,
AfterBackupBeforeMetaCommit}, constructed at the real commit-path call sites
but gated so the production build compiles them to a const-false no-op
(mirrors the existing should_fail_before_old_metadata_backup pattern).
- A parameterized scenario over {strict, relaxed} durability x {both crash
points} x {overwrite, fresh object}: seed, stage a replacement, inject,
reopen, and assert the object reads back as exactly the old version (or does
not exist when there was no old version) with its data dir intact, never the
half-committed new one. The un-injected run asserts the commit makes the new
version visible. Relaxed is held to the same old-or-new invariant as strict
(only the durability window widens), which is exactly the property the
durability-relaxation work must not break (rustfs/backlog#878 hard rule).
- Wire an explicit `ecstore-crash-consistency` gate into the destructive
profile of run_ecstore_validation_suite.sh so it runs in the standing suite
(coordinates with the #878 destructive profile rather than building a
parallel one).
Production behavior is unchanged: the injection guards are no-ops outside
tests, and the existing rename_data rollback tests still pass.
Refs: rustfs/backlog#935 (HP-14 crash-consistency harness), rustfs/backlog#896
(test plan), rustfs/backlog#878 (ECStore validation suite), rustfs/backlog#936
Co-authored-by: heihutu <heihutu@gmail.com>
* perf(ecstore): read bitrot hash+data in one pass on the shard path
BitrotReader::read issued two reads per block: one read_exact for the
32-byte hash, then a separate loop for the shard data. On the streaming
disk reader (a raw tokio File whose every read is a spawn_blocking
round-trip) that is two dispatches per block. Since the on-disk layout is
a contiguous [hash][data] run, pull both in a single pass into a reused
scratch buffer and split afterwards, halving the per-block dispatch count
on the streaming path. The no-hash path still reads straight into the
caller buffer with no extra copy, and an in-memory Cursor (inline/mmap)
just does a slice copy.
All existing invariants are preserved: a short read of either the hash or
the data maps to UnexpectedEof before and independent of the hash check
(backlog#799 B2), and the hash-mismatch / InvalidData semantics are
unchanged.
Correctness-only change; the per-dispatch latency win is platform
dependent and its default reliance is left to the warp size-bucket
benchmark gate tracked by backlog#935, per the backlog#933 acceptance
note.
Refs: rustfs/backlog#933 (HP-12 item 2), rustfs/backlog#936
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs: reword bitrot test comment to satisfy typos check
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
A dirty-subset refresh scans only the dirty disks, so its raw
`CapacityUpdate` carries just that subset's `total_used`/`file_count`.
`update_capacity` recomputed the correct cluster-wide total into a local
variable and wrote it to the cache, but never wrote it back to the
`CapacityUpdate` that `refresh_or_join`/`spawn_refresh_if_needed` return
and publish to joiners. The admin blocking path (`resolve_admin_used_capacity`
-> `refresh_or_join_admin_disks(allow_dirty_subset=true)`) consumes
`update.total_used`, so a single dirty disk in an N-disk cluster made
admin StorageInfo report only the scanned subset's bytes (a large
transient undercount for that request and same-cycle joiners).
`CachedDiskCapacity` also dropped each disk's `file_count`/`is_estimated`,
so subset refreshes could not recompute a correct cluster file count and
would launder an estimated per-disk value into an exact cluster total.
Fix:
- `update_capacity` now reconciles `total_used`, `file_count`, and
`is_estimated` from the full per-disk cache and returns the corrected
`CapacityUpdate`.
- `CachedDiskCapacity` stores `file_count`/`is_estimated` per disk;
`file_count` sums the cache and `is_estimated` is the OR across disks.
- `refresh_or_join` and `spawn_refresh_if_needed` rebind their result to
the reconciled update so the leader return value and the value
published to joiners both carry cluster totals.
Tests: extend the subset-refresh test to assert the returned update's
reconciled `total_used`/`file_count`/`is_estimated`, and add a
`refresh_or_join` dirty-subset test asserting the leader returns the
merged cluster total (not the subset sum) and matches the cache.
Refs: https://github.com/rustfs/backlog/issues/1011
Phase 5 Slice 4 (backlog#939): move the S3 region — a write-once identity
scalar — out of the GLOBAL_REGION process static into the per-instance
InstanceContext, so two instances can serve different regions.
- InstanceContext gains `region: OnceLock<Region>` with `set_region()` /
`region()`. `set_region` keeps the write-once fail-fast contract: a second
write panics, exactly as the process global did (not downgraded to a warn).
- global.rs `set_global_region` / `get_global_region` keep their signatures and
forward to the current instance's context; the GLOBAL_REGION static is
removed. Single-instance: startup writes the bootstrap context (which the
ECStore adopts), so reads are unchanged.
Tests: set/get round-trip, two contexts hold distinct regions, and a second
set_region panics (fail-fast preserved).
Verification: cargo test -p rustfs-ecstore (9 instance-context tests green),
cargo clippy -p rustfs-ecstore --all-targets (clean), make pre-commit (pass).
Refs: backlog#939 (Phase 5, Slice 4). Stacked on Slice 3 (#4417).
The erasure-set finalize block only gated completion on failed_objects, so a pass with transient skips (unmet quorum, DiskNotFound, SlowDown, OperationCanceled) but zero hard failures was marked completed, its resume/checkpoint state cleaned up and the per-disk healing marker cleared. That violated the Transient invariant: the skipped versions were never re-healed on a later pass.
Broaden the finalize gate to failed_objects > 0 || skipped_objects > 0 and reuse the existing bounded-retry path (schedule_retry + checkpoint reset_for_retry, returning Err so the caller preserves state and keeps the healing markers). Transient conditions are deferred to the next heal cycle, never hot-retried in place.
Also fix the object/EC-decode heal success paths, which passed object_size as the failed positional arg to update_progress, corrupting objects_failed and the admin-visible success rate; pass 0 instead.
Add heal tests for the transient-skip finalize behavior and a progress test asserting a successful heal reports zero failures.
Refs rustfs/backlog#1033
extract_params_header copied every request/response header verbatim into
the maps that feed audit entries (requestQuery/responseHeader) and
notification events (req_params). Sensitive headers such as
Authorization and X-Amz-Security-Token were serialized in plaintext and
forwarded to external sinks (webhook/kafka/file), leaking long-lived
credentials.
Redact credential-bearing headers at this single chokepoint: the header
name is kept for correlation, but its value is replaced with the shared
REDACTED_SECRET placeholder. A case-insensitive sensitive-header list
covers authorization, x-amz-security-token, x-amz-content-sha256,
cookie, and set-cookie. Non-sensitive headers keep their existing
behavior.
Refs: https://github.com/rustfs/backlog/issues/963
Moka evicts entries by TTL, time-to-idle, and capacity-LRU without going through invalidate_object, so the per-object identity index (by_object) never dropped the evicted keys and grew without bound, bypassing the memory cap. Register an async eviction listener that removes truly evicted keys from the identity index, holding only a Weak reference to avoid an Arc cycle.
Refs rustfs/backlog#1031
* fix(scanner): scope long walk timeouts
* fix(scanner): bound IAM config walks
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
`LiveEventHistory::snapshot_since` resumed from the oldest retained event
whenever a consumer's cursor pointed before it, without signaling that the
in-between events had been evicted from the ring buffer. Cursor-based
consumers therefore assumed the returned batch was contiguous with their
cursor and silently lost events.
Add a `gap` flag to `LiveEventBatch`, set when the consumer's cursor is
older than the oldest retained sequence, so consumers can detect the loss
and trigger a full re-sync or alert. Add a regression test that fills and
evicts past the ring-buffer capacity and asserts a lagging cursor reports
`gap = true` (and that contiguous/caught-up cursors do not).
Refs rustfs/backlog#969
fix(ecstore): stop ListMultipartUploads from returning one upload past max-uploads (backlog#954)
The result-collection loop in `SetDisks::list_multipart_uploads` pushed an
upload into the page and only then checked `ret_uploads.len() > max_uploads`,
so each page returned up to max_uploads + 1 uploads and pointed
`next_upload_id_marker` at that surplus entry, violating the S3
ListMultipartUploads max-uploads contract.
Move the cap check before the push (MinIO-style) so a page never exceeds
max_uploads, and derive `is_truncated` from the post-marker cursor position
(`upload_idx < uploads.len()`) rather than comparing the page length against
the full listing length. The old length comparison mis-reported truncation on
the final page whenever a marker had skipped earlier entries, which prevented
marker-based pagination from terminating.
Add a regression test that starts more in-progress uploads than a page holds
and asserts a single page returns exactly max_uploads with a correct
next-marker, that the exact-boundary case is not marked truncated, and that
paginating one upload at a time enumerates every upload once with no loss,
duplication, or non-termination.
Incidental: add the missing `ctx` field to the `new_multipart_lock_test_store`
cfg(test) helper so the ecstore test build compiles after the #4413/#4437
merge collision.
Refs: https://github.com/rustfs/backlog/issues/954
Statement equality drives merge/dedup of policy statements. Omitting NotResource let semantically-distinct statements be treated as duplicates and dropped, which can shrink Deny coverage and escalate privileges. Compare not_resources too and add regression tests.
Refs rustfs/backlog#1028
`create_file` writes erasure shard / multipart part data through the page
cache, so the commit-point `sync_dir_files` fdatasync in `rename_data` has to
flush the whole shard's dirty pages (profiling: create_file avg 67.52ms vs
1.51ms nosync; BitrotWriter::write p95 40.86ms), and concurrent PUTs on one
device stall each other's writeback inside the rename critical section.
Add an opt-in Linux O_DIRECT streaming writer that reuses the direct-io read
infrastructure (statx DIOALIGN probe, aligned bounce buffer, supported latch,
one-time fallback log). Incoming bytes are staged into an aligned buffer;
full aligned batches and the shutdown tail are flushed on the blocking pool
(same offloading posture as the buffered writer it replaces). The
sub-alignment tail is written buffered after clearing O_DIRECT via fcntl
(MinIO's recipe). Shard bytes reach the device during the write phase, so the
unchanged commit-point `sync_dir_files` fdatasync degrades to a cheap
metadata/device FLUSH instead of flushing ~2MiB of dirty pages.
Gated by `RUSTFS_OBJECT_DIRECT_IO_WRITE_ENABLE` (default false), Linux-only,
and durability-neutral: `sync_dir_files` still fdatasyncs every shard at the
commit point, so the strict/relaxed/none durability tiers keep their exact
guarantees. Filesystems that reject O_DIRECT (tmpfs, overlayfs, 9p) latch the
path off and fall back to buffered writes; an O_DIRECT open/write EINVAL is
never surfaced as `InvalidInput` (which `to_file_error` maps to `FileNotFound`
and would masquerade as a missing shard, triggering a spurious EC rebuild).
Unit tests cover the env gate, the alignment/staging-capacity and tail-split
math, an end-to-end `create_file` round trip across sizes crossing the
alignment and multi-batch boundaries (buffered fallback on macOS / O_DIRECT on
a block device), and the writer state machine over a plain file on Linux. Real
O_DIRECT throughput on Linux NVMe (ext4/xfs) is deferred to the azure 4-node
bench per the issue's GO/NO-GO gate; the development host is macOS and cannot
exercise O_DIRECT.
Refs: rustfs/backlog#927, rustfs/backlog#936
Co-authored-by: heihutu <heihutu@gmail.com>
perf(ecstore): per-bucket durability tier overrides (HP-5 phase 2)
Let a bucket override the process-wide RUSTFS_DURABILITY_MODE with its own
strict/relaxed/none tier, stored as a durability.json extension entry in the
bucket metadata file and resolved at commit points via effective_durability.
System-critical buckets (.rustfs.sys, .minio.sys) can never carry an override
and stay pinned to strict; the legacy full-off switch keeps its historical
semantics and per-bucket overrides do not apply under it. Overrides are
published and cleared through the existing bucket metadata cache-invalidation
path, and an admin GET/PUT handler exposes the configuration. Default behavior
is unchanged: with no override a bucket follows the global mode, which defaults
to strict and stays byte-for-byte identical to before.
Refs: https://github.com/rustfs/backlog/issues/938, https://github.com/rustfs/backlog/issues/936
Co-authored-by: heihutu <heihutu@gmail.com>
fix(object-capacity): use u128 intermediate to stop sampling estimate overflow (backlog#1012)
The three sampling extrapolation sites in scan.rs computed
`overflow_sampled_bytes.saturating_mul(overflow_count) / sampled_count`,
multiplying in u64 before dividing. On very large disks the product exceeds
u64::MAX and saturates to a constant, so dividing by a sampled_count that grows
with disk size yields an estimate that monotonically shrinks — bigger disks
report less capacity. It only sets is_estimated, with no alert.
Extract a shared `estimate_overflow_bytes` helper that performs the
multiplication in u128, divides, then clamps to u64::MAX, so the intermediate
product can never overflow. It also guards division by zero with `.max(1)` even
though callers only enter the branch when sampled_count > 0.
Adds unit tests covering the realistic ~105 TB regression scenario (asserting
the inputs actually overflow u64 and the fixed estimate dwarfs the saturated
value), monotonic scaling across disk sizes, extreme-value clamping, small
non-overflowing inputs (unchanged), and the zero-sampled-count guard, all
compared against a u128 reference implementation.
Refs: https://github.com/rustfs/backlog/issues/1012
The MQTT target spawned its rumqttc event loop exactly once through a
`OnceCell`. When the loop exited on a fatal protocol error, the finished
`JoinHandle` stayed in the cell, so `init()` never respawned it. The target
then went permanently silent: `publish` calls kept enqueuing but nothing was
ever delivered, while the target still looked healthy.
Replace the one-shot spawn with a supervisor task. The `OnceCell` now guards
a single supervisor that runs one event-loop session at a time and, when a
session exits, rebuilds the client and event loop from the latest
`MqttOptions` after an exponential backoff (1s..30s). A session that connected
resets the backoff so a transient drop reconnects promptly; repeated immediate
failures back off to avoid a reconnect storm. Cancellation from `close()` is
handled by the supervisor dropping the in-flight session future, so no
per-session cancel channel is needed.
Rebuilding options per session also keeps TLS hot-reload working across
reconnects. The backoff policy is a pure function and the supervisor loop is
generic over the session runner, so both are unit tested without a broker:
one test asserts the session restarts repeatedly until cancelled, another that
a live session stops promptly on cancel.
Refs: https://github.com/rustfs/backlog/issues/972
fix(targets): disable HTTP redirects on webhook delivery client (backlog#974)
The webhook reqwest client used the default redirect policy (follows up to
10 redirects). Even though the configured endpoint is validated against
outbound-egress rules, a malicious or compromised endpoint could return a
3xx response to bounce the outbound request to an internal address (e.g. the
cloud metadata service at 169.254.169.254), bypassing that validation (SSRF).
- Set `.redirect(reqwest::redirect::Policy::none())` on the delivery client
so redirects are never followed.
- Treat a 3xx response as a delivery failure (error carries the status code)
instead of silently succeeding.
- Add a regression test that stands up a loopback server returning a 302 to
an internal metadata address and asserts the client surfaces the 3xx rather
than following it.
Refs: https://github.com/rustfs/backlog/issues/974
* fix(swift): replace assert!(false) with panic! in expiration_worker test
clippy::assertions_on_constants fails the swift clippy CI job under -D warnings.
* fix(ecstore): add missing ctx field to multipart lock test store
The multipart list-parts lock test (#4437) constructs ECStore without the
ctx field added by the Phase 5 InstanceContext work (backlog#939). Both
landed on main independently, leaving a semantic conflict that breaks the
ecstore lib-test build (E0063). Adopt the process bootstrap context,
matching the existing bootstrap_ctx() test in store/mod.rs.
fix(ecstore): validate erasure distribution values to avoid shuffle index panic (backlog#949)
The element values of `erasure.distribution` read from `xl.meta` were never
range-checked. `FileInfo::is_valid()` and `MetaObjectV1::valid()` only verified
`distribution.len()` and the `erasure.index` bound, not that each distribution
value is a valid 1-based slot in `[1, N]`. The metadata shuffle helpers then use
these values directly as `distribution[k] - 1` indices, so a corrupt or
adversarial `xl.meta` carrying a `0` (usize underflow) or a value greater than N
(out-of-bounds) triggers a panic in the shuffle path, turning bad-disk metadata
that erasure coding is meant to tolerate into a request/task crash.
Fix, two layers:
- Validate distribution values at metadata acceptance: `is_valid_distribution`
now requires the distribution to be a permutation of `1..=N` (correct length,
every value in range, no duplicates). `FileInfo::is_valid()` and
`MetaObjectV1::valid()` use it, so `find_file_info_in_quorum` rejects corrupt
metadata and it surfaces as a clean `ErasureReadQuorum` error instead of an
index path.
- Defensive indexing in the shuffle helpers (`shuffle_disks_and_parts_metadata`,
`_by_index`, `_by_index_owned`, `shuffle_parts_metadata`, `shuffle_disks`,
`shuffle_check_parts`): out-of-range distribution values are skipped via
`checked_sub(1)` + bounds-checked slot access instead of a bare `idx - 1`
index, matching the existing pattern in
`collect_inline_data_shard_fileinfos_by_index`.
Regression tests: `is_valid_distribution`/`is_valid`/`valid` reject distributions
containing `0`, values greater than N, duplicates, and wrong length while
accepting valid permutations; the shuffle helpers no longer panic on corrupt
distributions and preserve output length.
Refs: https://github.com/rustfs/backlog/issues/949
fix(notify): serialize persisted config read-modify-write to prevent lost updates (backlog#968)
The notify config write path performed a read -> modify -> write over the
persisted server config with no serialization: two concurrent updates could
both read the same base config, apply disjoint changes, and race their
full-config writes. The later write silently overwrote the earlier one,
losing updates (e.g. adding two targets concurrently could drop one) with no
error surfaced.
Guard the whole read -> modify -> write with a process-global tokio Mutex
(NOTIFY_CONFIG_RMW_LOCK). The persisted config is a single process-global
resource, so serializing the RMW makes concurrent updates apply in sequence
and every change is preserved. The lock is only acquired inside
update_server_config and is released before the in-memory reload, so it never
nests with the per-manager config RwLock and introduces no lock-ordering risk.
The RMW sequence is extracted into serialized_read_modify_write with injected
read/save operations so the exact production serialization path is exercised
by a regression test that concurrently adds 32 distinct targets and asserts
all of them survive.
Refs: https://github.com/rustfs/backlog/issues/968
fix(audit): propagate dispatch delivery failures instead of swallowing them (backlog#962)
AuditPipeline::dispatch and dispatch_batch accumulated per-target save()
errors, logged them, and then unconditionally returned Ok(()). When every
configured target failed the audit entry was lost outright (for store-backed
targets a failed save() means the event was neither delivered nor persisted
for replay), yet callers such as dispatch_audit_log saw success and assumed
the audit trail was intact. Audit is a compliance-critical path, so a total
delivery failure that reports success is a silent data-loss bug.
Both methods now distinguish three outcomes: all targets succeeded (Ok),
partial failure where at least one target accepted the event (log a warning
and return Ok, since the entry is not lost), and total failure where no
delivery succeeded while errors were recorded (record the failure metric,
log an error, and return Err(AuditError::Target) carrying the first target
error). This lets the caller react instead of assuming success.
Adds regression tests with a FailingTarget mock asserting that dispatch and
dispatch_batch return Err on total failure and Ok on partial failure.
fix(targets): surface truncated store batches instead of silently returning partial entries (backlog#967)
get_multiple deserializes a persisted batch by pulling item_count items
from a concatenated JSON stream. When the stream ended early (a truncated
or corrupted batch file), the partial-read branch only logged a warn! and
broke out of the loop, returning Ok(partial). Callers then treated the
batch as fully delivered, deleted the store entry, and permanently lost
the remaining events with no error.
Return StoreError::Deserialization as soon as the stream ends before
item_count items are read, so the caller keeps the entry on disk and can
retry rather than silently dropping events. The empty-file case already
returned an error and remains covered by this path.
Add a regression test that writes a 3-item batch, truncates the file at a
clean boundary after two items, and asserts get_multiple returns Err and
leaves the batch file in place.
Refs: https://github.com/rustfs/backlog/issues/967
* fix(audit): resolve AB-BA lock-order deadlock between registry and stream_cancellers (backlog#961)
`AuditSystem::runtime_status_snapshot` acquired `stream_cancellers.read()`
before `registry.lock()`, while every other path that holds both locks
(`clear_runtime_targets`, `AuditRuntimeFacade::replace_targets`) acquires
`registry` first. Under concurrency (e.g. admin status polling racing a
config reload) this AB-BA ordering could deadlock the whole audit control
plane with no self-recovery.
Reorder `runtime_status_snapshot` to acquire `registry` before
`stream_cancellers`, unifying the global lock order to
`registry -> stream_cancellers` across all double-lock paths, and add a
multi-threaded regression test that hammers both critical sections with a
timeout guard to fail fast if the ordering regresses.
Refs: https://github.com/rustfs/backlog/issues/961
* docs(audit): reword AB-BA to ABBA to satisfy typos check (backlog#961)
fix(notify): strip rustfs/minio internal metadata from event userMetadata (backlog#964)
Notification event construction only stripped keys prefixed with
`x-amz-meta-internal-`, which is not a prefix RustFS actually uses. Real
internal metadata lives under `x-rustfs-internal-*` / `x-minio-internal-*`
(xl.meta compat, incl. `x-minio-internal-server-side-encryption-*`) and
server-side-encryption details under `x-rustfs-encryption-*` /
`x-minio-encryption-*`. None of these were filtered, so they leaked into
`s3.object.userMetadata` sent to downstream notification targets
(webhook/MQ) — an information disclosure of SSE/replication/healing
internal state.
Filter via the shared classifiers `rustfs_utils::http::is_internal_key`
and `is_encryption_metadata_key` (case-insensitive, covering both key
flavors), while retaining the legacy `x-amz-meta-internal-*` strip for
backward compat. Genuine user metadata (`x-amz-meta-*`, content-type, ...)
is preserved unchanged.
Adds a regression test asserting both internal prefixes, the SSE internal
key, both encryption prefixes (incl. mixed-case) and the legacy prefix are
stripped while user keys survive.
Refs: https://github.com/rustfs/backlog/issues/964
* refactor(ecstore): add per-instance InstanceContext, migrate erasure setup type
Phase 5 of the global-singleton consolidation (backlog#939): begin moving
runtime identity state out of process globals so multiple ECStore instances
can coexist in one process. Isolation is carried by the object graph
(ECStore -> Sets -> SetDisks holding an Arc<InstanceContext>), not a
task-local, which does not propagate across the many internal tokio::spawn
boundaries in the data/background paths.
This first slice migrates the erasure setup type -- previously three
independent process-global bools -- into a single per-instance
RwLock<SetupType> that derives is_erasure / is_dist_erasure / is_erasure_sd,
removing a triple source of truth that could drift out of sync.
- New runtime::instance module: InstanceContext + process bootstrap context.
- The legacy free-function facade (is_erasure/update_erasure_type/...) keeps
its signatures and forwards to the current instance's context, falling back
to the bootstrap context before a store is published.
- ECStore gains a pub(crate) ctx field and setup_is_* accessors; its
constructors adopt the bootstrap context (never mint a fresh one) so startup
writes and post-construction reads share one cell -- single-instance
behavior is byte-for-byte unchanged.
Tests: erasure predicate derivation vs the legacy behavior, object-graph
carrier isolation across two ECStore instances, and bootstrap adoption.
Refs: backlog#939 (Phase 5, Slice 1), backlog#653 (item 8)
* refactor(ecstore): thread InstanceContext down the object graph (Phase 5 Slice 2) (#4415)
* refactor(ecstore): source the namespace lock manager per-instance (#4417)
refactor(ecstore): source the namespace lock manager per-instance (Phase 5 Slice 3)
Phase 5 Slice 3 (backlog#939): give each instance its own lock namespace by
sourcing SetDisks' lock manager from the instance context instead of the
process singleton. This removes the false cross-instance mutual exclusion (and
attendant ABBA risk) that a shared GlobalLockManager would cause once multiple
instances coexist.
- InstanceContext gains a `lock_manager: Arc<GlobalLockManager>`. `new()` mints
a fresh manager (independent per-instance); `bootstrap_ctx()` aliases the
process singleton via get_global_lock_manager(), so a single-instance
deployment keeps exactly one shared namespace.
- SetDisks::new sources `local_lock_manager` from `ctx.lock_manager()` (the ctx
it already adopts), not `runtime_sources::global_lock_manager()`. Single
instance: same Arc as before, so behavior is unchanged.
- Remove the now-unused `runtime_sources::global_lock_manager()` wrapper.
Tests: bootstrap lock manager aliases the process singleton; two fresh contexts
own distinct managers; a SetDisks' lock manager is the one from its context and
aliases the global singleton in a single-instance build.
Verification: cargo test -p rustfs-ecstore (10 Phase 5 + set_disk locking
regressions green), cargo clippy -p rustfs-ecstore --all-targets (clean),
make pre-commit (pass).
Refs: backlog#939 (Phase 5, Slice 3). Stacked on #4415 (Slice 2).
fix(lock): fence write commit on lock loss (backlog#899 Phase 2)
Phase 0+1 (#4388) made object write locks refreshable and marks the guard
lost when the heartbeat can no longer refresh a quorum, but does not act on
it. Under a partition a long write's lock can expire on an unreachable node
and be reclaimed, letting a third party re-acquire it; the original writer
keeps going and both commit -- a double write.
Expose the loss signal through NamespaceLockGuard::is_lock_lost() and
ObjectLockDiagGuard::is_lock_lost(), and fence the commit in put_object and
complete_multipart_upload: immediately before rename_data (the atomic commit
point), abort with a retryable NamespaceLockQuorumUnavailable (503) if the
lock was lost. In multipart the check precedes cleanup_multipart_path so a
lost lock leaves the upload intact and retryable. A write that already
reached rename_data Ok is durable and never aborted.
The loss criterion is unchanged (reacts to Phase 1's signal). Heal and the
long-GET read side are deferred follow-ups.
EventName::mask() recursed forever for the three internal leaf events
(ObjectRemovedAbortMultipartUpload, ObjectCreatedCreateMultipartUpload,
ObjectRemovedDeleteObjects): their discriminants fall past
LAST_SINGLE_TYPE_VALUE so mask() took the compound branch, but expand()
returns vec![*self] for them, so mask() called itself with the same
value and overflowed the stack.
Detect the self-expanding leaf case (expand() yields exactly self) and
give it a dedicated bit derived from its discriminant. Those bits sit
above the single-type bits, so they never collide with each other or
with any compound 'All' mask.
Add exhaustive regression tests: every variant's mask() terminates, the
three internal masks are non-zero and mutually distinct and disjoint
from the single-type bits, and Everything covers all single-type bits.
Refs: https://github.com/rustfs/backlog/issues/965
The FTPS and WebDAV authentication handlers compared the client-supplied secret
key against the stored secret with `String::eq`, which short-circuits on the
first differing byte. A network attacker who knows (or enumerates) a valid
access key can recover the secret key byte-by-byte via response-timing analysis;
neither path is rate limited.
Switch both to a constant-time comparison using `subtle::ConstantTimeEq`, the
same primitive the SFTP handler and `rustfs/src/auth.rs::constant_time_eq`
already use. `subtle` is added to the `ftps` and `webdav` feature dependency
sets (it was previously gated on `sftp` only).
Addresses GHSA-3p3x-734c-h5vx.
The internode RPC HMAC secret is derived from the S3 credential pair via
`derive_rpc_secret` when `RUSTFS_RPC_SECRET` is unset. The derivation uses the
secret key as the HMAC key, so when the default secret key (`rustfsadmin`) is in
effect the derived RPC secret is a fixed, publicly computable value. Any network
peer can then forge valid `x-rustfs-signature` headers and invoke internode RPC
routes (e.g. `read_file_stream`), bypassing S3 IAM entirely.
`normalize_rpc_secret` already rejected the literal default when it was supplied
directly, but `resolve_rpc_secret` still derived a secret from the default
credential pair. Make the derivation path fail closed: refuse to derive while
the default secret key is in effect, forcing operators to set `RUSTFS_RPC_SECRET`
(or a non-default `RUSTFS_SECRET_KEY`). A default access key paired with a
non-default secret key remains safe and is still allowed.
Addresses GHSA-68cw-96m3-h2cf (incomplete-fix follow-up to CVE-2026-45039).
Replication deletes dropped the S3 `versionId` query parameter for every
replication request, conveying the version only via the internal
x-*-source-version-id header. A generic (non-MinIO/RustFS) S3 target ignores
that header, so a version purge degenerated into a delete-marker creation while
the source still stamped VersionPurgeStatus=Complete — a silent divergence.
Only omit the query `versionId` when propagating a delete-marker CREATION
(the target must mint its own marker); version purges, delete-marker purges and
force deletes now address the exact version. Extracted into
resolve_delete_api_version_id with unit coverage.
Fixesrustfs/backlog#857
Disk-replacement heal previously repaired only the latest version of each
object and never enumerated objects whose latest version is a delete marker,
so old versions were left unrepaired on a replaced drive.
Switch heal enumeration from list_objects_v2 (latest-only) to
list_object_versions (every version incl. delete markers), thread the concrete
version_id into the existing per-version heal_object, and make resume
cursor-based instead of positional: an opaque (marker, version_marker) paging
token persisted in ResumeState, a length-prefixed injective per-version dedup
key, schema_version bumps (v2) migrated independently in each of the two
persisted files, and a retry that resets both managers together (fixing a
latent rescan-skips-everything defect). Adds a real-disk-wipe e2e regression
suite proving old versions and delete-marker-latest objects are physically
restored.
Fixesrustfs/backlog#918Fixesrustfs/backlog#919Closesrustfs/backlog#854
* fix(ecstore): make post-commit old data dir cleanup best-effort (backlog#898)
A write is authoritatively committed once rename_data returns Ok (the new
version is durable on >= write_quorum disks and immediately readable). The
subsequent reclamation of the now-dereferenced old object/<data_dir> is pure
space reclamation, yet commit_rename_data_dir propagated a below-quorum GC
failure via `?` into ErasureWriteQuorum -> 503, producing a false-negative
ACK for an already-persisted write. This is a deliberate divergence from
MinIO (erasure-object.go:1577), which couples the two; the divergence is
justified by durability semantics, not parity.
Changes:
- commit_rename_data_dir now returns a structured OldDataDirCleanup receipt
and never returns Err. Adds an old==committed-dir anti-misdelete guard and
a committed_data_dir parameter. Classification is extracted into pure
functions (classify_old_data_dir_cleanup / map_cleanup_join_result /
is_cleanup_not_found) so it is unit-testable. Task panic/cancel is mapped to
a non-ignored DiskError::other (never DiskNotFound), and not-found is
normalized to reclaimed.
- object.rs / multipart.rs consume the receipt instead of `?`. The result
reverts to Ok, so the invalidate_get_object_metadata_cache self-heal and the
capacity/compression accounting that a `?` early-return previously skipped
now run on the cleanup-failure path too.
- On residue, report_old_data_dir_cleanup emits leak metrics and enqueues an
object heal over the existing heal channel (disk-health signal replacing the
503). heal_object -> reclaim_orphan_data_dirs already reclaims unreferenced
local data dirs, closing the loop end to end.
- Adds rustfs_old_data_dir_* counters (attempted/reclaimed/leaked/below_quorum)
as the operator-visible backstop for leaked residue.
- Adds a test-only (#[cfg(test)]) delete fault-injection seam; in production it
inlines to a no-op None and has no behavioral effect.
Tests: pure-function A/C group + join-error mapping + actions decision; A5/A5b
real-disk guard/reclaim integration; end-to-end overwrite returning 200 while
old-data-dir cleanup fails. #864 rollback guard test remains green.
* fix(ecstore): resolve merge conflicts with origin/main in io_primitives.rs
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
fix(protocols): allow clippy type_complexity in test mock struct
The MockExpirationObjectBackend test struct uses a nested generic type
that triggers clippy::type_complexity. Add #[allow(clippy::type_complexity)]
since this is test-only code where the type is inherent to the mock design.
Co-authored-by: houseme <housemecn@gmail.com>
Merge the hotpath-rs wall-time instrumentation from the backlog#936 analysis worktree behind an opt-in 'hotpath' cargo feature, keeping the default build at zero overhead and zero dependency.
- hotpath is an optional dependency everywhere (dep:hotpath feature syntax); the default dependency tree contains no hotpath crate at all
- 40+ measurement points across S3 handlers, ECStore/SetDisks object and multipart ops, erasure encode/decode, bitrot, LocalDisk I/O, FileMeta codec, and HashReader
- attribute sites use #[cfg_attr(feature = "hotpath", hotpath::measure)]; async_trait bodies use per-crate hp_guard! macros (ecstore + rustfs bin); rio gates measure_block! behind hp_measure_block!
- feature chain: rustfs -> rustfs-ecstore -> rustfs-rio / rustfs-filemeta, each crate owning its own gate
- hotpath-alloc is intentionally not wired up (hotpath 0.21.x TLS panic on cross-thread guard drop under tokio, see backlog#935); mimalloc stays the unconditional global allocator
- docs/development/hotpath-profiling.md documents building, HOTPATH_* env vars, SIGTERM report flow, and how to reproduce the backlog#936 timing reports
Refs: https://github.com/rustfs/backlog/issues/935 (HP-14, item 2), https://github.com/rustfs/backlog/issues/936
Co-authored-by: heihutu <heihutu@gmail.com>
* feat(ecstore): add stripe-advance handles for deferred bitrot readers
Give DeferredObjectReader a shared pending state and expose a
DeferredReaderStripeHandle that advances the still-unopened source by whole
bitrot blocks using the same bitrot_encoded_range geometry the reader was
created with (identity mapping when hash_size == 0). This lets the GET decode
path open a parity shard aligned to the stripe where a data shard failed
instead of reading every parity shard on every stripe (backlog#923).
An already-opened (or failed) reader rejects the advance so callers retire it
rather than engage it out of alignment; bitrot verification after an advance
checks the advanced stripe's block against that stripe's stored hash.
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(ecstore): read only data shards on healthy lockstep GET behind opt-in gate
PR #4289's lockstep fix made every reconstruction-verifying GET read all
data+parity shards per stripe; the parity blocks are read, bitrot-hashed and
then discarded, a deterministic 2x read-bytes/IOPS/hash-CPU amplification on
healthy 2+2 objects (backlog#923). With the new opt-in gate
RUSTFS_GET_LOCKSTEP_DATA_SHARDS_ONLY_ENABLE=true (default: false, behavior
identical to main):
- read_lockstep keeps only the data slots engaged while the object is
healthy; parity slots stay unopened deferred readers.
- When a data shard is missing or dies at stripe k, parity readers are
engaged mid-object by advancing their deferred stripe handle to stripe k,
preserving the lockstep alignment invariant from backlog#832.
- Degraded stripes engage one parity beyond the decode quorum so
reconstruction verification keeps an extra source to check against
(erasure.rs only verifies when available > data shards); an engaged parity
reader that errors is retired for the rest of the object like any other,
and a parity reader that cannot be realigned is retired instead of being
read out of position.
- fill_deferred_bitrot_readers records stripe handles for deferred slots and,
gate-on only, swaps eagerly opened parity readers for unopened deferred
ones so they remain engageable mid-object; ready/error bookkeeping used by
quorum decisions is untouched.
- Both GET paths (legacy duplex via Erasure::decode_with_stripe_handles,
codec streaming via ParallelReader::with_deferred_parity_handles) carry the
handles from reader setup.
Short-read -> UnexpectedEof -> whole-object retirement and the
inconsistent-source rejection are unchanged in both gate modes; tests lock
the healthy-path data-shards-only call counts, the default read-all-shards
behavior, mid-object parity engagement for streaming and hash_size==0
formats, and mid-stream inconsistent-parity rejection.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
fix(lock): renew distributed locks via heartbeat; wire server refresh (backlog#899)
A held distributed write lock had a fixed 30s TTL and was never renewed, so
any operation exceeding it got its per-node lease reclaimed and stolen by a
contender, causing split-brain writes. This implements Phase 0 + Phase 1 of the
#899 design (Phase 2 abort-on-loss is deferred and tracked in code comments).
Phase 0 (server refresh wiring, P0):
- node_service handle_refresh was a no-op stub that parsed args then always
returned success=true. Extract a testable `refresh_lock` free function that
actually delegates to the node's lock backend. Not-found maps to
success=false with no error_info, so RemoteClient::refresh keeps its
Ok(resp.success) semantics and yields a real not_found signal to the
coordinator heartbeat. Without this, client heartbeats were silently no-ops.
Phase 1 (client heartbeat + safe interval + observability):
- DistributedLockGuard spawns a heartbeat that refreshes every per-client lease
on a derived interval; interval is derived without Duration::clamp
(entries<=1 or interval>=ttl => no spawn), fixing the sub-second-ttl panic.
- Add LockLostSignal: declare the lock lost when not_found exceeds
entries.len() - refresh_quorum; RPC errors are not counted (absorbed by the
ttl > interval margin). Expose is_lock_lost()/lock_lost() for observers.
- disarm(), release(), and Drop now abort the heartbeat before releasing so no
refresh races the unlock (refresh only extends, never creates, a lease).
- Reclaim path stays behaviorally unchanged but now warns with owner/resource/
lease age and records a metric (#698 scavenger preserved).
- Add DEFAULT_LOCK_REFRESH_INTERVAL, LockRequest.refresh_interval (serde default
for RPC back-compat) + builder, and lock lifecycle metrics.
Open questions adopt the design's documented defaults (marked TODO in code):
refresh not-found -> Ok(false)/error_info=None; lost-quorum base entries.len();
DEFAULT_LOCK_REFRESH_INTERVAL=10s.
Tests: heartbeat keepalive/quorum-loss/jitter/boundary/disarm (lock crate),
server refresh delegation (rustfs), and end-to-end survives-past-ttl plus
crashed-owner-reclaim regressions (namespace).
* perf(ecstore): move speculative tmp cleanup off the PUT hot path
On a successful PUT, rename_data has already moved the data dir out of the tmp workspace, so the delete_all(RUSTFS_META_TMP_BUCKET) at the end of SetDisks::put_object is a speculative no-op safety net. It was awaited inline on the response path, where profiling (backlog#924 / HP-3) showed the same-disk queueing behind fsync-heavy load turns a ~49us no-op into ~9ms average (p99 77ms, macOS F_FULLFSYNC amplified) added to every PUT.
Run that cleanup on a spawned task instead, keeping it as a real backstop (rename_data's remove_std only removes empty dirs and silently ignores failures). The failure path (quorum loss / rollback) keeps the cleanup inline so a failed PUT never returns with tmp shards still on disk. If the process dies before the spawned task runs, cleanup_stale_tmp_objects (24h expiry, 5-minute loop) reclaims the entry.
Scope note: ops/multipart.rs delete_all on RUSTFS_META_MULTIPART_BUCKET is intentionally untouched; it removes real leftovers and deferring it would widen CompleteMultipartUpload/Abort races.
Regression tests (hermetic SetDisks on formatted local disks, no global state): PUT success drains the tmp workspace (polling the spawned task), and PUT failure (missing bucket volume, rename_data quorum error after tmp shards were written) cleans the workspace inline before returning.
Ref: https://github.com/rustfs/backlog/issues/924
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): do not retry NotFound in reliable_rename_inner
reliable_rename_inner blindly retried the rename once on any error. A NotFound retry cannot succeed: nothing recreates the missing source or parent directory between attempts, so the second rename fails identically and speculative cleanup renames (e.g. move_to_trash on an already-removed tmp path) always paid for two syscalls.
Extract the retry decision into should_retry_rename: NotFound returns immediately, any other error keeps the existing single retry. This helper is shared by the rename_data commit path via rename_all, so behavior there is covered by a new rename_all success regression test alongside the retry-predicate tests.
Ref: https://github.com/rustfs/backlog/issues/924
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Step 1 of 4 for rustfs/backlog#922 (HP-1): write_all_internal previously
coupled file-content durability (fdatasync) with directory-entry
durability (parent dir fsync) behind a single bool. For tmp files that
are immediately renamed away, the tmp parent dir fsync contributes
nothing to crash consistency: the safe-rename recipe only needs file
content fdatasync -> rename -> fsync of the destination parent, because
the rename removes the tmp directory entry and a crash before the
rename means the PUT was never acknowledged.
Replace the bool with a SyncMode enum (None / FileAndDir / FileOnly)
and use FileOnly at exactly the two write-then-rename tmp write points:
the tmp xl.meta write in the non-inline rename_data path and the tmp
write inside write_all_meta. write_all_public (format.json etc.) and
the old-metadata rollback backup keep FileAndDir since those files stay
where they are written. The rest of the commit sequence (shard
sync_dir_files, rename, destination parent fsync) is untouched, and
behavior with drive sync disabled is unchanged.
This saves one directory fsync per disk per non-inline PUT (4 on a
4-drive set). Unit tests assert, via a test-only fsync-dir recorder,
that tmp write points no longer fsync their parent while the public
write point, the rollback backup, and the commit-rename destination
parent still do.
Co-authored-by: heihutu <heihutu@gmail.com>
Under a full nextest run on loaded machines the legitimately serialized
cross-disk commits in concurrent_resend_same_part_commits_one_generation
exceed the acquire deadline by themselves: observed at the 5s default,
at the 30s production default (#4370), and on CI even at 60s — which is
a hard ceiling, because fast_lock clamps every requested timeout to
MAX_ACQUIRE_TIMEOUT (60s) while the Timeout error still reports the
requested value, so raising the env override higher is a no-op.
Request the full 60s ceiling explicitly and cap the queue depth at three
concurrent resends, bounding the last waiter to two serialized commits
(~12s each on the slowest observed CI runner). Three resends still race
the streaming phase and contend on the commit lock, which is all the
generation-mixing regression (backlog#853) needs; every correctness
assertion is unchanged.
Co-authored-by: heihutu <heihutu@gmail.com>
fix(protocols): factor swift expiration mock result type into alias
The swift feature clippy matrix on main fails with clippy::type_complexity on the MockExpirationObjectBackend test struct introduced with the expiration worker tests, blocking CI for every open PR. Introduce a MetadataResult type alias in the test module; no behavior change.
Co-authored-by: heihutu <heihutu@gmail.com>
MetaObject::into_fileinfo indexed part_sizes[i]/part_actual_sizes[i] by
part_numbers.len() without checking the arrays are the same length,
unlike the adjacent part_etags/part_indices which are length-guarded.
decode_from pushes the three arrays independently and the xl.meta CRC
only covers bytes, so a CRC-valid but internally inconsistent xl.meta
(foreign writer / MinIO interop) triggers an out-of-bounds panic on the
GET/HEAD/LIST decode path.
Guard the three arrays for equal length and return Err(FileCorrupt) so a
divergent shard is skipped and quorum uses the other disks, instead of
panicking the request task. Cascade into_fileinfo to Result across its
callers, and fix io_primitives early-return to derive the version id from
the merged header and fall into the per-disk loop (single-disk survival +
heal). The 2118 merge-first path is left as a documented follow-up.
Refs backlog#900 (filemeta-01).
test(interop): real-MinIO read + migration parity, Phase 1/2 (backlog#580)
Capture authentic on-disk fixtures from MinIO RELEASE.2025-07-23 (a bucket with
versioning, object-lock, lifecycle, tagging, quota, a public policy, SSE-S3
encryption, a webhook notification target, and a replication rule, plus inline /
versioned / multipart objects and a delete marker) and prove RustFS reads and
migrates them losslessly:
- filemeta parses_real_minio_object_xlmeta: small inline, two-object-version +
delete marker, and multipart object xl.meta parse to the expected FileInfo.
- ecstore parses_real_minio_bucket_metadata_blob_without_loss: the MinIO
.metadata.bin msgpack decodes via the PascalCase field names and
parse_all_configs loads all ten config types present (policy, lifecycle incl.
<ExpiryUpdatedAt>, object-lock, versioning, tagging, quota, notification,
encryption/SSE-S3, replication incl. DeleteMarkerReplication /
ExistingObjectReplication) without loss.
- ecstore reads_minio_inline_bucket_metadata_via_bitrot: MinIO inlines an object
body as [HighwayHash256 32B][body]; RustFS's BitrotReader with HighwayHash256S
verifies and yields the exact blob (the "inline_data 前缀不同" is that prefix).
- ecstore migrates_real_minio_bucket_metadata_end_to_end: on a throwaway 4-drive
local ECStore, a real MinIO .metadata.bin seeded under .minio.sys is migrated
into .rustfs.sys byte-identically for every config, exercising the Phase 2
source adapter (MIGRATING_META_BUCKET = ".minio.sys") through the object layer.
All four run as ordinary crate tests (nextest CI). Phase 4 (MinIO re-reading a
RustFS drive) is documented as out of scope for one-way migration.
Refs rustfs/backlog#580
ENV_OBJECT_DIRECT_IO_ENABLE, DEFAULT_OBJECT_DIRECT_IO_ENABLE,
ENV_OBJECT_DIRECT_IO_THRESHOLD and DEFAULT_OBJECT_DIRECT_IO_THRESHOLD
were never read anywhere in the workspace. The real O_DIRECT read path
landed in PR #4365 uses RUSTFS_OBJECT_DIRECT_IO_READ_ENABLE /
RUSTFS_OBJECT_DIRECT_IO_READ_THRESHOLD (crates/ecstore/src/disk/local.rs).
Keeping the near-identically named dead knobs invites operators to set
the wrong variable and believe O_DIRECT is enabled.
Co-authored-by: heihutu <heihutu@gmail.com>
* refactor(ecstore): extract LocalIoBackend trait behind LocalDisk
Model the LocalDisk per-file I/O hot path as a LocalIoBackend trait
(pread_bytes / open_read_stream / open_full_read / open_write) and move
the existing method bodies verbatim into a default StdBackend. LocalDisk
holds Arc<dyn LocalIoBackend> and the DiskAPI methods forward to it.
This is a behavior-preserving refactor: no logic, error-mapping, metrics,
or cfg-branch changes. It creates the seam for an alternative
runtime-probed io_uring backend (rustfs/backlog#894) without touching
DiskAPI callers. Commit-point durability (fdatasync -> rename ->
fsync-dir in rename_data) deliberately stays outside the trait.
Add a differential test asserting all four read shapes return identical
bytes across page-boundary and file-tail ranges.
Tracking: rustfs/backlog#891 (parent rustfs/backlog#897)
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(ecstore): true O_DIRECT read path with per-disk graceful fallback (#4366)
* feat(ecstore): true O_DIRECT read path with per-disk graceful fallback
Implement a real O_DIRECT positioned read inside StdBackend::pread_bytes
(Linux only) and wire up the previously dead RUSTFS_OBJECT_DIRECT_IO_READ_*
knobs, which had zero call sites.
Open with rustix OFlags::DIRECT, probe the DIO alignment once per disk via
statx STATX_DIOALIGN (4096 fallback), read the aligned superset into an
alignment-allocated bounce buffer with a short-read loop, and slice out the
exact logical range so padding never reaches BitrotReader.
Any O_DIRECT failure falls back to the buffered read methods; EINVAL or
EOPNOTSUPP (tmpfs, overlayfs, 9p) latches the path off per disk with one
warning. O_DIRECT errors never surface: EINVAL maps to FileNotFound in
to_file_error and would trigger spurious EC rebuilds.
Default behavior is unchanged (knob off). macOS keeps F_NOCACHE.
Tracking: rustfs/backlog#892 (parent rustfs/backlog#897)
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(ecstore): add P1.5 benchmark gate harness for O_DIRECT reads (#4369)
* test(ecstore): add P1.5 benchmark gate harness for O_DIRECT reads
Add an ignored, Linux-only release-mode test (direct_read_bench_gate)
that measures DiskAPI::read_file_mmap_copy through a real LocalDisk with
cold-cache enforcement (fadvise DONTNEED between rounds), reporting
p50/p95/p99/mean latency, wall time, process CPU time, and throughput as
one JSON line for A/B diffing between the buffered baseline and the
O_DIRECT candidate selected by the production env knobs.
Content is verified byte-for-byte before any timing starts.
Tracking: rustfs/backlog#893 (parent rustfs/backlog#897)
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(ecstore): add concurrency dimension to P1.5 bench harness
Model the EC GET shape (FuturesUnordered over concurrent shard reads)
via RUSTFS_BENCH_CONCURRENCY (default 1, sequential as before). Needed
to evaluate blocking-pool pressure for the io_uring gate decision.
Tracking: rustfs/backlog#893
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* ci: retrigger after cancelled required check
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): use is_multiple_of in O_DIRECT bench cache-drop check
clippy's manual_is_multiple_of (rust 1.96) fails -D warnings on the benchmark helper's `done % file_count == 0`.
Verification:
- cargo clippy -p rustfs-ecstore --all-targets
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
A node could show OFFLINE in the console with zero logs explaining why
(rustfs/backlog#888, reported in rustfs#4304): the consecutive-failure
threshold crossing in handle_server_info_failure marked the peer offline
silently, the per-call warnings only existed on the observing node and
never named the transition, and the PeerRestClient offline flag plus its
background recovery monitor ran without any start/success log.
Logging-only change, no behavior change:
- handle_server_info_failure / handle_storage_info_failure: WARN
event="peer_marked_offline" exactly once at the threshold crossing
(later failures while already offline stay DEBUG), and DEBUG
event="peer_probe_failure" while returning cached state below the
threshold.
- update_server_info_cache / update_storage_info_cache: INFO
event="peer_recovered_online" when a probe succeeds after the peer had
been reported offline.
- PeerRestClient::mark_offline_and_spawn_recovery: WARN
event="peer_connection_marked_offline" when the offline flag is first
set (guarded by the recovery_running CAS so repeated failures do not
spam), and INFO event="peer_connection_recovered" with the attempt
count when connectivity is restored.
An "offline then back" episode now leaves a complete, correlatable
trace: N probe failures -> peer_marked_offline -> recovery monitor ->
peer_recovered_online.
Verification:
- cargo test -p rustfs-ecstore --lib (notification + peer_rest suites)
- make pre-commit
Ref: rustfs/backlog#888, rustfs#4304
Co-authored-by: heihutu <heihutu@gmail.com>
GitHub renders single newlines inside a paragraph as line breaks, so hard-wrapped PR/issue/discussion prose shows up with ugly mid-sentence breaks. Add a rule under Git and PR Baseline to keep each paragraph on one line.
concurrent_resend_same_part_commits_one_generation keeps failing on CI
with Lock(Timeout ... 5s) even after the fast-lock lost-wakeup fix
(NOTIFY_WAIT_CAP re-polling) landed — observed on rustfs#4365's Test and
Lint runs. Two independent causes exist and both are real:
1. the lost/stolen wakeup stall — fixed in fast_lock::shard; and
2. the six legitimately serialized cross-disk commits exceeding the
small 5s default acquire timeout under full-suite CI disk load.
The lost-wakeup fix reverted the earlier 30s test override on the
assumption that cause 1 was the whole story; CI shows cause 2 stands on
its own. Restore the temp_env override (production-default 30s) around
the concurrent section so the regression guard asserts the correctness
property (exactly one intact generation), not CI disk latency. The
NOTIFY_WAIT_CAP fix stays untouched.
Verification:
- cargo test -p rustfs-ecstore --lib concurrent_resend_same_part_commits_one_generation
Ref: rustfs/backlog#882
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(iam): load IAM bootstrap snapshot without namespace locks
IAM bootstrap (init_iam_sys -> load_all) read every config object with
default ObjectOptions (no_lock=false), so each read acquired a
distributed namespace read lock. Lock quorum is counted over cluster
nodes and unreachable peers are hard failures, so during a sequential
restart the very first read failed with
"Quorum not reached: required 2, achieved 0" and IAM could not come up
until enough peers' lock RPC surfaces converged - even when the storage
read quorum was already satisfiable (rustfs#4304).
Extend the startup contract from rustfs#4056 ("startup metadata I/O must
not require namespace locks") to the IAM bootstrap path:
- Introduce LoadMode {Locked, BootstrapNoLock} and plumb it through the
load_all chain (groups, users, policies, mapped policies and their
concurrent variants) down to the storage read options.
- load_all now performs all reads with no_lock=true; on-line
single-object loads via the Store trait keep locked semantics.
- Fail-closed behavior is unchanged: any loader error still aborts the
whole snapshot load.
Safety: config objects are atomic whole-object writes, so a lock-free
read only observes an old or a new value; staleness is bounded by the
existing periodic IAM reload. Listing (walk) never took namespace locks,
and maybe_schedule_lazy_rewrite stays a best-effort background task.
Verification:
- cargo test -p rustfs-iam --lib (153 passed, incl. new LoadMode tests)
- cargo clippy -p rustfs-iam --all-targets
- cargo check -p rustfs
- make pre-commit
Ref: rustfs#4304; tracking rustfs/backlog#884, rustfs/backlog#885
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(iam): sequential-restart regression test for lock-free bootstrap
Add an integration test reproducing the rustfs#4304 failure mode against
a real 4-disk temp-dir ECStore:
- Seed IAM group data in single-node mode, then flip the runtime into
distributed-erasure mode. new_ns_lock now builds a distributed lock
over the set's (empty) lock-client list, so every namespace-locked
read fails exactly like a sequential restart with unreachable peers
(lock quorum unavailable, storage read quorum healthy).
- Assert the locked load_group path fails in that state, while the
bulk snapshot load_all (no_lock plumbing from the previous commit)
succeeds, and the data survives intact once single-node mode is
restored.
Reverse-verified: temporarily switching load_all back to the locked
mode makes the test fail, so it genuinely guards the contract.
Verification:
- cargo test -p rustfs-iam --test iam_bootstrap_no_lock_test
- cargo test -p rustfs-iam --lib
Ref: rustfs#4304; tracking rustfs/backlog#886
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(iam): route test ECStore imports through ecstore_test_compat boundary
The new integration test imported rustfs_ecstore facade paths directly,
tripping three architecture migration rules. Move every ECStore import
behind crates/iam/tests/ecstore_test_compat/mod.rs (the sanctioned
test-compat pattern), and register that module as a reviewed test-only
global-facade boundary in check_architecture_migration_rules.sh: the
sequential-restart regression test needs api::global::update_erasure_type
to flip into distributed-erasure mode for lock-quorum fault injection.
Verification:
- ./scripts/check_architecture_migration_rules.sh
- cargo test -p rustfs-iam --test iam_bootstrap_no_lock_test
- make pre-commit
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(server): expose readiness blocking reason + rolling-restart runbook
Operators hitting the rustfs#4304 sequential cold start could not tell
from the outside why a node stayed unavailable. Three additions:
- The readiness gate's 503 now names the blocking dependency in both the
body ("Service not ready: waiting for storage_quorum") and a new
x-rustfs-readiness-pending header (storage_quorum | iam |
startup_finalization), derived from the current startup stage.
/health/ready already returned details + degradedReasons; this covers
the plain S3 requests that hit the gate.
- IAM bootstrap retry logs now carry an actionable `hint` field that
classifies the failure (storage read quorum vs lock quorum vs
uninitialized metadata) instead of only echoing the storage error.
- New docs/operations/rolling-restart.md runbook: correct rolling
restart procedure, sequential cold-start expectations (degraded ->
auto-recovery), readiness signal reference, and
RUSTFS_STARTUP_READINESS_MAX_WAIT_SECS guidance.
Verification:
- cargo test -p rustfs --lib -- hint_tests service_not_ready readiness_pending
- make pre-commit
Ref: rustfs#4304; tracking rustfs/backlog#887
Co-Authored-By: heihutu <heihutu@gmail.com>
* upgrade deps version and improve import
* feat(iam): notification-path cache refreshes read without namespace locks (#4368)
P3 step 1 of rustfs/backlog#884 (scoped down from full MinIO readConfig
alignment after review): cross-node notification handlers
(group/policy/policy-mapping/user) refresh the local IAM cache with
single-object reads that previously took distributed namespace read
locks. These refreshes are asynchronous, best-effort, and already
stale-tolerant (the periodic reload converges them), so a node-counted
lock quorum failure or lock RPC hiccup on a peer must not fail them —
the same rationale as the lock-free bootstrap load_all (rustfs#4304).
- Store trait: add load_user_no_lock / load_group_no_lock /
load_policy_doc_no_lock / load_mapped_policy_no_lock with defaults
forwarding to the locked variants, so existing implementations and
test mocks keep their behavior.
- ObjectStore overrides them via the existing LoadMode::BootstrapNoLock
plumbing. Deletions triggered by the handlers keep locked writes.
- manager.rs: the four *_notification_handler paths (8 call sites)
switch to the lock-free variants.
- Integration test: while the lock quorum is unavailable (DistErasure
with empty lockers), load_group_no_lock must succeed exactly where
the locked load_group fails.
Request-path loads (check_key, verify_temp_user_persistence) and admin
write-then-reload paths intentionally stay locked: load_user_identity
embeds expiry deletions, so those need the side-effect extraction
tracked in rustfs/backlog#884 before going lock-free.
Verification:
- cargo test -p rustfs-iam --lib (156 passed)
- cargo test -p rustfs-iam --test iam_bootstrap_no_lock_test
- make pre-commit
Ref: rustfs/backlog#884, rustfs#4304
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(server): drop unused iam_bootstrap_failure_hint import in tests
The hint tests live in their own hint_tests module with a local import;
the stale re-import in mod tests failed clippy's -D warnings on the
Test and Lint CI variants.
Verification:
- cargo clippy -p rustfs --all-targets
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Drop the [target.'cfg(target_os = "linux")'.dependencies] tokio
"io-uring" feature from 6 crates and the rustfs binary. Because
.cargo/config.toml enables --cfg tokio_unstable globally, this feature
switched every Linux build's file I/O onto tokio's io_uring runtime
backend. Restricted Linux environments (Docker default seccomp, gVisor,
proot, old kernels) reject io_uring_setup with EACCES/ENOSYS, which
tokio surfaced as PermissionDenied and RustFS reported as
DiskAccessDenied at startup.
Add scripts/check_no_tokio_io_uring.sh so the feature cannot silently
return: it fails on any tokio dependency line enabling "io-uring", while
still allowing a future application-level io-uring crate dependency.
Wire it into make pre-commit/pre-pr/dev-check and CI.
Tracking: rustfs/backlog#890 (parent rustfs/backlog#897)
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(migration): decrypt MinIO IAM & server config on drop-in migration
MinIO encrypts IAM identity/service-account files and the server config at
rest with a key derived from the root credentials. The drop-in migration
paths read those blobs from the legacy `.minio.sys` bucket and parsed them
as plaintext JSON, so any encrypted blob failed to parse and was silently
skipped with "incompatible format". This is why users migrating from MinIO
kept their buckets/objects/policies but lost users and access keys (#2212).
The IAM load path already knows how to decrypt these blobs (RustFS master
keys plus MinIO-compatible legacy keys derived from the root credentials),
but that logic lived behind a private method and was never used by the
migration paths. Expose it as `rustfs_iam::try_decrypt_iam_blob` and inject
it into both migration paths via a `LegacyBlobDecryptFn` callback (ecstore
cannot depend on the IAM crate, so the closure is wired in the binary crate).
When a blob cannot be decrypted the raw bytes are used as-is, preserving the
previous plaintext-only behavior with no regression.
Also improve object-layer migration observability without changing control
flow: `try_migrate_format` now distinguishes "no legacy format" (a normal
fresh install) from "legacy format present but incompatible", and the caller
logs a loud error before initializing a fresh format that would leave the
existing MinIO objects unreadable. Topology/version skip reasons are promoted
from debug to warn.
Fixes a pre-existing test isolation race by marking
`test_recovery_falls_back_to_default_config_when_blob_stays_corrupt` serial,
since it reads a process-wide env var toggled by a sibling test.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(migration): box FormatV3 in LegacyFormatOutcome to satisfy clippy
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(ecstore): stabilize concurrent multipart resend lock timeout
concurrent_resend_same_part_commits_one_generation spawns 6 same-part
resends whose cross-disk commits serialize on the per-uploadId commit
lock. Under the full nextest suite the parallel disk load pushes those
serialized commits past the small default lock-acquire timeout (5s),
producing a spurious `Lock(Timeout ...)` unrelated to the property under
test (observed on CI at 5.775s vs ~0.5s in isolation).
Raise RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT to the production default (30s)
for the concurrent-commit section via temp_env, so the regression guard
reflects correctness (exactly one intact generation) rather than disk
latency under CI load. The meaningful assertions are unchanged, and
#[serial] keeps the process-wide env override isolated.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(lock): bound fast-lock notification wait to prevent lost-wakeup stall
The real cause of the concurrent_resend_same_part_commits_one_generation
failures was a lost wakeup in the fast-lock slow path, not disk latency:
raising the acquire timeout to 30s only delayed the failure (it then timed
out at 30s), proving a genuine stall rather than overload.
In acquire_lock_slow_path a waiter that reaches the notification phase did a
single `timeout(remaining, wait_for_write())` spanning the whole acquire
budget, and treated that wait's elapse as a hard `Timeout`. But the release
path only notifies when `writer_waiters > 0`, so if the holder releases in
the gap after the waiter's `try_acquire` fails and before it registers as a
waiter, no notification (and no stored permit, since the pooled `Notify` is
gated) is produced. The waiter then blocks until the deadline even though the
lock is free and stays free — a spurious lock-acquire timeout. The shared
process-wide notify pool makes it worse: a wakeup can be consumed by a waiter
of a different lock hashing to the same slot.
Bound each notification wait (NOTIFY_WAIT_CAP = 50ms) and, on elapse, loop
back and re-`try_acquire` instead of returning `Timeout`; the deadline check
at the top of the loop is the single source of truth for timing out. A
lost/stolen wakeup now degrades to bounded re-polling (acquire within ~50ms
of the lock becoming free) instead of stalling for the whole timeout.
Correctness (mutual exclusion) is unchanged — acquisition still only happens
via `try_acquire_*`.
Add a regression test that reproduces the stall (holder + late waiter across
many keys): it times out without the fix and passes in ~1s with it. Revert
the earlier acquire-timeout workaround in the multipart test now that the
underlying stall is fixed, so it runs under the default timeout again.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(server): raise default HTTP/1.1 idle keep-alive timeout for proxies
In hyper's HTTP/1.1 stack `header_read_timeout` is armed as soon as the
connection is ready to read the next request head, so on a keep-alive
connection it also bounds how long an idle upstream connection may sit
before RustFS closes it. The previous 5s default meant RustFS FIN'd pooled
upstream connections while reverse proxies (Caddy ~2min, Nginx 60s) still
believed they were alive. The proxy then reused a dead socket and clients
saw `socket hang up` / connection reset, most visibly on non-idempotent
`PUT` uploads that proxies will not transparently retry (issue #3076).
Raise DEFAULT_HTTP1_HEADER_READ_TIMEOUT to 75s (above common proxy
idle-keepalive windows) and document the coupling. Still overridable via
RUSTFS_HTTP1_HEADER_READ_TIMEOUT for deployments that expose RustFS
directly to untrusted slow clients and want tighter slowloris protection.
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(operations): add reverse-proxy deployment guide
Document the request/body semantics RustFS expects from a proxy layer, the
idle keep-alive mismatch that causes `socket hang up` on large PutObject
writes, and known-good Caddy/Nginx configs plus Cloudflare caveats. Closes
the documentation gap called out in issue #3076 and consolidates the
scattered findings from #609/#934/#1492/#1766.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(object): bound stalled PutObject request-body reads with diagnostics
When a reverse proxy or CDN forwards a partial request body and then goes
silent without closing the connection, the inner body stream neither yields
more bytes nor reports EOF, so RustFS waited forever for bytes that never
arrive and the client saw a silent hang/abort (issue #3076). A short body
that ends with a proper EOF was already rejected promptly; the gap was the
no-EOF stall case.
Add a single-point request-body guard on the PutObject path that wraps the
incoming StreamingBlob with an inter-chunk read timeout. The timer resets on
every chunk, so slow-but-progressing uploads are unaffected; it only fires
after RUSTFS_HTTP_REQUEST_BODY_READ_TIMEOUT (default 300s, 0 disables) of
complete silence. On timeout it logs a structured `put_object_body_read_stalled`
event with received/expected byte counts and fails the read with
ErrorKind::TimedOut instead of hanging. remaining_length/size_hint are
forwarded so wrapping is transparent to downstream length handling.
Covered by unit tests for the stall path, length-preserving pass-through, and
the disabled (timeout=0) pass-through.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(server): make startup readiness wait configurable and raise default (#4264)
The startup runtime-readiness wait was a hardcoded 30s constant with no env
override. On slow multi-node cold starts (Docker/K8s/Synology NAS) this window
is shorter than the internal startup budgets it depends on — the endpoint
DNS-retry window (~90s) and the format-load retry loop (~100s worst case) — so
readiness times out and the node exits with
`startup readiness timed out after 30s: storage_ready=false, lock_quorum_ready=false`
before storage/lock quorum can converge, feeding the restart storm reported in
the issue.
- Add `RUSTFS_STARTUP_READINESS_MAX_WAIT_SECS` (default 120s), documented in
rustfs-config health constants.
- Resolve the wait at runtime via `startup_runtime_readiness_max_wait()`; a
value of `0` falls back to the default instead of timing out instantly.
- Repoint `STARTUP_RUNTIME_READINESS_MAX_WAIT` at the shared config default so
there is a single source of truth, and cover the getter with unit tests.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): stop peer/disk background monitors on graceful shutdown (#4264)
Long-lived peer health/recovery and remote-disk monitors are detached
`tokio::spawn` tasks that each hold a `tracing::Span` via `.instrument(..)` for
their whole lifetime. Nothing cancelled them at shutdown, so on the normal
return path the Tokio runtime was dropped while they were still alive and their
`Span`s were dropped during worker-thread thread-local-storage (TLS)
destruction. At that point `tracing-subscriber`'s fmt `on_close` can touch an
already-destroyed TLS slot and panic with
`cannot access a Thread Local Storage value during or after destruction`, which
escalates to a panic-during-panic abort (SIGILL / exit 132) — the crash
reported on Synology in issue #4264, amplified by the restart storm.
- Add `cluster::rpc::background_monitor` with a process-global shutdown token,
`spawn_background_monitor()` (races the monitor future against that token so
its span drops while the runtime is alive), and public
`shutdown_background_monitors()`.
- Route every span-holding peer_s3 / peer_rest / remote_disk monitor spawn
through `spawn_background_monitor` instead of `tokio::spawn(..).instrument()`.
- Expose `rustfs_ecstore::shutdown_background_monitors()` and call it from the
graceful shutdown sequence (right after `ctx.cancel()`, before runtime
teardown) via the `storage_api` compatibility boundary.
Existing recovery-probe span-context tests still pass, confirming log
correlation is preserved.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
test(e2e): add SHA256 verify-on-write & HEAD-checksum regression for #4341
Issue #4341 reported (on beta.8) that RustFS accepted mismatched S3
SHA256 checksums on PutObject (HTTP 200 instead of 400 BadDigest) and
did not return ChecksumSHA256 on HeadObject with ChecksumMode=ENABLED.
Both behaviors are already correct on current main, but the existing
e2e coverage only asserted the happy path (upload succeeds + GetObject
content matches). It did not encode the issue's two acceptance criteria,
so a future regression would go undetected.
Add two focused e2e tests that assert exactly the issue's contract:
- test_put_object_rejects_mismatched_sha256: a body that does not match
the declared x-amz-checksum-sha256 must be rejected with BadDigest and
must not be stored.
- test_head_object_returns_stored_sha256: after a correct upload,
HeadObject with ChecksumMode=ENABLED must return the stored base64
SHA256 digest.
Both tests pass against a real rustfs server (mismatch -> HTTP 400
BadDigest verified).
Co-authored-by: heihutu <heihutu@gmail.com>
* perf(ecstore): replace linear merge scan with k-way heap and drop clone-to-parse (backlog#874 backlog#875)
merge_entry_channels advanced the k-way merge with a linear scan over all
channel heads (O(entries x channels)) and allocated two fresh Strings per
pairwise comparison via path::clean. Every step also cloned MetaCacheEntry
values, including entry.clone().xl_meta() clone-to-parse calls.
- Introduce MergeHead with a cached cleaned name (allocated only when the
raw name is not already clean) and drive the merge with a BinaryHeap of
boxed heads: O(log channels) per entry, allocation-free comparisons.
- Move entries through the merge instead of cloning; the winner is sent
without an intermediate copy.
- Remove the dead merge_file_meta_versions block: it only ran for
prefix-dir groups whose entries have empty metadata, so xl_meta() always
failed; cross-drive version merging happens in the resolve path.
- Keep legacy same-name semantics (dir groups collapse, objects shadow
prefix dirs, later object candidate wins) and add regression tests for
interleaved ordering, dir/object precedence, uncleaned-name grouping,
and prefix-dir collapse.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): honor ascending versions_sort in ListObjects walk (#4348)
* fix(ecstore): honor ascending versions_sort in walk and document ordering invariant (backlog#876)
The walk loop carried a bare `//TODO: SORT` inside the
`WalkVersionsSortOrder::Ascending` branch, so the requested ascending
order was silently ignored and versions streamed newest-first (the raw
FileMeta order). WalkOptions defaults to Ascending, so every default
walker -- notably replication resync, which replays versions and needs
oldest-first to preserve the version-stack order -- received the exact
opposite of the contract.
FileMeta maintains versions newest-first (sort_by_mod_time is
descending) and into_file_info_versions preserves that order, so
ascending emission is the exact reverse of file_info_versions output.
Reverse in place when ascending is requested and add a regression test
locking the newest-first invariant plus the reversal contract.
Key-ordering audit result (no gap found): per-disk walkers emit sorted
streams, merge_entry_channels performs an ordered k-way merge, and
gather_results only filters by marker/limit, so ListObjects key order is
guaranteed upstream and needs no post-sort.
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(ecstore): enable GET metadata early-stop by default (#4349)
* perf(ecstore): enable GET metadata early-stop by default with env opt-out (backlog#872)
The metadata early-stop fanout (read_all_fileinfo_early_stop) has been
implemented and instrumented for a while but stayed behind an opt-in
flag, so default GETs always waited for every disk to answer the
metadata read even after quorum agreement was reached.
Flip RUSTFS_GET_METADATA_EARLY_STOP_ENABLE to default-on. The gate stays
conservative: should_allow_metadata_early_stop only admits metadata-only
reads (read_data=false) without version_id, healing, or free-version
requirements, everything else falls back to the full-wait fanout, and
setting the env var to false restores the old behavior entirely. The
version-aware gate (RUSTFS_GET_METADATA_VERSION_EARLY_STOP_ENABLE)
remains opt-in because versioned reads carry a higher stale-selection
risk profile.
Also replace the stale "optimize concurrency" TODO in
get_object_fileinfo with a pointer to the early-stop implementation and
add regression tests for the new default plus the explicit opt-out path.
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(ecstore): lazily construct codec streaming multipart readers (#4350)
* perf(ecstore): lazily construct codec streaming multipart part readers (backlog#871)
get_object_decode_reader_with_fileinfo opened shard readers for every
part of a multipart object before returning the streaming reader, so
TTFB paid for parts x disks file opens up front and an early client
disconnect wasted the setup work for every unread part.
Replace the eager loop with LazyMultipartCodecStreamingReader: the first
part is still built eagerly so the dominant fallback conditions (missing
shards / read quorum) are detected before any byte is streamed and the
whole request can fall back to the legacy duplex path exactly as before.
Each subsequent part is built on demand -- when the previous part hits
EOF -- via a spawned task handle owned by the reader; dropping the
reader aborts an in-flight build so disconnects stop all further IO.
If a later part hits a fallback condition mid-stream (a shard vanished
after the request started), the reader surfaces an explicit read error
with a pipeline-failure metric instead of silently degrading; the
client's retry then detects the condition on the eager first-part setup
and takes the legacy path cleanly.
Adds unit tests for in-order streaming across lazy boundaries, deferred
construction (no build when the client stops within part 1), and the
mid-stream fallback error path.
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(ecstore): prefetch next multipart part reader setup during decode (#4351)
* perf(ecstore): prefetch next multipart part reader setup during decode (backlog#870)
get_object_with_fileinfo processed multipart parts strictly serially:
the next part's bitrot reader setup (file opens + read-quorum wait
across all disks) only started after the current part finished
decoding, so large multipart reads paid full setup latency between
every part.
Overlap the two stages with a depth-one pipeline: right after the
current part's readers are obtained, the next part's setup is spawned
(shared inputs behind Arc) and joined when the loop reaches that part.
The shared setup_multipart_part_readers helper keeps stage-duration
metrics semantics identical for both paths; a failed or stale prefetch
falls back to the synchronous setup, and the PrefetchedReaderSetup
guard aborts the in-flight task on error returns, early breaks, or
caller drop so disconnects stop background disk IO.
Gate: RUSTFS_GET_MULTIPART_READER_SETUP_PREFETCH (default on, env
opt-out). Adds a three-part end-to-end read test covering the prefetch
hit path and cross-part content ordering.
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(ecstore): move FileInfo through GET shuffle instead of cloning (#4352)
perf(ecstore): move FileInfo entries through the GET shuffle instead of cloning (backlog#873)
shuffle_disks_and_parts_metadata_by_index deep-cloned every valid
FileInfo (parts, erasure info, metadata map) once per disk on each GET.
Add an ownership-taking variant that runs the same by-index consistency
check as a read-only first pass and then moves entries into their
shuffled slots with mem::take, and switch get_object_with_fileinfo to
it -- that call site already owned the parts metadata vector. Disk
handles are Arc clones and stay cheap.
Scope notes from the backlog#873 audit:
- get_object_fileinfo's disks.clone() stays: DiskStore is Arc<Disk>, so
the clone is per-slot refcounting and correctly avoids holding the
RwLock read guard across the metadata fanout awaits.
- get_object_decode_reader_with_fileinfo keeps the borrowing shuffle:
its caller must retain files/disks for the legacy fallback path, so an
owned variant would just shift the same clone upstream.
- The metadata-cache hit path still clones parts_metadata; sharing the
cached entry via Arc changes the read-path return types and is left
as a follow-up.
Equivalence tests cover both the by-index placement and the mod-time
fallback against the borrowing variant.
Co-authored-by: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(ecstore): gate merge emission on cleaned key and clear clippy redundant_clone
Address review + CI findings on the ListObjects/GET optimization PR:
- merge_entry_channels gated emission on the raw entry name while the heap
orders by the cleaned key, so entries whose cleaned order and raw byte order
disagree (e.g. redundant slashes) could be dropped. Gate on the same cleaned
sort key the heap uses; add a regression test (`a//c` after `a/b`).
- Drop three redundant `.clone()` calls in test code flagged by
clippy::redundant_clone (owned-shuffle equivalence tests and the walk
ascending-versions contract test) that failed the CI clippy gate.
- Document the known mid-stream fallback limitation of the opt-in multipart
codec streaming reader (default off) and mark the in-place per-part legacy
degradation as a follow-up.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): force full metadata fanout for object tagging writes (backlog#872)
put_object_tags reads the object fileinfo with read_data=false and then
writes the updated tags to the online-disk set that read returned. With
metadata early-stop enabled by default, that read now returns as soon as
read quorum is reached, so the online-disk set is only a read-quorum
subset. Writing tags to that subset fails write quorum -> ErasureWriteQuorum
-> S3 SlowDown, which is exactly the s3-tests tagging failures
(PutObjectTagging/DeleteObjectTagging, reached max retries).
Thread a caller-controlled `allow_early_stop` gate through
read_all_fileinfo_observed/_inner and add get_object_fileinfo_gated;
put_object_tags calls it with allow_early_stop=false so the metadata read
does the full quorum fanout and returns the complete online-disk set as
the write target. Pure-read callers (GET/HEAD/tag read) keep the
early-stop fast path unchanged.
Extract metadata_early_stop_permitted() as the single gate and add a unit
test locking the invariant: caller opt-out (and observe=false, and data
reads) never early-stop even with the env flags on.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* feat(ecstore): reclaim orphaned data dirs during heal (backlog #3231/#3191)
Pre-#3510, an unversioned overwrite leaked one UUID-named data dir per PUT.
The write path now cleans up going forward, but pre-existing strays stay on
disk forever: heal's dangling logic only removes whole objects whose data is
missing, never surplus data dirs of an otherwise-healthy object. That leaked
space is what eventually made ListObjects scan tens of GB and time out.
Add SetDisks::reclaim_orphan_data_dirs, a fail-closed, quorum-safe sweep that
mirrors purge_orphan_dir_object:
- referenced set is the UNION of get_data_dirs() across every online replica's
xl.meta, so a dir named by any replica is kept;
- if a disk holds the object dir but its xl.meta is missing or unparseable the
object is treated as degraded and nothing is removed;
- only UUID-named subdirectories are ever considered.
Wire it into heal_object's healthy tail (under the object write lock) as a
best-effort step so the background heal scanner and admin heal recover the
leaked space automatically; a reclaim failure never fails the heal.
Covered by four unit tests: removal of an unreferenced dir, no-op when all dirs
are referenced, fail-closed abort on missing metadata, and cross-replica union
preservation.
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(ecstore): fix typo unparseable -> unparsable
Satisfies the repository typos CI check.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(ecstore): fsync inline rename_data rollback backup
The inline branch of LocalDisk::rename_data writes the previous version's
xl.meta rollback backup (<old_data_dir>/xl.meta.bkp) with a bare
std::fs::write, fsyncing neither the file nor its new directory entry, even
when drive_sync_enabled() is true. The same closure already syncs the new
xl.meta and the commit rename directory, and the non-inline branch persists
its backup durably.
That backup is the sole restore source for delete_version(undo_write=true)
on a set-level write-quorum failure. With sync enabled, an inline overwrite
plus power loss plus a quorum failure that triggers undo_write could restore
a lost or truncated backup and fail to roll back to the prior committed
object.
Write the inline backup via OpenOptions + write_all and, when sync is
enabled, sync_data() it and fsync_dir_std its parent, mirroring the
non-inline branch. Extend the inline backup test to assert the .bkp contents
equal the previous metadata bytes.
Refs backlog#868 (disk-durability-01).
* fix(ecstore): preserve to_file_error mapping on inline backup write
Address review: the rewritten inline rollback-backup write must keep the
same DiskError classification as the previous std::fs::write(...).map_err(
to_file_error)? call. Without it, io errors (PermissionDenied/NotFound/
StorageFull/...) would surface as DiskError::Io(_) instead of the specific
DiskError variants (FileAccessDenied/FileNotFound/DiskFull/...), since
From<io::Error> for DiskError only recovers variants that were wrapped via
to_file_error. Map open/write_all/sync_data/fsync_dir_std through
to_file_error, matching write_all_internal.
---------
Co-authored-by: houseme <housemecn@gmail.com>
The xl.meta version header `signature` was hardcoded to `[0,0,0,0]` on the
write path (`From<FileMetaVersion> for FileMetaVersionHeader`), and the
existing `get_signature` implementations only hashed `version_id` + `mod_time`
(+ `size`), so two versions that share a version_id and mod_time but differ in
body (e.g. a PutObjectTagging that reached only some disks) produced identical
headers. Such divergence was undetectable and unhealable in the strict merge
path (backlog#861, B12).
Compute a real signature over the full version body, mirroring MinIO's
`xlMetaV2Version.getSignature` semantics:
- Fill the signature on the write path via the `From` impl (covers
add_version / set_idx / update_object_version).
- `MetaObject::get_signature` / `MetaDeleteMarker::get_signature` now clone the
body, zero the per-disk `erasure_index`, fold `meta_user` / `meta_sys` in
with order-independent hashes (msgpack map order is not stable across disks),
marshal the rest and xxh64 it, then fold 64->32 bits. Empty vs all-empty
PartETags are normalized alike.
- Invalid/bodyless versions return an `err` sentinel rather than all-zero, so
they never collide with a real all-zero legacy signature.
Backward compatibility: the decode path preserves stored signature bytes, and
the normal (non-strict) merge path already zeroes signatures before grouping,
so existing all-zero xl.meta stays consistent across disks. Only strict
merge/heal compares signatures, where legacy data is uniformly zero (no split)
and genuine partial-write divergence is now correctly detected.
Adds unit tests covering: non-zero signature on write, erasure_index
independence, tag/metadata/size divergence detection, map-order independence,
PartETags normalization, delete-marker divergence, and the err sentinel.
* fix(ecstore): make delete_volume non-recursive by default to prevent bucket-heal wipe (backlog#799 B1)
`delete_volume` unconditionally `remove_dir_all`'d the whole bucket tree, and the
bucket-heal "remove" branch called it fire-and-forget on every local disk. A
mis-classified "dangling" bucket (or a non-force S3 DeleteBucket on a populated
bucket) was therefore recursively wiped — a potential whole-bucket data loss.
The `VolumeNotEmpty` -> recreate/`BucketNotEmpty` handling already present in
both delete_bucket paths was dead code because the primitive never refused.
Add an explicit `force_delete` flag to `DiskAPI::delete_volume` and default the
non-force path to a non-recursive `remove_dir` (rmdir), which fails atomically
with `VolumeNotEmpty` if the bucket still holds any object data. Only an explicit
force delete (S3 force bucket delete) removes recursively. Mirrors MinIO's
`xlStorage.DeleteVol` (`Remove` vs `RemoveAll`).
- Trait + all impls (local behavior, dispatch, disk_store, remote RPC) take the
flag; the gRPC `DeleteVolumeRequest` gains a `force` field (proto3 default
false → old peers get the safe non-recursive behavior on rolling upgrade).
- Heal remove branch passes `false` and no longer discards the result: a
`VolumeNotEmpty` refusal is logged (the bucket is not dangling) instead of
wiping data.
- Both `delete_bucket` paths pass `opts.force`, activating the previously-dead
`VolumeNotEmpty` -> `BucketNotEmpty`/recreate handling (correct S3 semantics).
Adds a regression test: non-force delete of a non-empty bucket returns
VolumeNotEmpty and preserves the data; force delete removes it.
Design converged by two independent expert reviews (MinIO-fidelity +
defense-in-depth) referencing MinIO xl-storage.go. Refs backlog#799 (B1),
issue rustfs/backlog#850. The safety expert's deeper hardening (typed
capability instead of a bool, trash-instead-of-in-place for force, quorum
re-verification of dangling) is noted on #850 as follow-up.
* fix(ecstore): reword 'mis-classified' -> 'misclassified' to satisfy typos (backlog#799 B1)
* fix(rustfs): thread force_delete through StorageDiskRpcExt::delete_volume + test literal (backlog#799 B1)
fix(filemeta): resolve P2 core-storage audit items B15/B16/B18 (backlog#863)
Fixes the three remaining actionable items from the core-storage reliability
audit (rustfs/backlog#863). All three are metadata-layer correctness defects in
the filemeta crate.
B15 — version sort tie-break consistency
`FileMeta::sort_by_mod_time` and the delete path used a hand-rolled comparator
whose version_type tie-break for equal mod_time was the OPPOSITE of the
canonical `FileMetaVersionHeader::sorts_before` (used by the metacache merge
and latest-version selection). At equal mod_time it sorted a delete marker
before its object, so the per-disk read path could treat an object as deleted
while the quorum-merge path treated it as present. Both sort sites now derive
their ordering from `sorts_before`, matching MinIO (object-first).
B16 — replication reset state persistence
The three sites that flush `reset_statuses_map` into `meta_sys` inserted each
key verbatim. A bare-ARN key (produced by `ObjectInfo::replication_state`) has
no internal prefix, so read-back — which only recognizes prefixed keys —
silently dropped the reset state; a rustfs-only key was invisible to
MinIO-compatible readers. New `persist_reset_statuses` normalizes every entry
to the canonical `replication-reset-<arn>` suffix and writes both the
`x-rustfs-internal-*` and `x-minio-internal-*` prefixes.
B18 — Legacy (V1Obj) body round trip
`FileMetaVersion::encode_to` never emitted the `V1Obj` field, so re-encoding a
Legacy version silently dropped its entire body. Adds symmetric `encode_to`
for `MetaObjectV1`/Stat/Erasure/ChecksumInfo/Part and a `write_msgp_time`
helper (ext8/type-5/12-byte, matching the decoder). `Mode` uses the strict
`write_u32` marker the decoder requires, and `Stat.ModTime` is written only
when present so a `None` never round-trips to `Some(UNIX_EPOCH)`.
Adds regression tests for each fix (138 filemeta tests pass). Verified with an
adversarial multi-expert review that could not break any of the three.
Refs rustfs/backlog#863 (B15, B16, B18).
The codec streaming GET reader divides by erasure.block_size in
build_codec_streaming_part_reader without validating the erasure
dimensions, unlike the legacy multipart path which already rejects
block_size==0 / data_shards==0. FileInfo::is_valid() does not check
block_size, so corrupted on-disk metadata (block_size==0, data_blocks>0)
passes validation and panics the read task with a divide-by-zero.
Add Erasure::has_valid_dimensions() and reject invalid dimensions at the
codec streaming entry before any disk access, mirroring the legacy guard
(which now reuses the same predicate).
Refs backlog#868 (868-1).
* feat(internode): P0 gRPC transport tuning, message limits, payload metrics
Land the P0 subtask from docs/grpc-optimization: close the client-vs-server
transport gaps and add instrumentation to size which unary RPCs need channel
isolation in P1.
Transport tuning (G3): the client `Endpoint` now disables Nagle and raises the
HTTP/2 stream/connection flow-control windows to mirror the server socket, so
small lock/health RPCs are not batched and larger metadata responses are not
throttled by the 64KiB default window. All env-overridable, 0 opts out.
Message-size limits (G1): both `NodeServiceClient` and `NodeServiceServer` set
max decode/encode size (default 100MiB) instead of tonic's silent 4MiB cap, so a
large multi-version xl.meta or aggregated ReadMultiple no longer fails
out_of_range. The server limit is set on `NodeServiceServer` before wrapping in
the auth `InterceptedService` (the interceptor type does not expose it).
Payload instrumentation (P1 prep): ReadAll/ReadMultiple record a payload-size
histogram plus a large-payload counter when a response crosses the configured
threshold (default 8MiB), feeding alerting on paths that contend with
latency-sensitive control-plane traffic on the shared channel. Threshold-only
counter, no per-call hot-path log.
Verification: cargo check/test on config, io-metrics, ecstore, rustfs; clippy
clean on touched files; make pre-commit green.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(internode): P1 control/bulk gRPC channel isolation (opt-in)
Land the P1 subtask from docs/grpc-optimization: physically separate large
bytes-carrying unary RPCs from latency-sensitive control-plane RPCs so a big
transfer can no longer head-of-line block a lock/health RPC on the shared
HTTP/2 connection (G2/G5).
Introduce ChannelClass { Control, Bulk } and get_channel_for_class in protos.
Control RPCs keep the per-peer connection keyed by the bare address; Bulk RPCs
(ReadAll/WriteAll/ReadMultiple/BatchReadVersion, via a new get_bulk_client) are
round-robined across a small per-peer bulk pool.
Rather than restructuring the global GLOBAL_CONN_MAP (and every consumer), bulk
channels are cached under a composite key (addr\0bulk\0idx). The NUL separator
cannot appear in a URL, so bulk keys never collide with the control key. This
keeps the blast radius small on a consistency-sensitive path. create_new_channel
is refactored into build_channel(dial_addr, cache_key) so several physically
distinct channels to one peer cache independently while dialing/TLS still use
the real address.
Gated by RUSTFS_INTERNODE_CHANNEL_ISOLATION (default OFF) so the default build
is byte-for-byte the pre-P1 behavior: bulk resolves to the control channel and
the switch is a single-env rollback. RUSTFS_INTERNODE_BULK_CHANNELS (default 2,
clamped >=1) sizes the pool. On failure, evict_failed_connection drops the whole
bulk pool for the peer (round-robin hides which index was used), avoiding
half-dead cached channels.
Lock RPCs (remote_locker) already use the default Control path, so lock
semantics and retry behavior are unchanged.
Verification: cargo check/test on config, protos, ecstore, rustfs; new protos
tests for bulk key routing and isolation-off passthrough; clippy clean on
touched files; make pre-commit green.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(internode): P2 msgpack/JSON codec observability + encode buffer presizing
Land the safe, wire-compatible slice of P2 from docs/grpc-optimization: the
observability prerequisite for retiring the redundant JSON fields, plus a codec
micro-optimization. No proto/wire-format change; JSON is still dual-written.
Internode RPCs today dual-encode each metadata value as both msgpack (`*_bin`)
and a JSON compatibility string, and decoders prefer `_bin` with a JSON
fallback. Before the JSON fields can ever be dropped (a cross-version change),
that fallback must be proven unused in production.
Add rustfs_system_network_internode_msgpack_json_fallback_total{direction,
message}: incremented whenever a decode falls back to the JSON field because the
msgpack payload was absent. Wired into both directions — the client decoding
peer responses (remote_disk.rs, incl. the list-level read_multiple/batch
fallbacks) and the server decoding peer requests (node_service/disk.rs). This
counter must read zero across a release window before send paths stop writing
JSON and the proto text fields are reserved/removed (the deferred P2-1 steps).
Also pre-size the msgpack encode buffers (Vec::with_capacity(512)) on both
sides, eliminating the repeated growth reallocations for typical FileInfo
payloads with zero added copy. Full thread_local buffer pooling is deferred: it
needs either an extra copy (unclear net win) or a send-path buffer-return
lifecycle, to be justified by a codec microbenchmark first.
Verification: cargo check/test on io-metrics, ecstore, rustfs; new fallback
counter smoke test; existing codec decode tests green; clippy clean on touched
files; make pre-commit green.
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(internode): add msgpack/JSON convergence observation runbook
Runbook driving the observation-gated retirement of the redundant JSON
compatibility fields on internode gRPC metadata RPCs (grpc-optimization P2-1).
Documents the shipped fallback counter
(rustfs_system_network_internode_msgpack_json_fallback_total{direction,message}),
the PromQL to confirm it reads zero across a release window, a standing alert,
and the staged flip/rollback procedure (env-gated msgpack-only send, then proto
field removal in N+1).
Includes the verified field -> peer-decoder audit: only fields whose peer
decodes _bin first may be converged. Notes DeleteVersion.opts (DeleteOptions) is
NOT convergence-ready — its server handler is not _bin-first and must gain a
decode_msgpack_or_json path first. This gates the send-side change so it cannot
empty a JSON field an old peer still needs.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(internode): env-gated msgpack-only send + DeleteVersion _bin support (P2-1)
Implements the send-side lever for retiring the redundant JSON compatibility
fields on internode gRPC metadata RPCs, plus the missing `_bin` support on the
delete path that it depends on (grpc-optimization P2-1). Default-off: the base
build is byte-for-byte the prior dual-write behavior.
Gated msgpack-only send (RUSTFS_INTERNODE_RPC_MSGPACK_ONLY, default false):
- New rustfs_protos::internode_rpc_msgpack_only() reads the flag.
- Client (remote_disk.rs) compat_json() and server (node_service/disk.rs)
compat_response_json() emit an empty JSON string when the flag is on, so only
the msgpack _bin payload is sent. The _bin field is always sent; decoders keep
the JSON read fallback. Applied only to fields with a confirmed _bin-first peer
decoder (WriteMetadata/UpdateMetadata/RenameData file_info, UpdateMetadata opts,
ReadOptions, ReadMultipleReq, BatchReadVersionReq; ReadVersion/ReadXL/RenameData
responses and the ReadMultiple/BatchReadVersion response lists).
- Only enable after the P2 fallback counter has read zero across a release window
(see docs/operations/internode-msgpack-json-convergence-runbook.md). Single-env
rollback; no wire-format break.
DeleteVersion(s) _bin support (prerequisite):
- The DeleteVersion/DeleteVersions protos had NO _bin fields. Add additive
(backward-compatible) bytes file_info_bin/opts_bin (DeleteVersion) and repeated
bytes versions_bin + bytes opts_bin (DeleteVersions); regenerate the checked-in
prost struct.
- Client dual-writes them; server decodes them _bin-first with JSON fallback.
- These delete fields are kept OUT of the msgpack-only set (always dual-write)
until their own fallback counter reads zero across a window with the new
decoders fully deployed. DeleteVersion.raw_file_info stays JSON-only (no _bin
field yet).
Verification: cargo check/test on protos, config, ecstore, rustfs (incl. the six
delete request handler tests and a compat_json default-path test); clippy clean
on touched files; make pre-commit green.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(internode): P3 cluster peer online/offline health metric
Land the safe observability core of P3 (grpc-optimization G6/G8): track each
internode peer's reachability and expose the offline count, for parity with
MinIO's minio_cluster_servers_offline_total. Pure instrumentation — peer
selection and quorum are unchanged.
- io-metrics: per-peer PeerHealthState { online, consecutive_failures } registry
plus record_peer_reachable/record_peer_unreachable. A peer flips offline after
N consecutive failures (dial failures or RPC-triggered evictions) and back
online on the next successful dial; the count of offline peers is published to
the rustfs_cluster_servers_offline_total gauge.
- config: RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD (default 3, clamped >= 1).
- protos: build_channel marks the peer reachable on a successful dial and
unreachable on a dial failure; evict_failed_connection feeds the failure signal
too. Keyed by the real peer address, so control and bulk channels to one peer
share health state.
Deferred (documented in docs/grpc-optimization P3): startup prewarm (no clean
topology-ready hook yet), the offline fast-bypass in peer routing (consistency-
sensitive; must not change quorum), and idempotent-read-only retry. This commit
is observability only.
Verification: cargo check/test on io-metrics, config, protos (new peer-health
state-machine and threshold-clamp tests); clippy clean on touched files; make
pre-commit green.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(internode): P3 control-channel prewarm + self-healing offline bypass
Add the remaining P3 connection-lifecycle levers (grpc-optimization G6/G8), both
env-gated and default-off so the base build is unchanged.
Prewarm (RUSTFS_INTERNODE_PREWARM, default off): RemoteDisk::new spawns a
best-effort background dial of the peer's control channel, deduped per peer
address, moving the connect cost off the first RPC. Failures fall through to the
existing lazy connect + recovery monitor.
Offline bypass (RUSTFS_INTERNODE_OFFLINE_BYPASS, default off): remote_disk
get_client/get_bulk_client fast-fail a peer already marked offline instead of
paying the connect timeout, so the erasure layer proceeds on quorum sooner. This
does NOT change quorum. It is self-healing: cluster_peer_should_bypass lets one
request per RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS (default 5s) through to recover
the peer even with no background monitor, and the recovery monitor's own probe
path calls the client directly so it is never bypassed.
io-metrics gains cluster_peer_is_offline / cluster_peer_should_bypass (with a
per-peer re-probe timestamp). Scope: data path only — remote_locker (lock RPCs,
most consistency-sensitive) is left dual-writing/unbypassed as a follow-up.
Verification: cargo check/test on io-metrics, config, ecstore (new self-healing
bypass tests; all 105 rpc tests green); clippy clean on touched files; make
pre-commit green.
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(internode): add A/B benchmark runbook for gRPC optimization stages
Reproducible before/after collection procedure for grpc-optimization P0–P3.
Since every stage is env-gated, before/after is the same binary with different
env — no rebuild. Documents, per stage: the exact env toggles (baseline vs
enabled column), which existing bench script to run
(run_internode_transport_baseline.sh / run_four_node_cluster_failover_bench.sh),
the Prometheus metrics to capture, and the acceptance gates from the design docs
(e.g. lock p99 down >= 20% for P1, msgpack fallback counter = 0 before enabling
P2, correct rustfs_cluster_servers_offline_total for P3).
Live runs require a multi-node cluster + load tool + Prometheus scrape and cannot
be produced in a single-process sandbox; artifacts land under target/bench
(gitignored) and attach to the PR.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(internode): P3-2 lock-path offline bypass + P3-3 idempotent read retry
Extend the offline bypass to the lock path and add opt-in retries for idempotent
reads (grpc-optimization P3-2/P3-3). Both env-gated and default-off/zero.
Offline bypass (lock path): factor the bypass decision into a shared pub(crate)
internode_offline_bypass_reason(addr) and call it from remote_locker::get_client
too, so lock RPCs to an offline peer fast-fail (letting dsync reach quorum
sooner) instead of paying the connect timeout. Does not change quorum; the
self-healing re-probe keeps peers recoverable. Gated by
RUSTFS_INTERNODE_OFFLINE_BYPASS (default off).
Idempotent read retry (P3-3): add execute_read_with_retry — a bounded,
exponential-backoff retry for read-only/reentrant RPCs on transient network
errors — and route disk_info through it. RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES
defaults to 0 (disabled). Write/lock RPCs are never retried (quorum/idempotency
safety, per CLAUDE.md); the wrapper requires an Fn closure so only reads that
rebuild their request from borrowed inputs qualify.
Deferred: grpc.health.v1 (optional ecosystem-compat only; needs a new
tonic-health dep and 3-way hybrid-service wiring — internal needs are met by the
existing Ping RPC).
Verification: cargo check/test on config, ecstore (105 rpc tests green incl.
disk_info now via the retry wrapper); clippy clean on touched files; make
pre-commit green.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(scripts): one-click internode gRPC A/B benchmark driver
Wrap the per-stage env matrix from the benchmark runbook into
scripts/run_internode_grpc_ab_bench.sh: given --stage <p0|p1|p2|p3> and --phase
<before|after>, it emits the stage/phase RUSTFS_INTERNODE_* server env to
<out-dir>/server-env.sh and runs the right underlying bench
(run_internode_transport_baseline.sh for p0/p1/p2, run_four_node_cluster_failover_bench.sh
for p3) into a labeled target/bench/internode-transport/<stage>-<phase>/.
Passthrough args after `--` reach the underlying bench; --dry-run previews the
env + command. The script is explicit that RUSTFS_INTERNODE_* are server env, so
for the load-driven stages the operator must restart rustfs with the emitted env
before the run; the docker four-node (p3) path exports them for a forwarding
compose. shellcheck-clean.
Runbook updated with a "One-click driver" section.
Co-Authored-By: heihutu <heihutu@gmail.com>
* chore(compose): forward RUSTFS_INTERNODE_* into the four-node cluster
The four-node local-build compose only forwarded a fixed whitelist of env, so
the internode gRPC knobs (grpc-optimization P0-P3) never reached the containers
and the A/B bench driver's "after" phase was a no-op. Forward the full
RUSTFS_INTERNODE_* set with defaults matching the binary defaults, so leaving
them unset is a no-op and the A/B driver can toggle a stage per phase.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(internode): address Copilot review — retry health action + poison-safe peer health
Two review nits on #4337:
- P3-3 idempotent read retry (remote_disk.rs): execute_read_with_retry ran every
attempt through execute_with_timeout_for_op, which hardcodes
FailureHealthAction::MarkFailure. So the first transient error could flip the
disk faulty and short-circuit the remaining retries, and each attempt
over-counted the failure. Route all but the final attempt through
execute_with_timeout_for_op_and_health_action with IgnoreFailure; only the last
attempt marks faulty/evicts. No default impact (retries default 0).
- Peer-health helpers (io-metrics): record_peer_reachable/record_peer_unreachable,
cluster_peer_is_offline and cluster_peer_should_bypass early-returned on a
poisoned mutex, permanently stalling the offline gauge and bypass state after a
single panic. Recover via PoisonError::into_inner().
Verification: cargo check/test on io-metrics + ecstore (105 rpc tests green);
clippy clean on touched files; make pre-commit green.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
feat(admin): add MinIO-compatible IAM/IDP admin endpoints
Register and implement MinIO admin API compatibility for IAM/IDP:
- PUT /v3/import-iam-v2 and POST /v3/revoke-tokens/{userProvider}
- generic /v3/idp-config/{type}[/{name}] CRUD mapped onto existing config
- LDAP/OpenID service-account, policy-entities, and list-access-keys flows
Adds IamSys::delete_temp_account primitive to back STS token revocation.
revoke-tokens requires the broader ListUsers admin capability for
cross-user revocation (self-revocation only needs RemoveServiceAccount),
mirroring the cross-user guard used by the service-account handlers.
Registers admin route-policy inventory entries for every new route.
Unsupported LDAP/OpenID backends return honest compatibility errors.
Refs rustfs/backlog#609 #610#616
feat(admin): add MinIO-compatible diagnostics and service-control endpoints
Register and implement MinIO admin API compatibility routes:
- POST /v3/service: restart/stop trigger the existing graceful-shutdown
path; freeze/unfreeze record advisory state (request admission is not
yet gated, surfaced honestly via effective=false)
- profiling/trace and healthinfo/obdinfo/log diagnostics endpoints
- GET /v3/top/locks and POST /v3/force-unlock, backed by new
FastObjectLockManager::list_locks()/force_unlock() introspection
- speedtest routes where feasible
Refs rustfs/backlog#604 #606#607#615
fix(replication): persist original mtime in MRF entries (backlog#867)
MRF delete entries did not persist the original delete-marker mtime, so
after a restart the recovery replay path reconstructed the delete without
a source timestamp. Downstream the replica delete-marker was stamped with
the replay time (now()) instead of the source mtime, causing delete-marker
timestamp divergence across clusters.
Extend the MrfReplicateEntry disk format with an optional deleteMarkerMtime
field (persisted as Unix nanoseconds) in both duplicate struct definitions
(rustfs-replication and rustfs-filemeta). DeletedObjectReplicationInfo now
persists delete_marker_mtime, and start_mrf_processor restores it onto the
reconstructed delete so the replica keeps the source timestamp.
Backward compatibility: the new key uses skip_serializing_if + serde
default, so historical MRF files without it decode to None and replay
falls back to the current time (pre-#867 behaviour). No panic or entry
loss on old files.
Closesrustfs/backlog#867
Add MinIO-compatible admin endpoints under the /v3 prefix, wired to real
RustFS subsystems where they exist and documented as compatibility-only
where they do not.
Replication (backlog#611):
- POST /v3/replication/diff: bounded on-demand scan of object versions,
returning versions whose replication status is PENDING or FAILED. Wired
to list_object_versions and the per-version replication status. Requires
the bucket to exist and have a replication configuration. The scan is
capped (REPLICATION_DIFF_MAX_SCAN) and reports IsTruncated when partial.
- GET /v3/replication/mrf: reports the failed-replication backlog (MinIO's
MRF concept) from the replication stats handle: aggregate failed count and
size per target plus in-queue counters. RustFS records failed replications
as aggregate counters and persists MRF entries without a runtime query API,
so per-object MRF rows are not enumerable; PerObjectEntriesAvailable is
always false to make that explicit.
Batch jobs (backlog#613):
- POST /v3/start-job, GET /v3/list-jobs, GET /v3/status-job,
GET /v3/describe-job, DELETE /v3/cancel-job.
- RustFS has no batch-job execution engine (no scheduler, queue, or worker),
so these implement MinIO request/response/error semantics only and never
fake execution or success: start-job parses and validates the job
definition then returns NotImplemented for known job types and
InvalidRequest for unknown ones; list-jobs returns an empty list;
status/describe/cancel validate jobId and return a no-such-job error.
All routes are registered with admin auth using existing AdminAction
variants (ReplicationDiff, GetReplicationMetricsAction, StartBatchJobAction,
ListBatchJobsAction, DescribeBatchJobAction, CancelBatchJobAction) and are
covered by the admin route-registration matrix and new handler unit tests.
Refs rustfs/backlog#611 #613
security(swift): remove placeholder SSE module (backlog#646)
The Swift `encryption` module was a non-functional stub: `encrypt_data`
returned the plaintext unchanged while labeling it AES-256-GCM in the
object metadata, and `generate_iv` derived the IV from a timestamp
rather than a CSPRNG. It had no production caller (only a `pub mod`
declaration and one integration test), so wiring it in as-is would have
silently shipped plaintext advertised as ciphertext.
We are not supporting Swift server-side encryption for now, so remove
the module outright rather than keep a dangerous stub around:
- delete crates/protocols/src/swift/encryption.rs
- drop `pub mod encryption;` from swift/mod.rs
- remove the encryption case (and unused import) from the swift
integration test
The module reached main dubiously: it was introduced together with the
whole Swift API in commit 86e93624 ("fix(heal): canonicalize scanner
object-dir repairs (#3864)"), a 1665-file squash whose PR description
only covered the heal change and never mentioned Swift or SSE.
Verified: cargo fmt; cargo test -p rustfs-protocols --features swift
--test swift_simple_integration (10 passed); arch guardrail scripts pass.
fix(multipart): serialize the part commit per uploadId to prevent mixed-generation shards (backlog#853)
Concurrently re-transmitting the same part could land two shard
generations across different disks: each shard is individually
bitrot-valid, so the corruption surfaces only as a silent mix at read
time.
The hazard is confined to rename_part, where two temp parts are moved
cross-disk onto the SAME final part path — interleaving there can leave
shards from two generations. Each concurrent stream already writes to its
own unique temp dir, so the encode/stream phase never conflicts and must
stay lock-free: holding a lock across it would serialize slow re-transmits
of the same part, break the S3 'last finisher wins' semantics, and cause
UploadPart lock-acquire timeouts (ServiceUnavailable).
Scope a write lock to the uploadId namespace around only the rename_part
commit, so each commit is atomic across disks and the last committer wins
consistently. Mirrors MinIO's per-uploadID NS lock; disjoint from the
object lock held by complete_multipart_upload (no lock-ordering cycle).
Honors opts.no_lock.
The pre-delete-before-rename half of backlog#853 was already fixed in
#4316; this completes the issue.
Adds an end-to-end regression test: six concurrent resends of the same
part must all succeed (no lock-acquire timeout), exactly one generation
stays visible, and the reassembled object equals one intact generation.
Closesrustfs/backlog#853
Three tests share the global list-objects mutation-sequence state (the
LIST_OBJECTS_MUTATION_SEQUENCE atomic and the per-bucket sequence map)
and all call reset_list_objects_mutation_sequences_for_test(), but only
observed_mutation_persists_namespace_journal_high_water was marked
#[serial_test::serial]. The serial lock only excludes tests that also
carry the attribute, so the unmarked tests still ran concurrently with
it: the journal test's persist of "bucket" -> 11 could land after the
recovery test's reset, making it observe 11 instead of the expected 9
(flaky under the default multi-threaded test runner, green with
--test-threads=1).
Add #[serial_test::serial] to the two unmarked tests:
- persistent_key_only_index_load_recovers_mutation_sequence_from_journal
- list_objects_mutation_sequence_advances_per_bucket
Unique bucket names alone would not fix this: the per-bucket assertions
in list_objects_mutation_sequence_advances_per_bucket check exact values
produced by the single global atomic counter, which any concurrent
observe call perturbs, and every reset zeroes that global state for
whichever peer is mid-flight.
Co-authored-by: heihutu <heihutu@gmail.com>
fix(filemeta): use struct-init to satisfy clippy::field_reassign_with_default
The replication_info_equals test constructed FileInfo via
`let mut x = FileInfo::default()` followed by field reassignment, which
clippy::field_reassign_with_default (deny-by-default under -D warnings)
rejects on the lib-test target, breaking `Test and Lint` CI on every PR.
Rewrite the three constructions using struct-init + ..Default::default().
Test-only change; behavior unchanged.
Add two durable architecture references grounded in current code:
- minio-rustfs-router-compatibility.md: MinIO cmd/api-router.go (S3
object/bucket) and cmd/admin-router.go (admin /v3, /v4) vs RustFS
implementation status, with per-row landing points and a gaps-only
checklist of still-missing admin endpoints (profiling, healthinfo,
LDAP IDP config, replication diff, MRF, batch, locks, speedtest,
log stream, top, trace).
- minio-file-format-compat.md: xl.meta (meta_ver 3 write, <=3 read,
XL2 magic, rs-vandermonde, HighwayHash256 bitrot, inline data) and
.metadata.bin bucket-metadata interop matrix for the backlog#580
items, plus old-RustFS -> new migration and a phased plan.
Cross-links s3-compatibility-matrix.md and admin-route-action-snapshot.md
and indexes both docs in docs/architecture/README.md.
Refs rustfs/backlog#596rustfs/backlog#603rustfs/backlog#580
The peer `DeleteBucketMetadata` RPC handler was a stub that returned
success without doing anything, and the delete-bucket flow never sent
the notification in the first place. As a result, after a bucket was
deleted other nodes kept serving its stale cached metadata.
Wire the whole path end to end:
- ecstore: add an in-memory `remove_bucket_metadata` (free fn) and
`BucketMetadataSys::remove`, the counterpart to `set_bucket_metadata`,
and export it through the `api::bucket::metadata_sys` facade.
- node_service: `handle_delete_bucket_metadata` now validates the bucket
name and actually drops the cached metadata for it.
- bucket_usecase: after a successful delete_bucket, notify peers via
`notification_sys.delete_bucket_metadata` in the background, symmetric
to the existing `notify_bucket_metadata_reload` path.
Also update the delete-bucket-metadata unit test to assert the
empty-bucket rejection instead of the old always-success stub, and drop
an unused `tracing::debug` test import left over from #4322.
Verified: cargo fmt; cargo check -p rustfs-ecstore; cargo test -p rustfs
--lib --features rio-v2 test_delete_bucket_metadata_empty_bucket; arch
guardrail scripts pass.
Two compounding defects let a truncated shard corrupt a GET (ranged GET could
return HTTP 200 with wrong bytes):
- `BitrotReader::read` returned `Ok(short_len)` when the shard stream hit EOF
before filling the caller's buffer. With `skip_verify` /
`HashAlgorithm::None` / parity=0 there is no hash to catch it, so the short
shard was accepted and every downstream byte shifted.
- `ParallelReader.offset` was set once and never advanced per stripe, so every
stripe after the first reused the first stripe's geometry and the last-stripe
length clamp was wrong.
Fix both (they are mutually required):
- `BitrotReader::read` now errors (`UnexpectedEof`) on a short read, before and
independent of the bitrot hash check, so it fires under skip-verify/no-hash
too. The caller sizes the buffer to the expected per-stripe shard length, so
"buffer not filled" == "shard truncated". A short read routes through the
existing `errs[i]` path, dropping that reader from the stripe so parity
reconstruction engages; with parity=0 the stripe fails read quorum and the GET
errors loudly instead of streaming shifted bytes. Mirrors MinIO's
`parallelReader.Read` (`n != shardSize` -> reader failed).
- `ParallelReader::read` / `read_lockstep` advance `self.offset += shard_size`
per stripe so the per-stripe expected length (incl. the shorter final stripe)
is exact, matching the correct pattern already used by heal.
Updates three bitrot tests that used an obsolete oversized-buffer pattern to
size per-stripe (as the real decode/heal paths do), and adds a regression test
that a truncated shard errors under None/HighwayHash + skip_verify.
Refs backlog#799 (B2), issue rustfs/backlog#851. Design converged by two
independent expert reviews referencing MinIO cmd/erasure-decode.go.
fix(heal): remove useless .into() conversion in heal error path
Clippy flagged a useless_conversion lint at heal.rs:491 where
is redundant because is already of type .
fix(filemeta): key round-tripped replication reset state by canonical header (backlog#799 B19->B16)
`get_internal_replication_state` stored the reset status under the bare ARN after
stripping the internal prefix, while the write and lookup sides
(`get_replication_state` / `ReplicationState::target_state`) use the full
`target_reset_header(arn)` key. A `target_state` fallback masked the lookup miss,
but the map stayed keyed inconsistently (bare on read, full on write), which can
drop reset state across merge/reflatten cycles.
Store the canonical `target_reset_header(arn)` key on read so the map is
consistent everywhere; the lookup fallback becomes belt-and-suspenders rather
than load-bearing. Adds a round-trip regression test.
Refs backlog#799 (B16), tracked in rustfs/backlog#863.
Second TODO-convergence round over the current tree (backlog#646). All
line numbers in the old inventory had gone stale after the set_disk /
diagnostics / cluster refactors, so this re-scans and reduces the marker
count from 144 to 99.
STALE removals (comment describes already-implemented behavior, or dead
commented-out blocks) across ecstore (set_disk ops/core, store,
cluster/rpc, bucket/metadata_sys, services), iam, filemeta, s3select and
rustfs auth/object_usecase. No behavior change.
Safe fills, each verified:
- filemeta: replication_info_equals now also compares
replication_state_internal (function currently has no callers; adds a
regression test).
- bitrot: drop the confirmed-unused `_want` parameter from bitrot_verify
and the now-unused `sum` on LocalDisk::bitrot_verify, removing a
Bytes::copy_from_slice allocation. Streaming verify uses the file's
embedded per-shard hash, never the passed sum.
- signer: rename v4_ignored_headers -> V4_IGNORED_HEADERS and drop the
non_upper_case_globals allow.
- admin/heal: test_decode was #[ignore]d and used serde_urlencoded on a
JSON body (would panic); rewire to serde_json::from_slice to match the
production decode path, add assertions, un-ignore.
Verified: cargo fmt; cargo check on touched crates; tests pass
(filemeta, signer, bitrot, heal::test_decode); arch guardrail scripts
pass.
fix(ecstore): reject renewing a misplaced drive into the wrong set (backlog#799 B19)
`renew_disk` located the drive's (set, disk) position from its own format via
`find_disk_index`, but never checked that the resolved `set_idx` matched this
`SetDisks`' own `set_index` before inserting the drive into `self.disks`. A
drive whose format places it in another set would be claimed by this set too,
so two sets could manage the same drive and degrade together.
Skip (with a warning) when `set_idx != self.set_index`.
Refs backlog#799 (B19), tracked in rustfs/backlog#863.
fix(replication): don't silently swallow resync status persistence failure (backlog#799 B23)
After a resync computes a new replication status, it persists it via
`put_object_metadata` but discarded the `Err` case (`if let Ok(u) = ...`). A
failure left the object's on-disk replication status disagreeing with the resync
result with no signal at all. Log the failure at warn level instead.
Refs backlog#799 (B23), tracked in rustfs/backlog#863.
heal_object wrote healed shards to .rustfs/tmp/<uuid>/ and only removed that tmp
dir on the success path (the final delete_all). Three early exits after the tmp
shards were written leaked the dir:
- `erasure.heal(...)?` failing midway,
- the `disks_to_heal_count == 0` early return, and
- the `?` on the post-rename remote data-dir delete.
Add the tmp cleanup before the first two, and downgrade the remote data-dir
cleanup failure to a warning (the healed shard is already renamed into place, so
that cleanup failing must not abort the heal or leak the tmp shards).
Refs backlog#799 (B20), tracked in rustfs/backlog#863.
`MetaDeleteMarker::decode_from` returned an error on any field key it didn't
recognize, unlike `MetaObject::decode_from` / `MetaObjectV1::decode_from`, which
skip unknown fields. That breaks forward compatibility: a delete marker written
by a newer version with an extra key fails to decode on an older binary.
Skip the unknown field's value (`skip_msgp_value`) and continue, matching the
object decoders. Adds a regression test.
Refs backlog#799 (B17), tracked in rustfs/backlog#863.
`rename_part` unconditionally ran `cleanup_multipart_path([part.N, part.N.meta])`
on every disk *before* the per-disk rename fan-out. The per-disk `rename_part`
already replaces `part.N` atomically (std::fs::rename) and rewrites `part.N.meta`,
so the pre-delete is redundant — and destructive: it removed an already-committed
(ACKed) part on all disks before the new rename landed, so a re-upload of the same
part that then failed write quorum destroyed the committed part outright.
Remove the pre-rename cleanup and rely on the atomic rename to overwrite in place;
the quorum-failure rollback below is unchanged. No behaviour change on the success
paths (first upload / retry both end with part.N replaced), verified by the
existing multipart suite.
Refs backlog#799 (B4). The remaining half of B4 — serializing concurrent
same-part uploads with an uploadId-scoped lock so two generations can't be mixed
across disks — is a larger concurrency change tracked as a follow-up on the issue.
MRF delete replay reconstructed the delete with `..Default::default()`, leaving
`replication_state = None`. `replicate_delete` derives its target set purely
from `replication_state.replicate_decision_str`, so an empty state produced an
empty decision -> zero targets: the replayed delete contacted no remote at all,
a silent no-op that left replicas permanently diverged after a restart.
The MRF entry doesn't persist the decision and the source object is already
gone, so re-derive it from the live bucket config at replay time via
`check_replicate_delete` — mirroring the object heal path
(`get_heal_replicate_object_info`) — and set it on the reconstructed delete's
`replication_state`. This is a contained fix with no change to the on-disk MRF
format.
Refs backlog#799 (B9).
Note: the source delete-marker mtime is still not persisted in the MRF entry,
so a replayed delete marker is stamped with the replay time on the target. That
is a separate, minor consistency nuance (the delete now propagates correctly)
and can be addressed by extending the MRF entry format in a follow-up.
The MRF persister accumulated overflow entries in `pending`, flushed them with
`flush_mrf_to_disk`, and cleared `pending` on success. But `flush_mrf_to_disk`
*overwrites* the whole MRF file with exactly the entries passed. After flushing
batch A (file = A) and clearing, the next flush wrote batch B and thereby
overwrote the file to contain only B — and the MRF file is only replayed (and
cleared) at startup, never during the run, so batch A's entries were silently
lost. A crash after the B flush lost all of batch A's pending replications.
Keep `pending` cumulative (the file must hold the full set of overflow entries
for the run) and rewrite the whole set on each flush instead of clearing after
success:
- flush eagerly once 1 000 *new* entries accumulate since the last write
(measured against the flushed length, so a large backlog isn't rewritten on
every add), and on the 10s tick when dirty;
- bound the in-memory/on-disk backlog with `MRF_PENDING_CAP` (200 000) and log
once when the cap is hit rather than growing without limit.
Refs backlog#799 (B10).
The resync result verification HEADed the target after replicating and counted
the outcome with inverted error handling:
- for a delete marker, ANY HEAD error (timeout, 5xx, auth, malformed) was
counted as replicated (success);
- for a versioned object against an AWS-style target, HEAD was sent with the
RustFS UUID versionId, which AWS rejects with 400, so a well-replicated
object was counted as failed.
Classify the error before counting:
- delete marker: only a definitive 404/NoSuchKey or 405/MethodNotAllowed
confirms the marker propagated (`is_retryable_delete_replication_head_error`
== false); any retryable/ambiguous error now counts as failed;
- versioned object with a version-id-format rejection: re-verify via
`head_object_fallback` (versionId-less HEAD) before deciding — present ->
replicated, absent/error -> failed;
- all other errors: failed, as before.
Reuses the existing, unit-tested classifier helpers. Verified against the
existing resyncer suite (24 tests).
Refs backlog#799 (B13).
During resync against an AWS-style target that rejects RustFS UUID versionIds,
the code retries HEAD without a versionId and compares ETags. On a match it set
`replication_action = None` ("already in sync, nothing to copy") but did not
return, so control fell through to `if replication_action != ReplicationAction::All`
— a branch meant only for the unsupported metadata-only case — and stamped the
object FAILED with "metadata-only replication is not implemented". The target
already held an identical object, yet the source recorded FAILED forever, so
AWS-style targets never converged and the MRF kept re-queuing.
Handle `ReplicationAction::None` explicitly before that branch: record it as
Completed (with the resync timestamp/`replication_resynced` bookkeeping for
ExistingObject + reset_id, mirroring the HEAD-success None path) and return.
Only `ReplicationAction::Metadata` now takes the metadata-unsupported failure
branch; `All` still proceeds to the copy. This path is the only way `None`
reaches that point (the HEAD-success None case already returns earlier).
Refs backlog#799 (B11).
`MultiWriter` recorded per-writer failures only in a private `errs` vector and
nulled the failed writer locally, but `put_object` never used that to prune the
disk set: `rename_data` ran over the full `shuffle_disks`, so a disk that took a
short/failed write still had its truncated shard renamed into place and counted
as an online disk. The object then claimed N good shards while one was
short/corrupt, so a single later disk failure could drop it below reconstructable
quorum — silent data loss.
- `write_shard`: null the writer on a generic write error too (not only on
`ShortWrite`), so a failed writer is uniformly represented as `None` — matching
`shutdown_writer`, which already does this.
- `put_object`: after encode, drop every disk whose writer failed
(`drop_failed_writer_disks`) from `shuffle_disks` before `rename_data`, and
re-check write quorum over the survivors. `rename_data` already re-checks
quorum and rolls back if too few disks remain, so excluded disks are neither
committed nor counted (MinIO sets failed writers to nil before `renameData`).
Adds unit tests for the exclusion/quorum accounting.
Refs backlog#799 (B3).
fix(heal): don't mark set healed / discard resume state when objects failed (backlog#855)
`execute_heal_with_resume` unconditionally called `mark_completed()` and
returned `Ok(())` after the bucket loop, even when objects failed. The caller
then treats `Ok` as success and cleans up the resume + checkpoint state, so a
heal run with per-object failures was reported as a clean completion and its
state (including the failed set) was destroyed with no retry.
Finalize based on the failure count instead:
- failed_objects == 0 -> mark completed (unchanged);
- failed_objects > 0 with retry budget left -> `schedule_retry()` (bump the
bounded retry counter + reset per-pass progress for a full re-scan) and
return Err, so `heal_erasure_set` preserves the resume/checkpoint state and
the caller keeps the healing markers for the next run;
- failed_objects > 0 with retries exhausted -> drop the resume state (no
zombie task) but still return Err so the markers survive for a later heal
cycle / the background scanner. Never silently claim a clean completion.
The failure identities are not persisted across pages (the per-page sets are
pruned in `complete_page`), so a retry is a bounded full re-scan; healed
objects are skipped quickly on the normal-scan pass.
Adds `ResumeState::reset_for_retry`, `ResumeManager::schedule_retry`,
`ResumeCheckpoint::reset_for_retry`, `CheckpointManager::reset_for_retry`, and
regression tests. This activates the failure path the caller already documents
at task.rs ("Keep the markers on failure ... the next run re-marks and
eventually clears them"), which was previously dead because heal never
returned Err on object failures.
Refs backlog#799 (B6).
PR #4220 added a purge for orphan empty-directory trees (folder keys with
no xl.meta) on the delete path, but the guard only accepted
object-not-found. Over the real HTTP DELETE path the guard is never
reached: `del_opts` pins `version_id = Uuid::nil()` for directory keys, so
the missing dir object fails the specific-version lookup with
version-not-found (FileVersionNotFound), not object-not-found. The guard
short-circuits, the store returns the error, and the API layer turns it
into a fake 204 — the ghost folder survives, exactly the #4189 symptom.
The existing unit tests passed because they call
`purge_orphan_dir_object` directly, bypassing the nil-version lookup.
Accept both misses in the guard (extracted as
`should_purge_orphan_dir_on_missing`) so the folder delete actually takes
effect. Non-directory keys and non-miss errors (e.g. quorum failures) are
unaffected; the cross-disk data-safety refusal in
`purge_orphan_dir_object` is unchanged.
Verified end-to-end against a running 4-disk erasure server: DELETE of a
planted orphan `ghost/` tree now purges it on all drives and clears the
listing, a real object under the same prefix is untouched, and a prefix
holding real data on any drive is refused (all drives preserved).
Adds predicate regression tests covering version-not-found /
object-not-found on directory keys, and the negative cases.
fix(heal): classify heal_object errors by type, not "not found" substring (backlog#856)
The heal page loop classified heal_object errors with a substring test
(`message.contains("not found")`). `StorageError::DiskNotFound` renders as
"disk not found", so an offline drive during a deep-scan heal matched the
"object absent" branch: the object was returned as `Ok(false)`, recorded into
the checkpoint's processed set, counted as a success, and permanently skipped
on resume — the object silently never got healed.
Replace the substring test with a typed classifier over the wrapped
`StorageError`:
- genuine object/version absence (FileNotFound / FileVersionNotFound /
ObjectNotFound / VersionNotFound) -> Absent (treated as handled);
- transient infrastructure conditions (quorum errors, DiskNotFound,
VolumeNotFound, SlowDown, OperationCanceled) -> TransientSkip, so the object
is retried on a later pass instead of recorded as processed;
- everything else -> Failed (recorded as failed).
Deliberately does NOT use `StorageError::is_not_found()`, which lumps
DiskNotFound/VolumeNotFound with object absence — the exact conflation that
caused the bug. Applied to both the deep-scan and normal-scan heal_object call
sites. Adds regression tests for the classifier.
Refs backlog#799 (B7).
P6 of the SetDisks God-Object split (tracking backlog#815, issue backlog#821;
depends on P5 #4288). Relocate the core object read/write hot-path contract
impls out of the set_disk/mod.rs God-Object into their own module home:
- impl ObjectIO for SetDisks (~1,010 lines) -> set_disk/ops/object.rs
- impl ObjectOperations for SetDisks (~1,137 lines) -> set_disk/ops/object.rs
- Registered pub(crate) mod object; under set_disk/ops.
Pure move — zero logic change. Both contracts stay implemented for SetDisks, so
their EcstoreObjectIO / EcstoreObjectOperations associated-type bounds are
unchanged and the contract-compat tests still guard them. Method bodies are
moved verbatim: a whitespace-insensitive token-stream diff of the two impl
blocks (with their #[async_trait::async_trait] attributes) against the pre-move
mod.rs source is byte-identical (5,754 tokens each) — NO visibility widening was
required, because the impls reach SetDisks helpers and the P5 io_primitives
through inherent self./Self:: calls that resolve across modules unchanged. The
two inherent impl SetDisks blocks that sat between the trait impls (lock batch
helpers) remain in mod.rs, verified present exactly once and uncorrupted.
The issue's borrow/Arc-clone-avoidance optimization is intentionally deferred:
it is a perf-sensitive change requiring the #738 benchmark and would risk the
'byte-level behavior unchanged' acceptance; this PR delivers the relocation.
Verification:
- cargo check / clippy -D warnings -p rustfs-ecstore --all-targets: clean
- cargo test -p rustfs-ecstore --lib: 1841 passed, 0 failed
- all five arch guard scripts: pass
- token-stream diff of moved impls vs original: identical (mod.rs diff is a pure
relocation; ObjectIO/ObjectOperations each defined exactly once post-move)
fix(ecstore): lockstep stripe read to stop EC GET desync truncation
On the reconstruction-verifying GET path, ParallelReader::read used a
data-first schedule: it read only `data_shards` readers per stripe and
pulled in a parity reader as a substitute on demand. Because the shard
readers are streaming (advanced only by being read, no seek), a parity
reader first used mid-object was still positioned at its stream start
(block 0) and returned an earlier stripe than the surviving data shards.
Every shard passed its own bitrot hash, yet the set was mutually
misaligned, so decode_data_with_reconstruction_verification correctly
rejected it with "inconsistent read source shards" and the large-object
GET truncated mid-stream (client "unexpected EOF").
Add read_lockstep(): on the verify_reconstruction path, read every live
shard reader once per stripe and wait for all of them, so all readers
advance one block per stripe and stay mutually aligned; any reader that
errors is retired for the rest of the object (a stream that failed
mid-block can no longer be trusted to be aligned). The adaptive
data-first path is unchanged for non-verifying callers (e.g. heal).
Adds a regression test reproducing a data shard that dies partway through
a multi-stripe object; it must still reconstruct byte-exact output.
Refs backlog#832.
* refactor(ecstore): sink write/rename/delete primitives into set_disk::core::io_primitives (backlog#820)
P5 step 2 of the SetDisks God-Object split (tracking backlog#815, issue
backlog#820; follows step 1 #4285). Relocate the entire set_disk/write.rs
primitive family into the core/io_primitives.rs module home established by
step 1:
- Module items: dangling_delete_grace() + its env consts, and the
OrphanDirScan scan-result enum.
- impl SetDisks write/rename/delete primitives shared across mod.rs and the
ops/ operation families: rename_data, commit_rename_data_dir,
cleanup_multipart_path, rename_part, eval_disks, write_unique_file_info,
update_object_meta(_with_opts), delete_if_dangling, delete_prefix,
scan_orphan_dir, purge_orphan_dir_object, check_write_precondition,
default_read_quorum, default_write_quorum.
- The dangling_delete_grace unit tests move with their subject.
set_disk/write.rs is removed and 'mod write;' dropped from mod.rs.
Pure move + visibility adjustment, zero logic change. Method bodies are moved
verbatim; pub(super) items are widened to pub(in crate::set_disk) so mod.rs /
ops still reach them (write.rs's super was set_disk; the deeper module needs the
explicit path to preserve identical reach). Callers use inherent self./Self::
calls, so no call sites change.
Verification:
- cargo check / clippy -D warnings -p rustfs-ecstore --all-targets: clean
- cargo test -p rustfs-ecstore --lib: 1841 passed, 0 failed (moved
dangling_delete_grace tests run as set_disk::core::io_primitives::tests::*)
- all five arch guard scripts: pass
- token-stream diff of moved block vs original write.rs: identical modulo
visibility tokens (2713 tokens each)
* refactor(ecstore): sink shared read primitives into set_disk::core::io_primitives (backlog#820)
P5 step 3 (final) of the SetDisks God-Object split (tracking backlog#815, issue
backlog#820; follows steps 1 #4285 and 2). Relocate the shared, low-level
metadata/erasure READ PRIMITIVE methods out of set_disk/read.rs into the
core/io_primitives.rs module home, leaving the object-read operation itself in
read.rs.
Moved into a new impl SetDisks block in io_primitives.rs (verbatim bodies):
- read_parts, read_all_fileinfo (+ _observed / _inner / _full_wait /
_early_stop variants), read_all_xl, load_file_info_versions_exact,
read_all_raw_file_info, pick_latest_quorum_files_info, read_multiple_files.
- The should_allow_metadata_early_stop free helper (called by the moved
read_all_fileinfo_observed).
Kept in read.rs: the object-read operation and its private helpers
(read_version_optimized, get_object_fileinfo, get_object_info_and_quorum,
try_get_object_direct_data_shards_with_fileinfo, get_object_with_fileinfo,
get_object_decode_reader_with_fileinfo, build_codec_streaming_part_reader) and
the metadata-cache helpers/tests. read.rs reaches the moved primitives through
the SetDisks core (inherent self./Self:: calls; the sole cross-boundary edge,
get_object_fileinfo -> read_all_fileinfo_observed, is handled by widening that
one method to pub(in crate::set_disk)).
Pure move + visibility adjustment, zero logic change. pub(super) primitives are
widened to pub(in crate::set_disk); the three internal-only fanout variants
(_inner/_full_wait/_early_stop) stay private; load_file_info_versions_exact
stays pub(crate). Method bodies are moved verbatim.
Verification:
- cargo check / clippy -D warnings -p rustfs-ecstore --all-targets: clean
- cargo test -p rustfs-ecstore --lib: 1841 passed, 0 failed
- all five arch guard scripts: pass
- token-stream diff of the three moved regions vs original read.rs: identical
modulo visibility tokens, the new impl-block scaffolding, and rustfmt
signature re-wrapping (read.rs diff is pure deletion, zero added lines)
* refactor(ecstore): extract read IO primitives to set_disk::core::io_primitives (backlog#820)
P5 step 1 of the SetDisks God-Object split (tracking backlog#815, issue
backlog#820). Relocate the module-level low-level read/erasure primitives out of
the 5,898-line set_disk/read.rs into a new core/io_primitives.rs module home:
- Metadata-fanout quorum machinery (MetadataQuorumAccumulator,
MetadataFanoutDiagnostics/Observation, MetadataEarlyStopDecision,
MetadataCacheLookup).
- Bitrot reader scheduling/creation (BitrotReaderSetup and the
create_bitrot_readers_* / schedule / fill helpers).
- Shard-cost classification, read-repair heal dedup, and codec-streaming
reader helpers.
Pure move + visibility adjustment, zero logic change. The moved block is
byte-identical to the pre-move read.rs source modulo the module header swap
(use super::* -> use super::super::*) and item visibility widening
(module-private / pub(super) -> pub(in crate::set_disk) so read.rs still
reaches them). read.rs's impl SetDisks body and imports are byte-identical to
before; mod.rs only gains 'mod core;' and repoints two
GetCodecStreamingReaderBuildOutcome references to the new path.
Verification:
- cargo check / clippy -D warnings -p rustfs-ecstore --all-targets: clean
- cargo test -p rustfs-ecstore --lib: 1840 passed, 0 failed
- normalized + token-stream diff of moved block vs original: identical modulo
visibility tokens and rustfmt signature re-wrapping
* fix(arch): allow io_primitives as READ_REPAIR_HEAL_CACHE owner (backlog#820)
The READ_REPAIR_HEAL_CACHE owner-path guard in
check_architecture_migration_rules.sh pinned the cache to
set_disk/read.rs. P5 step 1 relocated the cache (and its accessor
helpers) to set_disk/core/io_primitives.rs, so allow that path too.
Guard intent is unchanged: the cache stays behind the ECStore set-disk
read-owner helpers.
---------
Co-authored-by: houseme <housemecn@gmail.com>
fix(internode): relax aggressive HTTP/2 keepalive & RPC timeouts to stop large-object GET truncation
The internode gRPC channel used a 3s HTTP/2 keepalive timeout and a 10s
overall RPC timeout. Under high-concurrency large-object reads a saturated
peer's PING ACK is legitimately delayed past 3s, so the whole channel (and
every RPC/stream on it) is torn down as a 'dead peer'. In-flight peer shard
reads then fail mid-object and large GETs truncate after headers+Content-Length
are already sent, surfacing to clients as 'download error: unexpected EOF'.
Reproduced on a 4-node erasure cluster with pure GET-only warp (no concurrent
writes) at 64 concurrency: 10MiB GET ~31 unexpected-EOF / 3min. Raising the
internode keepalive timeout (3s->30s via env) alone cut that to ~7; also raising
the RPC timeout cut it to ~3.
- Raise DEFAULT_INTERNODE_HTTP2_KEEPALIVE_TIMEOUT_SECS 3 -> 20
- Raise DEFAULT_INTERNODE_RPC_TIMEOUT_SECS 10 -> 30
- Wire the data-plane rio HttpReader keepalive timeout (was hardcoded 3s) to the
same env/default so control- and data-plane stay consistent.
Refs backlog#832.
P1 of the SetDisks split (tracking backlog#815, depends on P0 backlog#816).
Give the Heal operation family its own module home: relocate set_disk/heal.rs
to set_disk/ops/heal.rs and move the HealOperations storage-api contract impl
beside its inherent helpers. The contract stays implemented for SetDisks so its
associated-type bounds are unchanged (ecstore_contract_compat_test still
covers it).
Method bodies are moved unchanged. The four inherent helpers widen from
pub(super) to pub(in crate::set_disk) to preserve their exact prior visibility
from the deeper module. get_pool_and_set now reads topology through
SetDisksCtx to keep the Heal family aligned with the P0 borrow pattern; the
read is provably identical (ctx.format()/pool_index() alias the core fields).
Runtime behavior is unchanged.
P0 of the SetDisks split (tracking backlog#815). Introduce SetDisksCtx<'a>,
a Copy borrow handle over the shared SetDisks core state (topology/config,
disks, locker trio), and validate the pattern by routing the List family's
delete_all through a ListOperations<'a> service unit.
No trait impl is moved and runtime behavior is byte-for-byte unchanged; the
public SetDisks::delete_all entry point is preserved and delegates to the
borrow-based unit.
* fix(rio): reject corrupted short compressed/encrypted blocks instead of panicking
DecompressReader::poll_read and DecryptReader::poll_read sliced the block
body with a fixed `[0..16]` index to read the length varint. The body length
comes from an untrusted 24-bit header field, so a corrupted/truncated block
shorter than 16 bytes made the slice panic and crash the request task — a
read-path DoS on GET of tiered/corrupted data.
Pass the whole (arbitrary-length-safe) slice to uvarint and reject a
non-positive or out-of-range length prefix with InvalidData. Adds a repro
test for each reader; all existing round-trip tests still pass.
Refs rustfs/backlog#812
* fix(utils): close SSRF bypass via IPv4-mapped IPv6 addresses
validate_outbound_ip branched on the IpAddr variant, and the V6 branch's
is_loopback/is_unicast_link_local/is_unique_local checks never inspect the
embedded IPv4 of an IPv4-mapped address (::ffff:a.b.c.d). The metadata guard
also only matched the plain V4 169.254.169.254. So ::ffff:127.0.0.1,
::ffff:10.0.0.5 and ::ffff:169.254.169.254 all passed the outbound guard,
letting an attacker reach loopback/private/metadata endpoints.
Normalize IPv4-mapped IPv6 to its embedded IPv4 (via to_ipv4_mapped, which
matches only the true mapped form) before classification. Adds reject tests
for mapped loopback/private/metadata and an allow test for public IPv6.
Refs rustfs/backlog#813
* fix(ecstore): streaming last-part loss, GCS tier Range/remove, stat_all_dirs alignment
Four confirmed data-reliability defects:
- put_object_multipart_stream: the CompleteMultipartUpload part-collection loop
used exclusive `1..total_parts_count`, dropping the final part (and collecting
zero parts for a single-part object) — silently truncating the completed object.
Extracted collect_complete_parts (1..=total_parts_count) with unit tests.
- GCS warm backend get() ignored the requested byte range, returning the whole
object for a Range GET; now applies ReadRange::segment like the other backends.
- GCS warm backend remove() was an empty stub, so deleting a tiered object left
it on GCS forever; now deletes via StorageControl (added a control-plane client),
and in_use() actually lists (prefix-scoped) instead of always returning false.
- stat_all_dirs skipped None disk slots and dropped JoinErrors, returning a
compressed, misaligned error vector; heal_object_dir then zipped it against the
full disks array and could make_volume on the WRONG disk. Now returns one
index-aligned entry per slot (None -> DiskNotFound), and heal no longer
pre-fills the drive report (which would double it). Added an alignment test.
Refs rustfs/backlog#807
* fix(kms): stop Vault backend from destroying/reviving keys on failure
Two confirmed key-safety defects in the Vault KV2 backend:
- get_key_material() 'self-healed' a decrypt or wrong-length failure by minting a
fresh random master key and overwriting the stored value. That destroys the
original key material, making every DEK ever wrapped by it permanently
undecryptable. Decryption must never mutate the stored key: both branches now
return a cryptographic_error instead. (The empty-material bootstrap path, which
only fills a never-initialized key, is intentionally left intact.)
- cancel_key_deletion() reset key_state to Enabled only in the returned response
and never persisted it, so the key stayed PendingDeletion in storage and would
still be reaped. It now writes the state back via update_key_metadata_in_storage
and fails the request if the write fails.
Adds ignored (Vault-requiring) integration tests documenting both behaviours.
The third item (VaultTransit key state only in memory -> revived as Enabled after
restart) is deferred: a fail-closed guard would break restart availability for all
transit keys; the correct fix needs a persistent metadata store + Vault integration
testing. Tracked in rustfs/backlog#808.
Refs rustfs/backlog#808
* fix(admin): clamp STS AssumeRole duration; persist ImportBucketMetadata to disk
Two confirmed admin-API defects:
- Standard AssumeRole used the raw client-supplied DurationSeconds with no upper
bound, so a caller could mint near-permanent temporary credentials. Clamp it to
the AWS/MinIO STS window [900, 43200] (with 0 -> default 3600) via a shared
clamp_assume_role_duration helper, and build the exp claim with saturating_add.
This matches the existing AssumeRoleWithWebIdentity path.
- ImportBucketMetadata only mutated an in-memory map and returned 200, silently
dropping every imported config. It now persists each non-empty config via
metadata_sys::update (which merges onto existing on-disk metadata) and returns
InternalError if a write fails. Mapping extracted to imported_configs_to_persist
with unit tests.
Refs rustfs/backlog#809
* fix(heal): enqueue displacing request in release builds
push_displacing_lower_priority folded the real enqueue call into
debug_assert_eq!(self.push(request), Accepted). In release builds
(debug_assertions off) the whole macro — including its argument — is compiled
out, so after evicting a lower-priority queued item the new high-priority
request was silently dropped and never healed. Hoist self.push(request) out of
the assertion so the side effect runs in all builds. Adds a --release regression
test.
Refs rustfs/backlog#811
* fix(iam): propagate real delete_policy backend errors instead of swallowing them
delete_policy's is_from_notify path had its error handling inverted: a real
backend failure (disk IO / insufficient quorum) evicted the cache and returned
Ok(()), reporting a phantom success while policy.json survived on disk (to be
reloaded on the next full IAM reload); NoSuchPolicy — which should be idempotent
success — returned Err. Propagate real errors and let NoSuchPolicy fall through
to the idempotent cache-evict + Ok, matching delete_user / the notification
handler in the same file. Adds a backend-error-injection regression test.
Refs rustfs/backlog#810
* fix(utils): also normalize IPv4-compatible IPv6 in the SSRF guard
The initial fix only unwrapped IPv4-mapped (::ffff:a.b.c.d) addresses; the
deprecated IPv4-compatible form (::a.b.c.d, e.g. ::127.0.0.1 / ::169.254.169.254)
still bypassed the guard. Reject pure-IPv6 specials (::, ::1, fe80::, fc00::)
first, then normalize BOTH embedded-IPv4 forms before the IPv4 rules. Adds tests
for compatible-form loopback/metadata and confirms ::1 / :: stay rejected.
Found by adversarial review of the initial fix. Refs rustfs/backlog#813
* fix(ecstore): fix the same last-part loss in the parallel streaming path
put_object_multipart_stream_parallel had the identical off-by-one
(1..total_parts_count) that truncated the last part / produced zero parts for a
single-part upload — reachable when concurrent stream parts are enabled. Reuse
collect_complete_parts, which now returns an error instead of panicking on a gap
in the parts map. Adds a missing-part error test.
Found by adversarial review of the initial fix. Refs rustfs/backlog#807
* fix(kms): local backend must preserve key material on status change
LocalKmsClient (the default KMS backend) regenerated the master key material on
enable_key/disable_key/schedule_key_deletion/cancel_key_deletion — a pure status
change. A single disable+enable cycle therefore destroyed the original key,
making every DEK ever wrapped by it permanently undecryptable (silent data loss,
no network needed). Preserve the existing material via get_key_material and
re-save with only the status changed. Adds a hermetic regression test that wraps
a DEK, cycles all four status methods, and asserts the DEK still decrypts.
Found by adversarial review of the Vault fix. Refs rustfs/backlog#808
* test(rio): cover the length-prefix guard; correct its comment
Add a DecompressReader test that feeds an unterminated length varint so uvarint
returns 0 and the new guard (not the downstream codec) produces the InvalidData
error, and reword the guard comment which overclaimed that the > len bound
prevents a reachable panic (it is belt-and-suspenders). No behavior change.
Found by adversarial review. Refs rustfs/backlog#812
* test(rio): build test block headers via vec! to satisfy clippy
The new corrupted-block tests built the header with Vec::new() + repeated push,
tripping clippy::vec_init_then_push (-D warnings in CI). Construct the fixed
header bytes with vec![] instead. No behavior change.
---------
Co-authored-by: houseme <housemecn@gmail.com>
* fix(object-capacity): stop cancelled/remote-disk refreshes from corrupting capacity
Two independent capacity-refresh bugs (backlog rustfs/backlog#805):
- refresh_or_join set the singleflight `running` flag then awaited the
refresh future with no drop guard. When the admin request that became
leader was cancelled (client disconnect) mid-await, `running` stayed true
forever: joiners blocked indefinitely and the 120s scheduled refresh could
never start again. catch_unwind covered panics but not cancellation. A
RefreshLeaderGuard now resets the state and publishes an error on drop.
- capacity_disk_refs mapped the cluster-wide storage_info disk list without
filtering non-local disks, so admin-triggered refreshes ran a local WalkDir
over remote disks' drive_path. On multi-node clusters this double-counted
local bytes (shared mount layout) or hit NotFound (per-node layouts), and
poisoned the per-disk cache the scheduled local-only refresh depends on,
making the cached total oscillate. Filter to local disks.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(rio): harden internode HTTP client build, cache, and PUT body integrity
Follow-ups from the internode HTTP review (backlog rustfs/backlog#805):
- handle_put_file accepted a truncated body as success: a HttpWriter dropped
mid-stream closes the chunked body cleanly, indistinguishable from EOF, and
the server never compared bytes copied against the declared size. Reject
size mismatches on the create path (append/unknown-size writes send size<=0
and are exempt).
- build_http_client used `.expect()` on ClientBuilder::build(), which runs
lazily on the first request and on every TLS generation bump (cert
rotation), so a build failure panicked a serving task. It now returns an
io::Error; get_http_client falls back to the previous TLS generation when a
rebuild fails instead of failing the request.
- CLIENT_CACHE was a tokio::Mutex taken on every stream open (data_shards
times per GET). Replaced with arc_swap::ArcSwapOption for lock-free reads on
the hot path; the generation-monotonic replacement guard is preserved via rcu.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
perf(get,heal): land verified fixes for backlog #800-#804
Five fixes from the GET-path performance audit and scanner/heal
completeness audit (rustfs/backlog#800..#804), each verified locally:
- backlog#800 (heal checkpoint O(N^2)): ResumeCheckpoint object sets are
now HashSet (Vec::contains was O(n) per healed object; 1.5ms at n=1M
vs 11ns measured), and per-object checkpoint/resume-state persistence
is batched (1000 mutations / 5s) instead of rewriting the whole file
per object. complete_page() prunes the sets at page boundaries so
memory stays bounded; positions still persist unconditionally and
legacy Vec-format checkpoints still deserialize.
- backlog#801 (DiskInfo.healing never set): erasure-set heal now writes
a healing marker (.rustfs.sys/healing.bin) on the disks it rebuilds
(endpoints plumbed via HealRequest/HealTask.heal_endpoints from the
auto disk scanner) and clears it on success. LocalDisk::disk_info
surfaces the marker, so scanner heal coordination, lock selection and
admin/metrics healing counts see the rebuild.
- backlog#802 (cache probe after data read): new GetObjectBodyCacheHook
in ecstore lets the app-layer object data cache serve the body inside
get_object_reader, after metadata quorum resolution (etag known) but
before the erasure shard read/decode. Previously a hit still paid the
full disk read. Hook is None/no-op when the cache is disabled.
- backlog#803 (GET hot-path redundant work): ObjectInfo is cloned for
event notification only when an event will actually be built (GET and
HEAD paths; events are currently suppressed so the clone was pure
waste); get_opts/put_opts/del_opts resolve bucket versioning with one
metadata-sys lookup instead of two; skip_verify_bitrot and
get_lock_acquire_timeout env reads are cached via OnceLock; the
io-priority metric is no longer double-counted; GetObject input fields
are cloned selectively instead of cloning the whole input.
- backlog#804 (disk permit starvation): the disk-read permit wait is now
bounded (RUSTFS_OBJECT_DISK_PERMIT_WAIT_TIMEOUT, default 5s, 0 =
previous unbounded behavior); on timeout the GET proceeds without a
permit and the bypass is counted. DiskReadPermitReader also releases
the permit at body EOF instead of holding it until the client drops
the stream.
Verification: make pre-commit; cargo clippy -D warnings on the four
changed crates; full rustfs lib suite (2096 tests) green; rustfs-heal
lib suite green with new unit tests for checkpoint pruning/legacy
format/throttle, permit EOF release, and the cache hook (hit + SSE
skip). The heal_integration_test and one set_disk listing test fail
identically on unmodified main (pre-existing global-state ordering
flakes, verified via git stash A/B).
Co-authored-by: houseme <housemecn@gmail.com>
* feat(cache): add object data cache engine
* feat(cache): wire app-layer object cache flow
* refactor(cache): streamline app-layer cache flow
* refactor(cache): tighten cache flow internals
* refactor: address final clippy cleanup
* chore(deps): update quick-xml to 0.41.0
* feat(cache): wire object data cache env config
* fix(cache): gate materialize fill by cache plan
* chore(cache): add object data cache benchmark gate
* fix(cache): guard object cache fill size mismatches
* refactor(cache): streamline object cache body planning
* fix(cache): align object cache rollout config
* test(cache): cover buffered object cache benchmark
* test(cache): isolate object cache benchmark metrics
* test(cache): mark materialize rollout experimental
* test(cache): tighten object cache benchmark gate
* fix(cache): address review findings for object data cache
- singleflight: clean up leader entry on cancellation (Drop impl) so a dropped GET future can no longer wedge all subsequent fills for the same key; switch the fill map to a std Mutex and add a regression test
- adapter: honor RUSTFS_OBJECT_DATA_CACHE_ENABLE=true by defaulting to hit_only when no explicit mode is set (explicit mode still wins)
- planner: treat nil version UUIDs as "no value" per repo convention so unversioned objects key under the canonical "null" instead of fragmenting the key space
- multipart: invalidate the object cache on the quota-exceeded rollback delete after complete-multipart, closing a stale-cache window
- layering: move the disabled-cache fallback into app::context and drop the new infra->app layer-dependency baseline entry
* fix(cache): close invalidation races and drop full-cache scan on writes
- index: make identity-index insert/remove/prune atomic via starshard compute_if_present/compute_if_absent so concurrent fills can no longer drop each other's keys (lost keys made entries unreachable to invalidation until TTL); add a concurrency regression test
- fill: register the key in the identity index before the entry becomes visible in the cache and re-check the index afterwards, undoing the fill when an invalidation raced in between (new skipped_invalidation_race fill result)
- invalidate: with the index now authoritative, remove the full-cache iter() fallback that made every PUT/DELETE of a never-cached object O(total cache entries) (two scans per PUT, 2N per batch delete)
- materialize-fill: fail the GET instead of falling back to the partially consumed stream after a mid-read error (the fallback would send a body missing its prefix under a full-length Content-Length), and log the same size-mismatch warning as the sibling buffering paths
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(storage): fix media-dependent buffer clamp expectation
test_concurrency_manager_multi_factor_strategy_buffer_clamp asserted media_cap.min(MI_B), but the implementation's final safety clamp is [32KiB, media_cap.max(MI_B)] — deliberately so a media cap above 1MiB (NVMe's 2MiB default) stays effective. The test only passed on machines detected as SSD/Unknown (cap == 1MiB) and failed on NVMe-backed CI runners with 2MiB != 1MiB. Assert the media cap itself, which is what the strategy actually guarantees on every environment.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(storage): format buffer clamp assertion
* chore(logging): update tier guardrail path
---------
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: cxymds <cxymds@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(rio): propagate http writer shutdown errors
* fix(ecstore): unify remote lock rpc deadlines
* fix(storage): reject corrupt read multiple payloads
* feat(rio): add internode http tuning profiles
* feat(metrics): add internode baseline signals
* feat(ecstore): observe shard locality topology
* feat(ecstore): gate shard locality scheduling
* feat(ecstore): gate batch read version rpc
* feat(ecstore): observe batch processor adaptation
* feat(ecstore): gate batch processor observation
* docs: add get benchmark regression analysis
* docs: add issue 797 execution plan status
* fix(ecstore): require explicit batch rpc support
* fix(ecstore): honor documented batch read gate
* fix(ecstore): keep batch read gate stable per call
* chore: update workspace dependencies
* feat(ecstore): log batch read gate decisions
* feat(ecstore): count batch read gate decisions
* test(issue-797): add local internode A/B runner
* test(rio): fix tuning profile spelling fixture
* fix(protocols): adapt sftp channel open callbacks
* fix(metrics): wrap batch processor observation args
* chore(docs): keep issue notes local only
* fix(storage): address internode review feedback
* fix(storage): address internode data-path review findings
- Run the BatchReadVersion auto-mode unary fallback outside the batch
RPC deadline so each read_version keeps its own per-op timeout and
health accounting instead of racing the whole batch against one
drive timeout.
- Cap adaptive batch-processor concurrency growth at a hard multiple
of the configured baseline so sustained fast batches cannot ratchet
past the configured limit.
- Parse RUSTFS_INTERNODE_HTTP_* tuning, RUSTFS_BATCH_PROCESSOR_ADAPTIVE,
and RUSTFS_METADATA_BATCH_READ once per process instead of re-reading
the environment on hot paths.
- Skip shard read-cost collection in observe mode when stage metrics
are disabled, and cache the local endpoint host list instead of
rebuilding it on every read.
- Allow --warp-extra-args values starting with -- and drop the unused
warp_hosts_csv helper in the issue-797 A/B runner.
* fix(storage): address internode data-path review findings
- Run the BatchReadVersion auto-mode unary fallback outside the batch
RPC deadline so each read_version keeps its own per-op timeout and
health accounting instead of racing the whole batch against one
drive timeout.
- Cap adaptive batch-processor concurrency growth at a hard multiple
of the configured baseline so sustained fast batches cannot ratchet
past the configured limit.
- Parse RUSTFS_INTERNODE_HTTP_* tuning, RUSTFS_BATCH_PROCESSOR_ADAPTIVE,
and RUSTFS_METADATA_BATCH_READ once per process instead of re-reading
the environment on hot paths.
- Skip shard read-cost collection in observe mode when stage metrics
are disabled, and cache the local endpoint host list instead of
rebuilding it on every read.
- Allow --warp-extra-args values starting with -- and drop the unused
warp_hosts_csv helper in the issue-797 A/B runner.
Co-Authored-By: heihutu<heihutu@gmail.com>
* fix(storage): align buffer clamp test with media cap
* fix(ecstore): release optimized read locks before streaming
---------
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Add crates/e2e_test/src/chaos.rs with in-process fault-injection
primitives for a single-node multi-disk RustFS server: take a disk
offline (rename to <dir>.offline), bring it back online, replace a
disk with a fresh empty directory, corrupt an object's erasure shard
(part.* byte flips, xl.meta untouched), and SIGKILL/restart the server
with the same volumes and port.
Add reliability tests on a 4-disk (EC 2+2) topology, verified via
sha256 manifests recorded at write time:
- degraded read/write with one disk offline (incl. multipart object)
- bitrot read-through with two corrupted shards per object
- fresh-disk replacement healed via admin deep heal after a SIGKILL
restart
Also reuse the shared signed_admin_post helper from chaos.rs in the
heal regression suite instead of a duplicated local copy.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(core-storage): fix critical correctness defects from core-storage audit
Fixes verified defects found in a deep audit of the core storage path
(erasure coding, disk persistence, quorum, heal, replication resync):
- ecstore/disk: rewrite live xl.meta atomically (temp+rename) in
delete_versions_internal and write_metadata instead of in-place
truncate, which exposed torn metadata to concurrent readers and
crashes on the DeleteObjects hot path
- ecstore/erasure: allow heal to reconstruct from exactly data_shards
bitrot-verified sources; requiring data_shards+1 made objects
permanently unhealable after losing parity_shards disks
- ecstore/set_disk: direct-memory inline GET applied the erasure
distribution permutation twice (shuffled inputs re-indexed through
distribution), concatenating wrong shards into the response body in
degraded reads; collect from canonical disk-ordered inputs
- ecstore/set_disk: heal now preserves the committed inline layout
instead of recomputing it with a hardcoded unversioned threshold,
which split quorum identity of healed replicas and caused endless
re-heal churn
- ecstore/replication: resync results channel switched from
broadcast(1) to mpsc; a lagged broadcast receiver ended the stats
collector and every subsequent failure went uncounted, letting
failed resyncs be marked completed
- ecstore/replication: ignore an empty persisted resync checkpoint;
resuming with one skipped every object and marked the resync
completed without replicating anything
- ecstore/replication: fix inverted not-found error classification in
replicate_object/replicate_delete logging paths
- ecstore/erasure: guard decode paths against zero block_size or
data_shards from corrupt on-disk metadata (divide-by-zero panic)
- ecstore/disk: os::read_dir no longer consumes the entry limit on
entries it does not return (is_empty_dir misjudgment); create_file
opens with O_TRUNC to avoid stale trailing bytes
- filemeta: treat Some(nil) version id as a null version in
matches_not_strict; disk-loaded headers never store None, so the
mod_time quorum guard for unversioned overwrites never fired and an
interrupted overwrite could displace the committed version in merge
- filemeta: fix msgpack skip lengths for fixext (missed the ext type
byte) and ext16/32 (over-skipped) unknown fields
- filemeta: return FileCorrupt instead of usize underflow when
xl.meta is truncated inside the CRC trailer
- filemeta: surface delete-marker insertion failure in delete_version
instead of reporting success when the data dir is shared
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(replication): drop duplicate cfg(test) etag import from boundary module
The test module already imports content_matches_by_etag locally, so the
top-level cfg(test) import is unused under -D warnings and fails clippy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
fix(ecstore): remove unused cfg(test) import of content_matches_by_etag
The #[cfg(test)] use of content_matches_by_etag at module level is
redundant — the test module already imports it locally at line 369.
This was causing a clippy unused-imports error under -D warnings.
fix(ecstore): purge orphan empty-directory trees on folder delete (#4189)
Empty "phantom" folders — on-disk directory trees with no xl.meta
anywhere — show up in listings as common prefixes but cannot be removed
by any S3 call: DeleteObject on the folder key encodes it to __XLDIR__,
finds no metadata, and returns NotFound, which the API layer masks as a
fake 204 while the tree survives on disk.
Make handle_delete_object attempt a guarded purge when a dir-object key
resolves to NotFound:
- sweep every erasure set of every pool (orphan fragments can sit on
any set, left behind by whichever sets stored the deleted children)
- per set, refuse if ANY disk holds a regular file under the prefix,
so degraded/healable objects are never destroyed
- remove empty directories children-first with non-recursive deletes;
a racing PutObject makes the rmdir fail with DirectoryNotEmpty and
the entry is skipped
This state is unrepresentable on AWS S3 (CommonPrefixes derive only
from real keys; DeleteObject is an idempotent 204), so the purge moves
visible behavior closer to S3. It deliberately goes beyond MinIO, whose
DeleteObject leaves these trees stranded and whose heal only removes
dangling (minority-presence) dirs — a long-standing complaint
(minio/minio#15257, #19516, #18444).
Also move the test-only content_matches_by_etag import in
replication_target_boundary.rs into the cfg(test) module; it failed
cargo clippy -D warnings and blocked the pre-PR gate.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
content_matches_by_etag is only used inside the #[cfg(test)] module, so
the lib-scope import from #4211 fails cargo clippy -D warnings on every
non-test build.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
fix(test): use subset check for capacity dirty scope test to handle global state contamination
The data_movement_put_object_marks_dirty_disks_for_capacity_manager test
asserts exact dirty disk count, but the global dirty scope registry is
process-wide. Other concurrent tests can add entries between the drain
and the assertion, causing flaky failures (8 instead of expected 4).
Use subset assertion to verify the expected disks are present without
being sensitive to entries from other tests.
Remove or consolidate 57 test cases that cannot catch regressions
(literal-constant asserts, construct-then-assert, derived-serde
round-trips, near-duplicate env/getter matrices) in common, config,
iam, madmin, and object-capacity, keeping all wire-format and
error-path guards. Add 13 tests for previously uncovered high-risk
behavior: filemeta version-sort determinism and merge resilience to
garbage headers, zip extraction path-traversal rejection and exact
limit boundaries, JWT tampered-signature rejection, and the bytes
variant of dual-key (rustfs/minio) metadata fallback and precedence.
Test-only change; no production code touched.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(ecstore): remove reachable panics in tiering, replication, and heal paths
- Parse x-amz-expiration leniently in tier PUT responses; any lifecycle
rule on the remote tier bucket returns an RFC1123 date that the previous
ISO8601 unwrap turned into a panic of the ILM transition worker
- Skip invalid user-metadata header values (with a warning) when building
tier and replication PUT headers instead of panicking on non-ASCII input
- Heal: tolerate absent data_dir for delete markers and remote objects
- transition_object: don't unwrap version_id on unversioned buckets when
recording partial writes for offline disks
- Admin server info: use port_or_known_default() so default-port (80/443)
endpoints don't panic is_server_resolvable
- Tier ListObjectsV2 client: decode response body with from_utf8_lossy
- walk_internal: log merge_entry_channels errors instead of dropping them
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(ecstore): fail heal explicitly when data_dir is missing
Address review feedback: unwrap_or_default() silently substituted a nil
UUID when latest metadata lacked data_dir. Delete markers and remote
objects legitimately have no data_dir and skip the data-heal block, but
for a regular object a missing data_dir means corrupt metadata — return
FileCorrupt with a descriptive log instead of building part paths under
a nil UUID directory.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Follow-ups to the compatibility harness rework:
- Weekly full sweep now runs as a matrix over both topologies: single
node and the 4-node distributed cluster. Manual dispatch keeps the
test-mode input.
- New mint workflow (weekly + dispatch) runs MinIO Mint against RustFS:
functional suites of real client SDKs and tools (awscli, mc,
aws-sdk-*, minio-*, s3cmd, ...), catching client-specific signing and
streaming edge cases that boto3-only ceph/s3-tests cannot. Report-only
for test failures with a per-suite summary table; fails only when mint
produces no results.
- New scripts/s3-tests/api_coverage.py quantifies API surface coverage
by diffing the s3s S3 trait (at the Cargo.toml pinned revision)
against the methods overridden in impl S3 for FS, distinguishing
NotImplemented defaults from delegating defaults (e.g. post_object).
Current state: 76/101 operations covered (75 overridden, 1 delegated).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Reworks the S3 compatibility test harness for reproducibility and faster
feedback:
- Pin ceph/s3-tests to a fixed commit (S3TESTS_REV, fetch-by-SHA) so the
449-test PR gate is reproducible; previously every run cloned upstream
master, letting test renames or assertion changes break CI silently.
- PR gate (ci.yml s3-implemented-tests): MAXFAIL=0 + XDIST=4 so a single
CI round reports every failure in parallel instead of stopping at the
first one serially.
- Add TEST_SCOPE=all to run.sh to run the entire upstream suite, and
report_compat.py to diff junit results against the classification
lists (regressions, promotion candidates, unclassified tests).
- Rewrite e2e-s3tests.yml: delegate execution to run.sh (single source
of truth; also fixes the broken config generation that left S3_PORT
empty), add a weekly scheduled full sweep that fails only on whitelist
regressions, and fix the multi-node topology to a real distributed
cluster (endpoint-style RUSTFS_VOLUMES) instead of four independent
single-node stores behind a load balancer.
- Docs: rewrite stale .github/s3tests/README.md (marker-era strategy),
update scripts/s3-tests/README.md, fix dead build_testexpr.sh
reference in S3_COMPAT_WORKFLOW.md, drop legacy non_standard_tests.txt.
All 747 classified test names verified present at the pinned revision;
13 upstream tests are currently unclassified and will surface in the
first full-sweep report.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Agent-instruction and architecture docs had drifted from the code:
- CLAUDE.md: slim to commands + pointers; fix wrong claim that
`make pre-commit` is the full pre-PR gate (that is `make pre-pr`);
drop stale pre-#3929 file paths and merged bug narratives
- AGENTS.md: drop dead `rust-refactor-helper` skill rule and the
hand-maintained (already stale) scoped-AGENTS index; link
architecture docs from Sources of Truth
- .github/AGENTS.md: replace the outdated copied CI command matrix
with a pointer to ci.yml
- crates/AGENTS.md: merge duplicated Testing sections
- ARCHITECTURE.md: resolve the utils->config contradiction (edges are
removed), mark volatile counts as snapshots, fix a bad path
- docs/architecture: add README router; move one-shot plans/trackers
(rebalance-decommission phases, migration-progress ledger, PR
template) to docs/superpowers/plans with archive headers; fix stale
source paths in kept inventories (core/sets.rs, core/pools.rs,
store/mod.rs, startup_* split from #3671)
- docs/operations/tier-ilm-debugging.md: extracted tier debugging
playbook with corrected paths
- scripts/check_doc_paths.sh: new guard failing pre-commit/pre-pr when
instruction/architecture docs reference nonexistent file paths
- .claude/skills: add tier-debug and arch-checks repo skills;
.gitignore now keeps .claude/skills and docs/operations committable
Verification: ./scripts/check_doc_paths.sh,
./scripts/check_architecture_migration_rules.sh (both pass)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Speeds up the CI pipeline without changing what is tested:
- Drop the global CARGO_BUILD_JOBS=2 and per-job --jobs 2 flags, which
halved compiler parallelism on the 4-core runners.
- Stop embedding hashFiles(Cargo.lock) in rust-cache shared keys. The
action already fingerprints lockfiles internally; putting the hash in
the key prefix meant any PR that touched Cargo.lock started from a
completely cold cache instead of reusing unchanged dependencies.
- Give the e2e-tests job the full setup action with dependency caching.
Its migration-proof step compiles the e2e_test crate (which pulls in
most of the workspace) and previously did so cold on every run with
no cache, on a 2-core runner.
- Set profile.dev debug = "line-tables-only": keeps usable backtraces
while cutting compile/link time, target-dir size (faster cache
save/restore), and debug-binary artifact upload/download.
- Split fmt + repo-script checks into a compile-free quick-checks job
on ubuntu-latest so contributors get feedback in ~1 minute instead
of after the full test run.
- Collapse the five per-filter migration-proof cargo test reruns into
one filtered nextest invocation; the same tests already run in the
full nextest pass.
- Remove the touch rustfs/build.rs + forced rebuild from the debug
binary jobs (version stamping is irrelevant for e2e binaries) and
the unreachable Windows cross-compile branch in build.yml.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
fix: simplify match arms to satisfy clippy manual-unwrap-or-default and manual-unwrap-or
Two match expressions in crates/replication/src/runtime.rs triggered
clippy lints (manual-unwrap-or-default, manual-unwrap-or) with the
project's -D warnings policy. Replace them with the idiomatic
.unwrap_or_default() and .unwrap_or() calls.
* fix(site-replication): Add Docker Compose setup for site replication testing
- Fixed the site replication flag issue and resolved the replication storm.
- Introduced a new directory for site replication tests with Docker Compose.
- Created `docker-compose.yml` to define three RustFS sites and a setup container.
- Added `README.md` to document the purpose, usage, and test flow of the replication setup.
- Implemented `run-object-flow-check.sh` script to verify object replication across sites.
- Configured health checks and volume permissions for the RustFS containers.
- Enabled customization of access keys, bucket names, and other parameters via environment variables.
* fix
* fix
* feat(get): SF01 - bucket validation cache
Add 5s TTL cache for bucket validation to avoid repeated stat_volume()
calls on every GET request.
Changes:
- Add BUCKET_VALIDATED_CACHE (OnceLock + RwLock + HashMap)
- Add invalidate_bucket_validation_cache() for cache invalidation
- Add invalidate_all_bucket_validation_cache() for bulk invalidation
- Update get_validated_store() to use cache
- Add cache invalidation in execute_delete_bucket()
Expected impact: 3-5x improvement for small file GET latency.
Closesrustfs/backlog#766
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(get): SF03 - metadata cache TTL increase
Increase metadata cache TTL from 250ms to 2s and capacity from 1024
to 4096 entries.
Changes:
- GET_OBJECT_METADATA_CACHE_TTL: 250ms -> 2s
- GET_OBJECT_METADATA_CACHE_MAX_ENTRIES: 1024 -> 4096
All mutation paths already call invalidate_get_object_metadata_cache,
so the longer TTL is safe.
Expected impact: 10-50x improvement for hot objects.
Closesrustfs/backlog#768
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(get): SF04 - remove unnecessary tokio::spawn in metadata fanout
Replace tokio::spawn with direct async future in read_all_fileinfo_full_wait.
join_all already provides concurrency, so tokio::spawn adds unnecessary
task creation and scheduling overhead.
Changes:
- Remove tokio::spawn from metadata fanout futures
- Update result handling for direct future results
Expected impact: 16-32us reduction per GET request.
Closesrustfs/backlog#769
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(get): SF06 - conditional lifecycle check
Only call resolve_put_object_expiration when the object has an
x-amz-expiration metadata marker. This avoids unnecessary lifecycle
configuration reads on every GET request.
Expected impact: 50-100us reduction per GET request.
Closesrustfs/backlog#771
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(get): SF07 - conditional metrics recording
Gate hot path metrics behind get_stage_metrics_enabled() to reduce
overhead when metrics are not needed.
Changes:
- Conditional record_zero_copy_read
- Conditional manager.record_disk_operation
- Conditional manager.record_access
- Conditional manager.record_transfer
Expected impact: 20-50us reduction per GET request.
Closesrustfs/backlog#772
Co-Authored-By: heihutu <heihutu@gmail.com>
* refactor(get): SF01 - use moka instead of dashmap for bucket cache
Replace OnceLock + RwLock + HashMap with moka::sync::Cache for bucket
validation cache. moka provides built-in TTL support and is already
available in the workspace.
Changes:
- Add moka dependency to rustfs crate
- Replace manual TTL management with moka's time_to_live
- Simplify cache operations
Closesrustfs/backlog#766
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(get): SF02 - inline data fast path
Add fast path for small inline objects that bypasses duplex pipe,
tokio::spawn, and bitrot reader creation when data is already in memory.
Changes:
- Add inline data detection before codec streaming gate
- Direct in-memory erasure decode for inline objects <= 128KB
- Add GET_OBJECT_PATH_INLINE_DIRECT metric path
- Skip duplex pipe and background task for inline data
Conditions for fast path:
- Single part object
- Inline data available
- Size <= 128KB
- Not encrypted/compressed/remote
- No range request
Expected impact: 2-3x improvement for small file GET latency.
Closesrustfs/backlog#767
Co-Authored-By: heihutu <heihutu@gmail.com>
* refactor: translate Chinese comments to English
Translate all Chinese comments to English in modified files:
- rustfs/src/storage/ecfs_extend.rs
- rustfs/src/app/bucket_usecase.rs
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix
* add
* fmt and improve import
* fmt
* feat(get): SF05 skip IO planning + refactor inline detection + adaptive bucket cache
SF05: Skip disk I/O semaphore for inline data fast path
- Reorder prepare_get_object_read_execution: read first, then decide semaphore
- Inline objects skip acquire_disk_read_permit() entirely (saves 100-200us)
- Add is_inline_fast_path field to GetObjectReadSetup
Refactor: Unify inline detection logic
- Add ObjectInfo::is_inline_fast_path_eligible() as single source of truth
- Version-aware thresholds: non-versioned 128KB, versioned 16KB (matches PUT)
- Eliminates divergent conditions between set_disk/mod.rs and object_usecase.rs
Refactor: Restore fault tolerance in metadata fanout
- Restore tokio::spawn + JoinError handling in read_all_fileinfo_full_wait
- Prevents single disk read panic from unwinding the entire operation
Refactor: Restore lifecycle check correctness
- Remove incorrect SF06 conditional that skipped lifecycle for most objects
- Always call resolve_put_object_expiration (original behavior)
Fix: make_bucket cache invalidation
- Invalidate bucket validation cache on create_bucket
Fix: erasure decode written validation
- Check decode() return value; error if 0 bytes written for non-empty object
Adaptive bucket cache
- Default: RwLock<HashMap> for < 100 buckets (low overhead)
- Opt-in: starshard::ShardedHashMap via RUSTFS_BUCKET_CACHE_STARSHARD=1
- 5s TTL with manual timestamp checking
Benchmark results (warp get, concurrency 32, 10s, 3 rounds):
- 10KiB: 25.10 MiB/s (+28.2% vs SF01-07)
- 100KiB: 221.81 MiB/s
- 1MiB: 1972.78 MiB/s
- vs main: -10% to -12% (inline path not triggered by warp)
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(versioning): use read lock for versioning config query + five-expert analysis
P0 fix: BucketVersioningSys::get() was using write lock on
GLOBAL_BucketMetadataSys for a pure read operation. This serialized
all concurrent GET requests (3 write-lock acquisitions per request).
Changed to read lock — get_versioning_config() handles its own
internal locking via metadata_map RwLock.
Five-expert analysis identified top bottlenecks:
1. Versioning write lock (P0, fixed)
2. Inline fast path not triggered (P0, needs verification)
3. Metadata fanout no early-stop (P1, early-stop has bug, reverted)
4. Request-level versioning cache (P1, pending)
5. Duplex pipe for small objects (P2, pending)
Benchmark (read-lock fix, warp concurrency 32):
- 1KiB: 2.29 MiB/s (vs 2.53 before, within variance)
- 10KiB: 25.00 MiB/s (same as before)
- 100KiB: 246.72 MiB/s (+11% vs 221.81)
- 1MiB: 2039.95 MiB/s (+3% vs 1972.78)
Co-Authored-By: heihutu <heihutu@gmail.com>
* chore: remove benchmark results from git, keep locally only
Remove docs/benchmark/*.md from version control.
Files remain on disk but are no longer tracked by git.
Added docs/benchmark/*.md to .gitignore.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(get): decode inline fast path through bitrot readers
---------
Co-authored-by: heihutu <heihutu@gmail.com>
docs: document global singleton migration plan
Add comments documenting the Tier A/B classification and migration
plan for global singletons in ecstore runtime.
Refs #730
* docs: document crypto RC version dependencies
Add comment explaining why aes-gcm and chacha20poly1305 use RC
versions and the migration path when stable versions are released.
Refs #732
* fix: remove all backlog links from code and comments
docs: document obs reverse dependency on ecstore
Add comment explaining why obs depends on ecstore and the scope
of work required to break this dependency.
Refs #735
Co-authored-by: houseme <housemecn@gmail.com>
docs: add documentation to storage-api public types
Add doc comments to public structs, enums, and traits in
storage-api crate to improve documentation coverage.
Refs #741
docs: document table_catalog mutex design rationale
Add comment explaining why the single mutex is intentional and
what alternatives to consider if contention becomes a bottleneck.
Refs #739
* feat: add rate limiting middleware framework
Add rate limiting middleware with token bucket algorithm for
per-client request rate limiting. This provides the foundation
for DoS protection.
Refs #737
* fix: address clippy lints in rate_limit.rs
- Collapse nested if statements into single if-let chains
- Use .is_multiple_of() instead of manual modulo check
test: add insta snapshot test for storage error display format
Add snapshot test to detect unexpected changes in StorageError
display format. This catches output format regressions that
traditional assert tests might miss.
Refs #740
* docs(storage-api): document filemeta dependency as known limitation
Add comment explaining why storage-api depends on filemeta and
the scope of work required to break this dependency (300+ files).
Refs https://github.com/rustfs/backlog/issues/731
* docs(storage-api): remove backlog link from comment
fix(ecstore): replace unwrap() with proper error handling in api_get_object_attributes
Replace unsafe unwrap() calls with proper error handling in
api_get_object_attributes.rs:
- HTTP header access now uses ok_or_else with descriptive messages
- String parsing now uses map_err with descriptive messages
- HeaderValue creation now uses expect with descriptive messages
Refs https://github.com/rustfs/backlog/issues/729
fix(ecstore): improve expect() messages in admin_server_info
Replace unwrap() with expect() for better error diagnostics in
admin_server_info.rs:
- URL host/port access now has descriptive messages
- HashMap get_mut calls now have descriptive messages
Refs https://github.com/rustfs/backlog/issues/729
fix(server): improve expect() messages in layer.rs
Replace unwrap() with expect(valid response body) for Response
builder calls in layer.rs.
Refs https://github.com/rustfs/backlog/issues/729
fix(ecstore): improve expect() messages in replication_resyncer
Replace unwrap() with expect() for better error diagnostics in
replication_resyncer.rs:
- HashMap get_mut calls now have descriptive expect messages
- format() calls now use unwrap_or_else for error handling
Refs https://github.com/rustfs/backlog/issues/729
fix(admin): improve expect() messages in user handler
Replace generic parse().unwrap() with parse().expect(valid header value)
for better error diagnostics in user.rs.
Refs https://github.com/rustfs/backlog/issues/729
fix(admin): replace unwrap() with safe pattern in bucket_meta handler
Replace 11 instances of HashMap.get_mut().unwrap() with
match pattern that continues to next iteration if key is missing.
Also improve expect() messages for header value parsing.
Refs https://github.com/rustfs/backlog/issues/729
fix(admin): replace unwrap() with proper error handling in tier handler
Replace 9 instances of args.{type}.clone().unwrap() with
ok_or_else() that returns a descriptive S3Error when the
tier configuration is missing.
Also improve expect() messages for header value parsing.
Refs https://github.com/rustfs/backlog/issues/729
fix(ecstore): replace k.unwrap() with safe pattern in bucket_target_sys
Replace unsafe k.unwrap().as_str() with if let Some(key_str) pattern
in 5 locations where HeaderMap iterator yields (Option<HeaderName>, Value).
This prevents potential panics if header names are invalid.
Refs https://github.com/rustfs/backlog/issues/729
revert: restore #![allow(dead_code)] - clippy -D warnings treats warn as error
The #742 PR changed #![allow(dead_code)] to #![warn(dead_code)], but
CI runs clippy with -D warnings which turns warnings into errors.
This caused CI failures across multiple PRs.
Reverting to #![allow(dead_code)] until the dead code is actually
cleaned up. The 189 warnings in ecstore should be fixed incrementally
by deleting dead code and adding item-level allows, not by changing
the crate-level policy.
fix(ecstore): replace unbounded metadata cache with moka
Replace the manual Arc<RwLock<HashMap>> metadata cache with
moka::future::Cache, which provides:
- Built-in LRU eviction when max_capacity is reached
- Automatic TTL expiry via time_to_live (250ms)
- Lock-free concurrent reads
- Non-blocking invalidation
Fixes the memory leak risk from unbounded HashMap and the
all-or-nothing eviction logic that cleared all entries at once.
Closes#743
Co-authored-by: houseme <housemecn@gmail.com>
* fix(ecstore): preserve multipart part quorum errors
* fix(ecstore): honor confirmed missing part quorum
* fix(ecstore): add multipart quorum diagnostics
* feat(replication): support insecure https bucket targets
* feat(replication): support custom ca bucket targets
* test(replication): satisfy e2e clippy for TLS helpers
* fix(replication): avoid native root panic for custom trust stores
* test(replication): decouple private IP target test from TLS roots
* test(replication): use target TLS client in private IP unit test
* refactor(runtime): route readiness lock clients via app context
* refactor(admin): route site replication iam via app context
* refactor(admin): route site replication oidc via app context
* fix(ecstore): retry transient endpoint DNS failures
* fix(ecstore): satisfy DNS retry clippy check
* fix(ecstore): harden endpoint DNS retry
* test(ecstore): stabilize DNS retry error test
* refactor: move delete object contracts to storage api
* refactor: narrow store api compatibility exports
* refactor: route table catalog test through storage compat
* fix: harden local SSE-S3 fallback
* test: update managed SSE-S3 assumptions
* chore: keep issue plan local only
* test(ci): seed local SSE key for s3-tests
* perf(put): add eager path metrics and isolation tooling
* fix(decommission): persist progress adaptively (#3497)
Persist decommission progress after either the existing time interval or a migrated-item threshold, and flush progress baselines after bucket and terminal-state saves.
Also stabilize the OIDC discovery mock used by the pre-commit gate.
* refactor: move bucket operations contract (#3507)
* fix(s3): handle multipart flexible checksums (#3508)
* fix(io-core): avoid blocking on pooled buffer return
* perf(put): add slow inflight diagnostics
* perf(put): fix 16KiB regression with threshold and pool bypass
- Lower SMALL_EAGER_PUT_MAX_SIZE from 256KB to 8KB so objects >8KiB
use the streaming BufReader path (matches baseline behavior)
- Add POOL_BYPASE_MAX_SIZE (16KiB) to bypass BytesPool for very small
objects, avoiding Small-tier Mutex contention under high concurrency
- Add read_small_put_body_exact_direct() for direct Vec<u8> allocation
- Fix stale test assertions to match new 8KB threshold
Root cause analysis: the 16KiB regression was primarily caused by
instrumentation overhead in set_disk.rs (4x Instant::now() + metrics
per PUT), not BytesPool contention. Lowering the threshold eliminates
the eager-path overhead for 16KiB+ objects.
* perf(put): gate stage metrics behind observability flag
Add put_stage_metrics_enabled() AtomicBool switch in io-metrics crate.
When disabled (default), record_put_object_path() and
record_put_object_stage_duration() are no-ops, avoiding unnecessary
histogram/counter macro overhead in the PUT hot path.
The flag is set to true during startup when OTEL metric export is
enabled (rustfs_obs::observability_metric_enabled() == true).
This eliminates the per-request metrics overhead that contributed
to the 16KiB PUT regression when metrics collection is not active.
* perf(put): comprehensive optimization - restore eager path, cache env, remove UUID
Change 1: Restore SMALL_EAGER_PUT_MAX_SIZE from 8KB to 1MB
- The try_lock() fix (d13a189e3) eliminates the blocking that caused
service health timeouts under 512KiB c64 load
- Eager path with BytesPool is now safe for objects up to 1MB
- Recovers the eager path benefit for 32KiB-256KiB objects
Change 2: Adjust POOL_BYPASE_MAX_SIZE from 16KB to 4KB
- With eager path restored to 1MB, objects 4KB-1MB benefit from pool reuse
- Only ≤4KB objects bypass the pool (allocation cost negligible)
Change 3: Cache RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES via OnceLock
- Eliminates per-encode std::env::var() syscall
- Env var still works (read once at first use)
Change 4: Replace Uuid::new_v4() with Uuid::nil() in Erasure construction
- _id field is unused in hot paths (documented in code)
- Eliminates CSPRNG syscall per PUT request
Change 5: Add concurrency-aware buffer sizing to PUT path
- Reuses get_concurrency_aware_buffer_size() from GET path
- Reduces buffer size under high concurrency (0.4x at >8 concurrent)
- Lowers memory pressure for >1MB streaming PUTs
* chore: add pyroscope feature flag and clean up imports
- Add pyroscope feature flag forwarding to rustfs-obs
- Remove unused allow(non_upper_case_globals) in globals.rs
- Sort imports and fix Cargo.toml formatting consistency
* style: fix import ordering and code formatting
- Sort imports alphabetically in globals.rs, encode.rs
- Fix indentation in erasure_coding encode/erasure
- Clean up HashReader formatting in object_usecase.rs
* fix(test): use tokio::test for request_logging_layer tests
The tests call tokio::spawn via RequestContextLayer, which requires a
Tokio runtime. Changed from #[test] + futures::executor::block_on to
#[tokio::test] + .await, and replaced tracing::subscriber::with_default
with tracing::subscriber::set_default to support async.
* fix(bench): normalize no-space throughput/latency parsing in to_bps/to_ms
When a benchmark tool prints throughput without a separator (e.g. 123MiB/s),
awk '{print $2}' returns empty because the whole string is one field,
causing to_bps to return N/A and losing valid measurements in CSV output.
Insert a space between number and unit via sed before awk field splitting.
Same fix applied to to_ms for latency values like '50ms'.
Also add TODO comment on PUT path noting that get_concurrency_aware_buffer_size
reads ACTIVE_GET_REQUESTS instead of PUT concurrency (PR #3514 review).
Refs: PR #3514 review comments by chatgpt-codex-connector
* fix(metrics): correct POOL_BYPASS comments and separate PUT vs generic stage metrics
- Fix 3 comment-code mismatches: POOL_BYPASS_MAX_SIZE is 4KiB, not 16KiB
- Add generic record_stage_duration() with separate histogram
(rustfs_internal_stage_duration_ms) for non-PUT paths
- Replace record_put_object_stage_duration with record_stage_duration in
metacache_set, store_list_objects, and bucket_lifecycle_ops to avoid
polluting PUT-specific dashboards with listing/lifecycle timings
- Fix flaky test: serialize tests mutating PUT_STAGE_METRICS_ENABLED with
METRICS_FLAG_LOCK mutex and explicitly set desired state at test start
Refs: PR #3514 review comments by chatgpt-codex-connector
* style: apply cargo fmt to metacache_set.rs
---------
Co-authored-by: cxymds <cxymds@gmail.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
Persist decommission progress after either the existing time interval or a migrated-item threshold, and flush progress baselines after bucket and terminal-state saves.
Also stabilize the OIDC discovery mock used by the pre-commit gate.
* perf(ecstore): reuse erasure-decode shard buffers across stripes
ParallelReader::read allocated and zero-filled a fresh
`vec![0u8; shard_size]` per shard on every erasure stripe. Erasure::decode
builds one ParallelReader and loops stripes, so those buffers can be
reused: BitrotReader::read overwrites buf[..n] and the caller truncates to
n, so leftover bytes from the prior stripe are never observed. The decode
loop now hands each stripe's shard buffers back to the reader, which
reuses them (resized to shard_size) on the next stripe. The heal path,
which consumes its buffers into Bytes, never hands them back and is
unchanged.
This path runs on every object GET. Streaming-decode benchmark
(Erasure::decode end-to-end, median):
4+2 16MiB 1MiB-block: -9.6% verify-on, -23.8% verify-off
6+3 24MiB 1MiB-block: -9.7% verify-on, -22.5% verify-off
4+2 4MiB 64KiB-block: -5.5% verify-on, -16.3% verify-off
Adds a streaming-decode benchmark covering the ParallelReader path and a
regression test that decodes a multi-stripe object with missing data
shards (so reconstructed buffers are recycled) with bitrot verification
both on and off, asserting byte-exact output.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* perf(ecstore): tighten decode buffer reuse and bench harness
Address PR review feedback on the erasure-decode buffer-reuse change:
- Only claim a recycled shard buffer inside the `Some(reader)` branch of
`ParallelReader::read`, so the take is co-located with the read that
uses it and missing shards no longer touch the recycle slot.
- Move per-shard `BitrotReader` construction into Criterion `iter_batched`
setup so reader/UUID construction stays out of the timed decode path,
and assert decode success plus full output length each iteration.
Re-measured with the cleaner harness (median, before -> after, all
p < 0.05): verify-on -5.3%..-11.7%, verify-off -15.8%..-25.3%.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* perf(ecstore): drop dead recycle slots and tidy decode bench
Address the second PR review pass:
- In `Erasure::decode`, clear recycled slots for missing-shard (None-reader)
indices before handing buffers back. `ParallelReader::read` only reuses
slots with an active reader, and `decode_data` always re-allocates the
reconstructed shard, so retaining those buffers only held memory that could
never be reused in degraded reads.
- Move the output sink into the benchmark's `iter_batched` setup and validate
decode success/output length once per config instead of inside the timed
loop, so neither sink construction nor assertions touch the measurement.
Re-measured with the final harness (median, before -> after, all p < 0.05):
verify-on -5.2%..-9.8%, verify-off -19.4%..-25.8%.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* build(deps): bump brotli from 8.0.3 to 8.0.4 in the dependencies group
Bumps the dependencies group with 1 update: [brotli](https://github.com/dropbox/rust-brotli).
Updates `brotli` from 8.0.3 to 8.0.4
- [Release notes](https://github.com/dropbox/rust-brotli/releases)
- [Commits](https://github.com/dropbox/rust-brotli/compare/8.0.3...8.0.4)
---
updated-dependencies:
- dependency-name: brotli
dependency-version: 8.0.4
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: dependencies
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump s3s from cf4c3346 to c59b62f7ba in the dependencies group
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
* fix(bucket-repl): persist MRF retry queue to disk and reload on startup
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(bucket-repl): address three blocking MRF issues from review
1. MRF replay loses delete operations — add `MrfOpKind` discriminator to
`MrfReplicateEntry` (Object | Delete, default=Object for backward
compat). `DeletedObjectReplicationInfo::to_mrf_entry` now persists
`op=Delete`, `version_id`, `delete_marker_version_id`, and
`delete_marker`. `start_mrf_processor` branches on `op`: delete
entries skip `get_object_info` and replay via
`schedule_replication_delete` with `ReplicationType::Heal`; object
entries follow the existing heal path.
2. `flush_mrf_to_disk` cleared the in-memory batch even on encode/write
failure — changed return type to `bool` and callers now only
`pending.clear()` on `true`, so a transient storage error retries
on the next tick instead of silently dropping the batch.
3. Add focused tests: encode/decode roundtrips for object, delete-marker,
versioned-delete, and mixed-batch entries; a routing test confirming
op-kind propagates correctly and that the default is Object for
legacy files; a legacy-compat test verifying old entries round-trip
cleanly through the new format.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* style: fix clippy redundant-clone in MRF tests
Replace &[entry.clone()] with std::slice::from_ref(&entry) in two
encode_mrf_file call sites flagged by clippy's redundant_clone lint
under --all-targets --features rio-v2.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* test(bucket-repl): strengthen legacy MRF compat test with hand-built msgpack
The previous mrf_legacy_file_without_op_field_decoded_as_object test
round-tripped through encode_mrf_file, so it exercised the new format
and never touched a truly-legacy payload.
Replace it with a hand-built msgpack payload that genuinely omits the
"op", "deleteMarker", and "deleteMarkerVersionID" keys — exactly what
the old binary would have written before MrfOpKind existed. The test
now fails if #[serde(default)] is removed from the op field, which
proves real backward compatibility rather than round-trip stability.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: houseme <housemecn@gmail.com>
fix(scanner): skip startup delay when replication is active so failed objects heal promptly after restart
Co-authored-by: houseme <housemecn@gmail.com>
fix(bucket-repl): honor op_type in replicate_object so ExistingObject resync respects DISABLED targets
replicate_object was calling filter_target_arns with hard-coded
op_type: Object and existing_object: false regardless of what was
stored in roi.op_type. This meant that a resync worker setting
roi.op_type = ExistingObject (resync_bucket, line 889) had no effect
on target filtering: all configured targets were included, even ones
whose rule had ExistingObjectReplicationStatus::DISABLED.
Fix: pass op_type: roi.op_type and derive existing_object from it
(true only for ExistingObject, not Heal — Heal intentionally bypasses
the existing-object opt-out to repair past failures).
Also add warn! logs at all four MRF channel-overflow sites that were
previously silently returning Missed with no observability.
Verified with a live two-instance test: after resync, objects reached
the ENABLED target and were correctly blocked from the DISABLED target.
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: houseme <housemecn@gmail.com>
* fix(site-repl): clamp replication_cfg_mismatch to owning deployments and add resync status branch
Fix 3: replication_cfg_mismatch was set on ALL deployments for a bucket when
any replication-config discrepancy existed, even on deployments that simply have
no config. mc computes "in sync" as max_buckets minus per-deployment mismatch
entries; with N deployments all flagged for 1 bucket, the result underflows to
1-N = -1. Now only deployments that own a replication config are flagged.
Fix 4: SiteReplicationResyncOpHandler accepted "start" and "cancel" but returned
an empty body (and later a parse error on the client) for any other operation
string. Add a SITE_REPL_RESYNC_STATUS="status" arm that returns the stored
SRResyncOpStatus or an explicit {"status":"not-found"} object so mc never sees
an unexpected end of JSON input.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(site-repl): derive sync_state from reachability and replication rule completeness
PeerInfo.sync_state was set to SyncStatus::Unknown at every construction site
and never updated from real signals. build_status_info now tracks which peers
were reachable during the metainfo fetch phase and, after all bucket stats are
merged, derives sync_state as:
- Enable : reachable AND no replication_cfg_mismatch for any bucket
- Disable : reachable BUT at least one bucket has incomplete/missing rules
- Unknown : unreachable (fetch failed)
The "Sync" column in mc admin replicate status will now reflect actual state
rather than always showing Unknown.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(site-repl): add SRRotateServiceAccountHandler for split-brain svc-acct repair
When site-replicator-0 gets desynced (different secret on different peers), calls
via the old service account return 403, blocking remove and replicate operations
with no in-band recovery path.
SiteReplicationRemoveHandler already purges local state unconditionally before
sending peer notifications, so force-remove works locally even when peers reject
the 403. The new POST /v3/site-replication/rotate-svc-acct endpoint provides the
missing recovery path: it generates a fresh service-account secret, applies it
locally, and pushes a peer/join to every member. Each peer's SRPeerJoinHandler
accepts the join idempotently (update if exists, create otherwise), repairing the
desynced credential without a full teardown + restart.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(site-repl): back-fill pre-existing buckets and objects on replicate add
SiteReplicationAddHandler and SRPeerJoinHandler both called persist_site_replication_state
and returned success without doing anything for buckets that existed before the
sites were linked. Only buckets created after the link (via site_replication_make_bucket_hook)
ever replicated.
Introduce backfill_existing_buckets_after_add which, after persisting the new
state, iterates every local bucket and:
1. Ensures versioning is enabled (required by replication).
2. Reconciles bucket targets so a target entry exists for each remote peer.
3. Reconciles the replication config so a rule pointing to each peer is present.
4. Broadcasts a make-bucket-hook to peers (idempotent) so they create the bucket.
5. Kicks start_site_bucket_resync toward every remote peer so pre-existing
objects travel across.
Errors per bucket are logged but never abort the overall add — manual resync
remains available as a fallback. Both the initiator and the receiving (join) side
run the backfill so convergence happens regardless of which side held data first.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(site-repl): reconcile replication config to restore bidirectional replication
Root cause: ensure_site_replication_bucket_replication_config bailed with Ok(())
the moment any replication config was found on the bucket. When bucket B was
propagated to site-2 via the make-with-versioning bucket-op, site-2's
configure-replication step loaded the freshly-written config and immediately
returned, never adding the reverse-direction rule pointing back to site-1. Result:
objects uploaded to site-2 failed with "replication head_object fallback failed
... service error" because no rule targeted the originating site.
Fix: drop the early-return. Instead load the existing rules, build the full
desired config via build_site_replication_config, and MERGE — adding only the
rules that are absent (identified by their "site-repl-<deployment_id>" id). Re-
number priorities after each merge to avoid conflicts. Existing non-site-repl
rules are preserved. The write is skipped entirely when all desired rules are
already present, so repeat calls remain cheap.
Together with Fix 1 (backfill), any file written to ANY member now replicates to
ALL members. The offline-and-recover case also benefits: when a peer returns,
the per-bucket rules are complete and the resyncer can catch up.
Also register the new rotate-svc-acct route in route_policy and
route_registration_test so the route inventory assertions stay green.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(replication): remove broken resync worker-signal gate
The resync_bucket task opened a new broadcast receiver via
worker_rx.resubscribe(), which positions the receiver at the current
write-head of the ring buffer — past all 10 bootstrap signals written
in ReplicationResyncer::new(). Every spawned resync task therefore
blocked on recv() forever, making `mc admin replicate resync start`
report "started" while no objects ever moved.
Remove the dead wait entirely. Each resync_bucket call is already
spawned on-demand (tokio::spawn in start_bucket_resync / load_resync),
so no additional gate is needed. The per-object concurrency limit is
already enforced by the inner mpsc worker channels (line ~877). Also
remove the now-dead worker_tx/worker_rx fields, bootstrap loop, and
signal-send in resync_bucket_mark_status.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* docs(site-repl): document rotate-svc-acct idempotency and partial-failure behavior
* style: cargo fmt and remove redundant clones
* fix(site-repl): preserve object-lock state when back-filling existing buckets
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: houseme <housemecn@gmail.com>
Adds an operator guide for the SFTP server: recommended
configuration, the path model, host keys on Unix and Windows, the
environment variable reference, session cleanup behaviour, IAM
policy requirements per SFTP operation, client compatibility
notes, multipart upload sizing and cleanup, and the log lines
worth alerting on.
Corrects documentation the platform change left stale. The module
overview and the UnsupportedPlatform error text still described
Windows as unsupported. A source comment referenced a document
that does not exist in the repository and now points at the new
guide. The changelog adds the two host-key reload variables
missing from its environment list, describes the banner variable
as the SSH identification string, and corrects the upload size
cap and compliance case count.
The session watchdog now selects its detection method per
platform. It previously probed kernel TCP state through a
Linux-only procfs path, so on macOS every healthy idle session
was killed within a minute, and on Windows the watchdog never
spawned at all, leaving wedged sessions with no cleanup. Linux
keeps its fast-kill watchdog unchanged. Other platforms get a
silence-only backstop that kills a session only at the
documented 30-minute idle ceiling.
The host-key loader now has a Windows arm. It loads OpenSSH
format host keys from the configured directory and logs a
one-time warning to restrict NTFS ACLs on the key directory,
the same operator-managed approach FTPS, WebDAV, KMS, and IAM
already use on Windows. Startup previously aborted with
UnsupportedPlatform because the Unix mode-bit permission check
has no Windows equivalent. tokio's io-uring feature is now
enabled only in Linux builds. io-uring is a Linux kernel
interface and enabling it unconditionally broke the Windows
build.
Co-authored-by: houseme <housemecn@gmail.com>
fix(security): add authorization check to ListRemoteTargetHandler
ListRemoteTargetHandler only verified that request credentials existed
but did not check whether the caller has replication or administrator
permissions. This allowed any authenticated user to list remote
replication target configuration, potentially leaking remote target
credentials (accessKey, secretKey, sessionToken).
Add validate_replication_admin_request() call with
GetBucketTargetAction, consistent with the other replication admin
handlers that already perform this check.
Fixes: GHSA-796f-j7xp-hwf4
Co-authored-by: houseme <housemecn@gmail.com>
fix(security): add IAM authorization to FTP read/metadata/cwd handlers
The FTP frontend's get() (RETR), metadata() (SIZE/MDTM), and cwd()
(CWD) handlers dispatched directly to the storage backend without
calling authorize_operation(). This allowed any authenticated FTP user,
including those with explicit Deny policies, to read arbitrary objects
and probe bucket existence regardless of IAM policy.
Add authorize_operation() calls matching the pattern used by the
write-path handlers (put, del, list, rmd) and the WebDAV driver:
- get(): S3Action::GetObject
- metadata() for files: S3Action::HeadObject
- metadata() for directories: S3Action::HeadBucket
- cwd(): S3Action::HeadBucket
Fixes: GHSA-3g29-xff2-92vp
Co-authored-by: houseme <housemecn@gmail.com>
Gate scanner-triggered deep heal behind a short new-object cooldown and retry transient bitrot shard size mismatches during local verification.
Add targeted tests for cooldown mode selection and size-mismatch classification, and log when deep scans are downgraded or size-mismatch retries occur.
* fix(ecstore): skip hidden metadata in walk limit
* fix(ecstore): match walk limit to visible versions
* fix(ecstore): avoid decoding all versions for limit
* fix: return 503 on lock contention instead of 500
When concurrent PUTs to the same key contend for the namespace write
lock, lock timeout and conflict errors were wrapped as
StorageError::other(...) → StorageError::Io(...), which fell through
to S3ErrorCode::InternalError (500) in the error mapping.
Only QuorumNotReached was correctly mapped to ServiceUnavailable (503);
all other lock errors (Timeout, AlreadyLocked, etc.) became 500.
Fix: map_namespace_lock_error now returns StorageError::Lock(err) for
non-quorum lock errors, and StorageError::Lock is mapped to
S3ErrorCode::ServiceUnavailable in the HTTP error conversion.
Affected paths:
- crates/ecstore/src/set_disk/lock.rs
- crates/ecstore/src/store/object.rs
- crates/ecstore/src/store/bucket.rs (make_bucket, delete_bucket)
- rustfs/src/app/object_usecase.rs (copy_object)
Regression test: test_concurrent_put_same_key_never_returns_500
* test: fail on unexpected lock contention errors
---------
Co-authored-by: houseme <housemecn@gmail.com>
* fix(lifecycle): harden scanner ILM expiry accounting
* fix(scanner): gate ILM action accounting on enqueue
* fix(metrics): avoid scanner source work argument list
* fix(scanner): gate local ILM accounting on enqueue
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
Log local endpoint disk-topology details during physical disk independence checks.
Include canonical paths, st_dev major:minor values, and resolved device ids in validation output to make Docker and bind-mount startup failures easier to diagnose.
Also assert the new diagnostics are present in the shared-disk regression test.
* fix(tier): stop sending nil/garbage versionId to warm backend S3
Three bugs caused NoSuchVersion errors when reading tiered objects:
1. warm_backend_s3sdk: GET and DELETE ignored rv/range opts entirely —
fixed to forward version_id and byte-range to the SDK request.
2. version.rs (MetaObject + MetaDeleteMarker): transition_version_id was
parsed with unwrap_or_default(), turning invalid/wrong-length bytes
into Uuid::nil(). The nil UUID was then serialized and sent as
?versionId=00000000-... to the tier backend -> NoSuchVersion.
Fixed: .and_then(.ok()).filter(!is_nil()) so only valid non-nil UUIDs
are forwarded as versionId.
3. bucket_lifecycle_ops: add debug/error logs in
get_transitioned_object_reader to record tier, tier_object, and
tier_version_id before and on failure of the tier GET.
Also adds tier transition fields to dump_fileinfo example for offline
xl.meta inspection, and fixes Docker build (cargo path + entrypoint).
Adds CLAUDE.md with tier architecture and debugging notes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* more fixes for versionId
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Marcelo Bartsch <marcelo@bartsch.cl>
* remove branch
* Add tests and fix cargo path, add load to build-docker
* update documentation (CLAUDE.md)
* more fixes for recover
* More fixes to ILM recover
* final fix
* chore: add missing-shard first-scene diagnostics (#3213)
chore(ecstore): add missing-shard first-scene diagnostics
Log rename_data quorum context behind RUSTFS_ISSUE3031_DIAG_ENABLE so partial-disk success can be correlated with later missing shard reads.
Also log put_object commit success and tmp cleanup boundaries to capture when successful quorum writes are followed by tmp_dir cleanup.
* fix test anmd fmt
* fix cargo path
fix test
* fix(tier): format copy_object self-copy guard
---------
Signed-off-by: Marcelo Bartsch <marcelo@bartsch.cl>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: cxymds <Cxymds@qq.com>
Co-authored-by: loverustfs <hello@rustfs.com>
* fix(admin): format policy JSON and improve error messages in service account API
- Use serde_json::to_string_pretty for policy serialization to match MinIO behavior
- Replace technical error messages with user-friendly ones for policy validation
- Fixesrustfs/rustfs#3233 (policy JSON not formatted)
- Fixesrustfs/rustfs#3232 (error messages not shown to users)
* Delete scripts/tempfile.zip
---------
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
* feat(scanner): expose checkpoint and source work status
* fix(scanner): count ignored checkpoints once per scan
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
* fix(allocator): restore validated jemalloc target gating
Restrict the global allocator and jemalloc profiling paths to linux-gnu-x86_64 so Linux ARM64 builds fall back to mimalloc again.
Update the runtime profiling, allocator reclaim, and admin profiling handlers to use the same target gating and avoid exposing jemalloc-only code on unsupported targets.
Verification:
- cargo fmt --all
- cargo fmt --all --check
- cargo check -p rustfs
- make pre-commit
* fix(profiling): restore non-jemalloc platform behavior
Restore CPU profiling support on macOS while keeping jemalloc-backed memory profiling restricted to validated linux-gnu-x86_64 targets.
Also restore Unix log directory permission hardening so the allocator regression fix does not roll back unrelated observability behavior on other supported platforms.
Verification in progress:
- cargo fmt --all
- cargo check -p rustfs
- make pre-commit (running)
* fix(profiling): qualify periodic memory log macros
Use explicit tracing macro paths in periodic jemalloc memory profiling logging so builds do not fail when the local target configuration excludes those branches from the current import set.
Verification:
- cargo fmt --all
- cargo check -p rustfs
* fix(profiling): gate unsupported memory helper
Restrict the unsupported memory profiling helper to non-linux-gnu-x86_64 targets so Linux builds do not emit dead_code warnings for an unreachable fallback.
Verification:
- cargo fmt --all
- cargo check -p rustfs
* fix(bucket-encryption): populate default KMS key for SSE-KMS without key ID
When PutBucketEncryption receives SSE-KMS configuration without a specific
KMS key ID, query the KMS service for the default key and populate
KMSMasterKeyID before storing. This ensures GetBucketEncryption responses
include the key ID, which S3 clients like mc rely on to distinguish SSE-KMS
from SSE-S3 in their display logic.
Fixes#3039
* test(kms): cover empty bucket encryption key ID
* fix(bucket-encryption): require default KMS key
* refactor(ecstore): add accessor methods for service globals
Phase 3 of global singleton consolidation. Add ECStore accessor
methods that delegate to process-global service singletons,
providing a unified API through ECStore.
New methods:
- notification_system() — delegates to notification_sys global
- bucket_metadata_sys() — delegates to bucket metadata global
- endpoints() — delegates to endpoints global
- region() — delegates to region global
- tier_config_mgr() — delegates to tier config global
- server_config() — delegates to server config global
- storage_class() — delegates to storage class global
1161 tests pass, clippy clean, formatting clean.
* fix: use imported functions instead of fully-qualified paths
Address review comments: use imported functions instead of
crate:: paths to avoid unused import warnings.
1160 tests pass, clippy clean.
* fix(ecstore): remove unused service imports
* fix(ecstore): skip hidden metadata in walk limit
* fix(ecstore): match walk limit to visible versions
* fix(ecstore): avoid decoding all versions for limit
* refactor(ecstore): migrate config globals into ECStore struct fields
Phase 2 of global singleton consolidation. Add server_config and
storage_class fields to ECStore, sharing the same underlying data
as the process-global LazyLock statics.
New ECStore fields:
- server_config: RwLock<Option<Config>>
- storage_class: RwLock<storageclass::Config>
New async accessor methods:
- get_server_config() / set_server_config()
- get_storage_class() / set_storage_class()
Fields initialized with defaults in ECStore::new(), synced from
globals after init() completes. Global functions preserved for
backward compatibility.
1160 tests pass, clippy clean, full workspace compiles.
* fix: address PR #3219 review comments - delegate to globals
Accessors now delegate to process-global statics to avoid state drift:
- get_server_config() delegates to get_global_server_config()
- set_server_config() updates both global and local field
- get_storage_class() delegates to get_global_storage_class()
- set_storage_class() updates both global and local field
- Removed stale sync code from init()
- Changed fields to std::sync::RwLock for sync access
1160 tests pass.
* fix: correct import order for cargo fmt
* fix: simplify config accessors - delegate to globals without local state
Remove local server_config and storage_class fields from ECStore.
Accessors now purely delegate to process-global statics, eliminating
the state drift risk identified in review.
1160 tests pass, formatting clean.
* chore(ecstore): add missing-shard first-scene diagnostics
Log rename_data quorum context behind RUSTFS_ISSUE3031_DIAG_ENABLE so partial-disk success can be correlated with later missing shard reads.
Also log put_object commit success and tmp cleanup boundaries to capture when successful quorum writes are followed by tmp_dir cleanup.
* chore(ecstore): add delete-objects lock batch diagnostics
Log DeleteObjects batch-lock request size, distributed lock quorum summary, and failed object keys behind RUSTFS_ISSUE3031_DIAG_ENABLE.
Include a Chinese operator checklist under docs/ that explains which logs to capture and how to interpret DeleteObjects lock_batch timeout incidents.
* chore: remove lock batch checklist from repo
Keep docs/issue-658-deleteobjects-lock-batch-checklist-zh.md only in the local checkout and drop it from the PR history.
* perf(erasure): remove UUID from clone + increase encode inflight budget
Two targeted optimizations for the erasure encoding hot path:
1. Erasure::clone() no longer generates Uuid::new_v4() per clone.
The _id field is unused in the hot path; reusing the original ID
eliminates a CSPRNG call per block encode (100 calls for a 100MB
object with 1MB blocks).
2. Default RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES raised from 8MB
to 32MB. This increases the encode pipeline depth from ~5 to ~20
blocks, allowing more read-ahead between the encoder and disk
writer stages. The per-request memory bound is still controlled
by the 8-block hard cap and the env var override.
3. Added encode_data_owned() utility method for zero-copy encoding
when the caller already owns a heap buffer (Vec<u8> → BytesMut
via Bytes::try_into_mut). Not used in the hot path yet but
available for future callers.
All 1157 ecstore tests pass. Criterion micro-benchmarks show no
regression (< 2% variance). Single-machine warp E2E tests were
inconclusive due to high variance; a dedicated multi-disk test
environment is needed for reliable E2E comparison.
Ref: https://github.com/rustfs/backlog/issues/659
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* chore: update Cargo.lock
* fix(erasure): align encode inflight cap and tests
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
chore(ecstore): add missing-shard first-scene diagnostics
Log rename_data quorum context behind RUSTFS_ISSUE3031_DIAG_ENABLE so partial-disk success can be correlated with later missing shard reads.
Also log put_object commit success and tmp cleanup boundaries to capture when successful quorum writes are followed by tmp_dir cleanup.
* fix(http): reduce header timeout log noise
Log peer_addr for HTTP connection errors so idle or probe traffic can be traced back to the source.
Downgrade HeaderTimeout connection logs from warn to info because these timeouts are often caused by incomplete probe traffic rather than a service-side fault.
* fix(ecstore): suppress spurious warn logs for missing trash cleanup sources
Extract `reliable_rename_inner` with a `warn_on_failure` flag so that
`rename_all_ignore_missing_source` can skip the warn log when the source
is absent. This completes the idempotent cleanup fix — the previous
implementation still emitted a `reliable_rename failed` warning before
the NotFound was caught and silenced.
Also restore the `recursive` branching in `move_to_trash` so that
non-recursive deletions use `fs::rename` directly (preserving original
semantics), while recursive deletions use the idempotent helper.
`is_safe_claim_policy_name` rejected any character other than
`[a-zA-Z0-9_-]`, silently dropping policy names containing colons.
This breaks Kubernetes workload identity where `claim_name=sub`
resolves to `system:serviceaccount:<namespace>:<sa-name>` — a valid
policy name that can be created via the admin API but is then
unreachable during STS session authorization.
Add `:` and `.` to the allowed character set. These characters are:
- Used in K8s service account `sub` claims (colons)
- Used in Java/DNS-style group names from OIDC providers (dots)
- Already accepted by the `add-canned-policy` admin API endpoint
Require at least one alphanumeric character to prevent meaningless
names (`.`, `..`, `-`, `_`, `:`, etc.) from resolving.
Still rejected: `/`, `\`, whitespace, `$`, `;`, `{`, `}` and other
chars that could enable path traversal or injection.
Signed-off-by: Alexander Kharkevich <alex@mara.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: GatewayJ <835269233@qq.com>
* fix(ecstore): remove dead _buf field from Erasure struct
_buf was allocated in new(), clone(), and Default but never read or written.
Each clone() was allocating block_size bytes (typically 10MB) for nothing.
- Remove _buf from struct, Default, Clone, new_with_options
- 51 erasure tests pass
* fix: remove empty line after doc comment
---------
Co-authored-by: houseme <housemecn@gmail.com>
* fix(iam): prevent transient IAM walk timeout from crashing startup
IAM startup performs a blocking full metadata walk on `.rustfs.sys/config/iam/`.
When that distributed walk times out (e.g. disk pressure after cluster reboot),
the old code treated the failure as fatal and exited the process, causing a
systemd restart loop.
Changes:
- Add `startup_iam.rs`: attempt IAM init, enter degraded mode on failure,
spawn background retry task with exponential backoff (5s→10s→20s→30s cap)
- Log level escalates to ERROR after 12 retries (~5 min) to aid diagnosis
- `/health/ready` returns 503 until IAM recovers; IAM-dependent ops return
`IamSysNotInitialized` (existing fail-closed behavior preserved)
- Fix admin path boundary matching: `/minio/administrator` no longer falsely
matches as admin prefix
- Normalize Content-Length: 0 for admin GET requests with empty body
Fixes#3175
* fix(iam): move constant assertion into const block
Fixes clippy::assertions-on-constants warning on
IAM_RETRY_ESCALATION_THRESHOLD assertion.
* fix(iam): address PR review comments
- Replace OnceLock with AtomicU64 sentinel for test isolation;
add reset_test_failure_counter() for integration tests
- Use u32::try_from() instead of `as u32` narrowing cast in
compute_backoff_interval
- Rename misleading test; update to verify finalize retry behavior
- Restructure spawn_iam_recovery_task into init-retry and
finalize-retry phases so transient readiness failures are retried
instead of leaving the server permanently degraded
* fix(iam): gate test hooks behind debug_assertions
- reset_test_failure_counter() now stores sentinel (u64::MAX) to
correctly trigger env var re-read on next call
- RUSTFS_TEST_IAM_FAIL_INIT_ATTEMPTS only honored in debug builds
- RUSTFS_TEST_IAM_RETRY_INTERVAL_MS only honored in debug builds
* test(iam): cover deferred bootstrap recovery
Add a dedicated embedded deferred-IAM integration test in a separate test binary to avoid process-global startup collisions.
Strengthen startup IAM recovery coverage with focused unit tests and keep the existing embedded smoke test isolated while carrying the manual test license header update in the same change set.
* fix(startup): tighten deferred IAM recovery path
Adopt follow-up review feedback by silencing misleading app-context warnings after IAM recovery, reusing boundary-aware path prefix checks in the readiness gate, and tying deferred IAM recovery retries to server shutdown tokens.
Keep the deferred IAM embedded integration coverage and startup recovery unit coverage green after the follow-up hardening.
* refactor(startup): simplify IAM recovery task
Collapse the deferred IAM recovery implementation back to a concrete production flow instead of keeping boxed callback seams in the runtime path.
Keep only stable backoff unit coverage in startup_iam and rely on the embedded deferred bootstrap integration test for end-to-end recovery behavior.
* refactor(startup): trim IAM recovery test scaffolding
Keep the concrete deferred IAM recovery path intact while removing bulky test-only async loop scaffolding from startup_iam.
Retain the stable backoff unit checks and rely on the embedded deferred bootstrap integration test for end-to-end recovery coverage.
* fix: apply code review improvements from PR #3188 review
- Simplify RecoveryFuture type alias by removing unnecessary lifetime
- Fix finalize_iam_recovery to return Err if app context unavailable
- Update bootstrap_or_defer_iam_init doc comment to reflect Err case
- Use boundary-aware has_path_prefix for admin path matching in utils.rs
- Add test for adminx boundary rejection in utils.rs and layer.rs
- Improve embedded deferred IAM test with timeout wrapper
* style: merge has_path_prefix import into existing use block
* fix(iam): address final review follow-ups
- fix main startup readiness publication to pass ServiceStateManager correctly
- centralize IAM test env keys in rustfs_config and reuse them in runtime/tests
- keep deferred IAM bootstrap validation aligned with the final review fixes
* fix: isolate listing timeouts from drive health
Keep walk_dir scanner timeouts request-scoped instead of marking local drives faulty.
Add regression coverage for follow-up bucket info, set-level list_path, and system-prefix listings after prior walk timeouts.
* test(iam): gate deferred bootstrap test to debug
Align the deferred IAM embedded integration test with debug-only IAM fault injection hooks so release-profile runs do not assert deferred bootstrap behavior that cannot be triggered.
* test(ecstore): bound prior walk timeout regressions
- set walk_dir stall timeout explicitly in prior-timeout listing tests
- keep the system-prefix follow-up listing scoped to the same base dir
- assert the expected directory entry so the timeout regression test stays fast and stable
* fmt
* fix(ecstore): retry transient walk dir stream errors
Retry RemoteDisk walk_dir once when the first failure looks transport-related so restart-time HttpReader stream errors do not immediately count as hard listing failures.
Improve HttpReader request and stream error messages with method and URL context, and add regression coverage for retry recovery and diagnostics.
* fix(ecstore): avoid retry after partial walk dir stream
Stop retrying walk_dir after copy_stream_with_buffer has already written partial bytes into the destination writer.
Keep the retry only for open_walk_dir failures and add a regression test that proves partial stream failures are returned without issuing a second stream.
Add `RUSTFS_IDENTITY_OPENID_HIDE_FROM_UI[_<SUFFIX>]` setting that
removes a provider from the login page while keeping it fully
functional for STS AssumeRoleWithWebIdentity and site-replication.
Changes:
- Add `hide_from_ui: bool` to `OidcProviderConfig`
- Add `list_visible_providers()` that filters hidden providers
(used by console login and /v3/oidc/providers endpoint)
- Keep `list_providers()` unfiltered for site-replication/admin config
- Extract `normalize_provider_config(config) -> config` to deduplicate
field normalization (accepts the struct directly, not 18 parameters)
- Add `parse_enable_state()` helper for consistent EnableState parsing
- Plumb through admin API request structs (`#[serde(default)]`)
- Expose in `OidcConfigView` for admin GET config round-trip
- Persist via `upsert_persisted_provider_config()`
Note: adding `hide_from_ui` to the public `OidcProviderConfig` struct
is a source-level change for code constructing it with struct literals.
This is acceptable for the current pre-1.0 release cycle.
Signed-off-by: Alexander Kharkevich <alex@mara.com>
Co-authored-by: GatewayJ <835269233@qq.com>
* feat(ecstore): add object lock diagnostics
Add configurable namespace lock diagnostics for object operations so production contention can be traced by operation, owner, and key.
Wrap object read/write lock acquisition in diagnostic guards across get, head, put, delete, copy, and multipart flows, and log slow acquisition and long hold durations behind new RUSTFS_OBJECT_LOCK_DIAG_* settings.
Verification:
- make pre-commit
* feat(obs): expose object lock diagnostics metrics
Add Prometheus metrics and Grafana panels for object namespace lock diagnostics, covering slow acquire counts, slow hold counts, acquire duration, hold duration, and the diagnostics-enabled state.
Adopt PR review feedback by keeping diagnostic guards alive through the guarded operation so long-hold warnings and metrics are emitted, reusing shared env helpers, and reducing default-path overhead when diagnostics are disabled.
Verification:
- cargo check -p rustfs-ecstore -p rustfs-io-metrics
- cargo test -p rustfs-ecstore store::object -- --nocapture
- make pre-commit
* perf(obs): reduce object lock diag overhead
Avoid repeated environment parsing on hot object-lock paths by caching the diagnostics-enabled flag, and stop allocating label strings for object lock metrics by recording static labels directly.
Strengthen io-metrics tests by using a local recorder and asserting that the expected object lock diagnostic counters, gauges, and histograms are emitted.
Verification:
- cargo check -p rustfs-ecstore -p rustfs-io-metrics
- cargo test -p rustfs-io-metrics -- --nocapture
- make pre-commit
* fix(rio): surface incomplete put bodies
Propagate incomplete PUT request bodies as IncompleteBody instead of allowing erasure encode to treat truncated input as a normal EOF.\n\n- mark premature EOFs in HardLimitReader with an explicit IncompleteBody error\n- preserve EOF error chains through read_full and map them to S3 IncompleteBody\n- stop erasure encode from swallowing UnexpectedEof on truncated input\n- add regression tests for reader, erasure encode, and API error mapping\n\nRefs: rustfs/backlog#654
* fix(s3): honor decoded length for aws chunked put
Use x-amz-decoded-content-length for aws-chunked PutObject requests so trailer-checksum uploads are sized against the decoded payload instead of the wire-encoded content-length.\n\n- prefer decoded content length for aws-chunked put bodies\n- add a regression test covering the size selection logic\n- keeps the incomplete body fix working for truly truncated uploads while restoring checksum trailer compatibility\n\nRefs: rustfs/backlog#654
* fix(io): follow up review comments on incompletebody handling
Address PR review feedback by restoring read_full's existing EOF contract, adding a dedicated read_full_or_eof helper for erasure encoding, covering nested incomplete-body error chains, and documenting plus hardening aws-chunked size selection.\n\n- keep read_full returning early EOF on empty reads\n- use read_full_or_eof only in erasure encoding paths\n- detect aws-chunked via content-encoding or transfer-encoding\n- add nested error-chain and aws-chunked regression tests\n\nRefs: rustfs/backlog#654
* fix(rio): surface incomplete put bodies
Propagate incomplete PUT request bodies as IncompleteBody instead of allowing erasure encode to treat truncated input as a normal EOF.\n\n- mark premature EOFs in HardLimitReader with an explicit IncompleteBody error\n- preserve EOF error chains through read_full and map them to S3 IncompleteBody\n- stop erasure encode from swallowing UnexpectedEof on truncated input\n- add regression tests for reader, erasure encode, and API error mapping\n\nRefs: rustfs/backlog#654
* fix(s3): honor decoded length for aws chunked put
Use x-amz-decoded-content-length for aws-chunked PutObject requests so trailer-checksum uploads are sized against the decoded payload instead of the wire-encoded content-length.\n\n- prefer decoded content length for aws-chunked put bodies\n- add a regression test covering the size selection logic\n- keeps the incomplete body fix working for truly truncated uploads while restoring checksum trailer compatibility\n\nRefs: rustfs/backlog#654
* fix(io): follow up review comments on incompletebody handling
Address PR review feedback by restoring read_full's existing EOF contract, adding a dedicated read_full_or_eof helper for erasure encoding, covering nested incomplete-body error chains, and documenting plus hardening aws-chunked size selection.\n\n- keep read_full returning early EOF on empty reads\n- use read_full_or_eof only in erasure encoding paths\n- detect aws-chunked via content-encoding or transfer-encoding\n- add nested error-chain and aws-chunked regression tests\n\nRefs: rustfs/backlog#654
* fix(rio): reject bytes beyond hard limit
* fix(ecstore): reject zero-sized erasure blocks
* fix(scanner,data-usage): fix add() logic inversion and usize underflow in reduce_children_of
Bug #1: add() had inverted logic — returned early when children were present,
making the recursive child traversal dead code. Leaf compaction path was
completely non-functional. Fixed by recursing into children when present and
collecting leaves only at leaf nodes.
Bug #2: reduce_children_of used which underflows in debug
(panic) or release (wraps to usize::MAX, compacts entire cache). Fixed with
saturating_sub.
Both bugs existed identically in crates/scanner and crates/data-usage.
Added 5 tests covering add() subtree traversal, edge cases, and
reduce_children_of compaction behavior.
Closesrustfs/backlog#652
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: collect internal nodes as compaction candidates, not leaves
The previous fix for add() collected leaf nodes (children empty) as
compaction candidates. This is incorrect: total_children_rec returns 0
for leaves, so compacting them never decrements — the
compaction loop makes no progress.
Collect internal nodes (children non-empty) instead. Compacting an
internal node removes its subtree via delete_recursive, which actually
reduces the total child count. Leaf nodes are skipped since they have
no children to remove.
Updated tests to match the corrected semantics.
* refactor: rename leaves→candidates, add data-usage regression tests
- Rename misleading variable/parameter to in
add() and reduce_children_of() for both crates
- Add 4 regression tests in crates/data-usage covering add() candidate
selection and reduce_children_of() compaction + saturating_sub
* fix: address data usage compaction review feedback
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: houseme <housemecn@gmail.com>
Add test assertions for all MinIO-compatible config admin routes
(get/set/del-config-kv, help-config-kv, list/clear/restore-config-history-kv,
GET/PUT /v3/config) and verify their compatibility alias paths.
Relates to rustfs/backlog#608
* fix(rebalance): require target goal before completion
Stop treating pools within a fixed tolerance of the rebalance goal as completed before any data has moved.
Add a regression test for the two-pool imbalance reported in issue 3137.
* chore(rebalance): clarify goal completion wording
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: houseme <housemecn@gmail.com>
* feat(admin): restore config admin compatibility
Co-authored-by: weisd <im@weisd.in>
* fix(admin): align config admin clean rebuild
Co-authored-by: weisd <im@weisd.in>
* fix(admin): align config history and peer signals
* fix(admin): harden config admin mutations
* fix(admin): tighten config review follow-ups
* perf(admin): reuse env snapshot in config render
* fix(ecstore): clean up config admin and listing error handling
Remove redundant is_all_volume_not_found check in list_merged, add
storage class encode/decode roundtrip tests, fresh boot integration
test, and config admin clean rebuild improvements.
Co-authored-by: hehutu <heihutu@gmail.com>
* fix(admin): sync global server config on mutation and reload
Change GLOBAL_SERVER_CONFIG from OnceLock to RwLock so config mutations
(set/del/restore/reload) are visible to readers without restart. Call
set_global_server_config after every store save and on snapshot reload.
Register storage_class as a dynamic config subsystem.
Co-authored-by: hehutu <heihutu@gmail.com>
* style: apply rustfmt to config and admin tests
Co-authored-by: hehutu <heihutu@gmail.com>
* fix(test): update signal_service test for storage_class dynamic subsystem
storage_class is now a valid dynamic config subsystem, so the
"requires object layer" test should expect "storage layer not initialized"
instead of "unsupported dynamic config subsystem".
Co-authored-by: hehutu <heihutu@gmail.com>
* fix(config): publish storage_class runtime config on dynamic reload
Change GLOBAL_STORAGE_CLASS from OnceLock to RwLock so runtime updates
are possible. apply_storage_class_runtime_config now actually publishes
the parsed config via set_global_storage_class instead of dropping it.
Addresses review feedback: storage_class was marked as dynamically
applied but the parsed result was discarded, so mc admin config set
returned config_applied=true while the runtime kept using stale parity
settings until restart.
Co-authored-by: hehutu <heihutu@gmail.com>
* fix(admin): harden config init, history ordering, and env redaction
- Change GLOBAL_SERVER_CONFIG from RwLock<Config> to RwLock<Option<Config>>
initialized with None, preserving "not initialized" detection via None
- Move save_server_config_history before save_server_config_to_store in
SetConfigKVHandler, DelConfigKVHandler, and SetConfigHandler so a
restore point exists before mutations are persisted
- Redact sensitive env override values with *redacted* instead of
silently omitting the line, improving admin visibility
- Add code comment explaining VolumeNotFound removal rationale in
list_merged for listing paths
Co-authored-by: hehutu <heihutu@gmail.com>
* fix(config): keep in-memory config in sync after set/restore/reload
GLOBAL_SERVER_CONFIG was a OnceLock set once at startup and never
updated. After mc admin config set writes to the store, any fallback
to get_global_server_config() returned stale init-time data. Similarly,
reload_runtime_config_snapshot read from the store but discarded the
result.
- Replace OnceLock with RwLock for GLOBAL_SERVER_CONFIG and
GLOBAL_STORAGE_CLASS so they can be updated at runtime
- Add set_global_server_config / set_global_storage_class setters
- Call set_global_server_config after every config save (set-kv,
del-kv, set-config, restore-history)
- Re-apply dynamic subsystems (storage_class, audit_webhook,
audit_mqtt) and signal peers in reload_runtime_config_snapshot
and full-config operations
- Fix render_selected_config scope boundary check: track per-scope
line count instead of checking global lines.is_empty()
- Include STORAGE_CLASS_SUB_SYS in is_dynamic_config_subsystem so
apply_storage_class_runtime_config is reachable
Co-authored-by: hehutu <heihutu@gmail.com>
* fix(storageclass): use CLASS_RRS key in lookup_config for RRS parity
lookup_config used kvs.get(RRS) where RRS="REDUCED_REDUNDANCY", but the
admin config path writes the key as CLASS_RRS="rrs". This caused RRS
values to never be read back, always falling back to default parity.
- Changed kvs.get(RRS) to kvs.get(CLASS_RRS) in lookup_config
- Added regression tests verifying RRS read/write consistency
Co-authored-by: hehutu <heihutu@gmail.com>
* fix(config): add peer-side logging and don't swallow apply errors
- Add tracing::warn! in reload_dynamic_config_runtime_state and
reload_runtime_config_snapshot when config read or subsystem apply
fails, so on-host diagnostics show which signal failed and why
- Change `let _ = apply_dynamic_config_for_subsystem(...)` to
`if let Err(err) = ... { warn!(...) }` in reload_runtime_config_snapshot
so per-subsystem failures are logged instead of silently swallowed
- Remove weak test global_server_config_returns_none_before_init that
had no meaningful assertion due to shared global state
Co-authored-by: hehutu <heihutu@gmail.com>
* style: apply rustfmt to config and storageclass tests
Co-authored-by: hehutu <heihutu@gmail.com>
---------
Co-authored-by: weisd <im@weisd.in>
Co-authored-by: hehutu <heihutu@gmail.com>
Clear persisted background heal active state after a deep scanner cycle completes so root heal status does not remain in progress between cycles.
Treat path-based stop for an already missing heal task as stopped while keeping client-token cancellation strict.
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
* perf: reduce spawn_blocking contention in PUT path (~23% throughput gain)
Flame graph profiling identified tokio blocking pool mutex contention
as the #1 bottleneck (17.3% of CPU time). Each spawn_blocking call must
acquire parking_lot::raw_mutex to enqueue work. With 16 concurrent PUTs
× 4 disks × 3+ spawn_blocking per disk, this became a serialization point.
Optimizations applied:
- Merge make_dir_all + file write into single spawn_blocking
- Merge read_file + parse + write + rename into single spawn_blocking
for inline objects (small files)
- Optimize reliable_rename to try rename first, mkdir only on ENOENT
- Optimize remove/remove_std to try remove_file first, EISDIR fallback
- Add encode_inline_small fast path for small objects
- Parallelize bitrot writer creation with join_all
Benchmark (4KiB PUT, 4-disk EC, 16 concurrent, 8 rounds, randomized A/B):
Baseline: ~950 obj/s → Optimized: ~1173 obj/s (+23%)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix: address review comments
- Return old_data_dir for non-inline rename_data path (was incorrectly None)
- Restore delete_all cleanup of PUT temp data on failure paths
- Fix cargo fmt formatting
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* style: apply rustfmt from stable 1.96.0
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix: collapse nested if-let chains for clippy compliance
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix: address Copilot review comments
- fs.rs: handle macOS EPERM from remove_file on directories
- os.rs: restore NotFound=Ok(()) semantics on first rename attempt
- local.rs: use try-rename-then-mkdir pattern for inline rename_data
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* test: add unit tests for encode_inline_small fast path
* test: fix comment and line length in encode_inline_small tests
* fix: revert reliable_rename and write_all_internal to match original
Restore the original reliable_rename logic (check parent exists, then
rename in loop) and the original write_all_internal (make_dir_all outside
spawn_blocking). The optimization changes caused a CI-only test failure
in capacity_dirty_scope_test that could not be reproduced locally.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix: update comment and fix formatting for CI
- Fix macOS/BSD comment to accurately say macOS only
- Fix encode_inline_small test formatting to match rustfmt 1.96.0
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix: propagate inline rename errors
* fix: retry rename when parent missing
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
* fix: preserve S3 listing pagination parity
S3 listing responses need stable wire semantics across encoded listings, version markers, and quorum-sensitive metadata reads. This tightens marker handling, response encoding, and version-marker pagination while keeping the changes scoped to listing paths and regression tests.
Constraint: Branch, code, commit, and PR text must avoid restricted upstream project naming.
Constraint: Verification required Rust 1.95 toolchain path because Homebrew cargo-clippy resolved to 1.94.
Rejected: Treat null version-id-marker as no marker | repeats or skips the marker boundary for null-version listings.
Rejected: Compare encoded next-marker values directly | encoded output can diverge from raw listing order.
Confidence: high
Scope-risk: moderate
Directive: Do not change listing marker semantics without covering V1, V2, versions, encoded responses, and null-version markers together.
Tested: cargo fmt --all --check
Tested: cargo clippy --workspace --all-features --all-targets -- -D warnings
Tested: make pre-commit with LC_ALL=en_US.UTF-8 and Rust 1.95 PATH
Tested: git diff --check
Not-tested: live distributed object-store compatibility test against a remote cluster
* fix: keep listing echo fields raw
S3 compatibility tests expect list response echo fields such as Prefix, Delimiter, StartAfter, and Marker to preserve the request value even when URL encoding is requested. Keep URL encoding scoped to object keys and common prefixes while preserving the next-marker raw comparison fix.
Constraint: PR branch was updated with latest main before this fix.
Constraint: Do not use restricted upstream project naming in commit text.
Rejected: Encode all response string fields | breaks compatibility tests for unreadable prefix values.
Confidence: high
Scope-risk: narrow
Tested: cargo test -p rustfs list_objects --lib
Tested: cargo fmt --all --check
Tested: cargo clippy --workspace --all-features --all-targets -- -D warnings
Tested: make pre-commit
Tested: git diff --check
Not-tested: live CI s3 compatibility rerun before push
* fix: accept empty listing continuation token
S3 compatibility tests treat an empty ListObjectsV2 continuation token as an explicit empty echo value, not as an invalid base64 token. Preserve the request echo while skipping decoded-token pagination for the empty string.
Constraint: Keep invalid non-empty continuation tokens rejected before store lookup.
Rejected: Drop empty continuation tokens from the response | compatibility tests assert the empty echo field is present.
Confidence: high
Scope-risk: narrow
Tested: cargo test -p rustfs list_objects --lib
Tested: cargo fmt --all --check
Tested: cargo clippy --workspace --all-features --all-targets -- -D warnings
Tested: make pre-commit
Tested: git diff --check
Not-tested: live CI s3 compatibility rerun after this push
* fix: delete explicit null object versions
S3 compatibility cleanup lists unversioned objects with VersionId=null and then sends that value back through DeleteObjects. The API layer keeps null as an internal sentinel, but the storage delete path must map it back to the stored null version instead of treating it as a real UUID.
Constraint: Keep the wire response able to echo null version IDs.
Rejected: Treat VersionId=null the same as an absent version id everywhere | versioned and suspended buckets need explicit null version semantics.
Confidence: high
Scope-risk: moderate
Tested: cargo test -p rustfs-ecstore delete_file_info_version_id_maps_explicit_null_version_to_stored_null
Tested: cargo test -p rustfs normalize_delete_objects_version_id_preserves_explicit_null_marker --lib
Tested: cargo fmt --all --check
Tested: cargo clippy --workspace --all-features --all-targets -- -D warnings
Tested: make pre-commit
Tested: git diff --check
Not-tested: live CI s3 compatibility rerun after this push
* fix: keep version marker scoped to marker key
Version listing cleanup can delete the key returned as the previous page marker before requesting the next page. When the marker key is no longer present, the version marker must not be applied to the first later key, otherwise each cleanup page can skip one null-version object and leave the bucket non-empty.
Constraint: Preserve version-marker pagination for marker keys that still exist.
Rejected: Drop version markers whenever the marker key was supplied | multi-version marker keys still need intra-key pagination.
Confidence: high
Scope-risk: narrow
Tested: cargo test -p rustfs-ecstore version_marker_is_applied_only_when_key_marker_entry_is_present
Tested: cargo test -p rustfs-ecstore delete_file_info_version_id_maps_explicit_null_version_to_stored_null
Tested: cargo test -p rustfs normalize_delete_objects_version_id_preserves_explicit_null_marker --lib
Tested: cargo fmt --all --check
Tested: cargo clippy --workspace --all-features --all-targets -- -D warnings
Tested: make pre-commit
Tested: git diff --check
Not-tested: live CI s3 compatibility rerun after this push
---------
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: loverustfs <hello@rustfs.com>
* feat(targets): support Kafka SASL auth
* fix(targets): infer Kafka SASL when enable is blank
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Set cross=true for the macos-x86_64 build matrix entry so that the
x86_64-apple-darwin target is cross-compiled on macOS ARM runners
instead of relying on a native x86_64 macOS runner.
* fix(utils): tolerate bavail greater than bfree on Linux
Treat f_bavail > f_bfree as a compatibility edge case instead of fatal corruption during startup disk info probing.
- keep corruption checks for other invalid counter relationships
- log a warning and clamp reserved blocks to 0 when bavail exceeds bfree
- add linux unit coverage for the compatibility and overflow paths
Refs #3025
* fix(utils): cap bavail to bfree for Linux statfs edge
* fix(utils): rate-limit Linux statfs compatibility warning
* test(utils): cover Linux statfs capacity math
Expand Linux statfs capacity math tests around normal, equal, zero, and fully-free block counter combinations.
Refs #3025
* fix(utils): avoid statfs warning guard reallocations
* test(utils): align statfs reserved block expectations
* fmt
* fix(utils): avoid warning guard path allocation
Check the warn-once set before allocating a PathBuf so repeated bavail/bfree compatibility warnings stay low-overhead.
Refs #3025
* chore(deps): bump workspace dependencies
- bump direct dependency versions in Cargo.toml\n- refresh Cargo.lock to resolve updated crates
* fix(deps): avoid pyroscope and pprof linker collision
- downgrade pyroscope from 2.0.6 to 2.0.5\n- keep a single pprof implementation path via pprof-pyroscope-fork\n- fix duplicate perf_signal_handler during test linking
* fix(ecstore): send valid ping body in remote locker
Build ping requests with a flatbuffer payload so health checks remain compatible with the ping response parser after restart.
* fix(bench): use multi-host warp target during failover
Normalize comma-separated warp host lists in run_object_batch_bench and let four-node failover bench pass BENCH_WARP_HOSTS so rolling restart does not pin load to a single restarting node.
* feat(health): add compat health probes with busy/KMS checks
- Add /health/live liveness probe endpoint
- Add busy protection (429) for readiness probes, gated by RUSTFS_HEALTH_COMPAT_BUSY_CHECK_ENABLE
- Add KMS readiness check for /health/ready, gated by RUSTFS_HEALTH_COMPAT_KMS_READY_CHECK_ENABLE
- Add lock quorum status caching with TTL to reduce RPC pressure
- Consolidate health response building into build_health_response_parts
- Register /health/live in console router and readiness gate
- Remove MinIO references from newly added health code
* fix(health): decouple kms readiness from lock quorum
* fix(site-replication): refresh peer TLS client cache
- rebuild site-replication peer reqwest client when outbound TLS generation changes\n- keep cached client/failed state per generation for low-overhead requests\n- accept --tls-path as alias of tls inspect --path for operator compatibility\n- add targeted parser/cache tests\n\nRefs #2723
* docs(issue-2723): add private-ca cert rotation retest
- add executable retest checklist for private CA site-replication flows\n- cover tls inspect compatibility (--path and --tls-path)\n- include post-rotation and post-restart pass criteria\n\nRefs #2723
* test(site-replication): isolate global TLS cache test
- serialize generation-sensitive peer TLS cache test\n- snapshot and restore global outbound TLS generation and static cache\n- avoid order-dependent interactions with parallel test execution\n\nRefs #2723
* chore: remove docs file from tracking, keep locally
The docs directory is already in .gitignore but this file was
previously committed. Remove it from git index to prevent docs
content from being included in the PR.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(site-replication): cover ready client cache hit path
Add a focused unit test for unchanged-generation Ready cache entries in site_replication_peer_client_cache_hit to validate steady-state client reuse behavior highlighted in PR #3109 review feedback.
---------
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Treat empty ping bodies as liveness probes, add a startup cleanup barrier for early walk_dir calls, and delay immediate background cleanup/capacity timers to reduce transient restart-time VolumeNotFound noise.
Also downgrade expected missing-path producer results during startup from generic errors to warnings while preserving existing storage semantics.
* fix(ecstore): harden issue3031 multipart validation path
- clear stale multipart part destinations before rename fan-out
- add repeated part overwrite regression coverage
- reduce remote disk startup false-fault escalation to suspect-first
- refine remote locker diagnostics and lower scanner leader-lock log noise
- add a dedicated 4-node issue3031 docker validation script
* refactor(admin): inline console version json macro
- drop the unused serde_json::json import in admin console
- call serde_json::json! inline in version_handler
- keep the console version response behavior unchanged
* fix(remote-disk): recover suspect health on probe success
- record probe success during remote disk health checks so suspect drives recover
- use async_with_vars for the remote disk health probe test
- make the missing-listener test assert the state transition more robustly
* fix(lock): isolate retry attempt lock ids
Generate a fresh lock id for each distributed lock retry attempt so late pending cleanup from an earlier attempt cannot release a lock acquired by a later retry.
Also add a regression test covering late-success cleanup versus retried acquisition.
* style(lock): normalize retry race regression test
Normalize the distributed lock retry race regression test formatting without changing behavior.
* test(lock): remove quorum-order assumption
Make the retry race regression test assert the safety property without assuming which delayed client enters the retry quorum first.
* fix(readiness): gate on lock quorum health
Include distributed lock quorum in runtime readiness decisions and expose the signal in health responses.
Map namespace lock quorum failures to ServiceUnavailable instead of InternalError so clients can safely retry while keeping quorum enforcement unchanged.
* fix(ecstore): restore namespace lock error decoding
Add the missing from_u32 arm for NamespaceLockQuorumUnavailable and cover the numeric roundtrip in error tests.
* fix(deploy): simplify systemd service startup
* docs(deploy): add nixos systemd service example
* docs(deploy): add nixos service guide
* chore(deps): update pulsar and pyroscope
* docs(deploy): refine nixos service guidance
* fix(deploy): use explicit rustfs server entrypoint
The service template only handled ClusterIP and NodePort via if/else-if
branches. When service.type=LoadBalancer was set, no branch matched and
the type field was omitted from the rendered Service, causing Kubernetes
to silently default to ClusterIP.
Add the missing LoadBalancer branch with support for:
- loadBalancerIP: request a specific IP (e.g. Cilium LB-IPAM, MetalLB)
- loadBalancerClass: select a specific LB implementation
- loadBalancerSourceRanges: restrict client source IPs
This is particularly useful for bare-metal / on-prem clusters using
software load balancers (Cilium LB-IPAM, MetalLB, kube-vip) where
a stable routable IP is preferable to an Ingress for S3 clients.
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: cxymds <Cxymds@qq.com>
Co-authored-by: majinghe <42570491+majinghe@users.noreply.github.com>
* fix(heal): restore single disk data during deep heal
* fix(heal): clarify erasure heal step logging
* fix(heal): avoid duplicate deep object heals
* fix(ecstore): preserve volume-not-found walk errors
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
Object repair rebuilt missing data shards but left missing parity shards unrecreated. A parity-only repair could therefore complete without restoring the shard contents.
Add a repair-specific reconstruction path that regenerates parity after data shards are available, while keeping normal read decoding data-only. Also shut down repair writers after all blocks are written.
Constraint: Preserve read-path decode behavior and limit parity regeneration to repair.
Confidence: high
Scope-risk: moderate
Directive: Do not route read decoding through parity regeneration without measuring the cost.
Tested: cargo test -p rustfs-ecstore decode_data_and_parity -- --nocapture
Tested: cargo test -p rustfs-ecstore heal_reconstructs_missing_parity_shard -- --nocapture
Tested: cargo test -p rustfs-ecstore erasure_coding -- --nocapture
Tested: cargo test -p rustfs heal_object_marks_missing_shard_disk_dirty_for_capacity_manager -- --nocapture
Tested: cargo test -p rustfs-heal -- --nocapture
Tested: cargo clippy -p rustfs-ecstore --all-targets -- -D warnings
Tested: cargo fmt --all --check
Tested: make pre-commit
Not-tested: Live multi-node disk replacement outside local test harness
* Preserve multipart replication recovery state
Multipart uploads previously only scheduled replication after completion, leaving no persisted pending state for scanner recovery if the initial async work was lost. Persist the same pending replication metadata during multipart initialization and let completion evaluate the object metadata that was actually stored.
The scanner heal path also treated ordinary pending objects as delete-replication candidates. Restrict that path to delete markers and version purge state so pending objects remain eligible for object replication heal.
Constraint: Bucket replication recovery depends on persisted object metadata after the async queue is unavailable.
Rejected: Rely only on immediate completion-time scheduling | it cannot recover after process restart or worker loss.
Confidence: high
Scope-risk: moderate
Directive: Keep multipart upload initialization aligned with single PUT replication metadata semantics.
Tested: cargo fmt --all --check
Tested: cargo clippy --workspace --all-features --all-targets -- -D warnings
Tested: LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8 make pre-commit
Tested: Runtime replication outage check confirmed multipart xl.meta stores PENDING status and timestamp.
* fix(replication): preserve version purge scanner state
Role-derived replication configs need target-scoped status strings before scanner heal can build per-target purge status. The duplicated replication-status assignment left version purge status unset, so scanner recovery could lose the target-level purge state.
Constraint: Scanner heal derives per-target purge decisions from version_purge_status_internal.
Rejected: Leave the duplicate as a harmless cleanup | it changes recovery behavior for role-only configs with version purge state.
Confidence: high
Scope-risk: narrow
Directive: Keep role-derived replication and version purge internal status mapping symmetric.
Tested: cargo fmt --all --check
Tested: cargo test -p rustfs-ecstore heal -- --nocapture
---------
Co-authored-by: wly <wlywly0735@126.com>
Co-authored-by: houseme <housemecn@gmail.com>
* perf(ecstore): use direct std writes for local disk
* fix(ecstore): avoid blocking disk writes in local disk path
* fix(ecstore): use tokio async write in local disk path
* chore(ecstore): remove unused AsyncWriteExt import
* fix(ecstore): preserve write semantics and avoid bytes copy
* fix(ecstore): optimize write_all_internal buffer paths
* fix(ecstore): sync write path in local disk
* fix(ecstore): align sync flag docs and avoid ref copy
* chore(rust): format local write path
---------
Co-authored-by: houseme <housemecn@gmail.com>
* refactor(config): centralize internode transport constants
* fix(bench): guard all ripgrep calls behind dry-run check
Move require_cmd rg and metrics collection inside the non-dry-run
path so that --dry-run works on hosts without rg installed.
* feat(tooling): cross-platform protoc setup for Linux and macOS
Make install-protoc.sh support Linux (x86_64, aarch64) alongside
macOS, and bump CI protoc from 29.3 to 33.1 to match the version
required by the gproto build script.
* fix(bench): record internode baseline error counts
* fix(skill): correct YAML frontmatter formatting for release-version-bump
* chore(ci): bump protoc version to 34.1
* fix(tooling): bump protoc 33.1 to 34.1 in install script, restore SKILL.md description
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
* chore(deps): refresh workspace deps and linux fs_type gating
- refresh workspace dependency pins and lockfile updates
- remove now-unused crate dependency entries in multiple Cargo.toml files
- enable profiling export defaults in config and scripts/run.sh
- gate os::fs_type module/function/tests to Linux to avoid non-Linux dead_code warnings
* fix(utils): simplify fs_type linux gating
- keep fs_type module-level linux cfg in os::mod
- remove redundant linux cfg on get_fs_type and test module
* chore(deps): bump s3s git revision
- update workspace s3s dependency to rev 507e1312b211c3ddc214b03875d6fabd15d22ed5
- refresh Cargo.lock source entry for s3s
* chore(dev): allow mysql_async git source and env overrides
- allow mysql_async git source in deny.toml allow-git list
- make scripts/run.sh core env vars overrideable via existing shell env
* fix(utils): import get_fs_type in fs_type tests
- add explicit super::get_fs_type import in fs_type test module
- fix Linux E0425 unresolved function errors in unit tests
* chore(dev): tune run script observability defaults
- make profiling export env overrideable in scripts/run.sh
- set RUSTFS_OBS_SAMPLE_RATIO default from 2.0 to 1.0
- update allow-git review window comments in deny.toml
* test(obs): stabilize profiling env alias tests
Move build_internode_data_transport_from_env() out of RemoteDisk::new()
into new_disk(), so RemoteDisk depends only on the
Arc<dyn InternodeDataTransport> trait object. TCP/HTTP remains the
default backend; all data paths unchanged.
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
* feat(internode): p0 transport baseline and ci hardening
* fix(internode): avoid double wrapping transport errors
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
* docs: add internode data transport RFC
* feat: add internode operation metrics
* fix feedback
* fix(ci): fallback protoc token to github.token
* feat(ecstore): add internode transport boundary and baseline runner
* feat(internode): harden data transport baseline
* Revert "feat(internode): harden data transport baseline"
This reverts commit 5b8d6b8aa4.
* fix(internode): address baseline review comments
* fix(ci): pin setup-protoc to stable release
* fix(ci): install protoc via apt on linux
* fix(ci): restore protoc install for macos and windows
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
* fix: keep scanner walk timeouts from offlining drives
Scanner walk operations can time out on large or slow directory listings without proving the backing drive is faulty. Keep the timeout error local to the scan while preserving failure marking for ordinary disk operations.
Constraint: Scanner walk_dir can include listing work that exceeds the drive timeout under slow storage.
Rejected: Disable timeout failure marking globally | real stuck disk operations must still affect drive health
Confidence: high
Scope-risk: narrow
Directive: Do not route scanner/listing timeout back into drive offline state without reproducing issue #2651
Tested: cargo test -p rustfs-ecstore timeout
Tested: cargo fmt --all --check
Tested: cargo clippy --workspace --all-features --all-targets -- -D warnings
Tested: make pre-commit with en_US.UTF-8 locale
Related: https://github.com/rustfs/rustfs/issues/2651
* fix: keep remote scanner walks from offlining drives
Remote walk_dir uses a streaming request that can hit total or stall timeouts during large scans without proving the remote drive is faulty. Route that scanner path through an explicit ignore action while preserving failure marking for ordinary remote operations.
Also treat Duration::ZERO as no operation timeout for remote health tracking, matching the existing local disk wrapper contract while still marking network-like errors on default paths.
Constraint: Scanner walk_dir streams can be slow because of directory size or consumer backpressure.
Rejected: Disable remote timeout failure marking globally | normal remote disk operations still need to evict failed connections and update drive health
Confidence: high
Scope-risk: narrow
Directive: Do not reintroduce remote scanner timeout failure marking without reproducing issue #2651 against remote disks.
Tested: cargo test -p rustfs-ecstore execute_with_timeout
Tested: cargo test -p rustfs-ecstore timeout
Tested: cargo fmt --all --check
Tested: cargo clippy --workspace --all-features --all-targets -- -D warnings
Tested: make pre-commit with en_US.UTF-8 locale
Related: https://github.com/rustfs/rustfs/issues/2651
* test: prove scanner walk backpressure keeps drives online
Add a local scanner walk regression test where the output writer never makes progress. The test confirms the walk timeout returns without marking the drive faulty, covering a stall source that is not caused by object count or network transfer speed.
Constraint: Scanner walk can block on downstream writer backpressure as well as disk or network IO.\nConfidence: high\nScope-risk: narrow\nTested: cargo test -p rustfs-ecstore walk_dir_writer_backpressure_timeout_does_not_mark_drive_failure\nTested: cargo test -p rustfs-ecstore timeout\nTested: cargo fmt --all --check\nTested: cargo clippy --workspace --all-features --all-targets -- -D warnings\nTested: make pre-commit
---------
Co-authored-by: houseme <housemecn@gmail.com>
* fix: unblock empty chunked admin rebalance start
Some admin clients send empty POST requests with chunked framing and no explicit content length. The admin rebalance route carries no request body, so normalize that known route before downstream validation sees a missing length.
Constraint: Keep normalization scoped to known empty-body admin routes
Rejected: Relax transfer-encoding handling for S3 routes | broader protocol risk
Confidence: high
Scope-risk: narrow
Tested: cargo test -p rustfs --lib server::layer::tests::
Tested: cargo fmt --all --check
Tested: git diff --check
* fix: keep rebalance listing from stalling
Large object migration was awaited inside the listing callback, so metacache readers could stop being drained while drive walk operations were still writing listing data. Move entry migration into bounded background tasks and wait for them after each set listing completes.
Constraint: Preserve existing rebalance cancellation and first-error propagation
Rejected: Only increase drive walk timeouts | masks callback backpressure and still fails for slower migrations
Confidence: medium
Scope-risk: moderate
Tested: cargo test -p rustfs-ecstore rebalance_unit_tests
Tested: cargo test -p rustfs-ecstore cache_value::metacache_set::tests
Tested: cargo test -p rustfs --lib server::layer::tests::
Tested: cargo clippy -p rustfs-ecstore --lib --all-features -- -D warnings
Tested: cargo clippy -p rustfs-ecstore --tests --all-features -- -D warnings
Tested: cargo fmt --all --check
Tested: git diff --check
---------
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: cxymds <Cxymds@qq.com>
Added a security philosophy section emphasizing transparency and community contributions. Updated the reporting process for vulnerabilities to ensure responsible disclosure.
Signed-off-by: loverustfs <hello@rustfs.com>
description: Resolve failures from the repository's architecture guard scripts — check_layer_dependencies.sh, check_architecture_migration_rules.sh, check_unsafe_code_allowances.sh, check_logging_guardrails.sh, check_doc_paths.sh. Use when make pre-commit / pre-pr or CI fails on one of these checks.
---
# Architecture Guard Checks
All five run in `make pre-commit` / `make pre-pr` and in CI. Fix the cause;
never weaken a check to get green.
## `check_layer_dependencies.sh` — layer DAG in `rustfs/src`
Enforces `interface (admin, storage/ecfs, storage/s3_api) → app → infra`; no
upward imports. Known legacy violations live in
`scripts/layer-dependency-baseline.txt`.
- **New violation**: restructure your change so the dependency points
downward (move the shared type/function to the lower layer).
- **You legitimately removed a baseline entry**: run
`./scripts/check_layer_dependencies.sh --update-baseline` and commit the
shrunken baseline. Never add new entries to the baseline to make a new
description: Verify code changes by identifying correctness, regression, security, and performance risks from diffs or patches, then produce prioritized findings with file/line evidence and concrete fixes. Use when reviewing commits, PRs, and merged patches before/after release.
---
# Code Change Verification
Use this skill to review code changes consistently before merge, before release, and during incident follow-up.
## Quick Start
1. Read the scope: commit, PR, patch, or file list.
2. Map each changed area by risk and user impact.
3. Inspect each risky change in context.
4. Report findings first, ordered by severity.
5. Close with residual risks and verification recommendations.
- If context is missing, state assumptions before deeper analysis.
- Focus only on requested scope; avoid reviewing unrelated files.
### 2) Risk map
- Prioritize in this order:
- Data correctness and user-visible behavior
- API/contract compatibility
- Security and authz/authn boundaries
- Concurrency and lifecycle correctness
- Performance and resource usage
- Give higher priority to stateful paths, migration logic, defaults, and error handling.
### 3) Evidence-based inspection
- Read each modified hunk with neighboring context.
- Trace call paths and call-site expectations.
- Check for:
- invariant breaks and missing guards
- unchecked assumptions and null/empty/error-path handling
- stale tests, fixtures, and configs
- hidden coupling to shared helpers/constants/features
- If a point is uncertain, mark it as an open question instead of guessing.
#### Rust-specific checks (apply to all Rust changes)
- **unwrap/expect in production**: Search changed files for `.unwrap()` and `.expect(` outside test modules. Every `unwrap()` in production code must have a justification comment or be replaced with `?`.
- **Silent type truncation**: Search for `as u8/u16/u32/u64/usize/i8/i16/i32/i64/isize` casts. Every `as` cast must be justified; negative-to-unsigned and large-to-small are bugs by default. Use `try_into()` or explicit clamping.
- **Unnecessary cloning**: Check `.clone()` calls in loops, per-request paths, and on structs with >5 heap-allocated fields. Consider `Arc`, references, or `Cow<str>`.
- **Lock ordering**: If the change acquires multiple locks, verify the order matches all other call sites. Document the order in a comment.
- **Locks across .await**: Flag any `tokio::sync::RwLock`/`Mutex` guard held across an `.await` point without bounded hold time.
- **Recursion depth**: If the change adds or modifies a recursive function, verify it has a depth limit or uses iterative traversal with an explicit stack.
- **Error types**: Flag `Result<_, String>`, `Box<dyn Error>`, and missing `Error::source()` implementations in public APIs.
- **Test assertions**: Every test function must have at least one `assert!`. Flag tests that only call code without verifying results.
- **println/eprintln**: Search changed files for `println!`/`eprintln!` outside test modules. Production code must use `tracing` macros.
- **Serde safety**: Structs deserialized from untrusted input (S3 API, user config) should have `#[serde(deny_unknown_fields)]`.
### 4) Findings-first output
- Order findings by severity:
- P0: critical failure, security breach, or data loss risk
- P1: high-impact regression
- P2: medium risk correctness gap
- P3: low risk/quality debt
- For each finding include:
- Severity
-`path:line` reference
- concise issue statement
- impact and likely failure mode
- specific fix or mitigation
- validation step to confirm
- If no issues exist, explicitly state `No findings` and why.
short_description:"Prioritize risks and verify code changes before merge."
default_prompt:"Inspect a patch or diff, identify correctness/security/regression risks, and return prioritized findings with file/line evidence and fixes."
description: Prepare PR-ready diffs by validating scope, checking required verification steps, drafting a compliant English PR title/body, and surfacing blockers before opening or updating a pull request in RustFS.
---
# PR Creation Checker
Use this skill before `gh pr create`, before `gh pr edit`, or when reviewing whether a branch is ready for PR.
## Read sources of truth first
- Read `AGENTS.md`.
- Read `.github/pull_request_template.md`.
- Use `Makefile` and `.config/make/` for local quality commands.
- Use `.github/workflows/ci.yml` for CI expectations.
- Do not restate long command matrices or template sections from memory when the files exist.
## Workflow
1. Collect PR context
- Confirm base branch, current branch, change goal, and scope.
- Confirm whether the task is: draft a new PR, update an existing PR, or preflight-check readiness.
- Confirm whether the branch includes only intended changes.
2. Inspect change scope
- Review the diff and summarize what changed.
- Call out unrelated edits, generated artifacts, logs, or secrets as blockers.
- Mark risky areas explicitly: auth, storage, config, network, migrations, breaking changes.
- Scan the diff for newly added string literals and confirm whether they duplicate values already defined as constants/enums/typed wrappers in the same module or shared modules.
- Treat introducing a new hardcoded literal where a project constant already exists as a likely regression risk; require either a refactor to reuse the constant or an explicit exception explanation in the PR body.
3. Verify readiness requirements
- Require `make pre-commit` before marking PRs ready when the diff changes Rust code, product behavior, CI behavior, runtime configuration, security-sensitive logic, migrations, storage, auth, networking, or other high-risk paths.
- For documentation-only, agent-instruction-only, or local developer-tooling-only changes, allow focused verification instead of `make pre-commit` when it directly validates the changed surface.
- For focused verification, explain why the full gate was not run and list the scope-specific commands in the PR body.
- If `make` is unavailable, use the equivalent commands from `.config/make/`.
- Add scope-specific verification commands when the changed area needs more than the baseline.
- If required checks fail, stop and return `BLOCKED`.
4. Draft PR metadata
- Write the PR title in English using Conventional Commits and keep it within 72 characters.
- If a generic PR workflow suggests a different title format, ignore it and follow the repository rule instead.
- In RustFS, do not use tool-specific prefixes such as `[codex]` when the repository requires Conventional Commits.
- Keep the PR body in English.
- Use the exact section headings from `.github/pull_request_template.md`.
- Fill non-applicable sections with `N/A`.
- Include verification commands in the PR description.
- Do not include local filesystem paths in the PR body unless the user explicitly asks for them.
- Prefer repo-relative paths, command names, and concise summaries over machine-specific paths such as `/Users/...`.
5. Prepare reviewer context
- Summarize why the change exists.
- Summarize what was verified.
- Call out risks, rollout notes, config impact, and rollback notes when applicable.
- Mention assumptions or missing context instead of guessing.
6. Prepare CLI-safe output
- When proposing `gh pr create` or `gh pr edit`, use `--body-file`, never inline `--body` for multiline markdown.
- Return a ready-to-save PR body plus a short title.
- If not ready, return blockers first and list the minimum steps needed to unblock.
## Output format
### Status
-`READY` or `BLOCKED`
### Title
-`<type>(<scope>): <summary>`
### PR Body
- Reproduce the repository template headings exactly.
- Fill every section.
- Omit local absolute paths unless explicitly required.
### Verification
- List each command run.
- State pass/fail.
### Risks
- List breaking changes, config changes, migration impact, or `N/A`.
## Blocker rules
- Return `BLOCKED` if a code, behavior, CI, runtime configuration, security-sensitive, migration, storage, auth, networking, or other high-risk change has not passed `make pre-commit`.
- Return `BLOCKED` if a documentation-only, agent-instruction-only, or local developer-tooling-only change lacks focused verification for the changed surface.
- Return `BLOCKED` if the diff contains unrelated changes that are not acknowledged.
- Return `BLOCKED` if required template sections are missing.
- Return `BLOCKED` if the title/body is not in English.
- Return `BLOCKED` if the title does not follow the repository's Conventional Commit rule.
- Return `BLOCKED` if the diff introduces string literals that should use existing constants but did not.
## Reference
- Use [pr-readiness-checklist.md](references/pr-readiness-checklist.md) for a short final pass before opening or editing the PR.
- Confirm no secrets, logs, temp files, or unrelated refactors are included.
- Confirm `make pre-commit` passed for code, behavior, CI, runtime configuration, security-sensitive, migration, storage, auth, networking, or other high-risk changes.
- For documentation-only, agent-instruction-only, or local developer-tooling-only changes, confirm focused verification covered the changed surface and the PR body explains why the full gate was not run.
- Confirm extra verification commands are listed for risky changes.
- Confirm the PR title uses Conventional Commits and stays within 72 characters.
- Confirm the PR title does not use tool-specific prefixes such as `[codex]`.
- Confirm the PR body is in English.
- Confirm the PR body keeps the exact headings from `.github/pull_request_template.md`.
- Confirm non-applicable sections are filled with `N/A`.
- Confirm the PR body does not include local absolute paths unless explicitly required.
- Confirm multiline GitHub CLI commands use `--body-file`.
- Confirm new hardcoded string literals were not introduced for values already represented by existing constants/enums (including protocol labels, error identifiers, headers, and metric names), or record a justified exception.
description: Enforce Rust-specific code quality rules on every code change. Use before merge to catch unwrap abuse, silent truncation, unnecessary cloning, lock ordering violations, recursion risks, and error type anti-patterns.
---
# Rust Code Quality Gate
Use this skill on every Rust code change to enforce quality rules that `cargo clippy` does not catch.
## Quick Start
1. Identify changed `.rs` files.
2. Run automated checks on changed files.
3. Run manual review checklist on the diff.
4. Report findings; block merge if P0/P1 issues exist.
## Automated Checks
Run these on every changed `.rs` file (excluding test modules):
description: Standardize and review RustFS logging with structured `tracing` events, lower noise on hot paths, preserve security-sensitive diagnostics, and extend guardrails to prevent legacy logging patterns from returning. Use when editing or reviewing RustFS logs, startup/config diagnostics, cloud metadata logs, request validation logs, or `scripts/check_logging_guardrails.sh`.
---
# RustFS Logging Governance
Use this skill when RustFS logging needs to be added, cleaned up, reviewed, or protected against regressions.
## Quick Start
1. Identify the files whose logs are changing.
2. Scan current `tracing` or `log` macros before editing.
3. Convert sentence-style logs to short event-style logs.
4. Demote hot-path success logs unless operators truly need them at `info`.
5. Preserve failure, fallback, and security-relevant diagnostics.
6. Update `scripts/check_logging_guardrails.sh` when a broad cleanup removes a legacy pattern class.
7. Validate with formatting, targeted checks/tests, and the logging guardrail script.
## Core Workflow
### 1. Scope the logging surface
- Read the changed module in full before touching log lines.
- Classify the log site:
- lifecycle/startup
- request or validation path
- background loop or hot path
- fallback/degraded behavior
- cloud metadata or external fetch path
- metrics/config summary
- Do not rewrite business logic to make logging easier.
### 2. Use the RustFS event shape
- Prefer fields first, message second.
- Use short labels, not prose paragraphs.
- Default field shape:
-`event`
-`component`
-`subsystem`
-`state` or `result`
- key context fields
- Reuse stable field names and avoid inventing near-duplicates.
See `references/logging-governance.md` for the event model, level policy, and anti-pattern list.
### 3. Choose the right level
-`error`: operation failure that affects behavior or security guarantees.
-`warn`: degraded path, fallback, suspicious input, or operator-actionable misconfiguration.
-`info`: low-frequency lifecycle or mode selection.
-`debug`: targeted diagnostics and low-volume detail.
-`trace`: hot-path and repetitive success-path events.
When in doubt, lower the verbosity of normal success paths and keep structured detail in fields.
### 4. Preserve security and privacy boundaries
- Do not log secrets, tokens, auth headers, raw credential payloads, or merged config dumps.
- Avoid logging raw forwarded headers or full trusted network inventories above `debug`.
- Keep warning/error logs useful without echoing attacker-controlled payloads unnecessarily.
### 5. Keep summaries aggregated
- Replace multi-line startup banners or checklist logs with one structured event.
- If metrics already express a concept, avoid duplicating it with many `info!` lines.
- Prefer counts, modes, and sources over inventories unless debug detail is truly needed.
### 6. Update guardrails when needed
- Broad logging cleanup should usually extend `scripts/check_logging_guardrails.sh`.
- Add forbidden patterns only for styles the repo has intentionally retired:
- sentence-style lifecycle logs
- noisy hot-path `info!`
- checklist-style summary logs
- legacy fallback wording that has been replaced by structured fields
Use `Cargo.toml``[workspace].members` as the source of truth for crate membership. When doing a broad logging sweep, classify crates by operational role so logs stay consistent within each role.
description: "Publish a RustFS alpha/beta/stable release with an auditable flow: confirm target version and scope, update workspace and release assets (including strict rustfs.spec changelog identity/date/version format), run required verification, and finish with commit, push, and GitHub PR creation."
---
# RustFS Release Version Bump
Use this skill to publish a RustFS release (alpha, beta, or stable) with a minimal, auditable diff and a complete ship flow (`edit -> verify -> commit -> push -> PR`).
Validated baseline: release pattern used in PR `#2957`.
## Required inputs
- Exact target version, for example `1.0.0-beta.4`.
- Delivery scope:
- Local only (`edit/verify`).
- Local + git (`commit/push`).
- Full GitHub flow (`commit/push/PR`).
If target version is missing or ambiguous, stop and ask before editing.
## Read before editing
-`AGENTS.md` (root and nearest path-specific files).
-`.github/pull_request_template.md`.
- Current branch status and diff against `origin/main`.
## Default release file scope
Treat the following file list as the default checklist for each release bump:
-`Cargo.toml`
-`Cargo.lock`
-`README.md`
-`README_ZH.md`
-`flake.nix`
-`helm/rustfs/Chart.yaml`
-`rustfs.spec`
Only drop a file when the current repository release process clearly no longer requires it.
## Hard release policy
- Docker doc tags use `<version>` (for example `rustfs/rustfs:1.0.0-beta.4`), not `v<version>`.
- Helm chart version mapping follows `beta.N -> 0.N.0`.
-`rustfs.spec``Release` uses prerelease suffix only (for example `beta.4`).
- Do not change these rules without explicit confirmation.
## Step-by-step workflow
1. Confirm intent and isolate scope
- Confirm target version string exactly.
- Confirm whether user requested local-only or full GitHub flow.
- Inspect current branch and ensure only release-related files are touched for this task.
2. Update workspace versions
- Bump `[workspace.package].version` in `Cargo.toml`.
- Bump internal workspace crate dependency versions in `Cargo.toml`.
- Update `Cargo.lock` so workspace package versions match target version.
- Re-scan for partial leftovers.
3. Update release assets
-`README.md` and `README_ZH.md`: update versioned Docker examples to target version.
-`flake.nix`: update package version to target version.
-`helm/rustfs/Chart.yaml`:
-`appVersion` = target version.
-`version` follows chart mapping rule, for example:
-`1.0.0-beta.3` -> `0.3.0`
-`1.0.0-beta.4` -> `0.4.0`
-`rustfs.spec`:
- Set `Release` to prerelease suffix (example `beta.4`).
- Add/update top changelog entry with exact format:
-`* Thu May 20 2026 houseme <housemecn@gmail.com>`
-`- Update RPM package to RustFS 1.0.0-beta.4`
- Changelog identity and time must come from current environment:
-`git config --get user.name`
-`git config --get user.email`
-`date '+%a %b %d %Y'`
- Changelog version text must match target release version exactly.
4. Verify before shipping
- Run:
-`cargo fmt --all`
-`cargo fmt --all --check`
-`make pre-commit`
- If verification passes, run `cargo clean`.
- If `make pre-commit` fails, return `BLOCKED` with root cause and do not silently widen scope to fix unrelated issues unless user asks.
5. Commit strategy
- Preferred split when both parts changed:
-`chore(release): prepare <version>` for `Cargo.toml` and `Cargo.lock`.
-`chore(release): align release assets for <version>` for docs and packaging files.
- If user asks for one commit, use one commit.
- Stage only intended release files; do not include unrelated working tree changes.
short_description:"Prepare RustFS release branches like PR #2957."
default_prompt:"Use $rustfs-release-version-bump to prepare a RustFS release version, ask about any unclear version policy, and finish the commit/push/PR flow."
description: Apply RustFS security lessons distilled from repository GitHub Security Advisories. Use when making or reviewing RustFS code changes, doing security checks, handling PR review for auth/authz, IAM, storage, RPC, logging, CORS, console/browser, encryption, policy, or endpoint changes, and when deciding which security regression tests are required.
---
# RustFS Security Advisory Lessons
Use this skill as a RustFS-specific security lens before changing or approving code. For the distilled advisory lessons and review patterns, read [advisory-patterns.md](references/advisory-patterns.md).
When currentness matters, fetch the live advisory inventory instead of relying on this skill as a status mirror:
```bash
gh api repos/rustfs/rustfs/security-advisories --paginate \
- Treat these paths as security-sensitive by default: `rustfs/src/admin/`, `rustfs/src/storage/`, `rustfs/src/auth.rs`, `rustfs/src/server/layer.rs`, `crates/iam/`, `crates/policy/`, `crates/credentials/`, `crates/ecstore/src/rpc/`, `crates/protocols/`, `crates/rio/`, and console preview/auth code.
### 2. Map to advisory classes
- Read [advisory-patterns.md](references/advisory-patterns.md) for matching GHSA lessons.
- Do not rely on advisory titles alone. Confirm whether the issue is authentication, authorization, input validation, storage invariant, browser isolation, logging, or operational hardening.
### 3. Verify fail-closed behavior
- Check that unauthenticated, wrong-permission, cross-user, cross-bucket, malformed-input, and default-config cases fail explicitly.
- Prefer exact action/permission checks over broad helper calls or inferred ownership.
- Confirm lower storage/RPC layers do not bypass checks done in upper layers.
### 4. Require regression evidence
- For behavior changes, add focused negative tests that reproduce the advisory class.
- For sensitive fixes, include tests for the bypass form, not only the happy path.
- If a test is impractical, explain the residual risk and provide a manual verification command.
### 5. Report clearly
- Lead with concrete findings and file/line evidence.
- Separate proven vulnerabilities from hardening risks.
- Avoid exaggerating unauthenticated impact when the code actually rejects unauthenticated requests but allows a low-privileged authenticated bypass.
## Advisory-Derived Guardrails
### Auth and admin authorization
- Every admin or diagnostic route needs an explicit authn and authz story. Route registration, router whitelist, and handler-level authorization must agree.
- Match the admin action to the operation exactly. Copy-paste action constants are a known RustFS vulnerability class.
- Avoid authentication-only helpers for state-changing admin APIs; use `validate_admin_request` or the established equivalent with the right `AdminAction`.
- Read-only admin APIs such as metrics, server info, and diagnostics still require admin authorization; checking only that credentials exist is not enough.
- Replication admin reads can expose remote target credentials; list/get target endpoints require replication/admin authorization and must not return secrets to low-privilege callers.
- Do not assume admin-action `Resource` scoping constrains blast radius unless the policy engine actually enforces resources for that action.
### IAM and service accounts
- Treat imported IAM payload fields as attacker-controlled: `parent`, `claims`, `accessKey`, `secretKey`, status, policy names, and groups.
- For service account create/update/import, prove parent ownership or root/admin authority before writing credentials or claims.
- Do not let `deny_only` or "no explicit deny" become an allow decision that skips required allow checks.
- Test cross-user list/update/import flows with wrong, correct, self, parent, and root identities.
### S3 copy, multipart, and presigned POST
- Multipart copy must enforce source `GetObject` and destination `PutObject` semantics equivalent to `CopyObject`, including copy-source and policy conditions.
- Do not let `CreateMultipartUpload`, `UploadPartCopy`, `CompleteMultipartUpload`, or `AbortMultipartUpload` return success without authorization.
- Presigned POST policies are server-side contracts. Enforce `content-length-range`, key prefix, exact metadata/content-type, and all signed policy conditions.
### Protocol frontends and IAM parity
- FTP/FTPS, SFTP, gateway, and other protocol drivers must enforce IAM per operation before calling storage backends; authentication to a protocol listener is not authorization.
- Match protocol commands to the same S3 actions as HTTP, such as `RETR` to `GetObject`, `SIZE`/`MDTM` to `HeadObject`, `MKD` to `CreateBucket`, and bucket probes to `ListBucket` or `HeadBucket`.
- Review every handler in a protocol driver, not only the changed handler, because RustFS advisories show mixed guarded and unguarded siblings in the same driver.
- Regression tests for protocol frontends should deny the shared authorization hook and prove the backend is not reached for the denied command.
- Compare protocol secrets in constant time, normalize invalid-user and invalid-secret failures where practical, and add rate limiting before exposing password-style protocol endpoints.
### Paths, object keys, and filesystem access
- Never join untrusted bucket/object/RPC path strings onto filesystem roots without normalization and boundary checks.
- Reject or safely handle `..`, absolute paths, URL-encoded traversal, platform separators, empty components, and paths that canonicalize outside the intended root.
- Validate both S3 object-key paths and internode/RPC disk paths; storage helpers can bypass S3 authorization if they trust already-parsed paths.
- Archive auto-extract paths are object keys too. Validate tar/zip entry names before IAM checks and before storage writes, and prove cleaned paths cannot cross bucket or prefix boundaries.
### Secrets, default credentials, and crypto
- Do not ship hard-coded shared tokens, HMAC secrets, private keys, or production test keys.
- Defaults for root credentials and internode/RPC auth must fail closed for network-reachable deployments or generate per-install random secrets; warnings alone are not a security boundary.
- Keep cryptographic roles separated: root S3 credentials, RPC HMAC keys, and STS/JWT signing keys must not be reused or deterministically derived from each other.
- License or token validation must use signatures with embedded public/verifying keys only; do not use private-key decryption as authenticity.
- Plan key rotation and key IDs when removing exposed keys.
### Logging and debug output
- Logs must never include access keys beyond safe identifiers, secret keys, session tokens, JWT claims, HMAC secrets, expected signatures, license secrets, or raw response bodies containing credentials.
- Treat `Debug` implementations, `?value` tracing, merged config dumps, and dependency-level HTTP body logging as leak surfaces.
- Add log-capture tests or targeted unit tests for redaction wrappers when changing credential structs or response bodies.
### RPC, parsing, and panic safety
- Treat all RPC payload bytes as attacker-controlled. Replace `unwrap`, `expect`, and panic-prone deserialization with typed errors.
- Malformed request tests should cover empty bytes, truncated MessagePack/protobuf, invalid enum values, stale timestamps, and invalid signatures.
- RPC authentication must be independently strong; do not depend on S3 admin credentials unless the fallback is explicit and safe.
- RPC signatures must bind the exact generated gRPC method path, timestamp, and request method. Service-prefix signatures must not authorize a different concrete NodeService call.
### Browser, CORS, and console surfaces
- Do not reflect arbitrary `Origin` while also allowing credentials. Default CORS should be no CORS unless explicitly configured.
- Do not render user-controlled object content in a same-origin iframe with console credentials available to JavaScript.
- Prefer origin separation for object preview/download, `nosniff`, CSP, strict content-type handling, and avoiding durable credentials in `localStorage`.
- Preview safety must be based on trusted content type and sandboxing, not object names or extensions such as `.pdf`.
- Console license/version-like metadata endpoints should expose only coarse public data unless authenticated, especially subject names and expiration timestamps.
### Profiling, debug, and health endpoints
- Profiling and debug endpoints are not health checks. They require admin auth, opt-in enablement, rate limiting, and safe responses.
- Do not return absolute filesystem paths or other deployment layout in unauthenticated or low-privilege responses.
- Ensure health endpoint allowlists cannot accidentally include expensive diagnostics.
### Trusted proxy and network identity
- Only honor `X-Forwarded-For` or `X-Real-IP` when the request came from a configured trusted proxy.
- Direct clients must use the socket peer address for `aws:SourceIp` and policy condition evaluation.
- Add tests for direct spoofed headers and trusted-proxy headers.
### SSE and storage invariants
- Encryption metadata is not proof that bytes were encrypted on disk.
- When touching reader/writer wrappers such as hashing, encryption, compression, or warp readers, verify wrapper order and inspect stored bytes in regression tests.
- Avoid helper shortcuts that unwrap nested readers and accidentally bypass encryption or integrity layers.
## Review Prompts
Use these prompts while reviewing a diff:
- Could a low-privileged authenticated user reach this path with the wrong action, parent, bucket, or source object?
- Does a non-HTTP protocol path call the same authorization boundary as the S3 API before touching storage?
- Does a public/default/empty config change security behavior from fail-closed to fail-open?
- Is any attacker-controlled value later used as a path, policy condition, credential identity, log field, URL, Origin, or response body?
- Does this response contain stored replication, remote target, or service credentials that need redaction or stricter authorization?
- Is an archive entry, object key, or policy resource normalized differently between authorization and storage?
- Is the same operation implemented in multiple paths, such as `CopyObject` vs `UploadPartCopy`, and do all paths enforce the same security contract?
- Does a preview or browser-surface fix preserve the original security invariant when adding alternate viewers or file-type detection?
- Does the test prove the exploit form is denied, or only that the intended form still works?
gh api repos/rustfs/rustfs/security-advisories/<GHSA_ID>
```
Update this file only when an advisory adds or changes a reusable lesson, affected surface, validation pattern, or regression-test expectation. Do not update it for state-only, URL-only, count-only, or timestamp-only changes.
## Pattern Index
### Admin authorization and route exposure
-`GHSA-pfcq-4gjr-6gjm`: notification target endpoints accepted authenticated users but skipped admin authorization. Lesson: distinguish authn from authz; admin target CRUD must call the operation-specific admin authorization path.
-`GHSA-mm2q-qcmx-gw4w`: `ListServiceAccount` used `UpdateServiceAccountAdminAction`, while update lacked target ownership checks. Lesson: exact action constants and ownership checks are both required; information disclosure can chain into secret rotation and takeover.
-`GHSA-vcwh-pff9-64cc`: `ImportIam` checked `ExportIAMAction` for an import/write operation. Lesson: every admin handler must authorize the action it actually performs.
-`GHSA-jqmc-mg33-v45g` and `GHSA-8784-9m7f-c6p6`: `/profile/cpu` and `/profile/memory` were whitelisted from auth and allowed expensive diagnostics plus path disclosure. Lesson: profiling/debug endpoints need admin auth, opt-in, rate limits, and non-sensitive responses.
-`GHSA-f5cv-v44x-2xgf`: `/rustfs/admin/v3/metrics` accepted any authenticated IAM user and skipped admin authorization. Lesson: read-only metrics and diagnostic admin endpoints still require an operation-specific admin action check.
-`GHSA-796f-j7xp-hwf4`: `/rustfs/admin/v3/list-remote-targets` checked only that credentials existed and leaked replication target credentials. Lesson: replication target reads are privileged admin operations, and stored remote credentials require strict authz plus response redaction review.
-`GHSA-xp32-gxq2-3v52`: console license metadata endpoint was public and exposed subject and expiration fields. Lesson: management metadata endpoints should require admin auth or return only coarse public status.
### IAM import, service accounts, and privilege boundaries
-`GHSA-566f-q62r-wcr8`: `ImportIam` accepted attacker-controlled service account `parent`, `claims`, `accessKey`, and `secretKey`, enabling persistent backdoor accounts under root. Lesson: imported IAM payloads are untrusted data and must be validated against privilege boundaries.
-`GHSA-xgr5-qc6w-vcg9`: `deny_only=true` skipped allow checks and let restricted service accounts mint unrestricted children. Lesson: deny-only logic must never become implicit allow for privilege creation.
-`GHSA-mm2q-qcmx-gw4w`: leaked service account access keys plus update-without-ownership formed an escalation chain. Lesson: service-account identifiers are security-sensitive because update APIs consume them.
### S3 copy, multipart, and upload policy validation
-`GHSA-mx42-j6wv-px98`: `UploadPartCopy` missed source authorization and allowed cross-bucket object exfiltration. Lesson: multipart copy must enforce the same source and destination contract as `CopyObject`.
-`GHSA-wfxj-ph3v-7mjf`: `UploadPartCopy` checked source and destination independently but missed destination copy-source policy constraints. Lesson: source read and destination write checks are not sufficient when policy constrains allowed copy sources.
-`GHSA-w5fh-f8xh-5x3p`: presigned POST accepted uploads without enforcing signed policy conditions. Lesson: parse and enforce all POST policy constraints server-side, including size, key prefix, and content type.
### Protocol frontends and IAM parity
-`GHSA-3g29-xff2-92vp`: FTP `RETR` and `SIZE`/`MDTM` read paths authenticated the user but skipped IAM before calling storage. Lesson: non-HTTP protocol frontends must enforce the same per-operation authorization as the S3 API before backend access.
-`GHSA-g3vq-vv42-f647`: FTPS `MKD` called `create_bucket` without checking `s3:CreateBucket`. Lesson: protocol command handlers need action-specific checks even when sibling handlers already authorize correctly.
-`GHSA-3p3x-734c-h5vx`: FTPS and WebDAV compared secret keys with early-return string equality, while FTPS also returned distinguishable invalid-user and invalid-password failures. Lesson: password-style protocol auth needs constant-time secret comparison, indistinguishable failures where practical, and rate limiting.
### Filesystem paths and object key traversal
-`GHSA-pq29-69jg-9mxc`: RPC `read_file_stream` joined untrusted paths under a volume directory without canonical boundary checks. Lesson: `PathBuf::join` plus length checks are not path security.
-`GHSA-8r6f-hmq2-28rg`: object keys containing traversal sequences bypassed bucket/object authorization when mapped to filesystem paths. Lesson: reject traversal at object-key parsing and verify final storage paths remain under the expected bucket/key root.
-`GHSA-f4vq-9ffr-m8m3`: Snowball auto-extract accepted archive entries such as `../victim-bucket/object`, authorized the raw attacker-bucket path, then storage path cleaning crossed bucket boundaries. Lesson: archive entries become object keys and need traversal rejection plus consistent authz/storage normalization before writes.
### Secrets, defaults, and cryptographic misuse
-`GHSA-j59h-h7q5-q348`: RustFS shipped known default root credentials that could authenticate to S3, admin APIs, IAM, KMS, and console surfaces. Lesson: root credentials must be operator-provided or generated per install; known defaults and warnings are not acceptable for network-reachable deployments.
-`GHSA-h956-rh7x-ppgj`: gRPC used the hard-coded token `rustfs rpc` on both client and server. Lesson: source-visible shared tokens are authentication bypasses.
-`GHSA-r5qv-rc46-hv8q`: internode RPC HMAC secret fell back to the public default `rustfsadmin`. Lesson: RPC/internode auth must fail closed instead of silently using public defaults.
-`GHSA-75fx-qg6f-8rm7` and `GHSA-68cw-96m3-h2cf`: internode RPC secrets were derivable from known root credentials, making raw storage RPC signatures forgeable when explicit RPC secrets were unset. Lesson: RPC auth keys must be independent random secrets, never derived from S3 root credentials, and raw storage RPC should not share the public S3 listener without an internode-only boundary.
-`GHSA-m77q-r63m-pj89`: STS JWTs used the root secret key as the shared token signing key, allowing token forgery when the root secret was known. Lesson: STS signing keys need key separation, rotation, and key IDs; do not reuse root credentials for JWT/HMAC signing.
-`GHSA-923g-jp7v-f97f`: license verification embedded a production RSA private key and used private-key decryption as authenticity. Lesson: ship verifying/public keys only and use real signature verification.
### Sensitive logging and debug output
-`GHSA-r54g-49rx-98cr`: STS credentials were logged at info level. Lesson: generated credentials must never be logged in plaintext.
-`GHSA-8cm2-h255-v749`: debug logs leaked session tokens, secret keys, JWT claims, and raw STS response bodies. Lesson: redaction must cover custom `Debug` implementations and dependency response-body logging.
-`GHSA-333v-68xh-8mmq`: invalid RPC signature logging included the shared HMAC secret and expected signature. Lesson: error paths often leak secrets; never log raw secrets or derived authenticators.
### RPC input validation and panic safety
-`GHSA-gw2x-q739-qhcr`: malformed gRPC `GetMetrics` payloads reached `unwrap()` on deserialization and caused remote DoS. Lesson: every network/RPC deserialization failure returns an error, not a panic.
-`GHSA-h956-rh7x-ppgj` and `GHSA-r5qv-rc46-hv8q`: weak RPC auth increased reachability of otherwise internal handlers. Lesson: panic bugs become more severe when internode auth is weak or defaulted.
-`GHSA-c667-rgrv-99vj`: NodeService authentication signed the service prefix instead of the concrete generated method path, so valid metadata for one RPC could be replayed to another method during the timestamp window. Lesson: RPC HMAC payloads must bind exact gRPC method path, HTTP method surrogate, timestamp, and secret.
### Browser, CORS, and console isolation
-`GHSA-v9fg-3cr2-277j`: object preview rendered attacker-controlled HTML in a same-origin iframe, exposing console credentials stored in `localStorage`. Lesson: user content must be origin-isolated from the console and protected with `nosniff`, CSP, and strict content-type handling.
-`GHSA-7gcx-wg4x-q9x6`: an incomplete preview fix reintroduced extension-based PDF detection and bypassed the sandboxed fallback for attacker-controlled content. Lesson: browser-surface fixes need regression tests for alternate viewers and file-type branches, and preview trust must come from validated content type plus sandboxing rather than object names.
-`GHSA-x5xv-223c-8vm7`: default CORS reflected arbitrary origins with credentials. Lesson: never combine reflected origins with `Access-Control-Allow-Credentials: true`; default should be fail-closed.
### Trusted proxy and source IP conditions
-`GHSA-fc6g-2gcp-2qrq`: `aws:SourceIp` trusted client-supplied `X-Forwarded-For` or `X-Real-IP`. Lesson: forwarded IP headers are valid only behind configured trusted proxies; direct clients use socket peer IP.
### SSE and on-disk storage invariants
-`GHSA-xrrf-67jm-3c2r`: SSE metadata reported encryption while reader composition bypassed `EncryptReader` and stored plaintext. Lesson: test actual bytes on disk and wrapper order, not only API metadata.
### Serde deserialization and input validation
- No `#[serde(deny_unknown_fields)]` found across the entire codebase. Lesson: all structs deserialized from untrusted input (S3 API XML/JSON, lifecycle rules, bucket policies, replication configs) should have `#[serde(deny_unknown_fields)]` to reject malformed or adversarial payloads.
-`#[serde(default)]` on security-critical fields silently accepts missing values as zero/empty. Lesson: when a field has security implications (retention days, permissions, limits), validate the deserialized value explicitly rather than relying on defaults.
- Integer fields deserialized from user input and cast with `as` (e.g., `i32 as u32`) can wrap negative values to large positives. Lesson: validate ranges before casting; use `try_into()` or clamp.
- XML config typos (e.g., `"NoncurentDays"` instead of `"NoncurrentDays"`) are silently accepted when `deny_unknown_fields` is absent. Lesson: strict deserialization prevents silent misconfiguration that could cause data loss or unexpected retention behavior.
## Useful Search Seeds
Use these targeted searches when a diff touches security-sensitive code:
- Authz fixes: include unauthenticated, valid low-privilege, wrong-action, correct-action, owner, non-owner, and root/admin cases as applicable.
- Protocol frontend authz fixes: include denied `RETR`, `SIZE`/`MDTM`, `MKD`, bucket probe, and sibling allowed-operation cases, and assert denied paths do not reach the storage backend.
- IAM fixes: include import/update/list service-account cases with attacker-controlled parent, claims, access key, secret key, and policy.
- Copy/upload fixes: include cross-bucket, cross-user, source-denied, destination-denied, copy-source-condition, and multipart completion cases.
- Path fixes: include encoded traversal, absolute path, nested traversal, archive entries with `..`, valid object keys that resemble traversal text but should be rejected, and canonical bucket/prefix boundary checks.
- Logging fixes: assert redacted output for structs and response bodies that may contain credentials.
- RPC auth fixes: include captured metadata replay across two concrete methods, stale timestamps, wrong path, wrong method surrogate, wrong secret, and valid same-method calls.
- Browser/CORS fixes: assert no credentials on reflected/default origins, correct behavior for explicit allowlists, and no same-origin script execution for previewed object content.
- SSE fixes: inspect stored bytes and verify API metadata, read-back behavior, and on-disk ciphertext together.
description: Run project coverage checks, rank high-risk gaps, and propose high-impact tests to improve regression confidence for changed and critical code paths before release.
---
# Test Coverage Improver
Use this skill when you need a prioritized, risk-aware plan to improve tests from coverage results.
## Usage assumptions
- Focus scope is either changed lines/files, a module, or the whole repository.
- Coverage artifact must be generated or provided in a supported format.
- If required context is missing, call out assumptions explicitly before proposing work.
## Workflow
1. Define scope and baseline
- Confirm target language, framework, and branch.
- Confirm whether the scope is changed files only or full-repo.
## —— Pre Commit Checks ----------------------------------------------------------------------------
.NOTPARALLEL:pre-commitpre-prdev-check
.PHONY:setup-hooks
setup-hooks:## Set up git hooks
@echo "🔧 Setting up git hooks..."
chmod +x .git/hooks/pre-commit
@echo "✅ Git hooks setup complete!"
.PHONY:doc-paths-check
doc-paths-check:## Check that instruction/architecture docs reference existing file paths
@echo "📄 Checking doc path references..."
./scripts/check_doc_paths.sh
.PHONY:pre-commit
pre-commit:fmt-checkunsafe-code-checkarchitecture-migration-checklogging-guardrails-checktokio-io-uring-checkdoc-paths-checkquick-check## Run fast pre-commit checks without clippy/full tests
@echo "✅ All pre-commit checks passed!"
.PHONY:pre-pr
pre-pr:fmt-checkunsafe-code-checkarchitecture-migration-checklogging-guardrails-checktokio-io-uring-checkdoc-paths-checkclippy-checktest## Run full pre-PR checks with clippy and tests
@echo "✅ All pre-PR checks passed!"
.PHONY:dev-check
dev-check:fmt-checkunsafe-code-checkarchitecture-migration-checklogging-guardrails-checktokio-io-uring-checkdoc-paths-checkquick-check## Run fast local development checks
This directory contains Docker configuration files and supporting infrastructure for building and running RustFS container images.
This directory contains the complete Docker infrastructure for building, deploying, and monitoring RustFS. It provides ready-to-use configurations for development, testing, and production-grade observability.
## 📁 Directory Structure
## 📂 Directory Structure
```
rustfs/
├── Dockerfile # Production image (Alpine + pre-built binaries)
├── Dockerfile.source # Development image (Debian + source build)
| `source` | Debian Bookworm | Source build | Medium | Custom builds with cross-compilation |
| `dev` | Debian Bookworm | Development tools | Large | Interactive development |
The following files in the project root are essential for Docker operations:
## 🚀 Usage Examples
### Build Scripts & Dockerfiles
### Quick Start (Production)
| File | Description | Usage |
| :--- | :--- | :--- |
| **`docker-buildx.sh`** | **Multi-Arch Build Script**<br>Automates building and pushing Docker images for `amd64` and `arm64`. Supports release and dev channels. | `./docker-buildx.sh --push` |
| **`Dockerfile.glibc`** | **Production Image (Ubuntu)**<br>Standard image using glibc. Useful if you need specific dynamic libraries. | `docker build -f Dockerfile.glibc .` |
| **`Dockerfile.source`** | **Development Image**<br>Builds RustFS from source code. Includes build tools. Ideal for local development and CI. | `docker build -f Dockerfile.source .` |
### Docker Compose Configurations
| File | Description | Usage |
| :--- | :--- | :--- |
| **`docker-compose.yml`** | **Main Development Setup**<br>Comprehensive setup with profiles for development, observability, and proxying. | `docker compose up -d`<br>`docker compose --profile observability up -d` |
| **`docker-compose-simple.yml`** | **Quick Start Setup**<br>Minimal configuration running a single RustFS instance with 4 volumes. Perfect for first-time users. | `docker compose -f docker-compose-simple.yml up -d` |
---
## 🌟 Observability Stack (Recommended)
Located in: [`.docker/observability/`](observability/README.md)
We provide a comprehensive, industry-standard observability stack designed for deep insights into RustFS performance. This is the recommended setup for both development and production monitoring.
This directory contains specialized Docker Compose configurations for different use cases.
This directory contains specialized Docker Compose configurations for specific testing scenarios.
## ⚠️ Important Note
**For Observability:**
We **strongly recommend** using the new, fully integrated observability stack located in `../observability/`. It provides a production-ready setup with Prometheus, Grafana, Tempo, Loki, and OpenTelemetry Collector, all with persistent storage and optimized configurations.
The `docker-compose.observability.yaml` in this directory is kept for legacy reference or specific minimal testing needs but is **not** the primary recommended setup.
## 📁 Configuration Files
This directory contains specialized Docker Compose configurations and their associated Dockerfiles, keeping related files organized together.
### Cluster Testing
### Main Configuration (Root Directory)
- **`docker-compose.cluster.yaml`**
- **Purpose**: Simulates a 4-node RustFS distributed cluster.
- **Use Case**: Testing distributed storage logic, consensus, and failover.
- **Nodes**: 4 RustFS instances.
- **Storage**: Uses local HTTP endpoints.
- **`../../docker-compose.yml`** - **Default Production Setup**
- Complete production-ready configuration
- Includes RustFS server + full observability stack
This directory contains the observability stack for the application. The stack is composed of the following components:
This directory contains the comprehensive observability stack for RustFS, designed to provide deep insights into application performance, logs, and traces.
- Prometheus v3.2.1
- Grafana 11.6.0
- Loki 3.4.2
- Jaeger 2.4.0
- Otel Collector 0.120.0 # 0.121.0 remove loki
## Components
## Prometheus
The stack is composed of the following best-in-class open-source components:
Prometheus is a monitoring and alerting toolkit. It scrapes metrics from instrumented jobs, either directly or via an
intermediary push gateway for short-lived jobs. It stores all scraped samples locally and runs rules over this data to
either aggregate and record new time series from existing data or generate alerts. Grafana or other API consumers can be
used to visualize the collected data.
- **Prometheus** (v2.53.1): The industry standard for metric collection and alerting.
- **Grafana** (v11.1.0): The leading platform for observability visualization.
- **Loki** (v3.1.0): A horizontally-scalable, highly-available, multi-tenant log aggregation system.
- **Tempo** (v2.5.0): A high-volume, minimal dependency distributed tracing backend.
- **Jaeger** (v1.59.0): Distributed tracing system (configured as a secondary UI/storage).
- **OpenTelemetry Collector** (v0.104.0): A vendor-agnostic implementation for receiving, processing, and exporting telemetry data.
## Grafana
By default, this stack uses Tempo in single-binary mode and does not require Kafka/Redpanda.
If you want the Kafka-backed HA Tempo path, use `docker-compose-example-for-rustfs.yml` together with `docker-compose-tempo-ha-override.yml`.
Grafana is a multi-platform open-source analytics and interactive visualization web application. It provides charts,
graphs, and alerts for the web when connected to supported data sources.
## Architecture
## Loki
1.**Telemetry Collection**: Applications send OTLP (OpenTelemetry Protocol) data (Metrics, Logs, Traces) to the **OpenTelemetry Collector**.
2.**Processing & Exporting**: The Collector processes the data (batching, memory limiting) and exports it to the respective backends:
The `prometheus` service mounts the `prometheus.yml` file to `/etc/prometheus/prometheus.yml`. The `grafana` service
mounts the `grafana/provisioning` directory to `/etc/grafana/provisioning`. You can modify these files to configure the
observability stack.
### Dashboard Usage
The dashboards are automatically provisioned when Grafana starts. They use the `${DS_PROMETHEUS}` datasource variable, so you need a Prometheus datasource configured in Grafana.
Key panels to monitor during optimization rollout:
1.**GET Latency by Reader Path** - Compare `codec_streaming` vs `legacy_duplex` latency
2.**Early-Stop Hit Rate** - Verify early-stop is triggering effectively
3.**Pipeline Failure Rate** - Detect any new failure modes introduced by optimizations
4.**Bitrot Verify Failures** - Ensure data integrity is maintained
## Quick Start
### Prerequisites
- Docker
- Docker Compose
### Deploy
Run the following command to start the entire stack:
```bash
docker compose up -d
```
### High Availability Tempo
The default `docker-compose.yml` is the single-node stack.
If you need the Kafka-backed HA Tempo configuration, start it with:
```bash
docker compose -f docker-compose-example-for-rustfs.yml -f docker-compose-tempo-ha-override.yml up -d
"expr":"sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_scheduled_sum[$__rate_interval])) / clamp_min(sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_scheduled_count[$__rate_interval])), 1) or vector(0)",
"expr":"sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_attempted_sum[$__rate_interval])) / clamp_min(sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_attempted_count[$__rate_interval])), 1) or vector(0)",
"expr":"sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_ready_sum[$__rate_interval])) / clamp_min(sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_ready_count[$__rate_interval])), 1) or vector(0)",
"legendFormat":"ready {{strategy}} / {{mode}}",
"refId":"C"
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"expr":"sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_deferred_sum[$__rate_interval])) / clamp_min(sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_deferred_count[$__rate_interval])), 1) or vector(0)",
"expr":"histogram_quantile(0.95, sum by (path, stage, size_bucket, le) (rate(rustfs_io_get_object_stage_duration_seconds_by_size_bucket{stage=~\"reader_setup|reader_setup_schedule|reader_setup_wait_quorum|reader_setup_drop_pending|reader_task_file_open|reader_task_reader_construction|reader_task_bitrot_reader_init\"}[$__rate_interval]))) or vector(0)",
"expr":"histogram_quantile(0.99, sum by (path, stage, size_bucket, le) (rate(rustfs_io_get_object_stage_duration_seconds_by_size_bucket{stage=~\"reader_setup|reader_setup_schedule|reader_setup_wait_quorum|reader_setup_drop_pending|reader_task_file_open|reader_task_reader_construction|reader_task_bitrot_reader_init\"}[$__rate_interval]))) or vector(0)",
"expr":"histogram_quantile(0.95, sum by (path, stage, le) (rate(rustfs_io_get_object_stage_duration_seconds_bucket{stage=~\"request_context|first_byte|response_handoff|full_body|body_build|output_poll|output_lock_wait|reader_open_mmap_copy_success|reader_open_mmap_copy_fallback|reader_open_stream|reader_stream_first_read|reader_mmap_blocking_wait|reader_mmap_blocking_task|reader_mmap_file_open|reader_mmap_map|reader_mmap_copy_buffer|reader_mmap_direct_read_copy\"}[$__rate_interval]))) or vector(0)",
"legendFormat":"p95 {{path}} / {{stage}}",
"refId":"A"
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"expr":"histogram_quantile(0.99, sum by (path, stage, le) (rate(rustfs_io_get_object_stage_duration_seconds_bucket{stage=~\"request_context|first_byte|response_handoff|full_body|body_build|output_poll|output_lock_wait|reader_open_mmap_copy_success|reader_open_mmap_copy_fallback|reader_open_stream|reader_stream_first_read|reader_mmap_blocking_wait|reader_mmap_blocking_task|reader_mmap_file_open|reader_mmap_map|reader_mmap_copy_buffer|reader_mmap_direct_read_copy\"}[$__rate_interval]))) or vector(0)",
"legendFormat":"p99 {{path}} / {{stage}}",
"refId":"B"
}
],
"title":"GET Reader, Handler and Body Latency",
"type":"timeseries"
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"fieldConfig":{
"defaults":{
"unit":"ops",
"custom":{
"drawStyle":"line",
"lineInterpolation":"smooth",
"fillOpacity":10,
"pointSize":5,
"lineWidth":1,
"showPoints":"auto",
"spanNulls":false,
"thresholdsStyle":{
"mode":"off"
}
},
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":null
}
]
}
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":0,
"y":24
},
"id":7,
"options":{
"legend":{
"calcs":[
"mean",
"max"
],
"displayMode":"table",
"placement":"bottom"
},
"reduceOptions":{
"calcs":[
"lastNotNull"
],
"fields":"",
"values":false
},
"tooltip":{
"mode":"multi",
"sort":"desc"
}
},
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"expr":"sum by (subpath) (rate(rustfs_io_get_object_direct_memory_subpath_total[$__rate_interval])) or vector(0)",
"legendFormat":"{{subpath}}",
"refId":"A"
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"expr":"sum by (subpath, size_bucket) (rate(rustfs_io_get_object_direct_memory_subpath_by_size_total[$__rate_interval])) or vector(0)",
"legendFormat":"{{subpath}} / {{size_bucket}}",
"refId":"B"
}
],
"title":"GET Direct-Memory Subpath Rate",
"type":"timeseries"
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"fieldConfig":{
"defaults":{
"unit":"ops",
"custom":{
"drawStyle":"line",
"lineInterpolation":"smooth",
"fillOpacity":10,
"pointSize":5,
"lineWidth":1,
"showPoints":"auto",
"spanNulls":false,
"thresholdsStyle":{
"mode":"off"
}
},
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":null
}
]
}
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":12,
"y":24
},
"id":8,
"options":{
"legend":{
"calcs":[
"mean",
"max"
],
"displayMode":"table",
"placement":"bottom"
},
"reduceOptions":{
"calcs":[
"lastNotNull"
],
"fields":"",
"values":false
},
"tooltip":{
"mode":"multi",
"sort":"desc"
}
},
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"expr":"sum by (path, stage, kind) (rate(rustfs_io_get_object_mmap_page_faults_total[$__rate_interval])) or vector(0)",
"title":"GET Latency by Reader Path (p50 / p95 / p99)",
"type":"timeseries",
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"expr":"histogram_quantile(0.50, sum(rate(rustfs_io_get_object_stage_duration_seconds_bucket{stage=\"total\"}[$__rate_interval])) by (le, path))",
"legendFormat":"p50 {{path}}",
"refId":"A"
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"expr":"histogram_quantile(0.95, sum(rate(rustfs_io_get_object_stage_duration_seconds_bucket{stage=\"total\"}[$__rate_interval])) by (le, path))",
"legendFormat":"p95 {{path}}",
"refId":"B"
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"expr":"histogram_quantile(0.99, sum(rate(rustfs_io_get_object_stage_duration_seconds_bucket{stage=\"total\"}[$__rate_interval])) by (le, path))",
"legendFormat":"p99 {{path}}",
"refId":"C"
}
]
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"fieldConfig":{
"defaults":{
"unit":"percentunit",
"min":0,
"max":1,
"custom":{
"drawStyle":"line",
"lineInterpolation":"smooth",
"fillOpacity":15,
"pointSize":5,
"lineWidth":1,
"showPoints":"auto",
"spanNulls":false,
"thresholdsStyle":{
"mode":"off"
}
},
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":null
},
{
"color":"yellow",
"value":0.3
},
{
"color":"red",
"value":0.7
}
]
}
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":12,
"y":0
},
"id":2,
"options":{
"reduceOptions":{
"values":false,
"calcs":["lastNotNull"],
"fields":""
},
"tooltip":{
"mode":"multi",
"sort":"desc"
},
"legend":{
"displayMode":"table",
"placement":"bottom",
"calcs":["mean","max"]
}
},
"title":"Early-Stop Hit Rate",
"type":"timeseries",
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"expr":"sum(rate(rustfs_io_get_object_metadata_early_stop_total{decision=\"hit\"}[$__rate_interval])) by (path) / clamp_min(sum(rate(rustfs_io_get_object_metadata_early_stop_total[$__rate_interval])) by (path), 1)",
"legendFormat":"hit rate {{path}}",
"refId":"A"
}
]
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"fieldConfig":{
"defaults":{
"unit":"percentunit",
"min":0,
"max":1,
"custom":{
"drawStyle":"line",
"lineInterpolation":"smooth",
"fillOpacity":15,
"pointSize":5,
"lineWidth":1,
"showPoints":"auto",
"spanNulls":false,
"thresholdsStyle":{
"mode":"off"
}
},
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":null
}
]
}
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":0,
"y":8
},
"id":3,
"options":{
"reduceOptions":{
"values":false,
"calcs":["lastNotNull"],
"fields":""
},
"tooltip":{
"mode":"multi",
"sort":"desc"
},
"legend":{
"displayMode":"table",
"placement":"bottom",
"calcs":["mean","max"]
}
},
"title":"Codec Streaming Usage Rate",
"type":"timeseries",
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"expr":"sum(rate(rustfs_io_get_object_codec_streaming_decision_total[$__rate_interval])) by (decision) / clamp_min(sum(rate(rustfs_io_get_object_codec_streaming_decision_total[$__rate_interval])), 1)",
"legendFormat":"codec_streaming {{decision}}",
"refId":"A"
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"expr":"sum(rate(rustfs_io_get_object_stream_strategy_total[$__rate_interval])) by (strategy) / clamp_min(sum(rate(rustfs_io_get_object_stream_strategy_total[$__rate_interval])), 1)",
"legendFormat":"stream_strategy {{strategy}}",
"refId":"B"
}
]
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"fieldConfig":{
"defaults":{
"unit":"ops",
"custom":{
"drawStyle":"bars",
"fillOpacity":80,
"lineWidth":0,
"pointSize":5,
"showPoints":"never",
"spanNulls":false,
"stacking":{
"mode":"normal",
"group":"A"
},
"thresholdsStyle":{
"mode":"off"
}
},
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":null
},
{
"color":"red",
"value":1
}
]
}
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":12,
"y":8
},
"id":4,
"options":{
"reduceOptions":{
"values":false,
"calcs":["lastNotNull"],
"fields":""
},
"tooltip":{
"mode":"multi",
"sort":"desc"
},
"legend":{
"displayMode":"table",
"placement":"bottom",
"calcs":["sum"]
}
},
"title":"Pipeline Failure Rate by Stage",
"type":"timeseries",
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"expr":"sum(rate(rustfs_io_get_object_pipeline_failures_total[$__rate_interval])) by (stage, reason)",
"legendFormat":"{{stage}} / {{reason}}",
"refId":"A"
}
]
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"fieldConfig":{
"defaults":{
"unit":"reqps",
"custom":{
"drawStyle":"line",
"lineInterpolation":"smooth",
"fillOpacity":10,
"pointSize":5,
"lineWidth":1,
"showPoints":"auto",
"spanNulls":false,
"thresholdsStyle":{
"mode":"off"
}
},
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":null
}
]
}
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":0,
"y":16
},
"id":5,
"options":{
"reduceOptions":{
"values":false,
"calcs":["lastNotNull"],
"fields":""
},
"tooltip":{
"mode":"multi",
"sort":"desc"
},
"legend":{
"displayMode":"table",
"placement":"bottom",
"calcs":["mean","max"]
}
},
"title":"GET Request Rate by Path",
"type":"timeseries",
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"expr":"sum(rate(rustfs_io_get_object_reader_path_total[$__rate_interval])) by (path)",
"legendFormat":"{{path}}",
"refId":"A"
}
]
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"fieldConfig":{
"defaults":{
"unit":"ops",
"custom":{
"drawStyle":"line",
"lineInterpolation":"smooth",
"fillOpacity":10,
"pointSize":5,
"lineWidth":1,
"showPoints":"auto",
"spanNulls":false,
"thresholdsStyle":{
"mode":"off"
}
},
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":null
}
]
}
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":12,
"y":16
},
"id":6,
"options":{
"reduceOptions":{
"values":false,
"calcs":["lastNotNull"],
"fields":""
},
"tooltip":{
"mode":"multi",
"sort":"desc"
},
"legend":{
"displayMode":"table",
"placement":"bottom",
"calcs":["mean","max"]
}
},
"title":"GET Total Latency (p50 / p95 / p99)",
"type":"timeseries",
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"expr":"histogram_quantile(0.50, sum(rate(rustfs_io_get_object_total_duration_seconds_bucket[$__rate_interval])) by (le))",
"legendFormat":"p50",
"refId":"A"
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"expr":"histogram_quantile(0.95, sum(rate(rustfs_io_get_object_total_duration_seconds_bucket[$__rate_interval])) by (le))",
"legendFormat":"p95",
"refId":"B"
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"expr":"histogram_quantile(0.99, sum(rate(rustfs_io_get_object_total_duration_seconds_bucket[$__rate_interval])) by (le))",
"expr":"sum by (path, eager_status, size_bucket, buffer_bucket, large_concurrency_tuning) (rate(rustfs_s3_put_object_diagnostics_total[$__rate_interval])) or vector(0)",
"expr":"sum by (path) (rate(rustfs_s3_put_object_path_total[$__rate_interval])) or vector(0)",
"legendFormat":"{{path}}",
"refId":"A"
}
],
"title":"PUT Path Rate",
"type":"timeseries"
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"fieldConfig":{
"defaults":{
"unit":"ms",
"custom":{
"drawStyle":"line",
"lineInterpolation":"smooth",
"fillOpacity":10,
"pointSize":5,
"lineWidth":1,
"showPoints":"auto",
"spanNulls":false,
"thresholdsStyle":{
"mode":"off"
}
},
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":null
}
]
}
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":0,
"y":8
},
"id":3,
"options":{
"legend":{
"calcs":[
"mean",
"max"
],
"displayMode":"table",
"placement":"bottom"
},
"reduceOptions":{
"calcs":[
"lastNotNull"
],
"fields":"",
"values":false
},
"tooltip":{
"mode":"multi",
"sort":"desc"
}
},
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"expr":"histogram_quantile(0.95, sum by (stage, le) (rate(rustfs_s3_put_object_stage_duration_ms_bucket{stage=~\"ingress_prepare|set_disk_writer_setup|set_disk_encode|set_disk_rename|set_disk_old_data_cleanup|multipart_ingress_prepare|multipart_set_disk_writer_setup|multipart_set_disk_encode|multipart_complete_tail\"}[$__rate_interval]))) or vector(0)",
"legendFormat":"p95 {{stage}}",
"refId":"A"
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"expr":"histogram_quantile(0.99, sum by (stage, le) (rate(rustfs_s3_put_object_stage_duration_ms_bucket{stage=~\"ingress_prepare|set_disk_writer_setup|set_disk_encode|set_disk_rename|set_disk_old_data_cleanup|multipart_ingress_prepare|multipart_set_disk_writer_setup|multipart_set_disk_encode|multipart_complete_tail\"}[$__rate_interval]))) or vector(0)",
"legendFormat":"p99 {{stage}}",
"refId":"B"
}
],
"title":"PUT Hot Stage Latency",
"type":"timeseries"
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"fieldConfig":{
"defaults":{
"unit":"bytes",
"custom":{
"drawStyle":"line",
"lineInterpolation":"smooth",
"fillOpacity":10,
"pointSize":5,
"lineWidth":1,
"showPoints":"auto",
"spanNulls":false,
"thresholdsStyle":{
"mode":"off"
}
},
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":null
}
]
}
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":12,
"y":8
},
"id":4,
"options":{
"legend":{
"calcs":[
"mean",
"max"
],
"displayMode":"table",
"placement":"bottom"
},
"reduceOptions":{
"calcs":[
"lastNotNull"
],
"fields":"",
"values":false
},
"tooltip":{
"mode":"multi",
"sort":"desc"
}
},
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"expr":"histogram_quantile(0.50, sum by (path, size_bucket, le) (rate(rustfs_s3_put_object_selected_buffer_size_bytes_bucket[$__rate_interval]))) or vector(0)",
"legendFormat":"p50 {{path}} / {{size_bucket}}",
"refId":"A"
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"expr":"histogram_quantile(0.95, sum by (path, size_bucket, le) (rate(rustfs_s3_put_object_selected_buffer_size_bytes_bucket[$__rate_interval]))) or vector(0)",
"legendFormat":"p95 {{path}} / {{size_bucket}}",
"refId":"B"
}
],
"title":"PUT Selected Buffer Size",
"type":"timeseries"
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"fieldConfig":{
"defaults":{
"unit":"percentunit",
"min":0,
"max":1,
"custom":{
"drawStyle":"line",
"lineInterpolation":"smooth",
"fillOpacity":15,
"pointSize":5,
"lineWidth":1,
"showPoints":"auto",
"spanNulls":false,
"thresholdsStyle":{
"mode":"off"
}
},
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":null
}
]
}
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":0,
"y":16
},
"id":5,
"options":{
"legend":{
"calcs":[
"mean",
"max"
],
"displayMode":"table",
"placement":"bottom"
},
"reduceOptions":{
"calcs":[
"lastNotNull"
],
"fields":"",
"values":false
},
"tooltip":{
"mode":"multi",
"sort":"desc"
}
},
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"expr":"sum(rate(rustfs_s3_put_object_zero_copy_eligible_total[$__rate_interval])) / clamp_min(sum(rate(rustfs_s3_put_object_total[$__rate_interval])), 1) or vector(0)",
"legendFormat":"eligible / total",
"refId":"A"
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"expr":"sum(rate(rustfs_s3_put_object_zero_copy_enabled_total[$__rate_interval])) / clamp_min(sum(rate(rustfs_s3_put_object_total[$__rate_interval])), 1) or vector(0)",
"legendFormat":"enabled / total",
"refId":"B"
}
],
"title":"PUT Zero-Copy Eligibility Ratio",
"type":"timeseries"
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"fieldConfig":{
"defaults":{
"unit":"ops",
"custom":{
"drawStyle":"line",
"lineInterpolation":"smooth",
"fillOpacity":10,
"pointSize":5,
"lineWidth":1,
"showPoints":"auto",
"spanNulls":false,
"thresholdsStyle":{
"mode":"off"
}
},
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":null
}
]
}
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":12,
"y":16
},
"id":6,
"options":{
"legend":{
"calcs":[
"mean",
"max"
],
"displayMode":"table",
"placement":"bottom"
},
"reduceOptions":{
"calcs":[
"lastNotNull"
],
"fields":"",
"values":false
},
"tooltip":{
"mode":"multi",
"sort":"desc"
}
},
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"expr":"sum(rate(rustfs_s3_put_object_total[$__rate_interval])) or vector(0)",
"legendFormat":"PUT total",
"refId":"A"
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"expr":"sum by (stage) (rate(rustfs_system_storage_erasure_write_quorum_failures_total[$__rate_interval])) or vector(0)",
"legendFormat":"write quorum failure {{stage}}",
"refId":"B"
}
],
"title":"PUT Throughput and Quorum Failure Signals",
These compose files prepare 4-node, 4-disk clusters for rio/rio-v2 storage format compatibility checks. All disks are bind-mounted under `.docker/compat/data` so the on-disk files remain available on the host.
## Clusters
```bash
docker compose -f .docker/compat/docker-compose.rustfs-beta5.yml up -d --build
docker compose -f .docker/compat/docker-compose.minio.yml up -d
docker compose -f .docker/compat/docker-compose.rustfs-rio-v2.yml up -d --build
```
Default API endpoints:
- RustFS `1.0.0-beta.5`: `http://127.0.0.1:9100`
- MinIO: `http://127.0.0.1:9200`
- current main with `rio-v2`: `http://127.0.0.1:9300`
## Reading old datasets with rio-v2
Stop the writer cluster before mounting its disks into the rio-v2 cluster.
```bash
docker compose -f .docker/compat/docker-compose.rustfs-beta5.yml down
RUSTFS_RIO_V2_DATASET=./data/rustfs-beta5 \
docker compose -f .docker/compat/docker-compose.rustfs-rio-v2.yml up -d --build
docker compose -f .docker/compat/docker-compose.minio.yml down
RUSTFS_RIO_V2_DATASET=./data/minio \
docker compose -f .docker/compat/docker-compose.rustfs-rio-v2.yml up -d --build
```
## 200G object mix
Use the same bucket/object matrix against the beta5 and MinIO endpoints, then read it back through the rio-v2 endpoint. A practical 200G mix is:
- 1 KiB x 1024
- 1 MiB x 1024
- 64 MiB x 512
- 1 GiB x 64
- 8 GiB x 12
- 6 GiB x 1
Compression is enabled by default for RustFS and MinIO. Server-side KMS/SSE settings are intentionally left to environment variables or mounted key directories so real key material is not committed. For SSE-C cases, run the clusters with TLS because MinIO requires HTTPS for SSE-C.
## High-concurrency write/read stress
Use `run_rw_compat_stress.sh` to generate a manifest on an old endpoint, then verify the same objects through the rio-v2 endpoint after mounting the old disks.
```bash
.docker/compat/run_rw_compat_stress.sh \
--mode write \
--endpoint http://127.0.0.1:9100 \
--access-key rustfsadmin \
--secret-key rustfsadmin \
--bucket compat-beta5 \
--concurrency 96
RUSTFS_RIO_V2_DATASET=./data/rustfs-beta5 \
docker compose -f .docker/compat/docker-compose.rustfs-rio-v2.yml up -d --build
For encrypted datasets, add `--encryption sse-s3`, `--encryption sse-kms --sse-kms-key-id <key-id>`, or `--encryption sse-c --sse-c-key-file <raw-32-byte-key-file>` to both the write and verify commands.
## 5 GiB encrypted compatibility run
The `5g` profile covers 1 KiB, 1 MiB, 16 MiB, 64 MiB, and 1 GiB objects and totals exactly 5 GiB. Generate `compat-key.key` under `.docker/compat/kms/rustfs-compat`, enable local KMS on both RustFS clusters, then use the same encryption arguments while writing with beta5 and verifying with rio-v2. Set non-default local test credentials first because distributed listeners reject the built-in default credentials.
This directory contains a local three-site RustFS site replication check. It is intended to verify the admin site-replication flow against real containers:
- three independent RustFS sites start successfully
- the MinIO-compatible `mc admin replicate add` command configures all three sites
- a bucket and object written to site 1 are replicated to site 2 and site 3
-`mc admin replicate status` can read the resulting site-replication state
The compose file uses named volumes so the test does not require preparing host bind-mount directories.
Because Docker named volumes usually share the same physical device in local desktop environments, this test compose defaults `RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true`. Keep that setting limited to local test and CI environments.
## Files
-`docker-compose.yml`: three RustFS sites, a volume permission helper, and a one-shot setup/check container
-`run-object-flow-check.sh`: host-side upload/download verification for replicated 10 MiB to 100 MiB objects
## Ports
Default host endpoints:
- Site 1 API: `http://127.0.0.1:9000`
- Site 1 Console: `http://127.0.0.1:9001`
- Site 2 API: `http://127.0.0.1:9010`
- Site 2 Console: `http://127.0.0.1:9011`
- Site 3 API: `http://127.0.0.1:9020`
- Site 3 Console: `http://127.0.0.1:9021`
Default credentials are `rustfsadmin` / `rustfsadmin`. These are for local testing only.
## Run
From the repository root:
```bash
docker compose -f .docker/test/site-replication/docker-compose.yml up
```
To test a locally built image instead of Docker Hub `rustfs/rustfs:latest`, set `RUSTFS_SITE_REPL_IMAGE`:
The script creates five local files and uploads them from different sites:
- 10 MiB from site 1
- 25 MiB from site 2
- 50 MiB from site 3
- 75 MiB from site 1
- 100 MiB from site 2
For each uploaded object, the script waits for replication to the other two sites, downloads the object from those sites, and verifies both byte size and SHA-256 checksum. It uses a temporary `mc` config directory, so it does not overwrite existing host aliases.
The default bucket is `site-repl-flow-check`. Override it when needed:
docker compose -f .docker/test/site-replication/docker-compose.yml down -v
```
Use `down -v` before rerunning the full setup from scratch. Site replication state is persisted in the named volumes, so rerunning without deleting volumes may attempt to add an already-configured replication topology.
<!-- Briefly describe the main changes and motivation for this PR -->
<!--
Briefly explain what changed and why reviewers should accept it.
Focus on behavior, compatibility, and review-relevant context.
-->
## Checklist
- [ ] I have read and followed the [CONTRIBUTING.md](CONTRIBUTING.md) guidelines
- [ ] Passed `make pre-commit`
- [ ] Added/updated necessary tests
- [ ] Documentation updated (if needed)
- [ ] CI/CD passed (if applicable)
## Verification
<!--
List the commands or checks you ran, for example:
-`make pre-commit`
Use N/A only when verification is not applicable.
-->
## Impact
- [ ] Breaking change (compatibility)
- [ ] Requires doc/config/deployment update
- [ ] Other impact:
<!--
Describe user-facing, compatibility, API, deployment, configuration, or
documentation impact. Use N/A when there is no expected impact.
-->
## Additional Notes
<!-- Any extra information for reviewers -->
<!-- Any extra information for reviewers, or N/A. -->
---
Thank you for your contribution! Please ensure your PR follows the community standards ([CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md)) and sign the CLA if this is your first contribution.
Thank you for your contribution! Please ensure your PR follows the community standards ([CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md)). If this is your first contribution, review the [CLA document](https://github.com/rustfs/cla/blob/main/cla/v1.md) and sign it by commenting `I have read and agree to the CLA.` on the PR.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Zhengchao An