* refactor(obs): make dial9 telemetry opt-in and actually record events
The dial9 Tokio-runtime profiler was disabled by default, yet every build
paid for it, and enabling it produced trace files with no events in them.
Recorded empty traces
---------------------
`build_traced_runtime` called `TracedRuntime::builder()...build(..)`, but dial9
only starts recording in `build_and_start*`. `build` still returns a live guard
whose `is_enabled()` reports true, and still creates and seals segment files —
they just contain a header and no events. It also skipped `with_trace_path`, so
the background worker driving the segment pipeline was never spawned.
Measured on the new smoke example: 310 bytes of bare segment header, against
5640 bytes for the same workload once recording actually starts.
Switch to `with_trace_path(..).build_and_start(..)`.
Cost was unconditional
----------------------
`--cfg tokio_unstable` was a global `[build] rustflags` entry and `rustfs-obs`
depended on `dial9-tokio-telemetry` unconditionally, so all builds depended on
Tokio's non-semver API. Worse, an environment `RUSTFLAGS` replaces (never
appends to) the config-file value, so any caller exporting their own RUSTFLAGS
silently dropped the flag — the long comment in build.yml was a scar from that.
dial9 is now an opt-in feature (`dial9`, plus `dial9-s3` and `dial9-taskdump`),
the global rustflag is gone, and `crates/obs/build.rs` fails the compile if the
feature is on without the flag. Telemetry builds go through `make build-profiling`.
Metrics that could not lie
--------------------------
`rustfs_dial9_{events_total,bytes_written_total,rotations_total,cpu_overhead_percent}`
were hard-coded to zero — a Counter pinned at 0 reads as "nothing happened".
Removed. `rustfs_dial9_enabled` was sourced from the environment, so it read 1
even when the traced runtime failed and the process fell back to a standard
runtime; it is replaced by `rustfs_dial9_supported` (compile-time),
`rustfs_dial9_configured` (intent) and `rustfs_dial9_active_sessions` (reality).
No `writer_healthy` gauge is exported: dial9's `RotatingWriter` can enter its
`Finished` state and stop writing, but exposes no way to observe that, so the
gauge could only ever be hard-coded to 1. Documented as a known gap instead.
Final events were lost
----------------------
The `TelemetryGuard` lived in a `static OnceLock`, which is never dropped, so
buffered events were never flushed at exit. `build_tokio_runtime` now returns
the guard and `run_process` drops it before any exit path.
Also
----
- `disk_usage_bytes` was a `read_dir` + per-file `stat` on the metrics
collection path. It is now sampled by a background task into an atomic.
- `SAMPLING_RATE`/`S3_BUCKET`/`S3_PREFIX` were parsed, warned about, and
discarded. S3 upload is now wired to dial9's `with_s3_uploader` behind
`dial9-s3`; `SAMPLING_RATE` has no upstream equivalent and is removed.
- Wire `with_task_dumps` (async backtraces of stalled tasks), configurable via
`RUSTFS_RUNTIME_DIAL9_TASK_DUMP_{ENABLED,IDLE_THRESHOLD_MS}`.
- Split `telemetry/dial9.rs` into `config`/`state`/`enabled`/`disabled`; the
stub keeps the public API identical so callers need no `#[cfg]`.
- Drop four print-only examples and the manual test bin that exercised the
removed `init_session` scaffolding.
Verified: cargo check/clippy/test across default, `dial9`, and `dial9-s3`;
build.rs correctly rejects `dial9` without `--cfg tokio_unstable`;
`make pre-commit` passes.
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(obs): document dial9 as an on-demand profiler
scripts/run.sh advertised a `SAMPLING_RATE` knob that was never passed to dial9,
and claimed "CPU overhead < 5% (with sampling rate 1.0)" and "lower values reduce
CPU overhead" on the strength of it. The knob is gone; the guidance built on it
had to go too.
Replace it with what is actually true: dial9 needs a `make build-profiling`
binary, its disk budget evicts oldest-first (so a high poll rate can overwrite
the incident you are chasing), and it cannot be toggled without a restart.
Add docs/operations/dial9-runtime-profiling.md covering the build variants, an
investigation walkthrough, the configuration table, how to read the three
supported/configured/active_sessions gauges against each other, and the upstream
gap that makes writer death only indirectly observable.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(obs): add a dial9 smoke example that proves events are recorded
The bug this guards against is invisible to every existing signal: with
`build` instead of `build_and_start`, dial9 creates the trace file, seals
segments, and reports `TelemetryGuard::is_enabled() == true` — it simply
records no events. Only the segment's byte count tells the two apart.
Measured on this workload: 5640 bytes when recording, 310 bytes (a bare
segment header) when not. The example asserts >= 2048 bytes, and was verified
to fail with the `build` call restored.
Also correct the comment on the `is_enabled` check in `finish_traced_runtime`.
It claimed to catch "recording silently off"; it does not. It only rejects the
inert guard a lenient config yields after a build failure. Recording is
guaranteed by `build_and_start`, not by that check.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(rustfs): accept Unsupported runtime telemetry capability
A binary built without the `dial9` feature now reports the runtime-telemetry
capability as `Unsupported` rather than `Disabled`. The distinction matters to
operators: `Disabled` implies the capability can be switched on by setting an
environment variable, which is not true here — telemetry needs a rebuild.
Widen the assertion and pin the new semantics: when `dial9::is_supported()` is
false, the state must be exactly `Unsupported`.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs): drop the dial9-s3 feature, its TLS stack is vulnerable
CI's Dependency Review and `cargo deny` both reject the branch: dial9's
`worker-s3` feature depends on aws-sdk-s3-transfer-manager 0.1.3, which pins
aws-smithy-http-client onto hyper-rustls 0.24 and rustls-webpki 0.101.7. That
webpki carries RUSTSEC-2026-0098, -0099 and -0104.
0.1.3 is the latest release of the transfer manager, and 1.2.0 the latest of the
smithy client, so there is nothing to upgrade to. Cargo's feature unification can
add features but cannot drop a transitive dependency, so it cannot be worked
around from here either — the rest of the workspace already resolves to the safe
rustls-webpki 0.103 / hyper-rustls 0.27.
Remove the `dial9-s3` feature and the `with_s3_uploader` wiring. The two S3
environment variables stay parsed and warned about, now naming the real reason
rather than a missing build feature. Trace segments are collected from the output
directory instead. Tracked as D9-14 in rustfs/backlog#1157.
With this, Cargo.lock is byte-identical to main: the PR no longer touches the
dependency graph at all.
Also correct the `dial9-taskdump` documentation. It claimed the feature "compiles
to a no-op elsewhere"; in fact `tokio/taskdump` raises a `compile_error!` on any
target other than linux/{aarch64,x86,x86_64}. Verified by trying to build it on
macOS, which is how the claim was found to be wrong.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
fix(ecstore): io_uring reads must reclaim the page cache like StdBackend (backlog#1145)
`StdBackend::pread_bytes` calls `fadvise(DONTNEED)` over the range it just
read whenever `should_reclaim_file_cache_after_read(length)` holds —
`RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE` (on by default) above a 4 MiB
threshold. Large object reads are usually cold, and leaving them resident
evicts everything else, so this is a deliberate policy rather than a side
effect of how StdBackend happens to read.
`pread_uring` and `pread_uring_direct` never did it. Enabling io_uring on a
disk therefore turned the policy off silently: shard reads at or above the
threshold stayed resident and grew the page cache without bound. This is the
same class of mistake as the O_DIRECT loss fixed earlier — copying the read
itself while dropping the policy attached to it.
It also badly distorts any benchmark. An end-to-end warp GET A/B on a
16-core host (4 MiB objects, conc 64) showed io_uring at 8782 MiB/s against
StdBackend's 116 MiB/s. That is not a 76x speedup: with io_uring the run
issued *zero* device reads and grew the page cache, while StdBackend read
2950 MiB from the device and shrank it. The two legs were not running the
same policy. (Disabling the mmap read path changed nothing — 1.01x — so the
mmap-vs-pread difference was not the cause either.)
Both io_uring read paths now reclaim the range they read, with the same gate,
the same range, and the same error handling as StdBackend. O_DIRECT should
leave nothing resident anyway, but a filesystem that quietly buffered the read
still has to honour the policy, so `pread_uring_direct` reclaims too.
The test measures the policy, not the call: it reads an 8 MiB file through
each backend and asks `mincore(2)` how much stayed resident. Crucially it also
asserts the reclaim-disabled case leaves the range resident — without that
gate, a backend that never reclaimed would pass silently. Verified: with the
fix removed, the test fails with "uring: reclaim is on ... 2048/2048 pages
remain"; with the fix it passes.
Verified on a real Linux host (16-core, real io_uring): clippy --tests
-D warnings clean; disk::local tests 143 passed, 0 failed.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(object-data-cache): bound cache config and make memory container-aware
Harden the object data cache configuration path so a single bad input
degrades to the disabled adapter instead of OOM-killing pods, panicking
at boot, or silently mis-sizing the cache (backlog#1110/1113/1114/1115/
1127/1140/1130).
- ODC-05: resolve capacity and the fill gate from the effective
(container-aware) memory. Prefer sysinfo cgroup_limits() and use
min(host, cgroup) as the total plus cgroup-derived availability, falling
back to host values when no cgroup limit constrains. Log the resolved
capacity and its basis once at startup.
- ODC-08: cap ttl/time_to_idle at 30 days in validate() so an unbounded
Duration can no longer trip moka's ~1000-year builder assertion.
- ODC-09: drop the max_entry_bytes floor from the derived-capacity clamp so
MAX_ENTRY_BYTES can no longer inflate total capacity above the safety
clamp; reject an entry larger than the resolved capacity instead.
- ODC-10: require an explicit max_bytes to clear max_entry_bytes plus the
weigher overhead so a fillable-but-unretainable cache is rejected.
- ODC-22: reject max_entry_bytes at/above the u32 weigher boundary.
- ODC-35: warn (not reject) when time_to_idle exceeds ttl and document the
min(ttl, time_to_idle) expiry interaction.
- ODC-25: give numeric env overrides two-valued semantics via a new
rustfs_utils::get_env_parse_outcome; a malformed value now disables the
whole cache with one aggregated warning instead of silently keeping
defaults.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(object-data-cache): require identity_keys_max of at least 2
The identity index admits a new key by evicting the oldest one, so a
budget of 1 evicts the previous key on every fill and can never hold two
live keys of one object at once — a versioned bucket alternating two
versions then hits a permanent 0% hit rate.
Reject the degenerate value at config validation time, where the adapter
turns it into a disabled cache with a warning, since the bounded-eviction
policy cannot rescue it at runtime.
Handed off from the identity-index batch (backlog#1128), which changed the
overflow policy but could not touch validate().
Refs: backlog#1115, backlog#1128
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(object-data-cache): let a zero free-memory floor opt out of the gate
Making the memory gate container-aware turned it live in CI: the runners
are Kubernetes pods, so the gate now reads the pod's cgroup free memory,
finds it below the 20% floor, and refuses the fill. Tests that assert a
fill succeeds then failed on CI while passing on a developer host, which
has no cgroup and falls back to host memory.
The gate is behaving correctly — the tests were the ones depending on a
live memory reading. Treat min_free_memory_percent = 0 as a deliberate
opt-out rather than an invalid value: allows_fill returns early before it
touches any snapshot, so admission becomes independent of where the suite
runs. Operators gain the same escape hatch.
Every test that requires a fill to succeed now sets the floor to 0. The
tests that exercise the gate keep it enabled via memory_gated_config, so
the ODC-05 coverage they provide is preserved rather than short-circuited.
Refs: backlog#1110
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(object-data-cache): opt the usecase fill tests out of the memory gate
The previous commit exempted the fill-dependent tests it could find, but
missed the six adapters built inside object_usecase.rs. Fixing the first
batch moved nextest's fail-fast point forward and CI surfaced them:
build_get_object_body_with_cache_materializes_once_and_hits_later and
..._uses_cached_body_without_reader_preread both assert a fill lands, and
both ran with the default 20% free-memory floor.
Set the floor to 0 on all six, matching the sibling test modules. Verified
by pinning the gate's snapshot to 0% available — harsher than any CI pod —
and confirming these tests still pass, which shows the exemption path never
reads memory at all.
An exhaustive sweep over every fill-enabled ObjectDataCacheConfig in the
tree now shows no remaining site: the only unexempted ones are
fill_enabled()'s matches! arm and two adapter tests that assert the adapter
is disabled and therefore never fill.
Refs: backlog#1110
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* feat(ecstore): run one io_uring ring per shard on each disk (backlog#1145)
A buffered read that hits the page cache completes inline inside
`io_uring_enter`, so the thread driving a ring performs that read's
memcpy. One ring per disk therefore capped cache-hit reads at a single
core's memory bandwidth: measured on a 16-core host, one driver thread sat
pinned at 100% CPU while throughput stayed flat at ~5 GB/s regardless of
read size, against 50 GB/s for the blocking-pool baseline.
rustfs/uring#6 taught the driver to hold N independent rings, each with its
own thread, pending table, backpressure semaphore, and eventfd. Wire it up:
`UringBackend::try_new` now calls `probe_and_start_sharded`, and
`RUSTFS_IO_URING_SHARDS` selects the count per disk.
The default is a quarter of the available parallelism clamped to `1..=4`,
because the cost is `disks × shards` driver threads (each normally blocked
in `poll(2)`). Any override is clamped to `1..=16`, so a mistyped value can
neither disable the driver (0) nor spawn threads without bound; an
unparseable value falls back to the default.
Effect (warm page cache, 16-core, rustfs/uring's concurrent_pread_bench):
1 MiB, conc 8: 1 shard 4911 MB/s -> 8 shards 47361 MB/s (9.6x);
the blocking-pool baseline is 50662 MB/s
64 KiB, conc 32: StdBackend 153678 IOPS, p999 3030 us
8 shards 345402 IOPS, p999 897 us
64 KiB, conc 128: StdBackend 135155 IOPS, p999 10716 us
8 shards 389047 IOPS, p999 4092 us
Sharding removes the throughput deficit *and* keeps io_uring's tail-latency
advantage, rather than trading one for the other.
Unchanged: io_uring read stays gray-off by default
(`RUSTFS_IO_URING_READ_ENABLE`), reads are byte-for-byte identical to
StdBackend, the per-disk degradation latches and probe cache (backlog#1101)
and the O_DIRECT tiered fallback (backlog#1102) all still apply. Rings stay
per-disk, so a stalled disk cannot starve another disk's rings
(backlog#1055). Bumps the rustfs-uring pin to the merged #6 commit.
Verified on a real Linux host (16-core, real io_uring): cargo clippy
--tests -D warnings clean; disk::local tests 132 passed, 0 failed —
including the existing io_uring and O_DIRECT cases now running on the
sharded driver, plus a new test covering the shard-count default, override,
and clamping.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(ecstore): cache part-file descriptors for io_uring reads (backlog#1145)
`pread_uring` opened the file on the blocking pool for every read, so each
read paid a `spawn_blocking` round trip — the very thread hop io_uring
exists to avoid. Sharding the driver (backlog#1145) removed the previous
ceiling and left this as the binding cost. Measured on a 16-core host with
a 4-shard driver, warm page cache:
64 KiB, conc 8: 143942 -> 263054 IOPS (+83%), p999 240 -> 65 us
64 KiB, conc 32: 150128 -> 204876 IOPS (+36%), p999 2508 -> 871 us
64 KiB, conc 128: 129172 -> 361287 IOPS (+180%), p999 15329 -> 3046 us
1 MiB, conc 32: 33875 -> 42301 IOPS (+25%)
At 64 KiB / conc 128 the open is what masked io_uring entirely: with it,
io_uring beat StdBackend by 3.5%; without it, by 189%.
Add a bounded per-disk descriptor cache used by the io_uring read path.
A hit takes no `open` and no `spawn_blocking`, so the read never leaves the
runtime worker.
Why caching a part-file descriptor is safe:
* only `<object>/<data_dir>/part.N` reaches this backend's `pread_bytes`;
`xl.meta` — the one path replaced in place — is read through `read_all`
/ `read_metadata` and never gets here;
* part files are never rewritten in place. A replacement is always
write-new-tmp then `rename`, which swaps the inode, so a cached
descriptor can never observe a torn shard.
Why invalidation is nevertheless REQUIRED: heal reuses the existing
version's `data_dir` and renames a rebuilt shard onto the SAME part path.
A cached descriptor would keep serving the pre-heal (corrupt) inode,
defeating the heal and eroding read quorum. `delete` likewise unlinks a
part that a cached descriptor would keep readable. So `rename_data`,
`rename_file`, and `delete` all call the new
`LocalIoBackend::invalidate_cached_fds` after they mutate, and a 5s TTL
bounds the blast radius should a future mutation path forget to.
Two preamble checks the miss path runs are not silently lost on a hit:
* bounds — the driver only short-reads at EOF (it resubmits otherwise),
so `bytes.len() != length` is exactly the old `meta.len() < end_offset`
check, and now yields the same `FileCorrupt`;
* volume access — skipped while an entry is live. An unreachable disk
keeps serving already-open descriptors for at most the TTL, after which
the re-open re-runs the check. Disk health is tracked independently of
this per-read probe.
Scope: buffered io_uring reads only. The O_DIRECT path keeps opening per
read (its reads are >= 4 MiB, so the open is a small fraction), and
StdBackend is untouched — it must take the blocking hop for the pread
regardless, so caching would buy it only 2-6% while carrying the same
invalidation risk. `RUSTFS_IO_URING_FD_CACHE=false` restores open-per-read.
Verified on a real Linux host (16-core, real io_uring): clippy --tests
-D warnings clean; disk::local tests 135 passed, 0 failed. The new
heal-staleness test first asserts a read still returns the PRE-heal bytes —
proving the cache is live and the hazard real — then that invalidation makes
the healed shard visible. A second test drives `rename_file` and `delete`
through `LocalDisk` to prove those paths actually invalidate, and a unit
test pins prefix invalidation to component boundaries (`a/b` must not drop
`a/bc`).
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
fix(object-data-cache): close identity-index races in fill/invalidate (backlog#1117/#1118/#1125/#1128/#1136)
The object body cache's fill-vs-invalidation contract (index.insert before
cache.insert, then re-check index membership and undo the fill on mismatch)
had several races where the identity index and the cache could drift, either
poisoning the race metric, breaking invalidation silently, or letting a
principal evict a hot body on demand.
ODC-12 (backlog#1117) — generation-aware index removal.
The eviction listener removed (identity, key) by key equality with no way to
tell an evicted old entry from a freshly refilled one under the same key.
Verified against moka 0.12.15: an insert over a TTL/TTI-expired but not yet
purged entry runs do_insert_with_hash inline during insert().await, and
do_post_update_steps awaits notify_upsert INLINE with cause Expired
(was_evicted() == true), so the listener deleted the index key the refill just
registered and the post-insert recheck then self-destructed the fresh entry
(SkippedInvalidationRace). A deferred Size notification for an old generation
had the same effect on a cleanly refilled key.
Chose Option A (generation token) over Option B (re-insert after
cache.contains_key): B removes the key first and only reconciles afterward,
which reopens a narrow window where a concurrent invalidate_object misses the
key while it is transiently absent from the index. A never removes a key whose
generation does not match, so the decision is atomic under the shard lock. The
token is the cache entry's Arc pointer captured at fill time (readable from the
value moka hands the listener), so no change to the entry type is needed.
ODC-13 (backlog#1118) — cancellation-safe insert/recheck/undo.
The recheck+undo sat after two awaits inside the S3 GET future; a client
disconnect could cancel the fill at the recheck await after cache.insert made
the entry visible, stranding a stale body with no index entry. The
register/insert/recheck/undo sequence now runs in a spawned task whose
JoinHandle the leader awaits: cancelling the await detaches the task, which
still finishes the recheck and undo. The dropped leader unblocks waiters as
before (SkippedSingleflightClosed), so no waiter is stranded.
ODC-20 (backlog#1125) — do not prune in-flight sibling keys.
prune_missing could remove another in-flight fill's index key (a different key
of the same identity that had registered but not yet published its cache
entry), making that fill's recheck fail and delete its own valid entry. Added a
read-only ObjectDataCacheSingleflight::has_inflight and consult it in the prune
predicate alongside cache.contains_key.
ODC-23 (backlog#1128) — bounded eviction instead of clear-and-reject.
Exceeding identity_keys_max drained the whole key set and rejected the incoming
key, so a principal able to GET identity_keys_max+1 distinct versions could
evict an object's hot body on demand, and identity_keys_max=1 on a versioned
bucket meant a permanent 0% hit rate. The key set now evicts only the oldest
key(s) (the Vec preserves insertion order) and inserts the new key. The insert
result carries evicted_keys so the backend removes exactly those from the cache
and still caches the newest body. The Overflow variant and the now-unused
SkippedIdentityOverflow construction are gone.
NOTE: the audit also asks for identity_keys_max >= 2 validation; that lives in
config.rs (out of scope here) and is deferred to the config batch.
ODC-31 (backlog#1136) — deterministic race coverage.
Added a #[cfg(test)] fill barrier between the index insert and the cache insert
so the fill-vs-invalidation recheck can be driven deterministically, plus a
companion test asserting the identity-budget eviction drops the evicted keys'
cache entries.
New tests:
- index: key_set_bounded_eviction_evicts_oldest_and_inserts_new,
key_set_removes_only_matching_generation_token,
key_set_duplicate_refreshes_generation_token
- starshard_index: identity_index_bounded_eviction_evicts_oldest,
identity_index_stale_eviction_preserves_refreshed_key,
identity_index_matching_eviction_removes_key
- moka_backend: moka_backend_refill_after_expiry_without_maintenance_inserts,
moka_backend_concurrent_fills_same_identity_both_insert,
moka_backend_identity_budget_evicts_oldest_and_caches_newest,
moka_backend_fill_loses_race_to_invalidation,
moka_backend_cancelled_fill_still_undoes_lost_race
cargo fmt --all, cargo check/clippy/test -p rustfs-object-data-cache all pass
(44 tests). Verified the refill-after-expiry and concurrent-fill tests fail
without their respective fixes.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(cache): gate GET body-cache hook to preserve ReadPlan output
The ecstore GET body-cache hook serves cached full-object plaintext
directly, bypassing ReadPlan/ReadTransform. That is only sound when the
normal read path returns that same plaintext byte-for-byte. Two probe
conditions were missing, plus two usecase-layer planner gaps.
ODC-01 (backlog#1108): raw/data-movement reads. ReadPlan::build returns
the STORED representation for raw_data_movement_read (e.g. compressed
bytes, length = oi.size), but the cache holds the post-decompression
body. Decommission (raw_data_movement_read: true) would receive
decompressed plaintext where raw compressed bytes are required, silently
corrupting the destination pool.
ODC-02 (backlog#1109): compressed objects. ReadTransform::Compressed
rewrites object_info.size to the decompressed length; on a hook hit
object_info is returned unchanged, so object_info.size is the compressed
size while the stream carries the decompressed body. UploadPartCopy then
uses src_info.size as the copy length and truncates the part.
Fix: gate the hook probe with should_probe_body_cache_hook, refusing
raw_data_movement_read, data_movement, and compressed objects, mirroring
the conditions get_small_object_direct_memory_decision already applies.
ODC-33 (backlog#1138): build_get_object_body_cache_plan lacked the
is_remote() exclusion the ecstore hook enforces; add it so transitioned
(remote-tier) objects are excluded uniformly.
ODC-C1 (backlog#1142): zero-length bodies save no I/O (ecstore returns an
empty body before the hook probe) yet the planner admitted them; change
the guard to response_content_length <= 0 so they plan Skip, mirroring
should_buffer_get_object_in_memory_with_threshold.
Tests: body_cache_hook_gate_tests (4) cover plain-probe plus
raw/data-movement/compressed skips; planner gains
plan_skips_remote_transitioned_objects and plan_skips_zero_length_objects.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(cache): allow compressed bodies via a fail-closed read allow-list
The body-cache hook probe was gated by a deny-list that refused compressed
objects outright, which cost the cache every compressed body — a growing
share of stored data. Replace it with an allow-list that returns the exact
plaintext length a hit may serve, or None.
full_object_plaintext_len() answers a single question: would the normal
ReadPlan produce this object's complete plaintext, and under which size?
Compressed objects now qualify, and the hit site publishes the returned
length as object_info.size, reproducing the contract ReadTransform::
Compressed establishes. A hit whose body length disagrees is refused and
falls through to the erasure read.
This also closes a gate the deny-list only covered by accident: a restore
read forces ReadPlan down the Plain branch, so a compressed object yields
STORED bytes under its compressed size. Refusing compressed objects hid
that; admitting them exposes it, so restore reads are refused explicitly.
Being fail-closed, a newly added ReadPlan branch bypasses the cache by
default rather than silently serving the wrong representation — the
structural defect behind both backlog#1108 and backlog#1109.
Refs: backlog#1108, backlog#1109
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
A buffered read that hits the page cache completes inline inside
`io_uring_enter`, so the thread driving a ring performs that read's
memcpy. One ring per disk therefore capped cache-hit reads at a single
core's memory bandwidth: measured on a 16-core host, one driver thread sat
pinned at 100% CPU while throughput stayed flat at ~5 GB/s regardless of
read size, against 50 GB/s for the blocking-pool baseline.
rustfs/uring#6 taught the driver to hold N independent rings, each with its
own thread, pending table, backpressure semaphore, and eventfd. Wire it up:
`UringBackend::try_new` now calls `probe_and_start_sharded`, and
`RUSTFS_IO_URING_SHARDS` selects the count per disk.
The default is a quarter of the available parallelism clamped to `1..=4`,
because the cost is `disks × shards` driver threads (each normally blocked
in `poll(2)`). Any override is clamped to `1..=16`, so a mistyped value can
neither disable the driver (0) nor spawn threads without bound; an
unparseable value falls back to the default.
Effect (warm page cache, 16-core, rustfs/uring's concurrent_pread_bench):
1 MiB, conc 8: 1 shard 4911 MB/s -> 8 shards 47361 MB/s (9.6x);
the blocking-pool baseline is 50662 MB/s
64 KiB, conc 32: StdBackend 153678 IOPS, p999 3030 us
8 shards 345402 IOPS, p999 897 us
64 KiB, conc 128: StdBackend 135155 IOPS, p999 10716 us
8 shards 389047 IOPS, p999 4092 us
Sharding removes the throughput deficit *and* keeps io_uring's tail-latency
advantage, rather than trading one for the other.
Unchanged: io_uring read stays gray-off by default
(`RUSTFS_IO_URING_READ_ENABLE`), reads are byte-for-byte identical to
StdBackend, the per-disk degradation latches and probe cache (backlog#1101)
and the O_DIRECT tiered fallback (backlog#1102) all still apply. Rings stay
per-disk, so a stalled disk cannot starve another disk's rings
(backlog#1055). Bumps the rustfs-uring pin to the merged #6 commit.
Verified on a real Linux host (16-core, real io_uring): cargo clippy
--tests -D warnings clean; disk::local tests 132 passed, 0 failed —
including the existing io_uring and O_DIRECT cases now running on the
sharded driver, plus a new test covering the shard-count default, override,
and clamping.
Co-authored-by: heihutu <heihutu@gmail.com>
The listing path no longer has a wrapper-level total timeout, so any
filesystem call the walk makes without a stall bound would have no liveness
bound at all. The first pass covered list_dir and read_metadata but left four
reads unbounded: the volume access() probe and fs::metadata() in walk_dir, the
xl.meta read in its dir-object branch, and is_empty_dir() in scan_dir.
Split the helper so reads that do not yield a Result go through the same
budget, and route the four remaining calls through it. A hung drive now fails
those reads with DiskError::Timeout instead of parking the walk until the
merge consumer's own stall timeout aborts it.
Co-authored-by: heihutu <heihutu@gmail.com>
fix(ecstore): keep O_DIRECT for eligible reads via native io_uring (rustfs/backlog#1102)
Wire ecstore's io_uring read backend to rustfs-uring's native aligned
O_DIRECT read (`read_at_direct`, merged in rustfs/uring#3), so an
O_DIRECT-eligible read keeps BOTH io_uring's async submission AND
O_DIRECT's page-cache bypass instead of trading one for the other.
`UringBackend::pread_bytes` now selects the read shape by a tiered,
per-disk-latched ladder — never a blanket downgrade:
1. io_uring latched off for this disk (backlog#1101) -> StdBackend.
2. O_DIRECT-eligible + native path usable -> pread_uring_direct: open
the file O_DIRECT, probe the device alignment once (cached), and
read via read_at_direct. Best path: async + no cache pollution.
3. O_DIRECT-eligible but the fs refused O_DIRECT earlier (latched)
-> StdBackend aligned path (which itself degrades to buffered).
4. non-O_DIRECT read -> buffered pread_uring.
Latching keeps a failure from being re-attempted per-read: an fs that
refuses O_DIRECT (EINVAL/EOPNOTSUPP on open) latches the native direct
path off for that disk; a restriction-class errno from the read latches
io_uring off wholesale (backlog#1101). Any other error falls back for
that one read without masking a genuine data problem as a permanent
downgrade. The alignment comes from the existing probe_direct_io_align
(statx STATX_DIOALIGN, default 4096) and is cached in a per-disk
OnceLock so the probe runs at most once.
This replaces the interim guard that routed every O_DIRECT-eligible read
to StdBackend (which disabled io_uring on the hottest shard reads); the
native path is now the default and StdBackend is only a fallback tier.
Bumps the rustfs-uring pin to the merged #3 commit.
Verified on a real Linux host (16-core, real io_uring + O_DIRECT):
cargo check --tests, clippy -D warnings, and disk::local tests all pass
(128 passed, 0 failed), including uring_preserves_o_direct_for_eligible_reads
across unaligned ranges.
Co-authored-by: heihutu <heihutu@gmail.com>
Foreground S3 listings wrapped the entire walk_dir stream in a single
wall-clock timeout (RUSTFS_DRIVE_WALKDIR_TIMEOUT_SECS, default 5s). That
budget measured how much data the walk had to produce rather than whether
the drive was still answering, so a healthy but large prefix on slow media
returned 500 InternalError / "Io error: timeout" to the client. The timer
also kept running while the walk was blocked writing to a slow consumer,
charging merge-side backpressure to the producer.
WalkDirOptions::stall_timeout_ms already carried the right semantics but was
only honored by the remote-disk RPC walk; LocalDisk::walk_dir ignored it
entirely. A single-drive deployment therefore had no way to distinguish a
hung drive from a big directory.
Teach LocalDisk::walk_dir to bound each individual drive read with the stall
budget, defaulting it from RUSTFS_DRIVE_WALKDIR_STALL_TIMEOUT_SECS when the
caller does not pin one, and let the foreground listing path skip the
wrapper-level total timeout. A walk that keeps making progress now runs to
completion; a drive that stops answering still fails with DiskError::Timeout.
Time spent blocked on the consumer stays outside the budget.
Heal and rebalance walks already skipped the total timeout and previously ran
unbounded on local drives. Give them an explicit, generous 60s stall budget so
this change does not tighten them from "no bound" to the 5s default.
Fixes#4644
Refs #2999, #3001
Co-authored-by: heihutu <heihutu@gmail.com>
fix(ecstore): keep O_DIRECT for eligible reads when io_uring is enabled (rustfs/backlog#1102)
UringBackend::pread_uring opens the file buffered, so with io_uring enabled an
O_DIRECT-eligible read (RUSTFS_OBJECT_DIRECT_IO_READ_ENABLE + length >= the
threshold) silently lost O_DIRECT and polluted the page cache — a behavior
change the moment io_uring was turned on for a disk that uses O_DIRECT. The
io_uring backend is gray-off by default, so no deployment is affected, but the
gap had to close before it can be enabled anywhere O_DIRECT is in use.
Route O_DIRECT-eligible reads back through StdBackend's aligned path (which
itself falls back to buffered when the filesystem rejects O_DIRECT). A native
aligned O_DIRECT read in the driver remains part of rustfs/backlog#1102.
Adds uring_preserves_o_direct_for_eligible_reads: with BOTH flags on and the
threshold at 1, unaligned ranges still return exactly the requested bytes.
Co-authored-by: heihutu <heihutu@gmail.com>
backlog#1052 S7 — the final piece: full bucket-namespace isolation
between embedded servers in one process.
Server B's requests resolved B's own ECStore (per-server dispatch landed
earlier), but the store's bucket operations still went through ambient
process facades, so both servers effectively operated on the FIRST
server's disks and metadata:
- LocalPeerS3Client::local_disks_for_pools() called all_local_disk()
(the ambient disk registry = the first published store's context), so
list/make/delete/heal bucket scanned and wrote the wrong volumes.
- BucketMetadata::save() persisted through the ambient object handle, so
a second server's bucket metadata landed in the first server's
.rustfs.sys; set/remove/get/created_at all used the ambient metadata
system.
Now the whole chain is bound to the owning store's InstanceContext:
- S3PeerSys/LocalPeerS3Client gain *_with_instance_ctx constructors and
operate on that context's registered disks; ECStore::new (and the test
store builder) pass the store's context. The legacy constructors keep
the bootstrap default.
- BucketMetadata::save_with_store persists through an explicit store;
BucketMetadataSys::persist_and_set uses the system's own api handle.
- metadata_sys gains instance-scoped variants (get_in / created_at_in /
set_bucket_metadata_in / remove_bucket_metadata_in) that resolve the
context's metadata system and fall back to the ambient default before
the instance cell is initialized (early startup, unchanged behavior).
- The store's bucket handlers (make/get_info/list/delete + the
table-bucket delete guard and emptiness check) use the per-context
variants and this instance's disks.
Acceptance (e2e): two embedded servers with different credentials are now
isolated end to end — each authenticates only its own key, neither sees
the other's buckets or objects, and both data planes stay intact. The
embedded module doc drops the shared-IAM caveat.
579 ecstore bucket/metadata/peer regressions plus the embedded basic and
deferred-IAM e2e stay green.
test(ecstore): cover the io_uring per-disk probe cache hit (rustfs/backlog#1101)
Follow-up to #4635. The per-disk negative probe cache was implemented but had
no dedicated test. Add uring_probe_cache_skips_known_unsupported_disk: a root
recorded in URING_UNSUPPORTED_DISKS makes try_new return None without a fresh
probe. Completes the #1101 acceptance ("cache hit does not re-probe").
Co-authored-by: heihutu <heihutu@gmail.com>
The startup topology convergence auto-detection (added in #4631) checked
Kubernetes before endpoint style, so a local path deployment (single or
multi-drive, non-distributed) running inside Kubernetes was classified as
orchestrated with an effectively unbounded wait window. Path-style
endpoints have no hostnames to resolve, so the wait never actually
triggers and runtime behavior is unchanged, but the startup log then
advertised mode=orchestrated / wait_timeout=MAX for a purely local
deployment, which is misleading and contradicts the documented policy
(local path endpoints -> fail-fast).
Order the auto-detection by endpoint style first: only distributed URL
endpoints, which resolve hostnames, pick orchestrated (Kubernetes) or
bounded (otherwise); local path endpoints stay fail-fast regardless of
the platform. Update the doc comment to match and add a policy case
locking Kubernetes + local path to fail-fast.
Co-authored-by: heihutu <heihutu@gmail.com>
feat(ecstore): io_uring runtime degradation latch + per-disk probe cache (rustfs/backlog#1101)
Follow-up to #1104. Add the runtime half of the io_uring degradation contract:
- Runtime latch: UringBackend gains an `active` flag (mirrors
DirectIoReadState::supported). A read that returns a restriction-class errno
(EPERM/EACCES/ENOSYS/EOPNOTSUPP) — io_uring became unusable on this disk —
trips the latch, and every further read goes straight to StdBackend with no
more per-read io_uring attempts. Data errors (EIO), missing files, and
parameter errors do NOT latch, since StdBackend would hit them too.
- Per-disk probe cache: a process-wide negative cache of disks whose probe
failed, so try_new skips re-probing (creating a ring + driver thread) on
reconnect.
Tests: the errno classifier (restriction-only) and a latched-off read that
still returns correct bytes via StdBackend; the #1104 differential test still
passes under real io_uring.
Co-authored-by: heihutu <heihutu@gmail.com>
The OnceLock introduced in 5fbd49a80 for bucket_metadata_sys panics when
multiple test modules initialize their own ECStore against the shared
bootstrap context. Switch to Mutex<Option> so test fixtures can
reinitialize safely.
Extract shared gating test environment to gating_test_env.rs to avoid
duplicate ECStore setup across delete_objects_stat_gating_test and
put_prelookup_gating_test.
Fixes 8 test failures:
- capacity_dirty_scope_test (2 tests)
- delete_objects_stat_gating_test (3 tests)
- put_prelookup_gating_test (3 tests)
* feat(embedded): allow multiple embedded servers to coexist in one process
backlog#1052 S5: turn the embedded startup guard into a sequential lock
and make every write-once shared cell tolerate a second embedded start.
Two RustFSServers on different ports and volumes now start, run, and
shut down independently in the same process.
- EMBEDDED_SERVER_STARTED is released once each startup hands off; a
second startup that runs after the first is no longer rejected.
- The second embedded server constructs its own InstanceContext instead
of adopting the process bootstrap one, so region/endpoints/deployment
id land on that context (the first server keeps adopting bootstrap to
keep single-instance ambient facades unchanged).
- Startup-time tolerant paths:
- action_credentials publish treats AlreadyInitialized as success
(per-server ActionCredentialHandle already holds the real creds).
- GLOBAL_RUSTFS_PORT warns instead of panicking on a second set.
- Observability install returns Ok when the process subscriber is
already set — the second server reuses it.
- New acceptance test proves two servers start, respond on their own
ports, and one can be shut down without disturbing the other; the
survivor keeps serving S3 requests. IAM and root-credential lookup
still share a process domain (a second server whose creds differ from
the first will fail signature validation), tracked as a follow-up.
- Embedded doc rewritten: 'Limitations' → 'Multi-instance status'; the
AlreadyStarted error is now scoped to concurrent startups only.
The remaining work in #1052 is the auth path per-server dispatch and the
matching data-plane routing so two servers with different credentials
serve independent buckets end-to-end.
* feat(app): per-server auth and application context for multiple embedded servers (#4633)
backlog#1052 S6: each embedded server now authenticates against — and
its request path resolves — its OWN application context, so two servers
with different root credentials each accept their own access key and
reject the other's.
- AppContext is per-server: ensure_startup_after_iam constructs a fresh
context around this server's store + IAM + KMS and installs it into the
server's own ServerContextSlot, then publishes it as the process
default first-writer-wins (publish_global_app_context) for legacy
ambient readers. The old 'reuse the global if present' path is gone.
- FS::check (the S3 data-plane access gate) resolves auth against
self.server_ctx's context: check_key_valid gains a _with_context
variant that takes the root credentials and IAM system from an explicit
context (None = ambient, unchanged for all 140+ existing callers). The
region and the server context slot are published into the request
extensions for downstream handlers.
- Each embedded server seeds its own root credentials into its context
(ActionCredentialHandle.publish) at startup, so credential validation
no longer falls back to the first server's process-global identity.
- The bucket/object/multipart use-cases resolve their store from the
server's context (bucket_usecase_for/object_usecase_for/... take &FS).
New acceptance test: two servers with distinct credentials each
authenticate with their own key and reject the other's.
KNOWN FOLLOW-UP: full bucket-namespace isolation still requires threading
the instance context through the lower ecstore data plane (peer_sys /
disk registry / bucket-metadata reads still resolve via the process
GLOBAL_OBJECT_API), so the two servers do not yet present independent
bucket listings even though each holds its own store. That deeper pass —
a continuation of the #939 object-graph ctx threading — is the remaining
work on #1052.
Stacked on the S5 guard change.
Multi-node startups (Kubernetes StatefulSets, bare-metal, VMs) can hit
transient DNS/local-host resolution failures while peers and headless
service records are still coming up. Previously endpoint resolution
retried DNS for a fixed 90s window and then returned an error, which
propagated out of run() and exited the process; in Kubernetes this drove
repeated kubelet restarts even though the cluster eventually converged.
Introduce a startup topology convergence policy with three modes:
- orchestrated: wait effectively indefinitely for recoverable
DNS/topology errors so the process stays Running (readiness stays
false) instead of exiting; defer the DNS-IP cross-port validation
because headless-service records can still be flapping.
- bounded: wait a finite window (default 3m) then fail with an
action-oriented error listing host/stage/elapsed and remediation hints.
- fail-fast: fail on the first non-transient resolution error.
Mode is auto-detected (Kubernetes -> orchestrated, distributed URL
endpoints -> bounded, local path endpoints -> fail-fast) and can be
overridden via RUSTFS_STARTUP_TOPOLOGY_WAIT_MODE,
RUSTFS_STARTUP_TOPOLOGY_WAIT_TIMEOUT and
RUSTFS_STARTUP_TOPOLOGY_RETRY_MAX_DELAY.
Also normalize divergent local/remote verdicts for endpoints that share
a host:port, throttle retry warnings, and keep the DNS-independent local
same-path checks in every mode. Configuration error handling is
unchanged: malformed volumes, mixed styles/schemes, duplicate or root
paths, and non-transient errors still fail fast.
Read the topology env vars through rustfs_utils::get_env_opt_str for
consistent alias handling, and simplify the log_once poisoned-lock branch
to unwrap_or_else(PoisonError::into_inner).
Co-authored-by: heihutu <heihutu@gmail.com>
feat(ecstore): add runtime-probed io_uring read backend, gray-off by default (rustfs/backlog#1104)
Wire the standalone rustfs-uring crate (https://github.com/rustfs/uring) into
LocalIoBackend as an opt-in read backend. When RUSTFS_IO_URING_READ_ENABLE is
set AND the per-disk io_uring probe succeeds, positioned reads go through the
cancel-safe UringDriver; otherwise — default, or on a restricted host, or on
any per-read driver error — reads fall back to StdBackend byte-for-byte, so the
default build is unchanged.
- Cargo.toml: rustfs-uring as a Linux-only git dependency (pinned rev). The
guard scripts/check_no_tokio_io_uring.sh allows an explicit io-uring
integration; only the tokio "io-uring" runtime feature is banned.
- UringBackend mirrors StdBackend::pread_bytes's resolution/access/bounds
preamble and only swaps the raw byte read; the other three trait methods
delegate to the inner StdBackend.
- Backend selection at LocalDisk construction via build_local_io_backend.
- Differential test uring_backend_reads_match_std: with the flag set, every
positioned read returns byte-identical data (driver-served when io_uring is
available, StdBackend fallback otherwise).
Verified: cargo check -p rustfs-ecstore on Linux; the differential test passes
under real io_uring (seccomp=unconfined). The runtime errno degradation latch,
per-disk probe cache, and productionization are tracked in
rustfs/backlog#1101/#1102.
Co-authored-by: heihutu <heihutu@gmail.com>
backlog#1052 S3, fourth slice. The AppContext's IamHandle already owned an
Arc<IamSys>, but the value it held was read back from the process
singleton after init — so a future second server's context would silently
bind to the first server's IAM domain (credentials, policies, users all
belong to a specific store's .rustfs.sys).
- rustfs_iam gains build_iam_sys(store): construct an IAM system bound to
a store without touching the singleton. init_iam_sys wraps it (build,
publish first-wins, return the handle — previously Result<()>).
- Startup threads the built handle: the inline bootstrap path passes the
Arc it just created into ensure_startup_after_iam, which constructs the
AppContext from the passed value instead of re-resolving the global.
The deferred-recovery path resolves the freshly published global
directly (context-first resolution cannot be used there: the AppContext
does not exist yet — creating it is the finalizer's job; caught by the
embedded deferred-IAM e2e).
- IamHandle::is_ready() reports the held system's readiness instead of
consulting the process singleton; the dead global re-resolver is
removed. token_signing_key stays on the credentials global until S4.
157 iam tests + embedded e2e (basic + deferred recovery) green.
backlog#1052 S3, third slice — the hard blocker for a second embedded
server's service stage. GLOBAL_BUCKET_METADATA_SYS was a process-global
OnceLock whose init ran .set().unwrap(): a second instance's service
startup panicked the whole process. The system it guards is inherently
per-store (it holds the store handle and that store's bucket→metadata
cache), so it now lives on the store's own InstanceContext:
- init_bucket_metadata_sys writes the cell of the store it was handed
(api.ctx) — no signature change — and keeps the double-init fail-fast,
now scoped to the instance (assert on the per-context set). The
dist-erasure check for the refresh loop reads the same context.
- get_global_bucket_metadata_sys / the crate-private getter behind the
~20 get_*_config helpers resolve through the current_ctx() facade —
the published store's context, or bootstrap before that — so every
existing reader keeps its exact single-instance behavior.
- New acceptance test: two real stores in one process each initialize
their own metadata system (the old global cell panicked right there),
plus a shared builder extracted from the store-graph test.
201 bucket/metadata regression tests green.
* refactor(app): give each context's server-config handle its own copy
backlog#1052 S3, first slice: establish the per-context state pattern on
the smallest subsystem. ServerConfigHandle was a unit struct forwarding
both get and set to the process-global GLOBAL_SERVER_CONFIG, so every
AppContext shared one config regardless of which context a caller held.
The handle now keeps its own copy: a set lands on the owning context and
on the process global (ambient readers — the config loader path, the
scanner — keep working), and a get prefers the owned copy, falling back
to the global while the initial load still publishes there. Single
instance: the owned copy and the global are written together, so reads
are unchanged. Two contexts that both published stay isolated even though
the shared global only remembers the last write (covered by a new test).
The global write in set() is the transitional bridge for ambient readers;
it goes away when those readers migrate and multi-instance flips on
(backlog#1052 S5).
* refactor(notify): hand out the notification system as an owned Arc
backlog#1052 S3, second slice. GLOBAL_NOTIFICATION_SYS stored the
NotificationSys by value and every accessor returned
Option<&'static NotificationSys> — a process-lifetime borrow that a
per-server AppContext can never hold. The global now stores an Arc and
the accessor chain (ecstore runtime sources, ECStore::notification_system,
the iam forwarders, the rustfs shims, NotificationSystemInterface and the
AppContext resolvers) hands out owned Arcs instead.
Call sites are unchanged apart from two borrow adjustments in the
rebalance admin handler (pass &Arc where a reference is expected; borrow
with as_ref() so the handle survives to the second use). Behavior is
identical — same single global instance, first init still wins — but the
type now permits a future per-server context to own its notification
system (backlog#1052 S3 follow-up).
Two follow-up fixes to #4607 (both verified real bugs, each with a regression
test):
1. The in-call server_info retry did not re-dial on network errors. A
network-like first failure runs through `finalize_result`, which sets the
client's `offline` gate; the retry then called `evict_connection` and
re-invoked `server_info`, but `get_client` short-circuits on that gate and
returns "temporarily offline" without dialing. Only the async recovery
monitor would clear it. So the retry re-dialed only in the timeout branch and
was a no-op in the transport/half-open branch it was meant to cover. Add
`PeerRestClient::prepare_retry` (evict + clear the offline gate) and use it
before the retry.
2. Synthesized/degraded drive lists went empty on hostname deployments.
`PeerRestClient::host` is an `XHost` that `hosts_sorted` builds via
`XHost::try_from` -> `to_socket_addrs`, so it is the resolved `IP:port`; but
`synthesized_disks`/`peer_disk_health` compared it against
`Endpoint::host_port()`, which is the raw `hostname:port`. On hostname
clusters the compare missed, the drive list came back empty, and
`unknownDisks` stayed 0 — reproducing the "drives vanish from the summary"
regression #4607 fixed (also affected the pre-existing offline path). Compare
through a shared `endpoint_host_matches` helper that canonicalizes the
endpoint side through the same `XHost` resolution.
Tests: `peer_rest_client_prepare_retry_clears_offline_gate`,
`endpoint_host_matches_direct_and_canonicalized` (uses localhost, no external
DNS).
Follow-up to rustfs/rustfs#4607; tracked in rustfs/backlog#1049.
Co-authored-by: heihutu <heihutu@gmail.com>
fix(ecstore): read the persisted SSE-C nonce in the GET decrypt resolver
#4576 moved SSE-C encryption to a random per-encryption nonce persisted
in object metadata and taught the rustfs-layer decrypt resolver to read
it back — but the byte-level GET decryption resolves its material in
crates/ecstore object_api/readers.rs, a duplicated twin that still
recomputed the deterministic legacy nonce. Encrypt with the random
nonce, decrypt with the deterministic one: the first AEAD block fails
and every SSE-C GET aborts its body after the response headers
(IncompleteRead(0 bytes) on clients; all 8 SSE-C cases in the
implemented s3-tests whitelist), which is what has been driving the
"S3 Implemented Tests" CI job into its 60-minute timeout since
2026-07-09.
Resolve the base nonce like the encrypt side: prefer the persisted IV
(x-rustfs-encryption-iv, then the case-insensitive MinIO interop key),
fall back to the deterministic nonce only for legacy objects with no
stored IV or an undecodable value. Full implemented s3-tests whitelist
is back to 451 passed / 0 failed against the fixed binary.
test(e2e): align object-lock overwrite tests with AWS new-version semantics
PR #3179 made put-like writes (PUT, CopyObject, multipart create/complete)
on versioned object-lock buckets create a new version instead of being
blocked by the current version's legal hold or COMPLIANCE retention, which
is the correct AWS behavior. Six e2e assertions in object_lock_test.rs
still encoded the old blocking semantics and, because e2e_test is excluded
from the CI test run, stayed red unnoticed since 2026-06-03.
Verified against a real server built from current main: all six failed at
their "should be blocked" assertions. Rewritten to assert the AWS
contract: the write succeeds with a distinct new version id, the new
content is current, the protected version keeps its content and lock
metadata, and version-targeted deletion of the protected version stays
blocked (for COMPLIANCE even with bypass_governance). Full object_lock
module is green again (33/33).
PR #4468 removed the last use of wildmatch (crates/notify swapped it for a
hand-rolled star-only glob matcher), but the workspace dependency declaration
in the root Cargo.toml was left behind as an orphan. cargo shear flags it as
not used by any workspace member. Drop the stray line.
Also refresh the comments in crates/notify/src/rules/pattern.rs that still
referred to the removed wildmatch crate, so they describe the current
hand-rolled matcher instead of a dependency that no longer exists.
Co-authored-by: heihutu <heihutu@gmail.com>
fix(admin): report unreachable info members as `unknown` with balanced drive counts
`GET /rustfs/admin/v3/info` synthesizes an entry for every peer, but a peer
whose properties RPC failed below the offline threshold with no cached snapshot
was reported as `initializing` with an empty drive list. That drive list being
empty meant its drives disappeared from the backend summary entirely, so
`onlineDisks + offlineDisks` no longer equaled `totalDrivesPerSet` (an operator
sees `onlineDisks: 3, offlineDisks: 0` for a 4-drive set), and `initializing` is
a misleading state for a member that has been running for hours — it reads like
a node was ejected when nothing is wrong (upstream rustfs/rustfs#4566).
Changes:
- madmin: add `ItemState::Unknown` ("unknown") and an `unknownDisks` bucket on
`ErasureBackend` (serde `default`, so older payloads still deserialize).
- ecstore diagnostics: `get_online_offline_disks_stats` now returns a third
`unknown` bucket instead of folding those drives into `offline`, keeping
`online + offline + unknown == totalDrivesPerSet`.
- ecstore notification_sys: a probe miss below the failure threshold with no
cache is now reported as `unknown` (not `initializing`) and carries the host's
drives from the pool topology (tagged `unknown`) so the summary stays balanced;
drive synthesis is factored into `synthesized_disks(host, endpoints, state)`,
shared by the offline and unknown paths.
Tracked in rustfs/backlog#1049 (P1-A + P0-A).
Co-authored-by: heihutu <heihutu@gmail.com>
* feat(utils): make DNS resolver cache TTL configurable via RUSTFS_DNS_CACHE_TTL_SECS
The resolver cache in crates/utils/src/net.rs had a hardcoded 300s TTL with
no way to tune or disable it. On orchestrated networks (Docker Swarm,
Kubernetes) a peer's DNS name is its only durable identity while its container
IP changes on reschedule, and the platform DNS always serves the current
answer. A fixed 300s application-side cache can keep a member dialing a dead IP
for up to five minutes after a peer restarts. Rust has no runtime DNS cache, so
on musl images this was the only cache in the stack and the only one that could
not be turned off.
Read the TTL from RUSTFS_DNS_CACHE_TTL_SECS (u64, default 300), matching the
existing RUSTFS_INTERNODE_* transport knobs. `0` disables caching entirely so
every get_host_ip consults the system resolver. The resolved value is logged
once at first use, removing an invisible cache from the triage surface. Change
is backward-compatible and confined to net.rs.
Also drop the two per-lookup info! logs on the resolve path: they would flood
the log when caching is disabled and added hot-path noise otherwise.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(helm): derive mTLS secret names from chart fullname (#4601)
* fix(helm): mtls certificate for server and client hardcode
* fix hardcode issue in deployment
* update typo yaml file
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: majinghe <42570491+majinghe@users.noreply.github.com>
fix(ecstore): emit CommonPrefix for object with same-named prefix in non-recursive listings (backlog#1042)
On single-disk / consistent deployments a non-recursive scan_dir that finds a/xl.meta classifies `a` as an object and does not descend, so the source never emits the prefix `a/`, dropping the CommonPrefix from delimiter listings even when `a/b` exists. This is the single-disk follow-up to backlog#880 (the multi-disk merge path was fixed in #4563).
Fix: in the scan_dir Ok branch, for a non-recursive listing of a plain object, probe whether `a/` holds a listing entry other than the object itself (new object_prefix_has_sibling_listing_entry, reusing directory_has_visible_listing_entry and skipping the object's own xl.meta and data dirs); if so, additionally schedule the prefix dir `a/`. The upstream fold in from_meta_cache_entries_sorted_infos already emits object `a` as Contents and dir `a/` as a CommonPrefix, so nothing changes there. Leaf objects (the common case) pay a single list_dir and return false immediately.
Verification: new scan_dir unit test (positive: `a` + `a/b` coexist; negative: leaf `c` produces no spurious `c/`); new end-to-end e2e (object `a` + `a/b` with delimiter "/" -> Contents `a` + CommonPrefix `a/`); existing list_objects_duplicates e2e (#1797 / #2439 regressions) stays green; set_disk::read 114 passed.
Note: lib-test compilation depends on #4573 (now merged), which added the allow_inplace_legacy_fallback argument to the codec streaming arity tests after #4560 / backlog#879.
Co-authored-by: heihutu <heihutu@gmail.com>
The three store::bucket::tests::bucket_delete_* tests drive make_bucket /
delete_bucket through the process-global local-disk registry
(GLOBAL_LOCAL_DISK_MAP) and lock client, and through Sets::new which reads the
process-global erasure mode. Under `cargo test` (single process, many threads)
they race other global-touching tests and fail deterministically with
InsufficientWriteQuorum.
serial_test's #[serial] with the crate default key serializes them across the
in-process suite. Measured at --test-threads=16: baseline fails all three every
round; with #[serial] all three pass every round. nextest's per-test processes
are covered separately by the ecstore-serial-flaky test-group in
.config/nextest.toml (#4558). Full instance-level isolation remains a backlog
#939 (InstanceContext) concern and is not attempted here.
Co-authored-by: heihutu <heihutu@gmail.com>
fix(ecstore): only evict remote lock channel on transport failures (#4567)
The remote lock RPC path evicted the cached gRPC channel on *any*
`tonic::Status`. Genuine transport failures (connect refused/reset,
keepalive/deadline) surface that way, but so do server-produced
application statuses on a perfectly healthy channel: `Unauthenticated`
/`PermissionDenied` from the signature interceptor, `Internal`
/`FailedPrecondition` when the peer lock service is not ready yet,
`InvalidArgument`, `ResourceExhausted`, `Unimplemented`, and so on.
Evicting the channel for those cannot help — the channel is fine — and
each spurious eviction re-dials the connection and advances the peer
toward the offline threshold via `record_peer_unreachable`, producing
background channel churn on an otherwise healthy cluster.
Classify the tonic status with `is_transport_failure` (underlying
transport source present, or one of `Unavailable`/`DeadlineExceeded`
/`Unknown`/`Cancelled`) and only evict the cached channel when it is a
real transport failure. The client-side timeout arm still evicts. This
mirrors the transport-vs-application distinction already used on the
remote-disk RPC path.
* refactor(ecstore): migrate the background-services cancel token into InstanceContext (Phase 5 Slice 13)
Phase 5 Slice 13 (backlog#939): move the background-services cancellation token
out of the process static into the per-instance InstanceContext, so cancelling
one instance's background workers (scanner/heal/tier/lifecycle) no longer
touches another instance.
- InstanceContext gains `background_cancel_token: OnceLock<CancellationToken>`
with `init_background_cancel_token` (set-once) and `background_cancel_token()`
returning an owned clone.
- global.rs `init_/get_/create_/shutdown_` helpers keep their signatures and
route through the current instance's context; the static is removed. The
getter now returns an owned `Option<CancellationToken>` instead of a
`Option<&'static _>`, which is what lets the token live in the context.
- Callers adapt to the owned token: the metadata-refresh loop drops `.cloned()`;
the lifecycle worker/loops take the owned token (a shared fallback is cloned
when the token is somehow uninitialized). No Arc cycle is introduced —
workers hold a token clone, not the instance context.
Single-instance behavior is unchanged: startup creates one token in the
bootstrap context the ECStore adopts, and shutdown cancels that same token.
Tests: the token is set-once and cancelling one instance's token leaves a
distinct instance uninitialized.
Verification: cargo test -p rustfs-ecstore (23 instance-context tests green),
cargo clippy -p rustfs-ecstore --all-targets (clean), make pre-commit (pass).
Refs: backlog#939 (Phase 5, Slice 13)
* test(ecstore): prove multi-instance isolation; document embedded guard retention (Phase 5 Slice 14) (#4588)
Phase 5 (backlog#939) capstone. The prior 13 slices moved every piece of
per-instance runtime state out of process globals into ECStore's
InstanceContext. This slice proves the result and records the remaining work.
- Add `two_instances_isolate_all_migrated_state`: an end-to-end acceptance test
that constructs two independent InstanceContexts and verifies NONE of the
migrated state is shared — erasure setup, lock manager, region, deployment id,
the four service handles (tier/notifier/expiry/transition), the local disk
registry, the bucket monitor, and the background cancel token. This is the
object-graph isolation carrier working end to end.
- Document why the embedded single-instance guard (EMBEDDED_SERVER_STARTED) is
intentionally retained: storage startup still publishes into the process-level
bootstrap context (write-once region/endpoints/deployment id) and the single
GLOBAL_OBJECT_API handle, so a second startup would fail-fast on that shared
state. Lifting the guard requires threading a per-instance context through
startup — a follow-up beyond migrating the globals. The guard is NOT removed:
rejecting the second start is safer than the panic it would otherwise become.
Verification: cargo test -p rustfs-ecstore (acceptance test + all instance-context
tests green), cargo clippy -p rustfs-ecstore --all-targets (clean), make
pre-commit (pass).
Refs: backlog#939 (Phase 5, Slice 14). Stacked on Slice 13 (#4586).
The internode HTTP/2 window/keepalive tuning (apply_http_client_tuning +
http2_keep_alive_* in build_http_client) only takes effect when the connection
negotiates HTTP/2, which happens via TLS-ALPN. Over plaintext internode
transport the connection is HTTP/1.1 and all the tuning is silently inert,
with no signal to the operator.
Make this honest without changing protocol behavior (no h2c, no
prior_knowledge, no opt-in flag):
- InternodeHttpClientTuning gains h2_tuning_explicit, true when the operator
set a window size, a non-default tuning profile, or the keepalive env.
- Add pure predicate should_warn_h2_inert(negotiated_is_http2,
h2_tuning_explicit, already_warned) and maybe_warn_h2_inert, gated by a
process-lifetime AtomicBool so the warn fires at most once.
- Call maybe_warn_h2_inert at both record_internode_http_version sites using
the actual negotiated resp.version().
- Document the TLS-ALPN requirement at apply_http_client_tuning and near the
ENV_INTERNODE_HTTP2_* constants.
Tests cover the should_warn_h2_inert truth table and that h2_tuning_explicit
reflects explicit windows and a non-default profile.
fix(ecstore): age LastMinuteLatency samples individually (backlog#806-16)
The rolling one-minute latency window stored bare Duration samples plus a
single, never-updated start_time. Its retain predicate ignored the element
and evaluated the constant now.duration_since(start_time) < 60s, so once the
window had existed for 60s every add() dropped ALL samples and the average
degenerated to the latest single sample.
Each sample now carries its own Instant and is retained by its individual
age. The type backs only in-memory EpHealth + admin display (the wire shape
is target::LatencyStat with curr/avg/max), so Serialize/Deserialize is kept
only to satisfy the enclosing LatencyStat derive; the sample Instant is
serde(skip) and restarts aging on reload.
fix(io-metrics): drop client-controlled bucket label from rustfs_s3_operations_total (backlog#806-22)
The bucket dimension on the rustfs_s3_operations_total counter is
client-controlled and unbounded, letting any client explode metric
cardinality (a Prometheus/OTEL cardinality DoS that grows the in-process
OTEL aggregation store even without a scrape). Drop the bucket label so the
series is keyed by the bounded op dimension only (<=122 variants), matching
MinIO which never labels its default operation counters with bucket.
BREAKING: rustfs_s3_operations_total no longer carries a "bucket" label.
record_s3_op(op, bucket) -> record_s3_op(op); all call sites in
rustfs/src/storage/{ecfs.rs,helper.rs} updated (bucket bindings retained
where still used by audit/notify paths). Add metrics-util debugging recorder
dev-dep and tests asserting the series carries only the op label and that
series count equals distinct-op count.
houseme