mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-07 13:53:12 +00:00
48c8d85f3baeb2dbe70175e38dbf35bb69c8b7fc
79 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 houseme
|
035ce5d784 |
feat(obs): add bounded metrics dimensions (#5645)
* feat(obs): add drive topology detail metrics Expose additive drive info, topology, state, and per-drive API metrics while preserving the existing drive metric label sets. Backlog: rustfs/backlog#1655 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs): preserve suspect drive runtime state Keep suspect as a bounded drive runtime state and avoid all-zero runtime_state samples for that storage health state. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs): skip unknown drive inode samples Avoid exporting zero inode gauges for missing or stale drive snapshots and ignore zero-count API latency buckets. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(obs): add scanner source work detail metrics Expose additive scanner source and cycle work metrics with bounded server/source/state labels while leaving the existing aggregate scanner metrics unchanged. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(obs): add ilm action detail metrics Expose additive ILM action/state task metrics with a server label while preserving the existing aggregate ILM series. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(obs): add delivery target server metrics Expose additive audit and notification delivery target metrics with server labels and extend removed-target tombstones for the server-aware series. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(obs): add replication target flow metrics Expose additive bucket replication target sent and failed-flow metrics while preserving existing bucket aggregates and target backlog series. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(obs): add request server metrics Expose additive API request metrics with server labels while preserving the existing request and traffic metric label sets. Co-Authored-By: heihutu <heihutu@gmail.com> * style(obs): apply rustfmt to metrics changes Apply rustfmt output to the metrics dimension changes without altering behavior. Co-Authored-By: heihutu <heihutu@gmail.com> * style(obs): reuse audit target label constant Use the exported audit target_id label constant for legacy audit target metrics. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(obs): populate drive disk metrics Co-Authored-By: heihutu <heihutu@gmail.com> * feat(obs): add scanner bucket drive result metrics Co-Authored-By: heihutu <heihutu@gmail.com> * feat(obs): add replication proxy server metrics Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs): address metric liveness review Use checked division for drive API latency aggregation and keep recovered drive, scanner current-cycle, replication flow, audit target, and notification target series from retaining stale values. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs): address metric dimension review Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs): address additional metric review Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs): count drive calls at start Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs): address metrics dimension review Co-Authored-By: heihutu <heihutu@gmail.com> * fix(metrics): address dimension review gaps Co-Authored-By: heihutu <heihutu@gmail.com> * fix(metrics): address scanner review follow-ups Co-Authored-By: heihutu <heihutu@gmail.com> * fix(metrics): address runtime review follow-ups Co-Authored-By: heihutu <heihutu@gmail.com> * fix(metrics): reduce disk metric contention Co-Authored-By: heihutu <heihutu@gmail.com> * fix(metrics): address runtime review follow-ups Co-Authored-By: heihutu <heihutu@gmail.com> * fix(metrics): retire stale dimension series Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
a08fb56607 |
test(perf): strengthen HotPath evidence contracts (#5581)
* test(perf): mark nonformal ABBA artifacts Co-Authored-By: heihutu <heihutu@gmail.com> * test(io-metrics): lock disabled EC stage accounting Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
749cb2c1a1 |
perf: harden HotPath closure evidence (#5573)
* test(ecstore): cover raw shard write errors Co-Authored-By: heihutu <heihutu@gmail.com> * perf(ecstore): track encode payload stage peaks Co-Authored-By: heihutu <heihutu@gmail.com> * test(perf): harden formal warp ABBA evidence Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
houseme
|
d5c6ba99d5 |
fix(ecstore): harden HotPath profiling boundaries (#5555)
* test(hotpath): gate mimalloc heap test by platform Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): attribute HotPath CPU measurements to impls Co-Authored-By: heihutu <heihutu@gmail.com> * feat(ecstore): trace raw shard I/O with HotPath Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs): redact profiler and OPA endpoint diagnostics Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): settle encoded queue accounting Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): abort encoder producer on cancellation Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
78d6918c52 |
feat: extend hotpath coverage across crates (#5505)
Add opt-in hotpath feature surfaces to every workspace crate and wire the root rustfs feature passthrough for function, allocation, and CPU profiling. Add a focused set of function-level measurements for scanner, heal, lock, target replay, IAM, KMS, Keystone, trusted proxy, and capacity paths without adding request-scoped primitive wrappers. Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
30dc04c94b |
fix(obs): label node-local metrics by server (#5465)
Add stable server labels to node-local Prometheus metrics and OTLP resource attributes so dashboards can distinguish per-node CPU, memory, host network, and internode traffic series. Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
Zhengchao An
|
719c0d6ef0 | feat(rpc): add replay-scoped internode authentication (#5455) | ||
|
houseme
|
683ff52a7b |
feat(internode): track msgpack decode traffic (#5309)
Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
Zhengchao An
|
61d4e04d65 |
feat(rpc): bind canonical body digest into internode mutating disk RPC signatures (#5234)
* feat(rpc): bind canonical body digest into internode mutating disk RPC signatures Binds a domain-separated, length-prefixed canonical request-body digest into the v2 HMAC signature scope for every mutating NodeService disk RPC, so an on-path attacker on the default-plaintext internode channel can no longer tamper with a mutation payload (or strip the msgpack `_bin` field to force the JSON fallback decode) without invalidating the signature. Covers 13 mutating disk RPCs: RenameData, DeleteVersion, DeleteVersions, WriteMetadata, UpdateMetadata, WriteAll, Delete, DeletePaths, RenameFile, RenamePart, DeleteVolume, MakeVolume, MakeVolumes. The digest covers both the msgpack `_bin` payloads and their JSON compatibility copies. Gated fail-open by default (RUSTFS_INTERNODE_RPC_BODY_DIGEST_STRICT) with a convergence counter, so rolling upgrades are byte-for-byte unaffected; the replay-cache capacity is now configurable and overflow fails closed with a metric. Refs https://github.com/rustfs/backlog/issues/1327 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(rpc): satisfy architecture-migration compat-marker guard Put the removal condition on the RUSTFS_COMPAT_TODO marker line itself, and stop backticking env-var/metric names in the cleanup-register entry so the guard's id extractor only sees the task-id. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
houseme
|
5988606e68 |
test(internode): extend fallback and transition coverage (#5232)
Add decode-error metrics for internode msgpack/json compatibility paths, extend the mixed fallback e2e assertions for transitioned multipart partNumber reads, and cover manual transition async status polling plus inactive-owner status behavior. Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
Henry Guo
|
a63b79004c |
fix(scanner): make distributed usage convergence authoritative (#5151)
* fix(scanner): make distributed usage cycles authoritative * fix(scanner): close distributed refresh races * fix(config): align scanner reload integration * fix(admin): scope config test helpers * fix(scanner): harden distributed usage convergence * fix(scanner): preserve rolling activity compatibility * fix(admin): expose non-secret optional config values * fix(scanner): acknowledge distributed dirty usage * fix(ecstore): make bucket mutations cancellation safe * fix(scanner): preserve pending dirty acknowledgements * test(obs): account for superseded scanner metric * fix(api): reject excess detached bucket mutations * test: close scanner convergence coverage gaps * fix(scanner): make path tracking cleanup one-shot --------- Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com> Co-authored-by: houseme <housemecn@gmail.com> |
||
|
Zhengchao An
|
ffd1b94e1f |
feat(rpc): add server-side internode v2 signature verification (fail-open, method-path binding groundwork) (#5160)
* feat(rpc): gate internode legacy signature fallback behind a convergence counter and strict env Add the backlog#1327 Plan-A rollout infrastructure on top of the already-merged v2 target-bound internode gRPC authentication: a rustfs_system_network_internode_signature_v1_fallback_total counter that increments only when a request without any v2 auth headers is accepted through the legacy constant-target signature, and a RUSTFS_INTERNODE_RPC_SIGNATURE_STRICT env (default false, compile-time asserted) that, when enabled later, closes the legacy fallback path. Default behavior is fail-open and byte-identical for legacy-only peers; requests carrying v2 headers are verified as v2 with no downgrade exactly as before. Refs https://github.com/rustfs/backlog/issues/1327 * ci: fix internode auth test lint failures * perf(ecstore): cache internode sig strict env --------- Co-authored-by: houseme <housemecn@gmail.com> |
||
|
Zhengchao An
|
8e214104f3 |
perf(ecstore): add guarded encrypted multipart range seeks (#5145)
* perf(ecstore): seek encrypted multipart Range GETs to the covering part boundary on the Legacy rio v1 backend Phase A of https://github.com/rustfs/backlog/issues/1316. Encrypted Range GETs on the default (non rio-v2) backend previously planned storage_offset=0, storage_length=oi.size and discarded the decrypted prefix, so a small Range on a large SSE object read, erasure-decoded, and decrypted the whole object. Eligible multipart objects now seek to the covering part boundary using only the per-part size/actual_size metadata facts; the multipart decrypt reader already handles streams starting at any part boundary, so rio and the on-disk format are untouched. Single-part objects, compressed payloads, defective parts tables, and zero-length ranges keep the previous full read, and RUSTFS_ENCRYPTED_RANGE_SEEK=false restores it globally. A new histogram rustfs_get_encrypted_range_read_amplification plus a full|part_seek path counter record the physical/plaintext amplification at the ReadPlan decision point. * fix(ecstore): guard encrypted multipart range seeks * fix(io-metrics): align encrypted range metric names with the rustfs_io_ prefix Every other metric in rustfs-io-metrics uses the rustfs_io_ prefix; the two encrypted-range-seek metrics were the only exception. Rename before first release so dashboards never see the unprefixed names. |
||
|
houseme
|
a8e7cce5e1 |
feat: expose list read-dir amplification metrics (#5103)
Record local read_dir entry counts and duration for live-walker ListObjects scans so wide root/prefix amplification can be measured below the cross-set merge layer. Add a focused LocalDisk scan_dir test showing a page limit of one still observes the whole parent directory enumeration, plus metric helper coverage. Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
7805cf5ae6 |
fix(cluster): clarify peer health and listing timeouts (#5086)
* fix(cluster): surface observed peer health Refs rustfs/backlog#1387 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(admin): distinguish unreported peer health Refs rustfs/backlog#1388 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): decouple metacache peek timeout Refs rustfs/backlog#1389 Co-Authored-By: heihutu <heihutu@gmail.com> * docs(ops): diagnose metacache listing timeouts Refs rustfs/backlog#1390 Co-Authored-By: heihutu <heihutu@gmail.com> * refactor(cluster): clarify observed peer health Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): route capability state through contract Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
56179210ab |
chore(deps): simplify dependency features (#4890)
* chore(deps): remove redundant dependency features Remove manifest feature entries that are implied by other requested features in the same dependency declaration. Verified that the resolved Cargo feature graph is unchanged after the cleanup. Co-Authored-By: heihutu <heihutu@gmail.com> * chore(deps): narrow tokio and reqwest features Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
f3a7a4b0da |
chore(deps): localize workspace dependency features (#4888)
Move workspace-level dependency feature lists into the member crates that consume each dependency while keeping required default-features flags at the workspace root. Also refresh starshard to 2.2.2 via cargo update and cargo upgrade --exclude ratelimit. Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
1553dc3f62 |
Address P2 follow-ups from the 2026-07-10..12 merged-PR review (backlog#1210-1220) (#4783)
* fix(obs): open cleaner compression source with O_NOFOLLOW The compressor opened the source log via File::open, which follows a symlink at the final path component. Between the scanner selecting a regular file and this open, an attacker with write access to the log directory could swap the entry for a symlink (TOCTOU) pointing at, say, /etc/shadow, whose contents would then be copied into an archive. Open the source with O_NOFOLLOW on Unix so such a swap fails with ELOOP; the temp/archive path already refused symlinks, this closes the source side. Refs rustfs/backlog#1210 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs): recompress instead of trusting leftover cleaner archives archive_header_ok only checked the first 2-4 magic bytes before treating an existing .gz/.zst as a completed prior result and letting the caller delete the source log. A file with valid magic but a truncated or forged body passes that check, so an attacker with write access to the log directory (or a crashed prior run) could plant such a stub and make the cleaner delete the real log without ever producing a usable archive — silent audit-data loss. Chosen fix: stop trusting cross-process leftovers entirely and always recompress the source in this pass, rather than fully decoding every leftover to validate it. Full-decode validation would add real CPU cost and decode-bug surface for a rare crash-recovery case; the existing atomic create_new+rename already overwrites whatever sits at the archive path (a planted symlink is replaced, never followed) with a freshly written, fsync'd archive, so a partial/forged leftover can never gate source deletion. This is the lowest-regression option. Refs rustfs/backlog#1211 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(object-data-cache): cap memory-gate reservation at cache growth headroom The memory gate subtracts `admitted_since_refresh` from the snapshot's available bytes so a burst arriving faster than the 5 s refresh cannot over-allocate. That counter is GROSS: it only rolls over on the refresh and never rolls back when a fill is later evicted, cancelled, or loses the invalidation race. Under sustained high-throughput churn (net footprint flat and far below `max_capacity`) the raw counter balloons past the memory the cache actually holds, so `effective_available` collapses and the gate reports false memory pressure — skipping the hottest fills with SkippedMemoryPressure until the next 5 s refresh. This only lowers hit rate; it never returns wrong data and self-heals each refresh. Fix direction 1 (minimal regression): cap the reservation deduction at the cache's own growth headroom (`max_capacity - weighted_size()`) instead of letting the unbounded gross counter shrink the system-available budget. The cache can never hold more than `max_capacity`, so a burst adds at most that headroom of real memory before moka evicts to stay bounded (net-zero churn beyond that point) — capping the deduction there keeps the reservation honest without treating gross churn as growth. Chosen over net-accounting (direction 2, releasing bytes on every failure/cancel/eviction path) because that only plugs the leak on failed fills and would not address the core defect: churn of *successful* insert/evict fills over the 5 s window. It also touches only the gate plus one call site rather than every failure path in moka_backend. The cap only ever raises `effective_available`, so real memory pressure (a low snapshot at refresh) still suppresses fills; when the cache is at capacity the headroom is 0 and the deduction vanishes, correctly reflecting net-zero churn. `MokaBackend` now stores `max_capacity` and passes the live headroom into `allows_fill`. Adds targeted gate tests: gross churn far above headroom no longer falsely suppresses, yet the reservation still bounds a burst while the cache can genuinely grow. Refs rustfs/backlog#1212 Co-Authored-By: heihutu <heihutu@gmail.com> * test(ecstore): assert native O_DIRECT path runs in uring read test uring_preserves_o_direct_for_eligible_reads only compared bytes through LocalDisk::read_file_mmap_copy. On a filesystem that rejects O_DIRECT the read silently degrades to the buffered StdBackend fallback and the byte check still passes, so the test could go green without the native read_at_direct path ever executing -- a vacuous pass. Add a per-disk native_direct_reads counter on UringBackend, incremented only when pread_uring_direct completes, and rebuild the test to drive a real UringBackend's pread_bytes and assert the counter is non-zero (every eligible read went through the native tier). When io_uring or O_DIRECT is unavailable on the host filesystem (restricted CI runners, tmpfs), the test skips loudly via eprintln instead of asserting a tautology, while still checking byte-correctness on whatever tier served the read. The counter also gives a gray release a positive signal that the O_DIRECT tier is serving reads, not just a fallback count. Refs rustfs/backlog#1213 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): warn + count read-time EINVAL on native O_DIRECT reads classify_direct_read_error is only reached from the read side: the O_DIRECT open in pread_uring_direct already succeeded (an open-time refusal is handled earlier as DirectOpenError::ODirectRefused). So an EINVAL/EOPNOTSUPP arriving here is a read-time error on an fd the kernel accepted for O_DIRECT -- far more likely an alignment bug in the aligned read path than an unsupported filesystem. The old code latched the disk's native path off with only a once-per-disk debug trace, hiding a potential correctness regression behind a silent buffered-read downgrade. Diagnostics only: the fallback behaviour is unchanged (the native path is still latched off and the caller still reads via StdBackend). This adds a rustfs_io_uring_direct_read_einval_total counter and promotes the once-per-disk trace from debug to warn so an operator can see an alignment regression instead of an unexplained latency/CPU shift. Refs rustfs/backlog#1214 Co-Authored-By: heihutu <heihutu@gmail.com> * docs(ecstore): document data-blocks-first default and its tail-latency cost DEFAULT_RUSTFS_GET_DATA_BLOCKS_FIRST_READER_SETUP is true and must stay true: deferred-parity is the deliberate, already-rolled-out full-object GET default from backlog#1159/#923. Flipping it back to false in code would silently revert that rollout for every deployment that has not set the env var, so this commit only documents -- no behaviour change. The added notes explain what data-blocks-first does (schedule data shards up front, engage parity lazily on a missing/corrupt data shard), the known trade-off (parity is engaged late, so a slow-but-not-dead data drive raises GET p99 because the faster parity shards are not raced against it until a data shard is declared missing), and the operational rollback switch (RUSTFS_GET_DATA_BLOCKS_FIRST_READER_SETUP=false), which is intentionally an env override rather than a code default change. No metric was added: the low-risk observability hook for "slow data drive engaged deferred parity" would live at the deferred-stripe engage point, which is out of this file's scope; this change stays documentation-only to avoid touching the hot GET path. Refs rustfs/backlog#1215 Co-Authored-By: heihutu <heihutu@gmail.com> * docs(ecstore): document wide-directory walk stall hazard and tuning list_dir enumerates a whole directory in one os::read_dir call (count = -1), and the walk caller bounds that entire enumeration with the per-read stall budget (default 5s) as if it were a single read. For a wide, flat prefix -- one directory holding millions of immediate children -- a single readdir can exceed the budget on a healthy disk, trip DiskError::Timeout, and surface as a ListObjects 500 quorum failure though the drive is fine (a #2999 sub-class). This is documented, not rewritten: turning the one-shot readdir into a streaming/batched enumeration that refreshes the stall deadline between chunks is an architecture-level change with high regression surface (ordering, the count contract, quorum merge) and belongs in a separate follow-up. The supported mitigation today is operational, so the comments point wide-directory deployments at RUSTFS_DRIVE_WALKDIR_STALL_TIMEOUT_SECS and the high-latency drive-timeout profile, which widen the budget with no code change. Notes were added at list_dir, the scan_dir call site, and get_drive_walkdir_stall_timeout. No behaviour change. Refs rustfs/backlog#1216 Co-Authored-By: heihutu <heihutu@gmail.com> * docs(ecstore): document consumer-peek vs producer-stall coupling In list_path_raw the consumer's peek_timeout is drawn from the same source and same value (walkdir_stall_timeout, default 5s) as the producer-side walk stall budget, but the two measure different things: the producer stall bounds a single drive read, while the consumer peek bounds the gap between two ADJACENT entries arriving from a reader. Because they share a value, the consumer cannot wait meaningfully longer for the next entry than the producer is allowed to spend producing one. Walking a region dense with non-listable internal items can make a HEALTHY drive miss the budget between visible entries; the consumer then declares it stalled and detaches it, dropping a good drive from the merge and capping the "large prefix succeeds" guarantee. Documented, not decoupled: giving the consumer peek an independent, strictly-larger budget would cut these false detaches but equally delays detaching a genuinely dead drive and shifts listing tail-latency semantics, so it wants soak data before changing the default. The comment records the invariant any such follow-up must keep -- consumer peek >= producer stall, never stricter -- so it can never fail a drive before the producer would. No behaviour change. Refs rustfs/backlog#1217 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(io-metrics): add time-based trigger for low-IOPS latency percentiles Percentiles were recomputed only every 128 IOs and seeded to 0, so a low-traffic deployment exported p95/p99 = 0/stale for a long time after startup. Add a 10s wall-clock trigger alongside the count throttle so the first recompute can fire before 128 samples accrue. Hot-path per-op mean update is unchanged. Refs rustfs/backlog#1218 Co-Authored-By: heihutu <heihutu@gmail.com> * test(e2e): cover codec-streaming parity under fault injection and NoSuchKey The codec-streaming compat A/B previously ran only against a healthy 4-disk EC set with successful full GETs: the DiskFaultHarness was constructed but never faulted, the error path was untested, and the range assertion silently compared legacy-vs-legacy (ranges always fall back to the duplex path), overstating what it proved. Add two genuinely-failable scenarios reusing the existing harness and fixtures: - Parity reconstruction A/B: take one data disk offline and re-run the full object matrix on both phases while the EC 2+2 set rebuilds each large object from the surviving shards. Assert codec == legacy byte-for-byte (sha256) and header-for-header, and assert the codec phase served the reconstructed objects with zero duplex-pipe fallback (the reader gate is drive-health-independent, so the codec fast path is really exercised through reconstruction). - NoSuchKey negative path: compare the HTTP status + S3 error code of a missing-key GET across the legacy and codec phases and require them to be identical (404/NoSuchKey), guarding against the codec env perturbing the error path. Also clarify the range-phase comment so it is not misread as codec-range correctness coverage: both sides are served by the same legacy range path, so the assertion only proves ranges keep working and keep falling back to legacy with the gates open. Verified: cargo check/--no-run pass and the test passes locally (1 passed; dup_codec=0 confirms the codec path ran). Refs rustfs/backlog#1219 Co-Authored-By: heihutu <heihutu@gmail.com> * ci(ecstore): exercise native O_DIRECT read path on an ext4 loopback The uring-integration leg ran on the runner's default TMPDIR, which may sit on tmpfs/overlayfs where open(O_DIRECT) fails and the native read_at_direct path silently latches off to the aligned StdBackend fallback. Mount a dedicated ext4 loopback and point TMPDIR at it so the real io_uring dep (bumped git->0.1.0->0.2.0->0.2.1) and the native O_DIRECT read path are actually covered rather than validated only by signature diffing. Refs rustfs/backlog#1220 Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
633c131cef |
perf(io-metrics): cache handles for hot label-less metric emitters (#4750)
The `metrics` macros re-run the recorder's `register_*` on every emission — a `RwLock` read, a name-key hash, and an `Arc` clone — even for a metric that never varies its key. For the hot, label-less recorders on the per-IO path (`record_data_transfer`, `record_io_latency`, `record_io_latency_p95/p99`, `record_io_queue_congestion`) that lookup is pure overhead once observability is on. Add `counter_increment_cached!` / `gauge_set_cached!` / `histogram_record_cached!` that resolve the handle once via `LazyLock` in production and reuse it. Under `cfg(test)` they re-resolve on every call, because the `metrics` crate resolves against a thread-local recorder that `with_local_recorder` swaps per test — a process-global cached handle would bind to whichever recorder was active first and break test capture. The macros only wrap FIXED (label-less) keys, and the `metrics_enabled()` gate still short-circuits before any emission when disabled. Verified: the only callers of these functions are the collector (io-metrics' own cfg(test) tests, which re-resolve) and production code; no cross-crate test captures them. rustfs-io-metrics builds on both cfg paths, 147 unit + 4 doctests pass, clippy clean. Addresses rustfs/backlog#1185 (P3, per-emission handle caching). Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
5282c71f86 |
perf(io-metrics): make internode peer-health read-mostly with an RwLock (#4746)
`cluster_peer_should_bypass` is called before every internode RPC when the offline-bypass feature is enabled (remote disk `get_client`, remote locker), and it took a single process-global `Mutex<HashMap>` even for the overwhelmingly common case of an online or unknown peer, which is read-only. That serialized all concurrent internode client acquisitions on one lock. Switch `CLUSTER_PEER_HEALTH` to a `RwLock`: - the hot check takes a shared read lock and returns immediately for unknown or online peers, so concurrent RPCs no longer serialize; - only an offline peer (rare) drops to the write lock to record a re-probe, with a re-fetch/re-check because the state can flip back online between releasing the read lock and taking the write lock; - the write paths (dial reachable/unreachable) and the read-only `cluster_peer_is_offline` move to `write()`/`read()` accordingly. Behavior is unchanged on every path (online/unknown -> not bypassed; offline -> bypass with one re-probe per interval); `Instant::now()` now runs only on the offline path. Poison recovery is preserved. All internode unit tests pass. Addresses rustfs/backlog#1185 (P2). Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
ce7d3119b2 |
perf(io-metrics): throttle collector percentile sort off the per-IO path (#4744)
`MetricsCollector::record_io_operation` recomputed P50/P95/P99 on every disk IO by taking a read lock, collecting up to `max_latency_samples` (1000) into a `Vec<u128>` and fully sorting it — an O(n log n) alloc+sort per operation on the GET read path (gated by stage metrics). Both consumers sample only periodically: the autotuner reads `avg_io_latency_us` on its tuning tick, and the P95/P99 values are never read internally — they only feed OTEL export. Nothing needs per-IO freshness. Split the two costs: - The window mean (stored in `avg_io_latency_us`, read by the autotuner) is now maintained in O(1) via a running sum kept in step with the window's push/pop, and refreshed on every op — no sort, no warm-up regression. - The P95/P99 sort is throttled to once per `PERCENTILE_RECOMPUTE_INTERVAL` (128) operations. A bounded lag is invisible to the periodic autotuner tick and OTEL export while the per-op sort cost is amortised ~128x away. Semantics are otherwise unchanged: same sliding window, same percentile indices, same mean value. Tests updated — the percentile test forces a recompute to exercise the math directly, and a new test asserts the mean tracks every op while the percentiles only recompute at the interval boundary. Addresses rustfs/backlog#1185 (P1). Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
a8eed1c44d |
perf(io-metrics): add a global switch to gate general metric emission (#4733)
The io-metrics crate already had per-stage GET/PUT switches, but the request headers/summaries and ~40 general recorders (I/O scheduler, bytes-pool, zero-copy, bandwidth, system-resource, error/timeout/retry) emitted unconditionally, paying their label allocations and arithmetic even when no metric recorder is installed. Add `METRICS_ENABLED: AtomicBool` (default false) with `set_metrics_enabled()` / `metrics_enabled()`, isomorphic to the existing stage switches, and gate: - GET headers/summaries via `get_stage_metrics_enabled()`: record_get_object_request_start / _request_started / _request_result / _timeout / _completion / _total_duration_with_path, plus legacy record_get_object. - PUT headers via `put_stage_metrics_enabled()`: record_put_object_request_start / _request_result, plus legacy record_put_object. - 41 general recorders via `metrics_enabled()`. Startup enables all three switches together under `observability_metric_enabled()` (startup_observability.rs), with a passthrough in startup_runtime_sources.rs. The global recorder is itself only installed when observability metric export is on, so gating off is pure cost savings when export is disabled and a no-op when it is enabled. Deliberately NOT gated (would break function, not just observation): the EC encode in-flight and GET buffered-bytes accounting guards (add/remove_ec_encode_inflight_bytes, track_get_object_buffered_bytes) whose values are read back by current_*; and the stateful sibling modules (lock/deadlock/backpressure/autotuner/adaptive_ttl/collector/internode). The console realtime metrics endpoint (/admin/v3/metrics) samples system state and internode metrics directly, so it does not depend on the gated recorders. Adds a toggle test exercising both the enabled and disabled paths, serialized on the existing METRICS_FLAG_LOCK to stay robust under nextest's shared-process model. Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
Zhengchao An
|
ed36d1ed97 |
fix(io-metrics): drop client-controlled bucket label from s3 ops counter (backlog#806) (#4580)
fix(io-metrics): drop client-controlled bucket label from rustfs_s3_operations_total (backlog#806-22)
The bucket dimension on the rustfs_s3_operations_total counter is
client-controlled and unbounded, letting any client explode metric
cardinality (a Prometheus/OTEL cardinality DoS that grows the in-process
OTEL aggregation store even without a scrape). Drop the bucket label so the
series is keyed by the bounded op dimension only (<=122 variants), matching
MinIO which never labels its default operation counters with bucket.
BREAKING: rustfs_s3_operations_total no longer carries a "bucket" label.
record_s3_op(op, bucket) -> record_s3_op(op); all call sites in
rustfs/src/storage/{ecfs.rs,helper.rs} updated (bucket bindings retained
where still used by audit/notify paths). Add metrics-util debugging recorder
dev-dep and tests asserting the series carries only the op label and that
series count equals distinct-op count.
|
||
|
houseme
|
506ded59ad |
fix(metrics): servers_offline_total counts each peer once (normalize peer-health key) (#4572)
fix(metrics): normalize peer-health key so servers_offline_total counts each peer once rustfs_cluster_servers_offline_total is the count of distinct keys in the addr-keyed CLUSTER_PEER_HEALTH map. The data path keys a peer by endpoint.grid_host() (scheme://host:port) while the lock path keys it by url::Url::to_string() (scheme://host:port/, trailing slash), so one physical peer becomes two health entries and a single downed node reports 2 (2N for N). Normalize the address (trim trailing slashes) at the map boundary in record_peer_reachable/record_peer_unreachable/cluster_peer_is_offline/ cluster_peer_should_bypass so both forms collapse to one canonical key. Pure observability fix; peer selection and quorum are unchanged. Fixes rustfs/backlog#1043 Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
Zhengchao An
|
cf2da0c44d |
refactor(object-capacity): remove the decorative SymlinkTracker and its no-op depth knob (#4571)
The tracker never influenced traversal: walkdir's follow behavior is fixed up front by follow_links(), and should_follow() only gated the tracker's own bookkeeping. Its 'depth limit' compared tree depth (not symlink chain depth), so RUSTFS_CAPACITY_MAX_SYMLINK_DEPTH was a complete no-op while its telemetry claimed symlinks were skipped that walkdir had in fact followed and counted; record_symlink was always called with size 0, so tracked_bytes never left zero (S12). Remove the tracker, its skipped/summary events, the symlink metric and the depth env knob end to end (the env's 'as u8' truncation goes with it), and document the real semantics at the walker: follow_links(true) counts targets with walkdir's ancestor-loop detection breaking cycles, follow_links(false) — the default — counts no symlink targets. The scan root itself is pre-resolved since backlog#1015. Ref: rustfs/backlog#1018 (S12 from audit rustfs/backlog#1010) |
||
|
Zhengchao An
|
0ebe00b383 |
fix(object-capacity): drive scan progress checks by visited entries and remove unreachable stall detector (#4557)
Progress/timeout checks only ran when file_count advanced, so trees of directory-only or error-dense entries (dirs, traversal errors, symlinks never increment file_count) bypassed the entire cooperative time budget and held the blocking thread for unbounded time, while each error entry logged its own warn — permission-dense trees flooded the log (S13). The stall detector was structurally unreachable: it fired on 'no file progress between checks', but checks themselves only ran when file progress happened, so RUSTFS_CAPACITY_STALL_TIMEOUT never did anything since its introduction (S09). - Progress checks (timeout + early-sampling entry) are now driven by a visited-entry counter that also advances on directories and errors, so every tree shape reaches the budget checks. - Remove the unreachable stall detector and its plumbing end to end (ProgressMonitor fields, ScanLimits, config getter, env const, stall metric). Genuine walker wedges are handled by the hard outer wall-clock budget from backlog#1017; no decorative protection is left. - Cap per-entry error warns at 10 per scan with an explicit suppression notice; had_partial_errors still records the condition. Ref: rustfs/backlog#1016 (S09+S13 from audit rustfs/backlog#1010) |
||
|
Zhengchao An
|
c6d07ffc59 |
fix(object-capacity): mark partial-failure refreshes degraded and merge over disk cache (#4499)
A full refresh with partial disk failures used to commit the surviving subset's sum as a fresh exact cluster total (no field carried the partial-failure fact), while the complete disk cache kept the failed disk's old value — so the reported capacity oscillated between the partial sum and the cache-merged total on alternating refreshes. - Add a degraded flag to CapacityUpdate/CachedCapacity, set when the scan behind the update had partial errors; expose it in refresh logs, admin capacity logs and a new rustfs_capacity_degraded_readings_total counter. - On a degraded full refresh, surface only the disks whose own scan fully succeeded and merge them over a complete disk cache, so failed disks keep their last-known values and the published total no longer dips and bounces back. The disk cache is never replaced from a degraded refresh. - Without a complete cache, keep the partial sum (unchanged #805 non-pollution behavior) but mark the reading degraded. Ref: rustfs/backlog#1014 (S06 from audit rustfs/backlog#1010) |
||
|
houseme
|
6cb47049e8 |
fix(obs): isolate process sampler windows (#4492)
* fix(obs): isolate process sampler windows Refs rustfs/backlog#1004 Refs rustfs/backlog#986 - add a reusable ProcessSampler so callers can own independent sysinfo refresh windows - wire separate sampler instances for obs metrics scheduling and memory observability - keep compatibility helpers while avoiding cross-task CPU and disk delta interference Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs): import process sampler bundle helper Refs rustfs/backlog#1004 Refs rustfs/backlog#986 - import collect_process_metric_bundle_with in the metrics scheduler - drop the stale collect_process_metric_bundle import after switching scheduler sampling to independent process samplers Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs): move process sampler into blocking task Refs rustfs/backlog#1004 Refs rustfs/backlog#986 - move the memory observability process sampler into the spawn_blocking closure - satisfy the closure static lifetime required by tokio while keeping the isolated sampler design intact Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
Zhengchao An
|
afc7f1d6f9 |
fix(ecstore): make post-commit old data dir cleanup best-effort (#4386)
* fix(ecstore): make post-commit old data dir cleanup best-effort (backlog#898) A write is authoritatively committed once rename_data returns Ok (the new version is durable on >= write_quorum disks and immediately readable). The subsequent reclamation of the now-dereferenced old object/<data_dir> is pure space reclamation, yet commit_rename_data_dir propagated a below-quorum GC failure via `?` into ErasureWriteQuorum -> 503, producing a false-negative ACK for an already-persisted write. This is a deliberate divergence from MinIO (erasure-object.go:1577), which couples the two; the divergence is justified by durability semantics, not parity. Changes: - commit_rename_data_dir now returns a structured OldDataDirCleanup receipt and never returns Err. Adds an old==committed-dir anti-misdelete guard and a committed_data_dir parameter. Classification is extracted into pure functions (classify_old_data_dir_cleanup / map_cleanup_join_result / is_cleanup_not_found) so it is unit-testable. Task panic/cancel is mapped to a non-ignored DiskError::other (never DiskNotFound), and not-found is normalized to reclaimed. - object.rs / multipart.rs consume the receipt instead of `?`. The result reverts to Ok, so the invalidate_get_object_metadata_cache self-heal and the capacity/compression accounting that a `?` early-return previously skipped now run on the cleanup-failure path too. - On residue, report_old_data_dir_cleanup emits leak metrics and enqueues an object heal over the existing heal channel (disk-health signal replacing the 503). heal_object -> reclaim_orphan_data_dirs already reclaims unreferenced local data dirs, closing the loop end to end. - Adds rustfs_old_data_dir_* counters (attempted/reclaimed/leaked/below_quorum) as the operator-visible backstop for leaked residue. - Adds a test-only (#[cfg(test)]) delete fault-injection seam; in production it inlines to a no-op None and has no behavioral effect. Tests: pure-function A/C group + join-error mapping + actions decision; A5/A5b real-disk guard/reclaim integration; end-to-end overwrite returning 200 while old-data-dir cleanup fails. #864 rollback guard test remains green. * fix(ecstore): resolve merge conflicts with origin/main in io_primitives.rs --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: houseme <housemecn@gmail.com> |
||
|
Zhengchao An
|
ddf197ba57 |
fix(lock): refresh object write locks with a heartbeat (#4388)
fix(lock): renew distributed locks via heartbeat; wire server refresh (backlog#899) A held distributed write lock had a fixed 30s TTL and was never renewed, so any operation exceeding it got its per-node lease reclaimed and stolen by a contender, causing split-brain writes. This implements Phase 0 + Phase 1 of the #899 design (Phase 2 abort-on-loss is deferred and tracked in code comments). Phase 0 (server refresh wiring, P0): - node_service handle_refresh was a no-op stub that parsed args then always returned success=true. Extract a testable `refresh_lock` free function that actually delegates to the node's lock backend. Not-found maps to success=false with no error_info, so RemoteClient::refresh keeps its Ok(resp.success) semantics and yields a real not_found signal to the coordinator heartbeat. Without this, client heartbeats were silently no-ops. Phase 1 (client heartbeat + safe interval + observability): - DistributedLockGuard spawns a heartbeat that refreshes every per-client lease on a derived interval; interval is derived without Duration::clamp (entries<=1 or interval>=ttl => no spawn), fixing the sub-second-ttl panic. - Add LockLostSignal: declare the lock lost when not_found exceeds entries.len() - refresh_quorum; RPC errors are not counted (absorbed by the ttl > interval margin). Expose is_lock_lost()/lock_lost() for observers. - disarm(), release(), and Drop now abort the heartbeat before releasing so no refresh races the unlock (refresh only extends, never creates, a lease). - Reclaim path stays behaviorally unchanged but now warns with owner/resource/ lease age and records a metric (#698 scavenger preserved). - Add DEFAULT_LOCK_REFRESH_INTERVAL, LockRequest.refresh_interval (serde default for RPC back-compat) + builder, and lock lifecycle metrics. Open questions adopt the design's documented defaults (marked TODO in code): refresh not-found -> Ok(false)/error_info=None; lost-quorum base entries.len(); DEFAULT_LOCK_REFRESH_INTERVAL=10s. Tests: heartbeat keepalive/quorum-loss/jitter/boundary/disarm (lock crate), server refresh delegation (rustfs), and end-to-end survives-past-ttl plus crashed-owner-reclaim regressions (namespace). |
||
|
houseme
|
6f613317f6 |
feat(internode): optimize gRPC transport (#4337)
* feat(internode): P0 gRPC transport tuning, message limits, payload metrics Land the P0 subtask from docs/grpc-optimization: close the client-vs-server transport gaps and add instrumentation to size which unary RPCs need channel isolation in P1. Transport tuning (G3): the client `Endpoint` now disables Nagle and raises the HTTP/2 stream/connection flow-control windows to mirror the server socket, so small lock/health RPCs are not batched and larger metadata responses are not throttled by the 64KiB default window. All env-overridable, 0 opts out. Message-size limits (G1): both `NodeServiceClient` and `NodeServiceServer` set max decode/encode size (default 100MiB) instead of tonic's silent 4MiB cap, so a large multi-version xl.meta or aggregated ReadMultiple no longer fails out_of_range. The server limit is set on `NodeServiceServer` before wrapping in the auth `InterceptedService` (the interceptor type does not expose it). Payload instrumentation (P1 prep): ReadAll/ReadMultiple record a payload-size histogram plus a large-payload counter when a response crosses the configured threshold (default 8MiB), feeding alerting on paths that contend with latency-sensitive control-plane traffic on the shared channel. Threshold-only counter, no per-call hot-path log. Verification: cargo check/test on config, io-metrics, ecstore, rustfs; clippy clean on touched files; make pre-commit green. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(internode): P1 control/bulk gRPC channel isolation (opt-in) Land the P1 subtask from docs/grpc-optimization: physically separate large bytes-carrying unary RPCs from latency-sensitive control-plane RPCs so a big transfer can no longer head-of-line block a lock/health RPC on the shared HTTP/2 connection (G2/G5). Introduce ChannelClass { Control, Bulk } and get_channel_for_class in protos. Control RPCs keep the per-peer connection keyed by the bare address; Bulk RPCs (ReadAll/WriteAll/ReadMultiple/BatchReadVersion, via a new get_bulk_client) are round-robined across a small per-peer bulk pool. Rather than restructuring the global GLOBAL_CONN_MAP (and every consumer), bulk channels are cached under a composite key (addr\0bulk\0idx). The NUL separator cannot appear in a URL, so bulk keys never collide with the control key. This keeps the blast radius small on a consistency-sensitive path. create_new_channel is refactored into build_channel(dial_addr, cache_key) so several physically distinct channels to one peer cache independently while dialing/TLS still use the real address. Gated by RUSTFS_INTERNODE_CHANNEL_ISOLATION (default OFF) so the default build is byte-for-byte the pre-P1 behavior: bulk resolves to the control channel and the switch is a single-env rollback. RUSTFS_INTERNODE_BULK_CHANNELS (default 2, clamped >=1) sizes the pool. On failure, evict_failed_connection drops the whole bulk pool for the peer (round-robin hides which index was used), avoiding half-dead cached channels. Lock RPCs (remote_locker) already use the default Control path, so lock semantics and retry behavior are unchanged. Verification: cargo check/test on config, protos, ecstore, rustfs; new protos tests for bulk key routing and isolation-off passthrough; clippy clean on touched files; make pre-commit green. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(internode): P2 msgpack/JSON codec observability + encode buffer presizing Land the safe, wire-compatible slice of P2 from docs/grpc-optimization: the observability prerequisite for retiring the redundant JSON fields, plus a codec micro-optimization. No proto/wire-format change; JSON is still dual-written. Internode RPCs today dual-encode each metadata value as both msgpack (`*_bin`) and a JSON compatibility string, and decoders prefer `_bin` with a JSON fallback. Before the JSON fields can ever be dropped (a cross-version change), that fallback must be proven unused in production. Add rustfs_system_network_internode_msgpack_json_fallback_total{direction, message}: incremented whenever a decode falls back to the JSON field because the msgpack payload was absent. Wired into both directions — the client decoding peer responses (remote_disk.rs, incl. the list-level read_multiple/batch fallbacks) and the server decoding peer requests (node_service/disk.rs). This counter must read zero across a release window before send paths stop writing JSON and the proto text fields are reserved/removed (the deferred P2-1 steps). Also pre-size the msgpack encode buffers (Vec::with_capacity(512)) on both sides, eliminating the repeated growth reallocations for typical FileInfo payloads with zero added copy. Full thread_local buffer pooling is deferred: it needs either an extra copy (unclear net win) or a send-path buffer-return lifecycle, to be justified by a codec microbenchmark first. Verification: cargo check/test on io-metrics, ecstore, rustfs; new fallback counter smoke test; existing codec decode tests green; clippy clean on touched files; make pre-commit green. Co-Authored-By: heihutu <heihutu@gmail.com> * docs(internode): add msgpack/JSON convergence observation runbook Runbook driving the observation-gated retirement of the redundant JSON compatibility fields on internode gRPC metadata RPCs (grpc-optimization P2-1). Documents the shipped fallback counter (rustfs_system_network_internode_msgpack_json_fallback_total{direction,message}), the PromQL to confirm it reads zero across a release window, a standing alert, and the staged flip/rollback procedure (env-gated msgpack-only send, then proto field removal in N+1). Includes the verified field -> peer-decoder audit: only fields whose peer decodes _bin first may be converged. Notes DeleteVersion.opts (DeleteOptions) is NOT convergence-ready — its server handler is not _bin-first and must gain a decode_msgpack_or_json path first. This gates the send-side change so it cannot empty a JSON field an old peer still needs. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(internode): env-gated msgpack-only send + DeleteVersion _bin support (P2-1) Implements the send-side lever for retiring the redundant JSON compatibility fields on internode gRPC metadata RPCs, plus the missing `_bin` support on the delete path that it depends on (grpc-optimization P2-1). Default-off: the base build is byte-for-byte the prior dual-write behavior. Gated msgpack-only send (RUSTFS_INTERNODE_RPC_MSGPACK_ONLY, default false): - New rustfs_protos::internode_rpc_msgpack_only() reads the flag. - Client (remote_disk.rs) compat_json() and server (node_service/disk.rs) compat_response_json() emit an empty JSON string when the flag is on, so only the msgpack _bin payload is sent. The _bin field is always sent; decoders keep the JSON read fallback. Applied only to fields with a confirmed _bin-first peer decoder (WriteMetadata/UpdateMetadata/RenameData file_info, UpdateMetadata opts, ReadOptions, ReadMultipleReq, BatchReadVersionReq; ReadVersion/ReadXL/RenameData responses and the ReadMultiple/BatchReadVersion response lists). - Only enable after the P2 fallback counter has read zero across a release window (see docs/operations/internode-msgpack-json-convergence-runbook.md). Single-env rollback; no wire-format break. DeleteVersion(s) _bin support (prerequisite): - The DeleteVersion/DeleteVersions protos had NO _bin fields. Add additive (backward-compatible) bytes file_info_bin/opts_bin (DeleteVersion) and repeated bytes versions_bin + bytes opts_bin (DeleteVersions); regenerate the checked-in prost struct. - Client dual-writes them; server decodes them _bin-first with JSON fallback. - These delete fields are kept OUT of the msgpack-only set (always dual-write) until their own fallback counter reads zero across a window with the new decoders fully deployed. DeleteVersion.raw_file_info stays JSON-only (no _bin field yet). Verification: cargo check/test on protos, config, ecstore, rustfs (incl. the six delete request handler tests and a compat_json default-path test); clippy clean on touched files; make pre-commit green. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(internode): P3 cluster peer online/offline health metric Land the safe observability core of P3 (grpc-optimization G6/G8): track each internode peer's reachability and expose the offline count, for parity with MinIO's minio_cluster_servers_offline_total. Pure instrumentation — peer selection and quorum are unchanged. - io-metrics: per-peer PeerHealthState { online, consecutive_failures } registry plus record_peer_reachable/record_peer_unreachable. A peer flips offline after N consecutive failures (dial failures or RPC-triggered evictions) and back online on the next successful dial; the count of offline peers is published to the rustfs_cluster_servers_offline_total gauge. - config: RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD (default 3, clamped >= 1). - protos: build_channel marks the peer reachable on a successful dial and unreachable on a dial failure; evict_failed_connection feeds the failure signal too. Keyed by the real peer address, so control and bulk channels to one peer share health state. Deferred (documented in docs/grpc-optimization P3): startup prewarm (no clean topology-ready hook yet), the offline fast-bypass in peer routing (consistency- sensitive; must not change quorum), and idempotent-read-only retry. This commit is observability only. Verification: cargo check/test on io-metrics, config, protos (new peer-health state-machine and threshold-clamp tests); clippy clean on touched files; make pre-commit green. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(internode): P3 control-channel prewarm + self-healing offline bypass Add the remaining P3 connection-lifecycle levers (grpc-optimization G6/G8), both env-gated and default-off so the base build is unchanged. Prewarm (RUSTFS_INTERNODE_PREWARM, default off): RemoteDisk::new spawns a best-effort background dial of the peer's control channel, deduped per peer address, moving the connect cost off the first RPC. Failures fall through to the existing lazy connect + recovery monitor. Offline bypass (RUSTFS_INTERNODE_OFFLINE_BYPASS, default off): remote_disk get_client/get_bulk_client fast-fail a peer already marked offline instead of paying the connect timeout, so the erasure layer proceeds on quorum sooner. This does NOT change quorum. It is self-healing: cluster_peer_should_bypass lets one request per RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS (default 5s) through to recover the peer even with no background monitor, and the recovery monitor's own probe path calls the client directly so it is never bypassed. io-metrics gains cluster_peer_is_offline / cluster_peer_should_bypass (with a per-peer re-probe timestamp). Scope: data path only — remote_locker (lock RPCs, most consistency-sensitive) is left dual-writing/unbypassed as a follow-up. Verification: cargo check/test on io-metrics, config, ecstore (new self-healing bypass tests; all 105 rpc tests green); clippy clean on touched files; make pre-commit green. Co-Authored-By: heihutu <heihutu@gmail.com> * docs(internode): add A/B benchmark runbook for gRPC optimization stages Reproducible before/after collection procedure for grpc-optimization P0–P3. Since every stage is env-gated, before/after is the same binary with different env — no rebuild. Documents, per stage: the exact env toggles (baseline vs enabled column), which existing bench script to run (run_internode_transport_baseline.sh / run_four_node_cluster_failover_bench.sh), the Prometheus metrics to capture, and the acceptance gates from the design docs (e.g. lock p99 down >= 20% for P1, msgpack fallback counter = 0 before enabling P2, correct rustfs_cluster_servers_offline_total for P3). Live runs require a multi-node cluster + load tool + Prometheus scrape and cannot be produced in a single-process sandbox; artifacts land under target/bench (gitignored) and attach to the PR. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(internode): P3-2 lock-path offline bypass + P3-3 idempotent read retry Extend the offline bypass to the lock path and add opt-in retries for idempotent reads (grpc-optimization P3-2/P3-3). Both env-gated and default-off/zero. Offline bypass (lock path): factor the bypass decision into a shared pub(crate) internode_offline_bypass_reason(addr) and call it from remote_locker::get_client too, so lock RPCs to an offline peer fast-fail (letting dsync reach quorum sooner) instead of paying the connect timeout. Does not change quorum; the self-healing re-probe keeps peers recoverable. Gated by RUSTFS_INTERNODE_OFFLINE_BYPASS (default off). Idempotent read retry (P3-3): add execute_read_with_retry — a bounded, exponential-backoff retry for read-only/reentrant RPCs on transient network errors — and route disk_info through it. RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES defaults to 0 (disabled). Write/lock RPCs are never retried (quorum/idempotency safety, per CLAUDE.md); the wrapper requires an Fn closure so only reads that rebuild their request from borrowed inputs qualify. Deferred: grpc.health.v1 (optional ecosystem-compat only; needs a new tonic-health dep and 3-way hybrid-service wiring — internal needs are met by the existing Ping RPC). Verification: cargo check/test on config, ecstore (105 rpc tests green incl. disk_info now via the retry wrapper); clippy clean on touched files; make pre-commit green. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(scripts): one-click internode gRPC A/B benchmark driver Wrap the per-stage env matrix from the benchmark runbook into scripts/run_internode_grpc_ab_bench.sh: given --stage <p0|p1|p2|p3> and --phase <before|after>, it emits the stage/phase RUSTFS_INTERNODE_* server env to <out-dir>/server-env.sh and runs the right underlying bench (run_internode_transport_baseline.sh for p0/p1/p2, run_four_node_cluster_failover_bench.sh for p3) into a labeled target/bench/internode-transport/<stage>-<phase>/. Passthrough args after `--` reach the underlying bench; --dry-run previews the env + command. The script is explicit that RUSTFS_INTERNODE_* are server env, so for the load-driven stages the operator must restart rustfs with the emitted env before the run; the docker four-node (p3) path exports them for a forwarding compose. shellcheck-clean. Runbook updated with a "One-click driver" section. Co-Authored-By: heihutu <heihutu@gmail.com> * chore(compose): forward RUSTFS_INTERNODE_* into the four-node cluster The four-node local-build compose only forwarded a fixed whitelist of env, so the internode gRPC knobs (grpc-optimization P0-P3) never reached the containers and the A/B bench driver's "after" phase was a no-op. Forward the full RUSTFS_INTERNODE_* set with defaults matching the binary defaults, so leaving them unset is a no-op and the A/B driver can toggle a stage per phase. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(internode): address Copilot review — retry health action + poison-safe peer health Two review nits on #4337: - P3-3 idempotent read retry (remote_disk.rs): execute_read_with_retry ran every attempt through execute_with_timeout_for_op, which hardcodes FailureHealthAction::MarkFailure. So the first transient error could flip the disk faulty and short-circuit the remaining retries, and each attempt over-counted the failure. Route all but the final attempt through execute_with_timeout_for_op_and_health_action with IgnoreFailure; only the last attempt marks faulty/evicts. No default impact (retries default 0). - Peer-health helpers (io-metrics): record_peer_reachable/record_peer_unreachable, cluster_peer_is_offline and cluster_peer_should_bypass early-returned on a poisoned mutex, permanently stalling the offline gauge and bypass state after a single panic. Recover via PoisonError::into_inner(). Verification: cargo check/test on io-metrics + ecstore (105 rpc tests green); clippy clean on touched files; make pre-commit green. Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
25193ee6cf |
feat(listobjects): add guarded metadata-fast listing (#4311)
* feat(listobjects): add phase 0 observability metrics * perf(listobjects): reduce gather metadata decoding * test(listobjects): add adversarial listing coverage * feat(listobjects): add source-aware cursor fields * feat(listobjects): define index source modes * feat(listobjects): model index rebuild health * feat(listobjects): add opt-in index fallback gate * feat(listobjects): verify index candidate pages * docs(listobjects): add rollout runbook * docs(listobjects): untrack baseline fixtures * feat(listobjects): add opt-in key-only provider * feat(listobjects): add index verification metrics * perf(listobjects): gate list metrics on get stage switch * feat(listobjects): bind provider health generation * feat(listobjects): add persistent key-only provider * feat(listobjects): bind persistent provider lifecycle * feat(listobjects): track persistent index mutations * feat(listobjects): persist index mutation checkpoints * fix(listobjects): aggregate benchmark metric snapshots * feat(listobjects): store journal in system namespace * chore(listobjects): report fallback reasons in bench * test(listobjects): verify metadata-fast stale fallbacks * feat(listobjects): serve metadata-fast snapshots behind guardrails * test(listobjects): add metadata-fast chaos bench * fix(listobjects): attribute metadata-fast fallback metrics * test(listobjects): add metadata-fast fallback chaos probes * test(listobjects): add quorum journal chaos guardrails * fix(listobjects): satisfy pre-pr clippy gate * docs(listobjects): untrack rollout runbook * fix(ecstore): remove redundant heal error conversion * test(filemeta): satisfy replication info clippy ---- Co-Authored-By: heihutu <heihutu@gmail.com> |
||
|
houseme
|
db277b17a4 |
fix: harden GET object performance paths (#4271)
* fix: harden GET object performance paths * fix: satisfy GET multipart layer guard * fix: keep v1 list markers S3 compatible * perf: tighten GET direct-memory decision * ci: isolate s3tests from scanner workload * refactor: simplify get object body lifecycle * fix: satisfy get object clippy |
||
|
houseme
|
25d80d7c60 |
feat(storage): harden internode data-path controls (#4224)
* fix(rio): propagate http writer shutdown errors * fix(ecstore): unify remote lock rpc deadlines * fix(storage): reject corrupt read multiple payloads * feat(rio): add internode http tuning profiles * feat(metrics): add internode baseline signals * feat(ecstore): observe shard locality topology * feat(ecstore): gate shard locality scheduling * feat(ecstore): gate batch read version rpc * feat(ecstore): observe batch processor adaptation * feat(ecstore): gate batch processor observation * docs: add get benchmark regression analysis * docs: add issue 797 execution plan status * fix(ecstore): require explicit batch rpc support * fix(ecstore): honor documented batch read gate * fix(ecstore): keep batch read gate stable per call * chore: update workspace dependencies * feat(ecstore): log batch read gate decisions * feat(ecstore): count batch read gate decisions * test(issue-797): add local internode A/B runner * test(rio): fix tuning profile spelling fixture * fix(protocols): adapt sftp channel open callbacks * fix(metrics): wrap batch processor observation args * chore(docs): keep issue notes local only * fix(storage): address internode review feedback * fix(storage): address internode data-path review findings - Run the BatchReadVersion auto-mode unary fallback outside the batch RPC deadline so each read_version keeps its own per-op timeout and health accounting instead of racing the whole batch against one drive timeout. - Cap adaptive batch-processor concurrency growth at a hard multiple of the configured baseline so sustained fast batches cannot ratchet past the configured limit. - Parse RUSTFS_INTERNODE_HTTP_* tuning, RUSTFS_BATCH_PROCESSOR_ADAPTIVE, and RUSTFS_METADATA_BATCH_READ once per process instead of re-reading the environment on hot paths. - Skip shard read-cost collection in observe mode when stage metrics are disabled, and cache the local endpoint host list instead of rebuilding it on every read. - Allow --warp-extra-args values starting with -- and drop the unused warp_hosts_csv helper in the issue-797 A/B runner. * fix(storage): address internode data-path review findings - Run the BatchReadVersion auto-mode unary fallback outside the batch RPC deadline so each read_version keeps its own per-op timeout and health accounting instead of racing the whole batch against one drive timeout. - Cap adaptive batch-processor concurrency growth at a hard multiple of the configured baseline so sustained fast batches cannot ratchet past the configured limit. - Parse RUSTFS_INTERNODE_HTTP_* tuning, RUSTFS_BATCH_PROCESSOR_ADAPTIVE, and RUSTFS_METADATA_BATCH_READ once per process instead of re-reading the environment on hot paths. - Skip shard read-cost collection in observe mode when stage metrics are disabled, and cache the local endpoint host list instead of rebuilding it on every read. - Allow --warp-extra-args values starting with -- and drop the unused warp_hosts_csv helper in the issue-797 A/B runner. Co-Authored-By: heihutu<heihutu@gmail.com> * fix(storage): align buffer clamp test with media cap * fix(ecstore): release optimized read locks before streaming --------- Co-authored-by: Zhengchao An <anzhengchao@gmail.com> |
||
|
houseme
|
7a075c91da | perf: avoid eager parity reader setup (#4133) | ||
|
houseme
|
b16120dbcc | feat: optimize small GET read paths (#4022) | ||
|
Zhengchao An
|
74c95920ca | fix: dual-write renamed IO metrics (#4077) | ||
|
houseme
|
46d7f9e1f2 |
feat(get): harden codec streaming rollout (#3981)
* feat(get): consolidate GET performance optimization Consolidated implementation of all GET performance optimizations into a single, well-organized commit replacing the previous patch-on-patch approach. ## Changes ### Configuration (set_disk/mod.rs) - Consolidated all GET optimization flags into a single organized section - Enabled by default: codec streaming, metadata early-stop, page cache reclaim - Added codec streaming multipart flag (default: disabled) - Added version-aware early-stop flag (default: disabled) - Added adaptive duplex buffer sizing based on object size - All flags use OnceLock caching with rollout percentage support ### Metadata Early-Stop (set_disk/read.rs) - Delete marker early-stop when quorum agrees - Version-aware early-stop for versioned GET requests - MetadataQuorumAccumulator enhanced with: - delete_marker_votes tracking - requested_version_id and matching_version_votes tracking - version_early_stop_decision() method - 6 new tests for version early-stop scenarios ### Codec Streaming (erasure/coding/decode_reader.rs) - DualInFlight (2-stripe lookahead) enabled by default ### Decode Pipeline (erasure/coding/decode.rs) - Stripe prefetch count configuration - Bitrot-decode overlap configuration ### Disk Layer (disk/local.rs) - O_DIRECT read configuration constants (preparation) ### Metrics (io-metrics/lib.rs) - BytesPool acquisition/return metrics - Metadata phase duration with early-stop label - Total duration with reader_path label ### Diagnostics (diagnostics/) - Early-stop reason constants - Pool tier/outcome label constants ### Observability (.docker/observability/) - 3 Grafana dashboards for GET optimization monitoring - Prometheus alert rules (6 alerts: 3 critical, 3 warning) - Updated README.md and README_ZH.md with usage docs ### Config (config/src/constants/runtime.rs) - Page cache reclaim read enabled by default ## Environment Variables | Variable | Default | Description | |----------|---------|-------------| | RUSTFS_GET_CODEC_STREAMING_ENABLE | true | Codec streaming base flag | | RUSTFS_GET_CODEC_STREAMING_ROLLOUT_PCT | 100 | Codec streaming rollout % | | RUSTFS_GET_CODEC_STREAMING_MULTIPART_ENABLE | false | Multipart codec streaming | | RUSTFS_GET_METADATA_EARLY_STOP_ENABLE | true | Early-stop base flag | | RUSTFS_GET_METADATA_EARLY_STOP_ROLLOUT_PCT | 100 | Early-stop rollout % | | RUSTFS_GET_METADATA_VERSION_EARLY_STOP_ENABLE | false | Version-aware early-stop | | RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE | true | Page cache reclaim | | RUSTFS_OBJECT_DIRECT_IO_READ_ENABLE | false | O_DIRECT (preparation) | | RUSTFS_GET_DECODE_STRIPE_PREFETCH_COUNT | 1 | Stripe prefetch | | RUSTFS_GET_BITROT_DECODE_OVERLAP_ENABLE | false | Bitrot-decode overlap | | RUSTFS_GET_CODEC_STREAMING_MAX_INFLIGHT | 2 | DualInFlight stripes | ## Rollback All optimizations can be disabled via environment variables: RUSTFS_GET_CODEC_STREAMING_ENABLE=false RUSTFS_GET_METADATA_EARLY_STOP_ENABLE=false RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE=false Co-Authored-By: heihutu <heihutu@gmail.com> * test(get): add stress test scripts for GET optimization validation - quick-validate-get-optimization.sh: Quick 5-minute validation - stress-test-get-optimization.sh: Full 30+ minute stress test - README-stress-test.md: Usage documentation Co-Authored-By: heihutu <heihutu@gmail.com> * test(ecstore): align file cache reclaim defaults * chore(deps): update redis and erasure codec * test(ecstore): align decode fill policy default * fix(get): wire codec streaming rollout gate * perf(get): skip metrics-off codec timers * test(get): capture codec streaming diagnostics * test(get): add multipart fallback probe * test(get): add encrypted fallback probe * test(get): add compressed fallback probe * test(get): add degraded read fallback probe * test(get): cover remote fallback probe * test(get): report warp request p99 * test(get): capture OTLP metric deltas * perf(get): align codec streaming inflight default * perf(get): reuse codec reader output buffers * test(get): count codec reader fill starts * perf(get): reuse codec reader fill worker * perf(get): lazy init rustfs codec reconstruct * test(get): cover rustfs codec source faults * docs(get): record rustfs codec fallback scope * feat(get): add multipart codec reader opt-in * test(get): add multipart codec smoke option * test(get): cover multipart codec degraded fallback * perf(get): bound multipart codec eager setup * test(get): satisfy codec hardening PR gate --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
27468ebfa9 |
feat(get): consolidate GET performance optimization (#3972)
* feat(get): consolidate GET performance optimization Consolidated implementation of all GET performance optimizations into a single, well-organized commit replacing the previous patch-on-patch approach. ## Changes ### Configuration (set_disk/mod.rs) - Consolidated all GET optimization flags into a single organized section - Enabled by default: codec streaming, metadata early-stop, page cache reclaim - Added codec streaming multipart flag (default: disabled) - Added version-aware early-stop flag (default: disabled) - Added adaptive duplex buffer sizing based on object size - All flags use OnceLock caching with rollout percentage support ### Metadata Early-Stop (set_disk/read.rs) - Delete marker early-stop when quorum agrees - Version-aware early-stop for versioned GET requests - MetadataQuorumAccumulator enhanced with: - delete_marker_votes tracking - requested_version_id and matching_version_votes tracking - version_early_stop_decision() method - 6 new tests for version early-stop scenarios ### Codec Streaming (erasure/coding/decode_reader.rs) - DualInFlight (2-stripe lookahead) enabled by default ### Decode Pipeline (erasure/coding/decode.rs) - Stripe prefetch count configuration - Bitrot-decode overlap configuration ### Disk Layer (disk/local.rs) - O_DIRECT read configuration constants (preparation) ### Metrics (io-metrics/lib.rs) - BytesPool acquisition/return metrics - Metadata phase duration with early-stop label - Total duration with reader_path label ### Diagnostics (diagnostics/) - Early-stop reason constants - Pool tier/outcome label constants ### Observability (.docker/observability/) - 3 Grafana dashboards for GET optimization monitoring - Prometheus alert rules (6 alerts: 3 critical, 3 warning) - Updated README.md and README_ZH.md with usage docs ### Config (config/src/constants/runtime.rs) - Page cache reclaim read enabled by default ## Environment Variables | Variable | Default | Description | |----------|---------|-------------| | RUSTFS_GET_CODEC_STREAMING_ENABLE | true | Codec streaming base flag | | RUSTFS_GET_CODEC_STREAMING_ROLLOUT_PCT | 100 | Codec streaming rollout % | | RUSTFS_GET_CODEC_STREAMING_MULTIPART_ENABLE | false | Multipart codec streaming | | RUSTFS_GET_METADATA_EARLY_STOP_ENABLE | true | Early-stop base flag | | RUSTFS_GET_METADATA_EARLY_STOP_ROLLOUT_PCT | 100 | Early-stop rollout % | | RUSTFS_GET_METADATA_VERSION_EARLY_STOP_ENABLE | false | Version-aware early-stop | | RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE | true | Page cache reclaim | | RUSTFS_OBJECT_DIRECT_IO_READ_ENABLE | false | O_DIRECT (preparation) | | RUSTFS_GET_DECODE_STRIPE_PREFETCH_COUNT | 1 | Stripe prefetch | | RUSTFS_GET_BITROT_DECODE_OVERLAP_ENABLE | false | Bitrot-decode overlap | | RUSTFS_GET_CODEC_STREAMING_MAX_INFLIGHT | 2 | DualInFlight stripes | ## Rollback All optimizations can be disabled via environment variables: RUSTFS_GET_CODEC_STREAMING_ENABLE=false RUSTFS_GET_METADATA_EARLY_STOP_ENABLE=false RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE=false Co-Authored-By: heihutu <heihutu@gmail.com> * test(get): add stress test scripts for GET optimization validation - quick-validate-get-optimization.sh: Quick 5-minute validation - stress-test-get-optimization.sh: Full 30+ minute stress test - README-stress-test.md: Usage documentation Co-Authored-By: heihutu <heihutu@gmail.com> * test(ecstore): align file cache reclaim defaults * chore(deps): update redis and erasure codec * test(ecstore): align decode fill policy default * test(ecstore): align metadata early-stop default * fix(ecstore): keep metadata early stop opt-in --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
bf03ff2869 |
feat(get): add limited opt-in rollout gates (#3963)
* feat(bench): harden cooled get ab harness * feat(get): add limited opt-in rollout gates * fix(get): tighten rollout gate fallbacks |
||
|
houseme
|
20f56af09c | feat(get): tune output response handoff (#3956) | ||
|
houseme
|
de86025f2c | feat(get): overlap read verify decode (#3945) | ||
|
houseme
|
8994a8ff00 |
feat(get): add local-first shard reads (#3924)
* feat(get): observe shard locality cost * feat(get): add local-first shard reads ---- Co-Authored-By: Heihutu <heihutu@gmail.com> |
||
|
houseme
|
c0ddc14bb8 |
feat(get): observe shard locality cost (#3922)
* feat(get): observe shard locality cost * fix(storage): avoid request counter underflow panic * fix(storage): import put guard in concurrency tests |
||
|
houseme
|
2c04807f05 |
feat(get): add guarded metadata early stop (#3916)
* feat(get): add v2 metrics compatibility harness * feat(get): observe metadata fanout quorum (#3915) * feat(get): observe metadata fanout quorum * fix(app): use storage ECStore type in app boundary * feat(get): add guarded metadata early stop |
||
|
houseme
|
5a78a9c416 |
feat(get): add v2 metrics compatibility harness (#3913)
* feat(get): add v2 metrics compatibility harness * feat(get): observe metadata fanout quorum (#3915) * feat(get): observe metadata fanout quorum * fix(app): use storage ECStore type in app boundary ---- Co-Authored-By: heihutu <heihutu@gmail.com> |
||
|
houseme
|
0321e4c6ca | perf(get): reduce metrics and streaming handoff overhead (#3907) | ||
|
houseme
|
7bf411f465 | obs(get): add GET read pipeline metrics (#3868) | ||
|
houseme
|
96b1f5c373 |
feat(storage): optimize gt1g get read path (#3860)
* feat(storage): tune gt1g get read path * docs(ops): add gt1g get benchmark guide * feat(scripts): add gt1g get fallback runner * test(storage): trim gt1g get helper warnings * test(storage): reduce gt1g get helper type complexity |
||
|
houseme
|
efebcb66be | feat(perf): add large PUT tuning and encode optimization (#3816) |