mirror of
https://github.com/rustfs/rustfs.git
synced 2026-07-26 16:28:15 +00:00
copilot/research-github-issue-5076
196 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 houseme
|
7805cf5ae6 |
fix(cluster): clarify peer health and listing timeouts (#5086)
* fix(cluster): surface observed peer health Refs rustfs/backlog#1387 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(admin): distinguish unreported peer health Refs rustfs/backlog#1388 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): decouple metacache peek timeout Refs rustfs/backlog#1389 Co-Authored-By: heihutu <heihutu@gmail.com> * docs(ops): diagnose metacache listing timeouts Refs rustfs/backlog#1390 Co-Authored-By: heihutu <heihutu@gmail.com> * refactor(cluster): clarify observed peer health Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): route capability state through contract Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
Zhengchao An
|
40c089f31b |
feat(server): add opt-in global connection cap to the S3 accept loop (#4957)
Follow-up to backlog#1191 (optional sub-item). The main accept loop spawned one task per socket with no global bound, so a connection flood could exhaust file descriptors and memory and take existing traffic down with it. - New RUSTFS_API_MAX_CONNECTIONS (default 0 = unlimited, no semaphore constructed, accept loop unchanged). - When set, the loop acquires an owned semaphore permit BEFORE accept(): at saturation it simply stops accepting and lets the kernel backlog absorb bursts (TCP-native backpressure) instead of accept-then-close churn. The permit moves into the connection task and is released by RAII on any exit path, including TLS handshake failures. - Saturation is observable via the rustfs_http_server_connection_cap_saturated_total counter, a connection_cap_state startup event, and a per-wait debug event; shutdown stays responsive while parked on the semaphore. - The cap covers everything on the main listener (S3, admin, console, internode gRPC); the constant docs carry sizing guidance. - E2E coverage: ten sequential Connection-close requests against cap 2 prove permits never leak; two stalled connections saturating cap 2 leave a third unserved until their permits are released, after which the queued request is accepted and answered. |
||
|
Zhengchao An
|
97edb2e5cf |
feat(api): add opt-in per-bucket dimension to the S3 API rate limiter (#4949)
Follow-up to #4895 (backlog#1191 deferred sub-item). The client-IP dimension gives per-client fairness; this adds a collective per-bucket budget so one hot bucket cannot monopolize the server regardless of how many client IPs the traffic is spread across. - Generalize RateLimiter over its key type with a Borrow-based check() so &str lookups against String bucket keys stay allocation-free on the hit path; client-IP call sites are unchanged in behavior. - RateLimitLayer now carries optional client and bucket limiters; a request must pass every configured dimension, and rejections report which one tripped via a new 'dimension' metric label. - Bucket extraction mirrors s3s host routing: virtual-hosted-style resolves the Host/authority prefix against the same expanded domain set (with port variants) the s3s router uses; otherwise the first path segment. Admin and table-catalog namespaces are never buckets. - New env vars RUSTFS_API_RATE_LIMIT_BUCKET_RPM/_BURST (default 0 = dimension off) under the existing enable switch; bucket-only configurations (client RPM 0) are supported. - Bucket names are attacker-chosen, so the bounded-shards design (100k keys, lossless idle sweeps, most-idle eviction) is the memory defense; a test floods 10k random names and asserts the cap holds. - Unit tests for extraction, shared bucket budgets across IPs, both- dimensions interaction, and the extended env matrix; e2e test proves bucket-only throttling on the real server with an unrelated bucket unaffected. |
||
|
Zhengchao An
|
78469aa63b |
fix(get): bound GET disk-read admission with a hard cap instead of unbounded permit bypass (#4935)
Refs https://github.com/rustfs/backlog/issues/1317 Previously, when the primary disk-read permit pool stayed saturated past `RUSTFS_OBJECT_DISK_PERMIT_WAIT_TIMEOUT` (default 5s) — which a handful of slow clients can cause because a permit is held for the whole body transfer — the GET path set `disk_permit = None` and continued reading with no admission token at all. That made the disk-read concurrency limit unbounded under exactly the overload it is meant to protect against: any number of GETs could pile onto the disks simultaneously. This replaces the permit-less bypass with a bounded overflow lane and a hard cap: - A new bounded degraded semaphore (`RUSTFS_OBJECT_DISK_DEGRADED_READ_CAP`, default mirrors the primary cap) is consulted only after the primary wait times out. A GET takes one degraded permit without blocking, or is rejected with `SlowDown`/503 once that lane is also full. The total number of GETs performing disk-active reads is therefore hard-capped at `primary_cap + degraded_cap`, and no GET ever reads without holding an admission token. - Admission is centralized in `ConcurrencyManager::admit_disk_read`, returning `Primary`/`Degraded`/`Unbounded`/`Rejected`. `Degraded` and `Rejected` are counted in metrics (`rustfs.get_object.disk_permit.degraded.total`, `rustfs.get_object.disk_permit.hard_reject.total`), replacing the removed `rustfs.get_object.disk_permit.bypass.total`. - A primary cap of `0` (disk-read throttling disabled) is preserved as the only intentional permit-less path via `Unbounded`, so that degenerate-but-served configuration is not turned into all-503. The `primary_wait == 0` wait-forever opt-out is likewise unchanged. Healthy GETs are unaffected: concurrency at or below the primary cap is admitted immediately from the primary pool exactly as before, with no new latency or rejection. The rejection is surfaced before response headers are constructed, so it is a clean pre-header 503, never a post-header 504 masquerade. Degraded-lane GETs use the identical streaming reader and forward-progress stall timeout as primary GETs, so a slow but progressing large download is not killed. Owned permits (primary or degraded) are held by `DiskReadPermitReader` and released on body EOF or client drop/cancel, so tokens are always returned. Body stall timeout already resets on forward progress and post-header failures already surface as body errors, so no timeout-split changes were needed here. Scope: this PR implements the minimal safe correctness core (eliminate unbounded bypass + hard cap + SlowDown). The two-level weighted-fair + aging scheduler (small setup cap feeding a size-aware data-producer fair queue) and the strictly-bounded producer/client buffer with cancellation propagation from the issue plan remain follow-ups. The black-box slow-client soak is deferred to the unbuilt facility in https://github.com/rustfs/backlog/issues/1325 rather than faked. Tests (white-box, virtual clock via `start_paused`): degrade-then-hard-reject with token release; 100 concurrent GETs against primary=1/degraded=1 never exceed 2 simultaneous admissions with every request either admitted or explicitly rejected; disabled cap serves unbounded; zero-wait blocks on the primary lane. Reverting the hard cap makes the concurrency invariant fail. |
||
|
Zhengchao An
|
3674f5f56e |
fix(ecstore): bound remote shard writers with a progress deadline so one black-hole peer cannot pin write quorum (#4925)
A PUT that fans out erasure shards to remote peers awaited every shard writer to completion on both the per-block write and the final shutdown, and the remote HttpWriter had no progress deadline. A peer that accepts the TCP connection but never drains the request body (or never sends a response) therefore wedges the writer forever once the bounded buffers fill, pinning an otherwise-healthy write quorum indefinitely — a cluster-level write-availability hazard triggered by a single bad peer (rustfs/backlog#1319, https://github.com/rustfs/backlog/issues/1319). MultiWriter now wraps each shard write and each shard-writer shutdown in a forward-progress deadline. The budget is re-armed on every block, so it bounds a stall rather than the total transfer time of a large object: a slow-but-honest writer that keeps completing shards is never killed, while a writer that makes no progress within the budget is failed and its disk dropped before commit. An optional absolute per-object cap (disabled by default) backstops a slow-drip peer that dribbles just enough progress to reset the per-block timer without ever converging; it is off by default so a legitimate large upload over a slow link is not killed on total time alone. Both knobs come from RUSTFS_OBJECT_DISK_WRITE_STALL_TIMEOUT (default 30s) and RUSTFS_OBJECT_DISK_WRITE_ABSOLUTE_CAP (default 0 = disabled); setting the stall timeout to 0 restores the previous wait-forever behavior for a conservative rollback. The deadline enforcement lives in MultiWriter (writer-agnostic), so it covers local and remote writers alike and keeps the existing control-flow shape: a timed-out shard is marked failed (Error::Timeout, which is not an ignored error) and excluded from the write quorum exactly like any other shard write failure, and the unchanged nil_count/quorum check then continues on quorum or fails cleanly. This deliberately stays out of the MultiWriter lifecycle / commit-coordinator territory owned by rustfs/backlog#1312. When a stalled writer is dropped to fail its shard, the remote HttpWriter must stop holding the connection and its buffered body. HttpWriter previously left its spawned request task running on drop; it now aborts that background task in Drop (it is no longer pin-projected, since every field is Unpin and the AsyncWrite impl already used get_mut). Bytes already handed to the transport cannot be unsent, but they land only in this upload's unique tmp path and are reclaimed by tmp GC — they never touch a committed object. Tests, all on a paused virtual clock so they are deterministic and non-flaky: - one black-hole writer still meets a 3/4 write quorum without hanging; two black holes fail the quorum cleanly (both for the per-block write and the shutdown paths). - a slow-but-honest writer that keeps making progress within the stall budget is never failed across many blocks. - the absolute cap bounds a slow-drip writer within a finite budget while the healthy writers keep quorum. - the default policy is armed by default and honors 0 as disabled. - HttpWriter aborts its background request task on drop against a hanging peer. The toxiproxy/black-hole 4x4 end-to-end acceptance depends on black-box test facilities from rustfs/backlog#1325, which are not built yet; that acceptance is deferred to #1325 and intentionally not faked here. |
||
|
Zhengchao An
|
c82ee6be58 |
feat(api): wire opt-in per-client S3 API rate limiting (429 + Retry-After) (#4895)
feat(api): wire opt-in per-client S3 API rate limiting (backlog#1191) RustFS shipped three rate-limiter implementations and none was wired to any request path: the tower layer never returned 429 (its over-limit branch passed requests through) and was never instantiated, the console env switches only logged, and the Swift token bucket was never called. Replace them with one working, default-off implementation: - Rewrite rustfs/src/server/rate_limit.rs as a sharded per-client-IP token-bucket limiter (32 mutex shards instead of one global RwLock write per request), bounded at 100k tracked IPs with lossless refilled-idle sweeps, returning 429 + Retry-After + x-ratelimit-* headers and an S3-style XML body. - Key on trusted-proxy-validated ClientInfo.real_ip, else the socket peer address; never read spoofable X-Forwarded-For/X-Real-IP headers. Requests without a resolvable identity fail open. The echoed request id is charset-gated to prevent reflected XML injection. - Wire the layer once at startup via option_layer between CatchPanicLayer and ReadinessGateLayer (external stack only), gated by new RUSTFS_API_RATE_LIMIT_ENABLE/_RPM/_BURST constants; health and profiling probes, internode RPC/gRPC, and the console are exempt. - Make RUSTFS_CONSOLE_RATE_LIMIT_ENABLE/_RPM actually enforce by reusing the same limiter core through an axum middleware. - Delete the dead Swift ratelimit module, its isolated tests, and the stale logging-guardrail entry; keep the live SwiftError 429 mapping. - Add unit tests (exhaustion/recovery with injected time, concurrency, cap eviction, spoofed-header and fail-open behavior, env matrix) and e2e tests proving 429 + Retry-After on the real server and zero behavior change with default configuration. |
||
|
houseme
|
f3a7a4b0da |
chore(deps): localize workspace dependency features (#4888)
Move workspace-level dependency feature lists into the member crates that consume each dependency while keeping required default-features flags at the workspace root. Also refresh starshard to 2.2.2 via cargo update and cargo upgrade --exclude ratelimit. Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
escapecode
|
a80699b6dd |
feat: add an opt-in NATS JetStream publish path for the notify and audit targets (#4634)
feat(targets): add an opt-in NATS JetStream publish path for the notify and audit targets The NATS notify and audit targets publish through NATS Core, which returns before the server has durably accepted the message. A broker restart or a connection drop between the publish and the flush loses the event, even though the send queue has already cleared it, and no acknowledgement gates that clear. An opt-in JetStream publish path clears a queued event only after the server returns a durable PublishAck, so delivery is at-least-once across a broker restart or a reconnect. It applies to both the notify and audit NATS targets, is off by default, and is byte-identical to the NATS Core path when disabled. The path includes durable store-and-forward, a stable dedup id sent as the Nats-Msg-Id header so a replayed event is collapsed by the stream duplicate window, pre-flight stream validation, and a bounded failed-events store for terminally-failed and retry-exhausted events. Three configuration keys per target select it: JETSTREAM_ENABLE, JETSTREAM_STREAM_NAME, and JETSTREAM_ACK_TIMEOUT_SECS, under the RUSTFS_NOTIFY_NATS_ and RUSTFS_AUDIT_NATS_ prefixes. The on-disk batch filename separator changes from colon to underscore so batch names are valid on Windows filesystems, with transparent read-back of files written under the previous separator. The migration affects the shared queue store for every target type and lands with this feature because the store gains its first Windows-exercised paths here. Co-authored-by: houseme <housemecn@gmail.com> |
||
|
houseme
|
95819cb986 |
docs(object-data-cache): state the correctness boundary and timing channel (#4695)
docs(object-data-cache): state the correctness boundary and the timing channel The crate doc still described itself as "a minimal skeleton for the initial rollout phase", which stopped being true several releases ago, and neither the crate nor the operator-facing env constant said anything about what the cache does or does not guarantee. Record two things a reader has to know. The correctness boundary: a hit is sound because the key matched metadata the caller just resolved from a read quorum, not because invalidation ran. Process-local invalidation is hygiene that frees capacity; the key is what keeps a stale body from being served. Anyone tempted to weaken the key should meet this sentence first. The timing side channel: a hit skips the erasure read, bitrot verify and decode, so it is reliably faster than a miss. A principal authorized to read an object can time a single GET and learn whether someone read that object within the entry's lifetime. This crosses no authorization boundary — the probe needs read access to that exact object, checked before the cache is consulted — but it does disclose a co-tenant's recent access pattern. Say so where operators look: on RUSTFS_OBJECT_DATA_CACHE_ENABLE. Timing noise would cost precisely the latency the cache exists to save, so the mitigation is to leave the cache off for buckets where access-pattern confidentiality matters. Refs: backlog#1139 Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
60ad15a7f9 |
fix(obs): remove the dial9 task-dump switch that could never work; correct measured claims (#4688)
* fix(obs): remove the dial9 task-dump switch that could never do anything Measured on a bench host (Linux x86_64) against the code merged in #4663: with `RUSTFS_RUNTIME_DIAL9_TASK_DUMP_ENABLED=true`, a `dial9-taskdump` build, and `--cfg tokio_taskdump`, dial9 recorded **zero** TaskDump events. dial9 captures a task dump only for futures it wrapped itself — those spawned through `dial9_tokio_telemetry::spawn`, which is where `TaskDumped<F>` gets applied. `tokio::spawn` gets no wrapper, and RustFS spawns with `tokio::spawn` throughout. Same workload, same binary, only the spawner changed: tokio::spawn -> 0 dumps dial9::spawn -> 14709 dumps, all with callchains Upstream documents this (README line 151) and tracks the doc gap at dial9-rs/dial9#477. I did not read it before wiring `with_task_dumps` in #4663, and so shipped exactly the kind of lying configuration knob that PR set out to delete. Remove it: the two environment variables, the config fields, the `with_task_dumps` call, and the `dial9-taskdump` feature — whose only effect was to constrain the build to Linux while recording nothing. Re-adding it only makes sense together with migrating the paths under investigation to dial9's spawner. Tracked as D9-16 in rustfs/backlog#1157. Also drop the `--cfg tokio_taskdump` requirement from the Makefile. Measured: dumps are captured with and without it (14709 vs 14674, within noise), and upstream never asked for it. That requirement was mine, invented and untested. Cargo.lock loses tokio's `backtrace` dependency, which `tokio/taskdump` pulled in. Co-Authored-By: heihutu <heihutu@gmail.com> * docs(obs): replace guessed dial9 retention numbers with measured ones Three corrections, all to claims I wrote in #4663 without measuring them. "Under a high poll rate that budget can wrap in minutes" was a guess. Measured on a single-node 4-drive cluster under warp mixed (66 MiB/s, 110 obj/s, 32 concurrent): 13023 events/s, 0.16 MiB/s, so the default 1 GiB budget wraps after roughly 108 minutes. Even at ten times the throughput that is ~11 minutes. State the measured rate and how to scale it instead. dial9 was described as the tool for drive stalls. It is not. RustFS does disk I/O on the blocking pool and through io_uring, never on an async worker, so a slow drive never lengthens a poll. Injecting 200 ms of latency on one of four drives cut throughput by 64% and left the poll distribution unchanged (polls >= 5 ms: 49 -> 56; p999: 2.67 ms -> 2.75 ms). Enabling dial9's CPU and sched profilers does not help: sched events are per-worker only, and the CPU profiler samples on-CPU while a stalled drive is an off-CPU wait. Say so plainly, and point at the `rustfs_io_*` metrics instead. What dial9 *is* good for, on the same traces: single polls of 418-625 ms with no fault injected at all — real worker stalls nothing else in the obs stack surfaces. Lead with that. Also link the two upstream issues filed for the gaps we documented: dial9-rs/dial9#658 (writer death unobservable) and #659 (worker-s3 CVEs). Measurements: rustfs/backlog#1157 (D9-11, D9-13, D9-18). Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
Zhengchao An
|
40875026df |
feat(ecstore): add RUSTFS_ILM_DEBUG_DAY_SECS lifecycle time-acceleration for tests (#4677)
feat(ecstore): add RUSTFS_ILM_DEBUG_DAY_SECS lifecycle time-acceleration for tests (backlog#1148 ilm-5) Model a Ceph rgw_lc_debug_interval-style switch: when RUSTFS_ILM_DEBUG_DAY_SECS is set to a positive integer N, lifecycle Days-based rule evaluation treats one 'day' as N seconds instead of 86400, so Days>=1 expiration/transition/noncurrent rules become exercisable in seconds. Unset => byte-identical to production. All Days->deadline conversions funnel through expected_expiry_time(); the new ilm_day_secs() helper (OnceLock-cached in production, env-read under cfg(test)) rescales both the day offset and the default rounding boundary. An explicit RUSTFS_ILM_PROCESS_TIME still wins for the rounding boundary. Absolute Date-based rules are deliberately NOT rescaled. Activation emits a WARN; test/debug only. Refs rustfs/backlog#1148 (ilm-5), rustfs/backlog#1155. |
||
|
houseme
|
00536da80c |
refactor(obs): make dial9 telemetry opt-in and actually record events (#4663)
* refactor(obs): make dial9 telemetry opt-in and actually record events
The dial9 Tokio-runtime profiler was disabled by default, yet every build
paid for it, and enabling it produced trace files with no events in them.
Recorded empty traces
---------------------
`build_traced_runtime` called `TracedRuntime::builder()...build(..)`, but dial9
only starts recording in `build_and_start*`. `build` still returns a live guard
whose `is_enabled()` reports true, and still creates and seals segment files —
they just contain a header and no events. It also skipped `with_trace_path`, so
the background worker driving the segment pipeline was never spawned.
Measured on the new smoke example: 310 bytes of bare segment header, against
5640 bytes for the same workload once recording actually starts.
Switch to `with_trace_path(..).build_and_start(..)`.
Cost was unconditional
----------------------
`--cfg tokio_unstable` was a global `[build] rustflags` entry and `rustfs-obs`
depended on `dial9-tokio-telemetry` unconditionally, so all builds depended on
Tokio's non-semver API. Worse, an environment `RUSTFLAGS` replaces (never
appends to) the config-file value, so any caller exporting their own RUSTFLAGS
silently dropped the flag — the long comment in build.yml was a scar from that.
dial9 is now an opt-in feature (`dial9`, plus `dial9-s3` and `dial9-taskdump`),
the global rustflag is gone, and `crates/obs/build.rs` fails the compile if the
feature is on without the flag. Telemetry builds go through `make build-profiling`.
Metrics that could not lie
--------------------------
`rustfs_dial9_{events_total,bytes_written_total,rotations_total,cpu_overhead_percent}`
were hard-coded to zero — a Counter pinned at 0 reads as "nothing happened".
Removed. `rustfs_dial9_enabled` was sourced from the environment, so it read 1
even when the traced runtime failed and the process fell back to a standard
runtime; it is replaced by `rustfs_dial9_supported` (compile-time),
`rustfs_dial9_configured` (intent) and `rustfs_dial9_active_sessions` (reality).
No `writer_healthy` gauge is exported: dial9's `RotatingWriter` can enter its
`Finished` state and stop writing, but exposes no way to observe that, so the
gauge could only ever be hard-coded to 1. Documented as a known gap instead.
Final events were lost
----------------------
The `TelemetryGuard` lived in a `static OnceLock`, which is never dropped, so
buffered events were never flushed at exit. `build_tokio_runtime` now returns
the guard and `run_process` drops it before any exit path.
Also
----
- `disk_usage_bytes` was a `read_dir` + per-file `stat` on the metrics
collection path. It is now sampled by a background task into an atomic.
- `SAMPLING_RATE`/`S3_BUCKET`/`S3_PREFIX` were parsed, warned about, and
discarded. S3 upload is now wired to dial9's `with_s3_uploader` behind
`dial9-s3`; `SAMPLING_RATE` has no upstream equivalent and is removed.
- Wire `with_task_dumps` (async backtraces of stalled tasks), configurable via
`RUSTFS_RUNTIME_DIAL9_TASK_DUMP_{ENABLED,IDLE_THRESHOLD_MS}`.
- Split `telemetry/dial9.rs` into `config`/`state`/`enabled`/`disabled`; the
stub keeps the public API identical so callers need no `#[cfg]`.
- Drop four print-only examples and the manual test bin that exercised the
removed `init_session` scaffolding.
Verified: cargo check/clippy/test across default, `dial9`, and `dial9-s3`;
build.rs correctly rejects `dial9` without `--cfg tokio_unstable`;
`make pre-commit` passes.
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(obs): document dial9 as an on-demand profiler
scripts/run.sh advertised a `SAMPLING_RATE` knob that was never passed to dial9,
and claimed "CPU overhead < 5% (with sampling rate 1.0)" and "lower values reduce
CPU overhead" on the strength of it. The knob is gone; the guidance built on it
had to go too.
Replace it with what is actually true: dial9 needs a `make build-profiling`
binary, its disk budget evicts oldest-first (so a high poll rate can overwrite
the incident you are chasing), and it cannot be toggled without a restart.
Add docs/operations/dial9-runtime-profiling.md covering the build variants, an
investigation walkthrough, the configuration table, how to read the three
supported/configured/active_sessions gauges against each other, and the upstream
gap that makes writer death only indirectly observable.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(obs): add a dial9 smoke example that proves events are recorded
The bug this guards against is invisible to every existing signal: with
`build` instead of `build_and_start`, dial9 creates the trace file, seals
segments, and reports `TelemetryGuard::is_enabled() == true` — it simply
records no events. Only the segment's byte count tells the two apart.
Measured on this workload: 5640 bytes when recording, 310 bytes (a bare
segment header) when not. The example asserts >= 2048 bytes, and was verified
to fail with the `build` call restored.
Also correct the comment on the `is_enabled` check in `finish_traced_runtime`.
It claimed to catch "recording silently off"; it does not. It only rejects the
inert guard a lenient config yields after a build failure. Recording is
guaranteed by `build_and_start`, not by that check.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(rustfs): accept Unsupported runtime telemetry capability
A binary built without the `dial9` feature now reports the runtime-telemetry
capability as `Unsupported` rather than `Disabled`. The distinction matters to
operators: `Disabled` implies the capability can be switched on by setting an
environment variable, which is not true here — telemetry needs a rebuild.
Widen the assertion and pin the new semantics: when `dial9::is_supported()` is
false, the state must be exactly `Unsupported`.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs): drop the dial9-s3 feature, its TLS stack is vulnerable
CI's Dependency Review and `cargo deny` both reject the branch: dial9's
`worker-s3` feature depends on aws-sdk-s3-transfer-manager 0.1.3, which pins
aws-smithy-http-client onto hyper-rustls 0.24 and rustls-webpki 0.101.7. That
webpki carries RUSTSEC-2026-0098, -0099 and -0104.
0.1.3 is the latest release of the transfer manager, and 1.2.0 the latest of the
smithy client, so there is nothing to upgrade to. Cargo's feature unification can
add features but cannot drop a transitive dependency, so it cannot be worked
around from here either — the rest of the workspace already resolves to the safe
rustls-webpki 0.103 / hyper-rustls 0.27.
Remove the `dial9-s3` feature and the `with_s3_uploader` wiring. The two S3
environment variables stay parsed and warned about, now naming the real reason
rather than a missing build feature. Trace segments are collected from the output
directory instead. Tracked as D9-14 in rustfs/backlog#1157.
With this, Cargo.lock is byte-identical to main: the PR no longer touches the
dependency graph at all.
Also correct the `dial9-taskdump` documentation. It claimed the feature "compiles
to a no-op elsewhere"; in fact `tokio/taskdump` raises a `compile_error!` on any
target other than linux/{aarch64,x86,x86_64}. Verified by trying to build it on
macOS, which is how the claim was found to be wrong.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
|
||
|
houseme
|
d1daa97ac5 |
fix(ecstore): keep local path startup fail-fast under Kubernetes (#4640)
The startup topology convergence auto-detection (added in #4631) checked Kubernetes before endpoint style, so a local path deployment (single or multi-drive, non-distributed) running inside Kubernetes was classified as orchestrated with an effectively unbounded wait window. Path-style endpoints have no hostnames to resolve, so the wait never actually triggers and runtime behavior is unchanged, but the startup log then advertised mode=orchestrated / wait_timeout=MAX for a purely local deployment, which is misleading and contradicts the documented policy (local path endpoints -> fail-fast). Order the auto-detection by endpoint style first: only distributed URL endpoints, which resolve hostnames, pick orchestrated (Kubernetes) or bounded (otherwise); local path endpoints stay fail-fast regardless of the platform. Update the doc comment to match and add a policy case locking Kubernetes + local path to fail-fast. Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
6a81353785 |
feat(ecstore): converge startup topology under a wait-mode policy (#4631)
Multi-node startups (Kubernetes StatefulSets, bare-metal, VMs) can hit transient DNS/local-host resolution failures while peers and headless service records are still coming up. Previously endpoint resolution retried DNS for a fixed 90s window and then returned an error, which propagated out of run() and exited the process; in Kubernetes this drove repeated kubelet restarts even though the cluster eventually converged. Introduce a startup topology convergence policy with three modes: - orchestrated: wait effectively indefinitely for recoverable DNS/topology errors so the process stays Running (readiness stays false) instead of exiting; defer the DNS-IP cross-port validation because headless-service records can still be flapping. - bounded: wait a finite window (default 3m) then fail with an action-oriented error listing host/stage/elapsed and remediation hints. - fail-fast: fail on the first non-transient resolution error. Mode is auto-detected (Kubernetes -> orchestrated, distributed URL endpoints -> bounded, local path endpoints -> fail-fast) and can be overridden via RUSTFS_STARTUP_TOPOLOGY_WAIT_MODE, RUSTFS_STARTUP_TOPOLOGY_WAIT_TIMEOUT and RUSTFS_STARTUP_TOPOLOGY_RETRY_MAX_DELAY. Also normalize divergent local/remote verdicts for endpoints that share a host:port, throttle retry warnings, and keep the DNS-independent local same-path checks in every mode. Configuration error handling is unchanged: malformed volumes, mixed styles/schemes, duplicate or root paths, and non-transient errors still fail fast. Read the topology env vars through rustfs_utils::get_env_opt_str for consistent alias handling, and simplify the log_once poisoned-lock branch to unwrap_or_else(PoisonError::into_inner). Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
GatewayJ
|
d238b2d24e | fix(admin): support external OIDC browser redirects (#4280) | ||
|
Zhengchao An
|
d8c32d3754 | fix(ecstore): retry idempotent reads to survive read-after-write races (#4597) | ||
|
Zhengchao An
|
c0e2c02e51 |
fix(rio): warn once when internode HTTP/2 tuning is inert on plaintext (backlog#805-C3) (#4587)
The internode HTTP/2 window/keepalive tuning (apply_http_client_tuning + http2_keep_alive_* in build_http_client) only takes effect when the connection negotiates HTTP/2, which happens via TLS-ALPN. Over plaintext internode transport the connection is HTTP/1.1 and all the tuning is silently inert, with no signal to the operator. Make this honest without changing protocol behavior (no h2c, no prior_knowledge, no opt-in flag): - InternodeHttpClientTuning gains h2_tuning_explicit, true when the operator set a window size, a non-default tuning profile, or the keepalive env. - Add pure predicate should_warn_h2_inert(negotiated_is_http2, h2_tuning_explicit, already_warned) and maybe_warn_h2_inert, gated by a process-lifetime AtomicBool so the warn fires at most once. - Call maybe_warn_h2_inert at both record_internode_http_version sites using the actual negotiated resp.version(). - Document the TLS-ALPN requirement at apply_http_client_tuning and near the ENV_INTERNODE_HTTP2_* constants. Tests cover the should_warn_h2_inert truth table and that h2_tuning_explicit reflects explicit windows and a non-default profile. |
||
|
Zhengchao An
|
cf2da0c44d |
refactor(object-capacity): remove the decorative SymlinkTracker and its no-op depth knob (#4571)
The tracker never influenced traversal: walkdir's follow behavior is fixed up front by follow_links(), and should_follow() only gated the tracker's own bookkeeping. Its 'depth limit' compared tree depth (not symlink chain depth), so RUSTFS_CAPACITY_MAX_SYMLINK_DEPTH was a complete no-op while its telemetry claimed symlinks were skipped that walkdir had in fact followed and counted; record_symlink was always called with size 0, so tracked_bytes never left zero (S12). Remove the tracker, its skipped/summary events, the symlink metric and the depth env knob end to end (the env's 'as u8' truncation goes with it), and document the real semantics at the walker: follow_links(true) counts targets with walkdir's ancestor-loop detection breaking cycles, follow_links(false) — the default — counts no symlink targets. The scan root itself is pre-resolved since backlog#1015. Ref: rustfs/backlog#1018 (S12 from audit rustfs/backlog#1010) |
||
|
Zhengchao An
|
0ebe00b383 |
fix(object-capacity): drive scan progress checks by visited entries and remove unreachable stall detector (#4557)
Progress/timeout checks only ran when file_count advanced, so trees of directory-only or error-dense entries (dirs, traversal errors, symlinks never increment file_count) bypassed the entire cooperative time budget and held the blocking thread for unbounded time, while each error entry logged its own warn — permission-dense trees flooded the log (S13). The stall detector was structurally unreachable: it fired on 'no file progress between checks', but checks themselves only ran when file progress happened, so RUSTFS_CAPACITY_STALL_TIMEOUT never did anything since its introduction (S09). - Progress checks (timeout + early-sampling entry) are now driven by a visited-entry counter that also advances on directories and errors, so every tree shape reaches the budget checks. - Remove the unreachable stall detector and its plumbing end to end (ProgressMonitor fields, ScanLimits, config getter, env const, stall metric). Genuine walker wedges are handled by the hard outer wall-clock budget from backlog#1017; no decorative protection is left. - Cap per-entry error warns at 10 per scan with an explicit suppression notice; had_partial_errors still records the condition. Ref: rustfs/backlog#1016 (S09+S13 from audit rustfs/backlog#1010) |
||
|
Zhengchao An
|
d1245ca33c | fix(config): default trusted proxies to loopback-only (#4508) | ||
|
houseme
|
d3660f9ded |
refactor(config): remove dead Direct I/O constants from zero_copy.rs (#4379)
ENV_OBJECT_DIRECT_IO_ENABLE, DEFAULT_OBJECT_DIRECT_IO_ENABLE, ENV_OBJECT_DIRECT_IO_THRESHOLD and DEFAULT_OBJECT_DIRECT_IO_THRESHOLD were never read anywhere in the workspace. The real O_DIRECT read path landed in PR #4365 uses RUSTFS_OBJECT_DIRECT_IO_READ_ENABLE / RUSTFS_OBJECT_DIRECT_IO_READ_THRESHOLD (crates/ecstore/src/disk/local.rs). Keeping the near-identically named dead knobs invites operators to set the wrong variable and believe O_DIRECT is enabled. Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
Zhengchao An
|
3ed414cdb4 | fix(storage): complete pending metadata and quorum fixes (#4375) | ||
|
houseme
|
88645c9169 |
fix(server): stop closing idle proxy keep-alive connections + reverse-proxy hardening (#3076) (#4360)
* fix(server): raise default HTTP/1.1 idle keep-alive timeout for proxies In hyper's HTTP/1.1 stack `header_read_timeout` is armed as soon as the connection is ready to read the next request head, so on a keep-alive connection it also bounds how long an idle upstream connection may sit before RustFS closes it. The previous 5s default meant RustFS FIN'd pooled upstream connections while reverse proxies (Caddy ~2min, Nginx 60s) still believed they were alive. The proxy then reused a dead socket and clients saw `socket hang up` / connection reset, most visibly on non-idempotent `PUT` uploads that proxies will not transparently retry (issue #3076). Raise DEFAULT_HTTP1_HEADER_READ_TIMEOUT to 75s (above common proxy idle-keepalive windows) and document the coupling. Still overridable via RUSTFS_HTTP1_HEADER_READ_TIMEOUT for deployments that expose RustFS directly to untrusted slow clients and want tighter slowloris protection. Co-Authored-By: heihutu <heihutu@gmail.com> * docs(operations): add reverse-proxy deployment guide Document the request/body semantics RustFS expects from a proxy layer, the idle keep-alive mismatch that causes `socket hang up` on large PutObject writes, and known-good Caddy/Nginx configs plus Cloudflare caveats. Closes the documentation gap called out in issue #3076 and consolidates the scattered findings from #609/#934/#1492/#1766. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(object): bound stalled PutObject request-body reads with diagnostics When a reverse proxy or CDN forwards a partial request body and then goes silent without closing the connection, the inner body stream neither yields more bytes nor reports EOF, so RustFS waited forever for bytes that never arrive and the client saw a silent hang/abort (issue #3076). A short body that ends with a proper EOF was already rejected promptly; the gap was the no-EOF stall case. Add a single-point request-body guard on the PutObject path that wraps the incoming StreamingBlob with an inter-chunk read timeout. The timer resets on every chunk, so slow-but-progressing uploads are unaffected; it only fires after RUSTFS_HTTP_REQUEST_BODY_READ_TIMEOUT (default 300s, 0 disables) of complete silence. On timeout it logs a structured `put_object_body_read_stalled` event with received/expected byte counts and fails the read with ErrorKind::TimedOut instead of hanging. remaining_length/size_hint are forwarded so wrapping is transparent to downstream length handling. Covered by unit tests for the stall path, length-preserving pass-through, and the disabled (timeout=0) pass-through. Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
f73880f354 |
fix(startup): survive slow multi-node cold starts (#4357)
* fix(server): make startup readiness wait configurable and raise default (#4264) The startup runtime-readiness wait was a hardcoded 30s constant with no env override. On slow multi-node cold starts (Docker/K8s/Synology NAS) this window is shorter than the internal startup budgets it depends on — the endpoint DNS-retry window (~90s) and the format-load retry loop (~100s worst case) — so readiness times out and the node exits with `startup readiness timed out after 30s: storage_ready=false, lock_quorum_ready=false` before storage/lock quorum can converge, feeding the restart storm reported in the issue. - Add `RUSTFS_STARTUP_READINESS_MAX_WAIT_SECS` (default 120s), documented in rustfs-config health constants. - Resolve the wait at runtime via `startup_runtime_readiness_max_wait()`; a value of `0` falls back to the default instead of timing out instantly. - Repoint `STARTUP_RUNTIME_READINESS_MAX_WAIT` at the shared config default so there is a single source of truth, and cover the getter with unit tests. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): stop peer/disk background monitors on graceful shutdown (#4264) Long-lived peer health/recovery and remote-disk monitors are detached `tokio::spawn` tasks that each hold a `tracing::Span` via `.instrument(..)` for their whole lifetime. Nothing cancelled them at shutdown, so on the normal return path the Tokio runtime was dropped while they were still alive and their `Span`s were dropped during worker-thread thread-local-storage (TLS) destruction. At that point `tracing-subscriber`'s fmt `on_close` can touch an already-destroyed TLS slot and panic with `cannot access a Thread Local Storage value during or after destruction`, which escalates to a panic-during-panic abort (SIGILL / exit 132) — the crash reported on Synology in issue #4264, amplified by the restart storm. - Add `cluster::rpc::background_monitor` with a process-global shutdown token, `spawn_background_monitor()` (races the monitor future against that token so its span drops while the runtime is alive), and public `shutdown_background_monitors()`. - Route every span-holding peer_s3 / peer_rest / remote_disk monitor spawn through `spawn_background_monitor` instead of `tokio::spawn(..).instrument()`. - Expose `rustfs_ecstore::shutdown_background_monitors()` and call it from the graceful shutdown sequence (right after `ctx.cancel()`, before runtime teardown) via the `storage_api` compatibility boundary. Existing recovery-probe span-context tests still pass, confirming log correlation is preserved. Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
6f613317f6 |
feat(internode): optimize gRPC transport (#4337)
* feat(internode): P0 gRPC transport tuning, message limits, payload metrics Land the P0 subtask from docs/grpc-optimization: close the client-vs-server transport gaps and add instrumentation to size which unary RPCs need channel isolation in P1. Transport tuning (G3): the client `Endpoint` now disables Nagle and raises the HTTP/2 stream/connection flow-control windows to mirror the server socket, so small lock/health RPCs are not batched and larger metadata responses are not throttled by the 64KiB default window. All env-overridable, 0 opts out. Message-size limits (G1): both `NodeServiceClient` and `NodeServiceServer` set max decode/encode size (default 100MiB) instead of tonic's silent 4MiB cap, so a large multi-version xl.meta or aggregated ReadMultiple no longer fails out_of_range. The server limit is set on `NodeServiceServer` before wrapping in the auth `InterceptedService` (the interceptor type does not expose it). Payload instrumentation (P1 prep): ReadAll/ReadMultiple record a payload-size histogram plus a large-payload counter when a response crosses the configured threshold (default 8MiB), feeding alerting on paths that contend with latency-sensitive control-plane traffic on the shared channel. Threshold-only counter, no per-call hot-path log. Verification: cargo check/test on config, io-metrics, ecstore, rustfs; clippy clean on touched files; make pre-commit green. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(internode): P1 control/bulk gRPC channel isolation (opt-in) Land the P1 subtask from docs/grpc-optimization: physically separate large bytes-carrying unary RPCs from latency-sensitive control-plane RPCs so a big transfer can no longer head-of-line block a lock/health RPC on the shared HTTP/2 connection (G2/G5). Introduce ChannelClass { Control, Bulk } and get_channel_for_class in protos. Control RPCs keep the per-peer connection keyed by the bare address; Bulk RPCs (ReadAll/WriteAll/ReadMultiple/BatchReadVersion, via a new get_bulk_client) are round-robined across a small per-peer bulk pool. Rather than restructuring the global GLOBAL_CONN_MAP (and every consumer), bulk channels are cached under a composite key (addr\0bulk\0idx). The NUL separator cannot appear in a URL, so bulk keys never collide with the control key. This keeps the blast radius small on a consistency-sensitive path. create_new_channel is refactored into build_channel(dial_addr, cache_key) so several physically distinct channels to one peer cache independently while dialing/TLS still use the real address. Gated by RUSTFS_INTERNODE_CHANNEL_ISOLATION (default OFF) so the default build is byte-for-byte the pre-P1 behavior: bulk resolves to the control channel and the switch is a single-env rollback. RUSTFS_INTERNODE_BULK_CHANNELS (default 2, clamped >=1) sizes the pool. On failure, evict_failed_connection drops the whole bulk pool for the peer (round-robin hides which index was used), avoiding half-dead cached channels. Lock RPCs (remote_locker) already use the default Control path, so lock semantics and retry behavior are unchanged. Verification: cargo check/test on config, protos, ecstore, rustfs; new protos tests for bulk key routing and isolation-off passthrough; clippy clean on touched files; make pre-commit green. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(internode): P2 msgpack/JSON codec observability + encode buffer presizing Land the safe, wire-compatible slice of P2 from docs/grpc-optimization: the observability prerequisite for retiring the redundant JSON fields, plus a codec micro-optimization. No proto/wire-format change; JSON is still dual-written. Internode RPCs today dual-encode each metadata value as both msgpack (`*_bin`) and a JSON compatibility string, and decoders prefer `_bin` with a JSON fallback. Before the JSON fields can ever be dropped (a cross-version change), that fallback must be proven unused in production. Add rustfs_system_network_internode_msgpack_json_fallback_total{direction, message}: incremented whenever a decode falls back to the JSON field because the msgpack payload was absent. Wired into both directions — the client decoding peer responses (remote_disk.rs, incl. the list-level read_multiple/batch fallbacks) and the server decoding peer requests (node_service/disk.rs). This counter must read zero across a release window before send paths stop writing JSON and the proto text fields are reserved/removed (the deferred P2-1 steps). Also pre-size the msgpack encode buffers (Vec::with_capacity(512)) on both sides, eliminating the repeated growth reallocations for typical FileInfo payloads with zero added copy. Full thread_local buffer pooling is deferred: it needs either an extra copy (unclear net win) or a send-path buffer-return lifecycle, to be justified by a codec microbenchmark first. Verification: cargo check/test on io-metrics, ecstore, rustfs; new fallback counter smoke test; existing codec decode tests green; clippy clean on touched files; make pre-commit green. Co-Authored-By: heihutu <heihutu@gmail.com> * docs(internode): add msgpack/JSON convergence observation runbook Runbook driving the observation-gated retirement of the redundant JSON compatibility fields on internode gRPC metadata RPCs (grpc-optimization P2-1). Documents the shipped fallback counter (rustfs_system_network_internode_msgpack_json_fallback_total{direction,message}), the PromQL to confirm it reads zero across a release window, a standing alert, and the staged flip/rollback procedure (env-gated msgpack-only send, then proto field removal in N+1). Includes the verified field -> peer-decoder audit: only fields whose peer decodes _bin first may be converged. Notes DeleteVersion.opts (DeleteOptions) is NOT convergence-ready — its server handler is not _bin-first and must gain a decode_msgpack_or_json path first. This gates the send-side change so it cannot empty a JSON field an old peer still needs. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(internode): env-gated msgpack-only send + DeleteVersion _bin support (P2-1) Implements the send-side lever for retiring the redundant JSON compatibility fields on internode gRPC metadata RPCs, plus the missing `_bin` support on the delete path that it depends on (grpc-optimization P2-1). Default-off: the base build is byte-for-byte the prior dual-write behavior. Gated msgpack-only send (RUSTFS_INTERNODE_RPC_MSGPACK_ONLY, default false): - New rustfs_protos::internode_rpc_msgpack_only() reads the flag. - Client (remote_disk.rs) compat_json() and server (node_service/disk.rs) compat_response_json() emit an empty JSON string when the flag is on, so only the msgpack _bin payload is sent. The _bin field is always sent; decoders keep the JSON read fallback. Applied only to fields with a confirmed _bin-first peer decoder (WriteMetadata/UpdateMetadata/RenameData file_info, UpdateMetadata opts, ReadOptions, ReadMultipleReq, BatchReadVersionReq; ReadVersion/ReadXL/RenameData responses and the ReadMultiple/BatchReadVersion response lists). - Only enable after the P2 fallback counter has read zero across a release window (see docs/operations/internode-msgpack-json-convergence-runbook.md). Single-env rollback; no wire-format break. DeleteVersion(s) _bin support (prerequisite): - The DeleteVersion/DeleteVersions protos had NO _bin fields. Add additive (backward-compatible) bytes file_info_bin/opts_bin (DeleteVersion) and repeated bytes versions_bin + bytes opts_bin (DeleteVersions); regenerate the checked-in prost struct. - Client dual-writes them; server decodes them _bin-first with JSON fallback. - These delete fields are kept OUT of the msgpack-only set (always dual-write) until their own fallback counter reads zero across a window with the new decoders fully deployed. DeleteVersion.raw_file_info stays JSON-only (no _bin field yet). Verification: cargo check/test on protos, config, ecstore, rustfs (incl. the six delete request handler tests and a compat_json default-path test); clippy clean on touched files; make pre-commit green. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(internode): P3 cluster peer online/offline health metric Land the safe observability core of P3 (grpc-optimization G6/G8): track each internode peer's reachability and expose the offline count, for parity with MinIO's minio_cluster_servers_offline_total. Pure instrumentation — peer selection and quorum are unchanged. - io-metrics: per-peer PeerHealthState { online, consecutive_failures } registry plus record_peer_reachable/record_peer_unreachable. A peer flips offline after N consecutive failures (dial failures or RPC-triggered evictions) and back online on the next successful dial; the count of offline peers is published to the rustfs_cluster_servers_offline_total gauge. - config: RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD (default 3, clamped >= 1). - protos: build_channel marks the peer reachable on a successful dial and unreachable on a dial failure; evict_failed_connection feeds the failure signal too. Keyed by the real peer address, so control and bulk channels to one peer share health state. Deferred (documented in docs/grpc-optimization P3): startup prewarm (no clean topology-ready hook yet), the offline fast-bypass in peer routing (consistency- sensitive; must not change quorum), and idempotent-read-only retry. This commit is observability only. Verification: cargo check/test on io-metrics, config, protos (new peer-health state-machine and threshold-clamp tests); clippy clean on touched files; make pre-commit green. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(internode): P3 control-channel prewarm + self-healing offline bypass Add the remaining P3 connection-lifecycle levers (grpc-optimization G6/G8), both env-gated and default-off so the base build is unchanged. Prewarm (RUSTFS_INTERNODE_PREWARM, default off): RemoteDisk::new spawns a best-effort background dial of the peer's control channel, deduped per peer address, moving the connect cost off the first RPC. Failures fall through to the existing lazy connect + recovery monitor. Offline bypass (RUSTFS_INTERNODE_OFFLINE_BYPASS, default off): remote_disk get_client/get_bulk_client fast-fail a peer already marked offline instead of paying the connect timeout, so the erasure layer proceeds on quorum sooner. This does NOT change quorum. It is self-healing: cluster_peer_should_bypass lets one request per RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS (default 5s) through to recover the peer even with no background monitor, and the recovery monitor's own probe path calls the client directly so it is never bypassed. io-metrics gains cluster_peer_is_offline / cluster_peer_should_bypass (with a per-peer re-probe timestamp). Scope: data path only — remote_locker (lock RPCs, most consistency-sensitive) is left dual-writing/unbypassed as a follow-up. Verification: cargo check/test on io-metrics, config, ecstore (new self-healing bypass tests; all 105 rpc tests green); clippy clean on touched files; make pre-commit green. Co-Authored-By: heihutu <heihutu@gmail.com> * docs(internode): add A/B benchmark runbook for gRPC optimization stages Reproducible before/after collection procedure for grpc-optimization P0–P3. Since every stage is env-gated, before/after is the same binary with different env — no rebuild. Documents, per stage: the exact env toggles (baseline vs enabled column), which existing bench script to run (run_internode_transport_baseline.sh / run_four_node_cluster_failover_bench.sh), the Prometheus metrics to capture, and the acceptance gates from the design docs (e.g. lock p99 down >= 20% for P1, msgpack fallback counter = 0 before enabling P2, correct rustfs_cluster_servers_offline_total for P3). Live runs require a multi-node cluster + load tool + Prometheus scrape and cannot be produced in a single-process sandbox; artifacts land under target/bench (gitignored) and attach to the PR. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(internode): P3-2 lock-path offline bypass + P3-3 idempotent read retry Extend the offline bypass to the lock path and add opt-in retries for idempotent reads (grpc-optimization P3-2/P3-3). Both env-gated and default-off/zero. Offline bypass (lock path): factor the bypass decision into a shared pub(crate) internode_offline_bypass_reason(addr) and call it from remote_locker::get_client too, so lock RPCs to an offline peer fast-fail (letting dsync reach quorum sooner) instead of paying the connect timeout. Does not change quorum; the self-healing re-probe keeps peers recoverable. Gated by RUSTFS_INTERNODE_OFFLINE_BYPASS (default off). Idempotent read retry (P3-3): add execute_read_with_retry — a bounded, exponential-backoff retry for read-only/reentrant RPCs on transient network errors — and route disk_info through it. RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES defaults to 0 (disabled). Write/lock RPCs are never retried (quorum/idempotency safety, per CLAUDE.md); the wrapper requires an Fn closure so only reads that rebuild their request from borrowed inputs qualify. Deferred: grpc.health.v1 (optional ecosystem-compat only; needs a new tonic-health dep and 3-way hybrid-service wiring — internal needs are met by the existing Ping RPC). Verification: cargo check/test on config, ecstore (105 rpc tests green incl. disk_info now via the retry wrapper); clippy clean on touched files; make pre-commit green. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(scripts): one-click internode gRPC A/B benchmark driver Wrap the per-stage env matrix from the benchmark runbook into scripts/run_internode_grpc_ab_bench.sh: given --stage <p0|p1|p2|p3> and --phase <before|after>, it emits the stage/phase RUSTFS_INTERNODE_* server env to <out-dir>/server-env.sh and runs the right underlying bench (run_internode_transport_baseline.sh for p0/p1/p2, run_four_node_cluster_failover_bench.sh for p3) into a labeled target/bench/internode-transport/<stage>-<phase>/. Passthrough args after `--` reach the underlying bench; --dry-run previews the env + command. The script is explicit that RUSTFS_INTERNODE_* are server env, so for the load-driven stages the operator must restart rustfs with the emitted env before the run; the docker four-node (p3) path exports them for a forwarding compose. shellcheck-clean. Runbook updated with a "One-click driver" section. Co-Authored-By: heihutu <heihutu@gmail.com> * chore(compose): forward RUSTFS_INTERNODE_* into the four-node cluster The four-node local-build compose only forwarded a fixed whitelist of env, so the internode gRPC knobs (grpc-optimization P0-P3) never reached the containers and the A/B bench driver's "after" phase was a no-op. Forward the full RUSTFS_INTERNODE_* set with defaults matching the binary defaults, so leaving them unset is a no-op and the A/B driver can toggle a stage per phase. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(internode): address Copilot review — retry health action + poison-safe peer health Two review nits on #4337: - P3-3 idempotent read retry (remote_disk.rs): execute_read_with_retry ran every attempt through execute_with_timeout_for_op, which hardcodes FailureHealthAction::MarkFailure. So the first transient error could flip the disk faulty and short-circuit the remaining retries, and each attempt over-counted the failure. Route all but the final attempt through execute_with_timeout_for_op_and_health_action with IgnoreFailure; only the last attempt marks faulty/evicts. No default impact (retries default 0). - Peer-health helpers (io-metrics): record_peer_reachable/record_peer_unreachable, cluster_peer_is_offline and cluster_peer_should_bypass early-returned on a poisoned mutex, permanently stalling the offline gauge and bypass state after a single panic. Recover via PoisonError::into_inner(). Verification: cargo check/test on io-metrics + ecstore (105 rpc tests green); clippy clean on touched files; make pre-commit green. Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
Zhengchao An
|
9959c828a1 |
fix(internode): relax keepalive/RPC timeouts to fix large-object GET EOF (#4284)
fix(internode): relax aggressive HTTP/2 keepalive & RPC timeouts to stop large-object GET truncation The internode gRPC channel used a 3s HTTP/2 keepalive timeout and a 10s overall RPC timeout. Under high-concurrency large-object reads a saturated peer's PING ACK is legitimately delayed past 3s, so the whole channel (and every RPC/stream on it) is torn down as a 'dead peer'. In-flight peer shard reads then fail mid-object and large GETs truncate after headers+Content-Length are already sent, surfacing to clients as 'download error: unexpected EOF'. Reproduced on a 4-node erasure cluster with pure GET-only warp (no concurrent writes) at 64 concurrency: 10MiB GET ~31 unexpected-EOF / 3min. Raising the internode keepalive timeout (3s->30s via env) alone cut that to ~7; also raising the RPC timeout cut it to ~3. - Raise DEFAULT_INTERNODE_HTTP2_KEEPALIVE_TIMEOUT_SECS 3 -> 20 - Raise DEFAULT_INTERNODE_RPC_TIMEOUT_SECS 10 -> 30 - Wire the data-plane rio HttpReader keepalive timeout (was hardcoded 3s) to the same env/default so control- and data-plane stay consistent. Refs backlog#832. |
||
|
Zhengchao An
|
92402a3bde |
perf(get,heal): fix GET hot-path overhead and heal checkpoint scaling (#4237)
perf(get,heal): land verified fixes for backlog #800-#804 Five fixes from the GET-path performance audit and scanner/heal completeness audit (rustfs/backlog#800..#804), each verified locally: - backlog#800 (heal checkpoint O(N^2)): ResumeCheckpoint object sets are now HashSet (Vec::contains was O(n) per healed object; 1.5ms at n=1M vs 11ns measured), and per-object checkpoint/resume-state persistence is batched (1000 mutations / 5s) instead of rewriting the whole file per object. complete_page() prunes the sets at page boundaries so memory stays bounded; positions still persist unconditionally and legacy Vec-format checkpoints still deserialize. - backlog#801 (DiskInfo.healing never set): erasure-set heal now writes a healing marker (.rustfs.sys/healing.bin) on the disks it rebuilds (endpoints plumbed via HealRequest/HealTask.heal_endpoints from the auto disk scanner) and clears it on success. LocalDisk::disk_info surfaces the marker, so scanner heal coordination, lock selection and admin/metrics healing counts see the rebuild. - backlog#802 (cache probe after data read): new GetObjectBodyCacheHook in ecstore lets the app-layer object data cache serve the body inside get_object_reader, after metadata quorum resolution (etag known) but before the erasure shard read/decode. Previously a hit still paid the full disk read. Hook is None/no-op when the cache is disabled. - backlog#803 (GET hot-path redundant work): ObjectInfo is cloned for event notification only when an event will actually be built (GET and HEAD paths; events are currently suppressed so the clone was pure waste); get_opts/put_opts/del_opts resolve bucket versioning with one metadata-sys lookup instead of two; skip_verify_bitrot and get_lock_acquire_timeout env reads are cached via OnceLock; the io-priority metric is no longer double-counted; GetObject input fields are cloned selectively instead of cloning the whole input. - backlog#804 (disk permit starvation): the disk-read permit wait is now bounded (RUSTFS_OBJECT_DISK_PERMIT_WAIT_TIMEOUT, default 5s, 0 = previous unbounded behavior); on timeout the GET proceeds without a permit and the bypass is counted. DiskReadPermitReader also releases the permit at body EOF instead of holding it until the client drops the stream. Verification: make pre-commit; cargo clippy -D warnings on the four changed crates; full rustfs lib suite (2096 tests) green; rustfs-heal lib suite green with new unit tests for checkpoint pruning/legacy format/throttle, permit EOF release, and the cache hook (hit + SSE skip). The heal_integration_test and one set_disk listing test fail identically on unmodified main (pre-existing global-state ordering flakes, verified via git stash A/B). Co-authored-by: houseme <housemecn@gmail.com> |
||
|
houseme
|
eebd16d8a4 |
feat(cache): add object data cache engine and app flow (#4187)
* feat(cache): add object data cache engine * feat(cache): wire app-layer object cache flow * refactor(cache): streamline app-layer cache flow * refactor(cache): tighten cache flow internals * refactor: address final clippy cleanup * chore(deps): update quick-xml to 0.41.0 * feat(cache): wire object data cache env config * fix(cache): gate materialize fill by cache plan * chore(cache): add object data cache benchmark gate * fix(cache): guard object cache fill size mismatches * refactor(cache): streamline object cache body planning * fix(cache): align object cache rollout config * test(cache): cover buffered object cache benchmark * test(cache): isolate object cache benchmark metrics * test(cache): mark materialize rollout experimental * test(cache): tighten object cache benchmark gate * fix(cache): address review findings for object data cache - singleflight: clean up leader entry on cancellation (Drop impl) so a dropped GET future can no longer wedge all subsequent fills for the same key; switch the fill map to a std Mutex and add a regression test - adapter: honor RUSTFS_OBJECT_DATA_CACHE_ENABLE=true by defaulting to hit_only when no explicit mode is set (explicit mode still wins) - planner: treat nil version UUIDs as "no value" per repo convention so unversioned objects key under the canonical "null" instead of fragmenting the key space - multipart: invalidate the object cache on the quota-exceeded rollback delete after complete-multipart, closing a stale-cache window - layering: move the disabled-cache fallback into app::context and drop the new infra->app layer-dependency baseline entry * fix(cache): close invalidation races and drop full-cache scan on writes - index: make identity-index insert/remove/prune atomic via starshard compute_if_present/compute_if_absent so concurrent fills can no longer drop each other's keys (lost keys made entries unreachable to invalidation until TTL); add a concurrency regression test - fill: register the key in the identity index before the entry becomes visible in the cache and re-check the index afterwards, undoing the fill when an invalidation raced in between (new skipped_invalidation_race fill result) - invalidate: with the index now authoritative, remove the full-cache iter() fallback that made every PUT/DELETE of a never-cached object O(total cache entries) (two scans per PUT, 2N per batch delete) - materialize-fill: fail the GET instead of falling back to the partially consumed stream after a mid-read error (the fallback would send a body missing its prefix under a full-length Content-Length), and log the same size-mismatch warning as the sibling buffering paths Co-Authored-By: heihutu <heihutu@gmail.com> * test(storage): fix media-dependent buffer clamp expectation test_concurrency_manager_multi_factor_strategy_buffer_clamp asserted media_cap.min(MI_B), but the implementation's final safety clamp is [32KiB, media_cap.max(MI_B)] — deliberately so a media cap above 1MiB (NVMe's 2MiB default) stays effective. The test only passed on machines detected as SSD/Unknown (cap == 1MiB) and failed on NVMe-backed CI runners with 2MiB != 1MiB. Assert the media cap itself, which is what the strategy actually guarantees on every environment. Co-Authored-By: heihutu <heihutu@gmail.com> * test(storage): format buffer clamp assertion * chore(logging): update tier guardrail path --------- Signed-off-by: houseme <housemecn@gmail.com> Co-authored-by: cxymds <cxymds@gmail.com> Co-authored-by: overtrue <anzhengchao@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
25d80d7c60 |
feat(storage): harden internode data-path controls (#4224)
* fix(rio): propagate http writer shutdown errors * fix(ecstore): unify remote lock rpc deadlines * fix(storage): reject corrupt read multiple payloads * feat(rio): add internode http tuning profiles * feat(metrics): add internode baseline signals * feat(ecstore): observe shard locality topology * feat(ecstore): gate shard locality scheduling * feat(ecstore): gate batch read version rpc * feat(ecstore): observe batch processor adaptation * feat(ecstore): gate batch processor observation * docs: add get benchmark regression analysis * docs: add issue 797 execution plan status * fix(ecstore): require explicit batch rpc support * fix(ecstore): honor documented batch read gate * fix(ecstore): keep batch read gate stable per call * chore: update workspace dependencies * feat(ecstore): log batch read gate decisions * feat(ecstore): count batch read gate decisions * test(issue-797): add local internode A/B runner * test(rio): fix tuning profile spelling fixture * fix(protocols): adapt sftp channel open callbacks * fix(metrics): wrap batch processor observation args * chore(docs): keep issue notes local only * fix(storage): address internode review feedback * fix(storage): address internode data-path review findings - Run the BatchReadVersion auto-mode unary fallback outside the batch RPC deadline so each read_version keeps its own per-op timeout and health accounting instead of racing the whole batch against one drive timeout. - Cap adaptive batch-processor concurrency growth at a hard multiple of the configured baseline so sustained fast batches cannot ratchet past the configured limit. - Parse RUSTFS_INTERNODE_HTTP_* tuning, RUSTFS_BATCH_PROCESSOR_ADAPTIVE, and RUSTFS_METADATA_BATCH_READ once per process instead of re-reading the environment on hot paths. - Skip shard read-cost collection in observe mode when stage metrics are disabled, and cache the local endpoint host list instead of rebuilding it on every read. - Allow --warp-extra-args values starting with -- and drop the unused warp_hosts_csv helper in the issue-797 A/B runner. * fix(storage): address internode data-path review findings - Run the BatchReadVersion auto-mode unary fallback outside the batch RPC deadline so each read_version keeps its own per-op timeout and health accounting instead of racing the whole batch against one drive timeout. - Cap adaptive batch-processor concurrency growth at a hard multiple of the configured baseline so sustained fast batches cannot ratchet past the configured limit. - Parse RUSTFS_INTERNODE_HTTP_* tuning, RUSTFS_BATCH_PROCESSOR_ADAPTIVE, and RUSTFS_METADATA_BATCH_READ once per process instead of re-reading the environment on hot paths. - Skip shard read-cost collection in observe mode when stage metrics are disabled, and cache the local endpoint host list instead of rebuilding it on every read. - Allow --warp-extra-args values starting with -- and drop the unused warp_hosts_csv helper in the issue-797 A/B runner. Co-Authored-By: heihutu<heihutu@gmail.com> * fix(storage): align buffer clamp test with media cap * fix(ecstore): release optimized read locks before streaming --------- Co-authored-by: Zhengchao An <anzhengchao@gmail.com> |
||
|
Zhengchao An
|
9dfeffc4c1 |
test: prune redundant cases and add high-risk coverage across crates (#4208)
Remove or consolidate 57 test cases that cannot catch regressions (literal-constant asserts, construct-then-assert, derived-serde round-trips, near-duplicate env/getter matrices) in common, config, iam, madmin, and object-capacity, keeping all wire-format and error-path guards. Add 13 tests for previously uncovered high-risk behavior: filemeta version-sort determinism and merge resilience to garbage headers, zip extraction path-traversal rejection and exact limit boundaries, JWT tampered-signature rejection, and the bytes variant of dual-key (rustfs/minio) metadata fallback and precedence. Test-only change; no production code touched. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
cxymds
|
cf33bcc742 | fix(ecstore): hold read locks for streaming readers (#4195) | ||
|
houseme
|
70e1d79dfd |
feat: replace jemalloc with mimalloc (#4174)
* feat: replace jemalloc with mimalloc * docs: record allocator rounds5 retest * fix(replication): satisfy clippy unwrap lints * docs: keep allocator migration plan local only * feat(profiling): rely on pyroscope cpu profiling * refactor(profiling): centralize unsupported pprof responses * chore(deps): update s3s revision |
||
|
Zhengchao An
|
8284e6a75c | config: add mmap read env alias (#4021) | ||
|
Zhengchao An
|
aa8b8b5706 |
docs: document crypto RC version dependencies (#732) (#4011)
* docs: document crypto RC version dependencies Add comment explaining why aes-gcm and chacha20poly1305 use RC versions and the migration path when stable versions are released. Refs #732 * fix: remove all backlog links from code and comments |
||
|
Zhengchao An
|
7238a937a9 | fix: correct misleading zero-copy/direct-io docs and internal naming (#733 Phase 1) (#3976) | ||
|
houseme
|
27468ebfa9 |
feat(get): consolidate GET performance optimization (#3972)
* feat(get): consolidate GET performance optimization Consolidated implementation of all GET performance optimizations into a single, well-organized commit replacing the previous patch-on-patch approach. ## Changes ### Configuration (set_disk/mod.rs) - Consolidated all GET optimization flags into a single organized section - Enabled by default: codec streaming, metadata early-stop, page cache reclaim - Added codec streaming multipart flag (default: disabled) - Added version-aware early-stop flag (default: disabled) - Added adaptive duplex buffer sizing based on object size - All flags use OnceLock caching with rollout percentage support ### Metadata Early-Stop (set_disk/read.rs) - Delete marker early-stop when quorum agrees - Version-aware early-stop for versioned GET requests - MetadataQuorumAccumulator enhanced with: - delete_marker_votes tracking - requested_version_id and matching_version_votes tracking - version_early_stop_decision() method - 6 new tests for version early-stop scenarios ### Codec Streaming (erasure/coding/decode_reader.rs) - DualInFlight (2-stripe lookahead) enabled by default ### Decode Pipeline (erasure/coding/decode.rs) - Stripe prefetch count configuration - Bitrot-decode overlap configuration ### Disk Layer (disk/local.rs) - O_DIRECT read configuration constants (preparation) ### Metrics (io-metrics/lib.rs) - BytesPool acquisition/return metrics - Metadata phase duration with early-stop label - Total duration with reader_path label ### Diagnostics (diagnostics/) - Early-stop reason constants - Pool tier/outcome label constants ### Observability (.docker/observability/) - 3 Grafana dashboards for GET optimization monitoring - Prometheus alert rules (6 alerts: 3 critical, 3 warning) - Updated README.md and README_ZH.md with usage docs ### Config (config/src/constants/runtime.rs) - Page cache reclaim read enabled by default ## Environment Variables | Variable | Default | Description | |----------|---------|-------------| | RUSTFS_GET_CODEC_STREAMING_ENABLE | true | Codec streaming base flag | | RUSTFS_GET_CODEC_STREAMING_ROLLOUT_PCT | 100 | Codec streaming rollout % | | RUSTFS_GET_CODEC_STREAMING_MULTIPART_ENABLE | false | Multipart codec streaming | | RUSTFS_GET_METADATA_EARLY_STOP_ENABLE | true | Early-stop base flag | | RUSTFS_GET_METADATA_EARLY_STOP_ROLLOUT_PCT | 100 | Early-stop rollout % | | RUSTFS_GET_METADATA_VERSION_EARLY_STOP_ENABLE | false | Version-aware early-stop | | RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE | true | Page cache reclaim | | RUSTFS_OBJECT_DIRECT_IO_READ_ENABLE | false | O_DIRECT (preparation) | | RUSTFS_GET_DECODE_STRIPE_PREFETCH_COUNT | 1 | Stripe prefetch | | RUSTFS_GET_BITROT_DECODE_OVERLAP_ENABLE | false | Bitrot-decode overlap | | RUSTFS_GET_CODEC_STREAMING_MAX_INFLIGHT | 2 | DualInFlight stripes | ## Rollback All optimizations can be disabled via environment variables: RUSTFS_GET_CODEC_STREAMING_ENABLE=false RUSTFS_GET_METADATA_EARLY_STOP_ENABLE=false RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE=false Co-Authored-By: heihutu <heihutu@gmail.com> * test(get): add stress test scripts for GET optimization validation - quick-validate-get-optimization.sh: Quick 5-minute validation - stress-test-get-optimization.sh: Full 30+ minute stress test - README-stress-test.md: Usage documentation Co-Authored-By: heihutu <heihutu@gmail.com> * test(ecstore): align file cache reclaim defaults * chore(deps): update redis and erasure codec * test(ecstore): align decode fill policy default * test(ecstore): align metadata early-stop default * fix(ecstore): keep metadata early stop opt-in --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
Zhengchao An
|
e1a4b9e0b6 | refactor: batch cluster lock and health readiness (#3936) | ||
|
cxymds
|
7820a55fdc |
fix: decouple readiness from cluster health (#3912)
* fix: include foreground write pressure * fix: split readiness and cluster health probes * fix: publish ready on node readiness * fix: bound cluster health collection * test: pin health probe collector routing * fix: qualify app storage ECStore type * test(admin): narrow runtime capabilities topology assertion --------- Co-authored-by: houseme <housemecn@gmail.com> |
||
|
cxymds
|
90638bfc19 | fix(heal): add backpressure to repair admission (#3900) | ||
|
cxymds
|
eef8f43251 | fix: harden lock timeouts, health, and heal paths (#3815) | ||
|
安正超
|
b4524033e3 | refactor(config): move global config accessors (#3360) | ||
|
安正超
|
2205991180 | refactor(config): extract server config model (#3351) | ||
|
Henry Guo
|
62e9c5b94f |
feat(scanner): add adaptive pacing controls (#3315)
* feat(scanner): add adaptive pacing controls * fix(scanner): bound pacing delay status precision --------- Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com> |
||
|
Henry Guo
|
e79c793803 |
feat(heal): add scanner-aware bitrot controls (#3297)
* feat(heal): add scanner-aware bitrot controls * fix(config): register heal defaults * test(admin): avoid heal config queue conflict --------- Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com> Co-authored-by: houseme <housemecn@gmail.com> |
||
|
Henry Guo
|
6850457707 |
feat(scanner): add runtime scanner controls and status (#3203)
* feat(scanner): add runtime scanner controls and status * fix(scanner): validate persisted scanner config * docs(scanner): clarify start delay behavior --------- Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com> Co-authored-by: houseme <housemecn@gmail.com> |
||
|
Henry Guo
|
ad1a489f75 |
feat(scanner): add scanner budget progress controls (#3185)
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com> |
||
|
houseme
|
f49827fc58 |
fix(iam): prevent transient IAM walk timeout from crashing startup (#3188)
* fix(iam): prevent transient IAM walk timeout from crashing startup
IAM startup performs a blocking full metadata walk on `.rustfs.sys/config/iam/`.
When that distributed walk times out (e.g. disk pressure after cluster reboot),
the old code treated the failure as fatal and exited the process, causing a
systemd restart loop.
Changes:
- Add `startup_iam.rs`: attempt IAM init, enter degraded mode on failure,
spawn background retry task with exponential backoff (5s→10s→20s→30s cap)
- Log level escalates to ERROR after 12 retries (~5 min) to aid diagnosis
- `/health/ready` returns 503 until IAM recovers; IAM-dependent ops return
`IamSysNotInitialized` (existing fail-closed behavior preserved)
- Fix admin path boundary matching: `/minio/administrator` no longer falsely
matches as admin prefix
- Normalize Content-Length: 0 for admin GET requests with empty body
Fixes #3175
* fix(iam): move constant assertion into const block
Fixes clippy::assertions-on-constants warning on
IAM_RETRY_ESCALATION_THRESHOLD assertion.
* fix(iam): address PR review comments
- Replace OnceLock with AtomicU64 sentinel for test isolation;
add reset_test_failure_counter() for integration tests
- Use u32::try_from() instead of `as u32` narrowing cast in
compute_backoff_interval
- Rename misleading test; update to verify finalize retry behavior
- Restructure spawn_iam_recovery_task into init-retry and
finalize-retry phases so transient readiness failures are retried
instead of leaving the server permanently degraded
* fix(iam): gate test hooks behind debug_assertions
- reset_test_failure_counter() now stores sentinel (u64::MAX) to
correctly trigger env var re-read on next call
- RUSTFS_TEST_IAM_FAIL_INIT_ATTEMPTS only honored in debug builds
- RUSTFS_TEST_IAM_RETRY_INTERVAL_MS only honored in debug builds
* test(iam): cover deferred bootstrap recovery
Add a dedicated embedded deferred-IAM integration test in a separate test binary to avoid process-global startup collisions.
Strengthen startup IAM recovery coverage with focused unit tests and keep the existing embedded smoke test isolated while carrying the manual test license header update in the same change set.
* fix(startup): tighten deferred IAM recovery path
Adopt follow-up review feedback by silencing misleading app-context warnings after IAM recovery, reusing boundary-aware path prefix checks in the readiness gate, and tying deferred IAM recovery retries to server shutdown tokens.
Keep the deferred IAM embedded integration coverage and startup recovery unit coverage green after the follow-up hardening.
* refactor(startup): simplify IAM recovery task
Collapse the deferred IAM recovery implementation back to a concrete production flow instead of keeping boxed callback seams in the runtime path.
Keep only stable backoff unit coverage in startup_iam and rely on the embedded deferred bootstrap integration test for end-to-end recovery behavior.
* refactor(startup): trim IAM recovery test scaffolding
Keep the concrete deferred IAM recovery path intact while removing bulky test-only async loop scaffolding from startup_iam.
Retain the stable backoff unit checks and rely on the embedded deferred bootstrap integration test for end-to-end recovery coverage.
* fix: apply code review improvements from PR #3188 review
- Simplify RecoveryFuture type alias by removing unnecessary lifetime
- Fix finalize_iam_recovery to return Err if app context unavailable
- Update bootstrap_or_defer_iam_init doc comment to reflect Err case
- Use boundary-aware has_path_prefix for admin path matching in utils.rs
- Add test for adminx boundary rejection in utils.rs and layer.rs
- Improve embedded deferred IAM test with timeout wrapper
* style: merge has_path_prefix import into existing use block
* fix(iam): address final review follow-ups
- fix main startup readiness publication to pass ServiceStateManager correctly
- centralize IAM test env keys in rustfs_config and reuse them in runtime/tests
- keep deferred IAM bootstrap validation aligned with the final review fixes
* fix: isolate listing timeouts from drive health
Keep walk_dir scanner timeouts request-scoped instead of marking local drives faulty.
Add regression coverage for follow-up bucket info, set-level list_path, and system-prefix listings after prior walk timeouts.
* test(iam): gate deferred bootstrap test to debug
Align the deferred IAM embedded integration test with debug-only IAM fault injection hooks so release-profile runs do not assert deferred bootstrap behavior that cannot be triggered.
* test(ecstore): bound prior walk timeout regressions
- set walk_dir stall timeout explicitly in prior-timeout listing tests
- keep the system-prefix follow-up listing scoped to the same base dir
- assert the expected directory entry so the timeout regression test stays fast and stable
* fmt
|
||
|
Alexander Kharkevich
|
ce6fcf39b1 |
feat(oidc): add HIDE_FROM_UI option to exclude providers from console login (#3162)
Add `RUSTFS_IDENTITY_OPENID_HIDE_FROM_UI[_<SUFFIX>]` setting that removes a provider from the login page while keeping it fully functional for STS AssumeRoleWithWebIdentity and site-replication. Changes: - Add `hide_from_ui: bool` to `OidcProviderConfig` - Add `list_visible_providers()` that filters hidden providers (used by console login and /v3/oidc/providers endpoint) - Keep `list_providers()` unfiltered for site-replication/admin config - Extract `normalize_provider_config(config) -> config` to deduplicate field normalization (accepts the struct directly, not 18 parameters) - Add `parse_enable_state()` helper for consistent EnableState parsing - Plumb through admin API request structs (`#[serde(default)]`) - Expose in `OidcConfigView` for admin GET config round-trip - Persist via `upsert_persisted_provider_config()` Note: adding `hide_from_ui` to the public `OidcProviderConfig` struct is a source-level change for code constructing it with struct literals. This is acceptable for the current pre-1.0 release cycle. Signed-off-by: Alexander Kharkevich <alex@mara.com> Co-authored-by: GatewayJ <835269233@qq.com> |
||
|
Henry Guo
|
cc07946782 |
feat(scanner): add cycle budget observability (#3166)
* feat(scanner): add cycle budget observability * fix(scanner): clear cycle state after budget stop * test(scanner): stabilize timeout-based scanner tests * fix(scanner): keep cycle ILM counts scanner-only * test(scanner): avoid test-only pending import --------- Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com> Co-authored-by: houseme <housemecn@gmail.com> Co-authored-by: 安正超 <anzhengchao@gmail.com> |
||
|
houseme
|
29fbdc2dbf |
feat(ecstore): add object lock diagnostics (#3178)
* feat(ecstore): add object lock diagnostics Add configurable namespace lock diagnostics for object operations so production contention can be traced by operation, owner, and key. Wrap object read/write lock acquisition in diagnostic guards across get, head, put, delete, copy, and multipart flows, and log slow acquisition and long hold durations behind new RUSTFS_OBJECT_LOCK_DIAG_* settings. Verification: - make pre-commit * feat(obs): expose object lock diagnostics metrics Add Prometheus metrics and Grafana panels for object namespace lock diagnostics, covering slow acquire counts, slow hold counts, acquire duration, hold duration, and the diagnostics-enabled state. Adopt PR review feedback by keeping diagnostic guards alive through the guarded operation so long-hold warnings and metrics are emitted, reusing shared env helpers, and reducing default-path overhead when diagnostics are disabled. Verification: - cargo check -p rustfs-ecstore -p rustfs-io-metrics - cargo test -p rustfs-ecstore store::object -- --nocapture - make pre-commit * perf(obs): reduce object lock diag overhead Avoid repeated environment parsing on hot object-lock paths by caching the diagnostics-enabled flag, and stop allocating label strings for object lock metrics by recording static labels directly. Strengthen io-metrics tests by using a local recorder and asserting that the expected object lock diagnostic counters, gauges, and histograms are emitted. Verification: - cargo check -p rustfs-ecstore -p rustfs-io-metrics - cargo test -p rustfs-io-metrics -- --nocapture - make pre-commit |