mirror of
https://github.com/rustfs/rustfs.git
synced 2026-07-26 16:28:15 +00:00
eaff17cadeae38ef0195f9a7c24e133472debef8
4142 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 houseme
|
eaff17cade | perf(ecstore): add strict/relaxed/none durability modes over the drive-sync switch (#4397) | ||
|
houseme
|
92bf55ce62 | perf(ecstore): right-size BytesMut encode ingest capacity to the EC-expanded block (#4396) | ||
|
Zhengchao An
|
afc7f1d6f9 |
fix(ecstore): make post-commit old data dir cleanup best-effort (#4386)
* fix(ecstore): make post-commit old data dir cleanup best-effort (backlog#898) A write is authoritatively committed once rename_data returns Ok (the new version is durable on >= write_quorum disks and immediately readable). The subsequent reclamation of the now-dereferenced old object/<data_dir> is pure space reclamation, yet commit_rename_data_dir propagated a below-quorum GC failure via `?` into ErasureWriteQuorum -> 503, producing a false-negative ACK for an already-persisted write. This is a deliberate divergence from MinIO (erasure-object.go:1577), which couples the two; the divergence is justified by durability semantics, not parity. Changes: - commit_rename_data_dir now returns a structured OldDataDirCleanup receipt and never returns Err. Adds an old==committed-dir anti-misdelete guard and a committed_data_dir parameter. Classification is extracted into pure functions (classify_old_data_dir_cleanup / map_cleanup_join_result / is_cleanup_not_found) so it is unit-testable. Task panic/cancel is mapped to a non-ignored DiskError::other (never DiskNotFound), and not-found is normalized to reclaimed. - object.rs / multipart.rs consume the receipt instead of `?`. The result reverts to Ok, so the invalidate_get_object_metadata_cache self-heal and the capacity/compression accounting that a `?` early-return previously skipped now run on the cleanup-failure path too. - On residue, report_old_data_dir_cleanup emits leak metrics and enqueues an object heal over the existing heal channel (disk-health signal replacing the 503). heal_object -> reclaim_orphan_data_dirs already reclaims unreferenced local data dirs, closing the loop end to end. - Adds rustfs_old_data_dir_* counters (attempted/reclaimed/leaked/below_quorum) as the operator-visible backstop for leaked residue. - Adds a test-only (#[cfg(test)]) delete fault-injection seam; in production it inlines to a no-op None and has no behavioral effect. Tests: pure-function A/C group + join-error mapping + actions decision; A5/A5b real-disk guard/reclaim integration; end-to-end overwrite returning 200 while old-data-dir cleanup fails. #864 rollback guard test remains green. * fix(ecstore): resolve merge conflicts with origin/main in io_primitives.rs --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: houseme <housemecn@gmail.com> |
||
|
Zhengchao An
|
07324e268b |
fix(protocols): allow clippy::type_complexity in expiration worker tests (#4395)
fix(protocols): allow clippy type_complexity in test mock struct The MockExpirationObjectBackend test struct uses a nested generic type that triggers clippy::type_complexity. Add #[allow(clippy::type_complexity)] since this is test-only code where the type is inherent to the mock design. Co-authored-by: houseme <housemecn@gmail.com> |
||
|
houseme
|
3f13d098b4 |
feat(observability): feature-gated hotpath instrumentation for the data path (#4394)
Merge the hotpath-rs wall-time instrumentation from the backlog#936 analysis worktree behind an opt-in 'hotpath' cargo feature, keeping the default build at zero overhead and zero dependency. - hotpath is an optional dependency everywhere (dep:hotpath feature syntax); the default dependency tree contains no hotpath crate at all - 40+ measurement points across S3 handlers, ECStore/SetDisks object and multipart ops, erasure encode/decode, bitrot, LocalDisk I/O, FileMeta codec, and HashReader - attribute sites use #[cfg_attr(feature = "hotpath", hotpath::measure)]; async_trait bodies use per-crate hp_guard! macros (ecstore + rustfs bin); rio gates measure_block! behind hp_measure_block! - feature chain: rustfs -> rustfs-ecstore -> rustfs-rio / rustfs-filemeta, each crate owning its own gate - hotpath-alloc is intentionally not wired up (hotpath 0.21.x TLS panic on cross-thread guard drop under tokio, see backlog#935); mimalloc stays the unconditional global allocator - docs/development/hotpath-profiling.md documents building, HOTPATH_* env vars, SIGTERM report flow, and how to reproduce the backlog#936 timing reports Refs: https://github.com/rustfs/backlog/issues/935 (HP-14, item 2), https://github.com/rustfs/backlog/issues/936 Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
bd5d3c5d92 |
perf(ecstore): data-shards-only lockstep GET reads with stripe-aligned deferred parity engagement (opt-in) (#4392)
* feat(ecstore): add stripe-advance handles for deferred bitrot readers Give DeferredObjectReader a shared pending state and expose a DeferredReaderStripeHandle that advances the still-unopened source by whole bitrot blocks using the same bitrot_encoded_range geometry the reader was created with (identity mapping when hash_size == 0). This lets the GET decode path open a parity shard aligned to the stripe where a data shard failed instead of reading every parity shard on every stripe (backlog#923). An already-opened (or failed) reader rejects the advance so callers retire it rather than engage it out of alignment; bitrot verification after an advance checks the advanced stripe's block against that stripe's stored hash. Co-Authored-By: heihutu <heihutu@gmail.com> * perf(ecstore): read only data shards on healthy lockstep GET behind opt-in gate PR #4289's lockstep fix made every reconstruction-verifying GET read all data+parity shards per stripe; the parity blocks are read, bitrot-hashed and then discarded, a deterministic 2x read-bytes/IOPS/hash-CPU amplification on healthy 2+2 objects (backlog#923). With the new opt-in gate RUSTFS_GET_LOCKSTEP_DATA_SHARDS_ONLY_ENABLE=true (default: false, behavior identical to main): - read_lockstep keeps only the data slots engaged while the object is healthy; parity slots stay unopened deferred readers. - When a data shard is missing or dies at stripe k, parity readers are engaged mid-object by advancing their deferred stripe handle to stripe k, preserving the lockstep alignment invariant from backlog#832. - Degraded stripes engage one parity beyond the decode quorum so reconstruction verification keeps an extra source to check against (erasure.rs only verifies when available > data shards); an engaged parity reader that errors is retired for the rest of the object like any other, and a parity reader that cannot be realigned is retired instead of being read out of position. - fill_deferred_bitrot_readers records stripe handles for deferred slots and, gate-on only, swaps eagerly opened parity readers for unopened deferred ones so they remain engageable mid-object; ready/error bookkeeping used by quorum decisions is untouched. - Both GET paths (legacy duplex via Erasure::decode_with_stripe_handles, codec streaming via ParallelReader::with_deferred_parity_handles) carry the handles from reader setup. Short-read -> UnexpectedEof -> whole-object retirement and the inconsistent-source rejection are unchanged in both gate modes; tests lock the healthy-path data-shards-only call counts, the default read-all-shards behavior, mid-object parity engagement for streaming and hash_size==0 formats, and mid-stream inconsistent-parity rejection. Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
Zhengchao An
|
65953bfdb3 | fix(ecstore): reduce rename data version signatures (#4383) | ||
|
Zhengchao An
|
ddf197ba57 |
fix(lock): refresh object write locks with a heartbeat (#4388)
fix(lock): renew distributed locks via heartbeat; wire server refresh (backlog#899) A held distributed write lock had a fixed 30s TTL and was never renewed, so any operation exceeding it got its per-node lease reclaimed and stolen by a contender, causing split-brain writes. This implements Phase 0 + Phase 1 of the #899 design (Phase 2 abort-on-loss is deferred and tracked in code comments). Phase 0 (server refresh wiring, P0): - node_service handle_refresh was a no-op stub that parsed args then always returned success=true. Extract a testable `refresh_lock` free function that actually delegates to the node's lock backend. Not-found maps to success=false with no error_info, so RemoteClient::refresh keeps its Ok(resp.success) semantics and yields a real not_found signal to the coordinator heartbeat. Without this, client heartbeats were silently no-ops. Phase 1 (client heartbeat + safe interval + observability): - DistributedLockGuard spawns a heartbeat that refreshes every per-client lease on a derived interval; interval is derived without Duration::clamp (entries<=1 or interval>=ttl => no spawn), fixing the sub-second-ttl panic. - Add LockLostSignal: declare the lock lost when not_found exceeds entries.len() - refresh_quorum; RPC errors are not counted (absorbed by the ttl > interval margin). Expose is_lock_lost()/lock_lost() for observers. - disarm(), release(), and Drop now abort the heartbeat before releasing so no refresh races the unlock (refresh only extends, never creates, a lease). - Reclaim path stays behaviorally unchanged but now warns with owner/resource/ lease age and records a metric (#698 scavenger preserved). - Add DEFAULT_LOCK_REFRESH_INTERVAL, LockRequest.refresh_interval (serde default for RPC back-compat) + builder, and lock lifecycle metrics. Open questions adopt the design's documented defaults (marked TODO in code): refresh not-found -> Ok(false)/error_info=None; lost-quorum base entries.len(); DEFAULT_LOCK_REFRESH_INTERVAL=10s. Tests: heartbeat keepalive/quorum-loss/jitter/boundary/disarm (lock crate), server refresh delegation (rustfs), and end-to-end survives-past-ttl plus crashed-owner-reclaim regressions (namespace). |
||
|
Zhengchao An
|
45435d83ab |
fix(keystone): accept Swift storage tokens (#4390)
fix(keystone): accept swift storage tokens |
||
|
Zhengchao An
|
f021e4f321 | fix(protocols): simplify swift expiration test types (#4385) | ||
|
houseme
|
e7cc719c17 |
perf(ecstore): move speculative PUT-tail tmp cleanup off the hot path (#4389)
* perf(ecstore): move speculative tmp cleanup off the PUT hot path On a successful PUT, rename_data has already moved the data dir out of the tmp workspace, so the delete_all(RUSTFS_META_TMP_BUCKET) at the end of SetDisks::put_object is a speculative no-op safety net. It was awaited inline on the response path, where profiling (backlog#924 / HP-3) showed the same-disk queueing behind fsync-heavy load turns a ~49us no-op into ~9ms average (p99 77ms, macOS F_FULLFSYNC amplified) added to every PUT. Run that cleanup on a spawned task instead, keeping it as a real backstop (rename_data's remove_std only removes empty dirs and silently ignores failures). The failure path (quorum loss / rollback) keeps the cleanup inline so a failed PUT never returns with tmp shards still on disk. If the process dies before the spawned task runs, cleanup_stale_tmp_objects (24h expiry, 5-minute loop) reclaims the entry. Scope note: ops/multipart.rs delete_all on RUSTFS_META_MULTIPART_BUCKET is intentionally untouched; it removes real leftovers and deferring it would widen CompleteMultipartUpload/Abort races. Regression tests (hermetic SetDisks on formatted local disks, no global state): PUT success drains the tmp workspace (polling the spawned task), and PUT failure (missing bucket volume, rename_data quorum error after tmp shards were written) cleans the workspace inline before returning. Ref: https://github.com/rustfs/backlog/issues/924 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): do not retry NotFound in reliable_rename_inner reliable_rename_inner blindly retried the rename once on any error. A NotFound retry cannot succeed: nothing recreates the missing source or parent directory between attempts, so the second rename fails identically and speculative cleanup renames (e.g. move_to_trash on an already-removed tmp path) always paid for two syscalls. Extract the retry decision into should_retry_rename: NotFound returns immediately, any other error keeps the existing single retry. This helper is shared by the rename_data commit path via rename_all, so behavior there is covered by a new rename_all success regression test alongside the retry-predicate tests. Ref: https://github.com/rustfs/backlog/issues/924 Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
062a68d151 |
perf(ecstore): skip tmp parent dir fsync for write-then-rename paths (#4387)
Step 1 of 4 for rustfs/backlog#922 (HP-1): write_all_internal previously coupled file-content durability (fdatasync) with directory-entry durability (parent dir fsync) behind a single bool. For tmp files that are immediately renamed away, the tmp parent dir fsync contributes nothing to crash consistency: the safe-rename recipe only needs file content fdatasync -> rename -> fsync of the destination parent, because the rename removes the tmp directory entry and a crash before the rename means the PUT was never acknowledged. Replace the bool with a SyncMode enum (None / FileAndDir / FileOnly) and use FileOnly at exactly the two write-then-rename tmp write points: the tmp xl.meta write in the non-inline rename_data path and the tmp write inside write_all_meta. write_all_public (format.json etc.) and the old-metadata rollback backup keep FileAndDir since those files stay where they are written. The rest of the commit sequence (shard sync_dir_files, rename, destination parent fsync) is untouched, and behavior with drive sync disabled is unchanged. This saves one directory fsync per disk per non-inline PUT (4 on a 4-drive set). Unit tests assert, via a test-only fsync-dir recorder, that tmp write points no longer fsync their parent while the public write point, the rollback backup, and the commit-rename destination parent still do. Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
9ae4ca5f99 |
test(ecstore): fit concurrent-resend guard inside lock acquire ceiling (#4384)
Under a full nextest run on loaded machines the legitimately serialized cross-disk commits in concurrent_resend_same_part_commits_one_generation exceed the acquire deadline by themselves: observed at the 5s default, at the 30s production default (#4370), and on CI even at 60s — which is a hard ceiling, because fast_lock clamps every requested timeout to MAX_ACQUIRE_TIMEOUT (60s) while the Timeout error still reports the requested value, so raising the env override higher is a no-op. Request the full 60s ceiling explicitly and cap the queue depth at three concurrent resends, bounding the last waiter to two serialized commits (~12s each on the slowest observed CI runner). Three resends still race the streaming phase and contend on the commit lock, which is all the generation-mixing regression (backlog#853) needs; every correctness assertion is unchanged. Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
86940a9452 |
fix(protocols): resolve clippy type_complexity in swift expiration worker tests (#4393)
fix(protocols): factor swift expiration mock result type into alias The swift feature clippy matrix on main fails with clippy::type_complexity on the MockExpirationObjectBackend test struct introduced with the expiration worker tests, blocking CI for every open PR. Introduce a MetadataResult type alias in the test module; no behavior change. Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
Zhengchao An
|
7efacbdf95 |
fix(filemeta): validate part array lengths in into_fileinfo (#4382)
MetaObject::into_fileinfo indexed part_sizes[i]/part_actual_sizes[i] by part_numbers.len() without checking the arrays are the same length, unlike the adjacent part_etags/part_indices which are length-guarded. decode_from pushes the three arrays independently and the xl.meta CRC only covers bytes, so a CRC-valid but internally inconsistent xl.meta (foreign writer / MinIO interop) triggers an out-of-bounds panic on the GET/HEAD/LIST decode path. Guard the three arrays for equal length and return Err(FileCorrupt) so a divergent shard is skipped and quorum uses the other disks, instead of panicking the request task. Cascade into_fileinfo to Result across its callers, and fix io_primitives early-return to derive the version id from the merged header and fall into the per-disk loop (single-disk survival + heal). The 2118 merge-first path is left as a documented follow-up. Refs backlog#900 (filemeta-01). |
||
|
Zhengchao An
|
a91d9cefc6 |
test(interop): add real MinIO read and migration parity tests (#4377)
test(interop): real-MinIO read + migration parity, Phase 1/2 (backlog#580) Capture authentic on-disk fixtures from MinIO RELEASE.2025-07-23 (a bucket with versioning, object-lock, lifecycle, tagging, quota, a public policy, SSE-S3 encryption, a webhook notification target, and a replication rule, plus inline / versioned / multipart objects and a delete marker) and prove RustFS reads and migrates them losslessly: - filemeta parses_real_minio_object_xlmeta: small inline, two-object-version + delete marker, and multipart object xl.meta parse to the expected FileInfo. - ecstore parses_real_minio_bucket_metadata_blob_without_loss: the MinIO .metadata.bin msgpack decodes via the PascalCase field names and parse_all_configs loads all ten config types present (policy, lifecycle incl. <ExpiryUpdatedAt>, object-lock, versioning, tagging, quota, notification, encryption/SSE-S3, replication incl. DeleteMarkerReplication / ExistingObjectReplication) without loss. - ecstore reads_minio_inline_bucket_metadata_via_bitrot: MinIO inlines an object body as [HighwayHash256 32B][body]; RustFS's BitrotReader with HighwayHash256S verifies and yields the exact blob (the "inline_data 前缀不同" is that prefix). - ecstore migrates_real_minio_bucket_metadata_end_to_end: on a throwaway 4-drive local ECStore, a real MinIO .metadata.bin seeded under .minio.sys is migrated into .rustfs.sys byte-identically for every config, exercising the Phase 2 source adapter (MIGRATING_META_BUCKET = ".minio.sys") through the object layer. All four run as ordinary crate tests (nextest CI). Phase 4 (MinIO re-reading a RustFS drive) is documented as out of scope for one-way migration. Refs rustfs/backlog#580 |
||
|
Zhengchao An
|
62a31e4ec4 | fix(storage): address pending metadata and health gaps (#4380) | ||
|
houseme
|
d3660f9ded |
refactor(config): remove dead Direct I/O constants from zero_copy.rs (#4379)
ENV_OBJECT_DIRECT_IO_ENABLE, DEFAULT_OBJECT_DIRECT_IO_ENABLE, ENV_OBJECT_DIRECT_IO_THRESHOLD and DEFAULT_OBJECT_DIRECT_IO_THRESHOLD were never read anywhere in the workspace. The real O_DIRECT read path landed in PR #4365 uses RUSTFS_OBJECT_DIRECT_IO_READ_ENABLE / RUSTFS_OBJECT_DIRECT_IO_READ_THRESHOLD (crates/ecstore/src/disk/local.rs). Keeping the near-identically named dead knobs invites operators to set the wrong variable and believe O_DIRECT is enabled. Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
Zhengchao An
|
742a59884d | test(ecstore): add validation suite coverage gates (#4378) | ||
|
Zhengchao An
|
3ed414cdb4 | fix(storage): complete pending metadata and quorum fixes (#4375) | ||
|
houseme
|
5533e080b0 |
feat(ecstore): LocalIoBackend trait and O_DIRECT read with fallback (#4365)
* refactor(ecstore): extract LocalIoBackend trait behind LocalDisk Model the LocalDisk per-file I/O hot path as a LocalIoBackend trait (pread_bytes / open_read_stream / open_full_read / open_write) and move the existing method bodies verbatim into a default StdBackend. LocalDisk holds Arc<dyn LocalIoBackend> and the DiskAPI methods forward to it. This is a behavior-preserving refactor: no logic, error-mapping, metrics, or cfg-branch changes. It creates the seam for an alternative runtime-probed io_uring backend (rustfs/backlog#894) without touching DiskAPI callers. Commit-point durability (fdatasync -> rename -> fsync-dir in rename_data) deliberately stays outside the trait. Add a differential test asserting all four read shapes return identical bytes across page-boundary and file-tail ranges. Tracking: rustfs/backlog#891 (parent rustfs/backlog#897) Co-Authored-By: heihutu <heihutu@gmail.com> * feat(ecstore): true O_DIRECT read path with per-disk graceful fallback (#4366) * feat(ecstore): true O_DIRECT read path with per-disk graceful fallback Implement a real O_DIRECT positioned read inside StdBackend::pread_bytes (Linux only) and wire up the previously dead RUSTFS_OBJECT_DIRECT_IO_READ_* knobs, which had zero call sites. Open with rustix OFlags::DIRECT, probe the DIO alignment once per disk via statx STATX_DIOALIGN (4096 fallback), read the aligned superset into an alignment-allocated bounce buffer with a short-read loop, and slice out the exact logical range so padding never reaches BitrotReader. Any O_DIRECT failure falls back to the buffered read methods; EINVAL or EOPNOTSUPP (tmpfs, overlayfs, 9p) latches the path off per disk with one warning. O_DIRECT errors never surface: EINVAL maps to FileNotFound in to_file_error and would trigger spurious EC rebuilds. Default behavior is unchanged (knob off). macOS keeps F_NOCACHE. Tracking: rustfs/backlog#892 (parent rustfs/backlog#897) Co-Authored-By: heihutu <heihutu@gmail.com> * test(ecstore): add P1.5 benchmark gate harness for O_DIRECT reads (#4369) * test(ecstore): add P1.5 benchmark gate harness for O_DIRECT reads Add an ignored, Linux-only release-mode test (direct_read_bench_gate) that measures DiskAPI::read_file_mmap_copy through a real LocalDisk with cold-cache enforcement (fadvise DONTNEED between rounds), reporting p50/p95/p99/mean latency, wall time, process CPU time, and throughput as one JSON line for A/B diffing between the buffered baseline and the O_DIRECT candidate selected by the production env knobs. Content is verified byte-for-byte before any timing starts. Tracking: rustfs/backlog#893 (parent rustfs/backlog#897) Co-Authored-By: heihutu <heihutu@gmail.com> * test(ecstore): add concurrency dimension to P1.5 bench harness Model the EC GET shape (FuturesUnordered over concurrent shard reads) via RUSTFS_BENCH_CONCURRENCY (default 1, sequential as before). Needed to evaluate blocking-pool pressure for the io_uring gate decision. Tracking: rustfs/backlog#893 Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> * ci: retrigger after cancelled required check Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): use is_multiple_of in O_DIRECT bench cache-drop check clippy's manual_is_multiple_of (rust 1.96) fails -D warnings on the benchmark helper's `done % file_count == 0`. Verification: - cargo clippy -p rustfs-ecstore --all-targets Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
8f4349793e |
fix(ecstore): log peer offline/online transitions for console reporting (#4367)
A node could show OFFLINE in the console with zero logs explaining why (rustfs/backlog#888, reported in rustfs#4304): the consecutive-failure threshold crossing in handle_server_info_failure marked the peer offline silently, the per-call warnings only existed on the observing node and never named the transition, and the PeerRestClient offline flag plus its background recovery monitor ran without any start/success log. Logging-only change, no behavior change: - handle_server_info_failure / handle_storage_info_failure: WARN event="peer_marked_offline" exactly once at the threshold crossing (later failures while already offline stay DEBUG), and DEBUG event="peer_probe_failure" while returning cached state below the threshold. - update_server_info_cache / update_storage_info_cache: INFO event="peer_recovered_online" when a probe succeeds after the peer had been reported offline. - PeerRestClient::mark_offline_and_spawn_recovery: WARN event="peer_connection_marked_offline" when the offline flag is first set (guarded by the recovery_running CAS so repeated failures do not spam), and INFO event="peer_connection_recovered" with the attempt count when connectivity is restored. An "offline then back" episode now leaves a complete, correlatable trace: N probe failures -> peer_marked_offline -> recovery monitor -> peer_recovered_online. Verification: - cargo test -p rustfs-ecstore --lib (notification + peer_rest suites) - make pre-commit Ref: rustfs/backlog#888, rustfs#4304 Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
Zhengchao An
|
9e6d7de0fa |
docs(agents): forbid hard-wrapping prose in PR/issue bodies (#4374)
GitHub renders single newlines inside a paragraph as line breaks, so hard-wrapped PR/issue/discussion prose shows up with ugly mid-sentence breaks. Add a rule under Git and PR Baseline to keep each paragraph on one line. |
||
|
houseme
|
2dfa3d3c36 |
test(ecstore): restore 30s lock-timeout guard for concurrent resend test (#4370)
concurrent_resend_same_part_commits_one_generation keeps failing on CI with Lock(Timeout ... 5s) even after the fast-lock lost-wakeup fix (NOTIFY_WAIT_CAP re-polling) landed — observed on rustfs#4365's Test and Lint runs. Two independent causes exist and both are real: 1. the lost/stolen wakeup stall — fixed in fast_lock::shard; and 2. the six legitimately serialized cross-disk commits exceeding the small 5s default acquire timeout under full-suite CI disk load. The lost-wakeup fix reverted the earlier 30s test override on the assumption that cause 1 was the whole story; CI shows cause 2 stands on its own. Restore the temp_env override (production-default 30s) around the concurrent section so the regression guard asserts the correctness property (exactly one intact generation), not CI disk latency. The NOTIFY_WAIT_CAP fix stays untouched. Verification: - cargo test -p rustfs-ecstore --lib concurrent_resend_same_part_commits_one_generation Ref: rustfs/backlog#882 Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
7001373316 |
fix(iam): load IAM bootstrap snapshot without namespace locks (#4363)
* fix(iam): load IAM bootstrap snapshot without namespace locks
IAM bootstrap (init_iam_sys -> load_all) read every config object with
default ObjectOptions (no_lock=false), so each read acquired a
distributed namespace read lock. Lock quorum is counted over cluster
nodes and unreachable peers are hard failures, so during a sequential
restart the very first read failed with
"Quorum not reached: required 2, achieved 0" and IAM could not come up
until enough peers' lock RPC surfaces converged - even when the storage
read quorum was already satisfiable (rustfs#4304).
Extend the startup contract from rustfs#4056 ("startup metadata I/O must
not require namespace locks") to the IAM bootstrap path:
- Introduce LoadMode {Locked, BootstrapNoLock} and plumb it through the
load_all chain (groups, users, policies, mapped policies and their
concurrent variants) down to the storage read options.
- load_all now performs all reads with no_lock=true; on-line
single-object loads via the Store trait keep locked semantics.
- Fail-closed behavior is unchanged: any loader error still aborts the
whole snapshot load.
Safety: config objects are atomic whole-object writes, so a lock-free
read only observes an old or a new value; staleness is bounded by the
existing periodic IAM reload. Listing (walk) never took namespace locks,
and maybe_schedule_lazy_rewrite stays a best-effort background task.
Verification:
- cargo test -p rustfs-iam --lib (153 passed, incl. new LoadMode tests)
- cargo clippy -p rustfs-iam --all-targets
- cargo check -p rustfs
- make pre-commit
Ref: rustfs#4304; tracking rustfs/backlog#884, rustfs/backlog#885
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(iam): sequential-restart regression test for lock-free bootstrap
Add an integration test reproducing the rustfs#4304 failure mode against
a real 4-disk temp-dir ECStore:
- Seed IAM group data in single-node mode, then flip the runtime into
distributed-erasure mode. new_ns_lock now builds a distributed lock
over the set's (empty) lock-client list, so every namespace-locked
read fails exactly like a sequential restart with unreachable peers
(lock quorum unavailable, storage read quorum healthy).
- Assert the locked load_group path fails in that state, while the
bulk snapshot load_all (no_lock plumbing from the previous commit)
succeeds, and the data survives intact once single-node mode is
restored.
Reverse-verified: temporarily switching load_all back to the locked
mode makes the test fail, so it genuinely guards the contract.
Verification:
- cargo test -p rustfs-iam --test iam_bootstrap_no_lock_test
- cargo test -p rustfs-iam --lib
Ref: rustfs#4304; tracking rustfs/backlog#886
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(iam): route test ECStore imports through ecstore_test_compat boundary
The new integration test imported rustfs_ecstore facade paths directly,
tripping three architecture migration rules. Move every ECStore import
behind crates/iam/tests/ecstore_test_compat/mod.rs (the sanctioned
test-compat pattern), and register that module as a reviewed test-only
global-facade boundary in check_architecture_migration_rules.sh: the
sequential-restart regression test needs api::global::update_erasure_type
to flip into distributed-erasure mode for lock-quorum fault injection.
Verification:
- ./scripts/check_architecture_migration_rules.sh
- cargo test -p rustfs-iam --test iam_bootstrap_no_lock_test
- make pre-commit
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(server): expose readiness blocking reason + rolling-restart runbook
Operators hitting the rustfs#4304 sequential cold start could not tell
from the outside why a node stayed unavailable. Three additions:
- The readiness gate's 503 now names the blocking dependency in both the
body ("Service not ready: waiting for storage_quorum") and a new
x-rustfs-readiness-pending header (storage_quorum | iam |
startup_finalization), derived from the current startup stage.
/health/ready already returned details + degradedReasons; this covers
the plain S3 requests that hit the gate.
- IAM bootstrap retry logs now carry an actionable `hint` field that
classifies the failure (storage read quorum vs lock quorum vs
uninitialized metadata) instead of only echoing the storage error.
- New docs/operations/rolling-restart.md runbook: correct rolling
restart procedure, sequential cold-start expectations (degraded ->
auto-recovery), readiness signal reference, and
RUSTFS_STARTUP_READINESS_MAX_WAIT_SECS guidance.
Verification:
- cargo test -p rustfs --lib -- hint_tests service_not_ready readiness_pending
- make pre-commit
Ref: rustfs#4304; tracking rustfs/backlog#887
Co-Authored-By: heihutu <heihutu@gmail.com>
* upgrade deps version and improve import
* feat(iam): notification-path cache refreshes read without namespace locks (#4368)
P3 step 1 of rustfs/backlog#884 (scoped down from full MinIO readConfig
alignment after review): cross-node notification handlers
(group/policy/policy-mapping/user) refresh the local IAM cache with
single-object reads that previously took distributed namespace read
locks. These refreshes are asynchronous, best-effort, and already
stale-tolerant (the periodic reload converges them), so a node-counted
lock quorum failure or lock RPC hiccup on a peer must not fail them —
the same rationale as the lock-free bootstrap load_all (rustfs#4304).
- Store trait: add load_user_no_lock / load_group_no_lock /
load_policy_doc_no_lock / load_mapped_policy_no_lock with defaults
forwarding to the locked variants, so existing implementations and
test mocks keep their behavior.
- ObjectStore overrides them via the existing LoadMode::BootstrapNoLock
plumbing. Deletions triggered by the handlers keep locked writes.
- manager.rs: the four *_notification_handler paths (8 call sites)
switch to the lock-free variants.
- Integration test: while the lock quorum is unavailable (DistErasure
with empty lockers), load_group_no_lock must succeed exactly where
the locked load_group fails.
Request-path loads (check_key, verify_temp_user_persistence) and admin
write-then-reload paths intentionally stay locked: load_user_identity
embeds expiry deletions, so those need the side-effect extraction
tracked in rustfs/backlog#884 before going lock-free.
Verification:
- cargo test -p rustfs-iam --lib (156 passed)
- cargo test -p rustfs-iam --test iam_bootstrap_no_lock_test
- make pre-commit
Ref: rustfs/backlog#884, rustfs#4304
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(server): drop unused iam_bootstrap_failure_hint import in tests
The hint tests live in their own hint_tests module with a local import;
the stale re-import in mod tests failed clippy's -D warnings on the
Test and Lint CI variants.
Verification:
- cargo clippy -p rustfs --all-targets
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix
---------
Co-authored-by: heihutu <heihutu@gmail.com>
|
||
|
houseme
|
2247823200 |
fix(runtime): remove tokio io-uring feature and add regression guard (#4364)
Drop the [target.'cfg(target_os = "linux")'.dependencies] tokio "io-uring" feature from 6 crates and the rustfs binary. Because .cargo/config.toml enables --cfg tokio_unstable globally, this feature switched every Linux build's file I/O onto tokio's io_uring runtime backend. Restricted Linux environments (Docker default seccomp, gVisor, proot, old kernels) reject io_uring_setup with EACCES/ENOSYS, which tokio surfaced as PermissionDenied and RustFS reported as DiskAccessDenied at startup. Add scripts/check_no_tokio_io_uring.sh so the feature cannot silently return: it fails on any tokio dependency line enabling "io-uring", while still allowing a future application-level io-uring crate dependency. Wire it into make pre-commit/pre-pr/dev-check and CI. Tracking: rustfs/backlog#890 (parent rustfs/backlog#897) Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
Henry Guo
|
0e61ba7c63 |
test(table-catalog): add scale fault rehearsal (#4359)
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com> Co-authored-by: houseme <housemecn@gmail.com> |
||
|
houseme
|
717cdd2abd |
fix(migration): decrypt MinIO IAM & server config on drop-in migration (#4358)
* fix(migration): decrypt MinIO IAM & server config on drop-in migration MinIO encrypts IAM identity/service-account files and the server config at rest with a key derived from the root credentials. The drop-in migration paths read those blobs from the legacy `.minio.sys` bucket and parsed them as plaintext JSON, so any encrypted blob failed to parse and was silently skipped with "incompatible format". This is why users migrating from MinIO kept their buckets/objects/policies but lost users and access keys (#2212). The IAM load path already knows how to decrypt these blobs (RustFS master keys plus MinIO-compatible legacy keys derived from the root credentials), but that logic lived behind a private method and was never used by the migration paths. Expose it as `rustfs_iam::try_decrypt_iam_blob` and inject it into both migration paths via a `LegacyBlobDecryptFn` callback (ecstore cannot depend on the IAM crate, so the closure is wired in the binary crate). When a blob cannot be decrypted the raw bytes are used as-is, preserving the previous plaintext-only behavior with no regression. Also improve object-layer migration observability without changing control flow: `try_migrate_format` now distinguishes "no legacy format" (a normal fresh install) from "legacy format present but incompatible", and the caller logs a loud error before initializing a fresh format that would leave the existing MinIO objects unreadable. Topology/version skip reasons are promoted from debug to warn. Fixes a pre-existing test isolation race by marking `test_recovery_falls_back_to_default_config_when_blob_stays_corrupt` serial, since it reads a process-wide env var toggled by a sibling test. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(migration): box FormatV3 in LegacyFormatOutcome to satisfy clippy Co-Authored-By: heihutu <heihutu@gmail.com> * test(ecstore): stabilize concurrent multipart resend lock timeout concurrent_resend_same_part_commits_one_generation spawns 6 same-part resends whose cross-disk commits serialize on the per-uploadId commit lock. Under the full nextest suite the parallel disk load pushes those serialized commits past the small default lock-acquire timeout (5s), producing a spurious `Lock(Timeout ...)` unrelated to the property under test (observed on CI at 5.775s vs ~0.5s in isolation). Raise RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT to the production default (30s) for the concurrent-commit section via temp_env, so the regression guard reflects correctness (exactly one intact generation) rather than disk latency under CI load. The meaningful assertions are unchanged, and #[serial] keeps the process-wide env override isolated. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(lock): bound fast-lock notification wait to prevent lost-wakeup stall The real cause of the concurrent_resend_same_part_commits_one_generation failures was a lost wakeup in the fast-lock slow path, not disk latency: raising the acquire timeout to 30s only delayed the failure (it then timed out at 30s), proving a genuine stall rather than overload. In acquire_lock_slow_path a waiter that reaches the notification phase did a single `timeout(remaining, wait_for_write())` spanning the whole acquire budget, and treated that wait's elapse as a hard `Timeout`. But the release path only notifies when `writer_waiters > 0`, so if the holder releases in the gap after the waiter's `try_acquire` fails and before it registers as a waiter, no notification (and no stored permit, since the pooled `Notify` is gated) is produced. The waiter then blocks until the deadline even though the lock is free and stays free — a spurious lock-acquire timeout. The shared process-wide notify pool makes it worse: a wakeup can be consumed by a waiter of a different lock hashing to the same slot. Bound each notification wait (NOTIFY_WAIT_CAP = 50ms) and, on elapse, loop back and re-`try_acquire` instead of returning `Timeout`; the deadline check at the top of the loop is the single source of truth for timing out. A lost/stolen wakeup now degrades to bounded re-polling (acquire within ~50ms of the lock becoming free) instead of stalling for the whole timeout. Correctness (mutual exclusion) is unchanged — acquisition still only happens via `try_acquire_*`. Add a regression test that reproduces the stall (holder + late waiter across many keys): it times out without the fix and passes in ~1s with it. Revert the earlier acquire-timeout workaround in the multipart test now that the underlying stall is fixed, so it runs under the default timeout again. Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
majinghe
|
796fbb47da | ci: replace docker image build runner with aks dind container (#4361) | ||
|
houseme
|
88645c9169 |
fix(server): stop closing idle proxy keep-alive connections + reverse-proxy hardening (#3076) (#4360)
* fix(server): raise default HTTP/1.1 idle keep-alive timeout for proxies In hyper's HTTP/1.1 stack `header_read_timeout` is armed as soon as the connection is ready to read the next request head, so on a keep-alive connection it also bounds how long an idle upstream connection may sit before RustFS closes it. The previous 5s default meant RustFS FIN'd pooled upstream connections while reverse proxies (Caddy ~2min, Nginx 60s) still believed they were alive. The proxy then reused a dead socket and clients saw `socket hang up` / connection reset, most visibly on non-idempotent `PUT` uploads that proxies will not transparently retry (issue #3076). Raise DEFAULT_HTTP1_HEADER_READ_TIMEOUT to 75s (above common proxy idle-keepalive windows) and document the coupling. Still overridable via RUSTFS_HTTP1_HEADER_READ_TIMEOUT for deployments that expose RustFS directly to untrusted slow clients and want tighter slowloris protection. Co-Authored-By: heihutu <heihutu@gmail.com> * docs(operations): add reverse-proxy deployment guide Document the request/body semantics RustFS expects from a proxy layer, the idle keep-alive mismatch that causes `socket hang up` on large PutObject writes, and known-good Caddy/Nginx configs plus Cloudflare caveats. Closes the documentation gap called out in issue #3076 and consolidates the scattered findings from #609/#934/#1492/#1766. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(object): bound stalled PutObject request-body reads with diagnostics When a reverse proxy or CDN forwards a partial request body and then goes silent without closing the connection, the inner body stream neither yields more bytes nor reports EOF, so RustFS waited forever for bytes that never arrive and the client saw a silent hang/abort (issue #3076). A short body that ends with a proper EOF was already rejected promptly; the gap was the no-EOF stall case. Add a single-point request-body guard on the PutObject path that wraps the incoming StreamingBlob with an inter-chunk read timeout. The timer resets on every chunk, so slow-but-progressing uploads are unaffected; it only fires after RUSTFS_HTTP_REQUEST_BODY_READ_TIMEOUT (default 300s, 0 disables) of complete silence. On timeout it logs a structured `put_object_body_read_stalled` event with received/expected byte counts and fails the read with ErrorKind::TimedOut instead of hanging. remaining_length/size_hint are forwarded so wrapping is transparent to downstream length handling. Covered by unit tests for the stall path, length-preserving pass-through, and the disabled (timeout=0) pass-through. Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
f73880f354 |
fix(startup): survive slow multi-node cold starts (#4357)
* fix(server): make startup readiness wait configurable and raise default (#4264) The startup runtime-readiness wait was a hardcoded 30s constant with no env override. On slow multi-node cold starts (Docker/K8s/Synology NAS) this window is shorter than the internal startup budgets it depends on — the endpoint DNS-retry window (~90s) and the format-load retry loop (~100s worst case) — so readiness times out and the node exits with `startup readiness timed out after 30s: storage_ready=false, lock_quorum_ready=false` before storage/lock quorum can converge, feeding the restart storm reported in the issue. - Add `RUSTFS_STARTUP_READINESS_MAX_WAIT_SECS` (default 120s), documented in rustfs-config health constants. - Resolve the wait at runtime via `startup_runtime_readiness_max_wait()`; a value of `0` falls back to the default instead of timing out instantly. - Repoint `STARTUP_RUNTIME_READINESS_MAX_WAIT` at the shared config default so there is a single source of truth, and cover the getter with unit tests. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): stop peer/disk background monitors on graceful shutdown (#4264) Long-lived peer health/recovery and remote-disk monitors are detached `tokio::spawn` tasks that each hold a `tracing::Span` via `.instrument(..)` for their whole lifetime. Nothing cancelled them at shutdown, so on the normal return path the Tokio runtime was dropped while they were still alive and their `Span`s were dropped during worker-thread thread-local-storage (TLS) destruction. At that point `tracing-subscriber`'s fmt `on_close` can touch an already-destroyed TLS slot and panic with `cannot access a Thread Local Storage value during or after destruction`, which escalates to a panic-during-panic abort (SIGILL / exit 132) — the crash reported on Synology in issue #4264, amplified by the restart storm. - Add `cluster::rpc::background_monitor` with a process-global shutdown token, `spawn_background_monitor()` (races the monitor future against that token so its span drops while the runtime is alive), and public `shutdown_background_monitors()`. - Route every span-holding peer_s3 / peer_rest / remote_disk monitor spawn through `spawn_background_monitor` instead of `tokio::spawn(..).instrument()`. - Expose `rustfs_ecstore::shutdown_background_monitors()` and call it from the graceful shutdown sequence (right after `ctx.cancel()`, before runtime teardown) via the `storage_api` compatibility boundary. Existing recovery-probe span-context tests still pass, confirming log correlation is preserved. Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
9ae233d552 |
test(e2e): add SHA256 write and HEAD checksum regressions (#4354)
test(e2e): add SHA256 verify-on-write & HEAD-checksum regression for #4341 Issue #4341 reported (on beta.8) that RustFS accepted mismatched S3 SHA256 checksums on PutObject (HTTP 200 instead of 400 BadDigest) and did not return ChecksumSHA256 on HeadObject with ChecksumMode=ENABLED. Both behaviors are already correct on current main, but the existing e2e coverage only asserted the happy path (upload succeeds + GetObject content matches). It did not encode the issue's two acceptance criteria, so a future regression would go undetected. Add two focused e2e tests that assert exactly the issue's contract: - test_put_object_rejects_mismatched_sha256: a body that does not match the declared x-amz-checksum-sha256 must be rejected with BadDigest and must not be stored. - test_head_object_returns_stored_sha256: after a correct upload, HeadObject with ChecksumMode=ENABLED must return the stored base64 SHA256 digest. Both tests pass against a real rustfs server (mismatch -> HTTP 400 BadDigest verified). Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
58114f49f2 |
perf(ecstore): k-way heap merge for ListObjects, drop clone-to-parse (#4347)
* perf(ecstore): replace linear merge scan with k-way heap and drop clone-to-parse (backlog#874 backlog#875) merge_entry_channels advanced the k-way merge with a linear scan over all channel heads (O(entries x channels)) and allocated two fresh Strings per pairwise comparison via path::clean. Every step also cloned MetaCacheEntry values, including entry.clone().xl_meta() clone-to-parse calls. - Introduce MergeHead with a cached cleaned name (allocated only when the raw name is not already clean) and drive the merge with a BinaryHeap of boxed heads: O(log channels) per entry, allocation-free comparisons. - Move entries through the merge instead of cloning; the winner is sent without an intermediate copy. - Remove the dead merge_file_meta_versions block: it only ran for prefix-dir groups whose entries have empty metadata, so xl_meta() always failed; cross-drive version merging happens in the resolve path. - Keep legacy same-name semantics (dir groups collapse, objects shadow prefix dirs, later object candidate wins) and add regression tests for interleaved ordering, dir/object precedence, uncleaned-name grouping, and prefix-dir collapse. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): honor ascending versions_sort in ListObjects walk (#4348) * fix(ecstore): honor ascending versions_sort in walk and document ordering invariant (backlog#876) The walk loop carried a bare `//TODO: SORT` inside the `WalkVersionsSortOrder::Ascending` branch, so the requested ascending order was silently ignored and versions streamed newest-first (the raw FileMeta order). WalkOptions defaults to Ascending, so every default walker -- notably replication resync, which replays versions and needs oldest-first to preserve the version-stack order -- received the exact opposite of the contract. FileMeta maintains versions newest-first (sort_by_mod_time is descending) and into_file_info_versions preserves that order, so ascending emission is the exact reverse of file_info_versions output. Reverse in place when ascending is requested and add a regression test locking the newest-first invariant plus the reversal contract. Key-ordering audit result (no gap found): per-disk walkers emit sorted streams, merge_entry_channels performs an ordered k-way merge, and gather_results only filters by marker/limit, so ListObjects key order is guaranteed upstream and needs no post-sort. Co-Authored-By: heihutu <heihutu@gmail.com> * perf(ecstore): enable GET metadata early-stop by default (#4349) * perf(ecstore): enable GET metadata early-stop by default with env opt-out (backlog#872) The metadata early-stop fanout (read_all_fileinfo_early_stop) has been implemented and instrumented for a while but stayed behind an opt-in flag, so default GETs always waited for every disk to answer the metadata read even after quorum agreement was reached. Flip RUSTFS_GET_METADATA_EARLY_STOP_ENABLE to default-on. The gate stays conservative: should_allow_metadata_early_stop only admits metadata-only reads (read_data=false) without version_id, healing, or free-version requirements, everything else falls back to the full-wait fanout, and setting the env var to false restores the old behavior entirely. The version-aware gate (RUSTFS_GET_METADATA_VERSION_EARLY_STOP_ENABLE) remains opt-in because versioned reads carry a higher stale-selection risk profile. Also replace the stale "optimize concurrency" TODO in get_object_fileinfo with a pointer to the early-stop implementation and add regression tests for the new default plus the explicit opt-out path. Co-Authored-By: heihutu <heihutu@gmail.com> * perf(ecstore): lazily construct codec streaming multipart readers (#4350) * perf(ecstore): lazily construct codec streaming multipart part readers (backlog#871) get_object_decode_reader_with_fileinfo opened shard readers for every part of a multipart object before returning the streaming reader, so TTFB paid for parts x disks file opens up front and an early client disconnect wasted the setup work for every unread part. Replace the eager loop with LazyMultipartCodecStreamingReader: the first part is still built eagerly so the dominant fallback conditions (missing shards / read quorum) are detected before any byte is streamed and the whole request can fall back to the legacy duplex path exactly as before. Each subsequent part is built on demand -- when the previous part hits EOF -- via a spawned task handle owned by the reader; dropping the reader aborts an in-flight build so disconnects stop all further IO. If a later part hits a fallback condition mid-stream (a shard vanished after the request started), the reader surfaces an explicit read error with a pipeline-failure metric instead of silently degrading; the client's retry then detects the condition on the eager first-part setup and takes the legacy path cleanly. Adds unit tests for in-order streaming across lazy boundaries, deferred construction (no build when the client stops within part 1), and the mid-stream fallback error path. Co-Authored-By: heihutu <heihutu@gmail.com> * perf(ecstore): prefetch next multipart part reader setup during decode (#4351) * perf(ecstore): prefetch next multipart part reader setup during decode (backlog#870) get_object_with_fileinfo processed multipart parts strictly serially: the next part's bitrot reader setup (file opens + read-quorum wait across all disks) only started after the current part finished decoding, so large multipart reads paid full setup latency between every part. Overlap the two stages with a depth-one pipeline: right after the current part's readers are obtained, the next part's setup is spawned (shared inputs behind Arc) and joined when the loop reaches that part. The shared setup_multipart_part_readers helper keeps stage-duration metrics semantics identical for both paths; a failed or stale prefetch falls back to the synchronous setup, and the PrefetchedReaderSetup guard aborts the in-flight task on error returns, early breaks, or caller drop so disconnects stop background disk IO. Gate: RUSTFS_GET_MULTIPART_READER_SETUP_PREFETCH (default on, env opt-out). Adds a three-part end-to-end read test covering the prefetch hit path and cross-part content ordering. Co-Authored-By: heihutu <heihutu@gmail.com> * perf(ecstore): move FileInfo through GET shuffle instead of cloning (#4352) perf(ecstore): move FileInfo entries through the GET shuffle instead of cloning (backlog#873) shuffle_disks_and_parts_metadata_by_index deep-cloned every valid FileInfo (parts, erasure info, metadata map) once per disk on each GET. Add an ownership-taking variant that runs the same by-index consistency check as a read-only first pass and then moves entries into their shuffled slots with mem::take, and switch get_object_with_fileinfo to it -- that call site already owned the parts metadata vector. Disk handles are Arc clones and stay cheap. Scope notes from the backlog#873 audit: - get_object_fileinfo's disks.clone() stays: DiskStore is Arc<Disk>, so the clone is per-slot refcounting and correctly avoids holding the RwLock read guard across the metadata fanout awaits. - get_object_decode_reader_with_fileinfo keeps the borrowing shuffle: its caller must retain files/disks for the legacy fallback path, so an owned variant would just shift the same clone upstream. - The metadata-cache hit path still clones parts_metadata; sharing the cached entry via Arc changes the read-path return types and is left as a follow-up. Equivalence tests cover both the by-index placement and the mod-time fallback against the borrowing variant. Co-authored-by: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> * fix(ecstore): gate merge emission on cleaned key and clear clippy redundant_clone Address review + CI findings on the ListObjects/GET optimization PR: - merge_entry_channels gated emission on the raw entry name while the heap orders by the cleaned key, so entries whose cleaned order and raw byte order disagree (e.g. redundant slashes) could be dropped. Gate on the same cleaned sort key the heap uses; add a regression test (`a//c` after `a/b`). - Drop three redundant `.clone()` calls in test code flagged by clippy::redundant_clone (owned-shuffle equivalence tests and the walk ascending-versions contract test) that failed the CI clippy gate. - Document the known mid-stream fallback limitation of the opt-in multipart codec streaming reader (default off) and mark the in-place per-part legacy degradation as a follow-up. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): force full metadata fanout for object tagging writes (backlog#872) put_object_tags reads the object fileinfo with read_data=false and then writes the updated tags to the online-disk set that read returned. With metadata early-stop enabled by default, that read now returns as soon as read quorum is reached, so the online-disk set is only a read-quorum subset. Writing tags to that subset fails write quorum -> ErasureWriteQuorum -> S3 SlowDown, which is exactly the s3-tests tagging failures (PutObjectTagging/DeleteObjectTagging, reached max retries). Thread a caller-controlled `allow_early_stop` gate through read_all_fileinfo_observed/_inner and add get_object_fileinfo_gated; put_object_tags calls it with allow_early_stop=false so the metadata read does the full quorum fanout and returns the complete online-disk set as the write target. Pure-read callers (GET/HEAD/tag read) keep the early-stop fast path unchanged. Extract metadata_early_stop_permitted() as the single gate and add a unit test locking the invariant: caller opt-out (and observe=false, and data reads) never early-stop even with the env flags on. Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
c9b976ad46 |
feat(ecstore): reclaim orphaned data dirs during heal (#4356)
* feat(ecstore): reclaim orphaned data dirs during heal (backlog #3231/#3191) Pre-#3510, an unversioned overwrite leaked one UUID-named data dir per PUT. The write path now cleans up going forward, but pre-existing strays stay on disk forever: heal's dangling logic only removes whole objects whose data is missing, never surplus data dirs of an otherwise-healthy object. That leaked space is what eventually made ListObjects scan tens of GB and time out. Add SetDisks::reclaim_orphan_data_dirs, a fail-closed, quorum-safe sweep that mirrors purge_orphan_dir_object: - referenced set is the UNION of get_data_dirs() across every online replica's xl.meta, so a dir named by any replica is kept; - if a disk holds the object dir but its xl.meta is missing or unparseable the object is treated as degraded and nothing is removed; - only UUID-named subdirectories are ever considered. Wire it into heal_object's healthy tail (under the object write lock) as a best-effort step so the background heal scanner and admin heal recover the leaked space automatically; a reclaim failure never fails the heal. Covered by four unit tests: removal of an unreferenced dir, no-op when all dirs are referenced, fail-closed abort on missing metadata, and cross-replica union preservation. Co-Authored-By: heihutu <heihutu@gmail.com> * docs(ecstore): fix typo unparseable -> unparsable Satisfies the repository typos CI check. Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
Zhengchao An
|
0ce0388fc0 |
fix(ecstore): fsync inline rename_data rollback backup (#4346)
* fix(ecstore): fsync inline rename_data rollback backup The inline branch of LocalDisk::rename_data writes the previous version's xl.meta rollback backup (<old_data_dir>/xl.meta.bkp) with a bare std::fs::write, fsyncing neither the file nor its new directory entry, even when drive_sync_enabled() is true. The same closure already syncs the new xl.meta and the commit rename directory, and the non-inline branch persists its backup durably. That backup is the sole restore source for delete_version(undo_write=true) on a set-level write-quorum failure. With sync enabled, an inline overwrite plus power loss plus a quorum failure that triggers undo_write could restore a lost or truncated backup and fail to roll back to the prior committed object. Write the inline backup via OpenOptions + write_all and, when sync is enabled, sync_data() it and fsync_dir_std its parent, mirroring the non-inline branch. Extend the inline backup test to assert the .bkp contents equal the previous metadata bytes. Refs backlog#868 (disk-durability-01). * fix(ecstore): preserve to_file_error mapping on inline backup write Address review: the rewritten inline rollback-backup write must keep the same DiskError classification as the previous std::fs::write(...).map_err( to_file_error)? call. Without it, io errors (PermissionDenied/NotFound/ StorageFull/...) would surface as DiskError::Io(_) instead of the specific DiskError variants (FileAccessDenied/FileNotFound/DiskFull/...), since From<io::Error> for DiskError only recovers variants that were wrapped via to_file_error. Map open/write_all/sync_data/fsync_dir_std through to_file_error, matching write_all_internal. --------- Co-authored-by: houseme <housemecn@gmail.com> |
||
|
yihong
|
e4ecb1bce5 | fix: ci failed (#4353) | ||
|
RustFS
|
85ba51ce72 | fix(ci): use direct hash comparison for sha256 verification on Windows (#4345) | ||
|
Zhengchao An
|
073bc96756 |
fix(filemeta): compute header signature instead of hardcoding zero (#4343)
The xl.meta version header `signature` was hardcoded to `[0,0,0,0]` on the write path (`From<FileMetaVersion> for FileMetaVersionHeader`), and the existing `get_signature` implementations only hashed `version_id` + `mod_time` (+ `size`), so two versions that share a version_id and mod_time but differ in body (e.g. a PutObjectTagging that reached only some disks) produced identical headers. Such divergence was undetectable and unhealable in the strict merge path (backlog#861, B12). Compute a real signature over the full version body, mirroring MinIO's `xlMetaV2Version.getSignature` semantics: - Fill the signature on the write path via the `From` impl (covers add_version / set_idx / update_object_version). - `MetaObject::get_signature` / `MetaDeleteMarker::get_signature` now clone the body, zero the per-disk `erasure_index`, fold `meta_user` / `meta_sys` in with order-independent hashes (msgpack map order is not stable across disks), marshal the rest and xxh64 it, then fold 64->32 bits. Empty vs all-empty PartETags are normalized alike. - Invalid/bodyless versions return an `err` sentinel rather than all-zero, so they never collide with a real all-zero legacy signature. Backward compatibility: the decode path preserves stored signature bytes, and the normal (non-strict) merge path already zeroes signatures before grouping, so existing all-zero xl.meta stays consistent across disks. Only strict merge/heal compares signatures, where legacy data is uniformly zero (no split) and genuine partial-write divergence is now correctly detected. Adds unit tests covering: non-zero signature on write, erasure_index independence, tag/metadata/size divergence detection, map-order independence, PartETags normalization, delete-marker divergence, and the err sentinel. |
||
|
Zhengchao An
|
5594b18912 |
fix(ecstore): make delete_volume non-recursive by default to prevent bucket-heal wipe (backlog#799 B1) (#4339)
* fix(ecstore): make delete_volume non-recursive by default to prevent bucket-heal wipe (backlog#799 B1) `delete_volume` unconditionally `remove_dir_all`'d the whole bucket tree, and the bucket-heal "remove" branch called it fire-and-forget on every local disk. A mis-classified "dangling" bucket (or a non-force S3 DeleteBucket on a populated bucket) was therefore recursively wiped — a potential whole-bucket data loss. The `VolumeNotEmpty` -> recreate/`BucketNotEmpty` handling already present in both delete_bucket paths was dead code because the primitive never refused. Add an explicit `force_delete` flag to `DiskAPI::delete_volume` and default the non-force path to a non-recursive `remove_dir` (rmdir), which fails atomically with `VolumeNotEmpty` if the bucket still holds any object data. Only an explicit force delete (S3 force bucket delete) removes recursively. Mirrors MinIO's `xlStorage.DeleteVol` (`Remove` vs `RemoveAll`). - Trait + all impls (local behavior, dispatch, disk_store, remote RPC) take the flag; the gRPC `DeleteVolumeRequest` gains a `force` field (proto3 default false → old peers get the safe non-recursive behavior on rolling upgrade). - Heal remove branch passes `false` and no longer discards the result: a `VolumeNotEmpty` refusal is logged (the bucket is not dangling) instead of wiping data. - Both `delete_bucket` paths pass `opts.force`, activating the previously-dead `VolumeNotEmpty` -> `BucketNotEmpty`/recreate handling (correct S3 semantics). Adds a regression test: non-force delete of a non-empty bucket returns VolumeNotEmpty and preserves the data; force delete removes it. Design converged by two independent expert reviews (MinIO-fidelity + defense-in-depth) referencing MinIO xl-storage.go. Refs backlog#799 (B1), issue rustfs/backlog#850. The safety expert's deeper hardening (typed capability instead of a bool, trash-instead-of-in-place for force, quorum re-verification of dangling) is noted on #850 as follow-up. * fix(ecstore): reword 'mis-classified' -> 'misclassified' to satisfy typos (backlog#799 B1) * fix(rustfs): thread force_delete through StorageDiskRpcExt::delete_volume + test literal (backlog#799 B1) |
||
|
Zhengchao An
|
28c0543f3c |
fix(filemeta): resolve core-storage audit P2 items B15/B16/B18 (backlog#863) (#4344)
fix(filemeta): resolve P2 core-storage audit items B15/B16/B18 (backlog#863) Fixes the three remaining actionable items from the core-storage reliability audit (rustfs/backlog#863). All three are metadata-layer correctness defects in the filemeta crate. B15 — version sort tie-break consistency `FileMeta::sort_by_mod_time` and the delete path used a hand-rolled comparator whose version_type tie-break for equal mod_time was the OPPOSITE of the canonical `FileMetaVersionHeader::sorts_before` (used by the metacache merge and latest-version selection). At equal mod_time it sorted a delete marker before its object, so the per-disk read path could treat an object as deleted while the quorum-merge path treated it as present. Both sort sites now derive their ordering from `sorts_before`, matching MinIO (object-first). B16 — replication reset state persistence The three sites that flush `reset_statuses_map` into `meta_sys` inserted each key verbatim. A bare-ARN key (produced by `ObjectInfo::replication_state`) has no internal prefix, so read-back — which only recognizes prefixed keys — silently dropped the reset state; a rustfs-only key was invisible to MinIO-compatible readers. New `persist_reset_statuses` normalizes every entry to the canonical `replication-reset-<arn>` suffix and writes both the `x-rustfs-internal-*` and `x-minio-internal-*` prefixes. B18 — Legacy (V1Obj) body round trip `FileMetaVersion::encode_to` never emitted the `V1Obj` field, so re-encoding a Legacy version silently dropped its entire body. Adds symmetric `encode_to` for `MetaObjectV1`/Stat/Erasure/ChecksumInfo/Part and a `write_msgp_time` helper (ext8/type-5/12-byte, matching the decoder). `Mode` uses the strict `write_u32` marker the decoder requires, and `Stat.ModTime` is written only when present so a `None` never round-trips to `Some(UNIX_EPOCH)`. Adds regression tests for each fix (138 filemeta tests pass). Verified with an adversarial multi-expert review that could not break any of the three. Refs rustfs/backlog#863 (B15, B16, B18). |
||
|
Zhengchao An
|
b813fc7739 |
fix(ecstore): reject zero erasure block_size in codec streaming read (#4340)
The codec streaming GET reader divides by erasure.block_size in build_codec_streaming_part_reader without validating the erasure dimensions, unlike the legacy multipart path which already rejects block_size==0 / data_shards==0. FileInfo::is_valid() does not check block_size, so corrupted on-disk metadata (block_size==0, data_blocks>0) passes validation and panics the read task with a divide-by-zero. Add Erasure::has_valid_dimensions() and reject invalid dimensions at the codec streaming entry before any disk access, mirroring the legacy guard (which now reuses the same predicate). Refs backlog#868 (868-1). |
||
|
Zhengchao An
|
8b2c67a100 | ci: cancel superseded main release builds (#4342) | ||
|
houseme
|
6f613317f6 |
feat(internode): optimize gRPC transport (#4337)
* feat(internode): P0 gRPC transport tuning, message limits, payload metrics Land the P0 subtask from docs/grpc-optimization: close the client-vs-server transport gaps and add instrumentation to size which unary RPCs need channel isolation in P1. Transport tuning (G3): the client `Endpoint` now disables Nagle and raises the HTTP/2 stream/connection flow-control windows to mirror the server socket, so small lock/health RPCs are not batched and larger metadata responses are not throttled by the 64KiB default window. All env-overridable, 0 opts out. Message-size limits (G1): both `NodeServiceClient` and `NodeServiceServer` set max decode/encode size (default 100MiB) instead of tonic's silent 4MiB cap, so a large multi-version xl.meta or aggregated ReadMultiple no longer fails out_of_range. The server limit is set on `NodeServiceServer` before wrapping in the auth `InterceptedService` (the interceptor type does not expose it). Payload instrumentation (P1 prep): ReadAll/ReadMultiple record a payload-size histogram plus a large-payload counter when a response crosses the configured threshold (default 8MiB), feeding alerting on paths that contend with latency-sensitive control-plane traffic on the shared channel. Threshold-only counter, no per-call hot-path log. Verification: cargo check/test on config, io-metrics, ecstore, rustfs; clippy clean on touched files; make pre-commit green. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(internode): P1 control/bulk gRPC channel isolation (opt-in) Land the P1 subtask from docs/grpc-optimization: physically separate large bytes-carrying unary RPCs from latency-sensitive control-plane RPCs so a big transfer can no longer head-of-line block a lock/health RPC on the shared HTTP/2 connection (G2/G5). Introduce ChannelClass { Control, Bulk } and get_channel_for_class in protos. Control RPCs keep the per-peer connection keyed by the bare address; Bulk RPCs (ReadAll/WriteAll/ReadMultiple/BatchReadVersion, via a new get_bulk_client) are round-robined across a small per-peer bulk pool. Rather than restructuring the global GLOBAL_CONN_MAP (and every consumer), bulk channels are cached under a composite key (addr\0bulk\0idx). The NUL separator cannot appear in a URL, so bulk keys never collide with the control key. This keeps the blast radius small on a consistency-sensitive path. create_new_channel is refactored into build_channel(dial_addr, cache_key) so several physically distinct channels to one peer cache independently while dialing/TLS still use the real address. Gated by RUSTFS_INTERNODE_CHANNEL_ISOLATION (default OFF) so the default build is byte-for-byte the pre-P1 behavior: bulk resolves to the control channel and the switch is a single-env rollback. RUSTFS_INTERNODE_BULK_CHANNELS (default 2, clamped >=1) sizes the pool. On failure, evict_failed_connection drops the whole bulk pool for the peer (round-robin hides which index was used), avoiding half-dead cached channels. Lock RPCs (remote_locker) already use the default Control path, so lock semantics and retry behavior are unchanged. Verification: cargo check/test on config, protos, ecstore, rustfs; new protos tests for bulk key routing and isolation-off passthrough; clippy clean on touched files; make pre-commit green. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(internode): P2 msgpack/JSON codec observability + encode buffer presizing Land the safe, wire-compatible slice of P2 from docs/grpc-optimization: the observability prerequisite for retiring the redundant JSON fields, plus a codec micro-optimization. No proto/wire-format change; JSON is still dual-written. Internode RPCs today dual-encode each metadata value as both msgpack (`*_bin`) and a JSON compatibility string, and decoders prefer `_bin` with a JSON fallback. Before the JSON fields can ever be dropped (a cross-version change), that fallback must be proven unused in production. Add rustfs_system_network_internode_msgpack_json_fallback_total{direction, message}: incremented whenever a decode falls back to the JSON field because the msgpack payload was absent. Wired into both directions — the client decoding peer responses (remote_disk.rs, incl. the list-level read_multiple/batch fallbacks) and the server decoding peer requests (node_service/disk.rs). This counter must read zero across a release window before send paths stop writing JSON and the proto text fields are reserved/removed (the deferred P2-1 steps). Also pre-size the msgpack encode buffers (Vec::with_capacity(512)) on both sides, eliminating the repeated growth reallocations for typical FileInfo payloads with zero added copy. Full thread_local buffer pooling is deferred: it needs either an extra copy (unclear net win) or a send-path buffer-return lifecycle, to be justified by a codec microbenchmark first. Verification: cargo check/test on io-metrics, ecstore, rustfs; new fallback counter smoke test; existing codec decode tests green; clippy clean on touched files; make pre-commit green. Co-Authored-By: heihutu <heihutu@gmail.com> * docs(internode): add msgpack/JSON convergence observation runbook Runbook driving the observation-gated retirement of the redundant JSON compatibility fields on internode gRPC metadata RPCs (grpc-optimization P2-1). Documents the shipped fallback counter (rustfs_system_network_internode_msgpack_json_fallback_total{direction,message}), the PromQL to confirm it reads zero across a release window, a standing alert, and the staged flip/rollback procedure (env-gated msgpack-only send, then proto field removal in N+1). Includes the verified field -> peer-decoder audit: only fields whose peer decodes _bin first may be converged. Notes DeleteVersion.opts (DeleteOptions) is NOT convergence-ready — its server handler is not _bin-first and must gain a decode_msgpack_or_json path first. This gates the send-side change so it cannot empty a JSON field an old peer still needs. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(internode): env-gated msgpack-only send + DeleteVersion _bin support (P2-1) Implements the send-side lever for retiring the redundant JSON compatibility fields on internode gRPC metadata RPCs, plus the missing `_bin` support on the delete path that it depends on (grpc-optimization P2-1). Default-off: the base build is byte-for-byte the prior dual-write behavior. Gated msgpack-only send (RUSTFS_INTERNODE_RPC_MSGPACK_ONLY, default false): - New rustfs_protos::internode_rpc_msgpack_only() reads the flag. - Client (remote_disk.rs) compat_json() and server (node_service/disk.rs) compat_response_json() emit an empty JSON string when the flag is on, so only the msgpack _bin payload is sent. The _bin field is always sent; decoders keep the JSON read fallback. Applied only to fields with a confirmed _bin-first peer decoder (WriteMetadata/UpdateMetadata/RenameData file_info, UpdateMetadata opts, ReadOptions, ReadMultipleReq, BatchReadVersionReq; ReadVersion/ReadXL/RenameData responses and the ReadMultiple/BatchReadVersion response lists). - Only enable after the P2 fallback counter has read zero across a release window (see docs/operations/internode-msgpack-json-convergence-runbook.md). Single-env rollback; no wire-format break. DeleteVersion(s) _bin support (prerequisite): - The DeleteVersion/DeleteVersions protos had NO _bin fields. Add additive (backward-compatible) bytes file_info_bin/opts_bin (DeleteVersion) and repeated bytes versions_bin + bytes opts_bin (DeleteVersions); regenerate the checked-in prost struct. - Client dual-writes them; server decodes them _bin-first with JSON fallback. - These delete fields are kept OUT of the msgpack-only set (always dual-write) until their own fallback counter reads zero across a window with the new decoders fully deployed. DeleteVersion.raw_file_info stays JSON-only (no _bin field yet). Verification: cargo check/test on protos, config, ecstore, rustfs (incl. the six delete request handler tests and a compat_json default-path test); clippy clean on touched files; make pre-commit green. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(internode): P3 cluster peer online/offline health metric Land the safe observability core of P3 (grpc-optimization G6/G8): track each internode peer's reachability and expose the offline count, for parity with MinIO's minio_cluster_servers_offline_total. Pure instrumentation — peer selection and quorum are unchanged. - io-metrics: per-peer PeerHealthState { online, consecutive_failures } registry plus record_peer_reachable/record_peer_unreachable. A peer flips offline after N consecutive failures (dial failures or RPC-triggered evictions) and back online on the next successful dial; the count of offline peers is published to the rustfs_cluster_servers_offline_total gauge. - config: RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD (default 3, clamped >= 1). - protos: build_channel marks the peer reachable on a successful dial and unreachable on a dial failure; evict_failed_connection feeds the failure signal too. Keyed by the real peer address, so control and bulk channels to one peer share health state. Deferred (documented in docs/grpc-optimization P3): startup prewarm (no clean topology-ready hook yet), the offline fast-bypass in peer routing (consistency- sensitive; must not change quorum), and idempotent-read-only retry. This commit is observability only. Verification: cargo check/test on io-metrics, config, protos (new peer-health state-machine and threshold-clamp tests); clippy clean on touched files; make pre-commit green. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(internode): P3 control-channel prewarm + self-healing offline bypass Add the remaining P3 connection-lifecycle levers (grpc-optimization G6/G8), both env-gated and default-off so the base build is unchanged. Prewarm (RUSTFS_INTERNODE_PREWARM, default off): RemoteDisk::new spawns a best-effort background dial of the peer's control channel, deduped per peer address, moving the connect cost off the first RPC. Failures fall through to the existing lazy connect + recovery monitor. Offline bypass (RUSTFS_INTERNODE_OFFLINE_BYPASS, default off): remote_disk get_client/get_bulk_client fast-fail a peer already marked offline instead of paying the connect timeout, so the erasure layer proceeds on quorum sooner. This does NOT change quorum. It is self-healing: cluster_peer_should_bypass lets one request per RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS (default 5s) through to recover the peer even with no background monitor, and the recovery monitor's own probe path calls the client directly so it is never bypassed. io-metrics gains cluster_peer_is_offline / cluster_peer_should_bypass (with a per-peer re-probe timestamp). Scope: data path only — remote_locker (lock RPCs, most consistency-sensitive) is left dual-writing/unbypassed as a follow-up. Verification: cargo check/test on io-metrics, config, ecstore (new self-healing bypass tests; all 105 rpc tests green); clippy clean on touched files; make pre-commit green. Co-Authored-By: heihutu <heihutu@gmail.com> * docs(internode): add A/B benchmark runbook for gRPC optimization stages Reproducible before/after collection procedure for grpc-optimization P0–P3. Since every stage is env-gated, before/after is the same binary with different env — no rebuild. Documents, per stage: the exact env toggles (baseline vs enabled column), which existing bench script to run (run_internode_transport_baseline.sh / run_four_node_cluster_failover_bench.sh), the Prometheus metrics to capture, and the acceptance gates from the design docs (e.g. lock p99 down >= 20% for P1, msgpack fallback counter = 0 before enabling P2, correct rustfs_cluster_servers_offline_total for P3). Live runs require a multi-node cluster + load tool + Prometheus scrape and cannot be produced in a single-process sandbox; artifacts land under target/bench (gitignored) and attach to the PR. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(internode): P3-2 lock-path offline bypass + P3-3 idempotent read retry Extend the offline bypass to the lock path and add opt-in retries for idempotent reads (grpc-optimization P3-2/P3-3). Both env-gated and default-off/zero. Offline bypass (lock path): factor the bypass decision into a shared pub(crate) internode_offline_bypass_reason(addr) and call it from remote_locker::get_client too, so lock RPCs to an offline peer fast-fail (letting dsync reach quorum sooner) instead of paying the connect timeout. Does not change quorum; the self-healing re-probe keeps peers recoverable. Gated by RUSTFS_INTERNODE_OFFLINE_BYPASS (default off). Idempotent read retry (P3-3): add execute_read_with_retry — a bounded, exponential-backoff retry for read-only/reentrant RPCs on transient network errors — and route disk_info through it. RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES defaults to 0 (disabled). Write/lock RPCs are never retried (quorum/idempotency safety, per CLAUDE.md); the wrapper requires an Fn closure so only reads that rebuild their request from borrowed inputs qualify. Deferred: grpc.health.v1 (optional ecosystem-compat only; needs a new tonic-health dep and 3-way hybrid-service wiring — internal needs are met by the existing Ping RPC). Verification: cargo check/test on config, ecstore (105 rpc tests green incl. disk_info now via the retry wrapper); clippy clean on touched files; make pre-commit green. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(scripts): one-click internode gRPC A/B benchmark driver Wrap the per-stage env matrix from the benchmark runbook into scripts/run_internode_grpc_ab_bench.sh: given --stage <p0|p1|p2|p3> and --phase <before|after>, it emits the stage/phase RUSTFS_INTERNODE_* server env to <out-dir>/server-env.sh and runs the right underlying bench (run_internode_transport_baseline.sh for p0/p1/p2, run_four_node_cluster_failover_bench.sh for p3) into a labeled target/bench/internode-transport/<stage>-<phase>/. Passthrough args after `--` reach the underlying bench; --dry-run previews the env + command. The script is explicit that RUSTFS_INTERNODE_* are server env, so for the load-driven stages the operator must restart rustfs with the emitted env before the run; the docker four-node (p3) path exports them for a forwarding compose. shellcheck-clean. Runbook updated with a "One-click driver" section. Co-Authored-By: heihutu <heihutu@gmail.com> * chore(compose): forward RUSTFS_INTERNODE_* into the four-node cluster The four-node local-build compose only forwarded a fixed whitelist of env, so the internode gRPC knobs (grpc-optimization P0-P3) never reached the containers and the A/B bench driver's "after" phase was a no-op. Forward the full RUSTFS_INTERNODE_* set with defaults matching the binary defaults, so leaving them unset is a no-op and the A/B driver can toggle a stage per phase. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(internode): address Copilot review — retry health action + poison-safe peer health Two review nits on #4337: - P3-3 idempotent read retry (remote_disk.rs): execute_read_with_retry ran every attempt through execute_with_timeout_for_op, which hardcodes FailureHealthAction::MarkFailure. So the first transient error could flip the disk faulty and short-circuit the remaining retries, and each attempt over-counted the failure. Route all but the final attempt through execute_with_timeout_for_op_and_health_action with IgnoreFailure; only the last attempt marks faulty/evicts. No default impact (retries default 0). - Peer-health helpers (io-metrics): record_peer_reachable/record_peer_unreachable, cluster_peer_is_offline and cluster_peer_should_bypass early-returned on a poisoned mutex, permanently stalling the offline gauge and bypass state after a single panic. Recover via PoisonError::into_inner(). Verification: cargo check/test on io-metrics + ecstore (105 rpc tests green); clippy clean on touched files; make pre-commit green. Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
Zhengchao An
|
8f11222a63 |
feat(admin): add MinIO-compatible IAM and IDP endpoints (#4334)
feat(admin): add MinIO-compatible IAM/IDP admin endpoints
Register and implement MinIO admin API compatibility for IAM/IDP:
- PUT /v3/import-iam-v2 and POST /v3/revoke-tokens/{userProvider}
- generic /v3/idp-config/{type}[/{name}] CRUD mapped onto existing config
- LDAP/OpenID service-account, policy-entities, and list-access-keys flows
Adds IamSys::delete_temp_account primitive to back STS token revocation.
revoke-tokens requires the broader ListUsers admin capability for
cross-user revocation (self-revocation only needs RemoveServiceAccount),
mirroring the cross-user guard used by the service-account handlers.
Registers admin route-policy inventory entries for every new route.
Unsupported LDAP/OpenID backends return honest compatibility errors.
Refs rustfs/backlog#609 #610 #616
|
||
|
Zhengchao An
|
3420f28c17 |
feat(admin): add MinIO-compatible diagnostics endpoints (#4333)
feat(admin): add MinIO-compatible diagnostics and service-control endpoints Register and implement MinIO admin API compatibility routes: - POST /v3/service: restart/stop trigger the existing graceful-shutdown path; freeze/unfreeze record advisory state (request admission is not yet gated, surfaced honestly via effective=false) - profiling/trace and healthinfo/obdinfo/log diagnostics endpoints - GET /v3/top/locks and POST /v3/force-unlock, backed by new FastObjectLockManager::list_locks()/force_unlock() introspection - speedtest routes where feasible Refs rustfs/backlog#604 #606 #607 #615 |
||
|
Zhengchao An
|
68f048b8fe |
fix(replication): persist delete marker mtime in MRF entries (#4331)
fix(replication): persist original mtime in MRF entries (backlog#867) MRF delete entries did not persist the original delete-marker mtime, so after a restart the recovery replay path reconstructed the delete without a source timestamp. Downstream the replica delete-marker was stamped with the replay time (now()) instead of the source mtime, causing delete-marker timestamp divergence across clusters. Extend the MrfReplicateEntry disk format with an optional deleteMarkerMtime field (persisted as Unix nanoseconds) in both duplicate struct definitions (rustfs-replication and rustfs-filemeta). DeletedObjectReplicationInfo now persists delete_marker_mtime, and start_mrf_processor restores it onto the reconstructed delete so the replica keeps the source timestamp. Backward compatibility: the new key uses skip_serializing_if + serde default, so historical MRF files without it decode to None and replay falls back to the current time (pre-#867 behaviour). No panic or entry loss on old files. Closes rustfs/backlog#867 |
||
|
Zhengchao An
|
b60686eb6f |
feat(admin): add replication diff/mrf and batch job admin endpoints (#4332)
Add MinIO-compatible admin endpoints under the /v3 prefix, wired to real RustFS subsystems where they exist and documented as compatibility-only where they do not. Replication (backlog#611): - POST /v3/replication/diff: bounded on-demand scan of object versions, returning versions whose replication status is PENDING or FAILED. Wired to list_object_versions and the per-version replication status. Requires the bucket to exist and have a replication configuration. The scan is capped (REPLICATION_DIFF_MAX_SCAN) and reports IsTruncated when partial. - GET /v3/replication/mrf: reports the failed-replication backlog (MinIO's MRF concept) from the replication stats handle: aggregate failed count and size per target plus in-queue counters. RustFS records failed replications as aggregate counters and persists MRF entries without a runtime query API, so per-object MRF rows are not enumerable; PerObjectEntriesAvailable is always false to make that explicit. Batch jobs (backlog#613): - POST /v3/start-job, GET /v3/list-jobs, GET /v3/status-job, GET /v3/describe-job, DELETE /v3/cancel-job. - RustFS has no batch-job execution engine (no scheduler, queue, or worker), so these implement MinIO request/response/error semantics only and never fake execution or success: start-job parses and validates the job definition then returns NotImplemented for known job types and InvalidRequest for unknown ones; list-jobs returns an empty list; status/describe/cancel validate jobId and return a no-such-job error. All routes are registered with admin auth using existing AdminAction variants (ReplicationDiff, GetReplicationMetricsAction, StartBatchJobAction, ListBatchJobsAction, DescribeBatchJobAction, CancelBatchJobAction) and are covered by the admin route-registration matrix and new handler unit tests. Refs rustfs/backlog#611 #613 |
||
|
Zhengchao An
|
511ad31ba0 |
fix(s3): normalize root double-slash ListBuckets requests (#4336)
test(s3): MinIO parity for double-slash ListBuckets, invalid copy-source date, region-aware CreateBucket - Add DoubleSlashListBucketsCompatLayer: rewrite `GET //` to `GET /` so it routes to ListBuckets, matching MinIO browser-client behavior (#601) - Add e2e regression coverage for invalid copy-source conditional date headers (#618) and region-aware MakeBucket(us-east-1) (#629) Refs rustfs/backlog#601 #618 #629 |
||
|
Zhengchao An
|
eb02486574 |
chore(swift): remove placeholder SSE module (#4330)
security(swift): remove placeholder SSE module (backlog#646)
The Swift `encryption` module was a non-functional stub: `encrypt_data`
returned the plaintext unchanged while labeling it AES-256-GCM in the
object metadata, and `generate_iv` derived the IV from a timestamp
rather than a CSPRNG. It had no production caller (only a `pub mod`
declaration and one integration test), so wiring it in as-is would have
silently shipped plaintext advertised as ciphertext.
We are not supporting Swift server-side encryption for now, so remove
the module outright rather than keep a dangerous stub around:
- delete crates/protocols/src/swift/encryption.rs
- drop `pub mod encryption;` from swift/mod.rs
- remove the encryption case (and unused import) from the swift
integration test
The module reached main dubiously: it was introduced together with the
whole Swift API in commit
|
||
|
Zhengchao An
|
3bc8d79fe5 |
fix(multipart): serialize put_object_part per uploadId (#4329)
fix(multipart): serialize the part commit per uploadId to prevent mixed-generation shards (backlog#853) Concurrently re-transmitting the same part could land two shard generations across different disks: each shard is individually bitrot-valid, so the corruption surfaces only as a silent mix at read time. The hazard is confined to rename_part, where two temp parts are moved cross-disk onto the SAME final part path — interleaving there can leave shards from two generations. Each concurrent stream already writes to its own unique temp dir, so the encode/stream phase never conflicts and must stay lock-free: holding a lock across it would serialize slow re-transmits of the same part, break the S3 'last finisher wins' semantics, and cause UploadPart lock-acquire timeouts (ServiceUnavailable). Scope a write lock to the uploadId namespace around only the rename_part commit, so each commit is atomic across disks and the last committer wins consistently. Mirrors MinIO's per-uploadID NS lock; disjoint from the object lock held by complete_multipart_upload (no lock-ordering cycle). Honors opts.no_lock. The pre-delete-before-rename half of backlog#853 was already fixed in #4316; this completes the issue. Adds an end-to-end regression test: six concurrent resends of the same part must all succeed (no lock-acquire timeout), exactly one generation stays visible, and the reassembled object equals one intact generation. Closes rustfs/backlog#853 |