fix(audit): propagate dispatch delivery failures instead of swallowing them (backlog#962)
AuditPipeline::dispatch and dispatch_batch accumulated per-target save()
errors, logged them, and then unconditionally returned Ok(()). When every
configured target failed the audit entry was lost outright (for store-backed
targets a failed save() means the event was neither delivered nor persisted
for replay), yet callers such as dispatch_audit_log saw success and assumed
the audit trail was intact. Audit is a compliance-critical path, so a total
delivery failure that reports success is a silent data-loss bug.
Both methods now distinguish three outcomes: all targets succeeded (Ok),
partial failure where at least one target accepted the event (log a warning
and return Ok, since the entry is not lost), and total failure where no
delivery succeeded while errors were recorded (record the failure metric,
log an error, and return Err(AuditError::Target) carrying the first target
error). This lets the caller react instead of assuming success.
Adds regression tests with a FailingTarget mock asserting that dispatch and
dispatch_batch return Err on total failure and Ok on partial failure.
fix(targets): surface truncated store batches instead of silently returning partial entries (backlog#967)
get_multiple deserializes a persisted batch by pulling item_count items
from a concatenated JSON stream. When the stream ended early (a truncated
or corrupted batch file), the partial-read branch only logged a warn! and
broke out of the loop, returning Ok(partial). Callers then treated the
batch as fully delivered, deleted the store entry, and permanently lost
the remaining events with no error.
Return StoreError::Deserialization as soon as the stream ends before
item_count items are read, so the caller keeps the entry on disk and can
retry rather than silently dropping events. The empty-file case already
returned an error and remains covered by this path.
Add a regression test that writes a 3-item batch, truncates the file at a
clean boundary after two items, and asserts get_multiple returns Err and
leaves the batch file in place.
Refs: https://github.com/rustfs/backlog/issues/967
* fix(audit): resolve AB-BA lock-order deadlock between registry and stream_cancellers (backlog#961)
`AuditSystem::runtime_status_snapshot` acquired `stream_cancellers.read()`
before `registry.lock()`, while every other path that holds both locks
(`clear_runtime_targets`, `AuditRuntimeFacade::replace_targets`) acquires
`registry` first. Under concurrency (e.g. admin status polling racing a
config reload) this AB-BA ordering could deadlock the whole audit control
plane with no self-recovery.
Reorder `runtime_status_snapshot` to acquire `registry` before
`stream_cancellers`, unifying the global lock order to
`registry -> stream_cancellers` across all double-lock paths, and add a
multi-threaded regression test that hammers both critical sections with a
timeout guard to fail fast if the ordering regresses.
Refs: https://github.com/rustfs/backlog/issues/961
* docs(audit): reword AB-BA to ABBA to satisfy typos check (backlog#961)
fix(notify): strip rustfs/minio internal metadata from event userMetadata (backlog#964)
Notification event construction only stripped keys prefixed with
`x-amz-meta-internal-`, which is not a prefix RustFS actually uses. Real
internal metadata lives under `x-rustfs-internal-*` / `x-minio-internal-*`
(xl.meta compat, incl. `x-minio-internal-server-side-encryption-*`) and
server-side-encryption details under `x-rustfs-encryption-*` /
`x-minio-encryption-*`. None of these were filtered, so they leaked into
`s3.object.userMetadata` sent to downstream notification targets
(webhook/MQ) — an information disclosure of SSE/replication/healing
internal state.
Filter via the shared classifiers `rustfs_utils::http::is_internal_key`
and `is_encryption_metadata_key` (case-insensitive, covering both key
flavors), while retaining the legacy `x-amz-meta-internal-*` strip for
backward compat. Genuine user metadata (`x-amz-meta-*`, content-type, ...)
is preserved unchanged.
Adds a regression test asserting both internal prefixes, the SSE internal
key, both encryption prefixes (incl. mixed-case) and the legacy prefix are
stripped while user keys survive.
Refs: https://github.com/rustfs/backlog/issues/964
* refactor(ecstore): add per-instance InstanceContext, migrate erasure setup type
Phase 5 of the global-singleton consolidation (backlog#939): begin moving
runtime identity state out of process globals so multiple ECStore instances
can coexist in one process. Isolation is carried by the object graph
(ECStore -> Sets -> SetDisks holding an Arc<InstanceContext>), not a
task-local, which does not propagate across the many internal tokio::spawn
boundaries in the data/background paths.
This first slice migrates the erasure setup type -- previously three
independent process-global bools -- into a single per-instance
RwLock<SetupType> that derives is_erasure / is_dist_erasure / is_erasure_sd,
removing a triple source of truth that could drift out of sync.
- New runtime::instance module: InstanceContext + process bootstrap context.
- The legacy free-function facade (is_erasure/update_erasure_type/...) keeps
its signatures and forwards to the current instance's context, falling back
to the bootstrap context before a store is published.
- ECStore gains a pub(crate) ctx field and setup_is_* accessors; its
constructors adopt the bootstrap context (never mint a fresh one) so startup
writes and post-construction reads share one cell -- single-instance
behavior is byte-for-byte unchanged.
Tests: erasure predicate derivation vs the legacy behavior, object-graph
carrier isolation across two ECStore instances, and bootstrap adoption.
Refs: backlog#939 (Phase 5, Slice 1), backlog#653 (item 8)
* refactor(ecstore): thread InstanceContext down the object graph (Phase 5 Slice 2) (#4415)
* refactor(ecstore): source the namespace lock manager per-instance (#4417)
refactor(ecstore): source the namespace lock manager per-instance (Phase 5 Slice 3)
Phase 5 Slice 3 (backlog#939): give each instance its own lock namespace by
sourcing SetDisks' lock manager from the instance context instead of the
process singleton. This removes the false cross-instance mutual exclusion (and
attendant ABBA risk) that a shared GlobalLockManager would cause once multiple
instances coexist.
- InstanceContext gains a `lock_manager: Arc<GlobalLockManager>`. `new()` mints
a fresh manager (independent per-instance); `bootstrap_ctx()` aliases the
process singleton via get_global_lock_manager(), so a single-instance
deployment keeps exactly one shared namespace.
- SetDisks::new sources `local_lock_manager` from `ctx.lock_manager()` (the ctx
it already adopts), not `runtime_sources::global_lock_manager()`. Single
instance: same Arc as before, so behavior is unchanged.
- Remove the now-unused `runtime_sources::global_lock_manager()` wrapper.
Tests: bootstrap lock manager aliases the process singleton; two fresh contexts
own distinct managers; a SetDisks' lock manager is the one from its context and
aliases the global singleton in a single-instance build.
Verification: cargo test -p rustfs-ecstore (10 Phase 5 + set_disk locking
regressions green), cargo clippy -p rustfs-ecstore --all-targets (clean),
make pre-commit (pass).
Refs: backlog#939 (Phase 5, Slice 3). Stacked on #4415 (Slice 2).
fix(lock): fence write commit on lock loss (backlog#899 Phase 2)
Phase 0+1 (#4388) made object write locks refreshable and marks the guard
lost when the heartbeat can no longer refresh a quorum, but does not act on
it. Under a partition a long write's lock can expire on an unreachable node
and be reclaimed, letting a third party re-acquire it; the original writer
keeps going and both commit -- a double write.
Expose the loss signal through NamespaceLockGuard::is_lock_lost() and
ObjectLockDiagGuard::is_lock_lost(), and fence the commit in put_object and
complete_multipart_upload: immediately before rename_data (the atomic commit
point), abort with a retryable NamespaceLockQuorumUnavailable (503) if the
lock was lost. In multipart the check precedes cleanup_multipart_path so a
lost lock leaves the upload intact and retryable. A write that already
reached rename_data Ok is durable and never aborted.
The loss criterion is unchanged (reacts to Phase 1's signal). Heal and the
long-GET read side are deferred follow-ups.
EventName::mask() recursed forever for the three internal leaf events
(ObjectRemovedAbortMultipartUpload, ObjectCreatedCreateMultipartUpload,
ObjectRemovedDeleteObjects): their discriminants fall past
LAST_SINGLE_TYPE_VALUE so mask() took the compound branch, but expand()
returns vec![*self] for them, so mask() called itself with the same
value and overflowed the stack.
Detect the self-expanding leaf case (expand() yields exactly self) and
give it a dedicated bit derived from its discriminant. Those bits sit
above the single-type bits, so they never collide with each other or
with any compound 'All' mask.
Add exhaustive regression tests: every variant's mask() terminates, the
three internal masks are non-zero and mutually distinct and disjoint
from the single-type bits, and Everything covers all single-type bits.
Refs: https://github.com/rustfs/backlog/issues/965
The FTPS and WebDAV authentication handlers compared the client-supplied secret
key against the stored secret with `String::eq`, which short-circuits on the
first differing byte. A network attacker who knows (or enumerates) a valid
access key can recover the secret key byte-by-byte via response-timing analysis;
neither path is rate limited.
Switch both to a constant-time comparison using `subtle::ConstantTimeEq`, the
same primitive the SFTP handler and `rustfs/src/auth.rs::constant_time_eq`
already use. `subtle` is added to the `ftps` and `webdav` feature dependency
sets (it was previously gated on `sftp` only).
Addresses GHSA-3p3x-734c-h5vx.
The internode RPC HMAC secret is derived from the S3 credential pair via
`derive_rpc_secret` when `RUSTFS_RPC_SECRET` is unset. The derivation uses the
secret key as the HMAC key, so when the default secret key (`rustfsadmin`) is in
effect the derived RPC secret is a fixed, publicly computable value. Any network
peer can then forge valid `x-rustfs-signature` headers and invoke internode RPC
routes (e.g. `read_file_stream`), bypassing S3 IAM entirely.
`normalize_rpc_secret` already rejected the literal default when it was supplied
directly, but `resolve_rpc_secret` still derived a secret from the default
credential pair. Make the derivation path fail closed: refuse to derive while
the default secret key is in effect, forcing operators to set `RUSTFS_RPC_SECRET`
(or a non-default `RUSTFS_SECRET_KEY`). A default access key paired with a
non-default secret key remains safe and is still allowed.
Addresses GHSA-68cw-96m3-h2cf (incomplete-fix follow-up to CVE-2026-45039).
Replication deletes dropped the S3 `versionId` query parameter for every
replication request, conveying the version only via the internal
x-*-source-version-id header. A generic (non-MinIO/RustFS) S3 target ignores
that header, so a version purge degenerated into a delete-marker creation while
the source still stamped VersionPurgeStatus=Complete — a silent divergence.
Only omit the query `versionId` when propagating a delete-marker CREATION
(the target must mint its own marker); version purges, delete-marker purges and
force deletes now address the exact version. Extracted into
resolve_delete_api_version_id with unit coverage.
Fixesrustfs/backlog#857
Disk-replacement heal previously repaired only the latest version of each
object and never enumerated objects whose latest version is a delete marker,
so old versions were left unrepaired on a replaced drive.
Switch heal enumeration from list_objects_v2 (latest-only) to
list_object_versions (every version incl. delete markers), thread the concrete
version_id into the existing per-version heal_object, and make resume
cursor-based instead of positional: an opaque (marker, version_marker) paging
token persisted in ResumeState, a length-prefixed injective per-version dedup
key, schema_version bumps (v2) migrated independently in each of the two
persisted files, and a retry that resets both managers together (fixing a
latent rescan-skips-everything defect). Adds a real-disk-wipe e2e regression
suite proving old versions and delete-marker-latest objects are physically
restored.
Fixesrustfs/backlog#918Fixesrustfs/backlog#919Closesrustfs/backlog#854
* fix(ecstore): make post-commit old data dir cleanup best-effort (backlog#898)
A write is authoritatively committed once rename_data returns Ok (the new
version is durable on >= write_quorum disks and immediately readable). The
subsequent reclamation of the now-dereferenced old object/<data_dir> is pure
space reclamation, yet commit_rename_data_dir propagated a below-quorum GC
failure via `?` into ErasureWriteQuorum -> 503, producing a false-negative
ACK for an already-persisted write. This is a deliberate divergence from
MinIO (erasure-object.go:1577), which couples the two; the divergence is
justified by durability semantics, not parity.
Changes:
- commit_rename_data_dir now returns a structured OldDataDirCleanup receipt
and never returns Err. Adds an old==committed-dir anti-misdelete guard and
a committed_data_dir parameter. Classification is extracted into pure
functions (classify_old_data_dir_cleanup / map_cleanup_join_result /
is_cleanup_not_found) so it is unit-testable. Task panic/cancel is mapped to
a non-ignored DiskError::other (never DiskNotFound), and not-found is
normalized to reclaimed.
- object.rs / multipart.rs consume the receipt instead of `?`. The result
reverts to Ok, so the invalidate_get_object_metadata_cache self-heal and the
capacity/compression accounting that a `?` early-return previously skipped
now run on the cleanup-failure path too.
- On residue, report_old_data_dir_cleanup emits leak metrics and enqueues an
object heal over the existing heal channel (disk-health signal replacing the
503). heal_object -> reclaim_orphan_data_dirs already reclaims unreferenced
local data dirs, closing the loop end to end.
- Adds rustfs_old_data_dir_* counters (attempted/reclaimed/leaked/below_quorum)
as the operator-visible backstop for leaked residue.
- Adds a test-only (#[cfg(test)]) delete fault-injection seam; in production it
inlines to a no-op None and has no behavioral effect.
Tests: pure-function A/C group + join-error mapping + actions decision; A5/A5b
real-disk guard/reclaim integration; end-to-end overwrite returning 200 while
old-data-dir cleanup fails. #864 rollback guard test remains green.
* fix(ecstore): resolve merge conflicts with origin/main in io_primitives.rs
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
fix(protocols): allow clippy type_complexity in test mock struct
The MockExpirationObjectBackend test struct uses a nested generic type
that triggers clippy::type_complexity. Add #[allow(clippy::type_complexity)]
since this is test-only code where the type is inherent to the mock design.
Co-authored-by: houseme <housemecn@gmail.com>
Merge the hotpath-rs wall-time instrumentation from the backlog#936 analysis worktree behind an opt-in 'hotpath' cargo feature, keeping the default build at zero overhead and zero dependency.
- hotpath is an optional dependency everywhere (dep:hotpath feature syntax); the default dependency tree contains no hotpath crate at all
- 40+ measurement points across S3 handlers, ECStore/SetDisks object and multipart ops, erasure encode/decode, bitrot, LocalDisk I/O, FileMeta codec, and HashReader
- attribute sites use #[cfg_attr(feature = "hotpath", hotpath::measure)]; async_trait bodies use per-crate hp_guard! macros (ecstore + rustfs bin); rio gates measure_block! behind hp_measure_block!
- feature chain: rustfs -> rustfs-ecstore -> rustfs-rio / rustfs-filemeta, each crate owning its own gate
- hotpath-alloc is intentionally not wired up (hotpath 0.21.x TLS panic on cross-thread guard drop under tokio, see backlog#935); mimalloc stays the unconditional global allocator
- docs/development/hotpath-profiling.md documents building, HOTPATH_* env vars, SIGTERM report flow, and how to reproduce the backlog#936 timing reports
Refs: https://github.com/rustfs/backlog/issues/935 (HP-14, item 2), https://github.com/rustfs/backlog/issues/936
Co-authored-by: heihutu <heihutu@gmail.com>
* feat(ecstore): add stripe-advance handles for deferred bitrot readers
Give DeferredObjectReader a shared pending state and expose a
DeferredReaderStripeHandle that advances the still-unopened source by whole
bitrot blocks using the same bitrot_encoded_range geometry the reader was
created with (identity mapping when hash_size == 0). This lets the GET decode
path open a parity shard aligned to the stripe where a data shard failed
instead of reading every parity shard on every stripe (backlog#923).
An already-opened (or failed) reader rejects the advance so callers retire it
rather than engage it out of alignment; bitrot verification after an advance
checks the advanced stripe's block against that stripe's stored hash.
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(ecstore): read only data shards on healthy lockstep GET behind opt-in gate
PR #4289's lockstep fix made every reconstruction-verifying GET read all
data+parity shards per stripe; the parity blocks are read, bitrot-hashed and
then discarded, a deterministic 2x read-bytes/IOPS/hash-CPU amplification on
healthy 2+2 objects (backlog#923). With the new opt-in gate
RUSTFS_GET_LOCKSTEP_DATA_SHARDS_ONLY_ENABLE=true (default: false, behavior
identical to main):
- read_lockstep keeps only the data slots engaged while the object is
healthy; parity slots stay unopened deferred readers.
- When a data shard is missing or dies at stripe k, parity readers are
engaged mid-object by advancing their deferred stripe handle to stripe k,
preserving the lockstep alignment invariant from backlog#832.
- Degraded stripes engage one parity beyond the decode quorum so
reconstruction verification keeps an extra source to check against
(erasure.rs only verifies when available > data shards); an engaged parity
reader that errors is retired for the rest of the object like any other,
and a parity reader that cannot be realigned is retired instead of being
read out of position.
- fill_deferred_bitrot_readers records stripe handles for deferred slots and,
gate-on only, swaps eagerly opened parity readers for unopened deferred
ones so they remain engageable mid-object; ready/error bookkeeping used by
quorum decisions is untouched.
- Both GET paths (legacy duplex via Erasure::decode_with_stripe_handles,
codec streaming via ParallelReader::with_deferred_parity_handles) carry the
handles from reader setup.
Short-read -> UnexpectedEof -> whole-object retirement and the
inconsistent-source rejection are unchanged in both gate modes; tests lock
the healthy-path data-shards-only call counts, the default read-all-shards
behavior, mid-object parity engagement for streaming and hash_size==0
formats, and mid-stream inconsistent-parity rejection.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
fix(lock): renew distributed locks via heartbeat; wire server refresh (backlog#899)
A held distributed write lock had a fixed 30s TTL and was never renewed, so
any operation exceeding it got its per-node lease reclaimed and stolen by a
contender, causing split-brain writes. This implements Phase 0 + Phase 1 of the
#899 design (Phase 2 abort-on-loss is deferred and tracked in code comments).
Phase 0 (server refresh wiring, P0):
- node_service handle_refresh was a no-op stub that parsed args then always
returned success=true. Extract a testable `refresh_lock` free function that
actually delegates to the node's lock backend. Not-found maps to
success=false with no error_info, so RemoteClient::refresh keeps its
Ok(resp.success) semantics and yields a real not_found signal to the
coordinator heartbeat. Without this, client heartbeats were silently no-ops.
Phase 1 (client heartbeat + safe interval + observability):
- DistributedLockGuard spawns a heartbeat that refreshes every per-client lease
on a derived interval; interval is derived without Duration::clamp
(entries<=1 or interval>=ttl => no spawn), fixing the sub-second-ttl panic.
- Add LockLostSignal: declare the lock lost when not_found exceeds
entries.len() - refresh_quorum; RPC errors are not counted (absorbed by the
ttl > interval margin). Expose is_lock_lost()/lock_lost() for observers.
- disarm(), release(), and Drop now abort the heartbeat before releasing so no
refresh races the unlock (refresh only extends, never creates, a lease).
- Reclaim path stays behaviorally unchanged but now warns with owner/resource/
lease age and records a metric (#698 scavenger preserved).
- Add DEFAULT_LOCK_REFRESH_INTERVAL, LockRequest.refresh_interval (serde default
for RPC back-compat) + builder, and lock lifecycle metrics.
Open questions adopt the design's documented defaults (marked TODO in code):
refresh not-found -> Ok(false)/error_info=None; lost-quorum base entries.len();
DEFAULT_LOCK_REFRESH_INTERVAL=10s.
Tests: heartbeat keepalive/quorum-loss/jitter/boundary/disarm (lock crate),
server refresh delegation (rustfs), and end-to-end survives-past-ttl plus
crashed-owner-reclaim regressions (namespace).
* perf(ecstore): move speculative tmp cleanup off the PUT hot path
On a successful PUT, rename_data has already moved the data dir out of the tmp workspace, so the delete_all(RUSTFS_META_TMP_BUCKET) at the end of SetDisks::put_object is a speculative no-op safety net. It was awaited inline on the response path, where profiling (backlog#924 / HP-3) showed the same-disk queueing behind fsync-heavy load turns a ~49us no-op into ~9ms average (p99 77ms, macOS F_FULLFSYNC amplified) added to every PUT.
Run that cleanup on a spawned task instead, keeping it as a real backstop (rename_data's remove_std only removes empty dirs and silently ignores failures). The failure path (quorum loss / rollback) keeps the cleanup inline so a failed PUT never returns with tmp shards still on disk. If the process dies before the spawned task runs, cleanup_stale_tmp_objects (24h expiry, 5-minute loop) reclaims the entry.
Scope note: ops/multipart.rs delete_all on RUSTFS_META_MULTIPART_BUCKET is intentionally untouched; it removes real leftovers and deferring it would widen CompleteMultipartUpload/Abort races.
Regression tests (hermetic SetDisks on formatted local disks, no global state): PUT success drains the tmp workspace (polling the spawned task), and PUT failure (missing bucket volume, rename_data quorum error after tmp shards were written) cleans the workspace inline before returning.
Ref: https://github.com/rustfs/backlog/issues/924
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): do not retry NotFound in reliable_rename_inner
reliable_rename_inner blindly retried the rename once on any error. A NotFound retry cannot succeed: nothing recreates the missing source or parent directory between attempts, so the second rename fails identically and speculative cleanup renames (e.g. move_to_trash on an already-removed tmp path) always paid for two syscalls.
Extract the retry decision into should_retry_rename: NotFound returns immediately, any other error keeps the existing single retry. This helper is shared by the rename_data commit path via rename_all, so behavior there is covered by a new rename_all success regression test alongside the retry-predicate tests.
Ref: https://github.com/rustfs/backlog/issues/924
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Step 1 of 4 for rustfs/backlog#922 (HP-1): write_all_internal previously
coupled file-content durability (fdatasync) with directory-entry
durability (parent dir fsync) behind a single bool. For tmp files that
are immediately renamed away, the tmp parent dir fsync contributes
nothing to crash consistency: the safe-rename recipe only needs file
content fdatasync -> rename -> fsync of the destination parent, because
the rename removes the tmp directory entry and a crash before the
rename means the PUT was never acknowledged.
Replace the bool with a SyncMode enum (None / FileAndDir / FileOnly)
and use FileOnly at exactly the two write-then-rename tmp write points:
the tmp xl.meta write in the non-inline rename_data path and the tmp
write inside write_all_meta. write_all_public (format.json etc.) and
the old-metadata rollback backup keep FileAndDir since those files stay
where they are written. The rest of the commit sequence (shard
sync_dir_files, rename, destination parent fsync) is untouched, and
behavior with drive sync disabled is unchanged.
This saves one directory fsync per disk per non-inline PUT (4 on a
4-drive set). Unit tests assert, via a test-only fsync-dir recorder,
that tmp write points no longer fsync their parent while the public
write point, the rollback backup, and the commit-rename destination
parent still do.
Co-authored-by: heihutu <heihutu@gmail.com>
Under a full nextest run on loaded machines the legitimately serialized
cross-disk commits in concurrent_resend_same_part_commits_one_generation
exceed the acquire deadline by themselves: observed at the 5s default,
at the 30s production default (#4370), and on CI even at 60s — which is
a hard ceiling, because fast_lock clamps every requested timeout to
MAX_ACQUIRE_TIMEOUT (60s) while the Timeout error still reports the
requested value, so raising the env override higher is a no-op.
Request the full 60s ceiling explicitly and cap the queue depth at three
concurrent resends, bounding the last waiter to two serialized commits
(~12s each on the slowest observed CI runner). Three resends still race
the streaming phase and contend on the commit lock, which is all the
generation-mixing regression (backlog#853) needs; every correctness
assertion is unchanged.
Co-authored-by: heihutu <heihutu@gmail.com>
fix(protocols): factor swift expiration mock result type into alias
The swift feature clippy matrix on main fails with clippy::type_complexity on the MockExpirationObjectBackend test struct introduced with the expiration worker tests, blocking CI for every open PR. Introduce a MetadataResult type alias in the test module; no behavior change.
Co-authored-by: heihutu <heihutu@gmail.com>
MetaObject::into_fileinfo indexed part_sizes[i]/part_actual_sizes[i] by
part_numbers.len() without checking the arrays are the same length,
unlike the adjacent part_etags/part_indices which are length-guarded.
decode_from pushes the three arrays independently and the xl.meta CRC
only covers bytes, so a CRC-valid but internally inconsistent xl.meta
(foreign writer / MinIO interop) triggers an out-of-bounds panic on the
GET/HEAD/LIST decode path.
Guard the three arrays for equal length and return Err(FileCorrupt) so a
divergent shard is skipped and quorum uses the other disks, instead of
panicking the request task. Cascade into_fileinfo to Result across its
callers, and fix io_primitives early-return to derive the version id from
the merged header and fall into the per-disk loop (single-disk survival +
heal). The 2118 merge-first path is left as a documented follow-up.
Refs backlog#900 (filemeta-01).
test(interop): real-MinIO read + migration parity, Phase 1/2 (backlog#580)
Capture authentic on-disk fixtures from MinIO RELEASE.2025-07-23 (a bucket with
versioning, object-lock, lifecycle, tagging, quota, a public policy, SSE-S3
encryption, a webhook notification target, and a replication rule, plus inline /
versioned / multipart objects and a delete marker) and prove RustFS reads and
migrates them losslessly:
- filemeta parses_real_minio_object_xlmeta: small inline, two-object-version +
delete marker, and multipart object xl.meta parse to the expected FileInfo.
- ecstore parses_real_minio_bucket_metadata_blob_without_loss: the MinIO
.metadata.bin msgpack decodes via the PascalCase field names and
parse_all_configs loads all ten config types present (policy, lifecycle incl.
<ExpiryUpdatedAt>, object-lock, versioning, tagging, quota, notification,
encryption/SSE-S3, replication incl. DeleteMarkerReplication /
ExistingObjectReplication) without loss.
- ecstore reads_minio_inline_bucket_metadata_via_bitrot: MinIO inlines an object
body as [HighwayHash256 32B][body]; RustFS's BitrotReader with HighwayHash256S
verifies and yields the exact blob (the "inline_data 前缀不同" is that prefix).
- ecstore migrates_real_minio_bucket_metadata_end_to_end: on a throwaway 4-drive
local ECStore, a real MinIO .metadata.bin seeded under .minio.sys is migrated
into .rustfs.sys byte-identically for every config, exercising the Phase 2
source adapter (MIGRATING_META_BUCKET = ".minio.sys") through the object layer.
All four run as ordinary crate tests (nextest CI). Phase 4 (MinIO re-reading a
RustFS drive) is documented as out of scope for one-way migration.
Refs rustfs/backlog#580
ENV_OBJECT_DIRECT_IO_ENABLE, DEFAULT_OBJECT_DIRECT_IO_ENABLE,
ENV_OBJECT_DIRECT_IO_THRESHOLD and DEFAULT_OBJECT_DIRECT_IO_THRESHOLD
were never read anywhere in the workspace. The real O_DIRECT read path
landed in PR #4365 uses RUSTFS_OBJECT_DIRECT_IO_READ_ENABLE /
RUSTFS_OBJECT_DIRECT_IO_READ_THRESHOLD (crates/ecstore/src/disk/local.rs).
Keeping the near-identically named dead knobs invites operators to set
the wrong variable and believe O_DIRECT is enabled.
Co-authored-by: heihutu <heihutu@gmail.com>
* refactor(ecstore): extract LocalIoBackend trait behind LocalDisk
Model the LocalDisk per-file I/O hot path as a LocalIoBackend trait
(pread_bytes / open_read_stream / open_full_read / open_write) and move
the existing method bodies verbatim into a default StdBackend. LocalDisk
holds Arc<dyn LocalIoBackend> and the DiskAPI methods forward to it.
This is a behavior-preserving refactor: no logic, error-mapping, metrics,
or cfg-branch changes. It creates the seam for an alternative
runtime-probed io_uring backend (rustfs/backlog#894) without touching
DiskAPI callers. Commit-point durability (fdatasync -> rename ->
fsync-dir in rename_data) deliberately stays outside the trait.
Add a differential test asserting all four read shapes return identical
bytes across page-boundary and file-tail ranges.
Tracking: rustfs/backlog#891 (parent rustfs/backlog#897)
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(ecstore): true O_DIRECT read path with per-disk graceful fallback (#4366)
* feat(ecstore): true O_DIRECT read path with per-disk graceful fallback
Implement a real O_DIRECT positioned read inside StdBackend::pread_bytes
(Linux only) and wire up the previously dead RUSTFS_OBJECT_DIRECT_IO_READ_*
knobs, which had zero call sites.
Open with rustix OFlags::DIRECT, probe the DIO alignment once per disk via
statx STATX_DIOALIGN (4096 fallback), read the aligned superset into an
alignment-allocated bounce buffer with a short-read loop, and slice out the
exact logical range so padding never reaches BitrotReader.
Any O_DIRECT failure falls back to the buffered read methods; EINVAL or
EOPNOTSUPP (tmpfs, overlayfs, 9p) latches the path off per disk with one
warning. O_DIRECT errors never surface: EINVAL maps to FileNotFound in
to_file_error and would trigger spurious EC rebuilds.
Default behavior is unchanged (knob off). macOS keeps F_NOCACHE.
Tracking: rustfs/backlog#892 (parent rustfs/backlog#897)
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(ecstore): add P1.5 benchmark gate harness for O_DIRECT reads (#4369)
* test(ecstore): add P1.5 benchmark gate harness for O_DIRECT reads
Add an ignored, Linux-only release-mode test (direct_read_bench_gate)
that measures DiskAPI::read_file_mmap_copy through a real LocalDisk with
cold-cache enforcement (fadvise DONTNEED between rounds), reporting
p50/p95/p99/mean latency, wall time, process CPU time, and throughput as
one JSON line for A/B diffing between the buffered baseline and the
O_DIRECT candidate selected by the production env knobs.
Content is verified byte-for-byte before any timing starts.
Tracking: rustfs/backlog#893 (parent rustfs/backlog#897)
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(ecstore): add concurrency dimension to P1.5 bench harness
Model the EC GET shape (FuturesUnordered over concurrent shard reads)
via RUSTFS_BENCH_CONCURRENCY (default 1, sequential as before). Needed
to evaluate blocking-pool pressure for the io_uring gate decision.
Tracking: rustfs/backlog#893
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* ci: retrigger after cancelled required check
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): use is_multiple_of in O_DIRECT bench cache-drop check
clippy's manual_is_multiple_of (rust 1.96) fails -D warnings on the benchmark helper's `done % file_count == 0`.
Verification:
- cargo clippy -p rustfs-ecstore --all-targets
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
A node could show OFFLINE in the console with zero logs explaining why
(rustfs/backlog#888, reported in rustfs#4304): the consecutive-failure
threshold crossing in handle_server_info_failure marked the peer offline
silently, the per-call warnings only existed on the observing node and
never named the transition, and the PeerRestClient offline flag plus its
background recovery monitor ran without any start/success log.
Logging-only change, no behavior change:
- handle_server_info_failure / handle_storage_info_failure: WARN
event="peer_marked_offline" exactly once at the threshold crossing
(later failures while already offline stay DEBUG), and DEBUG
event="peer_probe_failure" while returning cached state below the
threshold.
- update_server_info_cache / update_storage_info_cache: INFO
event="peer_recovered_online" when a probe succeeds after the peer had
been reported offline.
- PeerRestClient::mark_offline_and_spawn_recovery: WARN
event="peer_connection_marked_offline" when the offline flag is first
set (guarded by the recovery_running CAS so repeated failures do not
spam), and INFO event="peer_connection_recovered" with the attempt
count when connectivity is restored.
An "offline then back" episode now leaves a complete, correlatable
trace: N probe failures -> peer_marked_offline -> recovery monitor ->
peer_recovered_online.
Verification:
- cargo test -p rustfs-ecstore --lib (notification + peer_rest suites)
- make pre-commit
Ref: rustfs/backlog#888, rustfs#4304
Co-authored-by: heihutu <heihutu@gmail.com>
GitHub renders single newlines inside a paragraph as line breaks, so hard-wrapped PR/issue/discussion prose shows up with ugly mid-sentence breaks. Add a rule under Git and PR Baseline to keep each paragraph on one line.
concurrent_resend_same_part_commits_one_generation keeps failing on CI
with Lock(Timeout ... 5s) even after the fast-lock lost-wakeup fix
(NOTIFY_WAIT_CAP re-polling) landed — observed on rustfs#4365's Test and
Lint runs. Two independent causes exist and both are real:
1. the lost/stolen wakeup stall — fixed in fast_lock::shard; and
2. the six legitimately serialized cross-disk commits exceeding the
small 5s default acquire timeout under full-suite CI disk load.
The lost-wakeup fix reverted the earlier 30s test override on the
assumption that cause 1 was the whole story; CI shows cause 2 stands on
its own. Restore the temp_env override (production-default 30s) around
the concurrent section so the regression guard asserts the correctness
property (exactly one intact generation), not CI disk latency. The
NOTIFY_WAIT_CAP fix stays untouched.
Verification:
- cargo test -p rustfs-ecstore --lib concurrent_resend_same_part_commits_one_generation
Ref: rustfs/backlog#882
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(iam): load IAM bootstrap snapshot without namespace locks
IAM bootstrap (init_iam_sys -> load_all) read every config object with
default ObjectOptions (no_lock=false), so each read acquired a
distributed namespace read lock. Lock quorum is counted over cluster
nodes and unreachable peers are hard failures, so during a sequential
restart the very first read failed with
"Quorum not reached: required 2, achieved 0" and IAM could not come up
until enough peers' lock RPC surfaces converged - even when the storage
read quorum was already satisfiable (rustfs#4304).
Extend the startup contract from rustfs#4056 ("startup metadata I/O must
not require namespace locks") to the IAM bootstrap path:
- Introduce LoadMode {Locked, BootstrapNoLock} and plumb it through the
load_all chain (groups, users, policies, mapped policies and their
concurrent variants) down to the storage read options.
- load_all now performs all reads with no_lock=true; on-line
single-object loads via the Store trait keep locked semantics.
- Fail-closed behavior is unchanged: any loader error still aborts the
whole snapshot load.
Safety: config objects are atomic whole-object writes, so a lock-free
read only observes an old or a new value; staleness is bounded by the
existing periodic IAM reload. Listing (walk) never took namespace locks,
and maybe_schedule_lazy_rewrite stays a best-effort background task.
Verification:
- cargo test -p rustfs-iam --lib (153 passed, incl. new LoadMode tests)
- cargo clippy -p rustfs-iam --all-targets
- cargo check -p rustfs
- make pre-commit
Ref: rustfs#4304; tracking rustfs/backlog#884, rustfs/backlog#885
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(iam): sequential-restart regression test for lock-free bootstrap
Add an integration test reproducing the rustfs#4304 failure mode against
a real 4-disk temp-dir ECStore:
- Seed IAM group data in single-node mode, then flip the runtime into
distributed-erasure mode. new_ns_lock now builds a distributed lock
over the set's (empty) lock-client list, so every namespace-locked
read fails exactly like a sequential restart with unreachable peers
(lock quorum unavailable, storage read quorum healthy).
- Assert the locked load_group path fails in that state, while the
bulk snapshot load_all (no_lock plumbing from the previous commit)
succeeds, and the data survives intact once single-node mode is
restored.
Reverse-verified: temporarily switching load_all back to the locked
mode makes the test fail, so it genuinely guards the contract.
Verification:
- cargo test -p rustfs-iam --test iam_bootstrap_no_lock_test
- cargo test -p rustfs-iam --lib
Ref: rustfs#4304; tracking rustfs/backlog#886
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(iam): route test ECStore imports through ecstore_test_compat boundary
The new integration test imported rustfs_ecstore facade paths directly,
tripping three architecture migration rules. Move every ECStore import
behind crates/iam/tests/ecstore_test_compat/mod.rs (the sanctioned
test-compat pattern), and register that module as a reviewed test-only
global-facade boundary in check_architecture_migration_rules.sh: the
sequential-restart regression test needs api::global::update_erasure_type
to flip into distributed-erasure mode for lock-quorum fault injection.
Verification:
- ./scripts/check_architecture_migration_rules.sh
- cargo test -p rustfs-iam --test iam_bootstrap_no_lock_test
- make pre-commit
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(server): expose readiness blocking reason + rolling-restart runbook
Operators hitting the rustfs#4304 sequential cold start could not tell
from the outside why a node stayed unavailable. Three additions:
- The readiness gate's 503 now names the blocking dependency in both the
body ("Service not ready: waiting for storage_quorum") and a new
x-rustfs-readiness-pending header (storage_quorum | iam |
startup_finalization), derived from the current startup stage.
/health/ready already returned details + degradedReasons; this covers
the plain S3 requests that hit the gate.
- IAM bootstrap retry logs now carry an actionable `hint` field that
classifies the failure (storage read quorum vs lock quorum vs
uninitialized metadata) instead of only echoing the storage error.
- New docs/operations/rolling-restart.md runbook: correct rolling
restart procedure, sequential cold-start expectations (degraded ->
auto-recovery), readiness signal reference, and
RUSTFS_STARTUP_READINESS_MAX_WAIT_SECS guidance.
Verification:
- cargo test -p rustfs --lib -- hint_tests service_not_ready readiness_pending
- make pre-commit
Ref: rustfs#4304; tracking rustfs/backlog#887
Co-Authored-By: heihutu <heihutu@gmail.com>
* upgrade deps version and improve import
* feat(iam): notification-path cache refreshes read without namespace locks (#4368)
P3 step 1 of rustfs/backlog#884 (scoped down from full MinIO readConfig
alignment after review): cross-node notification handlers
(group/policy/policy-mapping/user) refresh the local IAM cache with
single-object reads that previously took distributed namespace read
locks. These refreshes are asynchronous, best-effort, and already
stale-tolerant (the periodic reload converges them), so a node-counted
lock quorum failure or lock RPC hiccup on a peer must not fail them —
the same rationale as the lock-free bootstrap load_all (rustfs#4304).
- Store trait: add load_user_no_lock / load_group_no_lock /
load_policy_doc_no_lock / load_mapped_policy_no_lock with defaults
forwarding to the locked variants, so existing implementations and
test mocks keep their behavior.
- ObjectStore overrides them via the existing LoadMode::BootstrapNoLock
plumbing. Deletions triggered by the handlers keep locked writes.
- manager.rs: the four *_notification_handler paths (8 call sites)
switch to the lock-free variants.
- Integration test: while the lock quorum is unavailable (DistErasure
with empty lockers), load_group_no_lock must succeed exactly where
the locked load_group fails.
Request-path loads (check_key, verify_temp_user_persistence) and admin
write-then-reload paths intentionally stay locked: load_user_identity
embeds expiry deletions, so those need the side-effect extraction
tracked in rustfs/backlog#884 before going lock-free.
Verification:
- cargo test -p rustfs-iam --lib (156 passed)
- cargo test -p rustfs-iam --test iam_bootstrap_no_lock_test
- make pre-commit
Ref: rustfs/backlog#884, rustfs#4304
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(server): drop unused iam_bootstrap_failure_hint import in tests
The hint tests live in their own hint_tests module with a local import;
the stale re-import in mod tests failed clippy's -D warnings on the
Test and Lint CI variants.
Verification:
- cargo clippy -p rustfs --all-targets
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Drop the [target.'cfg(target_os = "linux")'.dependencies] tokio
"io-uring" feature from 6 crates and the rustfs binary. Because
.cargo/config.toml enables --cfg tokio_unstable globally, this feature
switched every Linux build's file I/O onto tokio's io_uring runtime
backend. Restricted Linux environments (Docker default seccomp, gVisor,
proot, old kernels) reject io_uring_setup with EACCES/ENOSYS, which
tokio surfaced as PermissionDenied and RustFS reported as
DiskAccessDenied at startup.
Add scripts/check_no_tokio_io_uring.sh so the feature cannot silently
return: it fails on any tokio dependency line enabling "io-uring", while
still allowing a future application-level io-uring crate dependency.
Wire it into make pre-commit/pre-pr/dev-check and CI.
Tracking: rustfs/backlog#890 (parent rustfs/backlog#897)
Co-authored-by: heihutu <heihutu@gmail.com>
Zhengchao An