* fix(cache): gate GET body-cache hook to preserve ReadPlan output
The ecstore GET body-cache hook serves cached full-object plaintext
directly, bypassing ReadPlan/ReadTransform. That is only sound when the
normal read path returns that same plaintext byte-for-byte. Two probe
conditions were missing, plus two usecase-layer planner gaps.
ODC-01 (backlog#1108): raw/data-movement reads. ReadPlan::build returns
the STORED representation for raw_data_movement_read (e.g. compressed
bytes, length = oi.size), but the cache holds the post-decompression
body. Decommission (raw_data_movement_read: true) would receive
decompressed plaintext where raw compressed bytes are required, silently
corrupting the destination pool.
ODC-02 (backlog#1109): compressed objects. ReadTransform::Compressed
rewrites object_info.size to the decompressed length; on a hook hit
object_info is returned unchanged, so object_info.size is the compressed
size while the stream carries the decompressed body. UploadPartCopy then
uses src_info.size as the copy length and truncates the part.
Fix: gate the hook probe with should_probe_body_cache_hook, refusing
raw_data_movement_read, data_movement, and compressed objects, mirroring
the conditions get_small_object_direct_memory_decision already applies.
ODC-33 (backlog#1138): build_get_object_body_cache_plan lacked the
is_remote() exclusion the ecstore hook enforces; add it so transitioned
(remote-tier) objects are excluded uniformly.
ODC-C1 (backlog#1142): zero-length bodies save no I/O (ecstore returns an
empty body before the hook probe) yet the planner admitted them; change
the guard to response_content_length <= 0 so they plan Skip, mirroring
should_buffer_get_object_in_memory_with_threshold.
Tests: body_cache_hook_gate_tests (4) cover plain-probe plus
raw/data-movement/compressed skips; planner gains
plan_skips_remote_transitioned_objects and plan_skips_zero_length_objects.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(cache): allow compressed bodies via a fail-closed read allow-list
The body-cache hook probe was gated by a deny-list that refused compressed
objects outright, which cost the cache every compressed body — a growing
share of stored data. Replace it with an allow-list that returns the exact
plaintext length a hit may serve, or None.
full_object_plaintext_len() answers a single question: would the normal
ReadPlan produce this object's complete plaintext, and under which size?
Compressed objects now qualify, and the hit site publishes the returned
length as object_info.size, reproducing the contract ReadTransform::
Compressed establishes. A hit whose body length disagrees is refused and
falls through to the erasure read.
This also closes a gate the deny-list only covered by accident: a restore
read forces ReadPlan down the Plain branch, so a compressed object yields
STORED bytes under its compressed size. Refusing compressed objects hid
that; admitting them exposes it, so restore reads are refused explicitly.
Being fail-closed, a newly added ReadPlan branch bypasses the cache by
default rather than silently serving the wrong representation — the
structural defect behind both backlog#1108 and backlog#1109.
Refs: backlog#1108, backlog#1109
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
A buffered read that hits the page cache completes inline inside
`io_uring_enter`, so the thread driving a ring performs that read's
memcpy. One ring per disk therefore capped cache-hit reads at a single
core's memory bandwidth: measured on a 16-core host, one driver thread sat
pinned at 100% CPU while throughput stayed flat at ~5 GB/s regardless of
read size, against 50 GB/s for the blocking-pool baseline.
rustfs/uring#6 taught the driver to hold N independent rings, each with its
own thread, pending table, backpressure semaphore, and eventfd. Wire it up:
`UringBackend::try_new` now calls `probe_and_start_sharded`, and
`RUSTFS_IO_URING_SHARDS` selects the count per disk.
The default is a quarter of the available parallelism clamped to `1..=4`,
because the cost is `disks × shards` driver threads (each normally blocked
in `poll(2)`). Any override is clamped to `1..=16`, so a mistyped value can
neither disable the driver (0) nor spawn threads without bound; an
unparseable value falls back to the default.
Effect (warm page cache, 16-core, rustfs/uring's concurrent_pread_bench):
1 MiB, conc 8: 1 shard 4911 MB/s -> 8 shards 47361 MB/s (9.6x);
the blocking-pool baseline is 50662 MB/s
64 KiB, conc 32: StdBackend 153678 IOPS, p999 3030 us
8 shards 345402 IOPS, p999 897 us
64 KiB, conc 128: StdBackend 135155 IOPS, p999 10716 us
8 shards 389047 IOPS, p999 4092 us
Sharding removes the throughput deficit *and* keeps io_uring's tail-latency
advantage, rather than trading one for the other.
Unchanged: io_uring read stays gray-off by default
(`RUSTFS_IO_URING_READ_ENABLE`), reads are byte-for-byte identical to
StdBackend, the per-disk degradation latches and probe cache (backlog#1101)
and the O_DIRECT tiered fallback (backlog#1102) all still apply. Rings stay
per-disk, so a stalled disk cannot starve another disk's rings
(backlog#1055). Bumps the rustfs-uring pin to the merged #6 commit.
Verified on a real Linux host (16-core, real io_uring): cargo clippy
--tests -D warnings clean; disk::local tests 132 passed, 0 failed —
including the existing io_uring and O_DIRECT cases now running on the
sharded driver, plus a new test covering the shard-count default, override,
and clamping.
Co-authored-by: heihutu <heihutu@gmail.com>
The listing path no longer has a wrapper-level total timeout, so any
filesystem call the walk makes without a stall bound would have no liveness
bound at all. The first pass covered list_dir and read_metadata but left four
reads unbounded: the volume access() probe and fs::metadata() in walk_dir, the
xl.meta read in its dir-object branch, and is_empty_dir() in scan_dir.
Split the helper so reads that do not yield a Result go through the same
budget, and route the four remaining calls through it. A hung drive now fails
those reads with DiskError::Timeout instead of parking the walk until the
merge consumer's own stall timeout aborts it.
Co-authored-by: heihutu <heihutu@gmail.com>
fix(ecstore): keep O_DIRECT for eligible reads via native io_uring (rustfs/backlog#1102)
Wire ecstore's io_uring read backend to rustfs-uring's native aligned
O_DIRECT read (`read_at_direct`, merged in rustfs/uring#3), so an
O_DIRECT-eligible read keeps BOTH io_uring's async submission AND
O_DIRECT's page-cache bypass instead of trading one for the other.
`UringBackend::pread_bytes` now selects the read shape by a tiered,
per-disk-latched ladder — never a blanket downgrade:
1. io_uring latched off for this disk (backlog#1101) -> StdBackend.
2. O_DIRECT-eligible + native path usable -> pread_uring_direct: open
the file O_DIRECT, probe the device alignment once (cached), and
read via read_at_direct. Best path: async + no cache pollution.
3. O_DIRECT-eligible but the fs refused O_DIRECT earlier (latched)
-> StdBackend aligned path (which itself degrades to buffered).
4. non-O_DIRECT read -> buffered pread_uring.
Latching keeps a failure from being re-attempted per-read: an fs that
refuses O_DIRECT (EINVAL/EOPNOTSUPP on open) latches the native direct
path off for that disk; a restriction-class errno from the read latches
io_uring off wholesale (backlog#1101). Any other error falls back for
that one read without masking a genuine data problem as a permanent
downgrade. The alignment comes from the existing probe_direct_io_align
(statx STATX_DIOALIGN, default 4096) and is cached in a per-disk
OnceLock so the probe runs at most once.
This replaces the interim guard that routed every O_DIRECT-eligible read
to StdBackend (which disabled io_uring on the hottest shard reads); the
native path is now the default and StdBackend is only a fallback tier.
Bumps the rustfs-uring pin to the merged #3 commit.
Verified on a real Linux host (16-core, real io_uring + O_DIRECT):
cargo check --tests, clippy -D warnings, and disk::local tests all pass
(128 passed, 0 failed), including uring_preserves_o_direct_for_eligible_reads
across unaligned ranges.
Co-authored-by: heihutu <heihutu@gmail.com>
Foreground S3 listings wrapped the entire walk_dir stream in a single
wall-clock timeout (RUSTFS_DRIVE_WALKDIR_TIMEOUT_SECS, default 5s). That
budget measured how much data the walk had to produce rather than whether
the drive was still answering, so a healthy but large prefix on slow media
returned 500 InternalError / "Io error: timeout" to the client. The timer
also kept running while the walk was blocked writing to a slow consumer,
charging merge-side backpressure to the producer.
WalkDirOptions::stall_timeout_ms already carried the right semantics but was
only honored by the remote-disk RPC walk; LocalDisk::walk_dir ignored it
entirely. A single-drive deployment therefore had no way to distinguish a
hung drive from a big directory.
Teach LocalDisk::walk_dir to bound each individual drive read with the stall
budget, defaulting it from RUSTFS_DRIVE_WALKDIR_STALL_TIMEOUT_SECS when the
caller does not pin one, and let the foreground listing path skip the
wrapper-level total timeout. A walk that keeps making progress now runs to
completion; a drive that stops answering still fails with DiskError::Timeout.
Time spent blocked on the consumer stays outside the budget.
Heal and rebalance walks already skipped the total timeout and previously ran
unbounded on local drives. Give them an explicit, generous 60s stall budget so
this change does not tighten them from "no bound" to the 5s default.
Fixes#4644
Refs #2999, #3001
Co-authored-by: heihutu <heihutu@gmail.com>
fix(ecstore): keep O_DIRECT for eligible reads when io_uring is enabled (rustfs/backlog#1102)
UringBackend::pread_uring opens the file buffered, so with io_uring enabled an
O_DIRECT-eligible read (RUSTFS_OBJECT_DIRECT_IO_READ_ENABLE + length >= the
threshold) silently lost O_DIRECT and polluted the page cache — a behavior
change the moment io_uring was turned on for a disk that uses O_DIRECT. The
io_uring backend is gray-off by default, so no deployment is affected, but the
gap had to close before it can be enabled anywhere O_DIRECT is in use.
Route O_DIRECT-eligible reads back through StdBackend's aligned path (which
itself falls back to buffered when the filesystem rejects O_DIRECT). A native
aligned O_DIRECT read in the driver remains part of rustfs/backlog#1102.
Adds uring_preserves_o_direct_for_eligible_reads: with BOTH flags on and the
threshold at 1, unaligned ranges still return exactly the requested bytes.
Co-authored-by: heihutu <heihutu@gmail.com>
backlog#1052 S7 — the final piece: full bucket-namespace isolation
between embedded servers in one process.
Server B's requests resolved B's own ECStore (per-server dispatch landed
earlier), but the store's bucket operations still went through ambient
process facades, so both servers effectively operated on the FIRST
server's disks and metadata:
- LocalPeerS3Client::local_disks_for_pools() called all_local_disk()
(the ambient disk registry = the first published store's context), so
list/make/delete/heal bucket scanned and wrote the wrong volumes.
- BucketMetadata::save() persisted through the ambient object handle, so
a second server's bucket metadata landed in the first server's
.rustfs.sys; set/remove/get/created_at all used the ambient metadata
system.
Now the whole chain is bound to the owning store's InstanceContext:
- S3PeerSys/LocalPeerS3Client gain *_with_instance_ctx constructors and
operate on that context's registered disks; ECStore::new (and the test
store builder) pass the store's context. The legacy constructors keep
the bootstrap default.
- BucketMetadata::save_with_store persists through an explicit store;
BucketMetadataSys::persist_and_set uses the system's own api handle.
- metadata_sys gains instance-scoped variants (get_in / created_at_in /
set_bucket_metadata_in / remove_bucket_metadata_in) that resolve the
context's metadata system and fall back to the ambient default before
the instance cell is initialized (early startup, unchanged behavior).
- The store's bucket handlers (make/get_info/list/delete + the
table-bucket delete guard and emptiness check) use the per-context
variants and this instance's disks.
Acceptance (e2e): two embedded servers with different credentials are now
isolated end to end — each authenticates only its own key, neither sees
the other's buckets or objects, and both data planes stay intact. The
embedded module doc drops the shared-IAM caveat.
579 ecstore bucket/metadata/peer regressions plus the embedded basic and
deferred-IAM e2e stay green.
test(ecstore): cover the io_uring per-disk probe cache hit (rustfs/backlog#1101)
Follow-up to #4635. The per-disk negative probe cache was implemented but had
no dedicated test. Add uring_probe_cache_skips_known_unsupported_disk: a root
recorded in URING_UNSUPPORTED_DISKS makes try_new return None without a fresh
probe. Completes the #1101 acceptance ("cache hit does not re-probe").
Co-authored-by: heihutu <heihutu@gmail.com>
The startup topology convergence auto-detection (added in #4631) checked
Kubernetes before endpoint style, so a local path deployment (single or
multi-drive, non-distributed) running inside Kubernetes was classified as
orchestrated with an effectively unbounded wait window. Path-style
endpoints have no hostnames to resolve, so the wait never actually
triggers and runtime behavior is unchanged, but the startup log then
advertised mode=orchestrated / wait_timeout=MAX for a purely local
deployment, which is misleading and contradicts the documented policy
(local path endpoints -> fail-fast).
Order the auto-detection by endpoint style first: only distributed URL
endpoints, which resolve hostnames, pick orchestrated (Kubernetes) or
bounded (otherwise); local path endpoints stay fail-fast regardless of
the platform. Update the doc comment to match and add a policy case
locking Kubernetes + local path to fail-fast.
Co-authored-by: heihutu <heihutu@gmail.com>
feat(ecstore): io_uring runtime degradation latch + per-disk probe cache (rustfs/backlog#1101)
Follow-up to #1104. Add the runtime half of the io_uring degradation contract:
- Runtime latch: UringBackend gains an `active` flag (mirrors
DirectIoReadState::supported). A read that returns a restriction-class errno
(EPERM/EACCES/ENOSYS/EOPNOTSUPP) — io_uring became unusable on this disk —
trips the latch, and every further read goes straight to StdBackend with no
more per-read io_uring attempts. Data errors (EIO), missing files, and
parameter errors do NOT latch, since StdBackend would hit them too.
- Per-disk probe cache: a process-wide negative cache of disks whose probe
failed, so try_new skips re-probing (creating a ring + driver thread) on
reconnect.
Tests: the errno classifier (restriction-only) and a latched-off read that
still returns correct bytes via StdBackend; the #1104 differential test still
passes under real io_uring.
Co-authored-by: heihutu <heihutu@gmail.com>
The OnceLock introduced in 5fbd49a80 for bucket_metadata_sys panics when
multiple test modules initialize their own ECStore against the shared
bootstrap context. Switch to Mutex<Option> so test fixtures can
reinitialize safely.
Extract shared gating test environment to gating_test_env.rs to avoid
duplicate ECStore setup across delete_objects_stat_gating_test and
put_prelookup_gating_test.
Fixes 8 test failures:
- capacity_dirty_scope_test (2 tests)
- delete_objects_stat_gating_test (3 tests)
- put_prelookup_gating_test (3 tests)
* feat(embedded): allow multiple embedded servers to coexist in one process
backlog#1052 S5: turn the embedded startup guard into a sequential lock
and make every write-once shared cell tolerate a second embedded start.
Two RustFSServers on different ports and volumes now start, run, and
shut down independently in the same process.
- EMBEDDED_SERVER_STARTED is released once each startup hands off; a
second startup that runs after the first is no longer rejected.
- The second embedded server constructs its own InstanceContext instead
of adopting the process bootstrap one, so region/endpoints/deployment
id land on that context (the first server keeps adopting bootstrap to
keep single-instance ambient facades unchanged).
- Startup-time tolerant paths:
- action_credentials publish treats AlreadyInitialized as success
(per-server ActionCredentialHandle already holds the real creds).
- GLOBAL_RUSTFS_PORT warns instead of panicking on a second set.
- Observability install returns Ok when the process subscriber is
already set — the second server reuses it.
- New acceptance test proves two servers start, respond on their own
ports, and one can be shut down without disturbing the other; the
survivor keeps serving S3 requests. IAM and root-credential lookup
still share a process domain (a second server whose creds differ from
the first will fail signature validation), tracked as a follow-up.
- Embedded doc rewritten: 'Limitations' → 'Multi-instance status'; the
AlreadyStarted error is now scoped to concurrent startups only.
The remaining work in #1052 is the auth path per-server dispatch and the
matching data-plane routing so two servers with different credentials
serve independent buckets end-to-end.
* feat(app): per-server auth and application context for multiple embedded servers (#4633)
backlog#1052 S6: each embedded server now authenticates against — and
its request path resolves — its OWN application context, so two servers
with different root credentials each accept their own access key and
reject the other's.
- AppContext is per-server: ensure_startup_after_iam constructs a fresh
context around this server's store + IAM + KMS and installs it into the
server's own ServerContextSlot, then publishes it as the process
default first-writer-wins (publish_global_app_context) for legacy
ambient readers. The old 'reuse the global if present' path is gone.
- FS::check (the S3 data-plane access gate) resolves auth against
self.server_ctx's context: check_key_valid gains a _with_context
variant that takes the root credentials and IAM system from an explicit
context (None = ambient, unchanged for all 140+ existing callers). The
region and the server context slot are published into the request
extensions for downstream handlers.
- Each embedded server seeds its own root credentials into its context
(ActionCredentialHandle.publish) at startup, so credential validation
no longer falls back to the first server's process-global identity.
- The bucket/object/multipart use-cases resolve their store from the
server's context (bucket_usecase_for/object_usecase_for/... take &FS).
New acceptance test: two servers with distinct credentials each
authenticate with their own key and reject the other's.
KNOWN FOLLOW-UP: full bucket-namespace isolation still requires threading
the instance context through the lower ecstore data plane (peer_sys /
disk registry / bucket-metadata reads still resolve via the process
GLOBAL_OBJECT_API), so the two servers do not yet present independent
bucket listings even though each holds its own store. That deeper pass —
a continuation of the #939 object-graph ctx threading — is the remaining
work on #1052.
Stacked on the S5 guard change.
Multi-node startups (Kubernetes StatefulSets, bare-metal, VMs) can hit
transient DNS/local-host resolution failures while peers and headless
service records are still coming up. Previously endpoint resolution
retried DNS for a fixed 90s window and then returned an error, which
propagated out of run() and exited the process; in Kubernetes this drove
repeated kubelet restarts even though the cluster eventually converged.
Introduce a startup topology convergence policy with three modes:
- orchestrated: wait effectively indefinitely for recoverable
DNS/topology errors so the process stays Running (readiness stays
false) instead of exiting; defer the DNS-IP cross-port validation
because headless-service records can still be flapping.
- bounded: wait a finite window (default 3m) then fail with an
action-oriented error listing host/stage/elapsed and remediation hints.
- fail-fast: fail on the first non-transient resolution error.
Mode is auto-detected (Kubernetes -> orchestrated, distributed URL
endpoints -> bounded, local path endpoints -> fail-fast) and can be
overridden via RUSTFS_STARTUP_TOPOLOGY_WAIT_MODE,
RUSTFS_STARTUP_TOPOLOGY_WAIT_TIMEOUT and
RUSTFS_STARTUP_TOPOLOGY_RETRY_MAX_DELAY.
Also normalize divergent local/remote verdicts for endpoints that share
a host:port, throttle retry warnings, and keep the DNS-independent local
same-path checks in every mode. Configuration error handling is
unchanged: malformed volumes, mixed styles/schemes, duplicate or root
paths, and non-transient errors still fail fast.
Read the topology env vars through rustfs_utils::get_env_opt_str for
consistent alias handling, and simplify the log_once poisoned-lock branch
to unwrap_or_else(PoisonError::into_inner).
Co-authored-by: heihutu <heihutu@gmail.com>
feat(ecstore): add runtime-probed io_uring read backend, gray-off by default (rustfs/backlog#1104)
Wire the standalone rustfs-uring crate (https://github.com/rustfs/uring) into
LocalIoBackend as an opt-in read backend. When RUSTFS_IO_URING_READ_ENABLE is
set AND the per-disk io_uring probe succeeds, positioned reads go through the
cancel-safe UringDriver; otherwise — default, or on a restricted host, or on
any per-read driver error — reads fall back to StdBackend byte-for-byte, so the
default build is unchanged.
- Cargo.toml: rustfs-uring as a Linux-only git dependency (pinned rev). The
guard scripts/check_no_tokio_io_uring.sh allows an explicit io-uring
integration; only the tokio "io-uring" runtime feature is banned.
- UringBackend mirrors StdBackend::pread_bytes's resolution/access/bounds
preamble and only swaps the raw byte read; the other three trait methods
delegate to the inner StdBackend.
- Backend selection at LocalDisk construction via build_local_io_backend.
- Differential test uring_backend_reads_match_std: with the flag set, every
positioned read returns byte-identical data (driver-served when io_uring is
available, StdBackend fallback otherwise).
Verified: cargo check -p rustfs-ecstore on Linux; the differential test passes
under real io_uring (seccomp=unconfined). The runtime errno degradation latch,
per-disk probe cache, and productionization are tracked in
rustfs/backlog#1101/#1102.
Co-authored-by: heihutu <heihutu@gmail.com>
backlog#1052 S3, fourth slice. The AppContext's IamHandle already owned an
Arc<IamSys>, but the value it held was read back from the process
singleton after init — so a future second server's context would silently
bind to the first server's IAM domain (credentials, policies, users all
belong to a specific store's .rustfs.sys).
- rustfs_iam gains build_iam_sys(store): construct an IAM system bound to
a store without touching the singleton. init_iam_sys wraps it (build,
publish first-wins, return the handle — previously Result<()>).
- Startup threads the built handle: the inline bootstrap path passes the
Arc it just created into ensure_startup_after_iam, which constructs the
AppContext from the passed value instead of re-resolving the global.
The deferred-recovery path resolves the freshly published global
directly (context-first resolution cannot be used there: the AppContext
does not exist yet — creating it is the finalizer's job; caught by the
embedded deferred-IAM e2e).
- IamHandle::is_ready() reports the held system's readiness instead of
consulting the process singleton; the dead global re-resolver is
removed. token_signing_key stays on the credentials global until S4.
157 iam tests + embedded e2e (basic + deferred recovery) green.
backlog#1052 S3, third slice — the hard blocker for a second embedded
server's service stage. GLOBAL_BUCKET_METADATA_SYS was a process-global
OnceLock whose init ran .set().unwrap(): a second instance's service
startup panicked the whole process. The system it guards is inherently
per-store (it holds the store handle and that store's bucket→metadata
cache), so it now lives on the store's own InstanceContext:
- init_bucket_metadata_sys writes the cell of the store it was handed
(api.ctx) — no signature change — and keeps the double-init fail-fast,
now scoped to the instance (assert on the per-context set). The
dist-erasure check for the refresh loop reads the same context.
- get_global_bucket_metadata_sys / the crate-private getter behind the
~20 get_*_config helpers resolve through the current_ctx() facade —
the published store's context, or bootstrap before that — so every
existing reader keeps its exact single-instance behavior.
- New acceptance test: two real stores in one process each initialize
their own metadata system (the old global cell panicked right there),
plus a shared builder extracted from the store-graph test.
201 bucket/metadata regression tests green.
* refactor(app): give each context's server-config handle its own copy
backlog#1052 S3, first slice: establish the per-context state pattern on
the smallest subsystem. ServerConfigHandle was a unit struct forwarding
both get and set to the process-global GLOBAL_SERVER_CONFIG, so every
AppContext shared one config regardless of which context a caller held.
The handle now keeps its own copy: a set lands on the owning context and
on the process global (ambient readers — the config loader path, the
scanner — keep working), and a get prefers the owned copy, falling back
to the global while the initial load still publishes there. Single
instance: the owned copy and the global are written together, so reads
are unchanged. Two contexts that both published stay isolated even though
the shared global only remembers the last write (covered by a new test).
The global write in set() is the transitional bridge for ambient readers;
it goes away when those readers migrate and multi-instance flips on
(backlog#1052 S5).
* refactor(notify): hand out the notification system as an owned Arc
backlog#1052 S3, second slice. GLOBAL_NOTIFICATION_SYS stored the
NotificationSys by value and every accessor returned
Option<&'static NotificationSys> — a process-lifetime borrow that a
per-server AppContext can never hold. The global now stores an Arc and
the accessor chain (ecstore runtime sources, ECStore::notification_system,
the iam forwarders, the rustfs shims, NotificationSystemInterface and the
AppContext resolvers) hands out owned Arcs instead.
Call sites are unchanged apart from two borrow adjustments in the
rebalance admin handler (pass &Arc where a reference is expected; borrow
with as_ref() so the handle survives to the second use). Behavior is
identical — same single global instance, first init still wins — but the
type now permits a future per-server context to own its notification
system (backlog#1052 S3 follow-up).
Two follow-up fixes to #4607 (both verified real bugs, each with a regression
test):
1. The in-call server_info retry did not re-dial on network errors. A
network-like first failure runs through `finalize_result`, which sets the
client's `offline` gate; the retry then called `evict_connection` and
re-invoked `server_info`, but `get_client` short-circuits on that gate and
returns "temporarily offline" without dialing. Only the async recovery
monitor would clear it. So the retry re-dialed only in the timeout branch and
was a no-op in the transport/half-open branch it was meant to cover. Add
`PeerRestClient::prepare_retry` (evict + clear the offline gate) and use it
before the retry.
2. Synthesized/degraded drive lists went empty on hostname deployments.
`PeerRestClient::host` is an `XHost` that `hosts_sorted` builds via
`XHost::try_from` -> `to_socket_addrs`, so it is the resolved `IP:port`; but
`synthesized_disks`/`peer_disk_health` compared it against
`Endpoint::host_port()`, which is the raw `hostname:port`. On hostname
clusters the compare missed, the drive list came back empty, and
`unknownDisks` stayed 0 — reproducing the "drives vanish from the summary"
regression #4607 fixed (also affected the pre-existing offline path). Compare
through a shared `endpoint_host_matches` helper that canonicalizes the
endpoint side through the same `XHost` resolution.
Tests: `peer_rest_client_prepare_retry_clears_offline_gate`,
`endpoint_host_matches_direct_and_canonicalized` (uses localhost, no external
DNS).
Follow-up to rustfs/rustfs#4607; tracked in rustfs/backlog#1049.
Co-authored-by: heihutu <heihutu@gmail.com>
fix(ecstore): read the persisted SSE-C nonce in the GET decrypt resolver
#4576 moved SSE-C encryption to a random per-encryption nonce persisted
in object metadata and taught the rustfs-layer decrypt resolver to read
it back — but the byte-level GET decryption resolves its material in
crates/ecstore object_api/readers.rs, a duplicated twin that still
recomputed the deterministic legacy nonce. Encrypt with the random
nonce, decrypt with the deterministic one: the first AEAD block fails
and every SSE-C GET aborts its body after the response headers
(IncompleteRead(0 bytes) on clients; all 8 SSE-C cases in the
implemented s3-tests whitelist), which is what has been driving the
"S3 Implemented Tests" CI job into its 60-minute timeout since
2026-07-09.
Resolve the base nonce like the encrypt side: prefer the persisted IV
(x-rustfs-encryption-iv, then the case-insensitive MinIO interop key),
fall back to the deterministic nonce only for legacy objects with no
stored IV or an undecodable value. Full implemented s3-tests whitelist
is back to 451 passed / 0 failed against the fixed binary.
test(e2e): align object-lock overwrite tests with AWS new-version semantics
PR #3179 made put-like writes (PUT, CopyObject, multipart create/complete)
on versioned object-lock buckets create a new version instead of being
blocked by the current version's legal hold or COMPLIANCE retention, which
is the correct AWS behavior. Six e2e assertions in object_lock_test.rs
still encoded the old blocking semantics and, because e2e_test is excluded
from the CI test run, stayed red unnoticed since 2026-06-03.
Verified against a real server built from current main: all six failed at
their "should be blocked" assertions. Rewritten to assert the AWS
contract: the write succeeds with a distinct new version id, the new
content is current, the protected version keeps its content and lock
metadata, and version-targeted deletion of the protected version stays
blocked (for COMPLIANCE even with bypass_governance). Full object_lock
module is green again (33/33).
PR #4468 removed the last use of wildmatch (crates/notify swapped it for a
hand-rolled star-only glob matcher), but the workspace dependency declaration
in the root Cargo.toml was left behind as an orphan. cargo shear flags it as
not used by any workspace member. Drop the stray line.
Also refresh the comments in crates/notify/src/rules/pattern.rs that still
referred to the removed wildmatch crate, so they describe the current
hand-rolled matcher instead of a dependency that no longer exists.
Co-authored-by: heihutu <heihutu@gmail.com>
fix(admin): report unreachable info members as `unknown` with balanced drive counts
`GET /rustfs/admin/v3/info` synthesizes an entry for every peer, but a peer
whose properties RPC failed below the offline threshold with no cached snapshot
was reported as `initializing` with an empty drive list. That drive list being
empty meant its drives disappeared from the backend summary entirely, so
`onlineDisks + offlineDisks` no longer equaled `totalDrivesPerSet` (an operator
sees `onlineDisks: 3, offlineDisks: 0` for a 4-drive set), and `initializing` is
a misleading state for a member that has been running for hours — it reads like
a node was ejected when nothing is wrong (upstream rustfs/rustfs#4566).
Changes:
- madmin: add `ItemState::Unknown` ("unknown") and an `unknownDisks` bucket on
`ErasureBackend` (serde `default`, so older payloads still deserialize).
- ecstore diagnostics: `get_online_offline_disks_stats` now returns a third
`unknown` bucket instead of folding those drives into `offline`, keeping
`online + offline + unknown == totalDrivesPerSet`.
- ecstore notification_sys: a probe miss below the failure threshold with no
cache is now reported as `unknown` (not `initializing`) and carries the host's
drives from the pool topology (tagged `unknown`) so the summary stays balanced;
drive synthesis is factored into `synthesized_disks(host, endpoints, state)`,
shared by the offline and unknown paths.
Tracked in rustfs/backlog#1049 (P1-A + P0-A).
Co-authored-by: heihutu <heihutu@gmail.com>
* feat(utils): make DNS resolver cache TTL configurable via RUSTFS_DNS_CACHE_TTL_SECS
The resolver cache in crates/utils/src/net.rs had a hardcoded 300s TTL with
no way to tune or disable it. On orchestrated networks (Docker Swarm,
Kubernetes) a peer's DNS name is its only durable identity while its container
IP changes on reschedule, and the platform DNS always serves the current
answer. A fixed 300s application-side cache can keep a member dialing a dead IP
for up to five minutes after a peer restarts. Rust has no runtime DNS cache, so
on musl images this was the only cache in the stack and the only one that could
not be turned off.
Read the TTL from RUSTFS_DNS_CACHE_TTL_SECS (u64, default 300), matching the
existing RUSTFS_INTERNODE_* transport knobs. `0` disables caching entirely so
every get_host_ip consults the system resolver. The resolved value is logged
once at first use, removing an invisible cache from the triage surface. Change
is backward-compatible and confined to net.rs.
Also drop the two per-lookup info! logs on the resolve path: they would flood
the log when caching is disabled and added hot-path noise otherwise.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(helm): derive mTLS secret names from chart fullname (#4601)
* fix(helm): mtls certificate for server and client hardcode
* fix hardcode issue in deployment
* update typo yaml file
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: majinghe <42570491+majinghe@users.noreply.github.com>
fix(ecstore): emit CommonPrefix for object with same-named prefix in non-recursive listings (backlog#1042)
On single-disk / consistent deployments a non-recursive scan_dir that finds a/xl.meta classifies `a` as an object and does not descend, so the source never emits the prefix `a/`, dropping the CommonPrefix from delimiter listings even when `a/b` exists. This is the single-disk follow-up to backlog#880 (the multi-disk merge path was fixed in #4563).
Fix: in the scan_dir Ok branch, for a non-recursive listing of a plain object, probe whether `a/` holds a listing entry other than the object itself (new object_prefix_has_sibling_listing_entry, reusing directory_has_visible_listing_entry and skipping the object's own xl.meta and data dirs); if so, additionally schedule the prefix dir `a/`. The upstream fold in from_meta_cache_entries_sorted_infos already emits object `a` as Contents and dir `a/` as a CommonPrefix, so nothing changes there. Leaf objects (the common case) pay a single list_dir and return false immediately.
Verification: new scan_dir unit test (positive: `a` + `a/b` coexist; negative: leaf `c` produces no spurious `c/`); new end-to-end e2e (object `a` + `a/b` with delimiter "/" -> Contents `a` + CommonPrefix `a/`); existing list_objects_duplicates e2e (#1797 / #2439 regressions) stays green; set_disk::read 114 passed.
Note: lib-test compilation depends on #4573 (now merged), which added the allow_inplace_legacy_fallback argument to the codec streaming arity tests after #4560 / backlog#879.
Co-authored-by: heihutu <heihutu@gmail.com>
The three store::bucket::tests::bucket_delete_* tests drive make_bucket /
delete_bucket through the process-global local-disk registry
(GLOBAL_LOCAL_DISK_MAP) and lock client, and through Sets::new which reads the
process-global erasure mode. Under `cargo test` (single process, many threads)
they race other global-touching tests and fail deterministically with
InsufficientWriteQuorum.
serial_test's #[serial] with the crate default key serializes them across the
in-process suite. Measured at --test-threads=16: baseline fails all three every
round; with #[serial] all three pass every round. nextest's per-test processes
are covered separately by the ecstore-serial-flaky test-group in
.config/nextest.toml (#4558). Full instance-level isolation remains a backlog
#939 (InstanceContext) concern and is not attempted here.
Co-authored-by: heihutu <heihutu@gmail.com>
fix(ecstore): only evict remote lock channel on transport failures (#4567)
The remote lock RPC path evicted the cached gRPC channel on *any*
`tonic::Status`. Genuine transport failures (connect refused/reset,
keepalive/deadline) surface that way, but so do server-produced
application statuses on a perfectly healthy channel: `Unauthenticated`
/`PermissionDenied` from the signature interceptor, `Internal`
/`FailedPrecondition` when the peer lock service is not ready yet,
`InvalidArgument`, `ResourceExhausted`, `Unimplemented`, and so on.
Evicting the channel for those cannot help — the channel is fine — and
each spurious eviction re-dials the connection and advances the peer
toward the offline threshold via `record_peer_unreachable`, producing
background channel churn on an otherwise healthy cluster.
Classify the tonic status with `is_transport_failure` (underlying
transport source present, or one of `Unavailable`/`DeadlineExceeded`
/`Unknown`/`Cancelled`) and only evict the cached channel when it is a
real transport failure. The client-side timeout arm still evicts. This
mirrors the transport-vs-application distinction already used on the
remote-disk RPC path.
* refactor(ecstore): migrate the background-services cancel token into InstanceContext (Phase 5 Slice 13)
Phase 5 Slice 13 (backlog#939): move the background-services cancellation token
out of the process static into the per-instance InstanceContext, so cancelling
one instance's background workers (scanner/heal/tier/lifecycle) no longer
touches another instance.
- InstanceContext gains `background_cancel_token: OnceLock<CancellationToken>`
with `init_background_cancel_token` (set-once) and `background_cancel_token()`
returning an owned clone.
- global.rs `init_/get_/create_/shutdown_` helpers keep their signatures and
route through the current instance's context; the static is removed. The
getter now returns an owned `Option<CancellationToken>` instead of a
`Option<&'static _>`, which is what lets the token live in the context.
- Callers adapt to the owned token: the metadata-refresh loop drops `.cloned()`;
the lifecycle worker/loops take the owned token (a shared fallback is cloned
when the token is somehow uninitialized). No Arc cycle is introduced —
workers hold a token clone, not the instance context.
Single-instance behavior is unchanged: startup creates one token in the
bootstrap context the ECStore adopts, and shutdown cancels that same token.
Tests: the token is set-once and cancelling one instance's token leaves a
distinct instance uninitialized.
Verification: cargo test -p rustfs-ecstore (23 instance-context tests green),
cargo clippy -p rustfs-ecstore --all-targets (clean), make pre-commit (pass).
Refs: backlog#939 (Phase 5, Slice 13)
* test(ecstore): prove multi-instance isolation; document embedded guard retention (Phase 5 Slice 14) (#4588)
Phase 5 (backlog#939) capstone. The prior 13 slices moved every piece of
per-instance runtime state out of process globals into ECStore's
InstanceContext. This slice proves the result and records the remaining work.
- Add `two_instances_isolate_all_migrated_state`: an end-to-end acceptance test
that constructs two independent InstanceContexts and verifies NONE of the
migrated state is shared — erasure setup, lock manager, region, deployment id,
the four service handles (tier/notifier/expiry/transition), the local disk
registry, the bucket monitor, and the background cancel token. This is the
object-graph isolation carrier working end to end.
- Document why the embedded single-instance guard (EMBEDDED_SERVER_STARTED) is
intentionally retained: storage startup still publishes into the process-level
bootstrap context (write-once region/endpoints/deployment id) and the single
GLOBAL_OBJECT_API handle, so a second startup would fail-fast on that shared
state. Lifting the guard requires threading a per-instance context through
startup — a follow-up beyond migrating the globals. The guard is NOT removed:
rejecting the second start is safer than the panic it would otherwise become.
Verification: cargo test -p rustfs-ecstore (acceptance test + all instance-context
tests green), cargo clippy -p rustfs-ecstore --all-targets (clean), make
pre-commit (pass).
Refs: backlog#939 (Phase 5, Slice 14). Stacked on Slice 13 (#4586).
The internode HTTP/2 window/keepalive tuning (apply_http_client_tuning +
http2_keep_alive_* in build_http_client) only takes effect when the connection
negotiates HTTP/2, which happens via TLS-ALPN. Over plaintext internode
transport the connection is HTTP/1.1 and all the tuning is silently inert,
with no signal to the operator.
Make this honest without changing protocol behavior (no h2c, no
prior_knowledge, no opt-in flag):
- InternodeHttpClientTuning gains h2_tuning_explicit, true when the operator
set a window size, a non-default tuning profile, or the keepalive env.
- Add pure predicate should_warn_h2_inert(negotiated_is_http2,
h2_tuning_explicit, already_warned) and maybe_warn_h2_inert, gated by a
process-lifetime AtomicBool so the warn fires at most once.
- Call maybe_warn_h2_inert at both record_internode_http_version sites using
the actual negotiated resp.version().
- Document the TLS-ALPN requirement at apply_http_client_tuning and near the
ENV_INTERNODE_HTTP2_* constants.
Tests cover the should_warn_h2_inert truth table and that h2_tuning_explicit
reflects explicit windows and a non-default profile.
fix(ecstore): age LastMinuteLatency samples individually (backlog#806-16)
The rolling one-minute latency window stored bare Duration samples plus a
single, never-updated start_time. Its retain predicate ignored the element
and evaluated the constant now.duration_since(start_time) < 60s, so once the
window had existed for 60s every add() dropped ALL samples and the average
degenerated to the latest single sample.
Each sample now carries its own Instant and is retained by its individual
age. The type backs only in-memory EpHealth + admin display (the wire shape
is target::LatencyStat with curr/avg/max), so Serialize/Deserialize is kept
only to satisfy the enclosing LatencyStat derive; the sample Instant is
serde(skip) and restarts aging on reload.
fix(io-metrics): drop client-controlled bucket label from rustfs_s3_operations_total (backlog#806-22)
The bucket dimension on the rustfs_s3_operations_total counter is
client-controlled and unbounded, letting any client explode metric
cardinality (a Prometheus/OTEL cardinality DoS that grows the in-process
OTEL aggregation store even without a scrape). Drop the bucket label so the
series is keyed by the bounded op dimension only (<=122 variants), matching
MinIO which never labels its default operation counters with bucket.
BREAKING: rustfs_s3_operations_total no longer carries a "bucket" label.
record_s3_op(op, bucket) -> record_s3_op(op); all call sites in
rustfs/src/storage/{ecfs.rs,helper.rs} updated (bucket bindings retained
where still used by audit/notify paths). Add metrics-util debugging recorder
dev-dep and tests asserting the series carries only the op label and that
series count equals distinct-op count.
fix(rio): typed-first internode HTTP error classification (backlog#805)
classify_reqwest_error matched reqwest/hyper error text by English
substrings, brittle to library/OS-locale wording. Refactor to a pure,
testable classify_transport_error(err, is_timeout, is_connect, is_body)
that trusts structured signals first: caller timeout, then the io::Error
kind found anywhere in the source chain (typed wins over any string/body
signal so a real ConnectionRefused is never mislabeled DnsResolutionFailed),
then a DNS-only string heuristic gated behind is_connect, then body, then
Unknown.
Flip DnsResolutionFailed to retryable, mirroring MinIO IsNetworkOrHostDown
(*net.DNSError => network-or-host-down => retryable); RustFS already retries
transient DNS at endpoints.rs, and bounded retries cost at most one extra
attempt on permanent NXDOMAIN.
Swap the ecstore remote_disk non-retryable exemplar from DnsResolutionFailed
to Unknown accordingly.
fix(object-capacity): harden clock handling in write window, background intervals and scope registry
Three low-severity robustness gaps (S32+S33+#35):
- WriteRecord keyed its 60-bucket write window by wall-clock unix
seconds while the debounce used Instant. An NTP step backwards marked
recent buckets as future and silently suppressed write-triggered
refreshes until the wall clock caught up; a forward step aged the
whole window at once. Bucket keys now derive from a monotonic
per-record epoch, immune to clock steps.
- Background refresh/metrics intervals were only clamped at zero;
RUSTFS_CAPACITY_SCHEDULED_INTERVAL=u64::MAX overflowed
Instant + Duration and panicked the spawned task, silently killing
scheduled refreshes. Intervals now clamp into [1s, 30 days] via one
helper with a structured warn.
- record_capacity_scope merged new disks into an expired entry and
refreshed its recorded_at, resurrecting a scope take_capacity_scope
would have discarded (and the merge path never pruned). Expired
entries are now replaced, not merged into.
Ref: rustfs/backlog#1022 (S32+S33+#35 from audit rustfs/backlog#1010)
fix(metrics): normalize peer-health key so servers_offline_total counts each peer once
rustfs_cluster_servers_offline_total is the count of distinct keys in the
addr-keyed CLUSTER_PEER_HEALTH map. The data path keys a peer by
endpoint.grid_host() (scheme://host:port) while the lock path keys it by
url::Url::to_string() (scheme://host:port/, trailing slash), so one physical
peer becomes two health entries and a single downed node reports 2 (2N for N).
Normalize the address (trim trailing slashes) at the map boundary in
record_peer_reachable/record_peer_unreachable/cluster_peer_is_offline/
cluster_peer_should_bypass so both forms collapse to one canonical key.
Pure observability fix; peer selection and quorum are unchanged.
Fixesrustfs/backlog#1043
Co-authored-by: heihutu <heihutu@gmail.com>
The tracker never influenced traversal: walkdir's follow behavior is
fixed up front by follow_links(), and should_follow() only gated the
tracker's own bookkeeping. Its 'depth limit' compared tree depth (not
symlink chain depth), so RUSTFS_CAPACITY_MAX_SYMLINK_DEPTH was a
complete no-op while its telemetry claimed symlinks were skipped that
walkdir had in fact followed and counted; record_symlink was always
called with size 0, so tracked_bytes never left zero (S12).
Remove the tracker, its skipped/summary events, the symlink metric and
the depth env knob end to end (the env's 'as u8' truncation goes with
it), and document the real semantics at the walker: follow_links(true)
counts targets with walkdir's ancestor-loop detection breaking cycles,
follow_links(false) — the default — counts no symlink targets. The scan
root itself is pre-resolved since backlog#1015.
Ref: rustfs/backlog#1018 (S12 from audit rustfs/backlog#1010)
houseme