fix(ecstore): keep O_DIRECT for eligible reads when io_uring is enabled (rustfs/backlog#1102)
UringBackend::pread_uring opens the file buffered, so with io_uring enabled an
O_DIRECT-eligible read (RUSTFS_OBJECT_DIRECT_IO_READ_ENABLE + length >= the
threshold) silently lost O_DIRECT and polluted the page cache — a behavior
change the moment io_uring was turned on for a disk that uses O_DIRECT. The
io_uring backend is gray-off by default, so no deployment is affected, but the
gap had to close before it can be enabled anywhere O_DIRECT is in use.
Route O_DIRECT-eligible reads back through StdBackend's aligned path (which
itself falls back to buffered when the filesystem rejects O_DIRECT). A native
aligned O_DIRECT read in the driver remains part of rustfs/backlog#1102.
Adds uring_preserves_o_direct_for_eligible_reads: with BOTH flags on and the
threshold at 1, unaligned ranges still return exactly the requested bytes.
Co-authored-by: heihutu <heihutu@gmail.com>
backlog#1052 S7 — the final piece: full bucket-namespace isolation
between embedded servers in one process.
Server B's requests resolved B's own ECStore (per-server dispatch landed
earlier), but the store's bucket operations still went through ambient
process facades, so both servers effectively operated on the FIRST
server's disks and metadata:
- LocalPeerS3Client::local_disks_for_pools() called all_local_disk()
(the ambient disk registry = the first published store's context), so
list/make/delete/heal bucket scanned and wrote the wrong volumes.
- BucketMetadata::save() persisted through the ambient object handle, so
a second server's bucket metadata landed in the first server's
.rustfs.sys; set/remove/get/created_at all used the ambient metadata
system.
Now the whole chain is bound to the owning store's InstanceContext:
- S3PeerSys/LocalPeerS3Client gain *_with_instance_ctx constructors and
operate on that context's registered disks; ECStore::new (and the test
store builder) pass the store's context. The legacy constructors keep
the bootstrap default.
- BucketMetadata::save_with_store persists through an explicit store;
BucketMetadataSys::persist_and_set uses the system's own api handle.
- metadata_sys gains instance-scoped variants (get_in / created_at_in /
set_bucket_metadata_in / remove_bucket_metadata_in) that resolve the
context's metadata system and fall back to the ambient default before
the instance cell is initialized (early startup, unchanged behavior).
- The store's bucket handlers (make/get_info/list/delete + the
table-bucket delete guard and emptiness check) use the per-context
variants and this instance's disks.
Acceptance (e2e): two embedded servers with different credentials are now
isolated end to end — each authenticates only its own key, neither sees
the other's buckets or objects, and both data planes stay intact. The
embedded module doc drops the shared-IAM caveat.
579 ecstore bucket/metadata/peer regressions plus the embedded basic and
deferred-IAM e2e stay green.
test(ecstore): cover the io_uring per-disk probe cache hit (rustfs/backlog#1101)
Follow-up to #4635. The per-disk negative probe cache was implemented but had
no dedicated test. Add uring_probe_cache_skips_known_unsupported_disk: a root
recorded in URING_UNSUPPORTED_DISKS makes try_new return None without a fresh
probe. Completes the #1101 acceptance ("cache hit does not re-probe").
Co-authored-by: heihutu <heihutu@gmail.com>
Run `cargo update` followed by `cargo upgrade --exclude ratelimit` on the
latest main to pull the newest compatible versions.
Manifest bumps (Cargo.toml):
- md5 0.8.0 -> 0.8.1
- regex 1.12.4 -> 1.13.0
- sysinfo 0.39.5 -> 0.39.6
Lockfile-only bumps: der 0.8.1, dial9-tokio-telemetry 0.3.14, lru 0.18.1,
regex-automata 0.4.15, symbolic-common/symbolic-demangle 13.9.0, and
zlib-rs 0.6.6. Resolver re-selection moves crypto-common 0.1.7 -> 0.1.6
and generic-array 0.14.7 -> 0.14.9 (both semver-compatible). `ratelimit`
is held at 0.10.1 as requested; `pollster` 1.0.0 is an incompatible major
bump and is intentionally not taken.
Verification: cargo check --workspace --all-targets, cargo clippy
--all-features -D warnings, cargo nextest run --all --exclude e2e_test
(8467 passed), cargo test --all --doc, and cargo deny check
advisories/sources/bans/licenses all pass.
Co-authored-by: heihutu <heihutu@gmail.com>
The startup topology convergence auto-detection (added in #4631) checked
Kubernetes before endpoint style, so a local path deployment (single or
multi-drive, non-distributed) running inside Kubernetes was classified as
orchestrated with an effectively unbounded wait window. Path-style
endpoints have no hostnames to resolve, so the wait never actually
triggers and runtime behavior is unchanged, but the startup log then
advertised mode=orchestrated / wait_timeout=MAX for a purely local
deployment, which is misleading and contradicts the documented policy
(local path endpoints -> fail-fast).
Order the auto-detection by endpoint style first: only distributed URL
endpoints, which resolve hostnames, pick orchestrated (Kubernetes) or
bounded (otherwise); local path endpoints stay fail-fast regardless of
the platform. Update the doc comment to match and add a policy case
locking Kubernetes + local path to fail-fast.
Co-authored-by: heihutu <heihutu@gmail.com>
feat(ecstore): io_uring runtime degradation latch + per-disk probe cache (rustfs/backlog#1101)
Follow-up to #1104. Add the runtime half of the io_uring degradation contract:
- Runtime latch: UringBackend gains an `active` flag (mirrors
DirectIoReadState::supported). A read that returns a restriction-class errno
(EPERM/EACCES/ENOSYS/EOPNOTSUPP) — io_uring became unusable on this disk —
trips the latch, and every further read goes straight to StdBackend with no
more per-read io_uring attempts. Data errors (EIO), missing files, and
parameter errors do NOT latch, since StdBackend would hit them too.
- Per-disk probe cache: a process-wide negative cache of disks whose probe
failed, so try_new skips re-probing (creating a ring + driver thread) on
reconnect.
Tests: the errno classifier (restriction-only) and a latched-off read that
still returns correct bytes via StdBackend; the #1104 differential test still
passes under real io_uring.
Co-authored-by: heihutu <heihutu@gmail.com>
The OnceLock introduced in 5fbd49a80 for bucket_metadata_sys panics when
multiple test modules initialize their own ECStore against the shared
bootstrap context. Switch to Mutex<Option> so test fixtures can
reinitialize safely.
Extract shared gating test environment to gating_test_env.rs to avoid
duplicate ECStore setup across delete_objects_stat_gating_test and
put_prelookup_gating_test.
Fixes 8 test failures:
- capacity_dirty_scope_test (2 tests)
- delete_objects_stat_gating_test (3 tests)
- put_prelookup_gating_test (3 tests)
* feat(embedded): allow multiple embedded servers to coexist in one process
backlog#1052 S5: turn the embedded startup guard into a sequential lock
and make every write-once shared cell tolerate a second embedded start.
Two RustFSServers on different ports and volumes now start, run, and
shut down independently in the same process.
- EMBEDDED_SERVER_STARTED is released once each startup hands off; a
second startup that runs after the first is no longer rejected.
- The second embedded server constructs its own InstanceContext instead
of adopting the process bootstrap one, so region/endpoints/deployment
id land on that context (the first server keeps adopting bootstrap to
keep single-instance ambient facades unchanged).
- Startup-time tolerant paths:
- action_credentials publish treats AlreadyInitialized as success
(per-server ActionCredentialHandle already holds the real creds).
- GLOBAL_RUSTFS_PORT warns instead of panicking on a second set.
- Observability install returns Ok when the process subscriber is
already set — the second server reuses it.
- New acceptance test proves two servers start, respond on their own
ports, and one can be shut down without disturbing the other; the
survivor keeps serving S3 requests. IAM and root-credential lookup
still share a process domain (a second server whose creds differ from
the first will fail signature validation), tracked as a follow-up.
- Embedded doc rewritten: 'Limitations' → 'Multi-instance status'; the
AlreadyStarted error is now scoped to concurrent startups only.
The remaining work in #1052 is the auth path per-server dispatch and the
matching data-plane routing so two servers with different credentials
serve independent buckets end-to-end.
* feat(app): per-server auth and application context for multiple embedded servers (#4633)
backlog#1052 S6: each embedded server now authenticates against — and
its request path resolves — its OWN application context, so two servers
with different root credentials each accept their own access key and
reject the other's.
- AppContext is per-server: ensure_startup_after_iam constructs a fresh
context around this server's store + IAM + KMS and installs it into the
server's own ServerContextSlot, then publishes it as the process
default first-writer-wins (publish_global_app_context) for legacy
ambient readers. The old 'reuse the global if present' path is gone.
- FS::check (the S3 data-plane access gate) resolves auth against
self.server_ctx's context: check_key_valid gains a _with_context
variant that takes the root credentials and IAM system from an explicit
context (None = ambient, unchanged for all 140+ existing callers). The
region and the server context slot are published into the request
extensions for downstream handlers.
- Each embedded server seeds its own root credentials into its context
(ActionCredentialHandle.publish) at startup, so credential validation
no longer falls back to the first server's process-global identity.
- The bucket/object/multipart use-cases resolve their store from the
server's context (bucket_usecase_for/object_usecase_for/... take &FS).
New acceptance test: two servers with distinct credentials each
authenticate with their own key and reject the other's.
KNOWN FOLLOW-UP: full bucket-namespace isolation still requires threading
the instance context through the lower ecstore data plane (peer_sys /
disk registry / bucket-metadata reads still resolve via the process
GLOBAL_OBJECT_API), so the two servers do not yet present independent
bucket listings even though each holds its own store. That deeper pass —
a continuation of the #939 object-graph ctx threading — is the remaining
work on #1052.
Stacked on the S5 guard change.
Multi-node startups (Kubernetes StatefulSets, bare-metal, VMs) can hit
transient DNS/local-host resolution failures while peers and headless
service records are still coming up. Previously endpoint resolution
retried DNS for a fixed 90s window and then returned an error, which
propagated out of run() and exited the process; in Kubernetes this drove
repeated kubelet restarts even though the cluster eventually converged.
Introduce a startup topology convergence policy with three modes:
- orchestrated: wait effectively indefinitely for recoverable
DNS/topology errors so the process stays Running (readiness stays
false) instead of exiting; defer the DNS-IP cross-port validation
because headless-service records can still be flapping.
- bounded: wait a finite window (default 3m) then fail with an
action-oriented error listing host/stage/elapsed and remediation hints.
- fail-fast: fail on the first non-transient resolution error.
Mode is auto-detected (Kubernetes -> orchestrated, distributed URL
endpoints -> bounded, local path endpoints -> fail-fast) and can be
overridden via RUSTFS_STARTUP_TOPOLOGY_WAIT_MODE,
RUSTFS_STARTUP_TOPOLOGY_WAIT_TIMEOUT and
RUSTFS_STARTUP_TOPOLOGY_RETRY_MAX_DELAY.
Also normalize divergent local/remote verdicts for endpoints that share
a host:port, throttle retry warnings, and keep the DNS-independent local
same-path checks in every mode. Configuration error handling is
unchanged: malformed volumes, mixed styles/schemes, duplicate or root
paths, and non-transient errors still fail fast.
Read the topology env vars through rustfs_utils::get_env_opt_str for
consistent alias handling, and simplify the log_once poisoned-lock branch
to unwrap_or_else(PoisonError::into_inner).
Co-authored-by: heihutu <heihutu@gmail.com>
feat(ecstore): add runtime-probed io_uring read backend, gray-off by default (rustfs/backlog#1104)
Wire the standalone rustfs-uring crate (https://github.com/rustfs/uring) into
LocalIoBackend as an opt-in read backend. When RUSTFS_IO_URING_READ_ENABLE is
set AND the per-disk io_uring probe succeeds, positioned reads go through the
cancel-safe UringDriver; otherwise — default, or on a restricted host, or on
any per-read driver error — reads fall back to StdBackend byte-for-byte, so the
default build is unchanged.
- Cargo.toml: rustfs-uring as a Linux-only git dependency (pinned rev). The
guard scripts/check_no_tokio_io_uring.sh allows an explicit io-uring
integration; only the tokio "io-uring" runtime feature is banned.
- UringBackend mirrors StdBackend::pread_bytes's resolution/access/bounds
preamble and only swaps the raw byte read; the other three trait methods
delegate to the inner StdBackend.
- Backend selection at LocalDisk construction via build_local_io_backend.
- Differential test uring_backend_reads_match_std: with the flag set, every
positioned read returns byte-identical data (driver-served when io_uring is
available, StdBackend fallback otherwise).
Verified: cargo check -p rustfs-ecstore on Linux; the differential test passes
under real io_uring (seccomp=unconfined). The runtime errno degradation latch,
per-disk probe cache, and productionization are tracked in
rustfs/backlog#1101/#1102.
Co-authored-by: heihutu <heihutu@gmail.com>
The io_uring cancel-safety spike (imported in #4625) has been migrated to a
dedicated repository, https://github.com/rustfs/uring (crate `rustfs-uring`),
with the full per-issue commit history preserved.
The standalone repo has independent CI that this workspace cannot run: it
excludes the crate from the build graph, so its checks passed without ever
building or testing it. rustfs/uring CI is green on fmt + clippy, a native leg
that runs the suite with real io_uring and fails on any skip, and a docker leg
exercising both the graceful-degradation and real-io_uring paths.
Removing the in-tree copy keeps rustfs/rustfs clean; the crate will be wired in
later as a git dependency behind runtime probing (rustfs/backlog#1051).
Co-authored-by: heihutu <heihutu@gmail.com>
backlog#1052 S3, fifth slice (also part of what the plan called S4).
ActionCredentialHandle was a unit struct forwarding get to the process
singleton (rustfs_credentials' GLOBAL_ACTIVE_CRED), so every context
served the same credentials regardless of which server it belonged to.
The handle now keeps its own copy and gains publish():
- publish() lands on the owning context and tries to publish the process
global; readers prefer the owned copy and fall back to the global while
the initial startup publish still lands there — ambient callers (iam
token signing, request-signature validation) keep working.
- Single instance: both cells are written together, so reads are
unchanged; two contexts that both published stay isolated even though
the global remembers only the first (covered by a new test).
- The publish return signals "took effect": true if this handle newly
holds credentials OR the global was newly initialized, matching the
pre-existing init_global_action_credentials fail-fast contract, now
scoped to the handle.
The interface trait gains a default-body-friendly signature; the test
double implements it as a no-op. The startup publish path (S4) still
calls init_global_action_credentials directly; wiring it through the
handle is a follow-up.
* chore(experiments): import io_uring cancel-safety spike as audit baseline (backlog#894)
Import the Spike 0 io_uring cancel-safety prototype from the closed PR #4381
branch (houseme/p2-spike0-uring-cancel-safety) as the baseline for the
backlog#1051 audit remediation. This crate is a standalone workspace and is
deliberately kept out of the main Cargo.lock/build graph (NOT production code).
Subsequent commits apply the fixes tracked in backlog#1051 sub-issues, one
commit per issue.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(experiments/uring): drain probe SQE to its CQE before releasing buffer (rustfs/backlog#1053)
The probe path had no pending-table backstop: after pushing the read SQE,
any early return (`submit_and_wait` error, missing CQE) dropped the probe
buffer and file while the read could still be in flight in io-wq, and the
caller dropped/unmapped the ring on the error path. If the kernel then wrote
the 512-byte result into that freed heap block, it was a use-after-free — the
exact bug class this spike exists to prevent, living in its own probe path.
Fix: once the SQE is pushed, drain to its CQE via `drain_probe_cqe`, retrying
the WAIT on EINTR without re-pushing (the kernel consumed the SQE atomically
before the wait). A bounded attempt count prevents a probe against a hung
device from blocking forever; on any drain failure the buffer (and file) are
`mem::forget`-ed ("leak over UAF") so the kernel can never write into freed
memory. Unmapping the ring on its own is safe; only the user buffer must
survive.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(experiments/uring): split probe/runtime/transient errno classes, guard offset (rustfs/backlog#1059)
`is_expected_restriction` folded EINVAL into the "environment restricted"
class, but at runtime EINVAL is triple-meaning — offset > i64::MAX (signed
loff_t), O_DIRECT misalignment, and setup entries over the cap. Implementing
the rustfs/backlog#1048 permanent-degradation latch literally against this
class would fault a healthy disk off io_uring on one alignment retry or
offset-arithmetic bug.
Document that the class is probe-time ONLY and that P2 must split errnos into
probe-restriction / runtime-parameter-error / transient (EINTR/EAGAIN). Add a
concrete guard: `submit` rejects offset > i64::MAX with an InvalidInput error
instead of letting it reach the kernel as a runtime EINVAL. The probe EINTR
half of this issue is already handled by the drain loop from rustfs/backlog#1053
(retry the wait, never re-push).
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(experiments/uring): abort on driver-thread panic instead of freeing in-flight buffers (rustfs/backlog#1054)
The ownership model's "CQE is the only reclamation point" invariant held only
while the driver thread never unwound. On a panic inside drive(), Rust drop
order freed the `pending` table (every in-flight buffer) before the ring,
while the kernel could still be writing into those buffers → mass UAF.
`catch_unwind` cannot fix this: the destructors run during the unwind, before
the catch boundary.
Move ring + pending + backlog into a `DriverState` whose `Drop` checks
`thread::panicking()` and calls `process::abort()` BEFORE any field destructor
runs — leaving the ring mapped and the buffers allocated (leak over UAF). The
capacity-overflow panic that made this reachable (caller-controlled `len`) is
closed at the source in the len-guard commit (rustfs/backlog#1057).
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(experiments/uring): reject reads above MAX_RW_COUNT to stop u32 truncation (rustfs/backlog#1057)
The SQE length field is `len as u32`, so len == 4 GiB became a 0-byte read the
kernel answered with res=0 → an Ok(empty) the caller decodes as a false EOF
(and len > 4 GiB read only the low 32 bits). Silent truncation (CWE-197),
forbidden by the repo's rust-code-quality rules.
`submit` now rejects len > MAX_RW_COUNT (2 GiB - 4 KiB) with InvalidInput; the
`len as u32` cast in the driver is consequently lossless. This also closes the
caller-controlled capacity-overflow panic feeding rustfs/backlog#1054. P2 must
chunk reads larger than the cap.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(experiments/uring): resubmit short reads to satisfy the whole-range contract (rustfs/backlog#1058)
CQE res >= 0 was truncated and delivered as final with no resubmit loop, and
Pending did not even store the requested length. io_uring can legally
short-read a regular file (io-wq signal interruption, NOWAIT partial page
cache, O_DIRECT tail blocks), while LocalIoBackend::pread_bytes is a whole-
range contract — a short shard fed to EC bitrot verification surfaces as
intermittent, hard-to-attribute integrity/quorum errors.
Track offset/nread in Pending and drive a resubmit loop: a short non-EOF read
re-queues the remainder into buf[nread..], keeping the entry (and its buffer,
and in_flight) until the FINAL CQE of the logical read; res == 0 is treated as
a real EOF. The resubmitted SQE reuses the op's user_data, so a late
ASYNC_CANCEL from a dropped future still cancels the logical read cleanly.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(experiments/uring): bound the shutdown drain and record cancel outcomes (rustfs/backlog#1055)
Shutdown made "drain to in_flight == 0" a hard precondition for unmapping the
ring, but ASYNC_CANCEL is best-effort: it cannot interrupt a regular-file read
already executing in io-wq, so on a D-state/NFS-hung disk the CQE may never
arrive and drain-to-zero never terminates — the driver loops forever and
shutdown()/Drop join blocks the caller (and any tokio worker) permanently.
This is an internal contradiction (safe unmap needs drain; a bad disk makes
drain unbounded) in the very environment io_uring exists to handle.
Add a bounded-drain escape hatch: after DRAIN_TIMEOUT with ops still in flight,
leak the whole DriverState (ring stays mapped, buffers stay allocated — leak
over UAF) and exit so shutdown() returns. Soften shutdown()'s hard assert to a
warning for that degraded path; clean-drain tests still assert in_flight == 0
themselves. Also record the ASYNC_CANCEL three-state result
(succeeded/not-found/already-executing) so the hung-disk signal is observable
instead of discarded.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(experiments/uring): assert NODROP, monitor CQ overflow, handle EBUSY (rustfs/backlog#1056)
In-flight had no upper bound and could exceed CQ capacity (entries=64 → CQ=128)
with zero overflow handling: no NODROP check, no overflow read, no EBUSY
handling. A lost CQE means its pending entry is never reclaimed, drain never
completes and shutdown hangs — and the spike only avoided this by accidental
reliance on the io-uring crate's auto-flush + NODROP kernel + poll cadence,
all of which P2's eventfd/AsyncFd reaping removes.
Assert the NODROP feature at probe (degrade via ENOSYS otherwise), monitor the
kernel CQ-overflow counter each turn and surface a non-zero value as fatal, and
handle submit() EBUSY as CQ-overflow backpressure (keep the backlog, reap this
turn) instead of swallowing it. The hard in-flight bound (permits ≤ CQ capacity)
lands with the backpressure work in rustfs/backlog#1060.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(experiments/uring): add backpressure with permits released at the CQE (rustfs/backlog#1060)
Submission was unbounded (unbounded mpsc + uncapped pending/backlog), so a
concurrent large-object read storm had no memory ceiling. The subtler trap:
the planned SQ-depth semaphore, implemented the natural RAII way (permit held
by the ReadHandle/future), would release permits at future drop while orphan
buffers stay resident in the pending table awaiting slow-disk CQEs —
decoupling the permit count from resident memory and reopening the DoS surface
exactly in the EC quorum-drop hot path.
Add a `Backpressure` semaphore sized to the SQ depth (entries < CQ capacity, so
CQ overflow is structurally unreachable). `submit` acquires before handing the
op to the driver; the driver releases the permit at the CQE (pending-table
removal), NOT at future drop, tying the in-flight/memory bound to actual kernel
residency. Permits are balanced on the shutting-down reject and send-failure
paths, and the driver wakes all waiters on exit.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(experiments/uring): open the probe file via O_TMPFILE instead of a predictable path (rustfs/backlog#1061)
The probe wrote a predictable temp path (uring-spike-probe-{pid}-{seq}) with
std::fs::write (O_CREAT|O_TRUNC, no O_EXCL/O_NOFOLLOW): a local attacker could
pre-plant a symlink there and have the process — often root — truncate and
overwrite an arbitrary target (CWE-59/377), with a TOCTOU window between write
and open (CWE-367). This probe is the direct blueprint for P2's per-disk
startup probe, so copied verbatim it becomes a production vulnerability.
Open via O_TMPFILE (anonymous inode, no name → nothing to plant a symlink at,
no TOCTOU, no leftover), falling back to O_CREAT|O_EXCL|O_NOFOLLOW + 0600 +
per-process nonce + immediate unlink on filesystems without O_TMPFILE. P2's
per-disk probe should create inside the tested data-disk directory the same
way, which also validates that disk's filesystem + io_uring combination.
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(experiments/uring): correct invariant 2 mechanism, add invariants 6/7/8 (rustfs/backlog#1063)
Invariant 2 (the spike's flagship finding) mis-described the fd-reuse hazard:
it claimed the danger window is submission→CQE and that the kernel would
"write into someone else's file". Both are wrong — a submitted op holds a
struct file reference and is immune to fd close/reuse; the real window is SQE
construction (as_raw_fd) → io_uring_enter (backlog residency), and for a READ
the consequence is reading the WRONG file, not writing. A P2 optimization
reasoning from the false premise (drop Arc<File> after submit / registered-file
table) would step straight into it.
Correct the mechanism and add the invariants this audit hardened: driver-thread
unwind safety (6), backpressure permit released at the CQE (7), reused-buffer
content hygiene (8, detailed in rustfs/backlog#1062), plus the errno three-class
contract, bounded-drain escape hatch, and short-read resubmit responsibility.
Mark the now-remediated items in the leftover list.
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(experiments/uring): pin the reused-buffer content-hygiene invariant for P3 (rustfs/backlog#1062)
The spike leaks nothing today (fresh zeroed buffer per op + truncate to res),
but rustfs/backlog#1048's P3 constraint mandates a driver-owned aligned slab
whose buffers are reused across requests as dirty memory. Both the SPIKE
invariants and the #1048 constraint address only buffer LIFETIME (UAF), not
content hygiene: once buffers are reused, any path that forgets to bound the
caller-visible bytes to cqe.res (O_DIRECT full-block read sliced upstream,
error path returning the whole buffer) discloses a previous tenant's object
data (CWE-226) in an S3 store.
Pin invariant 8: reused-buffer bytes visible to the caller must be strictly
⊆ [0, cqe.res). Documented in SPIKE.md and marked at the delivery point in the
driver so P3 preserves it; needs a dirty-buffer + short-read regression test.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(experiments/uring): pin fd ownership and orphan-integrity directly (rustfs/backlog#1064)
The memory-safety assertions were all counter proxies, and invariant 2 (fd
owned by the pending table) had zero coverage — deleting Pending.file compiled
and left every test green because each test kept its own Arc<File> alive.
Add two direct observations: pending_table_owns_fd_after_caller_drop drops the
caller's Arc while the op is in flight and asserts F_GETFD still succeeds (only
the pending table's clone keeps the fd open; removing that field would close it
→ EBADF). orphan_in_flight_does_not_corrupt_delivered_reads keeps an orphaned
buffer in flight while 64 delivered reads must return byte-exact, asserting the
orphan buffer is not reclaimed early and its kernel writes corrupt nothing. A
driver-level poison/canary leg is noted as a P2 acceptance-matrix item (ASAN
cannot see a kernel write into a freed buffer).
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(experiments/uring): cover CQ-overflow safety and read boundaries (rustfs/backlog#1065)
The suite never approached CQ capacity and never touched EOF/len boundaries.
Add no_cq_overflow_under_load (300 ops through a CQ of 128 with backpressure
capping in-flight at 64, asserting cq_overflow stays 0 and all deliver),
boundary_reads (len==0, read at EOF, a cross-EOF short read delivered to a
live receiver exercising the positioned resubmit path, and the rejected
huge-len/huge-offset guards), and pipe_half_close_reads_eof (a closed write
end surfaces res==0 EOF).
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(experiments/uring): cover Drop-without-shutdown and de-flake cancel_stress (rustfs/backlog#1066)
All tests ended via explicit shutdown(), so the UringDriver Drop impl's
live-thread branch (send Shutdown before join) was never exercised; add
drop_without_shutdown_drains_and_cancels which drops the driver with ops in
flight and asserts the held futures resolve to ECANCELED (a join-first
regression or unbounded hang makes it hang).
Also de-flake cancel_stress: the exact assert delivered == OPS/2 raced the
driver — an even-i read can complete between read_at returning and drop(handle),
delivering to the still-live receiver and flipping the split. Relax to the
deterministic conservation identity plus delivered >= OPS/2.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(experiments/uring): make run-docker.sh assert each leg's real path (rustfs/backlog#1067)
Both legs ran the identical cargo test and checked only the exit code, and a
skip is indistinguishable from a real pass at that level: leg 1 depended on the
host Docker's default seccomp "usually" blocking io_uring, and leg 2 printed
"both legs passed" even if every test skipped (vacuous pass — zero real
io_uring coverage).
Add an explicit seccomp profile (seccomp-block-uring.json) that returns EPERM
for io_uring_setup/enter/register so leg 1 deterministically hits the
graceful-degradation path regardless of host defaults, and assert leg 1
actually degraded (SKIP lines present) while leg 2 did NOT skip a single test
(io_uring really ran). Either violation now fails the harness.
Co-Authored-By: heihutu <heihutu@gmail.com>
* style(experiments/uring): apply repo rustfmt (max_width=130) to the audit changes
Normalize the formatting of the remediation code to the repo rustfmt.toml.
Pure formatting; no behavior change. clippy --all-targets -D warnings is clean.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
backlog#1052 S3, fourth slice. The AppContext's IamHandle already owned an
Arc<IamSys>, but the value it held was read back from the process
singleton after init — so a future second server's context would silently
bind to the first server's IAM domain (credentials, policies, users all
belong to a specific store's .rustfs.sys).
- rustfs_iam gains build_iam_sys(store): construct an IAM system bound to
a store without touching the singleton. init_iam_sys wraps it (build,
publish first-wins, return the handle — previously Result<()>).
- Startup threads the built handle: the inline bootstrap path passes the
Arc it just created into ensure_startup_after_iam, which constructs the
AppContext from the passed value instead of re-resolving the global.
The deferred-recovery path resolves the freshly published global
directly (context-first resolution cannot be used there: the AppContext
does not exist yet — creating it is the finalizer's job; caught by the
embedded deferred-IAM e2e).
- IamHandle::is_ready() reports the held system's readiness instead of
consulting the process singleton; the dead global re-resolver is
removed. token_signing_key stays on the credentials global until S4.
157 iam tests + embedded e2e (basic + deferred recovery) green.
backlog#1052 S3, third slice — the hard blocker for a second embedded
server's service stage. GLOBAL_BUCKET_METADATA_SYS was a process-global
OnceLock whose init ran .set().unwrap(): a second instance's service
startup panicked the whole process. The system it guards is inherently
per-store (it holds the store handle and that store's bucket→metadata
cache), so it now lives on the store's own InstanceContext:
- init_bucket_metadata_sys writes the cell of the store it was handed
(api.ctx) — no signature change — and keeps the double-init fail-fast,
now scoped to the instance (assert on the per-context set). The
dist-erasure check for the refresh loop reads the same context.
- get_global_bucket_metadata_sys / the crate-private getter behind the
~20 get_*_config helpers resolve through the current_ctx() facade —
the published store's context, or bootstrap before that — so every
existing reader keeps its exact single-instance behavior.
- New acceptance test: two real stores in one process each initialize
their own metadata system (the old global cell panicked right there),
plus a shared builder extracted from the store-graph test.
201 bucket/metadata regression tests green.
* refactor(app): give each context's server-config handle its own copy
backlog#1052 S3, first slice: establish the per-context state pattern on
the smallest subsystem. ServerConfigHandle was a unit struct forwarding
both get and set to the process-global GLOBAL_SERVER_CONFIG, so every
AppContext shared one config regardless of which context a caller held.
The handle now keeps its own copy: a set lands on the owning context and
on the process global (ambient readers — the config loader path, the
scanner — keep working), and a get prefers the owned copy, falling back
to the global while the initial load still publishes there. Single
instance: the owned copy and the global are written together, so reads
are unchanged. Two contexts that both published stay isolated even though
the shared global only remembers the last write (covered by a new test).
The global write in set() is the transitional bridge for ambient readers;
it goes away when those readers migrate and multi-instance flips on
(backlog#1052 S5).
* refactor(notify): hand out the notification system as an owned Arc
backlog#1052 S3, second slice. GLOBAL_NOTIFICATION_SYS stored the
NotificationSys by value and every accessor returned
Option<&'static NotificationSys> — a process-lifetime borrow that a
per-server AppContext can never hold. The global now stores an Arc and
the accessor chain (ecstore runtime sources, ECStore::notification_system,
the iam forwarders, the rustfs shims, NotificationSystemInterface and the
AppContext resolvers) hands out owned Arcs instead.
Call sites are unchanged apart from two borrow adjustments in the
rebalance admin handler (pass &Arc where a reference is expected; borrow
with as_ref() so the handle survives to the second use). Behavior is
identical — same single global instance, first init still wins — but the
type now permits a future per-server context to own its notification
system (backlog#1052 S3 follow-up).
backlog#1052 S2, second slice: admin request dispatch.
Admin operations are registered as &'static dyn Operation, so unlike FS
they cannot carry per-server state. Instead, the (per-server) S3Router now
holds the server's ServerContextSlot and injects it into every request's
extensions at both dispatch points (check_access and call) — the same
extensions channel already used for ReqInfo/RemoteAddr.
- make_admin_route binds the slot; the HTTP builder passes the same slot
the S3 service (FS) uses, so both paths dispatch to the same server.
- admin/runtime_sources gains object_store_from_req / an extensions-based
variant for handlers that have already moved request fields out; both
fall back to the ambient process context when no slot was injected
(direct handler tests, non-router paths) — single-instance unchanged.
- 27 handler resolution sites across 12 files now resolve per-request;
helper functions without request access (11 sites: site_replication
state helpers, quota/table_catalog internals, object_zip_download
workers) stay on the ambient path until app subsystems become
per-server (backlog#1052 S3).
- One site (site-replication resync) resolves before the request body is
consumed, keeping the original error precedence.
New test: the router injects its slot into request extensions by pointer
identity. Embedded e2e (basic S3 + deferred-IAM recovery) still green.
Two follow-up fixes to #4607 (both verified real bugs, each with a regression
test):
1. The in-call server_info retry did not re-dial on network errors. A
network-like first failure runs through `finalize_result`, which sets the
client's `offline` gate; the retry then called `evict_connection` and
re-invoked `server_info`, but `get_client` short-circuits on that gate and
returns "temporarily offline" without dialing. Only the async recovery
monitor would clear it. So the retry re-dialed only in the timeout branch and
was a no-op in the transport/half-open branch it was meant to cover. Add
`PeerRestClient::prepare_retry` (evict + clear the offline gate) and use it
before the retry.
2. Synthesized/degraded drive lists went empty on hostname deployments.
`PeerRestClient::host` is an `XHost` that `hosts_sorted` builds via
`XHost::try_from` -> `to_socket_addrs`, so it is the resolved `IP:port`; but
`synthesized_disks`/`peer_disk_health` compared it against
`Endpoint::host_port()`, which is the raw `hostname:port`. On hostname
clusters the compare missed, the drive list came back empty, and
`unknownDisks` stayed 0 — reproducing the "drives vanish from the summary"
regression #4607 fixed (also affected the pre-existing offline path). Compare
through a shared `endpoint_host_matches` helper that canonicalizes the
endpoint side through the same `XHost` resolution.
Tests: `peer_rest_client_prepare_retry_clears_offline_gate`,
`endpoint_host_matches_direct_and_canonicalized` (uses localhost, no external
DNS).
Follow-up to rustfs/rustfs#4607; tracked in rustfs/backlog#1049.
Co-authored-by: heihutu <heihutu@gmail.com>
fix(ecstore): read the persisted SSE-C nonce in the GET decrypt resolver
#4576 moved SSE-C encryption to a random per-encryption nonce persisted
in object metadata and taught the rustfs-layer decrypt resolver to read
it back — but the byte-level GET decryption resolves its material in
crates/ecstore object_api/readers.rs, a duplicated twin that still
recomputed the deterministic legacy nonce. Encrypt with the random
nonce, decrypt with the deterministic one: the first AEAD block fails
and every SSE-C GET aborts its body after the response headers
(IncompleteRead(0 bytes) on clients; all 8 SSE-C cases in the
implemented s3-tests whitelist), which is what has been driving the
"S3 Implemented Tests" CI job into its 60-minute timeout since
2026-07-09.
Resolve the base nonce like the encrypt side: prefer the persisted IV
(x-rustfs-encryption-iv, then the case-insensitive MinIO interop key),
fall back to the deterministic nonce only for legacy objects with no
stored IV or an undecodable value. Full implemented s3-tests whitelist
is back to 451 passed / 0 failed against the fixed binary.
* feat(helm): support multiple server pools for capacity expansion
The server already understands multiple pools (RUSTFS_VOLUMES is split on
spaces, one pool expression each; rc admin pool/expand/rebalance/
decommission exist), but the chart could only render a single StatefulSet.
Add an opt-in pools mode:
- pools.enabled=false (default) renders byte-identical output to the
current chart (verified against five values variants)
- pools.list renders one StatefulSet per entry; entries may set
replicaCount (4 or 16) and a storageclass block, everything else is
inherited from the top-level values
- pool 0 keeps the legacy StatefulSet/pod/PVC names and its (immutable)
selector, so existing single-pool deployments can be expanded in place
without renaming or data loss; additional pools render as
<fullname>-poolN with a rustfs.com/pool label in their selector
- RUSTFS_VOLUMES and RUSTFS_SERVER_DOMAINS enumerate every pool
- pod anti-affinity is scoped per pool so pools may share nodes while
each pool still spreads its own pods across distinct nodes
- template validation fails fast on unsupported replicaCount, an empty
pools.list, or pools in standalone mode
- pools are append-only by design (list index = identity), documented in
values.yaml and the README
* fix(helm): truncate pool names DNS-safely, document PDB pool scope
Truncate <fullname>-poolN to 63 chars by shortening the base name rather
than the suffix, so the pool index always survives and two pools of a
max-length release cannot collide. Document that the single
PodDisruptionBudget deliberately spans all pools.
* fix(helm): pool-mode anti-affinity must be preferred, not required
Found by live-testing in-place expansion on a 5-node cluster (4-replica
pool 0 + 4-replica pool 1): with requiredDuringScheduling anti-affinity
the expansion deadlocks in a cycle that cannot self-heal -
1. the not-yet-rolled pool-0 pods still carry the unscoped required
rule and repel every rustfs pod from their nodes, so part of pool 1
stays Pending (no IP, no headless-DNS record);
2. every pod that already has the new RUSTFS_VOLUMES exits fatally on
'failed to lookup address information' for those Pending peers;
3. the StatefulSet rolling update is gated on the crashing pod going
Ready, so the remaining pool-0 pods never roll and keep repelling.
Because StatefulSets assign revisions per ordinal, even a subsequent
template fix cannot rescue a wedged rollout (Pending ordinals are only
recreated with the new template after the higher ordinals go Ready) -
the new pool's StatefulSet has to be recreated. Shipping the soft
affinity from the start avoids the state entirely; expansion was
re-tested end-to-end with it and converges.
Pool-mode rendering only - pools.enabled=false still renders the
existing required anti-affinity byte-identically.
---------
Co-authored-by: cxymds <Cxymds@qq.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: cxymds <cxymds@gmail.com>
test(e2e): align object-lock overwrite tests with AWS new-version semantics
PR #3179 made put-like writes (PUT, CopyObject, multipart create/complete)
on versioned object-lock buckets create a new version instead of being
blocked by the current version's legal hold or COMPLIANCE retention, which
is the correct AWS behavior. Six e2e assertions in object_lock_test.rs
still encoded the old blocking semantics and, because e2e_test is excluded
from the CI test run, stayed red unnoticed since 2026-06-03.
Verified against a real server built from current main: all six failed at
their "should be blocked" assertions. Rewritten to assert the AWS
contract: the write succeeds with a distinct new version id, the new
content is current, the protected version keeps its content and lock
metadata, and version-targeted deletion of the protected version stays
blocked (for COMPLIANCE even with bypass_governance). Full object_lock
module is green again (33/33).
PR #4468 removed the last use of wildmatch (crates/notify swapped it for a
hand-rolled star-only glob matcher), but the workspace dependency declaration
in the root Cargo.toml was left behind as an orphan. cargo shear flags it as
not used by any workspace member. Drop the stray line.
Also refresh the comments in crates/notify/src/rules/pattern.rs that still
referred to the removed wildmatch crate, so they describe the current
hand-rolled matcher instead of a dependency that no longer exists.
Co-authored-by: heihutu <heihutu@gmail.com>
fix(admin): report unreachable info members as `unknown` with balanced drive counts
`GET /rustfs/admin/v3/info` synthesizes an entry for every peer, but a peer
whose properties RPC failed below the offline threshold with no cached snapshot
was reported as `initializing` with an empty drive list. That drive list being
empty meant its drives disappeared from the backend summary entirely, so
`onlineDisks + offlineDisks` no longer equaled `totalDrivesPerSet` (an operator
sees `onlineDisks: 3, offlineDisks: 0` for a 4-drive set), and `initializing` is
a misleading state for a member that has been running for hours — it reads like
a node was ejected when nothing is wrong (upstream rustfs/rustfs#4566).
Changes:
- madmin: add `ItemState::Unknown` ("unknown") and an `unknownDisks` bucket on
`ErasureBackend` (serde `default`, so older payloads still deserialize).
- ecstore diagnostics: `get_online_offline_disks_stats` now returns a third
`unknown` bucket instead of folding those drives into `offline`, keeping
`online + offline + unknown == totalDrivesPerSet`.
- ecstore notification_sys: a probe miss below the failure threshold with no
cache is now reported as `unknown` (not `initializing`) and carries the host's
drives from the pool topology (tagged `unknown`) so the summary stays balanced;
drive synthesis is factored into `synthesized_disks(host, endpoints, state)`,
shared by the offline and unknown paths.
Tracked in rustfs/backlog#1049 (P1-A + P0-A).
Co-authored-by: heihutu <heihutu@gmail.com>
* feat(utils): make DNS resolver cache TTL configurable via RUSTFS_DNS_CACHE_TTL_SECS
The resolver cache in crates/utils/src/net.rs had a hardcoded 300s TTL with
no way to tune or disable it. On orchestrated networks (Docker Swarm,
Kubernetes) a peer's DNS name is its only durable identity while its container
IP changes on reschedule, and the platform DNS always serves the current
answer. A fixed 300s application-side cache can keep a member dialing a dead IP
for up to five minutes after a peer restarts. Rust has no runtime DNS cache, so
on musl images this was the only cache in the stack and the only one that could
not be turned off.
Read the TTL from RUSTFS_DNS_CACHE_TTL_SECS (u64, default 300), matching the
existing RUSTFS_INTERNODE_* transport knobs. `0` disables caching entirely so
every get_host_ip consults the system resolver. The resolved value is logged
once at first use, removing an invisible cache from the triage surface. Change
is backward-compatible and confined to net.rs.
Also drop the two per-lookup info! logs on the resolve path: they would flood
the log when caching is disabled and added hot-path noise otherwise.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(helm): derive mTLS secret names from chart fullname (#4601)
* fix(helm): mtls certificate for server and client hardcode
* fix hardcode issue in deployment
* update typo yaml file
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: majinghe <42570491+majinghe@users.noreply.github.com>
fix(ecstore): emit CommonPrefix for object with same-named prefix in non-recursive listings (backlog#1042)
On single-disk / consistent deployments a non-recursive scan_dir that finds a/xl.meta classifies `a` as an object and does not descend, so the source never emits the prefix `a/`, dropping the CommonPrefix from delimiter listings even when `a/b` exists. This is the single-disk follow-up to backlog#880 (the multi-disk merge path was fixed in #4563).
Fix: in the scan_dir Ok branch, for a non-recursive listing of a plain object, probe whether `a/` holds a listing entry other than the object itself (new object_prefix_has_sibling_listing_entry, reusing directory_has_visible_listing_entry and skipping the object's own xl.meta and data dirs); if so, additionally schedule the prefix dir `a/`. The upstream fold in from_meta_cache_entries_sorted_infos already emits object `a` as Contents and dir `a/` as a CommonPrefix, so nothing changes there. Leaf objects (the common case) pay a single list_dir and return false immediately.
Verification: new scan_dir unit test (positive: `a` + `a/b` coexist; negative: leaf `c` produces no spurious `c/`); new end-to-end e2e (object `a` + `a/b` with delimiter "/" -> Contents `a` + CommonPrefix `a/`); existing list_objects_duplicates e2e (#1797 / #2439 regressions) stays green; set_disk::read 114 passed.
Note: lib-test compilation depends on #4573 (now merged), which added the allow_inplace_legacy_fallback argument to the codec streaming arity tests after #4560 / backlog#879.
Co-authored-by: heihutu <heihutu@gmail.com>
The three store::bucket::tests::bucket_delete_* tests drive make_bucket /
delete_bucket through the process-global local-disk registry
(GLOBAL_LOCAL_DISK_MAP) and lock client, and through Sets::new which reads the
process-global erasure mode. Under `cargo test` (single process, many threads)
they race other global-touching tests and fail deterministically with
InsufficientWriteQuorum.
serial_test's #[serial] with the crate default key serializes them across the
in-process suite. Measured at --test-threads=16: baseline fails all three every
round; with #[serial] all three pass every round. nextest's per-test processes
are covered separately by the ecstore-serial-flaky test-group in
.config/nextest.toml (#4558). Full instance-level isolation remains a backlog
#939 (InstanceContext) concern and is not attempted here.
Co-authored-by: heihutu <heihutu@gmail.com>
fix(rustfs): sanitize user-metadata in metadata=true object listing (#2743)
The console listing path (`list-type=2&metadata=true`) serializes each
object's user-metadata key as a raw XML *element name* and its value as
XML text without validation. User-metadata keys derived from HTTP
headers can legally contain characters that are illegal in an XML `Name`
(space, `$`, `%`, `#`, a leading digit, control bytes), and values can
contain C0 control characters that are illegal in XML 1.0 text.
A single object carrying such a key or value produced a malformed
`ListBucketResult` document, which the console's XML parser rejected
wholesale — so every object under that prefix vanished from the Web UI,
while plain `ListObjectsV2` (which never serializes user metadata) kept
working. The breakage appeared the moment any one object in a prefix had
XML-unsafe metadata and cleared once that object was removed, matching
the report.
Guard the serialization in `ObjectMetadataExtension::serialize_content`
(shared by the list-objects and list-versions metadata outputs): skip
entries whose key is not a valid XML element name, and strip
XML-1.0-illegal control characters from values. A single poison object
can no longer corrupt the whole listing document.
fix(ecstore): only evict remote lock channel on transport failures (#4567)
The remote lock RPC path evicted the cached gRPC channel on *any*
`tonic::Status`. Genuine transport failures (connect refused/reset,
keepalive/deadline) surface that way, but so do server-produced
application statuses on a perfectly healthy channel: `Unauthenticated`
/`PermissionDenied` from the signature interceptor, `Internal`
/`FailedPrecondition` when the peer lock service is not ready yet,
`InvalidArgument`, `ResourceExhausted`, `Unimplemented`, and so on.
Evicting the channel for those cannot help — the channel is fine — and
each spurious eviction re-dials the connection and advances the peer
toward the offline threshold via `record_peer_unreachable`, producing
background channel churn on an otherwise healthy cluster.
Classify the tonic status with `is_transport_failure` (underlying
transport source present, or one of `Unavailable`/`DeadlineExceeded`
/`Unknown`/`Cancelled`) and only evict the cached channel when it is a
real transport failure. The client-side timeout arm still evicts. This
mirrors the transport-vs-application distinction already used on the
remote-disk RPC path.
* refactor(ecstore): migrate the background-services cancel token into InstanceContext (Phase 5 Slice 13)
Phase 5 Slice 13 (backlog#939): move the background-services cancellation token
out of the process static into the per-instance InstanceContext, so cancelling
one instance's background workers (scanner/heal/tier/lifecycle) no longer
touches another instance.
- InstanceContext gains `background_cancel_token: OnceLock<CancellationToken>`
with `init_background_cancel_token` (set-once) and `background_cancel_token()`
returning an owned clone.
- global.rs `init_/get_/create_/shutdown_` helpers keep their signatures and
route through the current instance's context; the static is removed. The
getter now returns an owned `Option<CancellationToken>` instead of a
`Option<&'static _>`, which is what lets the token live in the context.
- Callers adapt to the owned token: the metadata-refresh loop drops `.cloned()`;
the lifecycle worker/loops take the owned token (a shared fallback is cloned
when the token is somehow uninitialized). No Arc cycle is introduced —
workers hold a token clone, not the instance context.
Single-instance behavior is unchanged: startup creates one token in the
bootstrap context the ECStore adopts, and shutdown cancels that same token.
Tests: the token is set-once and cancelling one instance's token leaves a
distinct instance uninitialized.
Verification: cargo test -p rustfs-ecstore (23 instance-context tests green),
cargo clippy -p rustfs-ecstore --all-targets (clean), make pre-commit (pass).
Refs: backlog#939 (Phase 5, Slice 13)
* test(ecstore): prove multi-instance isolation; document embedded guard retention (Phase 5 Slice 14) (#4588)
Phase 5 (backlog#939) capstone. The prior 13 slices moved every piece of
per-instance runtime state out of process globals into ECStore's
InstanceContext. This slice proves the result and records the remaining work.
- Add `two_instances_isolate_all_migrated_state`: an end-to-end acceptance test
that constructs two independent InstanceContexts and verifies NONE of the
migrated state is shared — erasure setup, lock manager, region, deployment id,
the four service handles (tier/notifier/expiry/transition), the local disk
registry, the bucket monitor, and the background cancel token. This is the
object-graph isolation carrier working end to end.
- Document why the embedded single-instance guard (EMBEDDED_SERVER_STARTED) is
intentionally retained: storage startup still publishes into the process-level
bootstrap context (write-once region/endpoints/deployment id) and the single
GLOBAL_OBJECT_API handle, so a second startup would fail-fast on that shared
state. Lifting the guard requires threading a per-instance context through
startup — a follow-up beyond migrating the globals. The guard is NOT removed:
rejecting the second start is safer than the panic it would otherwise become.
Verification: cargo test -p rustfs-ecstore (acceptance test + all instance-context
tests green), cargo clippy -p rustfs-ecstore --all-targets (clean), make
pre-commit (pass).
Refs: backlog#939 (Phase 5, Slice 14). Stacked on Slice 13 (#4586).
The internode HTTP/2 window/keepalive tuning (apply_http_client_tuning +
http2_keep_alive_* in build_http_client) only takes effect when the connection
negotiates HTTP/2, which happens via TLS-ALPN. Over plaintext internode
transport the connection is HTTP/1.1 and all the tuning is silently inert,
with no signal to the operator.
Make this honest without changing protocol behavior (no h2c, no
prior_knowledge, no opt-in flag):
- InternodeHttpClientTuning gains h2_tuning_explicit, true when the operator
set a window size, a non-default tuning profile, or the keepalive env.
- Add pure predicate should_warn_h2_inert(negotiated_is_http2,
h2_tuning_explicit, already_warned) and maybe_warn_h2_inert, gated by a
process-lifetime AtomicBool so the warn fires at most once.
- Call maybe_warn_h2_inert at both record_internode_http_version sites using
the actual negotiated resp.version().
- Document the TLS-ALPN requirement at apply_http_client_tuning and near the
ENV_INTERNODE_HTTP2_* constants.
Tests cover the should_warn_h2_inert truth table and that h2_tuning_explicit
reflects explicit windows and a non-default profile.
fix(server): count in-flight HTTP requests with an RAII guard (backlog#806-35)
The active-requests gauge was maintained with tower-http TraceLayer hooks:
on_request (+1), on_response (-1) and on_failure (-1). With the default
ServerErrorsAsFailures classifier a 5xx response fires BOTH on_response and
on_failure, so every 5xx decremented the gauge twice (net -1); a streaming
200 that failed mid-body did the same. The gauge drifted downward and
saturated at zero, corrupting the readiness busy-protection signal
(alias_busy_threshold_exceeded).
Replace the hook arithmetic with an InFlightGuard (RAII): InFlightLayer wraps
each request future, incrementing on entry and decrementing exactly once when
the future resolves (any status) or errors/cancels. Removed the +1 from both
on_request hooks and the -1 from trace_on_response and both on_failure hooks,
keeping their tracing and failure-metric work intact. Applied to both the
external and internode stacks.
Added a test driving the layer for 2xx, 5xx and a no-response service error,
asserting the gauge nets to zero in every case (the 5xx path is the fix).
fix(ecstore): age LastMinuteLatency samples individually (backlog#806-16)
The rolling one-minute latency window stored bare Duration samples plus a
single, never-updated start_time. Its retain predicate ignored the element
and evaluated the constant now.duration_since(start_time) < 60s, so once the
window had existed for 60s every add() dropped ALL samples and the average
degenerated to the latest single sample.
Each sample now carries its own Instant and is retained by its individual
age. The type backs only in-memory EpHealth + admin display (the wire shape
is target::LatencyStat with curr/avg/max), so Serialize/Deserialize is kept
only to satisfy the enclosing LatencyStat derive; the sample Instant is
serde(skip) and restarts aging on reload.
houseme