A full refresh with partial disk failures used to commit the surviving
subset's sum as a fresh exact cluster total (no field carried the
partial-failure fact), while the complete disk cache kept the failed
disk's old value — so the reported capacity oscillated between the
partial sum and the cache-merged total on alternating refreshes.
- Add a degraded flag to CapacityUpdate/CachedCapacity, set when the
scan behind the update had partial errors; expose it in refresh logs,
admin capacity logs and a new rustfs_capacity_degraded_readings_total
counter.
- On a degraded full refresh, surface only the disks whose own scan
fully succeeded and merge them over a complete disk cache, so failed
disks keep their last-known values and the published total no longer
dips and bounces back. The disk cache is never replaced from a
degraded refresh.
- Without a complete cache, keep the partial sum (unchanged #805
non-pollution behavior) but mark the reading degraded.
Ref: rustfs/backlog#1014 (S06 from audit rustfs/backlog#1010)
Address a batch of correctness/security audit findings in the messaging
notification backends (mqtt/nats/kafka/amqp/redis/pulsar/webhook).
MQTT (backlog#971): wait for broker acknowledgement via publish_tracked +
wait_completion_async (PUBACK/PUBCOMP for QoS>=1, flush for QoS0) with a
30s timeout before reporting success, instead of treating the enqueue as
delivered. Replace substring error matching with typed ClientError /
PublishNoticeError classification.
NATS (backlog#971, #973, #983): flush() after publish to confirm the broker
received the message before the durable copy is deleted; classify publish
and flush failures by typed error kind; warn when credentials are sent
without TLS.
Kafka (backlog#973, #983): use Error::is_retriable() so transient broker
states (NotLeaderForPartition, LeaderNotAvailable, RequestTimedOut, ...) are
retried instead of dropped as permanent; add a bucket/object message key for
per-object partition ordering; build the producer without holding the cache
lock across the connect await.
AMQP (backlog#973, #980): classify permanent broker protocol errors
(404/403/406, NOT_ALLOWED, ...) as request-level errors rather than
connectivity errors to avoid reconnect storms; bound publish and
publisher-confirm waits with a timeout; check is_enabled in is_active; run
full close() cleanup even when the broker close fails; warn when
mandatory=false may silently drop unroutable messages.
Redis (backlog#982): warn when PUBLISH reaches 0 subscribers; reuse the
cached ConnectionManager for health probes instead of a fresh handshake;
warn when tls_allow_insecure disables certificate verification.
Pulsar (backlog#983): replace std::sync::Mutex + unwrap with parking_lot
Mutex so a panic while holding the guard cannot poison later accesses.
Webhook (backlog#983): drain the response body so the connection can be
reused (keep-alive). The #974 redirect-follow SSRF fix already landed.
Relates to rustfs/backlog#971
Relates to rustfs/backlog#973
Relates to rustfs/backlog#974
Relates to rustfs/backlog#980
Relates to rustfs/backlog#982
Relates to rustfs/backlog#983
Co-authored-by: heihutu <heihutu@gmail.com>
Land the remaining notify-crate audit fixes.
backlog#979(b): remove_target now enforces the same bucket-binding guard as
remove_target_config, refusing to delete a target still referenced by a bucket
rule so notification rules are not left orphaned.
backlog#984:
- event.rs: an unversioned object omits versionId entirely instead of
serializing versionId:"" (empty object/request versions treated as "no
version").
- notifier.rs: RUSTFS_NOTIFY_SEND_CONCURRENCY=0 coerces back to the default
instead of building a zero-permit semaphore that deadlocks every dispatch;
init_bucket_targets_shared closes the replaced targets instead of dropping
them without close() (connection leak).
- subscriber_index.rs: store_snapshot uses an atomic compute_if_absent upsert,
removing the get-then-insert TOCTOU that could clobber a concurrent
first-writer's snapshot cell.
- pipeline.rs: send_event assigns the history sequence and broadcasts to live
subscribers under one critical section so broadcast order matches recorded
sequence order.
- xml_config.rs: filter value length is bounded by character count, not byte
length, so valid multi-byte keys are no longer wrongly rejected.
- global.rs: a losing initialize() race shuts the just-initialized system down
instead of leaking its targets/replay workers.
backlog#970 (notify part): reload_config stops the running replay workers
before activating the new ones, so old and new workers do not concurrently
drain the same persisted stores. The full signal+join shutdown lives in the
targets crate under the same issue.
Tests: added regression coverage for each fix.
cargo build -p rustfs-notify, cargo test -p rustfs-notify --lib (98 passed),
cargo clippy -p rustfs-notify --all-targets (clean).
Relates to rustfs/backlog#979
Relates to rustfs/backlog#984
Relates to rustfs/backlog#970
Co-authored-by: heihutu <heihutu@gmail.com>
Addresses security/correctness audit findings in the target-plugin control
plane, TLS reload coordinator, and runtime extension registries.
control_plane (backlog#977):
- Install now preserves the currently installed revision as previous_revision
so Rollback actually restores the prior version instead of a no-op.
- Split the gate: circuit-breaker and runtime-activation checks apply only to
Install/Enable; Disable and Rollback stay available as break-glass
remediation while the breaker is open.
- Enforce the sidecar runtime protocol version for every external transport
(previously skippable by declaring a non-gRPC transport) and validate the
plugin api_compatibility_version at planning time.
- Download/signature/provenance host allowlisting now matches the full
host authority, so an allowlisted host never implicitly authorizes a
different host:port.
TLS reload coordinator/validate (backlog#981, #970-coordinator):
- Compute the fingerprint before building material in register() to remove the
TOCTOU that could permanently pin an old certificate; rotation self-heals.
- Always start a detection loop when reload is enabled; Watch mode no longer
returns success without any loop.
- validate_cert_key_pairing now verifies the private key matches the
certificate's public key instead of only parsing both files.
- Normalize a zero poll interval to a positive minimum to avoid a panic that
silently killed the poll loop.
- Serialize reload cycles with a per-target mutex; a first-step fingerprint
read failure now records last_error and a failure metric.
- Duplicate label registration stops and joins the previous loop before
publishing the replacement (stop-before-start), preventing orphaned loops.
runtime extension points (backlog#983 runtime subset):
- ops_profiler/ops_diagnostics authorize before probing the registry so an
unauthorized caller cannot learn whether a backend/surface exists.
- s3_hooks dispatch_post_auth actually traverses the registered hooks for the
point instead of unconditionally returning Continue.
- sidecar send_with_timeout redacts errors and uses the configurable failure
threshold via policy, and a successful send resets the breaker accounting.
Relates to rustfs/backlog#977
Relates to rustfs/backlog#981
Relates to rustfs/backlog#970
Relates to rustfs/backlog#983
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(obs): isolate process sampler windows
Refs rustfs/backlog#1004
Refs rustfs/backlog#986
- add a reusable ProcessSampler so callers can own independent sysinfo refresh windows
- wire separate sampler instances for obs metrics scheduling and memory observability
- keep compatibility helpers while avoiding cross-task CPU and disk delta interference
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs): import process sampler bundle helper
Refs rustfs/backlog#1004
Refs rustfs/backlog#986
- import collect_process_metric_bundle_with in the metrics scheduler
- drop the stale collect_process_metric_bundle import after switching scheduler sampling to independent process samplers
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs): move process sampler into blocking task
Refs rustfs/backlog#1004
Refs rustfs/backlog#986
- move the memory observability process sampler into the spawn_blocking closure
- satisfy the closure static lifetime required by tokio while keeping the isolated sampler design intact
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
refactor(ecstore,heal): return the local disk map as an owned read guard (Phase 5 prep)
Prerequisite for the Phase 5 disk-registry migration (backlog#939): the disk
map cannot move from the process global into the per-instance InstanceContext
while callers depend on a `'static` read guard borrowed from the global.
Change `local_disk_map_read` to return an owned guard
(`OwnedRwLockReadGuard`) via `Arc::read_owned` instead of
`RwLockReadGuard<'static, _>`:
- ecstore `runtime::sources::local_disk_map_read` now returns
`OwnedRwLockReadGuard<..>` (holds an Arc clone of the lock), not a `'static`
borrow of `GLOBAL_LOCAL_DISK_MAP`.
- heal's forwarding accessor and its callers (which hold the guard across
`.await` while clearing/writing per-disk markers) keep identical behavior —
the owned guard derefs to the same map, so iteration is unchanged.
This decouples the heal crate from the global's `'static` lifetime so a later
PR can source the map from the current instance's context. Single-instance
behavior is byte-for-byte unchanged; the same read lock is held across the same
awaits.
Verification: cargo test -p rustfs-heal (201 tests green), cargo clippy -p
rustfs-ecstore -p rustfs-heal --all-targets (clean), make pre-commit (pass).
Refs: backlog#939 (Phase 5, disk-registry prerequisite)
shard_read_costs_for_empty_disk_set_are_empty was a plain sync #[test], but
shard_read_costs_for_disks consults process-global topology state
(local_endpoint_hosts_for_shard_costs) whose fast-lock manager lazily spawns a
background cleanup task on first access. When this test was the first in a
process to touch that global — as under nextest's per-test isolation — the
tokio::spawn panicked with a TryCurrentError because no runtime was present,
making the test order-dependent flaky in CI (it passes only when some sibling
tokio test initializes the manager first).
Run it under a Tokio runtime like the sibling reservation tests
(#[tokio::test]), so the lazy init's spawn always has a runtime. Test-only
change; no production behavior change.
Verified failing before the change and passing after under both
`cargo test` and `cargo nextest run` in isolation.
Co-authored-by: heihutu <heihutu@gmail.com>
Phase 5 Slice 11 (backlog#939): move the background replication pool and stats —
the last service handles, and the only async ones — out of the process statics
into the per-instance InstanceContext.
- InstanceContext gains `replication_stats: OnceCell<Arc<ReplicationStats>>` and
`replication_pool: OnceCell<Arc<DynReplicationPool>>` (tokio async OnceCell),
with sync read accessors (`replication_stats`/`replication_pool`/
`replication_initialized`) and pub(crate) cell accessors for the async init.
- `init_background_replication` initializes the current instance's cells via the
same `get_or_init(async {…}).await` (workers still spawned once on first
init). The lifecycle owner helpers and the runtime-source accessors keep their
signatures and route through the current instance's context; the two statics
(and the now-unused lazy_static import) are removed. The replication-boundary
arch guard still passes.
Single-instance: init materializes one shared pool/stats via the bootstrap
context — byte-for-byte the same as the eager statics.
Tests: replication state is None until set and independent across instances.
Verification: cargo test -p rustfs-ecstore (22 instance-context tests green),
cargo clippy -p rustfs-ecstore --all-targets (clean), make pre-commit (pass).
Refs: backlog#939 (Phase 5, Slice 11). Stacked on Slice 10 (#4494).
refactor(ecstore): migrate the bucket bandwidth monitor into InstanceContext (Phase 5 Slice 10)
Phase 5 Slice 10 (backlog#939): move the bucket bandwidth monitor out of the
process static into the per-instance InstanceContext.
- InstanceContext gains `bucket_monitor: OnceLock<Arc<Monitor>>` with
`init_bucket_monitor(num_nodes)` (set-once; returns false if already set) and
`bucket_monitor() -> Option<..>`.
- global.rs `init_global_bucket_monitor` / `get_global_bucket_monitor` keep
their signatures and route through the current instance's context; the static
is removed. The ignore-and-warn-on-reinit behavior is preserved (the warn
stays in global.rs).
Tests: the monitor is None until initialized, set-once (second init is a no-op),
and independent across instances.
Verification: cargo test -p rustfs-ecstore (21 instance-context tests green),
cargo clippy -p rustfs-ecstore --all-targets (clean), make pre-commit (pass).
Refs: backlog#939 (Phase 5, Slice 10).
The non-inline rename_data commit path made the tmp xl.meta durable and then
fdatasynced the shard files in two sequential awaits, each its own blocking
round-trip. The two operations touch disjoint paths (the tmp xl.meta under the
tmp bucket vs the shard data dir) and have no ordering constraint between them:
both only need to be durable before the commit renames that follow.
Run them concurrently with tokio::join!, dropping a blocking round-trip from
the PUT commit critical path (backlog#922 step 2). The commit ordering is
unchanged — both futures complete before any rename, and a failure in either
aborts before the rename exactly as the sequential version did (tmp-meta error
is still surfaced first). Payload and metadata durability semantics are
identical under strict and relaxed tiers.
Validated by the rename_data crash-consistency harness (backlog#935): a crash
at any pre-commit point still reopens as old-or-new, never mixed. Existing
rename_data / durability-tier / disk::os tests are unchanged and pass.
Refs: rustfs/backlog#922 (HP-1 step 2), rustfs/backlog#936
Co-authored-by: heihutu <heihutu@gmail.com>
Each erasure block runs its Reed-Solomon encode through
tokio::task::block_in_place on the multi-threaded runtime. That parks the
worker and asks the scheduler to relocate other tasks, but the encode itself
is only ~110µs per 1MiB block (p99 ~542µs) — the profiling in backlog#932
flagged the scheduling disturbance as comparable to the compute it guards.
Call the encode closure inline on the multi-threaded runtime instead. The
CurrentThread (and any other) flavor keeps spawn_blocking so the sole executor
thread is never blocked and block_in_place's multi-thread-only requirement is
respected. Applied to both ingest paths (encode_block / Vec and
encode_block_bytes_mut / BytesMut); no change to encode output, quorum,
shutdown, or error handling.
Adds encode_works_on_multi_thread_runtime to cover the previously-untested
multi-threaded arm for both ingest paths, asserting streaming and batched
encode produce identical shard bytes.
This is the low-risk, correctness-neutral item that backlog#932's adversarial
verification recommended splitting out and doing first; the larger per-writer
pipeline restructure it belongs to stays gated on a Linux multi-disk baseline.
Refs: rustfs/backlog#932 (HP-11), rustfs/backlog#936
Co-authored-by: heihutu <heihutu@gmail.com>
* ci(perf): warp A/B relative-budget gate for the hotpath series
performance.yml only uploads a samply profile and a cargo-bench artifact with
no pass/fail, so a 26x-class write-path regression like #4221 or the GET perf
churn would sail through CI and only surface in customer load tests. This adds
the missing gate — the acceptance surface every queued HP change (#922/#923/
#925/#927/#930/#932) needs before its default can flip.
- scripts/hotpath_warp_ab_gate.sh: applies the relative budget to the
baseline_compare.csv that run_object_batch_bench_enhanced.sh already emits
(it computes deltas but never gates). A metric regressing past --fail-pct
fails, past --warn-pct warns; reqps/throughput are higher-is-better, latency
lower-is-better. --allow-regression downgrades a FAIL to an exempted WARN so
a deliberate correctness cost (e.g. #4221) is recorded, not blocked
(rustfs/backlog#935 correction 1). Unit-checked across pass/warn/fail/exempt.
- scripts/run_hotpath_warp_ab.sh: single-host baseline-binary vs candidate-
binary A/B over {put-4mib, get-4mib, mixed-256k} x {drive-sync on, off},
reusing the enhanced bench as the warp driver and the single-node local-disk
lifecycle. Has --dry-run; warp is assumed pre-installed as elsewhere in
scripts/. Drive-sync on/off keeps a sync-semantics change from being masked
by nosync numbers.
- .github/workflows/performance-ab.yml: nightly on main (post-merge detection)
plus opt-in pre-merge via the `perf-ab` label; `perf-deliberate-tradeoff`
runs the gate with --allow-regression; posts the gate table as a PR comment
and uploads the run. A Linux runner answers "do the macOS conclusions hold".
The gate logic is unit-validated with synthetic compare CSVs; the orchestrator
and workflow are shellcheck- and --dry-run-validated. The first real warp
measurement belongs on the Linux runner (no warp/multi-disk rig locally).
Refs: rustfs/backlog#935 (HP-14 warp A/B gate, item 4), rustfs/backlog#725
(cooled A/B harness precedent), rustfs/backlog#936
Co-Authored-By: heihutu <heihutu@gmail.com>
* ci(perf): pin upload-artifact, add external-cluster A/B mode + runbook
- Pin actions/upload-artifact to the repo-standard full-length SHA (# v6);
the previous @v4 float tripped the workflow-pin guard.
- Add an external deployment mode to run_hotpath_warp_ab.sh so warp can target
an already-running cluster instead of a throwaway single-node server:
--endpoint selects the cluster and --deploy-hook runs between phases to swap
in the phase's binary and drive-sync config (context passed via
HOTPATH_AB_PHASE / HOTPATH_AB_BINARY / HOTPATH_AB_DRIVE_SYNC). This maps onto
the team's ansible harness (cargo zigbuild -> ansible rustfs-manage --tags
stop,config,binary-copy,start -> warp).
- Restructure the matrix loop to bring a deployment up once per (phase,
drive-sync) and run all three workloads against it, instead of restarting per
workload — fewer server starts / cluster redeploys. Baseline medians are read
from a deterministic path, dropping the bash-4-only associative array so the
script (and its --dry-run self-check) runs on macOS bash 3.2 too.
- Add docs/operations/hotpath-warp-ab-runbook.md tying local mode, the ansible
external mode, and the budget/exemption together.
Verification: shellcheck clean; --dry-run in both modes prints the expected
4 deployments x 3 workloads and passes exactly 6 compare CSVs to the gate;
check_workflow_pins.sh, check_doc_paths.sh, and make pre-commit all pass.
Refs: rustfs/backlog#935, rustfs/backlog#936
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Refs rustfs/backlog#1006
- aggregate request traffic samples per type to avoid future counter collisions
- keep request schema and collector crate-internal until a production stats source exists
- preserve focused regression coverage for the internal request collector logic
Co-authored-by: heihutu <heihutu@gmail.com>
The rename_data commit sequence is the highest-risk durability path in the
store, and the HP-1/HP-4/HP-5 work (fsync coalescing, group commit, relaxed
durability tiers) all need a standing gate that proves a power loss mid-commit
can never leave a mixed or corrupt object. There was one graceful-rollback
failpoint but no crash-consistency coverage.
Add a deterministic harness that models a hard power loss — the commit
sequence stops dead at the armed step with no in-process cleanup — and then
reopens the disk to assert the raw on-disk state is coherent:
- Two pre-commit injection points, RenameDataCrashPoint::{AfterDataRename,
AfterBackupBeforeMetaCommit}, constructed at the real commit-path call sites
but gated so the production build compiles them to a const-false no-op
(mirrors the existing should_fail_before_old_metadata_backup pattern).
- A parameterized scenario over {strict, relaxed} durability x {both crash
points} x {overwrite, fresh object}: seed, stage a replacement, inject,
reopen, and assert the object reads back as exactly the old version (or does
not exist when there was no old version) with its data dir intact, never the
half-committed new one. The un-injected run asserts the commit makes the new
version visible. Relaxed is held to the same old-or-new invariant as strict
(only the durability window widens), which is exactly the property the
durability-relaxation work must not break (rustfs/backlog#878 hard rule).
- Wire an explicit `ecstore-crash-consistency` gate into the destructive
profile of run_ecstore_validation_suite.sh so it runs in the standing suite
(coordinates with the #878 destructive profile rather than building a
parallel one).
Production behavior is unchanged: the injection guards are no-ops outside
tests, and the existing rename_data rollback tests still pass.
Refs: rustfs/backlog#935 (HP-14 crash-consistency harness), rustfs/backlog#896
(test plan), rustfs/backlog#878 (ECStore validation suite), rustfs/backlog#936
Co-authored-by: heihutu <heihutu@gmail.com>
* perf(ecstore): read bitrot hash+data in one pass on the shard path
BitrotReader::read issued two reads per block: one read_exact for the
32-byte hash, then a separate loop for the shard data. On the streaming
disk reader (a raw tokio File whose every read is a spawn_blocking
round-trip) that is two dispatches per block. Since the on-disk layout is
a contiguous [hash][data] run, pull both in a single pass into a reused
scratch buffer and split afterwards, halving the per-block dispatch count
on the streaming path. The no-hash path still reads straight into the
caller buffer with no extra copy, and an in-memory Cursor (inline/mmap)
just does a slice copy.
All existing invariants are preserved: a short read of either the hash or
the data maps to UnexpectedEof before and independent of the hash check
(backlog#799 B2), and the hash-mismatch / InvalidData semantics are
unchanged.
Correctness-only change; the per-dispatch latency win is platform
dependent and its default reliance is left to the warp size-bucket
benchmark gate tracked by backlog#935, per the backlog#933 acceptance
note.
Refs: rustfs/backlog#933 (HP-12 item 2), rustfs/backlog#936
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs: reword bitrot test comment to satisfy typos check
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
A dirty-subset refresh scans only the dirty disks, so its raw
`CapacityUpdate` carries just that subset's `total_used`/`file_count`.
`update_capacity` recomputed the correct cluster-wide total into a local
variable and wrote it to the cache, but never wrote it back to the
`CapacityUpdate` that `refresh_or_join`/`spawn_refresh_if_needed` return
and publish to joiners. The admin blocking path (`resolve_admin_used_capacity`
-> `refresh_or_join_admin_disks(allow_dirty_subset=true)`) consumes
`update.total_used`, so a single dirty disk in an N-disk cluster made
admin StorageInfo report only the scanned subset's bytes (a large
transient undercount for that request and same-cycle joiners).
`CachedDiskCapacity` also dropped each disk's `file_count`/`is_estimated`,
so subset refreshes could not recompute a correct cluster file count and
would launder an estimated per-disk value into an exact cluster total.
Fix:
- `update_capacity` now reconciles `total_used`, `file_count`, and
`is_estimated` from the full per-disk cache and returns the corrected
`CapacityUpdate`.
- `CachedDiskCapacity` stores `file_count`/`is_estimated` per disk;
`file_count` sums the cache and `is_estimated` is the OR across disks.
- `refresh_or_join` and `spawn_refresh_if_needed` rebind their result to
the reconciled update so the leader return value and the value
published to joiners both carry cluster totals.
Tests: extend the subset-refresh test to assert the returned update's
reconciled `total_used`/`file_count`/`is_estimated`, and add a
`refresh_or_join` dirty-subset test asserting the leader returns the
merged cluster total (not the subset sum) and matches the cache.
Refs: https://github.com/rustfs/backlog/issues/1011
Phase 5 Slice 4 (backlog#939): move the S3 region — a write-once identity
scalar — out of the GLOBAL_REGION process static into the per-instance
InstanceContext, so two instances can serve different regions.
- InstanceContext gains `region: OnceLock<Region>` with `set_region()` /
`region()`. `set_region` keeps the write-once fail-fast contract: a second
write panics, exactly as the process global did (not downgraded to a warn).
- global.rs `set_global_region` / `get_global_region` keep their signatures and
forward to the current instance's context; the GLOBAL_REGION static is
removed. Single-instance: startup writes the bootstrap context (which the
ECStore adopts), so reads are unchanged.
Tests: set/get round-trip, two contexts hold distinct regions, and a second
set_region panics (fail-fast preserved).
Verification: cargo test -p rustfs-ecstore (9 instance-context tests green),
cargo clippy -p rustfs-ecstore --all-targets (clean), make pre-commit (pass).
Refs: backlog#939 (Phase 5, Slice 4). Stacked on Slice 3 (#4417).
The erasure-set finalize block only gated completion on failed_objects, so a pass with transient skips (unmet quorum, DiskNotFound, SlowDown, OperationCanceled) but zero hard failures was marked completed, its resume/checkpoint state cleaned up and the per-disk healing marker cleared. That violated the Transient invariant: the skipped versions were never re-healed on a later pass.
Broaden the finalize gate to failed_objects > 0 || skipped_objects > 0 and reuse the existing bounded-retry path (schedule_retry + checkpoint reset_for_retry, returning Err so the caller preserves state and keeps the healing markers). Transient conditions are deferred to the next heal cycle, never hot-retried in place.
Also fix the object/EC-decode heal success paths, which passed object_size as the failed positional arg to update_progress, corrupting objects_failed and the admin-visible success rate; pass 0 instead.
Add heal tests for the transient-skip finalize behavior and a progress test asserting a successful heal reports zero failures.
Refs rustfs/backlog#1033
extract_params_header copied every request/response header verbatim into
the maps that feed audit entries (requestQuery/responseHeader) and
notification events (req_params). Sensitive headers such as
Authorization and X-Amz-Security-Token were serialized in plaintext and
forwarded to external sinks (webhook/kafka/file), leaking long-lived
credentials.
Redact credential-bearing headers at this single chokepoint: the header
name is kept for correlation, but its value is replaced with the shared
REDACTED_SECRET placeholder. A case-insensitive sensitive-header list
covers authorization, x-amz-security-token, x-amz-content-sha256,
cookie, and set-cookie. Non-sensitive headers keep their existing
behavior.
Refs: https://github.com/rustfs/backlog/issues/963
Moka evicts entries by TTL, time-to-idle, and capacity-LRU without going through invalidate_object, so the per-object identity index (by_object) never dropped the evicted keys and grew without bound, bypassing the memory cap. Register an async eviction listener that removes truly evicted keys from the identity index, holding only a Weak reference to avoid an Arc cycle.
Refs rustfs/backlog#1031
* fix(scanner): scope long walk timeouts
* fix(scanner): bound IAM config walks
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
`LiveEventHistory::snapshot_since` resumed from the oldest retained event
whenever a consumer's cursor pointed before it, without signaling that the
in-between events had been evicted from the ring buffer. Cursor-based
consumers therefore assumed the returned batch was contiguous with their
cursor and silently lost events.
Add a `gap` flag to `LiveEventBatch`, set when the consumer's cursor is
older than the oldest retained sequence, so consumers can detect the loss
and trigger a full re-sync or alert. Add a regression test that fills and
evicts past the ring-buffer capacity and asserts a lagging cursor reports
`gap = true` (and that contiguous/caught-up cursors do not).
Refs rustfs/backlog#969
fix(ecstore): stop ListMultipartUploads from returning one upload past max-uploads (backlog#954)
The result-collection loop in `SetDisks::list_multipart_uploads` pushed an
upload into the page and only then checked `ret_uploads.len() > max_uploads`,
so each page returned up to max_uploads + 1 uploads and pointed
`next_upload_id_marker` at that surplus entry, violating the S3
ListMultipartUploads max-uploads contract.
Move the cap check before the push (MinIO-style) so a page never exceeds
max_uploads, and derive `is_truncated` from the post-marker cursor position
(`upload_idx < uploads.len()`) rather than comparing the page length against
the full listing length. The old length comparison mis-reported truncation on
the final page whenever a marker had skipped earlier entries, which prevented
marker-based pagination from terminating.
Add a regression test that starts more in-progress uploads than a page holds
and asserts a single page returns exactly max_uploads with a correct
next-marker, that the exact-boundary case is not marked truncated, and that
paginating one upload at a time enumerates every upload once with no loss,
duplication, or non-termination.
Incidental: add the missing `ctx` field to the `new_multipart_lock_test_store`
cfg(test) helper so the ecstore test build compiles after the #4413/#4437
merge collision.
Refs: https://github.com/rustfs/backlog/issues/954
Statement equality drives merge/dedup of policy statements. Omitting NotResource let semantically-distinct statements be treated as duplicates and dropped, which can shrink Deny coverage and escalate privileges. Compare not_resources too and add regression tests.
Refs rustfs/backlog#1028
`create_file` writes erasure shard / multipart part data through the page
cache, so the commit-point `sync_dir_files` fdatasync in `rename_data` has to
flush the whole shard's dirty pages (profiling: create_file avg 67.52ms vs
1.51ms nosync; BitrotWriter::write p95 40.86ms), and concurrent PUTs on one
device stall each other's writeback inside the rename critical section.
Add an opt-in Linux O_DIRECT streaming writer that reuses the direct-io read
infrastructure (statx DIOALIGN probe, aligned bounce buffer, supported latch,
one-time fallback log). Incoming bytes are staged into an aligned buffer;
full aligned batches and the shutdown tail are flushed on the blocking pool
(same offloading posture as the buffered writer it replaces). The
sub-alignment tail is written buffered after clearing O_DIRECT via fcntl
(MinIO's recipe). Shard bytes reach the device during the write phase, so the
unchanged commit-point `sync_dir_files` fdatasync degrades to a cheap
metadata/device FLUSH instead of flushing ~2MiB of dirty pages.
Gated by `RUSTFS_OBJECT_DIRECT_IO_WRITE_ENABLE` (default false), Linux-only,
and durability-neutral: `sync_dir_files` still fdatasyncs every shard at the
commit point, so the strict/relaxed/none durability tiers keep their exact
guarantees. Filesystems that reject O_DIRECT (tmpfs, overlayfs, 9p) latch the
path off and fall back to buffered writes; an O_DIRECT open/write EINVAL is
never surfaced as `InvalidInput` (which `to_file_error` maps to `FileNotFound`
and would masquerade as a missing shard, triggering a spurious EC rebuild).
Unit tests cover the env gate, the alignment/staging-capacity and tail-split
math, an end-to-end `create_file` round trip across sizes crossing the
alignment and multi-batch boundaries (buffered fallback on macOS / O_DIRECT on
a block device), and the writer state machine over a plain file on Linux. Real
O_DIRECT throughput on Linux NVMe (ext4/xfs) is deferred to the azure 4-node
bench per the issue's GO/NO-GO gate; the development host is macOS and cannot
exercise O_DIRECT.
Refs: rustfs/backlog#927, rustfs/backlog#936
Co-authored-by: heihutu <heihutu@gmail.com>
perf(ecstore): per-bucket durability tier overrides (HP-5 phase 2)
Let a bucket override the process-wide RUSTFS_DURABILITY_MODE with its own
strict/relaxed/none tier, stored as a durability.json extension entry in the
bucket metadata file and resolved at commit points via effective_durability.
System-critical buckets (.rustfs.sys, .minio.sys) can never carry an override
and stay pinned to strict; the legacy full-off switch keeps its historical
semantics and per-bucket overrides do not apply under it. Overrides are
published and cleared through the existing bucket metadata cache-invalidation
path, and an admin GET/PUT handler exposes the configuration. Default behavior
is unchanged: with no override a bucket follows the global mode, which defaults
to strict and stays byte-for-byte identical to before.
Refs: https://github.com/rustfs/backlog/issues/938, https://github.com/rustfs/backlog/issues/936
Co-authored-by: heihutu <heihutu@gmail.com>
Zhengchao An