test(ilm): re-enable test_transition_and_restore_flows; fix test-util disk-open and restore error-path lock (rustfs/backlog#1303)
The excluded test's 'missing xl.meta ... on disk2' was NOT an EC
metadata-distribution issue: after a transition all four shard disks hold
a fully consistent xl.meta (verified by decoding each shard). The panic
came from the tier test util's open_disk, which hardcoded disk_index 0
for every disk path; LocalDisk::new validates the endpoint's
(set_idx, disk_idx) against the disk's own format.json and rejected every
non-slot-0 disk with InconsistentDisk, which read_transition_meta
collapsed into 'missing xl.meta'. Derive the real indices from
format.json instead. This also un-breaks free_version_count /
wait_for_free_version_absence for non-first disks (silently 0 before).
With that fixed, the test advanced to the #4877 restore self-deadlock,
whose main paths #4886 already fixed. Complete that fix on the one path
it missed: update_restore_metadata (the restore-failure metadata
rewrite) still rebuilt copy_object options with no_lock=false and would
re-acquire the object write lock the restore handler already holds.
Propagate the caller's no_lock there too.
Remove the test from the serial-lane exclusion list; the four remaining
exclusions are unrelated known issues and stay.
fix(sse): surface unconfigured managed SSE as 400, fix POST SSE-S3 e2e
Managed SSE (SSE-S3 / bucket-default) on a server without KMS and without RUSTFS_SSE_S3_MASTER_KEY fails closed by design since #3564, but surfaced as 500 InternalError. Map the misconfiguration to InvalidRequest (400) and seed the local SSE master key in the anonymous POST-object SSE e2e tests; align the missing-from-policy test with the MinIO-compatible SSE field exemption (s3s#608). Re-admit the tests to the e2e-full profile (rustfs#4844).
Follow-up to #4895 (backlog#1191 deferred sub-item). The client-IP
dimension gives per-client fairness; this adds a collective per-bucket
budget so one hot bucket cannot monopolize the server regardless of how
many client IPs the traffic is spread across.
- Generalize RateLimiter over its key type with a Borrow-based check()
so &str lookups against String bucket keys stay allocation-free on
the hit path; client-IP call sites are unchanged in behavior.
- RateLimitLayer now carries optional client and bucket limiters; a
request must pass every configured dimension, and rejections report
which one tripped via a new 'dimension' metric label.
- Bucket extraction mirrors s3s host routing: virtual-hosted-style
resolves the Host/authority prefix against the same expanded domain
set (with port variants) the s3s router uses; otherwise the first
path segment. Admin and table-catalog namespaces are never buckets.
- New env vars RUSTFS_API_RATE_LIMIT_BUCKET_RPM/_BURST (default 0 =
dimension off) under the existing enable switch; bucket-only
configurations (client RPM 0) are supported.
- Bucket names are attacker-chosen, so the bounded-shards design
(100k keys, lossless idle sweeps, most-idle eviction) is the memory
defense; a test floods 10k random names and asserts the cap holds.
- Unit tests for extraction, shared bucket budgets across IPs, both-
dimensions interaction, and the extended env matrix; e2e test proves
bucket-only throttling on the real server with an unrelated bucket
unaffected.
fix(api): return descriptive InvalidArgument reason for Windows-unsupported object keys
On Windows hosts object keys containing NTFS-reserved characters or
Win32-unaddressable path segments are rejected up front, but the client
only saw a bare "Invalid argument" (issue #3299). Attach an explicit
reason to these rejections and surface non-empty InvalidArgument reasons
through the S3 error message.
An extracted entry whose tar header mtime is 0 was stored with
mod_time = UNIX_EPOCH, which xl.meta encodes as 0 nanos (= no
mod_time). On read-back the version failed valid() and parsing fell
into the legacy rmp_serde fallback, so every read of the object
returned 500 (invalid type: integer 0, expected an OffsetDateTime).
Treat mtime 0 as unset (tar convention) and fall back to the upload
time. Also fix two test-side issues uncovered behind the 500: the pax
fixture must use a ustar header for its XHeader entry, and the SSE-S3
extract tests must provision RUSTFS_SSE_S3_MASTER_KEY. Re-admit the 19
quarantined tests to the e2e-full merge gate.
Fixes#4842
Add Reuse Before You Write and Necessary Code Only sections to AGENTS.md,
promote quality probes into a dedicated seventh adversarial-validation
role (simplicity adversary), extend rust-code-quality checks, and dedupe
code-change-verification's restated Rust checklist into a pointer.
fix(ilm): serialize RestoreObject accepts with a CAS guard, not the whole copy-back
Implements the backlog#1304 decision: replace #4877's object write lock
held across the entire tier copy-back with an atomic compare-and-set on
the restore ongoing flag.
- ecstore: add ECStore::acquire_restore_accept_guard + RestoreAcceptGuard
(opaque, purpose-scoped object write lock with an is_lock_lost fence);
handle_restore_transitioned_object no longer locks the copy-back, so
HEAD/get_object_info stay non-blocking during a restore and the inner
put_object/complete_multipart_upload commit locks no longer self-deadlock.
- API: execute_restore_object holds the guard across the restore-status
read-check-write (no_lock inside the scope), drops it before spawning
the copy-back; SELECT-type restores keep the plain read-locked path;
the copy-back pins the resolved version so a concurrent PUT cannot
strand the flagged version at ongoing=true; concurrent restores are
rejected with 409 RestoreAlreadyInProgress (was a retryable 500) and
guard contention maps to 503 SlowDown.
- tests: drop the #4877 entry-blocking unit test (semantics deliberately
reversed), pin accept-guard mutual exclusion, update the ilm-8 restore
integration test to the final semantics, and add a concurrent
double-POST test asserting exactly one acceptance and one tier GET.
PUT admission since #4928 rejected any request whose Content-Encoding declares
aws-chunked but lacks x-amz-decoded-content-length with 400 UnexpectedContent.
Whether the body is actually chunk-framed is signalled by a STREAMING-*
x-amz-content-sha256, not by the declared encoding: the s3s auth layer only
de-frames streaming payloads and already requires the decoded length for them,
so a declared-only aws-chunked request (issue #1857 clients) carries an
unframed body whose wire Content-Length is the authoritative object size.
Admit it against that length; keep failing closed for genuinely framed bodies
without a decoded length, and use the decoded length for streaming payloads
even when Content-Encoding is absent.
Refs: https://github.com/rustfs/backlog/issues/1336
Issue #4810's production incident showed the drive timeout knobs are
undiscoverable from symptoms: a listing that outruns the walk budget
gives operators no signal pointing at RUSTFS_DRIVE_* tuning. The knobs
existed only as code constants in crates/config/src/constants/drive.rs
with no operator-facing documentation.
Add docs/operations/drive-timeout-tuning.md covering the per-operation
drive timeout knobs, resolution precedence, the high_latency profile,
and a dedicated section on listing truncation and the walk stall
budget - including the correction that since the stall-budget rework
the foreground listing path is governed by
RUSTFS_DRIVE_WALKDIR_STALL_TIMEOUT_SECS, not the total-timeout knob
the issue suggested. Link the guide from the README.
Ref #4810
docs(skills): single-commit release flow — preview and final tags share the validated commit
Version files are bumped once, directly to the final target version; the -preview.N suffix now exists only in tag names. The binary self-reports build::TAG and build.yml derives artifact naming and prerelease classification from the tag, so the preview tag and the final tag can point at the exact same commit — eliminating the post-validation version-bump commit that previously separated the validated hash from the released tag.
rustfs-release-version-bump gains a guard rejecting any -preview. target version.
test(e2e): add socket-level network fault-injection proxy (backlog#1325)
Add `FaultProxy`, an in-process async TCP proxy for black-box cluster E2E tests, under `crates/e2e_test/src/fault_proxy.rs`. It binds a random local port, forwards every accepted connection to a fixed target, and lets a test flip the wire behaviour at runtime.
Fault modes: `Pass` (transparent), `Latency(Duration)` (delay each forwarded chunk), `Blackhole` (accept but forward nothing either way and never respond), and `Partition(Direction)` (block exactly one direction while the other keeps flowing). Modes are switchable at runtime via `set_mode`, and `shutdown` stops the accept loop and drops the listener so the bound port is released.
This is the black-box counterpart to the in-process white-box hooks (rename barrier, disk call counters), which cannot reach across the process boundary into a spawned RustFS server. It serves the lock-plane one-way partition and accept-then-blackhole-peer acceptance for #1312/#1319 and the cross-process replay/tamper seam for #1327. Wiring it into the cluster harness (expose a node port through a proxy) plus the 5GiB / nightly CI-lane budget are follow-ups, since the harness multi-drive / 2-pool work lands separately (rustfs#4937); this block stays independent and self-tested.
Self-tests run against a loopback echo server that records received bytes, covering: pass round-trip identity, latency lower-bound delay, blackhole no-response-no-panic, one-way partition in both directions (request-path vs return-path), runtime mode switching on a live connection, and port release after shutdown. Timing assertions use loose lower bounds and bounded read timeouts only, never fixed-sleep absolute-value checks, to stay non-flaky. No product code changes; the module is `#[cfg(test)]`-gated.
test(e2e): add drivesPerNode and 2-pool cluster harness topology (backlog #1325)
Extend the e2e cluster harness so tests can declare per-node multiple drives (drivesPerNode) and a two-pool topology, alongside a smoke suite that boots such a cluster and round-trips a PUT/GET.
A new `ClusterTopology` describes node_count, drives_per_node, and pool membership, and `RustFSTestClusterEnvironment::with_topology` builds the matching on-disk drive directories and `RUSTFS_VOLUMES` string. `new(node_count)` now delegates to a single-pool single-drive topology, so existing single-drive cluster tests are byte-for-byte unchanged. `ClusterNode` gains `data_dirs` (all drives, `data_dirs[0] == data_dir`) and `pool_idx`; multi-drive/multi-pool clusters automatically add `RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true` because their drives share the same temp filesystem.
The `RUSTFS_VOLUMES` assembly matches the two forms the server parser accepts on a single localhost machine (verified against ecstore `DisksLayout::from_volumes`): a single pool is the explicit enumeration of every `(node, drive)` endpoint (no ellipses, one legacy DistErasure pool), while multiple pools each contribute one ellipses argument `http://<addr><node-base>/drive{0...N-1}`. A pool argument is a single URL template, so it cannot enumerate multiple distinct-port hosts; a pool striped across several localhost nodes would need a host ellipses that forces a shared on-disk path and collides physically. The topology validator therefore requires every pool in a multi-pool layout to own exactly one node and requires drives_per_node >= 2 (the parser rejects a single-drive `drive{0...0}` ellipses pool). Genuine multi-node pools need real multi-host infrastructure and are deferred to the nightly cluster lane (backlog #1313/#1314).
Unit tests assert the volumes-string layout for single-pool single-drive (backward compatible), single-pool multi-drive (8 explicit endpoints), and two-pool (one ellipses arg per pool), plus the validator rejections. The smoke suite `cluster_multidrive_pool_test` boots a 4-node x 2-drive single pool and a 2-node/2-pool cluster, asserts the volumes layout, and round-trips a PUT/GET using the harness readiness handshake (no fixed sleeps). It joins the six existing RustFSTestClusterEnvironment suites excluded from the merge-gate `e2e-full` profile so it runs in the nightly 4-node lane.
Network fault injection (toxiproxy / socket proxy) and 5GiB large-object budgets remain out of scope for this block.
test(ecstore): add rename/commit fan-out pause barrier and background-task introspection
Second white-box test-infra block for https://github.com/rustfs/backlog/issues/1325 (the first block landed the per-disk call counters in PR#4914). Adds a `#[cfg(test)]` awaitable pause barrier plus in-flight background-task introspection to the rename/commit fan-out in `crates/ecstore/src/set_disk/core/io_primitives.rs`, following the same dual-cfg seam style as the existing `disk_call_counters` and `cleanup_fault_injection` seams.
A test arms a barrier for `(object, disk_index, phase)`; the matching spawned fan-out task parks at its checkpoint until the test releases it, and the test awaits the pause through a deterministic `tokio::sync::Notify` handshake (no sleeps). A separate object-keyed task tracker reports how many rename/cleanup background disk tasks are still in flight, so a test can assert "a background disk write is still running" while paused and "no background disk write remains" once the fan-out drains. Both mechanisms live in one process-global registry keyed by object name, so concurrent tests using distinct object names stay isolated. Barriers are placed on the real `rename_data` fan-out (phase `rename`) and the `commit_rename_data_dir` old-data-dir cleanup fan-out (phase `cleanup`).
In production the barrier compiles to an immediately-ready `#[inline(always)]` no-op future and the task guard to `()`, so the fan-out control-flow shape and behavior are unchanged; only the `#[cfg(test)]` variants touch the registry. Coordinator lock-holding is asserted by the test at the store/coordinator layer via the guard it already holds; io_primitives has no handle to that namespace lock. Cross-process/black-box fault injection (toxiproxy, blackhole peers, 2-pool) remains a later cluster-harness block.
Serves the barrier-style white-box acceptances of #1312 (commit fencing: abort at the first-disk rename barrier, assert no background disk write remains after release), #1319, and #1313. Three demo tests drive the real fan-out functions and double as regression guards: neutralizing the barrier seam makes the pause await time out, and neutralizing the task guard pins the in-flight count at zero, so reverting either seam fails the demos.
Refs https://github.com/rustfs/backlog/issues/1317
Previously, when the primary disk-read permit pool stayed saturated past `RUSTFS_OBJECT_DISK_PERMIT_WAIT_TIMEOUT` (default 5s) — which a handful of slow clients can cause because a permit is held for the whole body transfer — the GET path set `disk_permit = None` and continued reading with no admission token at all. That made the disk-read concurrency limit unbounded under exactly the overload it is meant to protect against: any number of GETs could pile onto the disks simultaneously.
This replaces the permit-less bypass with a bounded overflow lane and a hard cap:
- A new bounded degraded semaphore (`RUSTFS_OBJECT_DISK_DEGRADED_READ_CAP`, default mirrors the primary cap) is consulted only after the primary wait times out. A GET takes one degraded permit without blocking, or is rejected with `SlowDown`/503 once that lane is also full. The total number of GETs performing disk-active reads is therefore hard-capped at `primary_cap + degraded_cap`, and no GET ever reads without holding an admission token.
- Admission is centralized in `ConcurrencyManager::admit_disk_read`, returning `Primary`/`Degraded`/`Unbounded`/`Rejected`. `Degraded` and `Rejected` are counted in metrics (`rustfs.get_object.disk_permit.degraded.total`, `rustfs.get_object.disk_permit.hard_reject.total`), replacing the removed `rustfs.get_object.disk_permit.bypass.total`.
- A primary cap of `0` (disk-read throttling disabled) is preserved as the only intentional permit-less path via `Unbounded`, so that degenerate-but-served configuration is not turned into all-503. The `primary_wait == 0` wait-forever opt-out is likewise unchanged.
Healthy GETs are unaffected: concurrency at or below the primary cap is admitted immediately from the primary pool exactly as before, with no new latency or rejection. The rejection is surfaced before response headers are constructed, so it is a clean pre-header 503, never a post-header 504 masquerade. Degraded-lane GETs use the identical streaming reader and forward-progress stall timeout as primary GETs, so a slow but progressing large download is not killed. Owned permits (primary or degraded) are held by `DiskReadPermitReader` and released on body EOF or client drop/cancel, so tokens are always returned.
Body stall timeout already resets on forward progress and post-header failures already surface as body errors, so no timeout-split changes were needed here.
Scope: this PR implements the minimal safe correctness core (eliminate unbounded bypass + hard cap + SlowDown). The two-level weighted-fair + aging scheduler (small setup cap feeding a size-aware data-producer fair queue) and the strictly-bounded producer/client buffer with cancellation propagation from the issue plan remain follow-ups. The black-box slow-client soak is deferred to the unbuilt facility in https://github.com/rustfs/backlog/issues/1325 rather than faked.
Tests (white-box, virtual clock via `start_paused`): degrade-then-hard-reject with token release; 100 concurrent GETs against primary=1/degraded=1 never exceed 2 simultaneous admissions with every request either admitted or explicitly rejected; disabled cap serves unbounded; zero-wait blocks on the primary lane. Reverting the hard cap makes the concurrency invariant fail.
The PutObject usecase computed `must_replicate_object` twice with the same inputs: once before commit to persist the pending replication metadata, and again after commit to drive `schedule_object_replication`. Besides repeating the versioning/config/target traversal on the hot path, the two computations read the replication configuration independently, so a config hot update landing between them could split the two phases into a pending-without-schedule or schedule-without-pending divergence.
Reuse the single immutable `ReplicateDecision` computed before commit for both the pending metadata and the post-commit schedule. The two former computations were already equivalent for a stable config (`must_replicate` reads only `opts.replication_request` from the options, which the pending-suffix insertion does not touch), so this preserves replication semantics while removing the redundant traversal and closing the config-race window. The decision carries only stable target/rule/status identifiers (arn, replicate, synchronous, id) and no secrets.
Add a white-box regression test that drives a real PutObject through the usecase and asserts, via a test-only invocation counter on `must_replicate_object`, that a single PUT computes the decision exactly once; reverting to the pre-commit + post-commit recompute makes the counter observe 2 and fails the test.
Refs: https://github.com/rustfs/backlog/issues/1320
perf(capacity): remove per-PUT global lock and per-disk allocation from write dirty-scope
Every successful write recorded its capacity dirty scope by allocating an endpoint/path String per online disk, deduplicating through a HashSet, entering the global dirty-scope Mutex, and — in the app response path — taking a global async RwLock to record the write frequency. Under small-object high concurrency this created a global serialization point and O(disks) allocation on the hot path (https://github.com/rustfs/backlog/issues/1315).
This change makes the steady-state write path allocation-free and lock-free without altering capacity accounting semantics:
- Memoize the per-set dirty scope. Each set resolves its disks' immutable endpoint/path identity lazily into a slot-indexed cache and reuses a shared `Arc<CapacityScope>`; steady-state writes clone the Arc under a read lock instead of rebuilding String/HashSet. The heal path keeps an ad-hoc scope builder because it passes disks in erasure-distribution order rather than physical-slot order.
- Add a monotonic generation to the global dirty-scope registry, advanced only when a non-empty drain removes disks. A set upgrades the global registry mutex only on the first write of each generation and then skips it while the generation is unchanged; the observed generation is read under the registry lock so a concurrent drain forces a re-mark, preventing lost updates. The write commits its bytes before recording the scope, so any drain that could remove the mark is ordered after the commit and the following refresh reads the committed bytes.
- Replace the write-frequency `RwLock<WriteRecord>` with lock-free atomics: per-second CAS buckets, an atomic last-write timestamp, and an atomic total counter. The frequency window and debounce semantics the refresh scheduler relies on are unchanged.
Capacity marking remains a conservative superset of the disks actually written, so admin/scan totals are byte-for-byte identical: extra dirty marks only trigger a re-read of a disk whose usage is unchanged. White-box tests assert the memoized scope equals the previous ad-hoc construction, that the global registry is upgraded exactly once per generation and re-marked after a drain, and that the lock-free write record is exact under concurrent contention.
Ref: https://github.com/rustfs/backlog/issues/1315
fix(quota): admit bucket hard quota against authoritative decoded size and fail closed on checker faults
Bucket-quota admission for PUT/POST previously ran before the authoritative object length was known and used the raw wire Content-Length: for an aws-chunked upload that length counts chunk framing (overcounting) and can be absent entirely, in which case the check was silently skipped. Separately, any quota-checker fault (bucket-config read, config parse, usage lookup) degraded to allow, which silently bypasses a configured hard quota.
Resolve the authoritative decoded/plain object length first — rejecting negative and unknown lengths, and requiring x-amz-decoded-content-length for aws-chunked instead of falling back to the framed wire length — then run quota admission exactly once against that size. This is the same basis the settle phase records via ObjectInfo.size, so admission and accounting agree. When no quota is configured the QuotaChecker keeps its zero-extra-I/O fast path; once a hard quota is set, checker faults now fail closed with a retryable ServiceUnavailable, increment rustfs_bucket_quota_check_failed_total, and keep the client-facing message generic so internal config/usage details are not leaked.
Size resolution and quota-outcome mapping are extracted into pure functions (resolve_put_object_authoritative_size, map_quota_check_outcome) with unit tests covering aws-chunked decoded-vs-wire, missing/negative/unknown lengths, plain PUT, the exact/over-limit admission split, and fail-closed on checker error. QuotaCheckResult is re-exported through the ecstore api::bucket::quota surface for the app layer. Cross-node reservation and overwrite-delta accounting remain out of scope (sibling issue). Also corrects one stale doc path (set_disk/core/local.rs -> disk/local.rs) flagged by the doc-paths guard.
Refs: https://github.com/rustfs/backlog/issues/1311
CompleteMultipartUpload enqueued a normal-priority heal whenever
`rename_data` returned a `Some(versions)` signature. But the per-disk
signature is produced for every object with <=10 versions, and a healthy
quorum reduces to `Some` as well, so the `Option<Vec<u8>>` return value
conflated two distinct facts — "a version signature exists" and "the
committed replicas need heal". The result: nearly every healthy MPU
completion self-enqueued a heal, while >10-version objects (signature
`None`) did not — an algorithmic heal amplification on the healthy path
(rustfs/backlog#1321).
Replace the overloaded `Option<Vec<u8>>` second element of
`SetDisks::rename_data` with an explicit `RenameConvergence` classification
computed after the write-quorum gate:
- AllSuccessIdentical — every attempted disk committed with an identical,
known signature (no heal).
- PartialCommit — write quorum met but a disk failed/offline; a committed
replica is missing or stale (heal).
- SignatureDivergent — all committed but signatures diverge, or mix signed
(<=10-version) with unsigned (>10-version) disks (heal).
- Unknown — all committed, no signature produced (>10 versions); latent
divergence is left to the scanner backstop, not self-enqueued.
`RenameConvergence::needs_heal()` is the single decision point. The version
signature is now only comparison material; it no longer doubles as a heal
flag. The old `select_rename_data_versions` / `reduce_common_versions` /
`rename_data_versions_key` machinery that carried the conflation is removed.
The heal submission in `complete_multipart_upload` moves off the ACK
critical path into a detached task: it runs after the object lock is
dropped and after the durable `rename_data` commit, survives cancellation
of the completion future, and coalesces through the existing bounded /
deduplicated / observable heal-channel admission (one submit per degraded
completion, at most). A completion cancelled in the narrow window between
the durable commit and reaching the enqueue is scanner-backstopped, as is
the Unknown (>10-version) case.
The PUT path (`object.rs`) binds the second element as `_` and is
unchanged. The change is orthogonal to and composes with the #1312 commit
fence on the same `rename_data` path (epoch rejection is a commit-gate
failure surfaced through `Result::Err`, convergence is a post-commit
signal); documented in docs/architecture/unified-object-generation.md.
Tests: `classify_rename_convergence` white-box cases cover the full
acceptance matrix (healthy 4/4 and 8/8, 3-same-1-divergent, failed/offline
disk, no-common-quorum split, >10-version all-success and with-failure,
mixed signed/unsigned) and fail on revert to the old "signature exists =>
heal" semantics. The decision function is tested directly rather than
through the process-global heal channel, whose receiver is owned
exclusively by the blackbox serial test (init_heal_channel is once per
binary).
Refs: https://github.com/rustfs/backlog/issues/1321
The legacy erasure-decode overlap path drove the speculative next-stripe read and the current-stripe emit with `tokio::join!`, which runs both futures to completion. When the current stripe's emit terminated the loop — a client disconnect or any emit error — the join still waited for the prefetch read, so a `Stop` could stall for a full shard-read deadline on a slow or wedged remote shard before the GET could fail.
Drive the two futures with a biased `select!` instead and, the moment emit reports `Stop`, drop the in-flight read future. Because the entire read pipeline is structured async (a `FuturesUnordered` of `read_shard` futures inside `ParallelReader::read`/`read_lockstep`, with no `tokio::spawn`), dropping the read future is a real cancellation: it drops every in-flight shard read and propagates cancellation down to the RemoteDisk/HTTP reader, leaving no background read behind. The `select!` is scoped so both pinned futures drop before `reader`/`shards` are reused, which is what performs the cancellation in the `Stop` case.
This only affects the overlap-enabled path. The default remains OFF (`prefetch_count == 1` and bitrot-decode overlap disabled), and the strictly-serial read -> reconstruct -> emit default branch is untouched and byte-for-byte identical. The `Continue` path preserves offset, short-tail, buffer recycle, and bitrot/reconstruction error ordering exactly as before.
Scope: cancel-safety only. The rollout decision (whether to enable overlap by default) still requires the Linux multi-node high-RTT three-size A/B from https://github.com/rustfs/backlog/issues/1310 and is deferred; this change does not flip the default or introduce any behavior that A/B must adjudicate.
White-box test `test_legacy_prefetch_cancels_next_read_on_emit_failure` drives the real `Erasure::decode` path with overlap enabled, a writer that fails emit, and shards that serve the first stripe then stall the next-stripe read far beyond the assertion window. Under the paused clock the read future is dropped and decode returns at virtual t~=0; reverting cancel-safety makes it wait out the shard-read timeout, so the test fails closed.
Refs: https://github.com/rustfs/backlog/issues/1310
fix(deps): pin tokio to 1.52.3 to fix dial9-tokio-telemetry build
tokio 1.52.4 made private, breaking
dial9-tokio-telemetry 0.3.14 which references it as public.
Pin tokio back to 1.52.3 until the upstream dependency fixes
compatibility.
Also fix a stale doc path reference in unified-object-generation.md
where is a planned file that does not exist yet.
A PUT that fans out erasure shards to remote peers awaited every shard writer to completion on both the per-block write and the final shutdown, and the remote HttpWriter had no progress deadline. A peer that accepts the TCP connection but never drains the request body (or never sends a response) therefore wedges the writer forever once the bounded buffers fill, pinning an otherwise-healthy write quorum indefinitely — a cluster-level write-availability hazard triggered by a single bad peer (rustfs/backlog#1319, https://github.com/rustfs/backlog/issues/1319).
MultiWriter now wraps each shard write and each shard-writer shutdown in a forward-progress deadline. The budget is re-armed on every block, so it bounds a stall rather than the total transfer time of a large object: a slow-but-honest writer that keeps completing shards is never killed, while a writer that makes no progress within the budget is failed and its disk dropped before commit. An optional absolute per-object cap (disabled by default) backstops a slow-drip peer that dribbles just enough progress to reset the per-block timer without ever converging; it is off by default so a legitimate large upload over a slow link is not killed on total time alone. Both knobs come from RUSTFS_OBJECT_DISK_WRITE_STALL_TIMEOUT (default 30s) and RUSTFS_OBJECT_DISK_WRITE_ABSOLUTE_CAP (default 0 = disabled); setting the stall timeout to 0 restores the previous wait-forever behavior for a conservative rollback.
The deadline enforcement lives in MultiWriter (writer-agnostic), so it covers local and remote writers alike and keeps the existing control-flow shape: a timed-out shard is marked failed (Error::Timeout, which is not an ignored error) and excluded from the write quorum exactly like any other shard write failure, and the unchanged nil_count/quorum check then continues on quorum or fails cleanly. This deliberately stays out of the MultiWriter lifecycle / commit-coordinator territory owned by rustfs/backlog#1312.
When a stalled writer is dropped to fail its shard, the remote HttpWriter must stop holding the connection and its buffered body. HttpWriter previously left its spawned request task running on drop; it now aborts that background task in Drop (it is no longer pin-projected, since every field is Unpin and the AsyncWrite impl already used get_mut). Bytes already handed to the transport cannot be unsent, but they land only in this upload's unique tmp path and are reclaimed by tmp GC — they never touch a committed object.
Tests, all on a paused virtual clock so they are deterministic and non-flaky:
- one black-hole writer still meets a 3/4 write quorum without hanging; two black holes fail the quorum cleanly (both for the per-block write and the shutdown paths).
- a slow-but-honest writer that keeps making progress within the stall budget is never failed across many blocks.
- the absolute cap bounds a slow-drip writer within a finite budget while the healthy writers keep quorum.
- the default policy is armed by default and honors 0 as disabled.
- HttpWriter aborts its background request task on drop against a hanging peer.
The toxiproxy/black-hole 4x4 end-to-end acceptance depends on black-box test facilities from rustfs/backlog#1325, which are not built yet; that acceptance is deferred to #1325 and intentionally not faked here.
test(ecstore): add per-disk call-counter registry for metadata fan-out
First landable piece of the backlog#1325 test-infrastructure work: a test-only, per-disk call-counter registry that can observe `read_version` RPC counts recorded inside `tokio::spawn` tasks. This unblocks the RPC-count assertions in backlog #1309 / #1314 / #1315, which the thread-local `CapturingRecorder` cannot serve because it is blind to metrics emitted from spawned tasks.
The new `#[cfg(test)]` module `disk_call_counters` (modeled on the existing `cleanup_fault_injection` seam) is a process-global registry keyed by object name; an RAII `observe(object)` scope collects per-disk counts and clears only its own on drop, so parallel tests using distinct object names stay isolated. A dual-`cfg` `SetDisks::record_read_version_call` seam records from inside both metadata-fanout `read_version` spawn sites for online disks only; the `#[cfg(not(test))]` variant is an empty `#[inline(always)]` fn, so production runtime behavior is unchanged.
Two demo/regression tests prove the facility works across worker threads and is revert-detecting (neutralizing the recorder makes them fail).
Refs: https://github.com/rustfs/backlog/issues/1325
* fix(get): enforce strict exact-length materialization for in-memory GET bodies (#1324)
The streaming GET path already fails a short read with UnexpectedEof, but several in-memory materialization branches only WARNed on a length mismatch and then served the body anyway: the encrypted-buffer path, the direct-memory/cache buffered-body path, and the seek-support buffer. Most dangerously, the seek-support path fell through to streaming the *same* reader after read_to_end had already drained K bytes on error, shipping a body missing its prefix (prefix-misaligned data — the closest this code came to returning wrong bytes rather than merely a short body).
Because the HTTP response commits to Content-Length == expected before the body is produced, any other body length is an unrecoverable, broken response. This change gives every in-memory source the same exact-length contract that the ODC materialize-fill path already had.
- Add strict_materialize_object_body, a shared helper that bounds the read to expected+1 (so an over-long stream is detected without buffering it unbounded), requires bytes_read == expected, and on any read error returns without reusing the partially consumed reader. The encrypted, seek, and ODC materialize branches now all route through it.
- The buffered-body branch hard-fails a length mismatch before headers instead of WARN-and-serve.
- MemoryTrackedBytesStream gains a defense-in-depth guard: a buffer whose length disagrees with the declared content length yields a stream error on first poll instead of a clean short body or a silently truncated over-long body. This backstops the cache-hit paths that hand bytes straight to the blob.
The compatibility boundary is preserved: expected is derived from get_actual_size(), the same value used for the committed Content-Length, so a legitimate object (including a legacy/backfilled-size object) whose decoded bytes equal its declared size still serves cleanly — only genuine short, over-long, or errored reads fail. This matches the streaming path, which already hard-fails short reads, so no new large-object behavior is introduced.
Tests: each source (encrypted/seek via the shared helper, cache via build_get_object_body_with_cache, buffered/memory via MemoryTrackedBytesStream) is exercised with expected N and actual N-1 / N / N+1 plus a read-K-then-error injection; only the exact-length read succeeds. Reversal is guarded: restoring WARN-and-serve or a partial fallback flips the short/over-long/error assertions from Err to Ok. Existing streaming UnexpectedEof tests are unchanged.
Refs: https://github.com/rustfs/backlog/issues/1324
* fix(app): reword WARNed comment to satisfy typos check
s3s parses a `Range` suffix length as `u64`, but the GET and HEAD handlers cast it straight to `i64` and bypass any satisfiability check. This truncates deterministically: `bytes=-18446744073709551615` wraps to `-1` and is then read as "last 1 byte", and `bytes=-0` produces a 0-length 206 instead of a 416.
Both handlers now share a single `range_to_http_range_spec` conversion that rejects a zero-length suffix with `InvalidRange` (416), clamps any suffix above `i64::MAX` to `i64::MAX` (such a suffix always covers the whole object, which `HTTPRangeSpec::get_length` then clamps to the real size), and keeps the int branch as a checked cast (s3s already caps `first`/`last` at `i64::MAX`). The scattered `as i64` casts are removed.
Note on ordering: a zero-length suffix is now rejected at conversion time, so `bytes=-0` on a missing object returns 416 rather than 404. This matches the handler's existing behavior of validating range shape (range + partNumber -> 400) before object existence.
Adds a table-driven unit test covering suffix `0/1/size/size+1/i64::MAX/i64::MAX+1/u64::MAX` over empty, 1-byte, and normal objects, asserting the InvalidRange (416) mapping, full-object return for over-size suffixes, and no regression of int first-last / open-ended ranges.
Refs: https://github.com/rustfs/backlog/issues/1322
Remove hard line wrapping from rustfs-release-publish prose (one logical line per sentence/paragraph, soft wrap for display) and link SemVer 2.0.0 spec including spec item 11 for prerelease precedence.
Require explicit target-version confirmation (AskUserQuestion with concrete semver candidates derived from the latest tag) before any release phase runs, and document SemVer 2.0.0 precedence and bump-type rules.
Adds a project skill that orchestrates the full release flow: preview tag first, CI/artifact verification, local run with console checks, rc client command validation, then final version bump and tag cut from the validated preview hash instead of latest main.
Add the shared design/contract document for backlog #1326: a single
per-object generation authority (the #1312 fencing epoch) that spans
commit fencing, read lease, prepared pool read, quota reservation, and
old-dir GC. Pins the five cross-cutting constraints once - RPC signature
binding with server-side nonce enforcement, xl.meta encoding contract
(no meta_ver bump, no positional FileInfo field, internal metadata map
under the dual-key contract), proto3 optional presence, mixed-version
fallback direction, and cluster-level capability negotiation - so the
implementation sub-issues follow them rather than each re-deciding.
No product code changes.
fix(docker): upgrade base-image packages in runtime stages
Trivy code scanning reports 40 open alerts against the published
container images, all from OS packages frozen at the base-image tag:
- musl image (alpine:3.23.4): openssl libssl3/libcrypto3 3.5.6-r0,
including HIGH CVE-2026-45447; fixed in 3.5.7-r0
- glibc image (ubuntu:24.04): tar, gzip, perl-base, ncurses CVEs with
fixes already published in the Ubuntu archive
The runtime stages only ever installed packages and never upgraded the
ones shipped with the base image, so distro security fixes could not
reach released images until the base tag itself moved. Run
`apk upgrade` / `apt-get upgrade -y` in the runtime stages of
Dockerfile, Dockerfile.glibc and Dockerfile.source so each build picks
up current security fixes.
Verified against the exact base tags: after upgrade, alpine 3.23.4
resolves libssl3/libcrypto3 3.5.7-r0 and ubuntu 24.04 resolves
tar 1.35+dfsg-3ubuntu0.2, gzip 1.12-1ubuntu3.2,
perl-base 5.38.2-3.2ubuntu0.3, ncurses 6.4+20240113-1ubuntu2.1 —
matching every fixed version demanded by the open alerts.
PR #4582 gated the update-latest-version job to stable release tags so prereleases could not overwrite the stable version pointer. But the project currently ships prerelease tags only (beta previews), so the job never runs and latest.json has been stale since. Run the job for every release tag again, and keep the honest half of the #4582 fix by writing release_type from the actual build type instead of hardcoding "stable".
* helm chart: one data drive per node - fix1
* refactor(helm): prevent negative or 0 replicaCount
Co-authored-by: Copilot <copilot@github.com>
* refactor(helm): remove env REPLICA_COUNT
* feat(helm): default no logs directory to force stdout
Co-authored-by: Copilot <copilot@github.com>
* feat(helm): add drivesPerNode
* feat(helm): correct default parity with 4 nodes / 1 drive per node
* conditional render of RUSTFS_OBS_LOG_DIRECTORY
* feat(chart): add table doc for parity
* fix(chart): handle invalid annotation objects
* fix(chart): move logging and obsevability options together; default for kubernetes output to stdout
* feat(chart): better table doc for parity
* fix(chart): leave RUSTFS_OBS_LOG_DIRECTORY empty, or the defaults will attempt to write to readonly fs
* fix(chart): chart defaults as the previous version: 4 nodes with 4 drives per node
* feat(chart): render RUSTFS_STORAGE_CLASS_STANDARD in configmap
* minor fix for pvcAnnotations
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Cristian Chiru <cristi.chiru@gmail.com>
* fix(chart): better drivesPerNode handling
* fix(chart): obs_log_directory default non empty again to preserve previous deployments compatibility
* fix(chart): proper drivesPerNode impl
* fix(chart): values clarification for empty vs unset `obs_log_directory`
* feat(chart): add service externalIPs
* feat(helm): add optional service labels
* fix(helm): merge service labels
* fix(helm): storageclass pvcAnnotations
* fix(chart): move logging and obsevability options together; default for kubernetes output to stdout
* fix(helm): remove duplicate keys
* fix(helm): default drivesPerNode null, keep legacy chart behavior
* feat(helm): add template test for externalIPs
* fix(helm): per-pool drivesPerNode inference, restore regression tests
Repairs the drivesPerNode feature from #2693 so it renders and stays
upgrade-safe:
- define the missing drives inference (rustfs.poolDrives) and compute it
per pool inside rustfs.pools, so mixed 4x4 + 16x1 pool deployments keep
their exact legacy volumeClaimTemplates when drivesPerNode is unset
- restore the $poolsEnabled definition dropped in the rebase (chart failed
to render at all)
- fix .Values references inside the pool range (dot is the pool there) and
keep per-pool storageclass pvcAnnotations overrides working
- make rustfs.volumes derive the drive range from pool drives instead of
the pod count, and relax the pools.list 4-or-16 restriction to >= 2
- reject drivesPerNode=0 explicitly instead of silently inferring
- keep default rendering identical to main: storage_class_standard stays
unrendered by default, obs_endpoint.use_stdout stays false, clusterDomain
value restored
- restore the clusterDomain/mTLS SAN/explicit-volumes regression tests that
the branch deleted, keeping the new topology tests
---------
Signed-off-by: Cristian Chiru <cristi.chiru@gmail.com>
Co-authored-by: Cristian Chiru <cristi.chiru@gmail.com>
Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Remove the Intel macOS entry from the Build and Release platform matrix so official release builds only publish the Apple Silicon macOS binary.
Co-authored-by: heihutu <heihutu@gmail.com>
* test(ilm): fix restore test object key to match transition filter
restore_object_usecase_reports_ongoing_conflict_and_completion used the
object key "restore/api-object.bin", but the shared
set_bucket_lifecycle_transition_with_tier helper only transitions objects
under the "test/" prefix. enqueue_transition_for_existing_objects therefore
matched nothing and wait_for_transition timed out at 15s, failing the test
deterministically.
The test was added in #4860 but its ILM Integration (serial) lane is
skipped on regular PRs, so it merged red and has failed on every main run
since. Move the object under the test/ prefix like every passing sibling
test in this file.
* ci(ilm): exclude broken RestoreObject API test from serial lane
restore_object_usecase_reports_ongoing_conflict_and_completion exposes a
real regression, not a test bug: the RestoreObject copy-back
(handle_restore_transitioned_object) now holds the object write lock added
in #4877 across the entire tier read-back, so it never releases in time and
the test's concurrent get_object_info times out with Lock(Timeout, 5s). The
failure is deterministic and independent of the mock tier's injected latency.
This is the same class of known-broken restore/transition failure already
tracked under backlog#1148 (three sibling scanner tests are excluded here by
name for the same reason), so exclude this one the same way until the restore
copy-back path is fixed or the #4877 lock scope is revisited. The prior commit
keeps its correct fix (the object key must live under the test/ transition
prefix); that was masking this deeper issue by never letting the object
transition in the first place.
Restore copy-back deadlock/hang under the #4877 lock is escalated separately
for a product-level decision (fix the copy-back vs. narrow/revert #4877).
* test(ilm): fix scanner restore test object keys to match transition filter
test_restore_chain_local_read_expiry_keeps_remote_and_allows_re_restore and
test_multipart_restore_preserves_parts_and_etag (both added in #4860) keyed
their objects under restore/ instead of the test/ prefix that
set_bucket_lifecycle_transition_with_tier filters on, so the objects never
transitioned and wait_for_transition timed out at 15s.
These surfaced only after the prior commit excluded the rustfs-side restore
API test: nextest runs -j1 fail-fast, so that earlier failure stopped the run
before these scanner tests executed. Unlike the excluded API test, both call
restore_transitioned_object().await sequentially and only read afterwards, so
they don't hit the concurrent-read-vs-#4877-write-lock timeout; the key
prefix was their only problem.
* ci(ilm): exclude the two remaining #4877-broken restore tests
test_multipart_restore_preserves_parts_and_etag and
test_restore_chain_local_read_expiry_keeps_remote_and_allows_re_restore both
call restore_transitioned_object().await, which since #4877 acquires the
object write lock and deterministically times out (Lock Timeout, 5s) against
an already-held lock, so restore never completes. They surfaced one at a time
because nextest runs -j1 fail-fast. The earlier prefix fix was necessary but
only advanced them from the transition wait to this restore-lock timeout.
Exclude both by name alongside their already-excluded sibling
test_transition_and_restore_flows (same root cause, tracked under
backlog#1148) so the ILM Integration (serial) lane goes green. The #4877 lock
scope still needs a product fix before any of these re-enable.
* docs(ilm): describe the excluded restore tests' symptom as a lock timeout, not a deadlock
The #4877 write lock is held across the tier read-back and outlives the 5s
lock timeout; nothing proves a true deadlock. Wording flagged by Copilot
review.
* fix(ecstore): stop restore copy-back self-deadlocking on the #4877 write lock
#4877 made handle_restore_transitioned_object hold the object write lock for
the whole restore and forward no_lock=true so the set layer would not
reacquire it. But the set-level copy-back rebuilds its own options
(put_restore_opts -> ropts, and the complete_multipart_upload opts) that
default no_lock=false, so the inner put_object / new_multipart_upload /
complete_multipart_upload each re-acquire this object's write lock in their
commit phase and block on the lock the restore already holds -> Lock(Timeout,
5s), and restore never completes.
Confirmed via RUSTFS_OBJECT_LOCK_DIAG_ENABLE: restore_transitioned_object
acquires the write lock, then holds it ~10.5s across two nested 5s acquire
timeouts before failing. This is a real product deadlock: a RestoreObject on
any transitioned object (multipart especially) hangs, not just the tests.
Propagate no_lock into the copy-back options so the inner writes inherit the
already-held lock. Use opts.no_lock (not a hardcoded true) so a caller that
restores without the outer lock still locks correctly. put_object_part is left
as-is: it locks the multipart upload-id resource, not the object key, so it
does not conflict. Verified test_multipart_restore_preserves_parts_and_etag
now passes (3.6s, was a 15s+ hang).
* ci(ilm): re-enable multipart restore test; scope remaining exclusions
The prior commit fixes the #4877 restore self-deadlock, so
test_multipart_restore_preserves_parts_and_etag passes again - drop it from
the serial-lane exclusion list and remove its 'currently excluded' note.
The other restore/transition tests still fail, but each on a DIFFERENT,
independent issue unrelated to the (now-fixed) lock, verified locally:
- test_restore_chain_...: DeleteRestoredAction sets expire_restored but no
delete path reads it, so cleanup deletes the whole object (unimplemented
semantics), not the local restored copy only.
- test_transition_and_restore_flows: transition xl.meta missing on one drive
(EC metadata distribution), not restore.
- restore_object_usecase_reports_ongoing_conflict_and_completion: asserts a
concurrent mid-restore ongoing=true read that #4877's read-vs-restore
serialization rules out (backlog#1148 ilm-8 criterion 1, an API-semantics
decision).
Comments and #[ignore] reasons updated to reflect each real cause. All remain
tracked under backlog#1148.
feat(api): wire opt-in per-client S3 API rate limiting (backlog#1191)
RustFS shipped three rate-limiter implementations and none was wired to
any request path: the tower layer never returned 429 (its over-limit
branch passed requests through) and was never instantiated, the console
env switches only logged, and the Swift token bucket was never called.
Replace them with one working, default-off implementation:
- Rewrite rustfs/src/server/rate_limit.rs as a sharded per-client-IP
token-bucket limiter (32 mutex shards instead of one global RwLock
write per request), bounded at 100k tracked IPs with lossless
refilled-idle sweeps, returning 429 + Retry-After + x-ratelimit-*
headers and an S3-style XML body.
- Key on trusted-proxy-validated ClientInfo.real_ip, else the socket
peer address; never read spoofable X-Forwarded-For/X-Real-IP headers.
Requests without a resolvable identity fail open. The echoed request
id is charset-gated to prevent reflected XML injection.
- Wire the layer once at startup via option_layer between
CatchPanicLayer and ReadinessGateLayer (external stack only), gated by
new RUSTFS_API_RATE_LIMIT_ENABLE/_RPM/_BURST constants; health and
profiling probes, internode RPC/gRPC, and the console are exempt.
- Make RUSTFS_CONSOLE_RATE_LIMIT_ENABLE/_RPM actually enforce by
reusing the same limiter core through an axum middleware.
- Delete the dead Swift ratelimit module, its isolated tests, and the
stale logging-guardrail entry; keep the live SwiftError 429 mapping.
- Add unit tests (exhaustion/recovery with injected time, concurrency,
cap eviction, spoofed-header and fail-open behavior, env matrix) and
e2e tests proving 429 + Retry-After on the real server and zero
behavior change with default configuration.
Zhengchao An