* fix(scanner): back off clean single-disk cycles
* fix(scanner): extend idle backoff across erasure clusters
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Replace stale dependency-depth and line-count snapshots with a domain overview based on the current Cargo workspace. This preserves the high-level architecture while avoiding references to removed crates and fragile size estimates.
Constraint: Workspace membership changes independently of architecture documentation updates
Rejected: Refresh the old numeric snapshots | they would immediately become stale again
Confidence: high
Scope-risk: narrow
Reversibility: clean
Directive: Keep Cargo.toml as the source of truth for workspace membership
Tested: git diff --check; scripts/check_doc_paths.sh; cargo metadata --no-deps --format-version 1
Not-tested: make pre-commit (documentation-only change)
* fix(sse): surface unconfigured managed SSE as 400, fix POST SSE-S3 e2e
Managed SSE (SSE-S3 / bucket-default) on a server without KMS and without RUSTFS_SSE_S3_MASTER_KEY fails closed by design since #3564, but surfaced as 500 InternalError. Map the misconfiguration to InvalidRequest (400) and seed the local SSE master key in the anonymous POST-object SSE e2e tests; align the missing-from-policy test with the MinIO-compatible SSE field exemption (s3s#608). Re-admit the tests to the e2e-full profile (rustfs#4844).
* fix(auth): POST-object lock fields and signed metadata=true listing 403s
Two authorization surfaces returned 403 for allowed requests (rustfs#4845): the PutObject access hook demanded PutObjectRetention/PutObjectLegalHold IAM actions for POST-object form uploads whose lock fields are governed by the validated POST policy, and the metadata=true listing route never resolved SigV4-verified credentials into ReqInfo so signed requests were evaluated as anonymous. Skip the PUT-header lock actions for POST form uploads and resolve credentials in the metadata route; re-admit the four e2e tests to the e2e-full profile.
Follow-up to backlog#1191 (optional sub-item). The main accept loop
spawned one task per socket with no global bound, so a connection flood
could exhaust file descriptors and memory and take existing traffic
down with it.
- New RUSTFS_API_MAX_CONNECTIONS (default 0 = unlimited, no semaphore
constructed, accept loop unchanged).
- When set, the loop acquires an owned semaphore permit BEFORE
accept(): at saturation it simply stops accepting and lets the
kernel backlog absorb bursts (TCP-native backpressure) instead of
accept-then-close churn. The permit moves into the connection task
and is released by RAII on any exit path, including TLS handshake
failures.
- Saturation is observable via the
rustfs_http_server_connection_cap_saturated_total counter, a
connection_cap_state startup event, and a per-wait debug event;
shutdown stays responsive while parked on the semaphore.
- The cap covers everything on the main listener (S3, admin, console,
internode gRPC); the constant docs carry sizing guidance.
- E2E coverage: ten sequential Connection-close requests against cap 2
prove permits never leak; two stalled connections saturating cap 2
leave a third unserved until their permits are released, after which
the queued request is accepted and answered.
POSIX lets rmdir report a non-empty directory as either ENOTEMPTY or EEXIST. Linux/macOS/Windows use ENOTEMPTY (ErrorKind::DirectoryNotEmpty); illumos/Solaris return EEXIST (errno 17), which Rust surfaces as ErrorKind::AlreadyExists and which a DirectoryNotEmpty match never catches.
LocalDisk::delete_file removes xl.meta and then recurses to rmdir the object directory, tolerating only NotFound and DirectoryNotEmpty. Since #4300 (transactional delete rollback-staging, new in beta9) the object directory still holds the rollback backup dir when that rmdir runs — the caller removes it only after write quorum is confirmed — so the rmdir legitimately reports "not empty". On Linux that is tolerated; on Solaris it is EEXIST, which fell through to the catch-all arm and became FileAccessDeniedWithContext. That failed the delete commit, rolled the metadata back, and left the object undeletable, so the client retried indefinitely with a spurious FileAccessDenied and no EACCES anywhere (rustfs/rustfs#4978). The same Linux-errno assumption also broke non-force DeleteBucket on a populated bucket on Solaris.
Add a portable is_dir_not_empty_error classifier (DirectoryNotEmpty kind plus raw ENOTEMPTY/EEXIST), mirroring MinIO's isSysErrNotEmpty, and use it at the two directory-removal sites via is_benign_object_rmdir_error (delete_file) and classify_delete_volume_error (delete_volume). The classifier is applied only at rmdir/remove_dir_all sites, where EEXIST unambiguously means "not empty", so EEXIST keeps its normal meaning everywhere else. rmdir never returns EEXIST on Linux/macOS/Windows, so the new raw-errno branch is unreachable there and the change is a strict no-op off illumos/Solaris.
Adds unit tests for the classifier (DirectoryNotEmpty/ENOTEMPTY/EEXIST match, EACCES/ENOENT reject, real non-empty rmdir against the host errno) and call-site decision tests that make a Solaris EEXIST regression detectable on Linux CI.
Fixes#4978
For a CopyObject whose source carries versioning, the response must echo the exact source version copied via x-amz-copy-source-version-id (SDK CopySourceVersionId), kept distinct from the newly created destination x-amz-version-id. RustFS set the destination version_id but left copy_source_version_id unset, so AWS SDK clients could not prove which source version was copied — the same field UploadPartCopy already populates.
Populate CopyObjectOutput.copy_source_version_id from the version actually read from the source, gated on the source bucket carrying versioning (enabled or suspended) and rendering the null version as "null", mirroring the GET/HEAD convention. Add an e2e regression test that copies a non-latest source version and asserts the source-version header equals the requested version, the destination header is a distinct new version, the destination bytes/size match the copied version, and the source version remains present.
Fixes#4976
test(ilm): re-enable test_transition_and_restore_flows; fix test-util disk-open and restore error-path lock (rustfs/backlog#1303)
The excluded test's 'missing xl.meta ... on disk2' was NOT an EC
metadata-distribution issue: after a transition all four shard disks hold
a fully consistent xl.meta (verified by decoding each shard). The panic
came from the tier test util's open_disk, which hardcoded disk_index 0
for every disk path; LocalDisk::new validates the endpoint's
(set_idx, disk_idx) against the disk's own format.json and rejected every
non-slot-0 disk with InconsistentDisk, which read_transition_meta
collapsed into 'missing xl.meta'. Derive the real indices from
format.json instead. This also un-breaks free_version_count /
wait_for_free_version_absence for non-first disks (silently 0 before).
With that fixed, the test advanced to the #4877 restore self-deadlock,
whose main paths #4886 already fixed. Complete that fix on the one path
it missed: update_restore_metadata (the restore-failure metadata
rewrite) still rebuilt copy_object options with no_lock=false and would
re-acquire the object write lock the restore handler already holds.
Propagate the caller's no_lock there too.
Remove the test from the serial-lane exclusion list; the four remaining
exclusions are unrelated known issues and stay.
fix(sse): surface unconfigured managed SSE as 400, fix POST SSE-S3 e2e
Managed SSE (SSE-S3 / bucket-default) on a server without KMS and without RUSTFS_SSE_S3_MASTER_KEY fails closed by design since #3564, but surfaced as 500 InternalError. Map the misconfiguration to InvalidRequest (400) and seed the local SSE master key in the anonymous POST-object SSE e2e tests; align the missing-from-policy test with the MinIO-compatible SSE field exemption (s3s#608). Re-admit the tests to the e2e-full profile (rustfs#4844).
Follow-up to #4895 (backlog#1191 deferred sub-item). The client-IP
dimension gives per-client fairness; this adds a collective per-bucket
budget so one hot bucket cannot monopolize the server regardless of how
many client IPs the traffic is spread across.
- Generalize RateLimiter over its key type with a Borrow-based check()
so &str lookups against String bucket keys stay allocation-free on
the hit path; client-IP call sites are unchanged in behavior.
- RateLimitLayer now carries optional client and bucket limiters; a
request must pass every configured dimension, and rejections report
which one tripped via a new 'dimension' metric label.
- Bucket extraction mirrors s3s host routing: virtual-hosted-style
resolves the Host/authority prefix against the same expanded domain
set (with port variants) the s3s router uses; otherwise the first
path segment. Admin and table-catalog namespaces are never buckets.
- New env vars RUSTFS_API_RATE_LIMIT_BUCKET_RPM/_BURST (default 0 =
dimension off) under the existing enable switch; bucket-only
configurations (client RPM 0) are supported.
- Bucket names are attacker-chosen, so the bounded-shards design
(100k keys, lossless idle sweeps, most-idle eviction) is the memory
defense; a test floods 10k random names and asserts the cap holds.
- Unit tests for extraction, shared bucket budgets across IPs, both-
dimensions interaction, and the extended env matrix; e2e test proves
bucket-only throttling on the real server with an unrelated bucket
unaffected.
fix(api): return descriptive InvalidArgument reason for Windows-unsupported object keys
On Windows hosts object keys containing NTFS-reserved characters or
Win32-unaddressable path segments are rejected up front, but the client
only saw a bare "Invalid argument" (issue #3299). Attach an explicit
reason to these rejections and surface non-empty InvalidArgument reasons
through the S3 error message.
An extracted entry whose tar header mtime is 0 was stored with
mod_time = UNIX_EPOCH, which xl.meta encodes as 0 nanos (= no
mod_time). On read-back the version failed valid() and parsing fell
into the legacy rmp_serde fallback, so every read of the object
returned 500 (invalid type: integer 0, expected an OffsetDateTime).
Treat mtime 0 as unset (tar convention) and fall back to the upload
time. Also fix two test-side issues uncovered behind the 500: the pax
fixture must use a ustar header for its XHeader entry, and the SSE-S3
extract tests must provision RUSTFS_SSE_S3_MASTER_KEY. Re-admit the 19
quarantined tests to the e2e-full merge gate.
Fixes#4842
Add Reuse Before You Write and Necessary Code Only sections to AGENTS.md,
promote quality probes into a dedicated seventh adversarial-validation
role (simplicity adversary), extend rust-code-quality checks, and dedupe
code-change-verification's restated Rust checklist into a pointer.
fix(ilm): serialize RestoreObject accepts with a CAS guard, not the whole copy-back
Implements the backlog#1304 decision: replace #4877's object write lock
held across the entire tier copy-back with an atomic compare-and-set on
the restore ongoing flag.
- ecstore: add ECStore::acquire_restore_accept_guard + RestoreAcceptGuard
(opaque, purpose-scoped object write lock with an is_lock_lost fence);
handle_restore_transitioned_object no longer locks the copy-back, so
HEAD/get_object_info stay non-blocking during a restore and the inner
put_object/complete_multipart_upload commit locks no longer self-deadlock.
- API: execute_restore_object holds the guard across the restore-status
read-check-write (no_lock inside the scope), drops it before spawning
the copy-back; SELECT-type restores keep the plain read-locked path;
the copy-back pins the resolved version so a concurrent PUT cannot
strand the flagged version at ongoing=true; concurrent restores are
rejected with 409 RestoreAlreadyInProgress (was a retryable 500) and
guard contention maps to 503 SlowDown.
- tests: drop the #4877 entry-blocking unit test (semantics deliberately
reversed), pin accept-guard mutual exclusion, update the ilm-8 restore
integration test to the final semantics, and add a concurrent
double-POST test asserting exactly one acceptance and one tier GET.
PUT admission since #4928 rejected any request whose Content-Encoding declares
aws-chunked but lacks x-amz-decoded-content-length with 400 UnexpectedContent.
Whether the body is actually chunk-framed is signalled by a STREAMING-*
x-amz-content-sha256, not by the declared encoding: the s3s auth layer only
de-frames streaming payloads and already requires the decoded length for them,
so a declared-only aws-chunked request (issue #1857 clients) carries an
unframed body whose wire Content-Length is the authoritative object size.
Admit it against that length; keep failing closed for genuinely framed bodies
without a decoded length, and use the decoded length for streaming payloads
even when Content-Encoding is absent.
Refs: https://github.com/rustfs/backlog/issues/1336
Issue #4810's production incident showed the drive timeout knobs are
undiscoverable from symptoms: a listing that outruns the walk budget
gives operators no signal pointing at RUSTFS_DRIVE_* tuning. The knobs
existed only as code constants in crates/config/src/constants/drive.rs
with no operator-facing documentation.
Add docs/operations/drive-timeout-tuning.md covering the per-operation
drive timeout knobs, resolution precedence, the high_latency profile,
and a dedicated section on listing truncation and the walk stall
budget - including the correction that since the stall-budget rework
the foreground listing path is governed by
RUSTFS_DRIVE_WALKDIR_STALL_TIMEOUT_SECS, not the total-timeout knob
the issue suggested. Link the guide from the README.
Ref #4810
docs(skills): single-commit release flow — preview and final tags share the validated commit
Version files are bumped once, directly to the final target version; the -preview.N suffix now exists only in tag names. The binary self-reports build::TAG and build.yml derives artifact naming and prerelease classification from the tag, so the preview tag and the final tag can point at the exact same commit — eliminating the post-validation version-bump commit that previously separated the validated hash from the released tag.
rustfs-release-version-bump gains a guard rejecting any -preview. target version.
test(e2e): add socket-level network fault-injection proxy (backlog#1325)
Add `FaultProxy`, an in-process async TCP proxy for black-box cluster E2E tests, under `crates/e2e_test/src/fault_proxy.rs`. It binds a random local port, forwards every accepted connection to a fixed target, and lets a test flip the wire behaviour at runtime.
Fault modes: `Pass` (transparent), `Latency(Duration)` (delay each forwarded chunk), `Blackhole` (accept but forward nothing either way and never respond), and `Partition(Direction)` (block exactly one direction while the other keeps flowing). Modes are switchable at runtime via `set_mode`, and `shutdown` stops the accept loop and drops the listener so the bound port is released.
This is the black-box counterpart to the in-process white-box hooks (rename barrier, disk call counters), which cannot reach across the process boundary into a spawned RustFS server. It serves the lock-plane one-way partition and accept-then-blackhole-peer acceptance for #1312/#1319 and the cross-process replay/tamper seam for #1327. Wiring it into the cluster harness (expose a node port through a proxy) plus the 5GiB / nightly CI-lane budget are follow-ups, since the harness multi-drive / 2-pool work lands separately (rustfs#4937); this block stays independent and self-tested.
Self-tests run against a loopback echo server that records received bytes, covering: pass round-trip identity, latency lower-bound delay, blackhole no-response-no-panic, one-way partition in both directions (request-path vs return-path), runtime mode switching on a live connection, and port release after shutdown. Timing assertions use loose lower bounds and bounded read timeouts only, never fixed-sleep absolute-value checks, to stay non-flaky. No product code changes; the module is `#[cfg(test)]`-gated.
test(e2e): add drivesPerNode and 2-pool cluster harness topology (backlog #1325)
Extend the e2e cluster harness so tests can declare per-node multiple drives (drivesPerNode) and a two-pool topology, alongside a smoke suite that boots such a cluster and round-trips a PUT/GET.
A new `ClusterTopology` describes node_count, drives_per_node, and pool membership, and `RustFSTestClusterEnvironment::with_topology` builds the matching on-disk drive directories and `RUSTFS_VOLUMES` string. `new(node_count)` now delegates to a single-pool single-drive topology, so existing single-drive cluster tests are byte-for-byte unchanged. `ClusterNode` gains `data_dirs` (all drives, `data_dirs[0] == data_dir`) and `pool_idx`; multi-drive/multi-pool clusters automatically add `RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true` because their drives share the same temp filesystem.
The `RUSTFS_VOLUMES` assembly matches the two forms the server parser accepts on a single localhost machine (verified against ecstore `DisksLayout::from_volumes`): a single pool is the explicit enumeration of every `(node, drive)` endpoint (no ellipses, one legacy DistErasure pool), while multiple pools each contribute one ellipses argument `http://<addr><node-base>/drive{0...N-1}`. A pool argument is a single URL template, so it cannot enumerate multiple distinct-port hosts; a pool striped across several localhost nodes would need a host ellipses that forces a shared on-disk path and collides physically. The topology validator therefore requires every pool in a multi-pool layout to own exactly one node and requires drives_per_node >= 2 (the parser rejects a single-drive `drive{0...0}` ellipses pool). Genuine multi-node pools need real multi-host infrastructure and are deferred to the nightly cluster lane (backlog #1313/#1314).
Unit tests assert the volumes-string layout for single-pool single-drive (backward compatible), single-pool multi-drive (8 explicit endpoints), and two-pool (one ellipses arg per pool), plus the validator rejections. The smoke suite `cluster_multidrive_pool_test` boots a 4-node x 2-drive single pool and a 2-node/2-pool cluster, asserts the volumes layout, and round-trips a PUT/GET using the harness readiness handshake (no fixed sleeps). It joins the six existing RustFSTestClusterEnvironment suites excluded from the merge-gate `e2e-full` profile so it runs in the nightly 4-node lane.
Network fault injection (toxiproxy / socket proxy) and 5GiB large-object budgets remain out of scope for this block.
test(ecstore): add rename/commit fan-out pause barrier and background-task introspection
Second white-box test-infra block for https://github.com/rustfs/backlog/issues/1325 (the first block landed the per-disk call counters in PR#4914). Adds a `#[cfg(test)]` awaitable pause barrier plus in-flight background-task introspection to the rename/commit fan-out in `crates/ecstore/src/set_disk/core/io_primitives.rs`, following the same dual-cfg seam style as the existing `disk_call_counters` and `cleanup_fault_injection` seams.
A test arms a barrier for `(object, disk_index, phase)`; the matching spawned fan-out task parks at its checkpoint until the test releases it, and the test awaits the pause through a deterministic `tokio::sync::Notify` handshake (no sleeps). A separate object-keyed task tracker reports how many rename/cleanup background disk tasks are still in flight, so a test can assert "a background disk write is still running" while paused and "no background disk write remains" once the fan-out drains. Both mechanisms live in one process-global registry keyed by object name, so concurrent tests using distinct object names stay isolated. Barriers are placed on the real `rename_data` fan-out (phase `rename`) and the `commit_rename_data_dir` old-data-dir cleanup fan-out (phase `cleanup`).
In production the barrier compiles to an immediately-ready `#[inline(always)]` no-op future and the task guard to `()`, so the fan-out control-flow shape and behavior are unchanged; only the `#[cfg(test)]` variants touch the registry. Coordinator lock-holding is asserted by the test at the store/coordinator layer via the guard it already holds; io_primitives has no handle to that namespace lock. Cross-process/black-box fault injection (toxiproxy, blackhole peers, 2-pool) remains a later cluster-harness block.
Serves the barrier-style white-box acceptances of #1312 (commit fencing: abort at the first-disk rename barrier, assert no background disk write remains after release), #1319, and #1313. Three demo tests drive the real fan-out functions and double as regression guards: neutralizing the barrier seam makes the pause await time out, and neutralizing the task guard pins the in-flight count at zero, so reverting either seam fails the demos.
Refs https://github.com/rustfs/backlog/issues/1317
Previously, when the primary disk-read permit pool stayed saturated past `RUSTFS_OBJECT_DISK_PERMIT_WAIT_TIMEOUT` (default 5s) — which a handful of slow clients can cause because a permit is held for the whole body transfer — the GET path set `disk_permit = None` and continued reading with no admission token at all. That made the disk-read concurrency limit unbounded under exactly the overload it is meant to protect against: any number of GETs could pile onto the disks simultaneously.
This replaces the permit-less bypass with a bounded overflow lane and a hard cap:
- A new bounded degraded semaphore (`RUSTFS_OBJECT_DISK_DEGRADED_READ_CAP`, default mirrors the primary cap) is consulted only after the primary wait times out. A GET takes one degraded permit without blocking, or is rejected with `SlowDown`/503 once that lane is also full. The total number of GETs performing disk-active reads is therefore hard-capped at `primary_cap + degraded_cap`, and no GET ever reads without holding an admission token.
- Admission is centralized in `ConcurrencyManager::admit_disk_read`, returning `Primary`/`Degraded`/`Unbounded`/`Rejected`. `Degraded` and `Rejected` are counted in metrics (`rustfs.get_object.disk_permit.degraded.total`, `rustfs.get_object.disk_permit.hard_reject.total`), replacing the removed `rustfs.get_object.disk_permit.bypass.total`.
- A primary cap of `0` (disk-read throttling disabled) is preserved as the only intentional permit-less path via `Unbounded`, so that degenerate-but-served configuration is not turned into all-503. The `primary_wait == 0` wait-forever opt-out is likewise unchanged.
Healthy GETs are unaffected: concurrency at or below the primary cap is admitted immediately from the primary pool exactly as before, with no new latency or rejection. The rejection is surfaced before response headers are constructed, so it is a clean pre-header 503, never a post-header 504 masquerade. Degraded-lane GETs use the identical streaming reader and forward-progress stall timeout as primary GETs, so a slow but progressing large download is not killed. Owned permits (primary or degraded) are held by `DiskReadPermitReader` and released on body EOF or client drop/cancel, so tokens are always returned.
Body stall timeout already resets on forward progress and post-header failures already surface as body errors, so no timeout-split changes were needed here.
Scope: this PR implements the minimal safe correctness core (eliminate unbounded bypass + hard cap + SlowDown). The two-level weighted-fair + aging scheduler (small setup cap feeding a size-aware data-producer fair queue) and the strictly-bounded producer/client buffer with cancellation propagation from the issue plan remain follow-ups. The black-box slow-client soak is deferred to the unbuilt facility in https://github.com/rustfs/backlog/issues/1325 rather than faked.
Tests (white-box, virtual clock via `start_paused`): degrade-then-hard-reject with token release; 100 concurrent GETs against primary=1/degraded=1 never exceed 2 simultaneous admissions with every request either admitted or explicitly rejected; disabled cap serves unbounded; zero-wait blocks on the primary lane. Reverting the hard cap makes the concurrency invariant fail.
The PutObject usecase computed `must_replicate_object` twice with the same inputs: once before commit to persist the pending replication metadata, and again after commit to drive `schedule_object_replication`. Besides repeating the versioning/config/target traversal on the hot path, the two computations read the replication configuration independently, so a config hot update landing between them could split the two phases into a pending-without-schedule or schedule-without-pending divergence.
Reuse the single immutable `ReplicateDecision` computed before commit for both the pending metadata and the post-commit schedule. The two former computations were already equivalent for a stable config (`must_replicate` reads only `opts.replication_request` from the options, which the pending-suffix insertion does not touch), so this preserves replication semantics while removing the redundant traversal and closing the config-race window. The decision carries only stable target/rule/status identifiers (arn, replicate, synchronous, id) and no secrets.
Add a white-box regression test that drives a real PutObject through the usecase and asserts, via a test-only invocation counter on `must_replicate_object`, that a single PUT computes the decision exactly once; reverting to the pre-commit + post-commit recompute makes the counter observe 2 and fails the test.
Refs: https://github.com/rustfs/backlog/issues/1320
perf(capacity): remove per-PUT global lock and per-disk allocation from write dirty-scope
Every successful write recorded its capacity dirty scope by allocating an endpoint/path String per online disk, deduplicating through a HashSet, entering the global dirty-scope Mutex, and — in the app response path — taking a global async RwLock to record the write frequency. Under small-object high concurrency this created a global serialization point and O(disks) allocation on the hot path (https://github.com/rustfs/backlog/issues/1315).
This change makes the steady-state write path allocation-free and lock-free without altering capacity accounting semantics:
- Memoize the per-set dirty scope. Each set resolves its disks' immutable endpoint/path identity lazily into a slot-indexed cache and reuses a shared `Arc<CapacityScope>`; steady-state writes clone the Arc under a read lock instead of rebuilding String/HashSet. The heal path keeps an ad-hoc scope builder because it passes disks in erasure-distribution order rather than physical-slot order.
- Add a monotonic generation to the global dirty-scope registry, advanced only when a non-empty drain removes disks. A set upgrades the global registry mutex only on the first write of each generation and then skips it while the generation is unchanged; the observed generation is read under the registry lock so a concurrent drain forces a re-mark, preventing lost updates. The write commits its bytes before recording the scope, so any drain that could remove the mark is ordered after the commit and the following refresh reads the committed bytes.
- Replace the write-frequency `RwLock<WriteRecord>` with lock-free atomics: per-second CAS buckets, an atomic last-write timestamp, and an atomic total counter. The frequency window and debounce semantics the refresh scheduler relies on are unchanged.
Capacity marking remains a conservative superset of the disks actually written, so admin/scan totals are byte-for-byte identical: extra dirty marks only trigger a re-read of a disk whose usage is unchanged. White-box tests assert the memoized scope equals the previous ad-hoc construction, that the global registry is upgraded exactly once per generation and re-marked after a drain, and that the lock-free write record is exact under concurrent contention.
Ref: https://github.com/rustfs/backlog/issues/1315
fix(quota): admit bucket hard quota against authoritative decoded size and fail closed on checker faults
Bucket-quota admission for PUT/POST previously ran before the authoritative object length was known and used the raw wire Content-Length: for an aws-chunked upload that length counts chunk framing (overcounting) and can be absent entirely, in which case the check was silently skipped. Separately, any quota-checker fault (bucket-config read, config parse, usage lookup) degraded to allow, which silently bypasses a configured hard quota.
Resolve the authoritative decoded/plain object length first — rejecting negative and unknown lengths, and requiring x-amz-decoded-content-length for aws-chunked instead of falling back to the framed wire length — then run quota admission exactly once against that size. This is the same basis the settle phase records via ObjectInfo.size, so admission and accounting agree. When no quota is configured the QuotaChecker keeps its zero-extra-I/O fast path; once a hard quota is set, checker faults now fail closed with a retryable ServiceUnavailable, increment rustfs_bucket_quota_check_failed_total, and keep the client-facing message generic so internal config/usage details are not leaked.
Size resolution and quota-outcome mapping are extracted into pure functions (resolve_put_object_authoritative_size, map_quota_check_outcome) with unit tests covering aws-chunked decoded-vs-wire, missing/negative/unknown lengths, plain PUT, the exact/over-limit admission split, and fail-closed on checker error. QuotaCheckResult is re-exported through the ecstore api::bucket::quota surface for the app layer. Cross-node reservation and overwrite-delta accounting remain out of scope (sibling issue). Also corrects one stale doc path (set_disk/core/local.rs -> disk/local.rs) flagged by the doc-paths guard.
Refs: https://github.com/rustfs/backlog/issues/1311
CompleteMultipartUpload enqueued a normal-priority heal whenever
`rename_data` returned a `Some(versions)` signature. But the per-disk
signature is produced for every object with <=10 versions, and a healthy
quorum reduces to `Some` as well, so the `Option<Vec<u8>>` return value
conflated two distinct facts — "a version signature exists" and "the
committed replicas need heal". The result: nearly every healthy MPU
completion self-enqueued a heal, while >10-version objects (signature
`None`) did not — an algorithmic heal amplification on the healthy path
(rustfs/backlog#1321).
Replace the overloaded `Option<Vec<u8>>` second element of
`SetDisks::rename_data` with an explicit `RenameConvergence` classification
computed after the write-quorum gate:
- AllSuccessIdentical — every attempted disk committed with an identical,
known signature (no heal).
- PartialCommit — write quorum met but a disk failed/offline; a committed
replica is missing or stale (heal).
- SignatureDivergent — all committed but signatures diverge, or mix signed
(<=10-version) with unsigned (>10-version) disks (heal).
- Unknown — all committed, no signature produced (>10 versions); latent
divergence is left to the scanner backstop, not self-enqueued.
`RenameConvergence::needs_heal()` is the single decision point. The version
signature is now only comparison material; it no longer doubles as a heal
flag. The old `select_rename_data_versions` / `reduce_common_versions` /
`rename_data_versions_key` machinery that carried the conflation is removed.
The heal submission in `complete_multipart_upload` moves off the ACK
critical path into a detached task: it runs after the object lock is
dropped and after the durable `rename_data` commit, survives cancellation
of the completion future, and coalesces through the existing bounded /
deduplicated / observable heal-channel admission (one submit per degraded
completion, at most). A completion cancelled in the narrow window between
the durable commit and reaching the enqueue is scanner-backstopped, as is
the Unknown (>10-version) case.
The PUT path (`object.rs`) binds the second element as `_` and is
unchanged. The change is orthogonal to and composes with the #1312 commit
fence on the same `rename_data` path (epoch rejection is a commit-gate
failure surfaced through `Result::Err`, convergence is a post-commit
signal); documented in docs/architecture/unified-object-generation.md.
Tests: `classify_rename_convergence` white-box cases cover the full
acceptance matrix (healthy 4/4 and 8/8, 3-same-1-divergent, failed/offline
disk, no-common-quorum split, >10-version all-success and with-failure,
mixed signed/unsigned) and fail on revert to the old "signature exists =>
heal" semantics. The decision function is tested directly rather than
through the process-global heal channel, whose receiver is owned
exclusively by the blackbox serial test (init_heal_channel is once per
binary).
Refs: https://github.com/rustfs/backlog/issues/1321
The legacy erasure-decode overlap path drove the speculative next-stripe read and the current-stripe emit with `tokio::join!`, which runs both futures to completion. When the current stripe's emit terminated the loop — a client disconnect or any emit error — the join still waited for the prefetch read, so a `Stop` could stall for a full shard-read deadline on a slow or wedged remote shard before the GET could fail.
Drive the two futures with a biased `select!` instead and, the moment emit reports `Stop`, drop the in-flight read future. Because the entire read pipeline is structured async (a `FuturesUnordered` of `read_shard` futures inside `ParallelReader::read`/`read_lockstep`, with no `tokio::spawn`), dropping the read future is a real cancellation: it drops every in-flight shard read and propagates cancellation down to the RemoteDisk/HTTP reader, leaving no background read behind. The `select!` is scoped so both pinned futures drop before `reader`/`shards` are reused, which is what performs the cancellation in the `Stop` case.
This only affects the overlap-enabled path. The default remains OFF (`prefetch_count == 1` and bitrot-decode overlap disabled), and the strictly-serial read -> reconstruct -> emit default branch is untouched and byte-for-byte identical. The `Continue` path preserves offset, short-tail, buffer recycle, and bitrot/reconstruction error ordering exactly as before.
Scope: cancel-safety only. The rollout decision (whether to enable overlap by default) still requires the Linux multi-node high-RTT three-size A/B from https://github.com/rustfs/backlog/issues/1310 and is deferred; this change does not flip the default or introduce any behavior that A/B must adjudicate.
White-box test `test_legacy_prefetch_cancels_next_read_on_emit_failure` drives the real `Erasure::decode` path with overlap enabled, a writer that fails emit, and shards that serve the first stripe then stall the next-stripe read far beyond the assertion window. Under the paused clock the read future is dropped and decode returns at virtual t~=0; reverting cancel-safety makes it wait out the shard-read timeout, so the test fails closed.
Refs: https://github.com/rustfs/backlog/issues/1310
fix(deps): pin tokio to 1.52.3 to fix dial9-tokio-telemetry build
tokio 1.52.4 made private, breaking
dial9-tokio-telemetry 0.3.14 which references it as public.
Pin tokio back to 1.52.3 until the upstream dependency fixes
compatibility.
Also fix a stale doc path reference in unified-object-generation.md
where is a planned file that does not exist yet.
A PUT that fans out erasure shards to remote peers awaited every shard writer to completion on both the per-block write and the final shutdown, and the remote HttpWriter had no progress deadline. A peer that accepts the TCP connection but never drains the request body (or never sends a response) therefore wedges the writer forever once the bounded buffers fill, pinning an otherwise-healthy write quorum indefinitely — a cluster-level write-availability hazard triggered by a single bad peer (rustfs/backlog#1319, https://github.com/rustfs/backlog/issues/1319).
MultiWriter now wraps each shard write and each shard-writer shutdown in a forward-progress deadline. The budget is re-armed on every block, so it bounds a stall rather than the total transfer time of a large object: a slow-but-honest writer that keeps completing shards is never killed, while a writer that makes no progress within the budget is failed and its disk dropped before commit. An optional absolute per-object cap (disabled by default) backstops a slow-drip peer that dribbles just enough progress to reset the per-block timer without ever converging; it is off by default so a legitimate large upload over a slow link is not killed on total time alone. Both knobs come from RUSTFS_OBJECT_DISK_WRITE_STALL_TIMEOUT (default 30s) and RUSTFS_OBJECT_DISK_WRITE_ABSOLUTE_CAP (default 0 = disabled); setting the stall timeout to 0 restores the previous wait-forever behavior for a conservative rollback.
The deadline enforcement lives in MultiWriter (writer-agnostic), so it covers local and remote writers alike and keeps the existing control-flow shape: a timed-out shard is marked failed (Error::Timeout, which is not an ignored error) and excluded from the write quorum exactly like any other shard write failure, and the unchanged nil_count/quorum check then continues on quorum or fails cleanly. This deliberately stays out of the MultiWriter lifecycle / commit-coordinator territory owned by rustfs/backlog#1312.
When a stalled writer is dropped to fail its shard, the remote HttpWriter must stop holding the connection and its buffered body. HttpWriter previously left its spawned request task running on drop; it now aborts that background task in Drop (it is no longer pin-projected, since every field is Unpin and the AsyncWrite impl already used get_mut). Bytes already handed to the transport cannot be unsent, but they land only in this upload's unique tmp path and are reclaimed by tmp GC — they never touch a committed object.
Tests, all on a paused virtual clock so they are deterministic and non-flaky:
- one black-hole writer still meets a 3/4 write quorum without hanging; two black holes fail the quorum cleanly (both for the per-block write and the shutdown paths).
- a slow-but-honest writer that keeps making progress within the stall budget is never failed across many blocks.
- the absolute cap bounds a slow-drip writer within a finite budget while the healthy writers keep quorum.
- the default policy is armed by default and honors 0 as disabled.
- HttpWriter aborts its background request task on drop against a hanging peer.
The toxiproxy/black-hole 4x4 end-to-end acceptance depends on black-box test facilities from rustfs/backlog#1325, which are not built yet; that acceptance is deferred to #1325 and intentionally not faked here.
test(ecstore): add per-disk call-counter registry for metadata fan-out
First landable piece of the backlog#1325 test-infrastructure work: a test-only, per-disk call-counter registry that can observe `read_version` RPC counts recorded inside `tokio::spawn` tasks. This unblocks the RPC-count assertions in backlog #1309 / #1314 / #1315, which the thread-local `CapturingRecorder` cannot serve because it is blind to metrics emitted from spawned tasks.
The new `#[cfg(test)]` module `disk_call_counters` (modeled on the existing `cleanup_fault_injection` seam) is a process-global registry keyed by object name; an RAII `observe(object)` scope collects per-disk counts and clears only its own on drop, so parallel tests using distinct object names stay isolated. A dual-`cfg` `SetDisks::record_read_version_call` seam records from inside both metadata-fanout `read_version` spawn sites for online disks only; the `#[cfg(not(test))]` variant is an empty `#[inline(always)]` fn, so production runtime behavior is unchanged.
Two demo/regression tests prove the facility works across worker threads and is revert-detecting (neutralizing the recorder makes them fail).
Refs: https://github.com/rustfs/backlog/issues/1325
* fix(get): enforce strict exact-length materialization for in-memory GET bodies (#1324)
The streaming GET path already fails a short read with UnexpectedEof, but several in-memory materialization branches only WARNed on a length mismatch and then served the body anyway: the encrypted-buffer path, the direct-memory/cache buffered-body path, and the seek-support buffer. Most dangerously, the seek-support path fell through to streaming the *same* reader after read_to_end had already drained K bytes on error, shipping a body missing its prefix (prefix-misaligned data — the closest this code came to returning wrong bytes rather than merely a short body).
Because the HTTP response commits to Content-Length == expected before the body is produced, any other body length is an unrecoverable, broken response. This change gives every in-memory source the same exact-length contract that the ODC materialize-fill path already had.
- Add strict_materialize_object_body, a shared helper that bounds the read to expected+1 (so an over-long stream is detected without buffering it unbounded), requires bytes_read == expected, and on any read error returns without reusing the partially consumed reader. The encrypted, seek, and ODC materialize branches now all route through it.
- The buffered-body branch hard-fails a length mismatch before headers instead of WARN-and-serve.
- MemoryTrackedBytesStream gains a defense-in-depth guard: a buffer whose length disagrees with the declared content length yields a stream error on first poll instead of a clean short body or a silently truncated over-long body. This backstops the cache-hit paths that hand bytes straight to the blob.
The compatibility boundary is preserved: expected is derived from get_actual_size(), the same value used for the committed Content-Length, so a legitimate object (including a legacy/backfilled-size object) whose decoded bytes equal its declared size still serves cleanly — only genuine short, over-long, or errored reads fail. This matches the streaming path, which already hard-fails short reads, so no new large-object behavior is introduced.
Tests: each source (encrypted/seek via the shared helper, cache via build_get_object_body_with_cache, buffered/memory via MemoryTrackedBytesStream) is exercised with expected N and actual N-1 / N / N+1 plus a read-K-then-error injection; only the exact-length read succeeds. Reversal is guarded: restoring WARN-and-serve or a partial fallback flips the short/over-long/error assertions from Err to Ok. Existing streaming UnexpectedEof tests are unchanged.
Refs: https://github.com/rustfs/backlog/issues/1324
* fix(app): reword WARNed comment to satisfy typos check
s3s parses a `Range` suffix length as `u64`, but the GET and HEAD handlers cast it straight to `i64` and bypass any satisfiability check. This truncates deterministically: `bytes=-18446744073709551615` wraps to `-1` and is then read as "last 1 byte", and `bytes=-0` produces a 0-length 206 instead of a 416.
Both handlers now share a single `range_to_http_range_spec` conversion that rejects a zero-length suffix with `InvalidRange` (416), clamps any suffix above `i64::MAX` to `i64::MAX` (such a suffix always covers the whole object, which `HTTPRangeSpec::get_length` then clamps to the real size), and keeps the int branch as a checked cast (s3s already caps `first`/`last` at `i64::MAX`). The scattered `as i64` casts are removed.
Note on ordering: a zero-length suffix is now rejected at conversion time, so `bytes=-0` on a missing object returns 416 rather than 404. This matches the handler's existing behavior of validating range shape (range + partNumber -> 400) before object existence.
Adds a table-driven unit test covering suffix `0/1/size/size+1/i64::MAX/i64::MAX+1/u64::MAX` over empty, 1-byte, and normal objects, asserting the InvalidRange (416) mapping, full-object return for over-size suffixes, and no regression of int first-last / open-ended ranges.
Refs: https://github.com/rustfs/backlog/issues/1322
Remove hard line wrapping from rustfs-release-publish prose (one logical line per sentence/paragraph, soft wrap for display) and link SemVer 2.0.0 spec including spec item 11 for prerelease precedence.
Require explicit target-version confirmation (AskUserQuestion with concrete semver candidates derived from the latest tag) before any release phase runs, and document SemVer 2.0.0 precedence and bump-type rules.
Adds a project skill that orchestrates the full release flow: preview tag first, CI/artifact verification, local run with console checks, rc client command validation, then final version bump and tag cut from the validated preview hash instead of latest main.
Add the shared design/contract document for backlog #1326: a single
per-object generation authority (the #1312 fencing epoch) that spans
commit fencing, read lease, prepared pool read, quota reservation, and
old-dir GC. Pins the five cross-cutting constraints once - RPC signature
binding with server-side nonce enforcement, xl.meta encoding contract
(no meta_ver bump, no positional FileInfo field, internal metadata map
under the dual-key contract), proto3 optional presence, mixed-version
fallback direction, and cluster-level capability negotiation - so the
implementation sub-issues follow them rather than each re-deciding.
No product code changes.
Henry Guo