HttpCompressionConfig::extract_filename_from_content_disposition() calls header_value.to_lowercase() and slices &header_value[start..] using the byte index from lower.find("filename="), panicking on multi-byte Unicode characters whose UTF-8 byte length changes under case mapping and matching unanchored *filename= parameter suffixes.
Root Cause:
In rustfs/src/server/compress.rs, extract_filename_from_content_disposition() computes lower = header_value.to_lowercase() and uses the byte offset idx from lower.find("filename=") to slice &header_value[start..]. Because Unicode case mapping can expand or shrink UTF-8 byte lengths (e.g., U+0130 "İ" is 2 bytes in UTF-8 but lowercases to 3 bytes; U+212A "K" is 3 bytes in UTF-8 but lowercases to 1 byte), start can land inside a multi-byte UTF-8 codepoint or past the end of header_value, causing a runtime panic. In addition, lower.find("filename=") matches any parameter ending in filename= (such as x-filename=).
Fix:
Use header_value.to_ascii_lowercase() so byte offsets remain 1:1 with header_value, and require filename= to appear either at the start of the header or after a semicolon parameter delimiter.
Signed-off-by: czcbangkai <czcbangkai@google.com>
StdBackend::pread_bytes caches FdCacheEntry { file, len } and skips fstat on cache hits, passing cached len directly to unsafe { mmap_options.map(file) }. However, StdBackend::open_write / UringBackend::open_write (create_file, append_file), LocalDisk::write_all_private (write_all), LocalDisk::rename_part, and directory LocalDisk::rename_file fail to invalidate FdCache, causing mmap past EOF (SIGBUS), false DiskError::FileCorrupt on grown files, or stale reads after renames.
Root Cause:
In crates/ecstore/src/disk/local.rs, StdBackend::pread_bytes caches open File handles and their initial fstat byte length in FdCache and uses the cached len directly for bounds checks and unsafe { mmap_options.map(file) }. While LocalDisk::rename_file (non-directory) and LocalDisk::delete invalidate FdCache, five mutation paths omit invalidation: (1) StdBackend::open_write (called by create_file with O_TRUNC and append_file), (2) LocalDisk::write_all_private (called by write_all with O_TRUNC), (3) LocalDisk::rename_part, (4) LocalDisk::rename_file when src_is_dir is true, and (5) UringBackend::open_write / unlink / rename which fail to forward invalidations to self.inner. Truncating a cached file across a page boundary causes mmap to map beyond EOF and trigger SIGBUS; growing a cached file causes false DiskError::FileCorrupt.
Fix:
Invalidate FdCache in StdBackend::open_write, UringBackend::open_write/unlink/rename, LocalDisk::write_all_private, LocalDisk::rename_part (both src and dst), and directory LocalDisk::rename_file (via invalidate_prefix on src and dst).
Signed-off-by: czcbangkai <czcbangkai@google.com>
* test(e2e): add MPU RPC quorum and recovery acceptance
Exercise separate storage and lock RPC delays across multipart phases,
including quorum boundaries, cancellation, bucket recreation, and bounded
eight-node load. Honor TMPDIR for isolated cluster drive directories.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(lock): probe healthy voters behind silent peers
Spread additional probes across the existing acquisition attempt budget
when the initial quorum-plus-spare wave stalls. Preserve quorum thresholds,
fast-path fan-out, and existing ownership and late-response cleanup.
Cover every four-node read-voter ordering, a stalled eight-node write
quorum, and normal fan-out. Extend MPU acceptance to EC12+4 at twelve and
eleven metadata acknowledgements.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Recover independent keys through a bounded rolling window while preserving per-key hint generations, admission budgets, cancellation retry, and the existing transaction decision engine.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(s3): preserve aws-chunked error responses
Map named aws-chunked failures through the shared bounded body classifier,
keeping typed source precedence and the dependency's S3 response contract.
Cover exact responses, upload publication, and valid signed/trailer limits.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(s3): classify upload stream errors once per cause
Reuse the typed UploadStreamError result for digest and incomplete-body
classification while preserving the shared classifier's marker priorities.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): refresh aws-chunked profile selections
Record the two new aws-chunked regression cases in smoke and full profiles.
Verify exact additions against the CI Linux inventory and native Darwin
listings, recovering each previous digest when those two cases are removed.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(rio): isolate internode streams with pooled HTTP/1.1
Default the internode HTTP data plane to reusable HTTP/1.1 connections so paused shard bodies cannot consume shared HTTP/2 receive credit. Retain explicit auto negotiation and the existing TLS, pooling, and recovery paths.
Refs rustfs/rustfs#8843 and rustfs/backlog#2934.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(rpc): frame bounded shard reads for connection reuse
Preserve the bounded read cap and stream-to-EOF sentinel while making connection completion independent of a chunked terminator. Cover exact reads without EOF polling, pooled TLS reuse, and partial-body truncation.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(deps): migrate signer and OIDC to patched crypto
Pin the requested signer fix and upstream OIDC/OAuth2 crypto migrations to immutable commits. Preserve registry SDK type identity, remove the old bigint and elliptic-curve chains, and verify independent login state, nonce, and PKCE values.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
ConditionalCorsLayer took the bucket from the first path segment only.
With RUSTFS_SERVER_DOMAINS set, a virtual-hosted-style request such as
OPTIONS http://mybucket.localhost:9000/x.jpg was evaluated against a
bucket named "x.jpg", so a preflight never matched the bucket's CORS
configuration and returned 200 without Access-Control-* headers, while
the same request in path style succeeded.
Resolve the bucket from the host prefix first, reusing the virtual-host
matching that the API rate limiter already uses, and fall back to the
first path segment for path-style requests. A bucket-level request on a
virtual host ("/") is now treated as a bucket request instead of the
server root.
Co-authored-by: cozima <8464261+cozima@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* test(scanner): isolate async checkpoint log capture
Scope captured tracing subscribers to each future poll and cover the
competing-dispatcher failure without weakening checkpoint assertions.
Verify exact bounded erasure-probe accounting around the 64-entry limit.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): classify erasure candidates in one budgeted walk
Continue directory classification in cooperative 64-entry batches until
part evidence or EOF, retaining normal checkpoint and namespace work.
Defer metadata accounting and namespace reuse until classification ends.
Distinguish vanished opens from incomplete iteration and preserve quota,
sampling, failed-object and non-destructive heal behavior.
Pin test callsite interest alongside poll-scoped capture to prevent an
unsubscribed sibling from caching the checkpoint callsite as disabled.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): purge every explicit delete-marker version
Keep explicit marker removals on the canonical version-delete path so
same-key batches are not truncated and pending replication does not retain
source markers. Preserve marker identity and timestamps in the response.
Add real four-disk regressions for mixed version cleanup, duplicate marker
IDs, pending replication, and null/UUID marker removal.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(filemeta): index explicit delete-marker batches
Reuse the batch index and final compaction for ordinary marker removals.
Keep small marker-only histories on the scalar path and preserve fallback
for free versions and complex mutations.
Add scalar-equivalence, boundary, corruption and deterministic scan-count
coverage, a real-disk mixed-history regression, and reproducible marker
benchmark cases. Preserve existing benchmark defaults while honoring CLI
sampling overrides.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(filemeta): remove redundant marker fixture clone
Move the original metadata into its final test fixture after the last
borrow, avoiding the redundant clone reported by strict Clippy checks.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(storage): validate explicit marker deletion and recovery
Exercise UUID/null and duplicate explicit marker removals over real HTTP
with retries disabled, including immediate four-node visibility and exact
retained payloads. Cover all-drive PREPARED/APPLIED/DECISION process
termination, rollback/commit recovery, and old-ID retries across a second
restart without deleting a fresh marker.
Use flat sampling and fixed per-case fixture batches in the metadata
benchmark. Update profile selection digests for the added regressions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(heal): synchronize crash fixture startup and flush readiness
Give disk setup and the durable flush boundary separate readiness gates,
keeping each phase bounded to ten seconds. Preserve the real SIGKILL and
restart assertions while reporting early child exits and retaining output.
Reap fixture children on assertion failure and cover early-exit diagnostics.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): roll a wedged pause backlog forward to its proven ledger
For rustfs/backlog#2873 defect E.
Two interleaved writers could leave a replica state that no claim could
leave. Writer B read writer A's partial commit and selected the stable
rollback point. A's commit then completed and A verified it. B's rollback
stabilization overwrote every member it had read after the commit, and A's
stabilization reached only the remaining member. No commit survived and the
stable fields differed, so every claim and every decommission retirement
failed with "neither a surviving membership commit nor a stable rollback
point".
- When no commit survives and stable consensus fails, select the unique
newest stable ledger by (writer epoch, generation). A stable field is
written only after its ledger was selected as authority, so that ledger is
the last proven one. The next claim stabilizes it and advances normally.
- Keep failing closed for an invalid or unreadable replica, for two
different stable ledgers at the newest key, and for a missing replica that
a stored commit names as a member.
- A rollback proof written without an authoritative commit names only
members that already store a record, so a failed first write to a new
member cannot turn it into a named member that reads as lost.
- Retirement refuses a rolled-forward ledger until the writer has
stabilized it, as a retryable handoff conflict rather than a terminal
authority conflict.
- Tests replay the interleaving on real storage: it converges and retires,
it converges after another expansion with a failed first write to a new
member, and a same-key fork or a lost named member still refuses.
* fix(scanner): let retirement stabilize a rolled-forward pause backlog
Follow-up to the defect E roll-forward for rustfs/backlog#2873.
- Retirement no longer waits for a scanner writer. When no commit
survives and the native selector over every read member (survivors,
retiring source and queued pools) rolls forward, the planner commits and
stabilizes that ledger on the survivors under the writers' object fence.
The previous retryable handoff could exhaust its ~13.5 s budget long
before a writer re-claimed, and failed the decommission terminally.
- A writer's roll-forward refuses when a stored commit names a member that
is missing or outside its read, such as a retiring source excluded from
the writable set. Retirement, which reads that member, decides instead.
- A writer stabilizing a rolled-forward ledger writes missing members
first and stops at the first failure. A member that only read as missing
rejects the create, so no surviving proof can omit its newer record.
- Docs describe the actual roll-forward preconditions and the retirement
exit.
- Tests: retirement converges the wedge with no writer; a writer refuses to
roll forward past an unread source that holds the newest ledger and
retirement keeps it; a spurious missing read changes no record and the
next claim converges on the newest ledger.
* fix(decommission): create missing pause backlog survivors first
Follow-up to the defect E retirement roll-forward for rustfs/backlog#2873.
- Retirement creates survivors that read as missing one at a time, before
any stored survivor, and stops at the first failed create. A transient
create failure on a new pool's set no longer leaves every other survivor
holding a commit that names a still-missing member, which no later
selection could leave.
- A rolled-forward retirement first seeds a rollback proof that names only
the survivors that store a record; the commit still names every survivor.
- Retirement rolls forward only when the survivors cannot select a ledger
on their own or select the same one. Survivors that agree on a different
ledger keep the source-authority check.
- The scanner writer applies the same missing-first, stop-at-first-failure
order to every stabilization with a missing member, including a seed from
an old commit selected as the rollback proof.
- Tests: a failed new-pool create during retirement retries to convergence;
a spurious missing read cannot seed an old commit over a newer member;
the rolled-forward precedence over source stable records is pinned.
* docs(decommission): state the all-missing survivor limit of backlog roll-forward
* test(e2e): use the resumed decommission completion budget
* test(heal): exercise checkpoints written by rc.5
* test(heal): route upgrade fixtures through the storage facade
* fix(deps): update ctutils to address constant-time advisory
* test(lifecycle): gate recovery I/O while advancing test time
* fix(decommission): count settled entries once and back off repeated pauses
Follow-up to rustfs/rustfs#8824 for rustfs/backlog#2873 defect F.
- Count an entry's versions when the entry settles (copied and cleaned up,
or retained) instead of as each version copies. A partially moved entry
that is deferred, paused and resumed, or interrupted, no longer counts its
already-moved versions again when the resumed pass finds them on the
target.
- Hold the watchdog relaunch of a decommission that pauses again on the
same cause without progress, doubling from 30 s up to 15 min, and warn with
the cause. A first pause still relaunches on the next pass; the pause never
becomes a terminal failure.
- Log each drive's error when a delete version loses write quorum, so the
cause of a sub-quorum delete-marker copy is recorded instead of only its
rollback.
- Tests: pause/resume counting, a target whose metadata is below read
quorum, the copy-quorum pause cause, the relaunch backoff, and a rollback
test that asserts the restore attempt instead of a tracing log.
* fix(decommission): measure pause progress by cleaned entries
Review follow-up on the pause/resume counting and relaunch backoff.
- A pause now carries this generation's progress counters into the
pool metadata it persists, like a progress checkpoint. Before, everything
counted since the last checkpoint was dropped at the pause.
- A retained entry (one whose source stays) is relisted by every resumed
pass. A per-instance ledger counts its versions once per decommission
generation, so retained versions no longer grow per round in one process.
- The relaunch backoff measures progress by entries moved off the pool and
completed buckets, which a relisted retained entry cannot advance. Before,
the recounted retained entry looked like progress and reset the backoff.
- Tests: exact totals across a pause and a resume from persisted metadata;
a target kept below quorum defers the entry with nothing counted; a
retained entry plus a repeated cause escalates the backoff over three
pauses, with the watchdog path relaunching a first pause and holding a
repeat.
* fix(ecstore): bound retained decommission accounting
* feat(scanner): configure startup recovery backoff cap
* test(scanner): compare recovery retry count without narrowing
* docs(scanner): document recovery backoff cap and expose it in Helm
List scanner.recovery_backoff_cap in the runtime controls table, drop the
fixed ten-minute wording from the follower recovery section, and map
config.rustfs.scanner.recovery_backoff_cap_secs to
RUSTFS_SCANNER_RECOVERY_BACKOFF_CAP_SECS in the chart.
Run two cargo update and cargo upgrade rounds from origin/main.
Upgrade the hotpath crate family to 0.29.0 and refresh compatible
AWS, Redis, profiling, crypto, UUID, and certificate dependencies.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(quota): retire degraded floors from durable usage reports
* docs(quota): clarify retained usage cache refresh behavior
* fix(admin): report unknown bucket usage as null in GetBucketQuota
GetBucketQuota serialized missing usage as size 0, which contradicted the
D3 rule that unknown usage is never reported as zero and, after retiring
degraded floors, made a bucket near its limit look empty. Make size
optional and stop collapsing a missing or unreadable usage snapshot to 0.
* fix(scanner): recover dual corruption with retained epoch fencing
* fix(scanner): preserve recovery markers and legacy quota floors
* test(scanner): expect retained usage recovery marker state
* docs(scanner): register dual-corruption recovery evidence objects
Add the .bloomcycle.bin.dual-recovery/ evidence family to the persisted
objects table and describe its owner, write-once lifecycle and recovery
rule in Deletion And Recovery Rules.
* fix(s3): expose typed causes for unavailable responses
Keep the existing S3 retry contract while distinguishing namespace lock
timeouts, other lock refusals, and storage write quorum failures. Reuse one
bounded typed-source walk and document multipart session attribution.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(s3): preserve XML type on attributed errors
Keep application/xml when the pinned serializer replaces generated headers
with the custom cause map. Assert the media type on direct and wrapped lock
timeouts and namespace/write-quorum error responses.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): unify multipart generation fence lifetimes
Own registry registration before asynchronous incarnation validation so
cancellation cannot preserve a memo beyond real lifecycle-lock coverage.
Reuse successful validation for overlapping PUT and multipart operations,
and retain the full owner through detached part commits and orphan cleanup.
Keep cold validation concurrent and preserve authoritative snapshot,
borrowed-fence, and disabled-lock checks.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): remove redundant pending closure
Pass std::future::pending directly to the incarnation validator in the
cancellation regression test so strict Clippy accepts the test target.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(decommission): keep partially moved versions readable and defer lost-quorum copies
A version copy whose target write kept losing drive quorum was counted as a
permanent per-object failure, and a partially moved entry made every copied
version unreadable: the retained source lists all versions while the target
lists only the copied ones, and the cross-pool identity check treated that
version-list difference as a same-time conflict (SlowDownRead).
- Compare the version-list fields (num_versions, successor_mod_time,
is_latest) only for exact identities; data-movement equivalent copies of a
version stay readable while the entry is partially moved.
- Defer a delete marker, object or tiered version copy that exhausts its
in-place retries on a drive quorum error to the end-of-round replay instead
of counting it failed; a persistent loss pauses the decommission, which the
local decommission watchdog resumes.
- Skip the backup rename when rolling back a delete transaction whose journal
recorded no original metadata, so a marker created on a missing key no
longer logs a NotFound that reads as the cause of the failed write.
Refs rustfs/backlog#2873
* fix(ecstore): close copy quorum and tiered read gaps
* test(ecstore): synchronize capacity fixtures by object
overtrue
