* fix(ecstore): retain namespace owners through local physical tails
(cherry picked from commit a2f242463316e87604feadbdac5e4148140e72c0)
* test(ecstore): expose stale fsync group cleanup
* fix(ecstore): capture complete fsync worker guard
(cherry picked from commit 1dc90bb836e20ea9ee45d0a629a9201e20d231c4)
* fix(ecstore): preserve successor fsync group registration
(cherry picked from commit c7dfaad90526052e56c57dafffa4813bdcde46ca)
* test(ecstore): mark physical owner fixtures as inline
* test(ecstore): wait for namespace owner release before asserting
The namespace owner tests decided that ownership had ended when the Weak probe stopped upgrading or when the mutation lease could be reacquired. Both signals fire before the owner guard's Drop decrements the pending counter: Arc releases its strong count before running Drop, and the lease drops its locks before its owner field. The rio-v2 lane hit that window in undo_fresh_version_keeps_physical_namespace_owner_after_timeout.
Extend every drain wait to also require namespace_commits_pending() to be false, so the assertions observe the completed release instead of racing it.
The batch `NewerNoncurrentVersions` expiry path took a lifecycle event
argument and ignored it: after `delete_objects` committed it only evicted
the cache and scheduled replication deletes, so a successful noncurrent
version expiry was invisible to notification subscribers while the
equivalent current-version path emitted a lifecycle expiration event.
Emit that event from the batch path too, reusing the existing lifecycle
audit sink and event contract. Only entries that actually mutated
something are announced, and cache eviction and replication scheduling
keep their existing order and admission — the event is derived from the
committed result and a send failure never rolls back a delete.
"No error" is not enough to prove a mutation: the disk layer skips an
absent version and reports success, so a batch entry for a version that
was already gone came back indistinguishable from a committed delete.
The delete plan already resolves whether the source exists, so carry that
`source_missing` result on `DeletedObject` and let the lifecycle path
stay silent for versions it did not remove.
backlog#2202
* fix(ecstore): persist merged checksum type for full-object multipart
complete_multipart_upload built the object-level checksum record from a
ChecksumType copied before the MULTIPART / INCLUDES_MULTIPART flags were
merged in. ChecksumType::merge takes &mut self, so the merge updated the
local variable while the copy already inside the Checksum struct stayed
behind. The composite branch rebuilt the Checksum from the merged type
and was unaffected; the full-object branch never rebuilt it, so those
flags never reached disk.
rustfs_rio::read_checksums only sets its multipart flag and only emits
the "x-amz-checksum-type" = "FULL_OBJECT" entry inside its MULTIPART
branch, so a full-object multipart object read back as non-multipart with
no type entry, and GetObject and HeadObject answered with no
x-amz-checksum-type header at all where AWS returns FULL_OBJECT.
Hand the full-object branch the merged type instead of rebuilding the
Checksum: the value must stay the running merge produced by add_part,
because hashing the concatenated part digests would yield the COMPOSITE
value, a different number than the one the client sent. The serialization
now lives in multipart_object_checksum_record so both shapes are covered
by unit tests.
Records written by earlier builds carry the bare algorithm type with no
MULTIPART flags and no trailing part block; they keep reading back to the
same checksum value, and the FULL_OBJECT reader arm predates this change
so older peers parse the new record shape correctly too.
Found while root-causing rustfs#6825.
* fix(s3): reject contradicting multipart checksum type as client error
A CompleteMultipartUpload declaring an x-amz-checksum-type that
contradicts the type recorded at CreateMultipartUpload answered 500
InternalError, telling the caller to retry a request that can only ever
fail. The storage layer does refuse the combination, but through a
generic error that maps to InternalError.
Validate the header against the recorded type in the usecase, where the
upload metadata returned by get_multipart_info is already in hand, and
answer InvalidRequest naming both types, matching AWS. The storage-layer
check stays as a backstop for non-HTTP callers.
Uploads created without a checksum algorithm record no type, so there is
nothing to contradict and the header is left alone rather than newly
rejected. Replication is unaffected: replication_put_object_options
already excludes x-amz-checksum-type from the metadata it forwards.
* test(e2e): cover full-object multipart checksum type round-trip
Adds an end-to-end test that a CRC32 FULL_OBJECT multipart upload reports
x-amz-checksum-type: FULL_OBJECT and the unsuffixed full-object value on
both GetObject and HeadObject, and one that a CompleteMultipartUpload
contradicting the recorded type is rejected as InvalidRequest while
leaving the upload intact. Extends the existing CRC64NVME multipart test
with the same checksum-type assertion.
* fix(s3): keep checksum-type validation off the s3s error macro
The s3s footprint ratchet (scripts/check_s3s_footprint.sh) counts
s3_error! invocation lines and is lower-only: new code must route
through the gateway abstractions rather than widen the direct s3s
surface the s3gate migration is shrinking.
Raise the contradiction through ApiError::invalid_request instead. The
response is byte-for-byte identical -- From<ApiError> for S3Error carries
the InvalidRequest code and the message through unchanged -- and the
usecase already returns ApiError elsewhere, so this is the idiomatic
path rather than a way around the counter.
The explanatory comment deliberately says "the s3s error macro" instead
of naming the macro: the ratchet counts raw matches, so spelling it out
in a comment tripped the same check.
* fix(ilm): enqueue committed tier free versions
* fix(ilm): stabilize causal cleanup CI coverage
* test(ilm): make expire GET race deterministic
* test(ilm): synchronize expiry with active GET
Six set_disk::ops tests failed non-deterministically only under
concurrent full-suite load, rotating between runs while each passed in
isolation. All six share one root cause: a lock-owning put_object
quorum-acks once the rename fanout reaches write quorum and lets a
detached tail task finish the lagging disks, so a fixture that inspects
per-disk state immediately after PUT can observe a disk the tail has not
reached yet.
The two heal report fixtures, the inline-commit fixture, and the
transaction-fencing fixture read or delete physical shards right after
PUT, and hit FileNotFound on a lagging disk. The two metadata-cache
fixtures prime the cache after PUT, and the read fanout refuses to publish
a cache entry while any disk still reports an error, so the priming read
observably published nothing.
Keep every affected setup PUT on the full-fanout commit path with
no_lock: true, following the existing precedent in this module, so PUT
returns only after every disk has committed. The option only governs lock
acquisition, so it does not weaken what any of these fixtures assert; the
transaction-fencing gate in particular is driven by the fleet proof and
env vars, never by the lock option. Where a fixture also depends on cache
publication, re-prime until the current generation is observably cached
instead of asserting on a single read that a loaded host can stall past
the cache TTL. The heal race fixture's shard damage injection is
best-effort by construction, so it now skips injection when the previous
round's tail still lags rather than unwrapping a read that may
legitimately race.
No production code changes, and no retries or sleeps added.
test(ecstore): assert inline fanout gate on deterministic scheduled metric
non_inline_data_read_early_stop_does_not_add_inline_fanout_on_unequal_layout
compared disk_call_counters::KIND_READ_VERSION totals between the two-phase
read-plan gate being off and on. That counter records inside each spawned
fanout task, so the single-pending inline hedge read races the early-stop
abort_all(): whether the hedge task gets its first poll before cancellation
decides a 4-vs-5 count per read. Under concurrent nextest load the two reads
can disagree (reproduced locally at ~5% when run beside one other test,
matching the CI failure on PR #6961).
Assert on the rustfs_io_get_object_metadata_fanout_scheduled histogram
instead, which records the scheduling decision synchronously in the fanout
loop and is deterministic, using the CapturingRecorder + current-thread
runtime pattern already used by the neighboring tests in this module.
A replicated version purge reaches the peer without the governance
bypass header, so a GOVERNANCE-retained version deleted on the source
with x-amz-bypass-governance-retention was rejected by the peer's WORM
deletion gate forever: retryStats ended at a permanent failed count and
the sites stayed diverged (issue #6850).
The source is authoritative for such a purge: the same WORM gate
already ran there, and GOVERNANCE retention with an authorized bypass
is the only lock state it can purge through. The peer's commit-time
deletion gate now treats an authorized replication delete addressed to
an explicit version as carrying that judged bypass, reusing the same
trust judgment as the replication write exemption
(ObjectOptions::replication_request, set only after the handler
authorized ReplicateDeleteAction). COMPLIANCE retention and legal hold
keep blocking replicated purges, and a plain client delete without the
bypass header stays rejected.
fix(ecstore): require verification source for degraded GET
Fail closed when reconstruction has only an exact decode quorum, because no surplus source remains to validate the rebuilt data. Cover both erasure engines and the data-shards-only rollout gate.
Co-authored-by: heihutu <heihutu@gmail.com>
fix(ecstore): hedge data-only GET with parity
Route the opt-in data-shards-only lockstep path through the bounded parity race and preserve deferred parity reserves across canceled hedges.
Co-authored-by: heihutu <heihutu@gmail.com>
Also replace deprecated Atomic::fetch_update calls with try_update so the
current Rust toolchain keeps lint and CI jobs warning-clean.
Co-authored-by: heihutu <heihutu@gmail.com>
A PUT with HTTP preconditions took the per-object namespace write lock
before ingesting the request body and held it until commit, so any
concurrent read of the same object queued behind client-paced body
ingestion until the 5s acquire timeout and surfaced as 503. Exposed as
a deterministic S3 Implemented Tests gate failure when #6770 routed
1 MB conditional writes onto the streaming path (rustfs/backlog#2074).
Keep a lock-free advisory precondition check before the body for fast
412/404, and evaluate the authoritative check under the put_object
commit lock, reusing the deferred shape data movement already uses.
Reads during ingestion now return the last committed version, and a
precondition invalidated mid-stream fails closed with 412 at commit.
* fix(s3): round-trip null-version delete-marker identity through listing and delete responses
On a versioning-suspended bucket, a null delete marker's identity was lost on the way back to the client at three points (issue #6745): ListObjectVersions advertised the marker's VersionId as the literal nil UUID instead of null; deleting by that id succeeded but the DeleteObjects/DeleteObject response reported the identity as null with no way to correlate it to the request; and the response lacked DeleteMarker/DeleteMarkerVersionId because the marker-ness comparison mixed the client-facing identity (Some(nil)) with the storage identity (None), so the removal also mis-recorded accounting and fired DeleteMarkerCreated semantics on later paths.
- Listing (bucket_usecase, s3_api/bucket, build_list_versions_next_marker) now maps the synthesized nil UUID to the literal null everywhere it reaches the wire, and VersionMarker::parse folds a nil-UUID marker from older listings into VersionMarker::Null so pagination resumes correctly.
- delete_objects normalizes both sides of the marker-ness comparison via delete_file_info_version_id (matching the adjacent explicit_delete_marker admission check) and reports DeleteMarkerVersionId as null for an explicit null-marker removal.
- resolve_delete_version_state reports delete_marker for an explicit-version delete whose target is a delete marker even when the bucket is versioning-suspended, fixing x-amz-delete-marker on the single-object path.
- The DeleteObjects response entry echoes the version identity the request addressed for marker removals, marker-removal accounting no longer records a marker creation, and notification events fire DeleteMarkerCreated only for actual marker creation.
Fixes#6745
* fix(s3): keep null-marker removal write shape undeleted and report marker semantics response-side
The first cut marked the storage delete request deleted for a null-marker removal, which FileMeta::delete_version interprets as the suspended-bucket delete-mints-a-marker write and re-creates the marker just removed. Carry marker-ness to responses via explicit_delete_removed_marker (single path) and a response-only branch flag (batch path) instead, keeping every storage write shape byte-identical to the pre-fix behavior. Adds an embedded end-to-end regression test covering the full issue #6745 round trip.
* refactor(ecstore,rustfs): reuse canonical starts_with_ignore_ascii_case
`crates/utils/src/http/metadata_compat.rs` owns the internal metadata key helpers, including `starts_with_ignore_ascii_case`. Two files carried their own byte-identical copies of that predicate: `SetDisks::starts_with_ignore_ascii_case` in ecstore and a free function in the S3 options layer. Both drive internal metadata key classification (`internal_metadata_suffix` and quorum hashing on one side, `should_skip_object_metadata_key` and `is_reserved_user_metadata_key` on the other), so keeping three implementations of one predicate is an avoidable drift risk on a path that decides whether an internal key is treated as user metadata.
Delete both local copies and call the canonical implementation. Every prefix used at these call sites is an ASCII constant or literal, where the canonical byte-slice comparison and the removed `str::get(..n)` form are equivalent; that equivalence was checked differentially over 4.6M (key, prefix) pairs, including keys with multi-byte characters straddling the prefix boundary. No other logic in `internal_metadata_suffix` or `should_skip_object_metadata_key` changed.
Add regression tests on both sides pinning the two properties the switch depends on: internal prefixes match case-insensitively (a mixed-case `X-RustFS-Internal-*` key stays internal), and keys shorter than a prefix never match (they stay ordinary user metadata).
Refs rustfs/backlog#2051
* fix(rustfs): avoid typos-checker false positive in prefix-length test
The test literal "x-rustfs-encryptio" (a deliberate truncation of the
x-rustfs-encryption- prefix, used to assert that a key shorter than every
internal prefix falls through to user metadata) reads as a likely typo of
"encryption" to the repo's typos CI check. Derive it from
RUSTFS_ENCRYPTION_PREFIX via slicing instead of a hand-typed literal, which
both satisfies the linter and ties the truncation to the real constant
instead of a copy-typed guess.
Refs rustfs/backlog#2051
* Revert "perf(ecstore): use AHashMap for FileInfo metadata fields (#6738)"
This reverts commit 13a2ae212e.
* fix(filemeta): restore standard HashMap metadata (#6742)
Remove the direct ahash dependency added for FileInfo metadata and revert the affected filemeta/ecstore call sites back to std::collections::HashMap.
Co-authored-by: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Zhengchao An