refactor(ecstore): make client base64 standard everywhere, drop the split
Self-review of the transition-client fixes: the Content-MD5 fix had only
converted the two transition call sites to a parallel base64_encode_standard,
leaving the same URL-safe, unpadded base64 bug on every other outbound value —
the SigV2/no-length multipart Content-MD5, the x-amz-checksum-* headers on the
parallel and no-length paths, api_remove's multi-delete Content-MD5, and the
checksum encode/decode helpers. Every base64 value this client emits or parses
is S3 wire format, which is standard base64; none is ever URL-safe.
Encode and decode both use base64_simd::STANDARD now, and the parallel
base64_encode_standard is gone. This fixes those seven latent call sites at the
root and removes the duplicate function. The transition path is functionally
unchanged (it already produced standard base64 via the helper), so the
end-to-end byte-for-byte result is preserved.
Also drop a stray Vec::with_capacity(part_size) that was allocated (up to
128 MiB) and immediately overwritten by read_multipart_part's own buffer.
Refs: rustfs/rustfs#4811, rustfs/backlog#1267
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(ecstore): treat ChecksumNone as unset so >128 MiB ILM transitions succeed
ILM transition of any object larger than 128 MiB to a RustFS-native tier
(rustfs/minio/aliyun/tencent/r2/azure/huaweicloud/s3 backends that use the
built-in TransitionClient) failed with "unsupported checksum type", while
objects <=128 MiB transitioned fine.
Root cause: `ChecksumMode::is_set()` reported `ChecksumNone` as a configured
checksum. `ChecksumNone` is the zeroth enum variant, so it occupies bit 0 of
the EnumSet repr and the `len() == 1` check treated "no checksum" as set. The
128 MiB boundary is the warm backend's `MIN_PART_SIZE`, which selects a single
PUT (<=128 MiB) versus a multipart PUT (>128 MiB). On the multipart path,
`put_object_multipart_stream_optional_checksum` saw `checksum.is_set() == true`,
disabled the Content-MD5 branch, and called `ChecksumNone.hasher()`, which
returns the "unsupported checksum type" error. The single-PUT path hit the same
misjudgement but never calls `hasher()`, so it silently succeeded (without a
checksum), which is why only >128 MiB objects failed.
Fix:
- `is_set()` returns false for `ChecksumNone` (and the bare `ChecksumFullObject`
flag, which has no base algorithm). This is the sole callers' intended
meaning: a concrete algorithm with a real hasher is selected.
- Defense in depth: guard the multipart checksum branch on
`auto_checksum.is_set()` so an unset mode uploads the part without a per-part
checksum header instead of hard-failing in `hasher()`.
Only the TransitionClient consumes this `ChecksumMode::is_set()`; the
server-side data path uses the unrelated `rustfs_rio::ChecksumType`.
Tests: is_set()/set_default semantics, hasher parity for every set mode, and a
`build_transition_put_options` invariant (checksum unset + Content-MD5 on).
Refs: rustfs/rustfs#4811, rustfs/backlog#1267
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): read exactly one part per multipart chunk in transition uploads
Second defect behind the >128 MiB ILM transition failure (rustfs/rustfs#4811),
uncovered while verifying the checksum fix.
`put_object_multipart_stream_optional_checksum` read each part with
`read_all()` / `to_vec()`, which drained the entire source into the first part
and left every later part empty. Any multipart upload of a streamed
(`ObjectBody`) source was therefore malformed. Objects <=128 MiB take the
single-part path and were unaffected; a 128 MiB + 1 byte object splits into a
128 MiB part plus a 1 byte part, so the first part received the whole object and
its declared Content-Length (part_size) did not match the body.
Verified empirically: `optimal_part_info(128 MiB + 1, 128 MiB)` yields 2 parts,
and `GetObjectReader::read_all()` on part 1 returns the full 134217729 bytes,
leaving 0 for part 2.
Fix:
- Add `read_multipart_part`, which reads exactly the requested part size (or
less at EOF) and advances the reader, for both `Body` (in-memory) and
`ObjectBody` (streamed) sources.
- Upload each part with the bytes actually read (`length`) as its size, and
account uploaded size by actual bytes, so a short read is detected instead of
masked.
The concurrent (`put_object_multipart_stream_parallel`) and SigV2
(`put_object_multipart`) paths share the same `read_all()` pattern but are not
exercised by transition; left untouched here and noted for follow-up.
Tests: `read_multipart_part` splits a 250-byte source into [100, 100, 50] for
both streamed and in-memory bodies, consumes the source fully, and stops at EOF
without overrun.
Refs: rustfs/rustfs#4811, rustfs/backlog#1267
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): complete the >128 MiB ILM transition multipart client
Docker end-to-end reproduction of rustfs/rustfs#4811 (two RustFS tiers, a
128 MiB + 1 byte object, zero-day transition) surfaced four more defects on the
multipart transition path, each masked by the previous one. With the checksum
and part-splitting fixes in place the transition now failed later and later,
and finally produced a 0-byte object with no error at all. Fixed together:
- initiate_multipart_upload discarded the CreateMultipartUpload response and
returned an empty UploadId, so the first UploadPart failed with "UploadID
cannot be empty". Parse the response XML (InitiateMultipartUploadResult now
derives Deserialize with PascalCase).
- Content-MD5 / x-amz-checksum-* were encoded with URL-safe, unpadded base64,
which the remote rejected as "Invalid content MD5: Base64Error". Add
base64_encode_standard and use it for those outbound header values.
- PutObjectOptions::default() set legalhold to OFF, so header() attached
x-amz-object-lock-legal-hold to every request and CompleteMultipartUpload was
rejected with "does not accept object lock or governance bypass headers".
Default to an empty (unset) status.
- CompleteMultipartUpload / CompletePart had no serde renames, so the request
body used Rust field names (<parts>/<part_num>/<etag>). The remote parsed
zero <Part> elements and completed a 0-byte object while returning 200. Emit
S3 element names (<Part>/<PartNumber>/<ETag>) and skip empty checksum fields.
Verified end-to-end: a 128 MiB + 1 byte object now transitions to the remote
tier and reads back (transparently restored) byte-for-byte identical
(sha256 match), with none of the four prior errors in the logs.
Refs: rustfs/rustfs#4811, rustfs/backlog#1267
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* feat(rio): wire XXHash3/64/128 and SHA-512 into ChecksumType (S2)
Add the AWS 2026-04 additional checksum algorithms as base types in
rustfs-rio's ChecksumType, covering every dispatch site (key, raw_byte_len,
hasher, Display, from_string_with_obj_type, BASE_CHECKSUM_TYPES) so no path
silently strips them. Derive BASE_TYPE_MASK from BASE_CHECKSUM_TYPES as the
single source of truth, allocate the new base-type bits append-only above
bit 9 to preserve the on-disk varint format, and add streaming hashers whose
digest uses the S3 canonical big-endian encoding (seed 0).
The new algorithms are COMPOSITE-only: an explicit FULL_OBJECT request is
rejected and they are never routed through add_part()/can_merge(). A
round-trip guardrail test asserts every base type survives all dispatch
sites, failing loudly if a future algorithm is added but a match arm or the
mask is forgotten.
Refs rustfs/backlog#1254 rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(rio): pin XXHash/SHA-512 digests to official vectors, big-endian (S3)
Lock the byte order and seed of the new algorithms against the OFFICIAL
upstream xxHash / SHA-512 empty-input test vectors (XXH3-64, XXH64, XXH3-128,
SHA-512), in big-endian, so the stored and echoed checksum is byte-for-byte
identical to what AWS SDKs (awscrt) compute — the interop correctness this
feature hinges on. Add a non-empty regression lock (official "fox" vectors)
that also asserts the encoded field is the standard-base64 of the raw digest.
Refs rustfs/backlog#1255rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(rio): lock on-disk checksum round-trip and forward-compat degrade (S8)
Cover the xl.meta varint (de)serialization for the new algorithms:
to_bytes() -> read_checksums() must recover the value under the Display key
for XXHASH3/64/128 and SHA512. Pin the rolling-upgrade contract that a node
reading a future, unknown base-type bit degrades safely — skips the entry and
returns without panicking or mis-decoding a length. Combined with the
append-only bit allocation from S2, this protects mixed-version clusters.
Refs rustfs/backlog#1260rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(head): echo XXHash/SHA-512 additional checksums on HeadObject (S5)
HeadObject with x-amz-checksum-mode: ENABLED now returns the XXHash3/64/128
and SHA-512 checksums that S3 stored, closing the head_object gap in #4800.
s3s HeadObjectOutput has no typed field for these, so they are emitted as raw
response headers via response.headers (the same mechanism RustFS already uses
for tagging-count), keyed by ChecksumType::key(). The existing five typed
algorithms are unchanged. Also carries the Cargo.lock update for the
xxhash-rust dependency introduced in S2.
Refs rustfs/backlog#1257rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(checksums): fail-closed on unknown checksum algorithm (S7)
A. Harden unknown/unsupported checksum algorithms to fail closed instead of
panicking. ChecksumMode::base() in the outbound S3 client
(crates/ecstore/src/client/checksum.rs) previously did
`panic!("enum err.")` for any mode without a concrete base algorithm (e.g.
a bare ChecksumFullObject flag); it now falls back to ChecksumNone. Added
unit tests proving base() never panics and hasher() returns Err for
unsupported modes. rustfs-checksums FromStr already returns Err on unknown
names; added a regression test asserting garbage/unknown names fail closed.
B. Extend rustfs-checksums ChecksumAlgorithm with the AWS 2026-04 additional
algorithms Sha512/Xxhash3/Xxhash64/Xxhash128. Updated FromStr, as_str,
into_impl, name constants, the x-amz-checksum-* header constants and the
HttpChecksum impls. Byte order/seed matches the server-side rustfs-rio
spec: xxh3/xxh64 as u64 big-endian (8 bytes, seed 0), xxh128 as u128
big-endian (16 bytes), sha512 via sha2::Sha512. Added tests validating each
digest against a direct library computation. MD5 stays intentionally
rejected (PR #4513) and is left untouched.
C. crates/ecstore/src/client/checksum.rs ChecksumMode is enumset repr="u8"
with 7 variants already consuming 7 bits; adding the 4 new algorithms would
overflow u8 and require a breaking repr change, so ChecksumMode is left
unchanged. The new algorithms are available through the rustfs-checksums
ChecksumAlgorithm path.
Refs rustfs/backlog#1259rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(get,put): echo XXHash/SHA-512 checksums on GetObject and PutObject (S5-GET, S4)
Complete the additional-checksum round-trip so AWS SDKs can verify integrity on
download and confirm it on upload:
- GetObject with x-amz-checksum-mode: ENABLED now returns XXHash3/64/128 and
SHA-512 checksums (the download-side path SDKs auto-verify). The values flow
from build_get_object_checksums through GetObjectOutputContext into
finalize_get_object_response and are emitted after wrap_response_with_cors.
- PutObject echoes the server-computed additional checksum on its response,
captured at the want_checksum set points before opts is moved.
Both reuse a single centralized helper, inject_additional_checksum_headers,
which HeadObject now also uses. This is the ONLY place that emits these headers,
so when s3s gains typed fields for these algorithms the migration is one spot
(fill the typed field, drop the insert) with no risk of duplicate headers.
The five s3s-typed algorithms are unchanged. Trailing-checksum PUT echo (value
lands after the body) is left for e2e coverage in S10.
Refs rustfs/backlog#1257rustfs/backlog#1256rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(multipart): support XXHash/SHA-512 composite multipart checksums (S9)
Make multipart uploads work end-to-end for the composite-only algorithms
(XXHash3/64/128, SHA-512):
- complete_part_checksum previously returned the outer None for any algorithm
outside the five typed ones, which failed CompleteMultipartUpload with
InvalidPart. It now accepts any valid base type with no double-check value
(Some(None)) — mirroring the missing-value path of the typed algorithms —
since s3s CompletePart has no field to carry a client-supplied per-part
value and the part was already verified server-side at UploadPart. Genuinely
unset/invalid types are still rejected.
- The existing COMPOSITE assembly (Checksum::new_from_data over the
concatenated per-part raw digests; full_object_requested() is false so
add_part() is correctly bypassed) already works for these algorithms via the
S2 wiring. A rio test locks the assembly and that add_part refuses them.
- UploadPart and CompleteMultipartUpload echo the new-algorithm checksum on
their responses via the shared inject_additional_checksum_headers helper
(now pub(crate)), since s3s has no typed output field.
Refs rustfs/backlog#1261rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(rio): add MD5 as an additional checksum (x-amz-checksum-md5) (S6)
Wire MD5 into ChecksumType as an additional (flexible) checksum, distinct from
the legacy Content-MD5 / ETag path: header x-amz-checksum-md5, 16-byte digest,
COMPOSITE-only, md-5 hasher. Pinned to the official empty-input MD5 vector.
Thanks to the single-source-of-truth wiring from S2, every dispatch site
(GetObject/HeadObject/PutObject echo, multipart complete_part_checksum and the
COMPOSITE assembly) picks MD5 up automatically via base()/key()/the catch-all
arm — no handler changes needed. Tests are extended to cover MD5 across them.
Coordination with #4513: that PR made the OUTBOUND rustfs-checksums client
reject "md5" so it could never silently fall back to CRC32. This change is on
the server-side rio path and never falls back — it implements MD5 correctly
rather than substituting another algorithm — so the #4513 intent is preserved,
and the outbound client keeps rejecting md5 (S7).
Refs rustfs/backlog#1258rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(rio): drop per-request to_uppercase alloc in checksum parsing (S11)
from_string_with_obj_type ran alg.to_uppercase() on every checksummed request,
allocating a String just to compare against a fixed set of algorithm names.
Replace it with eq_ignore_ascii_case, which is allocation-free and, for the
ASCII algorithm names involved, exactly equivalent. A test locks that
case-insensitivity, the CRC64NVME full-object assumption, composite-only
FULL_OBJECT rejection, and unknown/empty handling are all unchanged.
The other S11 notes are intentionally not acted on: the Phase-0 header scan is
N/A (we chose full support over rejection, so there is no reject guard), and
parallelizing the serialized hash passes is deferred pending a measured need.
Refs rustfs/backlog#1263rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* refactor(checksums): collapse 5 duplicated response-checksum loops into one
Review of the accumulated commits found the same "iterate decrypted checksums,
match five typed algorithms, drop the rest" loop copy-pasted across five
response paths (GetObject, HeadObject, GetObjectAttributes object-level and
part-level, CompleteMultipartUpload). That was patch-on-patch duplication.
Collapse it into a single source of truth:
- rustfs-rio gains ChecksumType::is_s3s_typed() — the one place that defines the
five-typed vs additional-algorithm split.
- object_usecase gains ResponseChecksums + classify_response_checksums(), which
performs the typed/extra split once. All five call sites now destructure its
result; additional_checksum_echo_pairs() also uses is_s3s_typed() instead of a
hand-rolled five-way comparison.
Behaviour is unchanged (GetObjectAttributes still cannot surface the additional
algorithms — an s3s XML-body limitation, now documented in one spot). One pass
over the map; extra pairs pushed only when a new-algorithm checksum is present.
Refs rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(checksums): unit tests for classifier/echo helpers + fix unused import
Add direct unit tests for the refactored single-source-of-truth helpers:
- rio ChecksumType::is_s3s_typed() — exhaustive typed-vs-additional split, and
that flags (FULL_OBJECT/MULTIPART) on a base type don't change classification.
- object_usecase classify_response_checksums() — typed fields vs `extra` headers,
the checksum-type marker, and empty input.
- additional_checksum_echo_pairs() — echo pair only for additional algorithms,
none for the five typed ones, none for None.
- inject_additional_checksum_headers() — writes all pairs; empty is a no-op.
Also drop the now-unused AMZ_CHECKSUM_TYPE import in multipart_usecase.rs left
by the classifier refactor (would fail the -D warnings gate).
Refs rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* style(rio): fix typo flagged by CI (mis-decoding -> decoding a wrong length)
The Typos CI check flagged "mis-decoding" (it reads "mis" as a word). Reword
the S8 forward-compat comment; no code change.
Refs rustfs/backlog#1260
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(e2e): integration test for XXHash/SHA-512/MD5 additional checksums (S10)
Permanent verify-on-write integration test in the e2e suite for the AWS 2026-04
additional algorithms. aws_sdk_s3 has no typed builder for these, so the
x-amz-checksum-<algo> header is injected via mutate_request (value from
rustfs-rio, byte-for-byte identical to awscrt). Uses a client with automatic
checksum calculation disabled (request_checksum_calculation=WhenRequired) so the
injected header is the only checksum on the wire. For each of XXHash3/64/128,
SHA-512 and MD5: a correct value is accepted and the object stored intact; a
mismatched value is rejected with BadDigest and nothing is stored.
Verified passing locally (1 passed) alongside a boto3+awscrt round-trip that
additionally confirms the HEAD/GET header echo (14/14).
Refs rustfs/backlog#1262rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* style(get): allow too_many_arguments on finalize_get_object_response
The classifier refactor added an extra_checksum_headers parameter, pushing
finalize_get_object_response to 8 args and tripping clippy::too_many_arguments
under CI's `-D warnings`. Add the same #[allow] the sibling GET helpers already
carry; no behavior change.
Refs rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* feat(object-data-cache): close write/delete-side invalidation gaps
The object data cache exposed only a single per-(bucket,object)
invalidation primitive and no write-side ecstore hook, so several
delete paths left dead bodies resident until TTL (hygiene/capacity, not
stale-serving: lookups follow a fresh metadata quorum and cannot serve a
gone object). This adds the missing primitives and wires them in.
ODC-26 (backlog#1131): add an `ObjectMutationHook` trait beside the GET
body hook, registered next to it at startup, and call it from the
ecstore-internal delete paths (`apply_expiry_on_non_transitioned_objects`,
`expire_transitioned_object` including the restored-copy branch, and
`delete_object_versions`). The app impl is one `invalidate_object` call
under a new `AfterLifecycleExpiry` reason.
ODC-27 (backlog#1132): force prefix delete now invalidates the whole
prefix, not just the prefix string. `store.delete_object(delete_prefix)`
returns no deleted-name list, so this uses a new prefix primitive rather
than the batch path.
ODC-28 (backlog#1133): DeleteBucket now flushes the bucket via a new
bucket-scope primitive (covers force and non-force, which share the
delete_bucket call).
ODC-C2 (backlog#1143): add `ObjectDataCache::clear()` and two admin
handlers (GET stats, POST flush) routed through admin runtime_sources.
The starshard identity index gains a single `remove_matching` full-scan
API backing prefix/bucket/clear; it is documented as admin/delete-path
only and never runs on the GET or fill hot path. New invalidation
reasons and metric labels added; outcome (removed/noop) labelling kept
correct for every new primitive.
Also fixes a pre-existing broken intra-doc link in memory.rs.
Co-Authored-By: heihutu <heihutu@gmail.com>
* refactor(ecstore): extract the shared HookSlot behind both cache hooks
This PR introduced object_mutation_hook.rs by mirroring body_cache_hook.rs,
which left two process-global registration slots whose register/get/clear
bodies were line-for-line identical except the trait type and the WARN string:
a RwLock<Option<Arc<dyn _>>>, an Arc::ptr_eq "different instance" warning, the
poison-recovery closure, and the same read-lock-and-clone read. Two copies of
the same swap-vs-warn logic can drift apart under maintenance.
Hoist it into a generic HookSlot<T: ?Sized> that owns the logic once. Each hook
module keeps its `static HOOK: HookSlot<dyn XxxHook>` and its thin, unchanged
public wrappers (register_/get_/clear_), so the crate's public surface and
every call site are untouched — this is an internal consolidation, not a
contract change.
The load-bearing #1126 guarantee (newest registration wins, so a rebuilt
AppContext is never stranded on a first-wins slot) previously had no direct
test — the hook tests only covered register-then-notify. HookSlot now has its
own unit tests including re_registration_swaps_to_the_latest_instance;
mutation-testing confirms a first-wins regression fails exactly that test.
No behavior change: the two hooks' existing tests, the P0 body_cache_hook_e2e
regressions, and the app-layer mutation-hook tests all pass unchanged.
Refs: backlog#1126, backlog#1131
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(admin): register the object-data-cache routes in the policy inventory
This PR added GET /object-data-cache/stats and POST /object-data-cache/flush
but did not list them in the two registries that must account for every
admin route: the route-policy inventory (route_policy.rs) and the route
matrix (route_registration_test.rs). Their coverage tests —
route_policy_inventory_covers_registered_routes and
test_admin_route_matrix_matches_registered_routes — failed on CI because a
registered route had no policy/matrix entry.
These two tests are not part of `make pre-commit` (which runs fmt + arch +
quick-check, not the full suite), so the gap passed local pre-commit and
only surfaced in the CI Test-and-Lint lane.
stats is a read (ServerInfoAdminAction, Sensitive); flush mutates
(ConfigUpdateAdminAction, High) — matching the actions the handlers already
enforce. The MinIO-alias matrix test is unaffected: these are native rustfs
endpoints with no MinIO equivalent.
Refs: backlog#1143
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
fix(object-data-cache): make GET body cache key write-unique and dedup lookups
Address four object-data-cache GET-path findings (backlog#1107 batch):
ODC-06 (backlog#1111): the cache key was content-unique, not write-unique.
Extend ObjectDataCacheKey with the resolved version's modification time
(i128 unix nanoseconds, None -> 0), derived once in the shared planner so the
ecstore hook and the usecase layer produce an identical key. An unversioned
overwrite advances mod_time, so a stale node can no longer serve old bytes for
up to the TTL under an MD5 collision; etag + size stay as belt-and-braces.
ODC-16 (backlog#1121): every cacheable GET planned and looked up twice (once in
the ecstore hook, once in the usecase layer), double-counting hits, hit_bytes
and lookups. GetObjectReader now carries a GetObjectBodySource marker
(Unprobed / HookMissed / HookServed); the hook stamps it, and
build_get_object_body_with_cache serves a hook-served body directly and skips
its lookup whenever the hook already probed. One hook-served GET now records
exactly one lookup.
ODC-19 (backlog#1124): ENABLE=true with no explicit mode defaulted to HitOnly,
which never fills and keeps a permanent 0% hit rate. Default to
FillBufferedOnly, log the resolved mode at startup, and warn when HitOnly is
selected explicitly.
ODC-24 (backlog#1129): max_entry_bytes above the in-memory GET fill limits was
silently inert. Clamp the planner's size eligibility to
min(max_entry_bytes, seek-support threshold, 64 MiB buffer cap) so ineligible
sizes plan SkipTooLarge instead of being reported eligible, and warn at startup
when the excess is inert.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(rio): reject corrupted short compressed/encrypted blocks instead of panicking
DecompressReader::poll_read and DecryptReader::poll_read sliced the block
body with a fixed `[0..16]` index to read the length varint. The body length
comes from an untrusted 24-bit header field, so a corrupted/truncated block
shorter than 16 bytes made the slice panic and crash the request task — a
read-path DoS on GET of tiered/corrupted data.
Pass the whole (arbitrary-length-safe) slice to uvarint and reject a
non-positive or out-of-range length prefix with InvalidData. Adds a repro
test for each reader; all existing round-trip tests still pass.
Refs rustfs/backlog#812
* fix(utils): close SSRF bypass via IPv4-mapped IPv6 addresses
validate_outbound_ip branched on the IpAddr variant, and the V6 branch's
is_loopback/is_unicast_link_local/is_unique_local checks never inspect the
embedded IPv4 of an IPv4-mapped address (::ffff:a.b.c.d). The metadata guard
also only matched the plain V4 169.254.169.254. So ::ffff:127.0.0.1,
::ffff:10.0.0.5 and ::ffff:169.254.169.254 all passed the outbound guard,
letting an attacker reach loopback/private/metadata endpoints.
Normalize IPv4-mapped IPv6 to its embedded IPv4 (via to_ipv4_mapped, which
matches only the true mapped form) before classification. Adds reject tests
for mapped loopback/private/metadata and an allow test for public IPv6.
Refs rustfs/backlog#813
* fix(ecstore): streaming last-part loss, GCS tier Range/remove, stat_all_dirs alignment
Four confirmed data-reliability defects:
- put_object_multipart_stream: the CompleteMultipartUpload part-collection loop
used exclusive `1..total_parts_count`, dropping the final part (and collecting
zero parts for a single-part object) — silently truncating the completed object.
Extracted collect_complete_parts (1..=total_parts_count) with unit tests.
- GCS warm backend get() ignored the requested byte range, returning the whole
object for a Range GET; now applies ReadRange::segment like the other backends.
- GCS warm backend remove() was an empty stub, so deleting a tiered object left
it on GCS forever; now deletes via StorageControl (added a control-plane client),
and in_use() actually lists (prefix-scoped) instead of always returning false.
- stat_all_dirs skipped None disk slots and dropped JoinErrors, returning a
compressed, misaligned error vector; heal_object_dir then zipped it against the
full disks array and could make_volume on the WRONG disk. Now returns one
index-aligned entry per slot (None -> DiskNotFound), and heal no longer
pre-fills the drive report (which would double it). Added an alignment test.
Refs rustfs/backlog#807
* fix(kms): stop Vault backend from destroying/reviving keys on failure
Two confirmed key-safety defects in the Vault KV2 backend:
- get_key_material() 'self-healed' a decrypt or wrong-length failure by minting a
fresh random master key and overwriting the stored value. That destroys the
original key material, making every DEK ever wrapped by it permanently
undecryptable. Decryption must never mutate the stored key: both branches now
return a cryptographic_error instead. (The empty-material bootstrap path, which
only fills a never-initialized key, is intentionally left intact.)
- cancel_key_deletion() reset key_state to Enabled only in the returned response
and never persisted it, so the key stayed PendingDeletion in storage and would
still be reaped. It now writes the state back via update_key_metadata_in_storage
and fails the request if the write fails.
Adds ignored (Vault-requiring) integration tests documenting both behaviours.
The third item (VaultTransit key state only in memory -> revived as Enabled after
restart) is deferred: a fail-closed guard would break restart availability for all
transit keys; the correct fix needs a persistent metadata store + Vault integration
testing. Tracked in rustfs/backlog#808.
Refs rustfs/backlog#808
* fix(admin): clamp STS AssumeRole duration; persist ImportBucketMetadata to disk
Two confirmed admin-API defects:
- Standard AssumeRole used the raw client-supplied DurationSeconds with no upper
bound, so a caller could mint near-permanent temporary credentials. Clamp it to
the AWS/MinIO STS window [900, 43200] (with 0 -> default 3600) via a shared
clamp_assume_role_duration helper, and build the exp claim with saturating_add.
This matches the existing AssumeRoleWithWebIdentity path.
- ImportBucketMetadata only mutated an in-memory map and returned 200, silently
dropping every imported config. It now persists each non-empty config via
metadata_sys::update (which merges onto existing on-disk metadata) and returns
InternalError if a write fails. Mapping extracted to imported_configs_to_persist
with unit tests.
Refs rustfs/backlog#809
* fix(heal): enqueue displacing request in release builds
push_displacing_lower_priority folded the real enqueue call into
debug_assert_eq!(self.push(request), Accepted). In release builds
(debug_assertions off) the whole macro — including its argument — is compiled
out, so after evicting a lower-priority queued item the new high-priority
request was silently dropped and never healed. Hoist self.push(request) out of
the assertion so the side effect runs in all builds. Adds a --release regression
test.
Refs rustfs/backlog#811
* fix(iam): propagate real delete_policy backend errors instead of swallowing them
delete_policy's is_from_notify path had its error handling inverted: a real
backend failure (disk IO / insufficient quorum) evicted the cache and returned
Ok(()), reporting a phantom success while policy.json survived on disk (to be
reloaded on the next full IAM reload); NoSuchPolicy — which should be idempotent
success — returned Err. Propagate real errors and let NoSuchPolicy fall through
to the idempotent cache-evict + Ok, matching delete_user / the notification
handler in the same file. Adds a backend-error-injection regression test.
Refs rustfs/backlog#810
* fix(utils): also normalize IPv4-compatible IPv6 in the SSRF guard
The initial fix only unwrapped IPv4-mapped (::ffff:a.b.c.d) addresses; the
deprecated IPv4-compatible form (::a.b.c.d, e.g. ::127.0.0.1 / ::169.254.169.254)
still bypassed the guard. Reject pure-IPv6 specials (::, ::1, fe80::, fc00::)
first, then normalize BOTH embedded-IPv4 forms before the IPv4 rules. Adds tests
for compatible-form loopback/metadata and confirms ::1 / :: stay rejected.
Found by adversarial review of the initial fix. Refs rustfs/backlog#813
* fix(ecstore): fix the same last-part loss in the parallel streaming path
put_object_multipart_stream_parallel had the identical off-by-one
(1..total_parts_count) that truncated the last part / produced zero parts for a
single-part upload — reachable when concurrent stream parts are enabled. Reuse
collect_complete_parts, which now returns an error instead of panicking on a gap
in the parts map. Adds a missing-part error test.
Found by adversarial review of the initial fix. Refs rustfs/backlog#807
* fix(kms): local backend must preserve key material on status change
LocalKmsClient (the default KMS backend) regenerated the master key material on
enable_key/disable_key/schedule_key_deletion/cancel_key_deletion — a pure status
change. A single disable+enable cycle therefore destroyed the original key,
making every DEK ever wrapped by it permanently undecryptable (silent data loss,
no network needed). Preserve the existing material via get_key_material and
re-save with only the status changed. Adds a hermetic regression test that wraps
a DEK, cycles all four status methods, and asserts the DEK still decrypts.
Found by adversarial review of the Vault fix. Refs rustfs/backlog#808
* test(rio): cover the length-prefix guard; correct its comment
Add a DecompressReader test that feeds an unterminated length varint so uvarint
returns 0 and the new guard (not the downstream codec) produces the InvalidData
error, and reword the guard comment which overclaimed that the > len bound
prevents a reachable panic (it is belt-and-suspenders). No behavior change.
Found by adversarial review. Refs rustfs/backlog#812
* test(rio): build test block headers via vec! to satisfy clippy
The new corrupted-block tests built the header with Vec::new() + repeated push,
tripping clippy::vec_init_then_push (-D warnings in CI). Construct the fixed
header bytes with vec![] instead. No behavior change.
---------
Co-authored-by: houseme <housemecn@gmail.com>
* fix(ecstore): remove reachable panics in tiering, replication, and heal paths
- Parse x-amz-expiration leniently in tier PUT responses; any lifecycle
rule on the remote tier bucket returns an RFC1123 date that the previous
ISO8601 unwrap turned into a panic of the ILM transition worker
- Skip invalid user-metadata header values (with a warning) when building
tier and replication PUT headers instead of panicking on non-ASCII input
- Heal: tolerate absent data_dir for delete markers and remote objects
- transition_object: don't unwrap version_id on unversioned buckets when
recording partial writes for offline disks
- Admin server info: use port_or_known_default() so default-port (80/443)
endpoints don't panic is_server_resolvable
- Tier ListObjectsV2 client: decode response body with from_utf8_lossy
- walk_internal: log merge_entry_channels errors instead of dropping them
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(ecstore): fail heal explicitly when data_dir is missing
Address review feedback: unwrap_or_default() silently substituted a nil
UUID when latest metadata lacked data_dir. Delete markers and remote
objects legitimately have no data_dir and skip the data-heal block, but
for a regular object a missing data_dir means corrupt metadata — return
FileCorrupt with a descriptive log instead of building part paths under
a nil UUID directory.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
fix(ecstore): replace unwrap() with proper error handling in api_get_object_attributes
Replace unsafe unwrap() calls with proper error handling in
api_get_object_attributes.rs:
- HTTP header access now uses ok_or_else with descriptive messages
- String parsing now uses map_err with descriptive messages
- HeaderValue creation now uses expect with descriptive messages
Refs https://github.com/rustfs/backlog/issues/729
* refactor: move delete object contracts to storage api
* refactor: narrow store api compatibility exports
* refactor: route table catalog test through storage compat
houseme