refactor(ecstore): make client base64 standard everywhere, drop the split
Self-review of the transition-client fixes: the Content-MD5 fix had only
converted the two transition call sites to a parallel base64_encode_standard,
leaving the same URL-safe, unpadded base64 bug on every other outbound value —
the SigV2/no-length multipart Content-MD5, the x-amz-checksum-* headers on the
parallel and no-length paths, api_remove's multi-delete Content-MD5, and the
checksum encode/decode helpers. Every base64 value this client emits or parses
is S3 wire format, which is standard base64; none is ever URL-safe.
Encode and decode both use base64_simd::STANDARD now, and the parallel
base64_encode_standard is gone. This fixes those seven latent call sites at the
root and removes the duplicate function. The transition path is functionally
unchanged (it already produced standard base64 via the helper), so the
end-to-end byte-for-byte result is preserved.
Also drop a stray Vec::with_capacity(part_size) that was allocated (up to
128 MiB) and immediately overwritten by read_multipart_part's own buffer.
Refs: rustfs/rustfs#4811, rustfs/backlog#1267
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(ecstore): treat ChecksumNone as unset so >128 MiB ILM transitions succeed
ILM transition of any object larger than 128 MiB to a RustFS-native tier
(rustfs/minio/aliyun/tencent/r2/azure/huaweicloud/s3 backends that use the
built-in TransitionClient) failed with "unsupported checksum type", while
objects <=128 MiB transitioned fine.
Root cause: `ChecksumMode::is_set()` reported `ChecksumNone` as a configured
checksum. `ChecksumNone` is the zeroth enum variant, so it occupies bit 0 of
the EnumSet repr and the `len() == 1` check treated "no checksum" as set. The
128 MiB boundary is the warm backend's `MIN_PART_SIZE`, which selects a single
PUT (<=128 MiB) versus a multipart PUT (>128 MiB). On the multipart path,
`put_object_multipart_stream_optional_checksum` saw `checksum.is_set() == true`,
disabled the Content-MD5 branch, and called `ChecksumNone.hasher()`, which
returns the "unsupported checksum type" error. The single-PUT path hit the same
misjudgement but never calls `hasher()`, so it silently succeeded (without a
checksum), which is why only >128 MiB objects failed.
Fix:
- `is_set()` returns false for `ChecksumNone` (and the bare `ChecksumFullObject`
flag, which has no base algorithm). This is the sole callers' intended
meaning: a concrete algorithm with a real hasher is selected.
- Defense in depth: guard the multipart checksum branch on
`auto_checksum.is_set()` so an unset mode uploads the part without a per-part
checksum header instead of hard-failing in `hasher()`.
Only the TransitionClient consumes this `ChecksumMode::is_set()`; the
server-side data path uses the unrelated `rustfs_rio::ChecksumType`.
Tests: is_set()/set_default semantics, hasher parity for every set mode, and a
`build_transition_put_options` invariant (checksum unset + Content-MD5 on).
Refs: rustfs/rustfs#4811, rustfs/backlog#1267
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): read exactly one part per multipart chunk in transition uploads
Second defect behind the >128 MiB ILM transition failure (rustfs/rustfs#4811),
uncovered while verifying the checksum fix.
`put_object_multipart_stream_optional_checksum` read each part with
`read_all()` / `to_vec()`, which drained the entire source into the first part
and left every later part empty. Any multipart upload of a streamed
(`ObjectBody`) source was therefore malformed. Objects <=128 MiB take the
single-part path and were unaffected; a 128 MiB + 1 byte object splits into a
128 MiB part plus a 1 byte part, so the first part received the whole object and
its declared Content-Length (part_size) did not match the body.
Verified empirically: `optimal_part_info(128 MiB + 1, 128 MiB)` yields 2 parts,
and `GetObjectReader::read_all()` on part 1 returns the full 134217729 bytes,
leaving 0 for part 2.
Fix:
- Add `read_multipart_part`, which reads exactly the requested part size (or
less at EOF) and advances the reader, for both `Body` (in-memory) and
`ObjectBody` (streamed) sources.
- Upload each part with the bytes actually read (`length`) as its size, and
account uploaded size by actual bytes, so a short read is detected instead of
masked.
The concurrent (`put_object_multipart_stream_parallel`) and SigV2
(`put_object_multipart`) paths share the same `read_all()` pattern but are not
exercised by transition; left untouched here and noted for follow-up.
Tests: `read_multipart_part` splits a 250-byte source into [100, 100, 50] for
both streamed and in-memory bodies, consumes the source fully, and stops at EOF
without overrun.
Refs: rustfs/rustfs#4811, rustfs/backlog#1267
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): complete the >128 MiB ILM transition multipart client
Docker end-to-end reproduction of rustfs/rustfs#4811 (two RustFS tiers, a
128 MiB + 1 byte object, zero-day transition) surfaced four more defects on the
multipart transition path, each masked by the previous one. With the checksum
and part-splitting fixes in place the transition now failed later and later,
and finally produced a 0-byte object with no error at all. Fixed together:
- initiate_multipart_upload discarded the CreateMultipartUpload response and
returned an empty UploadId, so the first UploadPart failed with "UploadID
cannot be empty". Parse the response XML (InitiateMultipartUploadResult now
derives Deserialize with PascalCase).
- Content-MD5 / x-amz-checksum-* were encoded with URL-safe, unpadded base64,
which the remote rejected as "Invalid content MD5: Base64Error". Add
base64_encode_standard and use it for those outbound header values.
- PutObjectOptions::default() set legalhold to OFF, so header() attached
x-amz-object-lock-legal-hold to every request and CompleteMultipartUpload was
rejected with "does not accept object lock or governance bypass headers".
Default to an empty (unset) status.
- CompleteMultipartUpload / CompletePart had no serde renames, so the request
body used Rust field names (<parts>/<part_num>/<etag>). The remote parsed
zero <Part> elements and completed a 0-byte object while returning 200. Emit
S3 element names (<Part>/<PartNumber>/<ETag>) and skip empty checksum fields.
Verified end-to-end: a 128 MiB + 1 byte object now transitions to the remote
tier and reads back (transparently restored) byte-for-byte identical
(sha256 match), with none of the four prior errors in the logs.
Refs: rustfs/rustfs#4811, rustfs/backlog#1267
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(rio): reject corrupted short compressed/encrypted blocks instead of panicking
DecompressReader::poll_read and DecryptReader::poll_read sliced the block
body with a fixed `[0..16]` index to read the length varint. The body length
comes from an untrusted 24-bit header field, so a corrupted/truncated block
shorter than 16 bytes made the slice panic and crash the request task — a
read-path DoS on GET of tiered/corrupted data.
Pass the whole (arbitrary-length-safe) slice to uvarint and reject a
non-positive or out-of-range length prefix with InvalidData. Adds a repro
test for each reader; all existing round-trip tests still pass.
Refs rustfs/backlog#812
* fix(utils): close SSRF bypass via IPv4-mapped IPv6 addresses
validate_outbound_ip branched on the IpAddr variant, and the V6 branch's
is_loopback/is_unicast_link_local/is_unique_local checks never inspect the
embedded IPv4 of an IPv4-mapped address (::ffff:a.b.c.d). The metadata guard
also only matched the plain V4 169.254.169.254. So ::ffff:127.0.0.1,
::ffff:10.0.0.5 and ::ffff:169.254.169.254 all passed the outbound guard,
letting an attacker reach loopback/private/metadata endpoints.
Normalize IPv4-mapped IPv6 to its embedded IPv4 (via to_ipv4_mapped, which
matches only the true mapped form) before classification. Adds reject tests
for mapped loopback/private/metadata and an allow test for public IPv6.
Refs rustfs/backlog#813
* fix(ecstore): streaming last-part loss, GCS tier Range/remove, stat_all_dirs alignment
Four confirmed data-reliability defects:
- put_object_multipart_stream: the CompleteMultipartUpload part-collection loop
used exclusive `1..total_parts_count`, dropping the final part (and collecting
zero parts for a single-part object) — silently truncating the completed object.
Extracted collect_complete_parts (1..=total_parts_count) with unit tests.
- GCS warm backend get() ignored the requested byte range, returning the whole
object for a Range GET; now applies ReadRange::segment like the other backends.
- GCS warm backend remove() was an empty stub, so deleting a tiered object left
it on GCS forever; now deletes via StorageControl (added a control-plane client),
and in_use() actually lists (prefix-scoped) instead of always returning false.
- stat_all_dirs skipped None disk slots and dropped JoinErrors, returning a
compressed, misaligned error vector; heal_object_dir then zipped it against the
full disks array and could make_volume on the WRONG disk. Now returns one
index-aligned entry per slot (None -> DiskNotFound), and heal no longer
pre-fills the drive report (which would double it). Added an alignment test.
Refs rustfs/backlog#807
* fix(kms): stop Vault backend from destroying/reviving keys on failure
Two confirmed key-safety defects in the Vault KV2 backend:
- get_key_material() 'self-healed' a decrypt or wrong-length failure by minting a
fresh random master key and overwriting the stored value. That destroys the
original key material, making every DEK ever wrapped by it permanently
undecryptable. Decryption must never mutate the stored key: both branches now
return a cryptographic_error instead. (The empty-material bootstrap path, which
only fills a never-initialized key, is intentionally left intact.)
- cancel_key_deletion() reset key_state to Enabled only in the returned response
and never persisted it, so the key stayed PendingDeletion in storage and would
still be reaped. It now writes the state back via update_key_metadata_in_storage
and fails the request if the write fails.
Adds ignored (Vault-requiring) integration tests documenting both behaviours.
The third item (VaultTransit key state only in memory -> revived as Enabled after
restart) is deferred: a fail-closed guard would break restart availability for all
transit keys; the correct fix needs a persistent metadata store + Vault integration
testing. Tracked in rustfs/backlog#808.
Refs rustfs/backlog#808
* fix(admin): clamp STS AssumeRole duration; persist ImportBucketMetadata to disk
Two confirmed admin-API defects:
- Standard AssumeRole used the raw client-supplied DurationSeconds with no upper
bound, so a caller could mint near-permanent temporary credentials. Clamp it to
the AWS/MinIO STS window [900, 43200] (with 0 -> default 3600) via a shared
clamp_assume_role_duration helper, and build the exp claim with saturating_add.
This matches the existing AssumeRoleWithWebIdentity path.
- ImportBucketMetadata only mutated an in-memory map and returned 200, silently
dropping every imported config. It now persists each non-empty config via
metadata_sys::update (which merges onto existing on-disk metadata) and returns
InternalError if a write fails. Mapping extracted to imported_configs_to_persist
with unit tests.
Refs rustfs/backlog#809
* fix(heal): enqueue displacing request in release builds
push_displacing_lower_priority folded the real enqueue call into
debug_assert_eq!(self.push(request), Accepted). In release builds
(debug_assertions off) the whole macro — including its argument — is compiled
out, so after evicting a lower-priority queued item the new high-priority
request was silently dropped and never healed. Hoist self.push(request) out of
the assertion so the side effect runs in all builds. Adds a --release regression
test.
Refs rustfs/backlog#811
* fix(iam): propagate real delete_policy backend errors instead of swallowing them
delete_policy's is_from_notify path had its error handling inverted: a real
backend failure (disk IO / insufficient quorum) evicted the cache and returned
Ok(()), reporting a phantom success while policy.json survived on disk (to be
reloaded on the next full IAM reload); NoSuchPolicy — which should be idempotent
success — returned Err. Propagate real errors and let NoSuchPolicy fall through
to the idempotent cache-evict + Ok, matching delete_user / the notification
handler in the same file. Adds a backend-error-injection regression test.
Refs rustfs/backlog#810
* fix(utils): also normalize IPv4-compatible IPv6 in the SSRF guard
The initial fix only unwrapped IPv4-mapped (::ffff:a.b.c.d) addresses; the
deprecated IPv4-compatible form (::a.b.c.d, e.g. ::127.0.0.1 / ::169.254.169.254)
still bypassed the guard. Reject pure-IPv6 specials (::, ::1, fe80::, fc00::)
first, then normalize BOTH embedded-IPv4 forms before the IPv4 rules. Adds tests
for compatible-form loopback/metadata and confirms ::1 / :: stay rejected.
Found by adversarial review of the initial fix. Refs rustfs/backlog#813
* fix(ecstore): fix the same last-part loss in the parallel streaming path
put_object_multipart_stream_parallel had the identical off-by-one
(1..total_parts_count) that truncated the last part / produced zero parts for a
single-part upload — reachable when concurrent stream parts are enabled. Reuse
collect_complete_parts, which now returns an error instead of panicking on a gap
in the parts map. Adds a missing-part error test.
Found by adversarial review of the initial fix. Refs rustfs/backlog#807
* fix(kms): local backend must preserve key material on status change
LocalKmsClient (the default KMS backend) regenerated the master key material on
enable_key/disable_key/schedule_key_deletion/cancel_key_deletion — a pure status
change. A single disable+enable cycle therefore destroyed the original key,
making every DEK ever wrapped by it permanently undecryptable (silent data loss,
no network needed). Preserve the existing material via get_key_material and
re-save with only the status changed. Adds a hermetic regression test that wraps
a DEK, cycles all four status methods, and asserts the DEK still decrypts.
Found by adversarial review of the Vault fix. Refs rustfs/backlog#808
* test(rio): cover the length-prefix guard; correct its comment
Add a DecompressReader test that feeds an unterminated length varint so uvarint
returns 0 and the new guard (not the downstream codec) produces the InvalidData
error, and reword the guard comment which overclaimed that the > len bound
prevents a reachable panic (it is belt-and-suspenders). No behavior change.
Found by adversarial review. Refs rustfs/backlog#812
* test(rio): build test block headers via vec! to satisfy clippy
The new corrupted-block tests built the header with Vec::new() + repeated push,
tripping clippy::vec_init_then_push (-D warnings in CI). Construct the fixed
header bytes with vec![] instead. No behavior change.
---------
Co-authored-by: houseme <housemecn@gmail.com>
* fix(ecstore): remove reachable panics in tiering, replication, and heal paths
- Parse x-amz-expiration leniently in tier PUT responses; any lifecycle
rule on the remote tier bucket returns an RFC1123 date that the previous
ISO8601 unwrap turned into a panic of the ILM transition worker
- Skip invalid user-metadata header values (with a warning) when building
tier and replication PUT headers instead of panicking on non-ASCII input
- Heal: tolerate absent data_dir for delete markers and remote objects
- transition_object: don't unwrap version_id on unversioned buckets when
recording partial writes for offline disks
- Admin server info: use port_or_known_default() so default-port (80/443)
endpoints don't panic is_server_resolvable
- Tier ListObjectsV2 client: decode response body with from_utf8_lossy
- walk_internal: log merge_entry_channels errors instead of dropping them
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(ecstore): fail heal explicitly when data_dir is missing
Address review feedback: unwrap_or_default() silently substituted a nil
UUID when latest metadata lacked data_dir. Delete markers and remote
objects legitimately have no data_dir and skip the data-heal block, but
for a regular object a missing data_dir means corrupt metadata — return
FileCorrupt with a descriptive log instead of building part paths under
a nil UUID directory.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit introduces a significant reorganization of the project structure to improve maintainability and clarity.
Key changes include:
- Adjusted the directory layout for a more logical module organization.
- Removed unused crate dependencies, reducing the overall project size and potentially speeding up build times.
- Updated import paths and configuration files to reflect the structural changes.
cxymds