refactor(ecstore): make client base64 standard everywhere, drop the split
Self-review of the transition-client fixes: the Content-MD5 fix had only
converted the two transition call sites to a parallel base64_encode_standard,
leaving the same URL-safe, unpadded base64 bug on every other outbound value —
the SigV2/no-length multipart Content-MD5, the x-amz-checksum-* headers on the
parallel and no-length paths, api_remove's multi-delete Content-MD5, and the
checksum encode/decode helpers. Every base64 value this client emits or parses
is S3 wire format, which is standard base64; none is ever URL-safe.
Encode and decode both use base64_simd::STANDARD now, and the parallel
base64_encode_standard is gone. This fixes those seven latent call sites at the
root and removes the duplicate function. The transition path is functionally
unchanged (it already produced standard base64 via the helper), so the
end-to-end byte-for-byte result is preserved.
Also drop a stray Vec::with_capacity(part_size) that was allocated (up to
128 MiB) and immediately overwritten by read_multipart_part's own buffer.
Refs: rustfs/rustfs#4811, rustfs/backlog#1267
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(ecstore): treat ChecksumNone as unset so >128 MiB ILM transitions succeed
ILM transition of any object larger than 128 MiB to a RustFS-native tier
(rustfs/minio/aliyun/tencent/r2/azure/huaweicloud/s3 backends that use the
built-in TransitionClient) failed with "unsupported checksum type", while
objects <=128 MiB transitioned fine.
Root cause: `ChecksumMode::is_set()` reported `ChecksumNone` as a configured
checksum. `ChecksumNone` is the zeroth enum variant, so it occupies bit 0 of
the EnumSet repr and the `len() == 1` check treated "no checksum" as set. The
128 MiB boundary is the warm backend's `MIN_PART_SIZE`, which selects a single
PUT (<=128 MiB) versus a multipart PUT (>128 MiB). On the multipart path,
`put_object_multipart_stream_optional_checksum` saw `checksum.is_set() == true`,
disabled the Content-MD5 branch, and called `ChecksumNone.hasher()`, which
returns the "unsupported checksum type" error. The single-PUT path hit the same
misjudgement but never calls `hasher()`, so it silently succeeded (without a
checksum), which is why only >128 MiB objects failed.
Fix:
- `is_set()` returns false for `ChecksumNone` (and the bare `ChecksumFullObject`
flag, which has no base algorithm). This is the sole callers' intended
meaning: a concrete algorithm with a real hasher is selected.
- Defense in depth: guard the multipart checksum branch on
`auto_checksum.is_set()` so an unset mode uploads the part without a per-part
checksum header instead of hard-failing in `hasher()`.
Only the TransitionClient consumes this `ChecksumMode::is_set()`; the
server-side data path uses the unrelated `rustfs_rio::ChecksumType`.
Tests: is_set()/set_default semantics, hasher parity for every set mode, and a
`build_transition_put_options` invariant (checksum unset + Content-MD5 on).
Refs: rustfs/rustfs#4811, rustfs/backlog#1267
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): read exactly one part per multipart chunk in transition uploads
Second defect behind the >128 MiB ILM transition failure (rustfs/rustfs#4811),
uncovered while verifying the checksum fix.
`put_object_multipart_stream_optional_checksum` read each part with
`read_all()` / `to_vec()`, which drained the entire source into the first part
and left every later part empty. Any multipart upload of a streamed
(`ObjectBody`) source was therefore malformed. Objects <=128 MiB take the
single-part path and were unaffected; a 128 MiB + 1 byte object splits into a
128 MiB part plus a 1 byte part, so the first part received the whole object and
its declared Content-Length (part_size) did not match the body.
Verified empirically: `optimal_part_info(128 MiB + 1, 128 MiB)` yields 2 parts,
and `GetObjectReader::read_all()` on part 1 returns the full 134217729 bytes,
leaving 0 for part 2.
Fix:
- Add `read_multipart_part`, which reads exactly the requested part size (or
less at EOF) and advances the reader, for both `Body` (in-memory) and
`ObjectBody` (streamed) sources.
- Upload each part with the bytes actually read (`length`) as its size, and
account uploaded size by actual bytes, so a short read is detected instead of
masked.
The concurrent (`put_object_multipart_stream_parallel`) and SigV2
(`put_object_multipart`) paths share the same `read_all()` pattern but are not
exercised by transition; left untouched here and noted for follow-up.
Tests: `read_multipart_part` splits a 250-byte source into [100, 100, 50] for
both streamed and in-memory bodies, consumes the source fully, and stops at EOF
without overrun.
Refs: rustfs/rustfs#4811, rustfs/backlog#1267
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): complete the >128 MiB ILM transition multipart client
Docker end-to-end reproduction of rustfs/rustfs#4811 (two RustFS tiers, a
128 MiB + 1 byte object, zero-day transition) surfaced four more defects on the
multipart transition path, each masked by the previous one. With the checksum
and part-splitting fixes in place the transition now failed later and later,
and finally produced a 0-byte object with no error at all. Fixed together:
- initiate_multipart_upload discarded the CreateMultipartUpload response and
returned an empty UploadId, so the first UploadPart failed with "UploadID
cannot be empty". Parse the response XML (InitiateMultipartUploadResult now
derives Deserialize with PascalCase).
- Content-MD5 / x-amz-checksum-* were encoded with URL-safe, unpadded base64,
which the remote rejected as "Invalid content MD5: Base64Error". Add
base64_encode_standard and use it for those outbound header values.
- PutObjectOptions::default() set legalhold to OFF, so header() attached
x-amz-object-lock-legal-hold to every request and CompleteMultipartUpload was
rejected with "does not accept object lock or governance bypass headers".
Default to an empty (unset) status.
- CompleteMultipartUpload / CompletePart had no serde renames, so the request
body used Rust field names (<parts>/<part_num>/<etag>). The remote parsed
zero <Part> elements and completed a 0-byte object while returning 200. Emit
S3 element names (<Part>/<PartNumber>/<ETag>) and skip empty checksum fields.
Verified end-to-end: a 128 MiB + 1 byte object now transitions to the remote
tier and reads back (transparently restored) byte-for-byte identical
(sha256 match), with none of the four prior errors in the logs.
Refs: rustfs/rustfs#4811, rustfs/backlog#1267
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
This commit introduces a significant reorganization of the project structure to improve maintainability and clarity.
Key changes include:
- Adjusted the directory layout for a more logical module organization.
- Removed unused crate dependencies, reducing the overall project size and potentially speeding up build times.
- Updated import paths and configuration files to reflect the structural changes.
houseme