mirror of
https://github.com/rustfs/rustfs.git
synced 2026-07-26 08:18:18 +00:00
houseme
be6859be55
* fix(ecstore): treat ChecksumNone as unset so >128 MiB ILM transitions succeed ILM transition of any object larger than 128 MiB to a RustFS-native tier (rustfs/minio/aliyun/tencent/r2/azure/huaweicloud/s3 backends that use the built-in TransitionClient) failed with "unsupported checksum type", while objects <=128 MiB transitioned fine. Root cause: `ChecksumMode::is_set()` reported `ChecksumNone` as a configured checksum. `ChecksumNone` is the zeroth enum variant, so it occupies bit 0 of the EnumSet repr and the `len() == 1` check treated "no checksum" as set. The 128 MiB boundary is the warm backend's `MIN_PART_SIZE`, which selects a single PUT (<=128 MiB) versus a multipart PUT (>128 MiB). On the multipart path, `put_object_multipart_stream_optional_checksum` saw `checksum.is_set() == true`, disabled the Content-MD5 branch, and called `ChecksumNone.hasher()`, which returns the "unsupported checksum type" error. The single-PUT path hit the same misjudgement but never calls `hasher()`, so it silently succeeded (without a checksum), which is why only >128 MiB objects failed. Fix: - `is_set()` returns false for `ChecksumNone` (and the bare `ChecksumFullObject` flag, which has no base algorithm). This is the sole callers' intended meaning: a concrete algorithm with a real hasher is selected. - Defense in depth: guard the multipart checksum branch on `auto_checksum.is_set()` so an unset mode uploads the part without a per-part checksum header instead of hard-failing in `hasher()`. Only the TransitionClient consumes this `ChecksumMode::is_set()`; the server-side data path uses the unrelated `rustfs_rio::ChecksumType`. Tests: is_set()/set_default semantics, hasher parity for every set mode, and a `build_transition_put_options` invariant (checksum unset + Content-MD5 on). Refs: rustfs/rustfs#4811, rustfs/backlog#1267 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): read exactly one part per multipart chunk in transition uploads Second defect behind the >128 MiB ILM transition failure (rustfs/rustfs#4811), uncovered while verifying the checksum fix. `put_object_multipart_stream_optional_checksum` read each part with `read_all()` / `to_vec()`, which drained the entire source into the first part and left every later part empty. Any multipart upload of a streamed (`ObjectBody`) source was therefore malformed. Objects <=128 MiB take the single-part path and were unaffected; a 128 MiB + 1 byte object splits into a 128 MiB part plus a 1 byte part, so the first part received the whole object and its declared Content-Length (part_size) did not match the body. Verified empirically: `optimal_part_info(128 MiB + 1, 128 MiB)` yields 2 parts, and `GetObjectReader::read_all()` on part 1 returns the full 134217729 bytes, leaving 0 for part 2. Fix: - Add `read_multipart_part`, which reads exactly the requested part size (or less at EOF) and advances the reader, for both `Body` (in-memory) and `ObjectBody` (streamed) sources. - Upload each part with the bytes actually read (`length`) as its size, and account uploaded size by actual bytes, so a short read is detected instead of masked. The concurrent (`put_object_multipart_stream_parallel`) and SigV2 (`put_object_multipart`) paths share the same `read_all()` pattern but are not exercised by transition; left untouched here and noted for follow-up. Tests: `read_multipart_part` splits a 250-byte source into [100, 100, 50] for both streamed and in-memory bodies, consumes the source fully, and stops at EOF without overrun. Refs: rustfs/rustfs#4811, rustfs/backlog#1267 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): complete the >128 MiB ILM transition multipart client Docker end-to-end reproduction of rustfs/rustfs#4811 (two RustFS tiers, a 128 MiB + 1 byte object, zero-day transition) surfaced four more defects on the multipart transition path, each masked by the previous one. With the checksum and part-splitting fixes in place the transition now failed later and later, and finally produced a 0-byte object with no error at all. Fixed together: - initiate_multipart_upload discarded the CreateMultipartUpload response and returned an empty UploadId, so the first UploadPart failed with "UploadID cannot be empty". Parse the response XML (InitiateMultipartUploadResult now derives Deserialize with PascalCase). - Content-MD5 / x-amz-checksum-* were encoded with URL-safe, unpadded base64, which the remote rejected as "Invalid content MD5: Base64Error". Add base64_encode_standard and use it for those outbound header values. - PutObjectOptions::default() set legalhold to OFF, so header() attached x-amz-object-lock-legal-hold to every request and CompleteMultipartUpload was rejected with "does not accept object lock or governance bypass headers". Default to an empty (unset) status. - CompleteMultipartUpload / CompletePart had no serde renames, so the request body used Rust field names (<parts>/<part_num>/<etag>). The remote parsed zero <Part> elements and completed a 0-byte object while returning 200. Emit S3 element names (<Part>/<PartNumber>/<ETag>) and skip empty checksum fields. Verified end-to-end: a 128 MiB + 1 byte object now transitions to the remote tier and reads back (transparently restored) byte-for-byte identical (sha256 match), with none of the four prior errors in the logs. Refs: rustfs/rustfs#4811, rustfs/backlog#1267 Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com>