mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-20 19:42:17 +00:00
35f3599992f71e80be8c61a1b64c6afad741dcc3
12 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 houseme
|
35f3599992 |
fix(ecstore): fence restore final commit by operation id (#5062)
Refs rustfs/backlog#1356 Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
18f0c161dd |
fix(ecstore): harden tier reader and restore cleanup races (#5035)
* fix(tier): hold generation lease through readers Refs rustfs/backlog#1354 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(restore): fence failed cleanup by source identity Refs rustfs/backlog#1356 Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
21049401fa |
fix(ilm): harden tier transition failure boundaries (#5031)
* fix(tier): fence generation-scoped operations Refs rustfs/backlog#1354 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ilm): verify transition upload streams Refs rustfs/backlog#1353 Co-Authored-By: heihutu <heihutu@gmail.com> * test(ecstore): expand transition fault matrix Refs rustfs/backlog#1355 Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
15f4e75870 |
fix(cache): harden object data cache coordination (#5004)
* fix(cache): enforce projected entry capacity Refs: rustfs/backlog#1335 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(cache): fence identity budget eviction by generation Refs rustfs/backlog#1334. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(cache): fence clear against concurrent fills Refs rustfs/backlog#1333 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(cache): linearize memory reservation claims Co-Authored-By: heihutu <heihutu@gmail.com> * fix(cache): retain allocation memory claims Co-Authored-By: heihutu <heihutu@gmail.com> * fix(cache): publish memory snapshots by epoch Co-Authored-By: heihutu <heihutu@gmail.com> * fix(cache): coordinate cold object fills Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): fence metadata cache transition races Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
Zhengchao An
|
c818177b54 |
test(ilm): re-enable test_transition_and_restore_flows; fix test-util disk-open and restore error-path lock (#4945)
test(ilm): re-enable test_transition_and_restore_flows; fix test-util disk-open and restore error-path lock (rustfs/backlog#1303) The excluded test's 'missing xl.meta ... on disk2' was NOT an EC metadata-distribution issue: after a transition all four shard disks hold a fully consistent xl.meta (verified by decoding each shard). The panic came from the tier test util's open_disk, which hardcoded disk_index 0 for every disk path; LocalDisk::new validates the endpoint's (set_idx, disk_idx) against the disk's own format.json and rejected every non-slot-0 disk with InconsistentDisk, which read_transition_meta collapsed into 'missing xl.meta'. Derive the real indices from format.json instead. This also un-breaks free_version_count / wait_for_free_version_absence for non-first disks (silently 0 before). With that fixed, the test advanced to the #4877 restore self-deadlock, whose main paths #4886 already fixed. Complete that fix on the one path it missed: update_restore_metadata (the restore-failure metadata rewrite) still rebuilt copy_object options with no_lock=false and would re-acquire the object write lock the restore handler already holds. Propagate the caller's no_lock there too. Remove the test from the serial-lane exclusion list; the four remaining exclusions are unrelated known issues and stay. |
||
|
houseme
|
be6859be55 |
fix(ecstore): handle ChecksumNone in >128 MiB ILM transitions (#4831)
* fix(ecstore): treat ChecksumNone as unset so >128 MiB ILM transitions succeed ILM transition of any object larger than 128 MiB to a RustFS-native tier (rustfs/minio/aliyun/tencent/r2/azure/huaweicloud/s3 backends that use the built-in TransitionClient) failed with "unsupported checksum type", while objects <=128 MiB transitioned fine. Root cause: `ChecksumMode::is_set()` reported `ChecksumNone` as a configured checksum. `ChecksumNone` is the zeroth enum variant, so it occupies bit 0 of the EnumSet repr and the `len() == 1` check treated "no checksum" as set. The 128 MiB boundary is the warm backend's `MIN_PART_SIZE`, which selects a single PUT (<=128 MiB) versus a multipart PUT (>128 MiB). On the multipart path, `put_object_multipart_stream_optional_checksum` saw `checksum.is_set() == true`, disabled the Content-MD5 branch, and called `ChecksumNone.hasher()`, which returns the "unsupported checksum type" error. The single-PUT path hit the same misjudgement but never calls `hasher()`, so it silently succeeded (without a checksum), which is why only >128 MiB objects failed. Fix: - `is_set()` returns false for `ChecksumNone` (and the bare `ChecksumFullObject` flag, which has no base algorithm). This is the sole callers' intended meaning: a concrete algorithm with a real hasher is selected. - Defense in depth: guard the multipart checksum branch on `auto_checksum.is_set()` so an unset mode uploads the part without a per-part checksum header instead of hard-failing in `hasher()`. Only the TransitionClient consumes this `ChecksumMode::is_set()`; the server-side data path uses the unrelated `rustfs_rio::ChecksumType`. Tests: is_set()/set_default semantics, hasher parity for every set mode, and a `build_transition_put_options` invariant (checksum unset + Content-MD5 on). Refs: rustfs/rustfs#4811, rustfs/backlog#1267 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): read exactly one part per multipart chunk in transition uploads Second defect behind the >128 MiB ILM transition failure (rustfs/rustfs#4811), uncovered while verifying the checksum fix. `put_object_multipart_stream_optional_checksum` read each part with `read_all()` / `to_vec()`, which drained the entire source into the first part and left every later part empty. Any multipart upload of a streamed (`ObjectBody`) source was therefore malformed. Objects <=128 MiB take the single-part path and were unaffected; a 128 MiB + 1 byte object splits into a 128 MiB part plus a 1 byte part, so the first part received the whole object and its declared Content-Length (part_size) did not match the body. Verified empirically: `optimal_part_info(128 MiB + 1, 128 MiB)` yields 2 parts, and `GetObjectReader::read_all()` on part 1 returns the full 134217729 bytes, leaving 0 for part 2. Fix: - Add `read_multipart_part`, which reads exactly the requested part size (or less at EOF) and advances the reader, for both `Body` (in-memory) and `ObjectBody` (streamed) sources. - Upload each part with the bytes actually read (`length`) as its size, and account uploaded size by actual bytes, so a short read is detected instead of masked. The concurrent (`put_object_multipart_stream_parallel`) and SigV2 (`put_object_multipart`) paths share the same `read_all()` pattern but are not exercised by transition; left untouched here and noted for follow-up. Tests: `read_multipart_part` splits a 250-byte source into [100, 100, 50] for both streamed and in-memory bodies, consumes the source fully, and stops at EOF without overrun. Refs: rustfs/rustfs#4811, rustfs/backlog#1267 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): complete the >128 MiB ILM transition multipart client Docker end-to-end reproduction of rustfs/rustfs#4811 (two RustFS tiers, a 128 MiB + 1 byte object, zero-day transition) surfaced four more defects on the multipart transition path, each masked by the previous one. With the checksum and part-splitting fixes in place the transition now failed later and later, and finally produced a 0-byte object with no error at all. Fixed together: - initiate_multipart_upload discarded the CreateMultipartUpload response and returned an empty UploadId, so the first UploadPart failed with "UploadID cannot be empty". Parse the response XML (InitiateMultipartUploadResult now derives Deserialize with PascalCase). - Content-MD5 / x-amz-checksum-* were encoded with URL-safe, unpadded base64, which the remote rejected as "Invalid content MD5: Base64Error". Add base64_encode_standard and use it for those outbound header values. - PutObjectOptions::default() set legalhold to OFF, so header() attached x-amz-object-lock-legal-hold to every request and CompleteMultipartUpload was rejected with "does not accept object lock or governance bypass headers". Default to an empty (unset) status. - CompleteMultipartUpload / CompletePart had no serde renames, so the request body used Rust field names (<parts>/<part_num>/<etag>). The remote parsed zero <Part> elements and completed a 0-byte object while returning 200. Emit S3 element names (<Part>/<PartNumber>/<ETag>) and skip empty checksum fields. Verified end-to-end: a 128 MiB + 1 byte object now transitions to the remote tier and reads back (transparently restored) byte-for-byte identical (sha256 match), with none of the four prior errors in the logs. Refs: rustfs/rustfs#4811, rustfs/backlog#1267 Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
Zhengchao An
|
763f246f8a |
test(ecstore): add shared MockWarmBackend test utility for lifecycle and tier tests (#4716)
* test(ecstore): extract shared MockWarmBackend into a test-util feature (backlog#1148 ilm-6) The tier/lifecycle integration tests carried two byte-for-byte copies of an in-memory WarmBackend mock — one in crates/scanner/tests and one in rustfs/src/app — plus duplicated register_mock_tier and polling helpers. Both implemented the same ecstore WarmBackend trait. Consolidate them into ecstore behind a new `test-util` feature, exposed via the `rustfs_ecstore::api::tier::test_util` facade: - MockWarmBackend: in-memory WarmBackend with an operation log (for ordering assertions such as "local delete precedes remote remove") and fault injection (FaultConfig): unreachable, HTTP 5xx, credential rejection, injected latency, plus external_remove to simulate an out-of-band remote deletion. - register_mock_tier / register_mock_tier_backend: register the mock into any TierConfigMgr handle (the global manager used by scanner tests or a per-instance one used by the app tests). - xl.meta transition assertion helpers: read_transition_meta, assert_transition_meta_consistent (cross-shard consistency of the status/tier/remote-key/remote-version-id tuple plus free-version count), and free_version_count. - polling helpers: wait_for_remote_absence, wait_for_object_count, wait_for_free_version_absence. Both existing copies now consume this single definition; `rg 'struct MockWarmBackend'` collapses to one. The feature is enabled only from [dev-dependencies], so it never links into the production binary (resolver 3). Designed for downstream ilm-8 (restore lifecycle) and ilm-11 (tier fault injection matrix). Coordinates with #4706 (ilm-2), which adds op-logging to the scanner mock — that op-logging is now part of this shared surface, so #4706 should rebase onto it. Refs rustfs/backlog#1148 (ilm-6), rustfs/backlog#1155. * test(ecstore): fix shared MockWarmBackend usage after main merge - Access stored objects via MockWarmBackend::contains() instead of the now private inner objects map (fixes E0609 after the shared test-util refactor). - Drop dead ReadCloser/ReaderImpl/DiskAPI imports and the unused transition_api test re-exports the mock extraction left behind. - Reword the scanner/rustfs test-util dependency comments so they no longer embed the literal rustfs_ecstore:: path that trips the ECStore architecture-migration guard. |
||
|
Zhengchao An
|
05fae6f939 |
test(ecstore): add TierConfigMgr state-machine unit coverage (#4713)
* test(ecstore): unit-test TierConfigMgr add/edit/remove/verify state machine (backlog#1148 ilm-4) Covers the tier config state machine and persistence paths that previously had only 4 codec tests and none for tier_config.rs: - add: non-uppercase name, duplicate name, unsupported type, missing backend payload, and a regression anchor documenting that AWS-reserved names (STANDARD) are not currently rejected. - edit: unknown tier, missing-credentials rejection for RustFS and MinIO. - remove: idempotent unknown-tier no-op, in-use rejection, empty-backend success, force skips the in_use probe, and probe-error surfacing. - verify: unknown tier, healthy backend, unhealthy backend. - pure query helpers (empty/is_tier_valid/tier_type/get/list_tiers). - persistence: JSON marshal/unmarshal roundtrip, external tier-config.bin roundtrip for Azure and GCS payload mapping, truncated/unknown-format/ unknown-version rejection, legacy v1 version-word acceptance, and encode failure on missing payload. Tests are hermetic: error paths return before backend construction, and a MockWarmBackend injected into driver_cache exercises remove/verify without any real remote. Refs backlog#1155. Co-authored-by: overtrue <anzhengchao@gmail.com> * fix(tier): reject reserved names STANDARD/RRS in TierConfigMgr::add (backlog#1148 ilm-4) (#4721) |
||
|
Zhengchao An
|
e3a8234bc9 |
fix: 12 P1 reliability/security defects from the full-repo audit (backlog#806) (#4256)
* fix(rio): reject corrupted short compressed/encrypted blocks instead of panicking DecompressReader::poll_read and DecryptReader::poll_read sliced the block body with a fixed `[0..16]` index to read the length varint. The body length comes from an untrusted 24-bit header field, so a corrupted/truncated block shorter than 16 bytes made the slice panic and crash the request task — a read-path DoS on GET of tiered/corrupted data. Pass the whole (arbitrary-length-safe) slice to uvarint and reject a non-positive or out-of-range length prefix with InvalidData. Adds a repro test for each reader; all existing round-trip tests still pass. Refs rustfs/backlog#812 * fix(utils): close SSRF bypass via IPv4-mapped IPv6 addresses validate_outbound_ip branched on the IpAddr variant, and the V6 branch's is_loopback/is_unicast_link_local/is_unique_local checks never inspect the embedded IPv4 of an IPv4-mapped address (::ffff:a.b.c.d). The metadata guard also only matched the plain V4 169.254.169.254. So ::ffff:127.0.0.1, ::ffff:10.0.0.5 and ::ffff:169.254.169.254 all passed the outbound guard, letting an attacker reach loopback/private/metadata endpoints. Normalize IPv4-mapped IPv6 to its embedded IPv4 (via to_ipv4_mapped, which matches only the true mapped form) before classification. Adds reject tests for mapped loopback/private/metadata and an allow test for public IPv6. Refs rustfs/backlog#813 * fix(ecstore): streaming last-part loss, GCS tier Range/remove, stat_all_dirs alignment Four confirmed data-reliability defects: - put_object_multipart_stream: the CompleteMultipartUpload part-collection loop used exclusive `1..total_parts_count`, dropping the final part (and collecting zero parts for a single-part object) — silently truncating the completed object. Extracted collect_complete_parts (1..=total_parts_count) with unit tests. - GCS warm backend get() ignored the requested byte range, returning the whole object for a Range GET; now applies ReadRange::segment like the other backends. - GCS warm backend remove() was an empty stub, so deleting a tiered object left it on GCS forever; now deletes via StorageControl (added a control-plane client), and in_use() actually lists (prefix-scoped) instead of always returning false. - stat_all_dirs skipped None disk slots and dropped JoinErrors, returning a compressed, misaligned error vector; heal_object_dir then zipped it against the full disks array and could make_volume on the WRONG disk. Now returns one index-aligned entry per slot (None -> DiskNotFound), and heal no longer pre-fills the drive report (which would double it). Added an alignment test. Refs rustfs/backlog#807 * fix(kms): stop Vault backend from destroying/reviving keys on failure Two confirmed key-safety defects in the Vault KV2 backend: - get_key_material() 'self-healed' a decrypt or wrong-length failure by minting a fresh random master key and overwriting the stored value. That destroys the original key material, making every DEK ever wrapped by it permanently undecryptable. Decryption must never mutate the stored key: both branches now return a cryptographic_error instead. (The empty-material bootstrap path, which only fills a never-initialized key, is intentionally left intact.) - cancel_key_deletion() reset key_state to Enabled only in the returned response and never persisted it, so the key stayed PendingDeletion in storage and would still be reaped. It now writes the state back via update_key_metadata_in_storage and fails the request if the write fails. Adds ignored (Vault-requiring) integration tests documenting both behaviours. The third item (VaultTransit key state only in memory -> revived as Enabled after restart) is deferred: a fail-closed guard would break restart availability for all transit keys; the correct fix needs a persistent metadata store + Vault integration testing. Tracked in rustfs/backlog#808. Refs rustfs/backlog#808 * fix(admin): clamp STS AssumeRole duration; persist ImportBucketMetadata to disk Two confirmed admin-API defects: - Standard AssumeRole used the raw client-supplied DurationSeconds with no upper bound, so a caller could mint near-permanent temporary credentials. Clamp it to the AWS/MinIO STS window [900, 43200] (with 0 -> default 3600) via a shared clamp_assume_role_duration helper, and build the exp claim with saturating_add. This matches the existing AssumeRoleWithWebIdentity path. - ImportBucketMetadata only mutated an in-memory map and returned 200, silently dropping every imported config. It now persists each non-empty config via metadata_sys::update (which merges onto existing on-disk metadata) and returns InternalError if a write fails. Mapping extracted to imported_configs_to_persist with unit tests. Refs rustfs/backlog#809 * fix(heal): enqueue displacing request in release builds push_displacing_lower_priority folded the real enqueue call into debug_assert_eq!(self.push(request), Accepted). In release builds (debug_assertions off) the whole macro — including its argument — is compiled out, so after evicting a lower-priority queued item the new high-priority request was silently dropped and never healed. Hoist self.push(request) out of the assertion so the side effect runs in all builds. Adds a --release regression test. Refs rustfs/backlog#811 * fix(iam): propagate real delete_policy backend errors instead of swallowing them delete_policy's is_from_notify path had its error handling inverted: a real backend failure (disk IO / insufficient quorum) evicted the cache and returned Ok(()), reporting a phantom success while policy.json survived on disk (to be reloaded on the next full IAM reload); NoSuchPolicy — which should be idempotent success — returned Err. Propagate real errors and let NoSuchPolicy fall through to the idempotent cache-evict + Ok, matching delete_user / the notification handler in the same file. Adds a backend-error-injection regression test. Refs rustfs/backlog#810 * fix(utils): also normalize IPv4-compatible IPv6 in the SSRF guard The initial fix only unwrapped IPv4-mapped (::ffff:a.b.c.d) addresses; the deprecated IPv4-compatible form (::a.b.c.d, e.g. ::127.0.0.1 / ::169.254.169.254) still bypassed the guard. Reject pure-IPv6 specials (::, ::1, fe80::, fc00::) first, then normalize BOTH embedded-IPv4 forms before the IPv4 rules. Adds tests for compatible-form loopback/metadata and confirms ::1 / :: stay rejected. Found by adversarial review of the initial fix. Refs rustfs/backlog#813 * fix(ecstore): fix the same last-part loss in the parallel streaming path put_object_multipart_stream_parallel had the identical off-by-one (1..total_parts_count) that truncated the last part / produced zero parts for a single-part upload — reachable when concurrent stream parts are enabled. Reuse collect_complete_parts, which now returns an error instead of panicking on a gap in the parts map. Adds a missing-part error test. Found by adversarial review of the initial fix. Refs rustfs/backlog#807 * fix(kms): local backend must preserve key material on status change LocalKmsClient (the default KMS backend) regenerated the master key material on enable_key/disable_key/schedule_key_deletion/cancel_key_deletion — a pure status change. A single disable+enable cycle therefore destroyed the original key, making every DEK ever wrapped by it permanently undecryptable (silent data loss, no network needed). Preserve the existing material via get_key_material and re-save with only the status changed. Adds a hermetic regression test that wraps a DEK, cycles all four status methods, and asserts the DEK still decrypts. Found by adversarial review of the Vault fix. Refs rustfs/backlog#808 * test(rio): cover the length-prefix guard; correct its comment Add a DecompressReader test that feeds an unterminated length varint so uvarint returns 0 and the new guard (not the downstream codec) produces the InvalidData error, and reword the guard comment which overclaimed that the > len bound prevents a reachable panic (it is belt-and-suspenders). No behavior change. Found by adversarial review. Refs rustfs/backlog#812 * test(rio): build test block headers via vec! to satisfy clippy The new corrupted-block tests built the header with Vec::new() + repeated push, tripping clippy::vec_init_then_push (-D warnings in CI). Construct the fixed header bytes with vec![] instead. No behavior change. --------- Co-authored-by: houseme <housemecn@gmail.com> |
||
|
Zhengchao An
|
726bf142e0 | refactor(runtime): remove stale tier transport comments (#4050) | ||
|
Zhengchao An
|
4111ce1547 | refactor(runtime): wrap tier config global (#4045) | ||
|
Zhengchao An
|
0a5b1b1b3a |
refactor: consolidate ecstore owner module layout (#3934)
* refactor: shrink ecstore root owner facades * refactor: remove ecstore core store root shims * refactor: move ecstore erasure owner modules * refactor: remove ecstore root rpc facade * refactor: move ecstore services domain modules |