mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-15 17:43:13 +00:00
Zhengchao An
ffe889ad59
* fix(storage): restore multipart disk compression and make the legacy decompressor resumable Multipart uploads have bypassed disk compression since #5169 removed the session marker as a stopgap for mid-stream GET failures. The actual root cause was never the multipart layout: the legacy DecompressReader reset its payload consumption state on every poll re-entry, so a Poll::Pending in the middle of a block payload (routine under the erasure duplex) desynchronized the block framing and surfaced as LZ4 frameType errors. This rewrites the decoder as a resumable state machine, restores the multipart session compression marker, reports logical part sizes in ListParts, and makes the rebalance migration read raw stored bytes so compressed and encrypted objects survive migration verbatim. Fixes #5957. Internal tracking: backlog#1848, backlog#1850. * feat(storage): stage multipart compression behind RUSTFS_COMPRESSION_MULTIPART_ENABLED Review follow-up: a rolling-upgrade window must not create new compressed multipart objects while pre-fix nodes (whose decompressor is not resumable) may still serve reads. The session marker is now additionally gated on RUSTFS_COMPRESSION_MULTIPART_ENABLED, default off, so the restored capability stays dark until the operator confirms fleet convergence. The default flips per the multipart-compression-default-off-window entry in docs/architecture/compat-cleanup-register.md once the minimum supported direct-upgrade release ships the resumable decoder. * chore(compat): satisfy the cleanup-register guard for the multipart compression switch The architecture guard requires every backticked identifier in a register entry to carry a RUSTFS_COMPAT_TODO source marker: keep only the entry slug in backticks, and add the marker (with its literal Remove-after condition) at the switch definition. * chore(rio): drop a dead store in the poison guard and note the end-block branch Review follow-up: the poison gate re-assigned an already-true flag, and the COMPRESS_TYPE_END branch reads as dead without stating that the writer never emits an end block — that absence is exactly what lets concatenated per-part streams decode as one. * fix(s3): report empty compressed multipart part size * fix(s3): report empty encrypted multipart part size
Architecture Documentation
Durable architecture reference for RustFS: migration guardrails, runtime contracts, boundary rules, and support matrices.
Two rules keep this directory healthy:
- Durable reference only. One-shot implementation plans, task trackers, and PR templates do not belong in the repository — keep them in the issue tracker or your local worktree. When their work closes, delete them rather than archiving them here.
- No copies of other sources of truth. Crate lists come from
Cargo.toml, CI steps from.github/workflows/ci.yml, code structure from the code.scripts/check_doc_paths.shfails the pre-commit gate when a doc here references a file path that no longer exists.
Start here
- overview.md — migration baseline, phase order, core principles
CI-enforced core (required by scripts/check_architecture_migration_rules.sh)
- crate-boundaries.md — dependency direction, PR types, re-export contracts
- runtime-lifecycle.md — startup/shutdown sequencing, readiness guarantees
- readiness-matrix.md — request/dependency behavior, probe semantics
- storage-control-data-plane.md — storage API contracts, control-plane boundaries
- global-state-crate-split-plan.md — remaining global-state owners and split evaluation
- ecstore-module-split-plan.md — ECStore decomposition rules and facade contracts
Contracts & invariants
- erasure-coding.md — normative erasure-coding algorithm and on-disk (
xl.meta) compatibility contract; the frozen invariants for all user-data read/write, encode/decode, quorum, heal, and decode tolerance - placement-repair-invariants.md
- unified-object-generation.md — single per-object generation authority (fencing epoch, transport/encoding/proto/mixed-version contracts)
- runtime-capability-contracts.md
- workload-admission-contracts.md
- background-controller-contract.md
- config-model-boundary-adr.md
- ecstore-layout-boundary.md
- decommission-compatibility.md
- kms-bulk-rekey-contract.md — object-side DEK re-wrap job: work unit, idempotency model, exclusion rules, and the never-destroy-old-key-versions constraint
Support matrices (release-facing, keep current)
- s3-compatibility-matrix.md
- s3-tables-support-matrix.md
- minio-rustfs-router-compatibility.md
- minio-file-format-compat.md
Inventories & baselines (snapshots that feed migration work)
- global-state-inventory.md
- ecstore-api-facade-inventory.md
- ecstore-config-consumer-inventory.md
- obs-ecstore-dependency-inventory.md
- background-services-inventory.md
- admin-route-action-snapshot.md
- compat-cleanup-register.md
Historical plans and trackers (rebalance/decommission phases, migration-progress ledger, and the one-shot migration snapshots that fed it — startup timeline, scheduler baseline, profiling/NUMA capability inventory, KMS development defaults inventory) were retired in 2026-07 once the architecture-review ledger they served closed out (backlog#660/#665). Planning documents are no longer kept in the repository.