mirror of
https://github.com/rustfs/rustfs.git
synced 2026-07-26 16:28:15 +00:00
Zhengchao An
2c113542f8
* docs(architecture): add normative erasure-coding algorithm and on-disk compatibility contract Adds docs/architecture/erasure-coding.md as the source of truth for how RustFS erasure-codes, stores, reads, reconstructs, and heals user data, and the on-disk (xl.meta) / decode compatibility contract every future change must preserve. Grounded in the baseline (main) implementation with file:line anchors; cross-links (does not duplicate) the existing placement, MinIO-format-compat, layout-boundary, decommission, and tier-ILM docs, and the AGENTS.md cross-cutting invariants. Covers: Reed-Solomon over GF(2^8) modern vs GF(2^16) legacy backends and how each is selected; pool/set/drive geometry with the 2..=16 set-size and per-pool parity invariants; the key-derived distribution permutation (1..=N); 1 MiB block size and the modern/legacy shard-size formulas; HighwayHash256S interleaved bitrot layout; the full xl.meta container/header/version-body schema and internal dual-key convention; write/read/heal quorum rules; and, newly codified as a first-class contract, the decode-tolerance invariants (nil-UUID/epoch-mod_time to None, skip unknown fields, hard-guard only length-critical arrays, tolerate the negative actual_size compressed sentinel, tolerate a malformed transitioned-versionID). Linked from the architecture README under "Contracts & invariants". Docs-only; check_doc_paths.sh passes. * docs(architecture): anchor erasure spec by symbol name, not line numbers Line numbers rot as code changes and are not validated by check_doc_paths.sh, so they would silently mislead the very code changes this normative spec is meant to guide. Replace all file:line citations with file-path + symbol-name references (functions/consts/types are greppable and rename only on deliberate changes; format byte offsets are kept). Add an explicit "references work here" note and a §13 rule that governed changes must update this spec in the same PR. * docs(architecture): correct erasure spec after multi-expert adversarial review Four independent adversarial reviewers fact-checked every claim against the code. Fixes: - CRITICAL (found independently by two reviewers): §1 mislabeled the GF(2^16) reed-solomon-simd "legacy" backend as the reader for "older MinIO-lineage format". It is the opposite — that backend serves RustFS's own older main-branch (rmp_serde, uses_legacy_checksum) objects; MinIO-migrated data uses the same rs-vandermonde GF(2^8) scheme and is decoded by the modern backend. The old wording contradicted §1's own MinIO-interop line, §12, minio-file-format-compat.md, and the source comments, and would have misled the highest-stakes decode-routing decision. - §2.1/§12: set size 2..=16 holds for multi-drive layouts; single-drive deployments run at N=1 (parity 0) outside is_valid_set_size. - §2.2: validate_parity_inner enforces parity <= N/2 only for N > 2 (user storage-class parity flows through it); the standalone validate_parity is unconditional but only applied to the resolved default. - §6.2/§12: header format is dispatched by header_ver; array length (4/5/7) is a per-version validation, not the discriminator. - §6.3: the part-array length guard applies on the all_parts decode path. - §6.5: get_bytes matches only the two canonical lowercase keys; only is_internal_key/get_str are case-insensitive. - §6.5: transitioned-versionID — state the load-bearing invariant (non-16-byte decodes to None, never fatal); string-form recovery is optional, and transitioned-xl.meta interop is out of scope. - §11: typo RustSF -> RustFS. All other claims across §1-§14 were verified accurate against code (distribution formula, quorum formulas, on-disk key set/endianness/markers, decode-tolerance invariants, version anchors, standard references).