docs(ecstore): pin streaming-only bitrot layout invariant (ECA-18) (#4553)

bitrot_shard_file_size only counts per-block checksum bytes for the two
streaming Highway variants, while BitrotWriter::write interleaves a hash
for any hash_algo.size() > 0 and bitrot_verify's read loop assumes an
interleaved hash per block. The three disagree for non-streaming
algorithms (SHA256/HighwayHash256/BLAKE2b512/Md5), but the divergence is
unreachable in production: every write path hardcodes HighwayHash256S and
ErasureInfo::get_checksum_info defaults to HighwayHash256S.

Per the audit decision (backlog#959), do NOT change the size formula:
it is a byte-for-byte port of MinIO's bitrotShardFileSize and its bare
return for non-streaming algorithms is correct for MinIO whole-file
bitrot; changing it would break legacy interop. Instead, document the
per-algorithm layout contract at bitrot_shard_file_size, BitrotWriter,
and bitrot_verify, and add regression tests that pin the invariants:
get_checksum_info defaults to HighwayHash256S, and the size formula
counts per-block hash bytes for streaming variants only while returning
the bare size for non-streaming ones. No disk layout or formula change.

Co-authored-by: heihutu <heihutu@gmail.com>
This commit is contained in:
houseme
2026-07-09 02:25:05 +08:00
committed by GitHub
parent 20d61c73bc
commit f96314a1d5
2 changed files with 177 additions and 0 deletions
+32
View File
@@ -715,6 +715,38 @@ mod tests {
use proptest::collection::{hash_map, vec};
use proptest::prelude::*;
// backlog#959 / ECA-18: the interleaved per-block bitrot subsystem in
// rustfs-ecstore (BitrotWriter / bitrot_verify / bitrot_shard_file_size) is
// only self-consistent for the streaming Highway variants. That safety rests
// on production always resolving a streaming checksum algorithm, so pin the
// default here: a part with no explicit ChecksumInfo must resolve to
// HighwayHash256S. If this default ever changes, the ecstore bitrot layout
// assumptions must be revisited in lockstep.
#[test]
fn get_checksum_info_defaults_to_highwayhash256s() {
let ei = ErasureInfo::default();
assert!(ei.checksums.is_empty(), "default ErasureInfo carries no checksums");
let info = ei.get_checksum_info(1);
assert_eq!(
info.algorithm,
HashAlgorithm::HighwayHash256S,
"missing ChecksumInfo must default to the streaming HighwayHash256S algorithm"
);
// A present entry is returned as-is; the default only applies on miss.
let ei = ErasureInfo {
checksums: vec![ChecksumInfo {
part_number: 2,
algorithm: HashAlgorithm::SHA256,
hash: Bytes::new(),
}],
..Default::default()
};
assert_eq!(ei.get_checksum_info(2).algorithm, HashAlgorithm::SHA256);
// A part_number with no entry still falls back to the streaming default.
assert_eq!(ei.get_checksum_info(99).algorithm, HashAlgorithm::HighwayHash256S);
}
// backlog#949: distribution range/permutation validation.
#[test]
fn is_valid_distribution_accepts_permutation() {