mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-30 16:59:52 +00:00
fix(storage): resolve erasure parity per pool (#4977)
* fix(filemeta): add state-aware file info validation
* fix(filemeta): validate shard arithmetic and delete paths
* fix(ecstore): add fallible erasure construction
* fix(ecstore): resolve storage parity per pool
* fix(storage): report heterogeneous erasure layouts
* fix(admin): publish prepared storage config atomically
* fix(storage): harden per-pool parity boundaries
* fix(storage): address pre-PR validation findings
* test(ci): fix strict-topology validation fixtures
* fix(heal): preserve delete markers during repair
* refactor(filemeta): drop unused ValidatedFileInfo witness
ValidatedFileInfo wrapped an unread `_file_info` reference alongside an `Option<ValidatedErasureLayout>`, but only the layout was ever consumed. Return the layout directly from `FileInfo::validate` so the sole production consumer (`LocalDisk::check_parts`) and the two unit tests read it without the extra witness type and lifetime.
No behavior change.
* fix(filemeta): keep compressed and MinIO-migrated tiered objects readable
The new decode-path validation rejected several legitimate on-disk shapes that older RustFS and MinIO-migrated data carry, turning readable objects into FileCorrupt:
- Compressed objects written with an unknown upload size persist a negative per-part actual_size (the documented "unknown size" sentinel that ObjectInfo::get_actual_size already tolerates). validate_collection_contents rejected it via usize::try_from; now a negative actual_size skips shard validation and only real, non-negative sizes are checked.
- MinIO-migrated objects transitioned to a versioned remote tier store the tier version id as a UUID string, not 16 raw bytes. MetaObject::into_fileinfo returned FileCorrupt (main tolerated it as None), making all versions of the object unreadable; MetaDeleteMarker free-version records took a Some(nil) sentinel path with the same effect, which also breaks free-version expiry (remote-tier leak). Both now decode through a shared transitioned_version_id_from_meta_sys helper: 16 raw bytes or a UUID string are accepted, anything else is tolerated as None instead of failing the read.
Regression tests updated to assert the readable/compat behavior, with new tests covering MinIO string-form recovery.
* fix(scanner): build the delete-marker test fixture without erasure geometry
get_size_counts_delete_markers_separately_from_versions built its delete marker with `FileInfo::new(object, 1, 1)`, which attaches erasure geometry (data=1/parity=1/distribution). This PR classifies versions by shape via `is_storage_delete_marker()` (no geometry) rather than the raw `deleted` flag, so a geometry-bearing "delete marker" is correctly serialized as a purge-pending payload Object and counted as a version — CI saw summary.versions=3, expected 2.
Real delete markers carry no erasure geometry (delete paths build them as `FileInfo { deleted: true, ..Default::default() }`), so construct the fixture the same way. It then classifies as a storage delete marker and the counts (versions=2, delete_markers=1) hold. This keeps the PR's more-correct classification, which prevents a purge-pending object's geometry from being dropped when serialized as a bare delete marker.
* docs(changelog): note per-pool parity fix and storage-class startup upgrade caveat
Records the #4801 per-pool erasure parity fix under Fixed, and documents the upgrade behavior where a persisted storage class that a small or heterogeneous pool cannot satisfy now fails startup — with the RUSTFS_STORAGE_CLASS_STANDARD recovery steps. Docs-only; covers R4 from the on-disk compatibility audit.
* fix(heal): report parity from erasure geometry, not is_valid()
heal_object set HealResultItem.parity_blocks via `if lfi.is_valid()`, which was missed by the migration of the other quorum/metadata predicates. With the new `is_valid()` semantics (full payload validation; delete markers now return false), a delete marker or a geometry-bearing version with a benign collection quirk would misreport parity as the pool default instead of its own. Use `has_valid_erasure_geometry()` — the narrow "does this carry erasure geometry" predicate the rest of the migration uses — so reporting matches the object's actual layout. Reporting-only; no data-path change.
* fix(filemeta): do not silently serialize a non-canonical deleted FileInfo as an Object
`From<FileInfo> for FileMetaVersion` classifies by `is_storage_delete_marker()` (shape), which correctly routes canonical delete markers to Delete and purge-pending payloads (deleted=true with real erasure geometry) to Object. But a `deleted` FileInfo that is neither a canonical marker nor a valid erasure payload would silently serialize as a zero-geometry MetaObject that later fails `validate_for_metadata_read`. Write paths validate first (`validate_for_erasure_write` / `validate_for_metadata_read`), so this is a caller bug; `From` is infallible, so surface it with a structured `warn!` on the malformed branch instead of writing corrupt metadata silently. Legitimate purge-pending objects (valid geometry) are unaffected — the guard only fires for `deleted && !has_valid_erasure_geometry()`.
* test(filemeta): assert real historical xl.meta versions pass metadata-read validation
Empirical companion to the code-reasoned decode-tolerance invariants (docs/architecture/erasure-coding.md §11) and the rolling-upgrade / MinIO-migration compatibility concern: the tightened `validate_for_metadata_read` runs on every local disk read and peer-RPC-decoded FileInfo, so it must accept every version of real historically-written xl.meta, never reject it as FileCorrupt.
Loads five real fixtures — MinIO small-inline, MinIO versioned (two object versions + a delete marker), MinIO large multipart, a legacy V1 (xl.json-derived) object, and a legacy meta_ver 2 object — decodes every version with parts materialized, and asserts validate_for_metadata_read() is Ok for each. Reverting the tolerant handling (delete-marker shape, legacy per-part checksums, string/short transitioned-versionID, negative actual_size) turns this red.
* fix(ci): remove duplicate storage test re-exports
---------
Co-authored-by: overtrue <anzhengchao@gmail.com>
This commit is contained in:
cxymds
@@ -224,6 +224,14 @@ pub(crate) const RUSTFS_MULTIPART_BUCKET_KEY: &str = "x-rustfs-internal-multipar
|
||||
pub(crate) const RUSTFS_MULTIPART_OBJECT_KEY: &str = "x-rustfs-internal-multipart-object";
|
||||
const ENV_ISSUE3031_DIAG_ENABLE: &str = "RUSTFS_ISSUE3031_DIAG_ENABLE";
|
||||
|
||||
/// Validate disk metadata at a boundary that may legitimately return a delete
|
||||
/// marker. Disk/RPC decode boundaries perform the full collection validation
|
||||
/// once; repeated quorum passes use the cheap erasure-geometry predicate for
|
||||
/// payload entries and the canonical marker predicate for pure delete markers.
|
||||
pub(in crate::set_disk) fn file_info_is_valid_for_metadata(file_info: &FileInfo) -> bool {
|
||||
file_info.has_valid_metadata_shape()
|
||||
}
|
||||
|
||||
struct ObjectLockDiagGuard {
|
||||
guard: NamespaceLockGuard,
|
||||
enabled: bool,
|
||||
@@ -1971,25 +1979,209 @@ fn issue3031_diag_enabled() -> bool {
|
||||
rustfs_utils::get_env_bool(ENV_ISSUE3031_DIAG_ENABLE, false)
|
||||
}
|
||||
|
||||
fn build_tiered_decommission_file_info(
|
||||
bucket: &str,
|
||||
object: &str,
|
||||
fi: &FileInfo,
|
||||
disk_count: usize,
|
||||
default_parity_count: usize,
|
||||
storage_class: Option<&str>,
|
||||
) -> (FileInfo, usize) {
|
||||
let parity_drives = runtime_sources::storage_class_parity(storage_class).unwrap_or(default_parity_count);
|
||||
let data_drives = disk_count - parity_drives;
|
||||
let mut write_quorum = data_drives;
|
||||
if data_drives == parity_drives {
|
||||
write_quorum += 1;
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub(super) struct WriteLayout {
|
||||
data_drives: usize,
|
||||
parity_drives: usize,
|
||||
write_quorum: usize,
|
||||
}
|
||||
|
||||
impl WriteLayout {
|
||||
fn from_parity(drive_count: usize, parity_drives: usize) -> Result<Self> {
|
||||
let max_parity = drive_count / 2;
|
||||
if parity_drives > max_parity {
|
||||
return Err(Error::other(format!(
|
||||
"write parity {parity_drives} exceeds the maximum {max_parity} for {drive_count} drives"
|
||||
)));
|
||||
}
|
||||
|
||||
let data_drives = drive_count
|
||||
.checked_sub(parity_drives)
|
||||
.filter(|&data_drives| data_drives > 0 && parity_drives <= data_drives)
|
||||
.ok_or_else(|| Error::other(format!("invalid write layout with {drive_count} drives and parity {parity_drives}")))?;
|
||||
let write_quorum = data_drives
|
||||
.checked_add(usize::from(data_drives == parity_drives))
|
||||
.filter(|&write_quorum| write_quorum <= drive_count)
|
||||
.ok_or_else(|| Error::other(format!("invalid write quorum for {drive_count} drives and parity {parity_drives}")))?;
|
||||
|
||||
Ok(Self {
|
||||
data_drives,
|
||||
parity_drives,
|
||||
write_quorum,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
pub(super) fn resolve_write_layout(
|
||||
config: &storageclass::Config,
|
||||
pool_index: usize,
|
||||
drive_count: usize,
|
||||
fallback_parity: usize,
|
||||
storage_class: Option<&str>,
|
||||
max_parity: bool,
|
||||
) -> Result<WriteLayout> {
|
||||
let configured_parity = if config.is_initialized() {
|
||||
config
|
||||
.parity_for_pool(storage_class.unwrap_or_default(), pool_index, drive_count)
|
||||
.ok_or_else(|| {
|
||||
Error::other(format!("storage class layout does not match pool {pool_index} with {drive_count} drives"))
|
||||
})?
|
||||
} else {
|
||||
fallback_parity
|
||||
};
|
||||
let parity_drives = if max_parity { drive_count / 2 } else { configured_parity };
|
||||
|
||||
WriteLayout::from_parity(drive_count, parity_drives)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod write_layout_tests {
|
||||
use super::{WriteLayout, resolve_write_layout};
|
||||
use crate::config::storageclass::{
|
||||
CLASS_RRS, CLASS_STANDARD, INLINE_BLOCK_ENV, OPTIMIZE_ENV, RRS, RRS_ENV, STANDARD_ENV, lookup_config_for_pools,
|
||||
lookup_config_for_pools_without_env,
|
||||
};
|
||||
use arc_swap::ArcSwap;
|
||||
use rustfs_config::server_config::KVS;
|
||||
use std::sync::Arc;
|
||||
|
||||
#[test]
|
||||
fn automatic_standard_layout_is_resolved_per_pool() {
|
||||
let config = lookup_config_for_pools_without_env(&KVS::new(), &[4, 2])
|
||||
.expect("automatic storage class should resolve for both pools");
|
||||
|
||||
assert_eq!(
|
||||
resolve_write_layout(&config, 0, 4, 2, None, false).expect("first pool should resolve"),
|
||||
WriteLayout {
|
||||
data_drives: 2,
|
||||
parity_drives: 2,
|
||||
write_quorum: 3,
|
||||
}
|
||||
);
|
||||
assert_eq!(
|
||||
resolve_write_layout(&config, 1, 2, 1, None, false).expect("second pool should resolve"),
|
||||
WriteLayout {
|
||||
data_drives: 1,
|
||||
parity_drives: 1,
|
||||
write_quorum: 2,
|
||||
}
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn reduced_redundancy_layout_allows_single_disk_zero_parity() {
|
||||
let config =
|
||||
lookup_config_for_pools_without_env(&KVS::new(), &[4, 1]).expect("reduced redundancy should resolve for both pools");
|
||||
|
||||
assert_eq!(
|
||||
resolve_write_layout(&config, 0, 4, 2, Some(RRS), false).expect("four-drive RRS pool should resolve"),
|
||||
WriteLayout {
|
||||
data_drives: 3,
|
||||
parity_drives: 1,
|
||||
write_quorum: 3,
|
||||
}
|
||||
);
|
||||
assert_eq!(
|
||||
resolve_write_layout(&config, 1, 1, 0, Some(RRS), false).expect("single-drive RRS pool should resolve"),
|
||||
WriteLayout {
|
||||
data_drives: 1,
|
||||
parity_drives: 0,
|
||||
write_quorum: 1,
|
||||
}
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn write_layout_rejects_unknown_topology_and_invalid_parity() {
|
||||
let config = lookup_config_for_pools_without_env(&KVS::new(), &[4, 2]).expect("test storage class should resolve");
|
||||
|
||||
assert!(resolve_write_layout(&config, 2, 2, 1, None, false).is_err());
|
||||
assert!(resolve_write_layout(&config, 1, 4, 2, None, false).is_err());
|
||||
assert!(WriteLayout::from_parity(4, 3).is_err());
|
||||
assert!(WriteLayout::from_parity(0, 0).is_err());
|
||||
|
||||
let mut zero_parity_kvs = KVS::new();
|
||||
zero_parity_kvs.insert(CLASS_STANDARD.to_string(), "EC:0".to_string());
|
||||
zero_parity_kvs.insert(CLASS_RRS.to_string(), "EC:0".to_string());
|
||||
let zero_parity = lookup_config_for_pools_without_env(&zero_parity_kvs, &[4]).expect("zero-parity config should resolve");
|
||||
assert_eq!(
|
||||
resolve_write_layout(&zero_parity, 0, 4, 2, None, true).expect("max parity should override configured parity"),
|
||||
WriteLayout {
|
||||
data_drives: 2,
|
||||
parity_drives: 2,
|
||||
write_quorum: 3,
|
||||
}
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn only_uninitialized_config_falls_back_to_pool_startup_parity() {
|
||||
let uninitialized = crate::config::storageclass::Config::default();
|
||||
assert_eq!(
|
||||
resolve_write_layout(&uninitialized, 99, 2, 0, None, false)
|
||||
.expect("uninitialized config should preserve the pool's startup fallback"),
|
||||
WriteLayout {
|
||||
data_drives: 2,
|
||||
parity_drives: 0,
|
||||
write_quorum: 2,
|
||||
}
|
||||
);
|
||||
|
||||
let initialized = lookup_config_for_pools_without_env(&KVS::new(), &[4, 2]).expect("initialized config should resolve");
|
||||
assert!(resolve_write_layout(&initialized, 99, 2, 1, None, false).is_err());
|
||||
assert!(resolve_write_layout(&initialized, 1, 4, 2, None, false).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial_test::serial(storage_class_env)]
|
||||
fn held_snapshot_keeps_parity_and_inline_policy_consistent_across_reload() {
|
||||
let old = temp_env::with_vars(
|
||||
[
|
||||
(STANDARD_ENV, Some("")),
|
||||
(RRS_ENV, Some("")),
|
||||
(OPTIMIZE_ENV, None),
|
||||
(INLINE_BLOCK_ENV, Some("1KiB")),
|
||||
],
|
||||
|| lookup_config_for_pools(&KVS::new(), &[4, 2]),
|
||||
)
|
||||
.expect("old config should resolve");
|
||||
let new = temp_env::with_vars(
|
||||
[
|
||||
(STANDARD_ENV, Some("EC:1")),
|
||||
(RRS_ENV, Some("EC:1")),
|
||||
(OPTIMIZE_ENV, None),
|
||||
(INLINE_BLOCK_ENV, Some("0B")),
|
||||
],
|
||||
|| lookup_config_for_pools(&KVS::new(), &[4, 2]),
|
||||
)
|
||||
.expect("new config should resolve");
|
||||
|
||||
let published = ArcSwap::from_pointee(old);
|
||||
let held = published.load_full();
|
||||
published.store(Arc::new(new));
|
||||
|
||||
let held_layout = resolve_write_layout(&held, 0, 4, 2, None, false).expect("held snapshot should remain valid");
|
||||
assert_eq!(held_layout.parity_drives, 2);
|
||||
assert!(held.should_inline(512, false));
|
||||
|
||||
let current = published.load_full();
|
||||
let current_layout = resolve_write_layout(¤t, 0, 4, 2, None, false).expect("new snapshot should resolve");
|
||||
assert_eq!(current_layout.parity_drives, 1);
|
||||
assert!(!current.should_inline(512, false));
|
||||
}
|
||||
}
|
||||
|
||||
fn build_tiered_decommission_file_info(bucket: &str, object: &str, fi: &FileInfo, layout: WriteLayout) -> FileInfo {
|
||||
let WriteLayout {
|
||||
data_drives,
|
||||
parity_drives,
|
||||
..
|
||||
} = layout;
|
||||
|
||||
let mut updated = fi.clone();
|
||||
updated.erasure = FileInfo::new([bucket, object].join("/").as_str(), data_drives, parity_drives).erasure;
|
||||
|
||||
(updated, write_quorum)
|
||||
updated
|
||||
}
|
||||
|
||||
fn resolve_tiered_decommission_write_quorum_result(
|
||||
@@ -2036,6 +2228,8 @@ pub struct SetDisks {
|
||||
/// writes skip the global registry mutex (backlog#1315). `Arc` so clones of
|
||||
/// a set share one generation marker.
|
||||
capacity_dirty_generation: Arc<AtomicU64>,
|
||||
#[cfg(test)]
|
||||
storage_class_config_override: Arc<std::sync::RwLock<Option<Arc<storageclass::Config>>>>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Eq, PartialEq)]
|
||||
@@ -2168,6 +2362,28 @@ impl DiskHealthEntry {
|
||||
}
|
||||
|
||||
impl SetDisks {
|
||||
fn storage_class_config_snapshot(&self) -> Arc<storageclass::Config> {
|
||||
#[cfg(test)]
|
||||
if let Some(config) = self
|
||||
.storage_class_config_override
|
||||
.read()
|
||||
.expect("test storage class override lock should not be poisoned")
|
||||
.as_ref()
|
||||
{
|
||||
return config.clone();
|
||||
}
|
||||
|
||||
runtime_sources::storage_class_config_snapshot()
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) fn set_test_storage_class_config(&self, config: storageclass::Config) {
|
||||
*self
|
||||
.storage_class_config_override
|
||||
.write()
|
||||
.expect("test storage class override lock should not be poisoned") = Some(Arc::new(config));
|
||||
}
|
||||
|
||||
fn get_object_metadata_cache_hash(&self, bucket: &str, object: &str) -> u64 {
|
||||
let mut hasher = self.get_object_metadata_cache_hash_builder.build_hasher();
|
||||
bucket.hash(&mut hasher);
|
||||
@@ -2372,6 +2588,8 @@ impl SetDisks {
|
||||
ctx,
|
||||
capacity_scope_cache: Arc::new(std::sync::RwLock::new(CapacityScopeCache::default())),
|
||||
capacity_dirty_generation: Arc::new(AtomicU64::new(u64::MAX)),
|
||||
#[cfg(test)]
|
||||
storage_class_config_override: Arc::new(std::sync::RwLock::new(None)),
|
||||
})
|
||||
}
|
||||
|
||||
@@ -2888,7 +3106,7 @@ fn collect_inline_data_shard_fileinfos_by_index<'a>(
|
||||
if block_index == 0 || block_index > data_shards {
|
||||
continue;
|
||||
}
|
||||
if !file_info.is_valid() {
|
||||
if !file_info.has_valid_erasure_geometry() {
|
||||
continue;
|
||||
}
|
||||
if file_info.data.as_ref().is_none_or(|data| data.is_empty()) {
|
||||
@@ -3322,6 +3540,7 @@ impl SetDisks {
|
||||
fi: &FileInfo,
|
||||
opts: &ObjectOptions,
|
||||
) -> Result<()> {
|
||||
let storage_class_config = self.storage_class_config_snapshot();
|
||||
let _lock_guard = if !opts.no_lock {
|
||||
Some(
|
||||
self.new_ns_lock(bucket, object)
|
||||
@@ -3336,8 +3555,16 @@ impl SetDisks {
|
||||
|
||||
let disks = self.disks.read().await.clone();
|
||||
let storage_class = opts.user_defined.get(AMZ_STORAGE_CLASS).map(String::as_str);
|
||||
let (fi, write_quorum) =
|
||||
build_tiered_decommission_file_info(bucket, object, fi, disks.len(), self.default_parity_count, storage_class);
|
||||
let layout = resolve_write_layout(
|
||||
&storage_class_config,
|
||||
self.pool_index,
|
||||
disks.len(),
|
||||
self.default_parity_count,
|
||||
storage_class,
|
||||
opts.max_parity,
|
||||
)?;
|
||||
let fi = build_tiered_decommission_file_info(bucket, object, fi, layout);
|
||||
let write_quorum = layout.write_quorum;
|
||||
let parts_metadata = vec![fi.clone(); disks.len()];
|
||||
let (shuffle_disks, parts_metadata) = Self::shuffle_disks_and_parts_metadata(&disks, &parts_metadata, &fi);
|
||||
|
||||
@@ -3407,13 +3634,13 @@ fn is_object_dangling(
|
||||
let mut valid_meta = FileInfo::default();
|
||||
|
||||
for fi in meta_arr.iter() {
|
||||
if fi.is_valid() {
|
||||
if file_info_is_valid_for_metadata(fi) {
|
||||
valid_meta = fi.clone();
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
if !valid_meta.is_valid() {
|
||||
if !file_info_is_valid_for_metadata(&valid_meta) {
|
||||
let data_blocks = meta_arr.len().div_ceil(2);
|
||||
if not_found_parts_errs > data_blocks {
|
||||
return (valid_meta, true);
|
||||
@@ -3426,7 +3653,7 @@ fn is_object_dangling(
|
||||
return (valid_meta, false);
|
||||
}
|
||||
|
||||
if valid_meta.deleted {
|
||||
if valid_meta.is_canonical_delete_marker() {
|
||||
let data_blocks = errs.len().div_ceil(2);
|
||||
return (valid_meta, not_found_meta_errs > data_blocks);
|
||||
}
|
||||
@@ -3539,12 +3766,12 @@ async fn disks_with_all_parts(
|
||||
// Check for inconsistent erasure distribution
|
||||
let mut inconsistent = 0;
|
||||
for (index, meta) in parts_metadata.iter().enumerate() {
|
||||
if !meta.is_valid() {
|
||||
if !file_info_is_valid_for_metadata(meta) {
|
||||
// Since for majority of the cases erasure.Index matches with erasure.Distribution we can
|
||||
// consider the offline disks as consistent.
|
||||
continue;
|
||||
}
|
||||
if !meta.deleted {
|
||||
if !meta.is_canonical_delete_marker() {
|
||||
if meta.erasure.distribution.len() != online_disks.len() {
|
||||
// Erasure distribution seems to have lesser
|
||||
// number of items than number of online disks.
|
||||
@@ -3604,7 +3831,7 @@ async fn disks_with_all_parts(
|
||||
}
|
||||
|
||||
if erasure_distribution_reliable {
|
||||
if !meta.is_valid() {
|
||||
if !file_info_is_valid_for_metadata(meta) {
|
||||
info!(
|
||||
"disks_with_all_partsv2: metadata is not valid, object_name={}, index: {index}",
|
||||
object_name
|
||||
@@ -3615,7 +3842,7 @@ async fn disks_with_all_parts(
|
||||
continue;
|
||||
}
|
||||
|
||||
if !meta.deleted && meta.erasure.distribution.len() != online_disks_len {
|
||||
if !meta.is_canonical_delete_marker() && meta.erasure.distribution.len() != online_disks_len {
|
||||
// Erasure distribution is not the same as onlineDisks
|
||||
// attempt a fix if possible, assuming other entries
|
||||
// might have the right erasure distribution.
|
||||
@@ -3658,7 +3885,7 @@ async fn disks_with_all_parts(
|
||||
};
|
||||
|
||||
let meta = &mut parts_metadata[index];
|
||||
if meta.deleted || meta.is_remote() {
|
||||
if meta.is_canonical_delete_marker() || meta.is_remote() {
|
||||
continue;
|
||||
}
|
||||
|
||||
@@ -3788,7 +4015,7 @@ pub fn should_heal_object_on_disk(
|
||||
return (true, true, Some(DiskError::OutdatedXLMeta));
|
||||
}
|
||||
|
||||
if !meta.deleted && !meta.is_remote() {
|
||||
if !meta.is_canonical_delete_marker() && !meta.is_remote() {
|
||||
let err_vec = [CHECK_PART_FILE_NOT_FOUND, CHECK_PART_FILE_CORRUPT];
|
||||
for part_err in parts_errs.iter() {
|
||||
if err_vec.contains(part_err) {
|
||||
@@ -6356,6 +6583,7 @@ mod tests {
|
||||
erasure: ErasureInfo {
|
||||
data_blocks: 4,
|
||||
parity_blocks: 2,
|
||||
block_size: 4,
|
||||
index: 1, // Must be > 0 for is_valid() to return true
|
||||
distribution: vec![1, 2, 3, 4, 5, 6], // Must match data_blocks + parity_blocks
|
||||
..Default::default()
|
||||
@@ -6368,6 +6596,7 @@ mod tests {
|
||||
erasure: ErasureInfo {
|
||||
data_blocks: 6,
|
||||
parity_blocks: 3,
|
||||
block_size: 4,
|
||||
index: 1, // Must be > 0 for is_valid() to return true
|
||||
distribution: vec![1, 2, 3, 4, 5, 6, 7, 8, 9], // Must match data_blocks + parity_blocks
|
||||
..Default::default()
|
||||
@@ -6380,6 +6609,7 @@ mod tests {
|
||||
erasure: ErasureInfo {
|
||||
data_blocks: 2,
|
||||
parity_blocks: 1,
|
||||
block_size: 4,
|
||||
index: 1, // Must be > 0 for is_valid() to return true
|
||||
distribution: vec![1, 2, 3], // Must match data_blocks + parity_blocks
|
||||
..Default::default()
|
||||
@@ -6399,6 +6629,102 @@ mod tests {
|
||||
assert_eq!(parities[2], 1); // half of total shards (3/2 = 1) for zero size file
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn delete_markers_participate_in_four_disk_metadata_quorum_without_erasure_geometry() {
|
||||
let marker = FileInfo {
|
||||
name: "bucket/deleted".to_string(),
|
||||
deleted: true,
|
||||
version_id: Some(Uuid::new_v4()),
|
||||
mod_time: Some(OffsetDateTime::now_utc()),
|
||||
..Default::default()
|
||||
};
|
||||
let parts_metadata = vec![marker; 4];
|
||||
let errs = vec![None; 4];
|
||||
|
||||
assert!(parts_metadata.iter().all(file_info_is_valid_for_metadata));
|
||||
assert!(parts_metadata.iter().all(|metadata| !metadata.is_valid()));
|
||||
assert_eq!(SetDisks::list_object_parities(&parts_metadata, &errs), vec![2; 4]);
|
||||
assert_eq!(
|
||||
SetDisks::object_quorum_from_meta(&parts_metadata, &errs, 2)
|
||||
.expect("four matching delete markers must reach metadata quorum"),
|
||||
(2, 3)
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn metadata_boundary_does_not_relax_non_delete_or_malformed_delete_metadata() {
|
||||
assert!(!file_info_is_valid_for_metadata(&FileInfo::default()));
|
||||
|
||||
let mut transitioned = FileInfo::new("bucket/transitioned", 2, 2);
|
||||
transitioned.erasure.index = 1;
|
||||
transitioned.transition_status = TRANSITION_COMPLETE.to_string();
|
||||
assert!(file_info_is_valid_for_metadata(&transitioned));
|
||||
|
||||
transitioned.erasure = ErasureInfo::default();
|
||||
assert!(
|
||||
!file_info_is_valid_for_metadata(&transitioned),
|
||||
"transition state must not relax local erasure validation"
|
||||
);
|
||||
|
||||
let mut purge_pending = FileInfo::new("bucket/purge-pending", 2, 2);
|
||||
purge_pending.erasure.index = 1;
|
||||
purge_pending.deleted = true;
|
||||
purge_pending.parts.push(ObjectPartInfo {
|
||||
number: 1,
|
||||
..Default::default()
|
||||
});
|
||||
assert!(
|
||||
file_info_is_valid_for_metadata(&purge_pending),
|
||||
"purge-pending payload metadata must retain its valid erasure vote"
|
||||
);
|
||||
assert!(!purge_pending.is_canonical_delete_marker());
|
||||
|
||||
let mut malformed_marker = FileInfo {
|
||||
deleted: true,
|
||||
..Default::default()
|
||||
};
|
||||
malformed_marker.parts = vec![
|
||||
ObjectPartInfo {
|
||||
number: 1,
|
||||
..Default::default()
|
||||
},
|
||||
ObjectPartInfo {
|
||||
number: 1,
|
||||
..Default::default()
|
||||
},
|
||||
];
|
||||
assert!(!file_info_is_valid_for_metadata(&malformed_marker));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn purge_pending_payload_uses_its_erasure_parity_for_metadata_quorum() {
|
||||
let version_id = Uuid::new_v4();
|
||||
let mod_time = OffsetDateTime::now_utc();
|
||||
let parts_metadata = (1..=6)
|
||||
.map(|disk_index| {
|
||||
let mut purge_pending = FileInfo::new("bucket/purge-pending", 5, 1);
|
||||
purge_pending.name = "bucket/purge-pending".to_string();
|
||||
purge_pending.version_id = Some(version_id);
|
||||
purge_pending.mod_time = Some(mod_time);
|
||||
purge_pending.size = 1;
|
||||
purge_pending.deleted = true;
|
||||
purge_pending.erasure.index = disk_index;
|
||||
purge_pending.add_object_part(1, "part-etag-1".to_string(), 1, None, 1, None, None);
|
||||
purge_pending
|
||||
})
|
||||
.collect::<Vec<_>>();
|
||||
let errs = vec![None; 6];
|
||||
|
||||
assert!(parts_metadata.iter().all(file_info_is_valid_for_metadata));
|
||||
assert!(parts_metadata.iter().all(|metadata| !metadata.is_canonical_delete_marker()));
|
||||
assert_eq!(SetDisks::list_object_parities(&parts_metadata, &errs), vec![1; 6]);
|
||||
assert_eq!(
|
||||
SetDisks::object_quorum_from_meta(&parts_metadata, &errs, 3)
|
||||
.expect("purge-pending payload should retain its EC:1 quorum"),
|
||||
(5, 5)
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_conv_part_err_to_int() {
|
||||
// Test error conversion to integer codes
|
||||
@@ -7141,7 +7467,8 @@ mod tests {
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
let (updated, write_quorum) = build_tiered_decommission_file_info("bucket", "object", &original, 16, 4, None);
|
||||
let layout = WriteLayout::from_parity(16, 4).expect("tiered write layout should be valid");
|
||||
let updated = build_tiered_decommission_file_info("bucket", "object", &original, layout);
|
||||
|
||||
assert_eq!(updated.version_id, original.version_id);
|
||||
assert_eq!(updated.transition_status, original.transition_status);
|
||||
@@ -7150,7 +7477,7 @@ mod tests {
|
||||
assert_eq!(updated.transition_version_id, original.transition_version_id);
|
||||
assert_eq!(updated.erasure.data_blocks, 12);
|
||||
assert_eq!(updated.erasure.parity_blocks, 4);
|
||||
assert_eq!(write_quorum, 12);
|
||||
assert_eq!(layout.write_quorum, 12);
|
||||
assert_ne!(updated.erasure.distribution, original.erasure.distribution);
|
||||
}
|
||||
|
||||
@@ -8395,11 +8722,15 @@ mod tests {
|
||||
}
|
||||
|
||||
async fn make_local_bucket_test_set_disks() -> Arc<SetDisks> {
|
||||
let format = FormatV3::new(1, 2);
|
||||
make_local_bucket_test_set_disks_with_drive_count(2).await
|
||||
}
|
||||
|
||||
async fn make_local_bucket_test_set_disks_with_drive_count(drive_count: usize) -> Arc<SetDisks> {
|
||||
let format = FormatV3::new(1, drive_count);
|
||||
let mut endpoints = Vec::new();
|
||||
let mut disks = Vec::new();
|
||||
|
||||
for disk_idx in 0..2 {
|
||||
for disk_idx in 0..drive_count {
|
||||
let dir = tempfile::tempdir().expect("tempdir should be created");
|
||||
let mut endpoint =
|
||||
Endpoint::try_from(dir.path().to_str().expect("tempdir path should be utf8")).expect("endpoint should parse");
|
||||
@@ -8428,18 +8759,23 @@ mod tests {
|
||||
disks.push(Some(disk));
|
||||
}
|
||||
|
||||
SetDisks::new(
|
||||
let set_disks = SetDisks::new(
|
||||
"test-owner".to_string(),
|
||||
Arc::new(RwLock::new(disks)),
|
||||
2,
|
||||
1,
|
||||
drive_count,
|
||||
drive_count / 2,
|
||||
0,
|
||||
0,
|
||||
endpoints,
|
||||
format,
|
||||
Vec::new(),
|
||||
)
|
||||
.await
|
||||
.await;
|
||||
set_disks.set_test_storage_class_config(
|
||||
storageclass::lookup_config_for_pools_without_env(&rustfs_config::server_config::KVS::new(), &[drive_count])
|
||||
.expect("test storage class should resolve for the local drive count"),
|
||||
);
|
||||
set_disks
|
||||
}
|
||||
|
||||
async fn make_local_bucket_test_set_disks_with_missing_format() -> Arc<SetDisks> {
|
||||
@@ -8909,7 +9245,7 @@ mod tests {
|
||||
|
||||
#[tokio::test]
|
||||
async fn set_level_versioned_delete_marker_hides_object_without_corrupting_version_metadata() {
|
||||
let set_disks = make_local_bucket_test_set_disks().await;
|
||||
let set_disks = make_local_bucket_test_set_disks_with_drive_count(4).await;
|
||||
let bucket = "bucket-versioned-delete";
|
||||
let object = "object.txt";
|
||||
let opts = ObjectOptions {
|
||||
|
||||
Reference in New Issue
Block a user