fix(lifecycle): safely expire all object versions (#6291)

* fix(lifecycle): safely expire all object versions * fix(lifecycle): preserve delete-all replication purges * fix(lifecycle): remove dead replication journal * fix(ci): avoid lifecycle transition test stack overflow * fix(lifecycle): release recovery locks before tier IO * test(lifecycle): align object-lock error assertions * test(lifecycle): avoid scanner restore stack overflow * test(scanner): avoid stack overflow in transition and restore flow test (#6300) * refactor(scanner): split remote_scanner.rs into stream child module (#6289) Split the 3080-line remote_scanner.rs (47% inline tests) into a canonical foo.rs + foo/ module tree with zero behavior change: - remote_scanner.rs (~320): protocol constants, process statics, and the request decode/validate/admit/preflight/claim API plus root re-exports - remote_scanner/stream.rs (~1340): wire/frame types, replay cache, FrameAuthenticator, serve path, local bucket scan + persist, client scan, and the bounded stream plumbing - remote_scanner/stream/tests.rs (~1470): the inline test module as a child module of stream so it can reach both parents' private items All crate paths are unchanged: lib.rs re-exports (serve_remote_scanner_request, RemoteScannerRequest, ...) resolve through root re-exports, and scanner_io's crate::remote_scanner:: {scan_remote_bucket, RemoteScannerScanSpec, RemoteScannerOutcome} paths resolve through pub(crate) re-exports. Cross-module items gain pub(super), whose scope equals the old single-module privacy domain; no item's effective visibility widens. Code is moved verbatim apart from those markers, per-module import headers, and rustfmt line re-wraps. Co-authored-by: heihutu <heihutu@gmail.com> * refactor(heal): split resume.rs into focused child modules (#6290) Split the 4242-line resume.rs (46% inline tests) into a canonical foo.rs + foo/ module tree with zero behavior change: - resume.rs (~1020): state file constants, PersistThrottle, ResumeState, ResumeManager core (constructors, load/discovery, progress mutators, ordinary persistence) plus root re-exports - resume/replacement.rs (~690): replacement-intent/proof types and the ResumeManager replacement-lifecycle methods - resume/checkpoint.rs (~350): ResumeCheckpoint + CheckpointManager - resume/utils.rs (~310): ResumeUtils statics - resume/tests.rs (~1980): the inline test module as a child module All module paths are unchanged (heal::resume::CheckpointManager and friends resolve through root re-exports), so no consumer inside or outside the crate changes. Items defined in child modules keep module-private visibility; only the ten cross-module helpers gain pub(super), which is not part of the crate API. Code is moved verbatim apart from those visibility markers, four super::storage_api path fixes, and the new per-module import headers. Co-authored-by: heihutu <heihutu@gmail.com> * refactor(scanner): split scanner_io.rs into child modules (#6294) Split the 5369-line scanner_io.rs (39% inline tests) into a canonical scanner_io.rs + scanner_io/ module tree with zero behavior change: - scanner_io.rs (~660): constants, metadata-error constructors, the bucket scan plan, cycle-status classification helpers, the ScannerIO / ScannerIOCache / ScannerIODisk traits, and ScannerCycleResult - scanner_io/dirty_usage.rs (~300): process-wide dirty-usage statics and the acknowledgment protocol - scanner_io/guards.rs (~270): concurrency gauges and RAII guards - scanner_io/cache.rs (~410): scanner cache locks and the snapshot persist/publish path - scanner_io/io_cycle.rs (~390), io_cache.rs (~1160), io_disk.rs (~230): the ECStore / SetDisks / Disk trait implementations - scanner_io/publish_gate_tests.rs (~750) and tests.rs (~1340): the two inline test modules as child modules All crate paths are unchanged: the lib.rs scanner_io re-exports and every crate::scanner_io:: consumer (scanner.rs, remote_scanner, scanner_folder, and cross-crate rustfs users) resolve through root re-exports with their original visibilities (pub stays pub, pub(crate) stays pub(crate)). Cross-module items gain pub(super), whose scope equals the old single-module privacy domain. Code is moved verbatim apart from those markers, per-module import headers, and rustfmt re-wraps. The logging-guardrail nsscanner_disk skip-set_disks rule now points at scanner_io/io_disk.rs where the function moved; the pattern and thresholds are unchanged. Co-authored-by: heihutu <heihutu@gmail.com> * refactor(scanner): split data_usage_define persistence and tests (#6292) Split the 3655-line data_usage_define.rs (59% inline tests) into a canonical foo.rs + foo/ module tree with zero behavior change: - data_usage_define.rs (~950): cache constants and revision helpers, the data-usage tree types, DataUsageCacheInfo with its hand-written Serialize, the in-memory tree operations, dui, and marshal/unmarshal - data_usage_define/persistence.rs (~580): the load/backup/restore ladder (load, try_load_inner, revision_for_path) and the CAS save path with its retry policy and save metrics - data_usage_define/tests.rs (~2155): the inline test module as a child module All module paths are unchanged (the lib.rs data_usage_define::* glob re-export and every crate::data_usage_define:: consumer resolve as before). The hand-written map-encoded Serialize for DataUsageCacheInfo is moved byte-for-byte per the AGENTS.md cross-cutting invariant; on-disk names and the cache key format const stay in the root. Four persistence helpers used by tests gain pub(super), whose scope equals the old single-module privacy domain. Code is moved verbatim apart from those markers, per-module import headers, and rustfmt re-wraps. Co-authored-by: heihutu <heihutu@gmail.com> * chore(deps): bump datafusion to 55.0.0 (#6288) * refactor(heal): split task.rs per heal kind (#6293) * feat(ecstore): batch small file fdatasync commits (#6297) * feat(ecstore): batch small file fdatasync commits Add a default-off experimental file fdatasync group commit path for small rename_data shard directories. The coordinator batches same-disk waiters into one blocking task while preserving per-directory source fsync after shard contents are durable. Co-Authored-By: heihutu <heihutu@gmail.com> * test(e2e): wait for compression S3 readiness Reuse the shared S3 API readiness probe for compression test servers so multipart requests do not race the startup readiness gate after the TCP port opens. Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> * fix(tier): recover multi-committed mutation intents (#6296) * fix(tier): recover multi-committed mutation intents * fix(tier): recover committed mutations on standalone nodes * test(scanner): avoid stack overflow in transition test --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: cxymds <cxymds@gmail.com> --------- Co-authored-by: houseme <housemecn@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-30 16:59:52 +00:00 · 2026-08-20 17:42:09 +08:00
parent 621fcb93c7
commit 319a03e638
15 changed files with 3251 additions and 529 deletions
@@ -202,10 +202,76 @@ impl ECStore {
    }

    pub(super) async fn delete_prefix(&self, bucket: &str, object: &str, opts: &ObjectOptions) -> Result<()> {
+        if opts.lifecycle_delete_all.is_some() {
+            let mut preflight_opts = opts.clone();
+            preflight_opts
+                .lifecycle_delete_all
+                .as_mut()
+                .ok_or(StorageError::PreconditionFailed)?
+                .phase = crate::object_api::LifecycleDeleteAllPhase::Preflight;
+            for pool in &self.pools {
+                #[cfg(test)]
+                lifecycle_delete_all_test_failure(crate::object_api::LifecycleDeleteAllPhase::Preflight, pool.pool_idx)?;
+                pool.delete_object(bucket, object, preflight_opts.clone()).await?;
+            }
+
+            opts.lifecycle_delete_all_journal()
+                .ok_or(StorageError::PreconditionFailed)?
+                .lock()
+                .mark_mutation_started();
+            let mut non_trigger_opts = opts.clone();
+            non_trigger_opts
+                .lifecycle_delete_all
+                .as_mut()
+                .ok_or(StorageError::PreconditionFailed)?
+                .phase = crate::object_api::LifecycleDeleteAllPhase::History;
+            for pool in &self.pools {
+                #[cfg(test)]
+                lifecycle_delete_all_test_failure(crate::object_api::LifecycleDeleteAllPhase::History, pool.pool_idx)?;
+                let mut pool_opts = non_trigger_opts.clone();
+                pool_opts.delete_prefix = true;
+                pool.delete_object(bucket, object, pool_opts).await?;
+            }
+
+            let mut final_preflight_opts = opts.clone();
+            final_preflight_opts
+                .lifecycle_delete_all
+                .as_mut()
+                .ok_or(StorageError::PreconditionFailed)?
+                .phase = crate::object_api::LifecycleDeleteAllPhase::FinalPreflight;
+            let mut trigger_pools = Vec::new();
+            for (pool_index, pool) in self.pools.iter().enumerate() {
+                #[cfg(test)]
+                lifecycle_delete_all_test_failure(crate::object_api::LifecycleDeleteAllPhase::FinalPreflight, pool.pool_idx)?;
+                let result = pool.delete_object(bucket, object, final_preflight_opts.clone()).await?;
+                if !result.name.is_empty() {
+                    trigger_pools.push(pool_index);
+                }
+            }
+            if trigger_pools.is_empty() {
+                return Err(StorageError::PreconditionFailed);
+            }
+
+            let mut trigger_opts = opts.clone();
+            trigger_opts
+                .lifecycle_delete_all
+                .as_mut()
+                .ok_or(StorageError::PreconditionFailed)?
+                .phase = crate::object_api::LifecycleDeleteAllPhase::Trigger;
+            for pool_index in trigger_pools {
+                #[cfg(test)]
+                lifecycle_delete_all_test_failure(crate::object_api::LifecycleDeleteAllPhase::Trigger, pool_index)?;
+                let mut pool_opts = trigger_opts.clone();
+                pool_opts.delete_prefix = true;
+                self.pools[pool_index].delete_object(bucket, object, pool_opts).await?;
+            }
+            return Ok(());
+        }
+
        let mut first_error = None;
        let mut first_volume_error = None;
        let mut has_success = false;
-        for pool in self.pools.iter() {
+        for pool in &self.pools {
            let mut opts = opts.clone();
            opts.delete_prefix = true;
            match pool.delete_object(bucket, object, opts).await {
@@ -774,6 +840,22 @@ impl ECStore {
    }
 }

+#[cfg(test)]
+static LIFECYCLE_DELETE_ALL_TEST_FAILURE: std::sync::Mutex<Option<(crate::object_api::LifecycleDeleteAllPhase, usize)>> =
+    std::sync::Mutex::new(None);
+
+#[cfg(test)]
+fn lifecycle_delete_all_test_failure(phase: crate::object_api::LifecycleDeleteAllPhase, pool_index: usize) -> Result<()> {
+    if LIFECYCLE_DELETE_ALL_TEST_FAILURE
+        .lock()
+        .expect("lifecycle delete-all failure hook should not poison")
+        .is_some_and(|failure| failure == (phase, pool_index))
+    {
+        return Err(StorageError::PreconditionFailed);
+    }
+    Ok(())
+}
+
 #[cfg(test)]
 mod tests {
    use super::*;
@@ -781,23 +863,28 @@ mod tests {
    use crate::disk::error::DiskError;
    use crate::layout::endpoint::Endpoint;
    use crate::layout::endpoints::{EndpointServerPools, Endpoints, PoolEndpoints};
+    use crate::object_api::ObjectLockConfigSnapshot;
    use crate::storage_api_contracts::bucket::MakeBucketOptions;
+    use crate::storage_api_contracts::object::ObjectIO as _;
    use arc_swap::ArcSwap;
    use rustfs_config::server_config::KVS;
+    use rustfs_filemeta::FileInfo;
    use std::sync::Arc;
    use tokio_util::sync::CancellationToken;

-    #[tokio::test]
-    async fn delete_prefix_attempts_later_pools_after_an_earlier_pool_error() {
-        let temp_dir = tempfile::tempdir().expect("multi-pool delete test directory should be created");
-        let mut pools = Vec::with_capacity(2);
-        for (pool_index, drives_per_set) in [2, 4].into_iter().enumerate() {
+    async fn setup_multi_pool_test_store(
+        name: &str,
+        drives_per_pool: &[usize],
+    ) -> (tempfile::TempDir, Arc<ECStore>, CancellationToken) {
+        let temp_dir = tempfile::tempdir().expect("multi-pool test directory should be created");
+        let mut pools = Vec::with_capacity(drives_per_pool.len());
+        for (pool_index, drives_per_set) in drives_per_pool.iter().copied().enumerate() {
            let mut endpoints = Vec::with_capacity(drives_per_set);
            for disk_index in 0..drives_per_set {
                let disk_path = temp_dir.path().join(format!("pool{pool_index}-disk{disk_index}"));
                tokio::fs::create_dir_all(&disk_path)
                    .await
-                    .expect("multi-pool delete test disk should be created");
+                    .expect("multi-pool test disk should be created");
                let mut endpoint =
                    Endpoint::try_from(disk_path.to_str().expect("disk path should be utf8")).expect("endpoint should parse");
                endpoint.set_pool_index(pool_index);
@@ -810,7 +897,7 @@ mod tests {
                set_count: 1,
                drives_per_set,
                endpoints: Endpoints::from(endpoints),
-                cmd_line: format!("delete-prefix-pool-{pool_index}"),
+                cmd_line: format!("{name}-pool-{pool_index}"),
                platform: "test".to_string(),
            });
        }
@@ -830,6 +917,92 @@ mod tests {
        .await
        .expect("multi-pool store should initialize");
        crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await;
+        (temp_dir, store, shutdown)
+    }
+
+    struct LifecycleDeleteAllFailureGuard;
+
+    impl Drop for LifecycleDeleteAllFailureGuard {
+        fn drop(&mut self) {
+            *LIFECYCLE_DELETE_ALL_TEST_FAILURE
+                .lock()
+                .expect("lifecycle delete-all failure hook should not poison") = None;
+        }
+    }
+
+    async fn seed_multi_pool_delete_all(store: &Arc<ECStore>, bucket: &str, object: &str) -> ObjectOptions {
+        let trigger_id = Uuid::new_v4();
+        for (pool_index, pool) in store.pools.iter().enumerate() {
+            let mut history_reader = PutObjReader::from_vec(format!("{object}-history-{pool_index}").into_bytes());
+            pool.put_object(
+                bucket,
+                object,
+                &mut history_reader,
+                &ObjectOptions {
+                    versioned: true,
+                    version_id: Some(Uuid::new_v4().to_string()),
+                    mod_time: Some(OffsetDateTime::UNIX_EPOCH + time::Duration::seconds(1)),
+                    ..Default::default()
+                },
+            )
+            .await
+            .expect("history should be stored");
+            let mut trigger_reader = PutObjReader::from_vec(format!("{object}-trigger-{pool_index}").into_bytes());
+            pool.put_object(
+                bucket,
+                object,
+                &mut trigger_reader,
+                &ObjectOptions {
+                    versioned: true,
+                    version_id: Some(trigger_id.to_string()),
+                    mod_time: Some(OffsetDateTime::UNIX_EPOCH + time::Duration::seconds(2)),
+                    ..Default::default()
+                },
+            )
+            .await
+            .expect("shared trigger should be stored");
+        }
+        let mut opts = ObjectOptions {
+            delete_prefix: true,
+            delete_prefix_object: true,
+            versioned: true,
+            lifecycle_delete_all: Some(crate::object_api::LifecycleDeleteAllRequest {
+                version_id: Some(trigger_id),
+                delete_marker: false,
+                action: rustfs_common::metrics::IlmAction::DeleteAllVersionsAction,
+                rule_id: "rule".to_string(),
+                phase: crate::object_api::LifecycleDeleteAllPhase::Preflight,
+            }),
+            object_lock_config_snapshot: Some(Arc::new(ObjectLockConfigSnapshot::new(
+                crate::bucket::metadata_sys::ObjectLockConfigState::ConfirmedAbsent,
+            ))),
+            delete_replication_config_snapshot: Some(Arc::new(
+                crate::bucket::replication::DeleteReplicationConfigSnapshot::default(),
+            )),
+            ..Default::default()
+        };
+        opts.ensure_lifecycle_delete_all_journal();
+        opts
+    }
+
+    async fn ordinary_version_count(store: &ECStore, pool_index: usize, bucket: &str, object: &str) -> usize {
+        store.pools[pool_index].disk_set[0]
+            .load_file_info_versions_exact(bucket, object)
+            .await
+            .expect("pool metadata should load")
+            .map(|versions| {
+                versions
+                    .versions
+                    .iter()
+                    .filter(|version| !version.tier_free_version())
+                    .count()
+            })
+            .unwrap_or_default()
+    }
+
+    #[tokio::test]
+    async fn delete_prefix_attempts_later_pools_after_an_earlier_pool_error() {
+        let (_temp_dir, store, shutdown) = setup_multi_pool_test_store("delete-prefix", &[2, 4]).await;
        let bucket = format!("delete-prefix-{}", Uuid::new_v4().simple());
        store
            .make_bucket(&bucket, &MakeBucketOptions::default())
@@ -921,6 +1094,171 @@ mod tests {
        shutdown.cancel();
    }

+    #[tokio::test]
+    #[serial_test::serial]
+    async fn lifecycle_delete_all_history_failure_preserves_trigger_and_retry_converges() {
+        let (_temp_dir, store, shutdown) = setup_multi_pool_test_store("lifecycle-delete-all", &[4, 4]).await;
+        let bucket = format!("lifecycle-delete-all-{}", Uuid::new_v4().simple());
+        let object = "object";
+        store
+            .make_bucket(&bucket, &MakeBucketOptions::default())
+            .await
+            .expect("bucket should be created in both pools");
+
+        for pool_index in 0..2 {
+            let mut reader = PutObjReader::from_vec(format!("pool-{pool_index}-history").into_bytes());
+            store.pools[pool_index]
+                .put_object(
+                    &bucket,
+                    object,
+                    &mut reader,
+                    &ObjectOptions {
+                        versioned: true,
+                        ..Default::default()
+                    },
+                )
+                .await
+                .expect("historical version should be stored");
+        }
+        let marker = store.pools[0]
+            .delete_object(
+                &bucket,
+                object,
+                ObjectOptions {
+                    versioned: true,
+                    ..Default::default()
+                },
+            )
+            .await
+            .expect("trigger marker should be stored in the first pool");
+        let marker_id = marker.version_id.expect("trigger marker should have a version id");
+        let mut opts = ObjectOptions {
+            delete_prefix: true,
+            delete_prefix_object: true,
+            versioned: true,
+            lifecycle_delete_all: Some(crate::object_api::LifecycleDeleteAllRequest {
+                version_id: Some(marker_id),
+                delete_marker: true,
+                action: rustfs_common::metrics::IlmAction::DelMarkerDeleteAllVersionsAction,
+                rule_id: "rule".to_string(),
+                phase: crate::object_api::LifecycleDeleteAllPhase::Preflight,
+            }),
+            object_lock_config_snapshot: Some(Arc::new(ObjectLockConfigSnapshot::new(
+                crate::bucket::metadata_sys::ObjectLockConfigState::ConfirmedAbsent,
+            ))),
+            delete_replication_config_snapshot: Some(Arc::new(
+                crate::bucket::replication::DeleteReplicationConfigSnapshot::default(),
+            )),
+            ..Default::default()
+        };
+        opts.ensure_lifecycle_delete_all_journal();
+
+        let _failure_guard = LifecycleDeleteAllFailureGuard;
+        *LIFECYCLE_DELETE_ALL_TEST_FAILURE
+            .lock()
+            .expect("lifecycle delete-all failure hook should not poison") =
+            Some((crate::object_api::LifecycleDeleteAllPhase::History, 1));
+        let err = store
+            .delete_prefix(&bucket, object, &opts)
+            .await
+            .expect_err("a later pool history failure must stop before trigger deletion");
+        assert_eq!(err, StorageError::PreconditionFailed);
+        assert!(
+            opts.lifecycle_delete_all_journal()
+                .expect("delete-all journal should be initialized")
+                .lock()
+                .mutation_started()
+        );
+
+        let first_pool = store.pools[0].disk_set[0]
+            .load_file_info_versions_exact(&bucket, object)
+            .await
+            .expect("first pool metadata should load")
+            .expect("the trigger should remain");
+        let first_pool_ordinary: Vec<&FileInfo> = first_pool
+            .versions
+            .iter()
+            .filter(|version| !version.tier_free_version())
+            .collect();
+        assert_eq!(first_pool_ordinary.len(), 1);
+        assert_eq!(first_pool_ordinary[0].version_id, Some(marker_id));
+        assert!(first_pool_ordinary[0].deleted);
+
+        *LIFECYCLE_DELETE_ALL_TEST_FAILURE
+            .lock()
+            .expect("lifecycle delete-all failure hook should not poison") = None;
+        store
+            .delete_prefix(&bucket, object, &opts)
+            .await
+            .expect("retry should delete remaining history and its trigger owner");
+        for pool in &store.pools {
+            assert!(
+                pool.disk_set[0]
+                    .load_file_info_versions_exact(&bucket, object)
+                    .await
+                    .expect("pool metadata should load after retry")
+                    .is_none(),
+                "all ordinary versions should be removed after retry"
+            );
+        }
+
+        shutdown.cancel();
+    }
+
+    #[tokio::test]
+    #[serial_test::serial]
+    async fn lifecycle_delete_all_phase_failures_preserve_barriers_and_retry() {
+        let (_temp_dir, store, shutdown) = setup_multi_pool_test_store("lifecycle-delete-all-phases", &[4, 4]).await;
+        let bucket = format!("lifecycle-delete-all-phases-{}", Uuid::new_v4().simple());
+        store
+            .make_bucket(&bucket, &MakeBucketOptions::default())
+            .await
+            .expect("bucket should be created in both pools");
+        let _failure_guard = LifecycleDeleteAllFailureGuard;
+
+        for (object, phase, expected_counts, mutation_started) in [
+            ("preflight-failure", crate::object_api::LifecycleDeleteAllPhase::Preflight, [2, 2], false),
+            (
+                "final-preflight-failure",
+                crate::object_api::LifecycleDeleteAllPhase::FinalPreflight,
+                [1, 1],
+                true,
+            ),
+            ("trigger-failure", crate::object_api::LifecycleDeleteAllPhase::Trigger, [0, 1], true),
+        ] {
+            let opts = seed_multi_pool_delete_all(&store, &bucket, object).await;
+            *LIFECYCLE_DELETE_ALL_TEST_FAILURE
+                .lock()
+                .expect("lifecycle delete-all failure hook should not poison") = Some((phase, 1));
+            let err = store
+                .delete_prefix(&bucket, object, &opts)
+                .await
+                .expect_err("injected phase failure should stop the transaction");
+            assert_eq!(err, StorageError::PreconditionFailed);
+            assert_eq!(
+                opts.lifecycle_delete_all_journal()
+                    .expect("delete-all journal should be initialized")
+                    .lock()
+                    .mutation_started(),
+                mutation_started
+            );
+            assert_eq!(ordinary_version_count(&store, 0, &bucket, object).await, expected_counts[0]);
+            assert_eq!(ordinary_version_count(&store, 1, &bucket, object).await, expected_counts[1]);
+
+            *LIFECYCLE_DELETE_ALL_TEST_FAILURE
+                .lock()
+                .expect("lifecycle delete-all failure hook should not poison") = None;
+            store
+                .delete_prefix(&bucket, object, &opts)
+                .await
+                .expect("retry should converge after the injected failure is removed");
+            assert_eq!(ordinary_version_count(&store, 0, &bucket, object).await, 0);
+            assert_eq!(ordinary_version_count(&store, 1, &bucket, object).await, 0);
+        }
+
+        shutdown.cancel();
+    }
+
    fn assert_backend_layout_empty(info: &rustfs_madmin::BackendInfo) {
        assert!(info.standard_sc_parities.is_empty());
        assert!(info.standard_sc_data.is_empty());