fix(targets): make persistent queue store crash-safe and replay lifecycle correct (#4505)

* fix(targets): make persistent queue store crash-safe and replay lifecycle correct

Harden the target notification persistent queue (store.rs) and the replay
worker lifecycle (runtime) against data loss, silent truncation, ordering
drift, orphaned tasks, and a few low-risk robustness gaps.

store.rs
- Atomic, durable writes: write to a per-key temp file, fsync (sync_all),
  then rename into place; best-effort parent-dir fsync. A crash mid-write
  can no longer lose an acknowledged event or leave a half-written payload
  that reads as a valid entry.
- open() now removes leftover .tmp residue and zero-byte files, and only
  indexes files matching the queue extension, so ghosts/foreign files are
  never replayed.
- FIFO ordering is derived from time-ordered UUIDv7 entry names instead of
  coarse, clock-dependent file mtimes, so replay order is stable and
  identical after a restart.
- Clamp HashMap/Vec pre-allocation derived from untrusted inputs
  (entry_limit, batch item_count) to avoid capacity-overflow panics / giant
  allocations.

target/mod.rs
- QueuedPayload::decode validates body length against the recorded
  payload_len, rejecting torn/truncated writes instead of delivering a
  silently truncated body.
- send_from_store purges a NotFound/empty entry (index + file) instead of
  skipping it, so it cannot occupy a queue slot and be replayed forever.
- sanitize_queue_dir_component appends a stable hash suffix when the id was
  lossy, so distinct target ids can no longer collapse onto the same queue
  directory; path-safe ids are unchanged (no migration).

runtime
- Replay backoff, idle waits, and inter-scan pauses are now cancel-aware, so
  reload/shutdown is not blocked for the full retry delay.
- ReplayWorkerManager::stop_all signals cancellation and then awaits each
  worker's exit (bounded, with abort fallback), preventing orphaned tasks
  and overlapping drain of the same store.
- Fix the always-true replay flush condition so batching is real
  (size/timeout based, one semaphore permit per batch) rather than one
  permit per entry; dedup keys already pending in the batch.
- clear_and_close aggregates and reports per-target close failures instead
  of swallowing them; explicit shutdown surfaces them.

Relates to rustfs/backlog#966
Relates to rustfs/backlog#967
Relates to rustfs/backlog#975
Relates to rustfs/backlog#970
Relates to rustfs/backlog#983

Co-Authored-By: heihutu <heihutu@gmail.com>

* style(targets): apply rustfmt to replay batch dedup guard

Fixes the Quick Checks rustfmt failure on the multi-line `.iter().any(...)`
closure in the replay batch dedup guard.

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>

crates/targets/src/runtime/adapter.rs

@@ -142,8 +142,15 @@ where
         replay_workers: &mut ReplayWorkerManager,
         activation: RuntimeActivation<E>,
     ) -> Result<(), TargetError> {
+        // Stop (and join) the old replay workers before installing the new set so
+        // no two workers ever drain the same store concurrently, then close the
+        // old targets. A close failure during reload is logged but does not abort
+        // the reload — the new configuration must still take effect.
         self.stop_replay_workers(replay_workers).await;
-        runtime.clear_and_close().await;
+        let close_errors = runtime.clear_and_close().await;
+        if !close_errors.is_empty() {
+            tracing::warn!(failed_targets = close_errors.len(), "Some targets failed to close during runtime reload");
+        }
 
         for target in activation.targets {
             runtime.add_arc(target);
@@ -174,8 +181,19 @@ where
         runtime: &mut TargetRuntimeManager<E>,
         replay_workers: &mut ReplayWorkerManager,
     ) -> Result<(), TargetError> {
+        // On explicit shutdown, propagate any close/flush failures instead of
+        // swallowing them: the runtime is still fully torn down, but the caller
+        // learns that a target could not be flushed/closed cleanly.
         self.stop_replay_workers(replay_workers).await;
-        runtime.clear_and_close().await;
+        let close_errors = runtime.clear_and_close().await;
+        if !close_errors.is_empty() {
+            let detail = close_errors
+                .into_iter()
+                .map(|(target_id, err)| format!("{target_id}: {err}"))
+                .collect::<Vec<_>>()
+                .join("; ");
+            return Err(TargetError::Storage(format!("Failed to close {detail}")));
+        }
         Ok(())
     }
 }