test(targets,notify,audit): share a builder MockTarget testkit (#6717)

* test(targets): ship a builder MockTarget testkit and retire the in-crate Target mocks

Adds crates/targets/src/testkit.rs with a builder-style MockTarget implementing Target<E> for every E: PluginEvent, with orthogonal off-by-default knobs: disabled/active override, health delay plus health-started signal plus a drop-guard counter proving a cancelled probe future was dropped, an init failure budget (usize::MAX = always fail) plus blocking init plus an init counter, a close counter/signal/semaphore gate with a runtime block toggle, a save counter plus save failure budget, caller-supplied store and failed-store handles, and a shared final-failure counter. Clones and clone_dyn share all counters, so an observer clone keeps watching a target after it is boxed into a runtime.

The module is gated as #[cfg(any(test, feature = "test-support"))]: in-crate unit tests get it via cfg(test), and downstream test suites opt in through the new off-by-default test-support cargo feature (test-support = [], activating no dependencies).

Migrates the five in-crate duplicate mocks onto it: the plugin.rs registry-factory TestTarget, the runtime/adapter.rs lifecycle TestTarget (init/close/store knobs), the runtime/mod.rs TestTarget plus HealthDropGuard (close gating and health-probe tests), the target/mod.rs MoveTestTarget (folded into the test_support helper constructors used by the NATS JetStream failed-store tests), and the target/mod.rs StoreBackedTarget (the default send_from_store purge test; the mock deliberately does not override send_from_store or handle_terminal_failure). The forced init failure now uses TargetError::Initialization instead of the old adapter mock's Configuration; the adapter derives its redacted failure summary from the target id alone, so the migrated assertions are unchanged.

Leak guard: testkit unit tests assert the crate manifest still declares default = [] and that test-support = [] stays a pure cfg gate, complementing the compile-level cfg gate that keeps the mock out of production builds.

Part of rustfs/backlog#1846 (cluster 3, step 1).

* test(notify,audit): migrate the Target mocks onto the shared testkit

Retires the hand-written Target mocks in crates/notify and crates/audit in favor of rustfs_targets::testkit::MockTarget: notify's notifier.rs TestTarget/DeferredTestTarget/ClosableTestTarget, lifecycle.rs BlockingInitTarget/RetryInitTarget (rebuilt as observed MockTarget templates cloned by their plugin-descriptor factories, sharing the init signal, close counter, and single-failure init budget across generations), runtime_view.rs TestTarget, runtime_facade.rs TestTarget, and audit's pipeline.rs MockTarget, system.rs TestTarget, registry.rs CloseTestTarget, plus TestTarget and FailingTarget in audit/tests/pipeline_layer_test.rs. Removing the two integration-test mocks also removes their respelled PluginEvent bound (E: Send + Sync + 'static + Clone + Serialize + DeserializeOwned), which the plugin-contract rules require to be spelled only via PluginEvent.

lifecycle.rs ReplayTarget stays bespoke on purpose: its generation tags, mpsc observation channels, gated send_raw delivery, and ObservedQueueStore model the replay pipeline itself and would contort a general-purpose mock. The other bespoke mocks named out of scope in PR-3a (ProgrammedTarget, ClassifyingTarget, the ReloadableTargetTls fakes) are likewise untouched.

New testkit knobs, each defaulted off and unit-tested: with_id (rename a clone while keeping the shared counters, for factory templates), with_first_save_gate (the first save notifies entered and waits on release; several mocks may share one pair), with_health_gate (is_active waits on a release handle after notifying health_started), with_delivery_snapshot (fixed snapshot overriding the store-derived default), with_close_failures (close-failure budget, default TargetError::Storage) with with_close_failure_error to shape the variant (audit's registry test pins TargetError::Unknown), and an always-on is_enabled call counter exposed as enabled_call_count (notifier's generation tests count dispatcher selections through it).

Both crates enable the testkit through a dev-dependency on rustfs-targets with the test-support feature; the feature stays out of default and activates no dependencies, so production builds are unchanged.

Part of rustfs/backlog#1846 (cluster 3, step 2).

crates/audit/tests/pipeline_layer_test.rs

@@ -12,136 +12,16 @@
 //  See the License for the specific language governing permissions and
 //  limitations under the License.
 
-use async_trait::async_trait;
 use rustfs_audit::{AuditEntry, AuditError, AuditPipeline, AuditRegistry, AuditRuntimeFacade, AuditRuntimeView};
-use rustfs_targets::arn::TargetID;
-use rustfs_targets::store::{Key, Store};
-use rustfs_targets::target::{EntityTarget, QueuedPayload, QueuedPayloadMeta};
-use rustfs_targets::{SharedTarget, StoreError, Target, TargetError};
-use serde::{Serialize, de::DeserializeOwned};
+use rustfs_targets::SharedTarget;
+use rustfs_targets::testkit::MockTarget;
 use std::sync::Arc;
-use std::sync::atomic::{AtomicUsize, Ordering};
 use tokio::sync::{Mutex, RwLock};
 
-#[derive(Clone)]
-struct TestTarget {
-    close_calls: Arc<AtomicUsize>,
-    id: TargetID,
-    init_calls: Arc<AtomicUsize>,
-}
-
-impl TestTarget {
-    fn new(id: &str, name: &str) -> Self {
-        Self {
-            close_calls: Arc::new(AtomicUsize::new(0)),
-            id: TargetID::new(id.to_string(), name.to_string()),
-            init_calls: Arc::new(AtomicUsize::new(0)),
-        }
-    }
-}
-
-#[async_trait]
-impl<E> Target<E> for TestTarget
-where
-    E: Send + Sync + 'static + Clone + Serialize + DeserializeOwned,
-{
-    fn id(&self) -> TargetID {
-        self.id.clone()
-    }
-
-    async fn is_active(&self) -> Result<bool, TargetError> {
-        Ok(true)
-    }
-
-    async fn save(&self, _event: Arc<EntityTarget<E>>) -> Result<(), TargetError> {
-        Ok(())
-    }
-
-    async fn send_raw_from_store(&self, _key: Key, _body: Vec<u8>, _meta: QueuedPayloadMeta) -> Result<(), TargetError> {
-        Ok(())
-    }
-
-    async fn close(&self) -> Result<(), TargetError> {
-        self.close_calls.fetch_add(1, Ordering::SeqCst);
-        Ok(())
-    }
-
-    fn store(&self) -> Option<&(dyn Store<QueuedPayload, Error = StoreError, Key = Key> + Send + Sync)> {
-        None
-    }
-
-    fn clone_dyn(&self) -> Box<dyn Target<E> + Send + Sync> {
-        Box::new(self.clone())
-    }
-
-    async fn init(&self) -> Result<(), TargetError> {
-        self.init_calls.fetch_add(1, Ordering::SeqCst);
-        Ok(())
-    }
-
-    fn is_enabled(&self) -> bool {
-        true
-    }
-}
-
-/// A target whose `save()` always fails, used to exercise the dispatch
+/// Builds a target whose `save()` always fails, used to exercise the dispatch
 /// failure-propagation paths.
-#[derive(Clone)]
-struct FailingTarget {
-    id: TargetID,
-    save_calls: Arc<AtomicUsize>,
-}
-
-impl FailingTarget {
-    fn new(id: &str, name: &str) -> Self {
-        Self {
-            id: TargetID::new(id.to_string(), name.to_string()),
-            save_calls: Arc::new(AtomicUsize::new(0)),
-        }
-    }
-}
-
-#[async_trait]
-impl<E> Target<E> for FailingTarget
-where
-    E: Send + Sync + 'static + Clone + Serialize + DeserializeOwned,
-{
-    fn id(&self) -> TargetID {
-        self.id.clone()
-    }
-
-    async fn is_active(&self) -> Result<bool, TargetError> {
-        Ok(true)
-    }
-
-    async fn save(&self, _event: Arc<EntityTarget<E>>) -> Result<(), TargetError> {
-        self.save_calls.fetch_add(1, Ordering::SeqCst);
-        Err(TargetError::Storage("disk full".to_string()))
-    }
-
-    async fn send_raw_from_store(&self, _key: Key, _body: Vec<u8>, _meta: QueuedPayloadMeta) -> Result<(), TargetError> {
-        Ok(())
-    }
-
-    async fn close(&self) -> Result<(), TargetError> {
-        Ok(())
-    }
-
-    fn store(&self) -> Option<&(dyn Store<QueuedPayload, Error = StoreError, Key = Key> + Send + Sync)> {
-        None
-    }
-
-    fn clone_dyn(&self) -> Box<dyn Target<E> + Send + Sync> {
-        Box::new(self.clone())
-    }
-
-    async fn init(&self) -> Result<(), TargetError> {
-        Ok(())
-    }
-
-    fn is_enabled(&self) -> bool {
-        true
-    }
+fn failing_target(id: &str, name: &str) -> MockTarget {
+    MockTarget::new(id, name).with_save_failures(usize::MAX)
 }
 
 fn pipeline_with_targets(targets: Vec<(&str, SharedTarget<AuditEntry>)>) -> AuditPipeline {
@@ -154,8 +34,8 @@ fn pipeline_with_targets(targets: Vec<(&str, SharedTarget<AuditEntry>)>) -> Audi
 
 #[tokio::test]
 async fn audit_pipeline_dispatch_propagates_total_failure() {
-    let failing = FailingTarget::new("primary", "webhook");
-    let save_calls = Arc::clone(&failing.save_calls);
+    let failing = failing_target("primary", "webhook");
+    let observer = failing.clone();
     let pipeline = pipeline_with_targets(vec![("primary:webhook", Arc::new(failing))]);
 
     let result = pipeline.dispatch(Arc::new(AuditEntry::default())).await;
@@ -164,13 +44,13 @@ async fn audit_pipeline_dispatch_propagates_total_failure() {
         matches!(result, Err(AuditError::Target(_))),
         "dispatch must surface an error when every target fails, got {result:?}"
     );
-    assert_eq!(save_calls.load(Ordering::SeqCst), 1, "the failing target should have been invoked");
+    assert_eq!(observer.save_call_count(), 1, "the failing target should have been invoked");
 }
 
 #[tokio::test]
 async fn audit_pipeline_dispatch_tolerates_partial_failure() {
-    let failing = FailingTarget::new("primary", "webhook");
-    let healthy = TestTarget::new("secondary", "webhook");
+    let failing = failing_target("primary", "webhook");
+    let healthy = MockTarget::new("secondary", "webhook");
     let pipeline = pipeline_with_targets(vec![
         ("primary:webhook", Arc::new(failing)),
         ("secondary:webhook", Arc::new(healthy)),
@@ -186,7 +66,7 @@ async fn audit_pipeline_dispatch_tolerates_partial_failure() {
 
 #[tokio::test]
 async fn audit_pipeline_dispatch_batch_propagates_total_failure() {
-    let failing = FailingTarget::new("primary", "webhook");
+    let failing = failing_target("primary", "webhook");
     let pipeline = pipeline_with_targets(vec![("primary:webhook", Arc::new(failing))]);
 
     let entries = vec![Arc::new(AuditEntry::default()), Arc::new(AuditEntry::default())];
@@ -200,8 +80,8 @@ async fn audit_pipeline_dispatch_batch_propagates_total_failure() {
 
 #[tokio::test]
 async fn audit_pipeline_dispatch_batch_tolerates_partial_failure() {
-    let failing = FailingTarget::new("primary", "webhook");
-    let healthy = TestTarget::new("secondary", "webhook");
+    let failing = failing_target("primary", "webhook");
+    let healthy = MockTarget::new("secondary", "webhook");
     let pipeline = pipeline_with_targets(vec![
         ("primary:webhook", Arc::new(failing)),
         ("secondary:webhook", Arc::new(healthy)),
@@ -266,9 +146,8 @@ async fn audit_runtime_facade_activates_empty_target_list() {
 async fn audit_runtime_view_upsert_and_remove_target() {
     let registry = Arc::new(Mutex::new(AuditRegistry::new()));
     let runtime_view = AuditRuntimeView::new(registry.clone());
-    let target = TestTarget::new("primary", "webhook");
-    let init_calls = Arc::clone(&target.init_calls);
-    let close_calls = Arc::clone(&target.close_calls);
+    let target = MockTarget::new("primary", "webhook");
+    let observer = target.clone();
 
     runtime_view
         .upsert_target("primary:webhook".to_string(), Box::new(target))
@@ -276,7 +155,7 @@ async fn audit_runtime_view_upsert_and_remove_target() {
         .expect("upsert should succeed");
 
     assert_eq!(runtime_view.list_targets().await, vec!["primary:webhook".to_string()]);
-    assert_eq!(init_calls.load(Ordering::SeqCst), 1);
+    assert_eq!(observer.init_call_count(), 1);
 
     runtime_view
         .remove_target("primary:webhook")
@@ -284,7 +163,7 @@ async fn audit_runtime_view_upsert_and_remove_target() {
         .expect("remove should succeed");
 
     assert!(runtime_view.list_targets().await.is_empty());
-    assert_eq!(close_calls.load(Ordering::SeqCst), 1);
+    assert_eq!(observer.close_call_count(), 1);
 }
 
 #[tokio::test]
@@ -292,7 +171,7 @@ async fn audit_runtime_facade_replace_targets_commits_runtime_state() {
     let registry = Arc::new(Mutex::new(AuditRegistry::new()));
     let replay_workers = Arc::new(RwLock::new(rustfs_targets::ReplayWorkerManager::new()));
     let facade = AuditRuntimeFacade::new(registry.clone(), replay_workers.clone());
-    let target = TestTarget::new("primary", "webhook");
+    let target = MockTarget::new("primary", "webhook");
     let activation = rustfs_targets::RuntimeActivation {
         replay_workers: rustfs_targets::ReplayWorkerManager::new(),
         targets: vec![Arc::new(target) as rustfs_targets::SharedTarget<rustfs_audit::AuditEntry>],