* test(targets): ship a builder MockTarget testkit and retire the in-crate Target mocks
Adds crates/targets/src/testkit.rs with a builder-style MockTarget implementing Target<E> for every E: PluginEvent, with orthogonal off-by-default knobs: disabled/active override, health delay plus health-started signal plus a drop-guard counter proving a cancelled probe future was dropped, an init failure budget (usize::MAX = always fail) plus blocking init plus an init counter, a close counter/signal/semaphore gate with a runtime block toggle, a save counter plus save failure budget, caller-supplied store and failed-store handles, and a shared final-failure counter. Clones and clone_dyn share all counters, so an observer clone keeps watching a target after it is boxed into a runtime.
The module is gated as #[cfg(any(test, feature = "test-support"))]: in-crate unit tests get it via cfg(test), and downstream test suites opt in through the new off-by-default test-support cargo feature (test-support = [], activating no dependencies).
Migrates the five in-crate duplicate mocks onto it: the plugin.rs registry-factory TestTarget, the runtime/adapter.rs lifecycle TestTarget (init/close/store knobs), the runtime/mod.rs TestTarget plus HealthDropGuard (close gating and health-probe tests), the target/mod.rs MoveTestTarget (folded into the test_support helper constructors used by the NATS JetStream failed-store tests), and the target/mod.rs StoreBackedTarget (the default send_from_store purge test; the mock deliberately does not override send_from_store or handle_terminal_failure). The forced init failure now uses TargetError::Initialization instead of the old adapter mock's Configuration; the adapter derives its redacted failure summary from the target id alone, so the migrated assertions are unchanged.
Leak guard: testkit unit tests assert the crate manifest still declares default = [] and that test-support = [] stays a pure cfg gate, complementing the compile-level cfg gate that keeps the mock out of production builds.
Part of rustfs/backlog#1846 (cluster 3, step 1).
* test(notify,audit): migrate the Target mocks onto the shared testkit
Retires the hand-written Target mocks in crates/notify and crates/audit in favor of rustfs_targets::testkit::MockTarget: notify's notifier.rs TestTarget/DeferredTestTarget/ClosableTestTarget, lifecycle.rs BlockingInitTarget/RetryInitTarget (rebuilt as observed MockTarget templates cloned by their plugin-descriptor factories, sharing the init signal, close counter, and single-failure init budget across generations), runtime_view.rs TestTarget, runtime_facade.rs TestTarget, and audit's pipeline.rs MockTarget, system.rs TestTarget, registry.rs CloseTestTarget, plus TestTarget and FailingTarget in audit/tests/pipeline_layer_test.rs. Removing the two integration-test mocks also removes their respelled PluginEvent bound (E: Send + Sync + 'static + Clone + Serialize + DeserializeOwned), which the plugin-contract rules require to be spelled only via PluginEvent.
lifecycle.rs ReplayTarget stays bespoke on purpose: its generation tags, mpsc observation channels, gated send_raw delivery, and ObservedQueueStore model the replay pipeline itself and would contort a general-purpose mock. The other bespoke mocks named out of scope in PR-3a (ProgrammedTarget, ClassifyingTarget, the ReloadableTargetTls fakes) are likewise untouched.
New testkit knobs, each defaulted off and unit-tested: with_id (rename a clone while keeping the shared counters, for factory templates), with_first_save_gate (the first save notifies entered and waits on release; several mocks may share one pair), with_health_gate (is_active waits on a release handle after notifying health_started), with_delivery_snapshot (fixed snapshot overriding the store-derived default), with_close_failures (close-failure budget, default TargetError::Storage) with with_close_failure_error to shape the variant (audit's registry test pins TargetError::Unknown), and an always-on is_enabled call counter exposed as enabled_call_count (notifier's generation tests count dispatcher selections through it).
Both crates enable the testkit through a dev-dependency on rustfs-targets with the test-support feature; the feature stays out of default and activates no dependencies, so production builds are unchanged.
Part of rustfs/backlog#1846 (cluster 3, step 2).
`audit_runtime_facade_stops_empty_replay_workers` called the stop path and checked nothing, the same shape as the notify facade test in the previous commit. It now asserts the worker manager is empty afterwards and that a second call — which shutdown paths make — stays harmless.
The two heal timestamp tests bound their fields to `_`. Both timestamps come from `SystemTime::now().duration_since(UNIX_EPOCH).unwrap_or_default()`, so a pre-epoch clock yields 0; binding to `_` could not tell that apart from a real reading, which is precisely what the tests said they were guarding. They now require the value to be past 2020-01-01, and `last_update` not to predate `start_time`.
`test_config_parsing_with_multiple_instances` is left as it was — see the issue comment. Asserting on it turned up something bigger than a missing assertion.
Refs backlog#1836
fix(audit): propagate dispatch delivery failures instead of swallowing them (backlog#962)
AuditPipeline::dispatch and dispatch_batch accumulated per-target save()
errors, logged them, and then unconditionally returned Ok(()). When every
configured target failed the audit entry was lost outright (for store-backed
targets a failed save() means the event was neither delivered nor persisted
for replay), yet callers such as dispatch_audit_log saw success and assumed
the audit trail was intact. Audit is a compliance-critical path, so a total
delivery failure that reports success is a silent data-loss bug.
Both methods now distinguish three outcomes: all targets succeeded (Ok),
partial failure where at least one target accepted the event (log a warning
and return Ok, since the entry is not lost), and total failure where no
delivery succeeded while errors were recorded (record the failure metric,
log an error, and return Err(AuditError::Target) carrying the first target
error). This lets the caller react instead of assuming success.
Adds regression tests with a FailingTarget mock asserting that dispatch and
dispatch_batch return Err on total failure and Ok on partial failure.
Zhengchao An