diff --git a/CHANGELOG.md b/CHANGELOG.md index b1cfeec37..05b19e177 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **Helm Ingress**: `customAnnotations` are now merged with class-specific annotations (nginx/traefik) instead of being ignored when `ingress.className` is set. ### Added +- **NATS JetStream Publish Path**: Opt-in at-least-once delivery for the NATS notify and audit targets. A NATS Core publish flushes to the connection without awaiting a broker acknowledgement, so an event can be lost across a broker restart or a reconnect after the send queue has already cleared it. A queued event now clears only after the JetStream `PublishAck`, so bucket notifications survive those interruptions. Off by default and byte-identical to the NATS Core path when disabled. + - Three configuration keys per target: `JETSTREAM_ENABLE`, `JETSTREAM_STREAM_NAME`, and `JETSTREAM_ACK_TIMEOUT_SECS`, under the `RUSTFS_NOTIFY_NATS_` and `RUSTFS_AUDIT_NATS_` prefixes + - Durable store-and-forward with a stable dedup id sent as the `Nats-Msg-Id` header, so a replay after a crash is collapsed by the server duplicate window + - Pre-flight stream validation, and a bounded failed-events store (count and TTL). Only a non-retryable rejection is recorded in the failed-events store. A retryable condition keeps the entry on the live queue until it is delivered + - Operator guide at `docs/operations/nats-jetstream.md` - **OpenStack Keystone Authentication Integration**: Full support for OpenStack Keystone authentication via X-Auth-Token headers - Tower-based middleware (`KeystoneAuthLayer`) self-contained within `rustfs-keystone` crate - Task-local storage for async-safe credential passing between middleware and auth handlers diff --git a/Cargo.lock b/Cargo.lock index e1a680218..c7fed349a 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -9971,6 +9971,7 @@ dependencies = [ "s3s", "serde", "serde_json", + "sha2 0.11.0", "snap", "sysinfo", "tempfile", @@ -9979,6 +9980,7 @@ dependencies = [ "tokio-postgres", "tokio-postgres-rustls", "tracing", + "tracing-subscriber", "url", "urlencoding", "uuid", diff --git a/crates/audit/src/pipeline.rs b/crates/audit/src/pipeline.rs index 9325e0cc4..6a83317c8 100644 --- a/crates/audit/src/pipeline.rs +++ b/crates/audit/src/pipeline.rs @@ -32,6 +32,7 @@ const EVENT_AUDIT_BATCH_DISPATCH_COMPLETED: &str = "audit_batch_dispatch_complet const EVENT_AUDIT_TARGET_STATE_CHANGED: &str = "audit_target_state_changed"; const EVENT_AUDIT_REPLAY_DELIVERED: &str = "audit_replay_delivered"; const EVENT_AUDIT_REPLAY_RETRY_SCHEDULED: &str = "audit_replay_retry_scheduled"; +const EVENT_AUDIT_REPLAY_RETRY_EXHAUSTED: &str = "audit_replay_retry_exhausted"; const EVENT_AUDIT_REPLAY_DROPPED: &str = "audit_replay_dropped"; const EVENT_AUDIT_REPLAY_STREAM_STATUS: &str = "audit_replay_stream_status"; @@ -281,6 +282,7 @@ impl AuditPipeline { let delivery = target.delivery_snapshot(); AuditTargetMetricSnapshot { failed_messages: delivery.failed_messages, + failed_store_length: delivery.failed_store_length, queue_length: delivery.queue_length, target_id: target.id().to_string(), total_messages: delivery.total_messages, @@ -463,18 +465,16 @@ impl AuditRuntimeFacade { target.record_final_failure(); observability::record_target_failure(); } - ReplayEvent::RetryExhausted { key, target } => { + ReplayEvent::RetryExhausted { detail, key, target } => { warn!( - event = EVENT_AUDIT_REPLAY_DROPPED, + event = EVENT_AUDIT_REPLAY_RETRY_EXHAUSTED, component = LOG_COMPONENT_AUDIT, subsystem = LOG_SUBSYSTEM_PIPELINE, target_id = %target.id(), replay_key = %key, - reason = "retry_exhausted", - "audit replay delivery" + error = %detail, + "audit replay retry budget exhausted, entry stays queued and retries" ); - target.record_final_failure(); - observability::record_target_failure(); } ReplayEvent::UnreadableEntry { key, error, target } => { warn!( diff --git a/crates/audit/src/system.rs b/crates/audit/src/system.rs index ead1f5157..4d79a04e6 100644 --- a/crates/audit/src/system.rs +++ b/crates/audit/src/system.rs @@ -30,6 +30,7 @@ const EVENT_AUDIT_CONFIG_RELOADED: &str = "audit_config_reloaded"; #[derive(Debug, Clone, Default, PartialEq, Eq)] pub struct AuditTargetMetricSnapshot { pub failed_messages: u64, + pub failed_store_length: u64, pub queue_length: u64, pub target_id: String, pub total_messages: u64, diff --git a/crates/config/src/audit/nats.rs b/crates/config/src/audit/nats.rs index c9657c62b..eccde6125 100644 --- a/crates/config/src/audit/nats.rs +++ b/crates/config/src/audit/nats.rs @@ -25,8 +25,11 @@ pub const ENV_AUDIT_NATS_TLS_CLIENT_KEY: &str = "RUSTFS_AUDIT_NATS_TLS_CLIENT_KE pub const ENV_AUDIT_NATS_TLS_REQUIRED: &str = "RUSTFS_AUDIT_NATS_TLS_REQUIRED"; pub const ENV_AUDIT_NATS_QUEUE_DIR: &str = "RUSTFS_AUDIT_NATS_QUEUE_DIR"; pub const ENV_AUDIT_NATS_QUEUE_LIMIT: &str = "RUSTFS_AUDIT_NATS_QUEUE_LIMIT"; +pub const ENV_AUDIT_NATS_JETSTREAM_ENABLE: &str = "RUSTFS_AUDIT_NATS_JETSTREAM_ENABLE"; +pub const ENV_AUDIT_NATS_JETSTREAM_STREAM_NAME: &str = "RUSTFS_AUDIT_NATS_JETSTREAM_STREAM_NAME"; +pub const ENV_AUDIT_NATS_JETSTREAM_ACK_TIMEOUT_SECS: &str = "RUSTFS_AUDIT_NATS_JETSTREAM_ACK_TIMEOUT_SECS"; -pub const ENV_AUDIT_NATS_KEYS: &[&str; 13] = &[ +pub const ENV_AUDIT_NATS_KEYS: &[&str; 16] = &[ ENV_AUDIT_NATS_ENABLE, ENV_AUDIT_NATS_ADDRESS, ENV_AUDIT_NATS_SUBJECT, @@ -40,6 +43,9 @@ pub const ENV_AUDIT_NATS_KEYS: &[&str; 13] = &[ ENV_AUDIT_NATS_TLS_REQUIRED, ENV_AUDIT_NATS_QUEUE_DIR, ENV_AUDIT_NATS_QUEUE_LIMIT, + ENV_AUDIT_NATS_JETSTREAM_ENABLE, + ENV_AUDIT_NATS_JETSTREAM_STREAM_NAME, + ENV_AUDIT_NATS_JETSTREAM_ACK_TIMEOUT_SECS, ]; pub const AUDIT_NATS_KEYS: &[&str] = &[ @@ -56,5 +62,8 @@ pub const AUDIT_NATS_KEYS: &[&str] = &[ crate::NATS_TLS_REQUIRED, crate::NATS_QUEUE_DIR, crate::NATS_QUEUE_LIMIT, + crate::NATS_JETSTREAM_ENABLE, + crate::NATS_JETSTREAM_STREAM_NAME, + crate::NATS_JETSTREAM_ACK_TIMEOUT_SECS, crate::COMMENT_KEY, ]; diff --git a/crates/config/src/constants/targets.rs b/crates/config/src/constants/targets.rs index 0d3e14a32..58e92067b 100644 --- a/crates/config/src/constants/targets.rs +++ b/crates/config/src/constants/targets.rs @@ -79,6 +79,13 @@ pub const NATS_TLS_CLIENT_KEY: &str = "tls_client_key"; pub const NATS_TLS_REQUIRED: &str = "tls_required"; pub const NATS_QUEUE_DIR: &str = "queue_dir"; pub const NATS_QUEUE_LIMIT: &str = "queue_limit"; +pub const NATS_JETSTREAM_ENABLE: &str = "jetstream_enable"; +pub const NATS_JETSTREAM_STREAM_NAME: &str = "jetstream_stream_name"; +pub const NATS_JETSTREAM_ACK_TIMEOUT_SECS: &str = "jetstream_ack_timeout_secs"; + +pub const NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS: u64 = 30; +pub const NATS_JETSTREAM_ACK_TIMEOUT_MIN_SECS: u64 = 10; +pub const NATS_JETSTREAM_ACK_TIMEOUT_MAX_SECS: u64 = 120; pub const PULSAR_BROKER: &str = "broker"; pub const PULSAR_TOPIC: &str = "topic"; diff --git a/crates/config/src/notify/nats.rs b/crates/config/src/notify/nats.rs index d9464128d..092d1ab9c 100644 --- a/crates/config/src/notify/nats.rs +++ b/crates/config/src/notify/nats.rs @@ -26,6 +26,9 @@ pub const NOTIFY_NATS_KEYS: &[&str] = &[ crate::NATS_TLS_REQUIRED, crate::NATS_QUEUE_DIR, crate::NATS_QUEUE_LIMIT, + crate::NATS_JETSTREAM_ENABLE, + crate::NATS_JETSTREAM_STREAM_NAME, + crate::NATS_JETSTREAM_ACK_TIMEOUT_SECS, crate::COMMENT_KEY, ]; @@ -42,8 +45,11 @@ pub const ENV_NOTIFY_NATS_TLS_CLIENT_KEY: &str = "RUSTFS_NOTIFY_NATS_TLS_CLIENT_ pub const ENV_NOTIFY_NATS_TLS_REQUIRED: &str = "RUSTFS_NOTIFY_NATS_TLS_REQUIRED"; pub const ENV_NOTIFY_NATS_QUEUE_DIR: &str = "RUSTFS_NOTIFY_NATS_QUEUE_DIR"; pub const ENV_NOTIFY_NATS_QUEUE_LIMIT: &str = "RUSTFS_NOTIFY_NATS_QUEUE_LIMIT"; +pub const ENV_NOTIFY_NATS_JETSTREAM_ENABLE: &str = "RUSTFS_NOTIFY_NATS_JETSTREAM_ENABLE"; +pub const ENV_NOTIFY_NATS_JETSTREAM_STREAM_NAME: &str = "RUSTFS_NOTIFY_NATS_JETSTREAM_STREAM_NAME"; +pub const ENV_NOTIFY_NATS_JETSTREAM_ACK_TIMEOUT_SECS: &str = "RUSTFS_NOTIFY_NATS_JETSTREAM_ACK_TIMEOUT_SECS"; -pub const ENV_NOTIFY_NATS_KEYS: &[&str; 13] = &[ +pub const ENV_NOTIFY_NATS_KEYS: &[&str; 16] = &[ ENV_NOTIFY_NATS_ENABLE, ENV_NOTIFY_NATS_ADDRESS, ENV_NOTIFY_NATS_SUBJECT, @@ -57,4 +63,7 @@ pub const ENV_NOTIFY_NATS_KEYS: &[&str; 13] = &[ ENV_NOTIFY_NATS_TLS_REQUIRED, ENV_NOTIFY_NATS_QUEUE_DIR, ENV_NOTIFY_NATS_QUEUE_LIMIT, + ENV_NOTIFY_NATS_JETSTREAM_ENABLE, + ENV_NOTIFY_NATS_JETSTREAM_STREAM_NAME, + ENV_NOTIFY_NATS_JETSTREAM_ACK_TIMEOUT_SECS, ]; diff --git a/crates/ecstore/src/config/audit.rs b/crates/ecstore/src/config/audit.rs index 3bb4ee545..b7ed83323 100644 --- a/crates/ecstore/src/config/audit.rs +++ b/crates/ecstore/src/config/audit.rs @@ -23,17 +23,19 @@ use rustfs_config::{ MQTT_QUEUE_LIMIT, MQTT_RECONNECT_INTERVAL, MQTT_TLS_CA, MQTT_TLS_CLIENT_CERT, MQTT_TLS_CLIENT_KEY, MQTT_TLS_POLICY, MQTT_TLS_TRUST_LEAF_AS_CA, MQTT_TOPIC, MQTT_USERNAME, MQTT_WS_PATH_ALLOWLIST, MYSQL_DSN_STRING, MYSQL_FORMAT, MYSQL_MAX_OPEN_CONNECTIONS, MYSQL_QUEUE_DIR, MYSQL_QUEUE_LIMIT, MYSQL_TABLE, MYSQL_TLS_CA, MYSQL_TLS_CLIENT_CERT, - MYSQL_TLS_CLIENT_KEY, NATS_ADDRESS, NATS_CREDENTIALS_FILE, NATS_PASSWORD, NATS_QUEUE_DIR, NATS_QUEUE_LIMIT, NATS_SUBJECT, - NATS_TLS_CA, NATS_TLS_CLIENT_CERT, NATS_TLS_CLIENT_KEY, NATS_TLS_REQUIRED, NATS_TOKEN, NATS_USERNAME, POSTGRES_DSN_STRING, - POSTGRES_FORMAT, POSTGRES_QUEUE_DIR, POSTGRES_QUEUE_LIMIT, POSTGRES_TABLE, POSTGRES_TLS_CA, POSTGRES_TLS_CLIENT_CERT, - POSTGRES_TLS_CLIENT_KEY, POSTGRES_TLS_REQUIRED, PULSAR_AUTH_TOKEN, PULSAR_BROKER, PULSAR_PASSWORD, PULSAR_QUEUE_DIR, - PULSAR_QUEUE_LIMIT, PULSAR_TLS_ALLOW_INSECURE, PULSAR_TLS_CA, PULSAR_TLS_HOSTNAME_VERIFICATION, PULSAR_TOPIC, - PULSAR_USERNAME, REDIS_CHANNEL, REDIS_CONNECTION_TIMEOUT, REDIS_KEEP_ALIVE_INTERVAL, REDIS_MAX_RETRY_ATTEMPTS, - REDIS_MAX_RETRY_DELAY, REDIS_MIN_RETRY_DELAY, REDIS_PASSWORD, REDIS_PIPELINE_BUFFER_SIZE, REDIS_QUEUE_DIR, REDIS_QUEUE_LIMIT, - REDIS_RECONNECT_RETRY_ATTEMPTS, REDIS_RESPONSE_TIMEOUT, REDIS_TLS_ALLOW_INSECURE, REDIS_TLS_CA, REDIS_TLS_CLIENT_CERT, - REDIS_TLS_CLIENT_KEY, REDIS_TLS_POLICY, REDIS_URL, REDIS_USERNAME, WEBHOOK_AUTH_TOKEN, WEBHOOK_BATCH_SIZE, WEBHOOK_CLIENT_CA, - WEBHOOK_CLIENT_CERT, WEBHOOK_CLIENT_KEY, WEBHOOK_ENDPOINT, WEBHOOK_HTTP_TIMEOUT, WEBHOOK_MAX_RETRY, WEBHOOK_QUEUE_DIR, - WEBHOOK_QUEUE_LIMIT, WEBHOOK_RETRY_INTERVAL, WEBHOOK_SKIP_TLS_VERIFY, + MYSQL_TLS_CLIENT_KEY, NATS_ADDRESS, NATS_CREDENTIALS_FILE, NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS, + NATS_JETSTREAM_ACK_TIMEOUT_SECS, NATS_JETSTREAM_ENABLE, NATS_JETSTREAM_STREAM_NAME, NATS_PASSWORD, NATS_QUEUE_DIR, + NATS_QUEUE_LIMIT, NATS_SUBJECT, NATS_TLS_CA, NATS_TLS_CLIENT_CERT, NATS_TLS_CLIENT_KEY, NATS_TLS_REQUIRED, NATS_TOKEN, + NATS_USERNAME, POSTGRES_DSN_STRING, POSTGRES_FORMAT, POSTGRES_QUEUE_DIR, POSTGRES_QUEUE_LIMIT, POSTGRES_TABLE, + POSTGRES_TLS_CA, POSTGRES_TLS_CLIENT_CERT, POSTGRES_TLS_CLIENT_KEY, POSTGRES_TLS_REQUIRED, PULSAR_AUTH_TOKEN, PULSAR_BROKER, + PULSAR_PASSWORD, PULSAR_QUEUE_DIR, PULSAR_QUEUE_LIMIT, PULSAR_TLS_ALLOW_INSECURE, PULSAR_TLS_CA, + PULSAR_TLS_HOSTNAME_VERIFICATION, PULSAR_TOPIC, PULSAR_USERNAME, REDIS_CHANNEL, REDIS_CONNECTION_TIMEOUT, + REDIS_KEEP_ALIVE_INTERVAL, REDIS_MAX_RETRY_ATTEMPTS, REDIS_MAX_RETRY_DELAY, REDIS_MIN_RETRY_DELAY, REDIS_PASSWORD, + REDIS_PIPELINE_BUFFER_SIZE, REDIS_QUEUE_DIR, REDIS_QUEUE_LIMIT, REDIS_RECONNECT_RETRY_ATTEMPTS, REDIS_RESPONSE_TIMEOUT, + REDIS_TLS_ALLOW_INSECURE, REDIS_TLS_CA, REDIS_TLS_CLIENT_CERT, REDIS_TLS_CLIENT_KEY, REDIS_TLS_POLICY, REDIS_URL, + REDIS_USERNAME, WEBHOOK_AUTH_TOKEN, WEBHOOK_BATCH_SIZE, WEBHOOK_CLIENT_CA, WEBHOOK_CLIENT_CERT, WEBHOOK_CLIENT_KEY, + WEBHOOK_ENDPOINT, WEBHOOK_HTTP_TIMEOUT, WEBHOOK_MAX_RETRY, WEBHOOK_QUEUE_DIR, WEBHOOK_QUEUE_LIMIT, WEBHOOK_RETRY_INTERVAL, + WEBHOOK_SKIP_TLS_VERIFY, }; use std::sync::LazyLock; @@ -350,6 +352,21 @@ pub static DEFAULT_AUDIT_NATS_KVS: LazyLock = LazyLock::new(|| { value: DEFAULT_LIMIT.to_string(), hidden_if_empty: false, }, + KV { + key: NATS_JETSTREAM_ENABLE.to_owned(), + value: EnableState::Off.to_string(), + hidden_if_empty: false, + }, + KV { + key: NATS_JETSTREAM_STREAM_NAME.to_owned(), + value: "".to_owned(), + hidden_if_empty: false, + }, + KV { + key: NATS_JETSTREAM_ACK_TIMEOUT_SECS.to_owned(), + value: NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS.to_string(), + hidden_if_empty: false, + }, KV { key: COMMENT_KEY.to_owned(), value: "".to_owned(), diff --git a/crates/ecstore/src/config/notify.rs b/crates/ecstore/src/config/notify.rs index a7130becd..4bb76e394 100644 --- a/crates/ecstore/src/config/notify.rs +++ b/crates/ecstore/src/config/notify.rs @@ -23,16 +23,18 @@ use rustfs_config::{ MQTT_QUEUE_LIMIT, MQTT_RECONNECT_INTERVAL, MQTT_TLS_CA, MQTT_TLS_CLIENT_CERT, MQTT_TLS_CLIENT_KEY, MQTT_TLS_POLICY, MQTT_TLS_TRUST_LEAF_AS_CA, MQTT_TOPIC, MQTT_USERNAME, MQTT_WS_PATH_ALLOWLIST, MYSQL_DSN_STRING, MYSQL_FORMAT, MYSQL_MAX_OPEN_CONNECTIONS, MYSQL_QUEUE_DIR, MYSQL_QUEUE_LIMIT, MYSQL_TABLE, MYSQL_TLS_CA, MYSQL_TLS_CLIENT_CERT, - MYSQL_TLS_CLIENT_KEY, NATS_ADDRESS, NATS_CREDENTIALS_FILE, NATS_PASSWORD, NATS_QUEUE_DIR, NATS_QUEUE_LIMIT, NATS_SUBJECT, - NATS_TLS_CA, NATS_TLS_CLIENT_CERT, NATS_TLS_CLIENT_KEY, NATS_TLS_REQUIRED, NATS_TOKEN, NATS_USERNAME, POSTGRES_DSN_STRING, - POSTGRES_FORMAT, POSTGRES_QUEUE_DIR, POSTGRES_QUEUE_LIMIT, POSTGRES_TABLE, POSTGRES_TLS_CA, POSTGRES_TLS_CLIENT_CERT, - POSTGRES_TLS_CLIENT_KEY, POSTGRES_TLS_REQUIRED, PULSAR_AUTH_TOKEN, PULSAR_BROKER, PULSAR_PASSWORD, PULSAR_QUEUE_DIR, - PULSAR_QUEUE_LIMIT, PULSAR_TLS_ALLOW_INSECURE, PULSAR_TLS_CA, PULSAR_TLS_HOSTNAME_VERIFICATION, PULSAR_TOPIC, - PULSAR_USERNAME, REDIS_CHANNEL, REDIS_CONNECTION_TIMEOUT, REDIS_KEEP_ALIVE_INTERVAL, REDIS_MAX_RETRY_ATTEMPTS, - REDIS_MAX_RETRY_DELAY, REDIS_MIN_RETRY_DELAY, REDIS_PASSWORD, REDIS_PIPELINE_BUFFER_SIZE, REDIS_QUEUE_DIR, REDIS_QUEUE_LIMIT, - REDIS_RECONNECT_RETRY_ATTEMPTS, REDIS_RESPONSE_TIMEOUT, REDIS_TLS_ALLOW_INSECURE, REDIS_TLS_CA, REDIS_TLS_CLIENT_CERT, - REDIS_TLS_CLIENT_KEY, REDIS_TLS_POLICY, REDIS_URL, REDIS_USERNAME, WEBHOOK_AUTH_TOKEN, WEBHOOK_CLIENT_CA, - WEBHOOK_CLIENT_CERT, WEBHOOK_CLIENT_KEY, WEBHOOK_ENDPOINT, WEBHOOK_QUEUE_DIR, WEBHOOK_QUEUE_LIMIT, WEBHOOK_SKIP_TLS_VERIFY, + MYSQL_TLS_CLIENT_KEY, NATS_ADDRESS, NATS_CREDENTIALS_FILE, NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS, + NATS_JETSTREAM_ACK_TIMEOUT_SECS, NATS_JETSTREAM_ENABLE, NATS_JETSTREAM_STREAM_NAME, NATS_PASSWORD, NATS_QUEUE_DIR, + NATS_QUEUE_LIMIT, NATS_SUBJECT, NATS_TLS_CA, NATS_TLS_CLIENT_CERT, NATS_TLS_CLIENT_KEY, NATS_TLS_REQUIRED, NATS_TOKEN, + NATS_USERNAME, POSTGRES_DSN_STRING, POSTGRES_FORMAT, POSTGRES_QUEUE_DIR, POSTGRES_QUEUE_LIMIT, POSTGRES_TABLE, + POSTGRES_TLS_CA, POSTGRES_TLS_CLIENT_CERT, POSTGRES_TLS_CLIENT_KEY, POSTGRES_TLS_REQUIRED, PULSAR_AUTH_TOKEN, PULSAR_BROKER, + PULSAR_PASSWORD, PULSAR_QUEUE_DIR, PULSAR_QUEUE_LIMIT, PULSAR_TLS_ALLOW_INSECURE, PULSAR_TLS_CA, + PULSAR_TLS_HOSTNAME_VERIFICATION, PULSAR_TOPIC, PULSAR_USERNAME, REDIS_CHANNEL, REDIS_CONNECTION_TIMEOUT, + REDIS_KEEP_ALIVE_INTERVAL, REDIS_MAX_RETRY_ATTEMPTS, REDIS_MAX_RETRY_DELAY, REDIS_MIN_RETRY_DELAY, REDIS_PASSWORD, + REDIS_PIPELINE_BUFFER_SIZE, REDIS_QUEUE_DIR, REDIS_QUEUE_LIMIT, REDIS_RECONNECT_RETRY_ATTEMPTS, REDIS_RESPONSE_TIMEOUT, + REDIS_TLS_ALLOW_INSECURE, REDIS_TLS_CA, REDIS_TLS_CLIENT_CERT, REDIS_TLS_CLIENT_KEY, REDIS_TLS_POLICY, REDIS_URL, + REDIS_USERNAME, WEBHOOK_AUTH_TOKEN, WEBHOOK_CLIENT_CA, WEBHOOK_CLIENT_CERT, WEBHOOK_CLIENT_KEY, WEBHOOK_ENDPOINT, + WEBHOOK_QUEUE_DIR, WEBHOOK_QUEUE_LIMIT, WEBHOOK_SKIP_TLS_VERIFY, }; use std::sync::LazyLock; @@ -328,6 +330,21 @@ pub static DEFAULT_NOTIFY_NATS_KVS: LazyLock = LazyLock::new(|| { value: DEFAULT_LIMIT.to_string(), hidden_if_empty: false, }, + KV { + key: NATS_JETSTREAM_ENABLE.to_owned(), + value: EnableState::Off.to_string(), + hidden_if_empty: false, + }, + KV { + key: NATS_JETSTREAM_STREAM_NAME.to_owned(), + value: "".to_owned(), + hidden_if_empty: false, + }, + KV { + key: NATS_JETSTREAM_ACK_TIMEOUT_SECS.to_owned(), + value: NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS.to_string(), + hidden_if_empty: false, + }, KV { key: COMMENT_KEY.to_owned(), value: "".to_owned(), diff --git a/crates/notify/src/integration.rs b/crates/notify/src/integration.rs index 1328f238e..a97f39cf7 100644 --- a/crates/notify/src/integration.rs +++ b/crates/notify/src/integration.rs @@ -80,6 +80,7 @@ pub struct NotificationMetricSnapshot { #[derive(Debug, Clone, Default, PartialEq, Eq)] pub struct NotificationTargetMetricSnapshot { pub failed_messages: u64, + pub failed_store_length: u64, pub queue_length: u64, pub target_id: String, pub target_type: String, diff --git a/crates/notify/src/runtime_facade.rs b/crates/notify/src/runtime_facade.rs index f59f20972..c6383844a 100644 --- a/crates/notify/src/runtime_facade.rs +++ b/crates/notify/src/runtime_facade.rs @@ -19,12 +19,13 @@ use rustfs_targets::{ use std::sync::Arc; use std::time::Duration; use tokio::sync::{RwLock, Semaphore}; -use tracing::{debug, info}; +use tracing::{debug, info, warn}; const LOG_COMPONENT_NOTIFY: &str = "notify"; const LOG_SUBSYSTEM_RUNTIME: &str = "runtime"; const EVENT_NOTIFY_RUNTIME_LIFECYCLE: &str = "notify_runtime_lifecycle"; const EVENT_NOTIFY_RUNTIME_SHUTDOWN_FAILED: &str = "notify_runtime_shutdown_failed"; +const EVENT_NOTIFY_REPLAY_RETRY_EXHAUSTED: &str = "notify_replay_retry_exhausted"; #[derive(Clone)] pub struct NotifyRuntimeFacade { @@ -48,9 +49,18 @@ impl NotifyRuntimeFacade { match event { ReplayEvent::Delivered { .. } => metrics.increment_processed(), ReplayEvent::RetryableError { .. } => {} - ReplayEvent::Dropped { target, .. } - | ReplayEvent::PermanentFailure { target, .. } - | ReplayEvent::RetryExhausted { target, .. } => { + ReplayEvent::RetryExhausted { detail, key, target } => { + warn!( + event = EVENT_NOTIFY_REPLAY_RETRY_EXHAUSTED, + component = LOG_COMPONENT_NOTIFY, + subsystem = LOG_SUBSYSTEM_RUNTIME, + target_id = %target.id(), + replay_key = %key, + error = %detail, + "notify replay retry budget exhausted, entry stays queued and retries" + ); + } + ReplayEvent::Dropped { target, .. } | ReplayEvent::PermanentFailure { target, .. } => { target.record_final_failure(); metrics.increment_failed(); } diff --git a/crates/notify/src/runtime_view.rs b/crates/notify/src/runtime_view.rs index 0ad80baa1..e8530b1b3 100644 --- a/crates/notify/src/runtime_view.rs +++ b/crates/notify/src/runtime_view.rs @@ -51,6 +51,7 @@ impl NotifyRuntimeView { .into_iter() .map(|snapshot| NotificationTargetMetricSnapshot { failed_messages: snapshot.failed_messages, + failed_store_length: snapshot.failed_store_length, queue_length: snapshot.queue_length, target_id: snapshot.target_id, target_type: snapshot.target_type, @@ -90,6 +91,7 @@ mod tests { active: bool, enabled: bool, failed_messages: Arc, + failed_store_length: u64, id: TargetID, total_messages: Arc, } @@ -100,6 +102,7 @@ mod tests { active: true, enabled: true, failed_messages: Arc::new(AtomicU64::new(0)), + failed_store_length: 0, id: TargetID::new(id.to_string(), name.to_string()), total_messages: Arc::new(AtomicU64::new(0)), } @@ -110,6 +113,11 @@ mod tests { self } + fn with_failed_store_length(mut self, failed_store_length: u64) -> Self { + self.failed_store_length = failed_store_length; + self + } + fn with_enabled(mut self, enabled: bool) -> Self { self.enabled = enabled; self @@ -168,6 +176,7 @@ mod tests { fn delivery_snapshot(&self) -> TargetDeliverySnapshot { TargetDeliverySnapshot { failed_messages: self.failed_messages.load(Ordering::Relaxed), + failed_store_length: self.failed_store_length, queue_length: 0, total_messages: self.total_messages.load(Ordering::Relaxed), } @@ -206,7 +215,7 @@ mod tests { let target_list = Arc::new(RwLock::new(TargetList::new())); let replay_workers = Arc::new(RwLock::new(ReplayWorkerManager::new())); - let online = Arc::new(TestTarget::new("primary", "webhook")); + let online = Arc::new(TestTarget::new("primary", "webhook").with_failed_store_length(7)); online.record_successes(3); online.record_failures(1); @@ -239,9 +248,11 @@ mod tests { assert_eq!(metric_snapshots.len(), 2); assert_eq!(metric_snapshots[0].target_id, "backup:mqtt"); assert_eq!(metric_snapshots[0].failed_messages, 0); + assert_eq!(metric_snapshots[0].failed_store_length, 0); assert_eq!(metric_snapshots[0].total_messages, 2); assert_eq!(metric_snapshots[1].target_id, "primary:webhook"); assert_eq!(metric_snapshots[1].failed_messages, 1); + assert_eq!(metric_snapshots[1].failed_store_length, 7); assert_eq!(metric_snapshots[1].total_messages, 3); let health_snapshots = runtime_view.snapshot_target_health().await; diff --git a/crates/obs/src/metrics/collectors/audit.rs b/crates/obs/src/metrics/collectors/audit.rs index 35a8129e2..0f7e1f600 100644 --- a/crates/obs/src/metrics/collectors/audit.rs +++ b/crates/obs/src/metrics/collectors/audit.rs @@ -33,6 +33,8 @@ pub struct AuditTargetStats { pub target_id: String, /// Number of messages that failed to send pub failed_messages: u64, + /// Number of messages held in the failed-events store + pub failed_store_length: u64, /// Number of unsent messages in queue pub queue_length: u64, /// Total number of messages sent @@ -48,7 +50,7 @@ pub fn collect_audit_metrics(stats: &[AuditTargetStats]) -> Vec = Cow::Owned(stat.target_id.clone()); @@ -56,6 +58,10 @@ pub fn collect_audit_metrics(stats: &[AuditTargetStats]) -> Vec return Vec::new(); } - let mut metrics = Vec::with_capacity(stats.len() * 3); + let mut metrics = Vec::with_capacity(stats.len() * 4); for stat in stats { let target_id: Cow<'static, str> = Cow::Owned(stat.target_id.clone()); let target_type: Cow<'static, str> = Cow::Owned(stat.target_type.clone()); @@ -45,6 +46,11 @@ pub fn collect_notification_target_metrics(stats: &[NotificationTargetStats]) -> .with_label(TARGET_ID, target_id.clone()) .with_label(TARGET_TYPE, target_type.clone()), ); + metrics.push( + PrometheusMetric::from_descriptor(&NOTIFICATION_TARGET_FAILED_STORE_LENGTH_MD, stat.failed_store_length as f64) + .with_label(TARGET_ID, target_id.clone()) + .with_label(TARGET_TYPE, target_type.clone()), + ); metrics.push( PrometheusMetric::from_descriptor(&NOTIFICATION_TARGET_QUEUE_LENGTH_MD, stat.queue_length as f64) .with_label(TARGET_ID, target_id.clone()) @@ -69,6 +75,7 @@ mod tests { fn test_collect_notification_target_metrics() { let stats = vec![NotificationTargetStats { failed_messages: 2, + failed_store_length: 3, queue_length: 4, target_id: "primary:webhook".to_string(), target_type: "webhook".to_string(), @@ -77,7 +84,15 @@ mod tests { let metrics = collect_notification_target_metrics(&stats); - assert_eq!(metrics.len(), 3); + assert_eq!(metrics.len(), 4); + assert!(metrics.iter().any(|metric| { + metric.value == 3.0 + && metric.name == NOTIFICATION_TARGET_FAILED_STORE_LENGTH_MD.get_full_metric_name() + && metric + .labels + .iter() + .any(|(key, value)| *key == TARGET_ID && value == "primary:webhook") + })); assert!(metrics.iter().any(|metric| { metric.value == 42.0 && metric @@ -94,6 +109,7 @@ mod tests { #[test] fn notification_target_totals_are_exported_as_gauges() { assert_eq!(NOTIFICATION_TARGET_FAILED_MESSAGES_MD.metric_type, MetricType::Gauge); + assert_eq!(NOTIFICATION_TARGET_FAILED_STORE_LENGTH_MD.metric_type, MetricType::Gauge); assert_eq!(NOTIFICATION_TARGET_QUEUE_LENGTH_MD.metric_type, MetricType::Gauge); assert_eq!(NOTIFICATION_TARGET_TOTAL_MESSAGES_MD.metric_type, MetricType::Gauge); } diff --git a/crates/obs/src/metrics/scheduler.rs b/crates/obs/src/metrics/scheduler.rs index 2fbfd7b55..dd306f1a7 100644 --- a/crates/obs/src/metrics/scheduler.rs +++ b/crates/obs/src/metrics/scheduler.rs @@ -1222,6 +1222,7 @@ pub fn init_metrics_runtime(token: CancellationToken) { .into_iter() .map(|snapshot| AuditTargetStats { failed_messages: snapshot.failed_messages, + failed_store_length: snapshot.failed_store_length, queue_length: snapshot.queue_length, target_id: snapshot.target_id, total_messages: snapshot.total_messages, @@ -1274,6 +1275,7 @@ pub fn init_metrics_runtime(token: CancellationToken) { .into_iter() .map(|snapshot| NotificationTargetStats { failed_messages: snapshot.failed_messages, + failed_store_length: snapshot.failed_store_length, queue_length: snapshot.queue_length, target_id: snapshot.target_id, target_type: snapshot.target_type, diff --git a/crates/obs/src/metrics/schema/audit.rs b/crates/obs/src/metrics/schema/audit.rs index a5d738c6a..ceeb9045e 100644 --- a/crates/obs/src/metrics/schema/audit.rs +++ b/crates/obs/src/metrics/schema/audit.rs @@ -33,6 +33,15 @@ pub static AUDIT_FAILED_MESSAGES_MD: LazyLock = LazyLock::new( ) }); +pub static AUDIT_FAILED_STORE_LENGTH_MD: LazyLock = LazyLock::new(|| { + new_gauge_md( + MetricName::AuditFailedStoreLength, + "Number of audit messages held in the failed-events store for target", + &[TARGET_ID], + subsystems::AUDIT, + ) +}); + pub static AUDIT_TARGET_QUEUE_LENGTH_MD: LazyLock = LazyLock::new(|| { new_gauge_md( MetricName::AuditTargetQueueLength, diff --git a/crates/obs/src/metrics/schema/entry/metric_name.rs b/crates/obs/src/metrics/schema/entry/metric_name.rs index 1eb890edb..9c0cbf2a2 100644 --- a/crates/obs/src/metrics/schema/entry/metric_name.rs +++ b/crates/obs/src/metrics/schema/entry/metric_name.rs @@ -175,6 +175,7 @@ pub enum MetricName { // Audit metrics AuditFailedMessages, + AuditFailedStoreLength, AuditTargetQueueLength, AuditTotalMessages, @@ -221,6 +222,7 @@ pub enum MetricName { NotificationEventsSentTotal, NotificationEventsSkippedTotal, NotificationTargetFailedMessages, + NotificationTargetFailedStoreLength, NotificationTargetQueueLength, NotificationTargetTotalMessages, @@ -581,6 +583,7 @@ impl MetricName { Self::ApiTrafficRecvBytes => "traffic_received_bytes".to_string(), Self::AuditFailedMessages => "failed_messages".to_string(), + Self::AuditFailedStoreLength => "failed_store_length".to_string(), Self::AuditTargetQueueLength => "target_queue_length".to_string(), Self::AuditTotalMessages => "total_messages".to_string(), @@ -627,6 +630,7 @@ impl MetricName { Self::NotificationEventsSentTotal => "events_sent_total".to_string(), Self::NotificationEventsSkippedTotal => "events_skipped_total".to_string(), Self::NotificationTargetFailedMessages => "failed_messages".to_string(), + Self::NotificationTargetFailedStoreLength => "failed_store_length".to_string(), Self::NotificationTargetQueueLength => "target_queue_length".to_string(), Self::NotificationTargetTotalMessages => "total_messages".to_string(), diff --git a/crates/obs/src/metrics/schema/notification_target.rs b/crates/obs/src/metrics/schema/notification_target.rs index ceb801c13..3a1c6e491 100644 --- a/crates/obs/src/metrics/schema/notification_target.rs +++ b/crates/obs/src/metrics/schema/notification_target.rs @@ -31,6 +31,15 @@ pub static NOTIFICATION_TARGET_FAILED_MESSAGES_MD: LazyLock = ) }); +pub static NOTIFICATION_TARGET_FAILED_STORE_LENGTH_MD: LazyLock = LazyLock::new(|| { + new_gauge_md( + MetricName::NotificationTargetFailedStoreLength, + "Number of notification messages held in the failed-events store for target", + &NOTIFICATION_TARGET_LABELS, + subsystems::NOTIFICATION, + ) +}); + pub static NOTIFICATION_TARGET_QUEUE_LENGTH_MD: LazyLock = LazyLock::new(|| { new_gauge_md( MetricName::NotificationTargetQueueLength, diff --git a/crates/targets/Cargo.toml b/crates/targets/Cargo.toml index 470362719..297fb591c 100644 --- a/crates/targets/Cargo.toml +++ b/crates/targets/Cargo.toml @@ -34,6 +34,7 @@ rustls-native-certs = { workspace = true } s3s = { workspace = true } serde = { workspace = true } serde_json = { workspace = true } +sha2 = { workspace = true } snap = { workspace = true } thiserror = { workspace = true } tokio = { workspace = true, features = ["fs", "rt-multi-thread", "sync", "time"] } @@ -56,6 +57,7 @@ metrics = { workspace = true } criterion = { workspace = true } tempfile = { workspace = true } rcgen = { workspace = true } +tracing-subscriber = { workspace = true } [[bench]] name = "queue_store_benchmark" diff --git a/crates/targets/src/check.rs b/crates/targets/src/check.rs index 7de161224..c56c14895 100644 --- a/crates/targets/src/check.rs +++ b/crates/targets/src/check.rs @@ -100,11 +100,27 @@ pub async fn check_nats_server_available(args: &crate::target::nats::NATSArgs) - .flush() .await .map_err(|e| crate::TargetError::Network(format!("NATS connection check failed: {e}")))?; + // Validate the configured stream on the live connection before the drain closes it, so a + // missing or non-writable stream is rejected here rather than at first publish. The lookup is + // read-only and never creates a stream. The drain runs regardless of the validation outcome so + // the check connection is closed gracefully, and the validation error then propagates. + let validation = if args.jetstream_enable.unwrap_or(false) { + let mut context = async_nats::jetstream::new(client.clone()); + // Match every other context construction site: the ack timeout is the per-operation await + // bound, so the validation get_stream lookup uses it rather than the library default. + context.set_timeout(std::time::Duration::from_secs( + args.jetstream_ack_timeout_secs + .unwrap_or(rustfs_config::NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS), + )); + crate::target::nats::validate_jetstream_stream(&context, args, "nats-check", None).await + } else { + Ok(()) + }; client .drain() .await .map_err(|e| crate::TargetError::Network(format!("Failed to close NATS check connection: {e}")))?; - Ok(()) + validation }) .await .unwrap_or_else(|_| Err(crate::TargetError::Timeout("NATS connection timed out".to_string()))) diff --git a/crates/targets/src/config/common.rs b/crates/targets/src/config/common.rs index ff3e850da..46f365410 100644 --- a/crates/targets/src/config/common.rs +++ b/crates/targets/src/config/common.rs @@ -13,13 +13,15 @@ // limitations under the License. use crate::TargetError; +use crate::target::nats::validate_jetstream_settings; use crate::target::pulsar::validate_pulsar_broker; use async_nats::ServerAddr; use rustfs_config::server_config::KVS; use rustfs_config::{ - DEFAULT_DELIMITER, ENABLE_KEY, EnableState, NATS_CREDENTIALS_FILE, NATS_PASSWORD, NATS_QUEUE_DIR, NATS_SUBJECT, NATS_TLS_CA, - NATS_TLS_CLIENT_CERT, NATS_TLS_CLIENT_KEY, NATS_TOKEN, NATS_USERNAME, PULSAR_AUTH_TOKEN, PULSAR_PASSWORD, PULSAR_QUEUE_DIR, - PULSAR_TLS_CA, PULSAR_TOPIC, PULSAR_USERNAME, + DEFAULT_DELIMITER, ENABLE_KEY, EnableState, NATS_CREDENTIALS_FILE, NATS_JETSTREAM_ACK_TIMEOUT_SECS, NATS_JETSTREAM_ENABLE, + NATS_JETSTREAM_STREAM_NAME, NATS_PASSWORD, NATS_QUEUE_DIR, NATS_SUBJECT, NATS_TLS_CA, NATS_TLS_CLIENT_CERT, + NATS_TLS_CLIENT_KEY, NATS_TOKEN, NATS_USERNAME, PULSAR_AUTH_TOKEN, PULSAR_PASSWORD, PULSAR_QUEUE_DIR, PULSAR_TLS_CA, + PULSAR_TOPIC, PULSAR_USERNAME, }; use rustfs_utils::egress::validate_outbound_url; use std::collections::HashSet; @@ -130,10 +132,42 @@ pub(super) fn validate_nats_server_config(server: &ServerAddr, config: &KVS, def return Err(TargetError::Configuration("NATS queue directory must be an absolute path".to_string())); } + let jetstream_enable = parse_jetstream_enable(config)?.unwrap_or(false); + let jetstream_stream_name = config.lookup(NATS_JETSTREAM_STREAM_NAME).unwrap_or_default(); + let jetstream_ack_timeout_secs = parse_jetstream_ack_timeout_secs(config)?; + validate_jetstream_settings(jetstream_enable, &jetstream_stream_name, &queue_dir, jetstream_ack_timeout_secs)?; + let _ = server; Ok(()) } +/// Parses the jetstream_enable flag, distinguishing an absent or blank key (None, disabled) from a +/// key that is present but carries an unparsable value, which is a configuration error rather than +/// a silent disable. Unwrapping an unparsable value to disabled would drop the durability guarantee +/// the operator asked for with no signal. +pub(super) fn parse_jetstream_enable(config: &KVS) -> Result, TargetError> { + match config.lookup(NATS_JETSTREAM_ENABLE) { + Some(value) if !value.trim().is_empty() => parse_target_bool(Some(value.as_str())).map(Some).ok_or_else(|| { + TargetError::Configuration(format!( + "Invalid {NATS_JETSTREAM_ENABLE} boolean value: {value} (accepted values include on, off, true, false, yes, no, 1, 0)" + )) + }), + _ => Ok(None), + } +} + +pub(super) fn parse_jetstream_ack_timeout_secs(config: &KVS) -> Result, TargetError> { + match config.lookup(NATS_JETSTREAM_ACK_TIMEOUT_SECS) { + Some(value) if !value.trim().is_empty() => { + let secs = value.trim().parse::().map_err(|_| { + TargetError::Configuration(format!("{NATS_JETSTREAM_ACK_TIMEOUT_SECS} must be a whole number of seconds")) + })?; + Ok(Some(secs)) + } + _ => Ok(None), + } +} + pub(super) fn validate_pulsar_broker_config(broker: &str, config: &KVS, default_queue_dir: &str) -> Result<(), TargetError> { let url = validate_pulsar_broker(broker)?; @@ -193,15 +227,26 @@ pub(super) fn validate_outbound_http_url(value: &Url, field_label: &str) -> Resu #[cfg(test)] mod tests { - use super::{validate_nats_server_config, validate_pulsar_broker_config}; + use super::{parse_jetstream_enable, validate_nats_server_config, validate_pulsar_broker_config}; use async_nats::ServerAddr; use rustfs_config::server_config::KVS; use rustfs_config::{ - NATS_PASSWORD, NATS_QUEUE_DIR, NATS_SUBJECT, NATS_TOKEN, NATS_USERNAME, PULSAR_TLS_ALLOW_INSECURE, PULSAR_TLS_CA, - PULSAR_TLS_HOSTNAME_VERIFICATION, PULSAR_TOPIC, + NATS_JETSTREAM_ACK_TIMEOUT_SECS, NATS_JETSTREAM_ENABLE, NATS_JETSTREAM_STREAM_NAME, NATS_PASSWORD, NATS_QUEUE_DIR, + NATS_SUBJECT, NATS_TOKEN, NATS_USERNAME, PULSAR_TLS_ALLOW_INSECURE, PULSAR_TLS_CA, PULSAR_TLS_HOSTNAME_VERIFICATION, + PULSAR_TOPIC, }; use std::str::FromStr; + fn nats_server() -> ServerAddr { + ServerAddr::from_str("nats://127.0.0.1:4222").expect("valid nats address") + } + + // Absolute on Linux, macOS, and Windows. temp_dir needs no filesystem to exist for a + // validation-only test, and Path::is_absolute stays true across platforms. + fn nats_queue_dir() -> String { + std::env::temp_dir().join("rustfs-nats-queue").to_string_lossy().into_owned() + } + #[test] fn validate_nats_server_config_rejects_multiple_auth_methods() { let server = ServerAddr::from_str("nats://127.0.0.1:4222").expect("valid nats address"); @@ -228,6 +273,131 @@ mod tests { assert!(err.to_string().contains("absolute path")); } + #[test] + fn validate_nats_server_config_rejects_jetstream_without_stream() { + let mut config = KVS::new(); + config.insert(NATS_SUBJECT.to_string(), "events".to_string()); + config.insert(NATS_QUEUE_DIR.to_string(), nats_queue_dir()); + config.insert(NATS_JETSTREAM_ENABLE.to_string(), "on".to_string()); + + let err = validate_nats_server_config(&nats_server(), &config, "") + .expect_err("enabled jetstream without a stream should be rejected"); + assert!(err.to_string().contains(NATS_JETSTREAM_STREAM_NAME)); + } + + #[test] + fn validate_nats_server_config_rejects_jetstream_without_queue_dir() { + let mut config = KVS::new(); + config.insert(NATS_SUBJECT.to_string(), "events".to_string()); + config.insert(NATS_JETSTREAM_ENABLE.to_string(), "on".to_string()); + config.insert(NATS_JETSTREAM_STREAM_NAME.to_string(), "RUSTFS_EVENTS".to_string()); + + let err = validate_nats_server_config(&nats_server(), &config, "") + .expect_err("enabled jetstream without a queue_dir should be rejected"); + assert!(err.to_string().contains(NATS_QUEUE_DIR)); + } + + #[test] + fn validate_nats_server_config_rejects_out_of_range_ack_timeout() { + let mut config = KVS::new(); + config.insert(NATS_SUBJECT.to_string(), "events".to_string()); + config.insert(NATS_JETSTREAM_ACK_TIMEOUT_SECS.to_string(), "5".to_string()); + + let err = + validate_nats_server_config(&nats_server(), &config, "").expect_err("out-of-range ack timeout should be rejected"); + assert!(err.to_string().contains("between")); + } + + #[test] + fn validate_nats_server_config_accepts_enabled_jetstream_with_stream_and_queue() { + let mut config = KVS::new(); + config.insert(NATS_SUBJECT.to_string(), "events".to_string()); + config.insert(NATS_QUEUE_DIR.to_string(), nats_queue_dir()); + config.insert(NATS_JETSTREAM_ENABLE.to_string(), "on".to_string()); + config.insert(NATS_JETSTREAM_STREAM_NAME.to_string(), "RUSTFS_EVENTS".to_string()); + + validate_nats_server_config(&nats_server(), &config, "").expect("a stream name and queue dir accept enabling jetstream"); + } + + #[test] + fn validate_nats_server_config_audit_parity_matches_notify() { + let mut config = KVS::new(); + config.insert(NATS_SUBJECT.to_string(), "events".to_string()); + config.insert(NATS_QUEUE_DIR.to_string(), nats_queue_dir()); + config.insert(NATS_JETSTREAM_ENABLE.to_string(), "on".to_string()); + + // The same invalid combination (enabled, stream name missing) is rejected identically on the + // notify and the audit path, proving one shared validator with no drift. + let notify = validate_nats_server_config(&nats_server(), &config, "/opt/rustfs/events").expect_err("notify rejects"); + let audit = validate_nats_server_config(&nats_server(), &config, "/opt/rustfs/audit").expect_err("audit rejects"); + assert_eq!(notify.to_string(), audit.to_string()); + } + + #[test] + fn validate_nats_server_config_accepts_jetstream_absent() { + let mut config = KVS::new(); + config.insert(NATS_SUBJECT.to_string(), "events".to_string()); + config.insert(NATS_QUEUE_DIR.to_string(), nats_queue_dir()); + + validate_nats_server_config(&nats_server(), &config, "").expect("baseline config validates"); + } + + #[test] + fn parse_jetstream_enable_accepts_the_standard_boolean_forms() { + // An absent or blank key is None, legitimately disabled. + assert_eq!(parse_jetstream_enable(&KVS::new()).unwrap(), None, "an absent key parses as unset"); + let mut blank = KVS::new(); + blank.insert(NATS_JETSTREAM_ENABLE.to_string(), " ".to_string()); + assert_eq!(parse_jetstream_enable(&blank).unwrap(), None, "a blank value parses as unset"); + + for (value, expected) in [ + ("on", true), + ("off", false), + ("true", true), + ("false", false), + ("yes", true), + ("no", false), + ("1", true), + ("0", false), + ("enabled", true), + ("disabled", false), + ] { + let mut config = KVS::new(); + config.insert(NATS_JETSTREAM_ENABLE.to_string(), value.to_string()); + assert_eq!( + parse_jetstream_enable(&config).unwrap(), + Some(expected), + "value {value} parses as {expected}" + ); + } + } + + #[test] + fn parse_jetstream_enable_rejects_an_unparsable_value_naming_the_key() { + // A typo must fail validation rather than silently disabling the durability guarantee. + for value in ["y", "t", "enabl", "10"] { + let mut config = KVS::new(); + config.insert(NATS_JETSTREAM_ENABLE.to_string(), value.to_string()); + let err = parse_jetstream_enable(&config).expect_err("an unparsable value is rejected"); + assert!( + err.to_string().contains(NATS_JETSTREAM_ENABLE), + "the error names the key for value {value}: {err}" + ); + } + } + + #[test] + fn validate_nats_server_config_rejects_an_invalid_jetstream_enable_value() { + let mut config = KVS::new(); + config.insert(NATS_SUBJECT.to_string(), "events".to_string()); + config.insert(NATS_QUEUE_DIR.to_string(), nats_queue_dir()); + config.insert(NATS_JETSTREAM_ENABLE.to_string(), "y".to_string()); + + let err = validate_nats_server_config(&nats_server(), &config, "") + .expect_err("an unparsable enable value fails validation instead of passing as disabled"); + assert!(err.to_string().contains(NATS_JETSTREAM_ENABLE), "the error names the key: {err}"); + } + #[test] fn validate_pulsar_broker_config_rejects_tls_ca_without_tls_scheme() { let mut config = KVS::new(); diff --git a/crates/targets/src/config/target_args.rs b/crates/targets/src/config/target_args.rs index f32741e0e..6f50c5d6e 100644 --- a/crates/targets/src/config/target_args.rs +++ b/crates/targets/src/config/target_args.rs @@ -13,7 +13,8 @@ // limitations under the License. use super::common::{ - parse_target_bool, parse_url, validate_nats_server_config, validate_outbound_http_url, validate_pulsar_broker_config, + parse_jetstream_ack_timeout_secs, parse_jetstream_enable, parse_target_bool, parse_url, validate_nats_server_config, + validate_outbound_http_url, validate_pulsar_broker_config, }; use crate::error::TargetError; use crate::target::{ @@ -38,17 +39,17 @@ use rustfs_config::{ MQTT_KEEP_ALIVE_INTERVAL, MQTT_PASSWORD, MQTT_QOS, MQTT_QUEUE_DIR, MQTT_QUEUE_LIMIT, MQTT_RECONNECT_INTERVAL, MQTT_TLS_CA, MQTT_TLS_CLIENT_CERT, MQTT_TLS_CLIENT_KEY, MQTT_TLS_POLICY, MQTT_TLS_TRUST_LEAF_AS_CA, MQTT_TOPIC, MQTT_USERNAME, MQTT_WS_PATH_ALLOWLIST, MYSQL_DSN_STRING, MYSQL_FORMAT, MYSQL_MAX_OPEN_CONNECTIONS, MYSQL_QUEUE_DIR, MYSQL_QUEUE_LIMIT, - MYSQL_TABLE, MYSQL_TLS_CA, MYSQL_TLS_CLIENT_CERT, MYSQL_TLS_CLIENT_KEY, NATS_ADDRESS, NATS_CREDENTIALS_FILE, NATS_PASSWORD, - NATS_QUEUE_DIR, NATS_QUEUE_LIMIT, NATS_SUBJECT, NATS_TLS_CA, NATS_TLS_CLIENT_CERT, NATS_TLS_CLIENT_KEY, NATS_TLS_REQUIRED, - NATS_TOKEN, NATS_USERNAME, POSTGRES_DSN_STRING, POSTGRES_FORMAT, POSTGRES_QUEUE_DIR, POSTGRES_QUEUE_LIMIT, POSTGRES_TABLE, - POSTGRES_TLS_CA, POSTGRES_TLS_CLIENT_CERT, POSTGRES_TLS_CLIENT_KEY, POSTGRES_TLS_REQUIRED, PULSAR_AUTH_TOKEN, PULSAR_BROKER, - PULSAR_PASSWORD, PULSAR_QUEUE_DIR, PULSAR_QUEUE_LIMIT, PULSAR_TLS_ALLOW_INSECURE, PULSAR_TLS_CA, - PULSAR_TLS_HOSTNAME_VERIFICATION, PULSAR_TOPIC, PULSAR_USERNAME, REDIS_CHANNEL, REDIS_CONNECTION_TIMEOUT, - REDIS_KEEP_ALIVE_INTERVAL, REDIS_MAX_RETRY_ATTEMPTS, REDIS_MAX_RETRY_DELAY, REDIS_MIN_RETRY_DELAY, REDIS_PASSWORD, - REDIS_PIPELINE_BUFFER_SIZE, REDIS_QUEUE_DIR, REDIS_QUEUE_LIMIT, REDIS_RECONNECT_RETRY_ATTEMPTS, REDIS_RESPONSE_TIMEOUT, - REDIS_TLS_ALLOW_INSECURE, REDIS_TLS_CA, REDIS_TLS_CLIENT_CERT, REDIS_TLS_CLIENT_KEY, REDIS_TLS_POLICY, REDIS_URL, - REDIS_USERNAME, RUSTFS_WEBHOOK_SKIP_TLS_VERIFY_DEFAULT, WEBHOOK_AUTH_TOKEN, WEBHOOK_CLIENT_CA, WEBHOOK_CLIENT_CERT, - WEBHOOK_CLIENT_KEY, WEBHOOK_ENDPOINT, WEBHOOK_QUEUE_DIR, WEBHOOK_QUEUE_LIMIT, WEBHOOK_SKIP_TLS_VERIFY, + MYSQL_TABLE, MYSQL_TLS_CA, MYSQL_TLS_CLIENT_CERT, MYSQL_TLS_CLIENT_KEY, NATS_ADDRESS, NATS_CREDENTIALS_FILE, + NATS_JETSTREAM_STREAM_NAME, NATS_PASSWORD, NATS_QUEUE_DIR, NATS_QUEUE_LIMIT, NATS_SUBJECT, NATS_TLS_CA, NATS_TLS_CLIENT_CERT, + NATS_TLS_CLIENT_KEY, NATS_TLS_REQUIRED, NATS_TOKEN, NATS_USERNAME, POSTGRES_DSN_STRING, POSTGRES_FORMAT, POSTGRES_QUEUE_DIR, + POSTGRES_QUEUE_LIMIT, POSTGRES_TABLE, POSTGRES_TLS_CA, POSTGRES_TLS_CLIENT_CERT, POSTGRES_TLS_CLIENT_KEY, + POSTGRES_TLS_REQUIRED, PULSAR_AUTH_TOKEN, PULSAR_BROKER, PULSAR_PASSWORD, PULSAR_QUEUE_DIR, PULSAR_QUEUE_LIMIT, + PULSAR_TLS_ALLOW_INSECURE, PULSAR_TLS_CA, PULSAR_TLS_HOSTNAME_VERIFICATION, PULSAR_TOPIC, PULSAR_USERNAME, REDIS_CHANNEL, + REDIS_CONNECTION_TIMEOUT, REDIS_KEEP_ALIVE_INTERVAL, REDIS_MAX_RETRY_ATTEMPTS, REDIS_MAX_RETRY_DELAY, REDIS_MIN_RETRY_DELAY, + REDIS_PASSWORD, REDIS_PIPELINE_BUFFER_SIZE, REDIS_QUEUE_DIR, REDIS_QUEUE_LIMIT, REDIS_RECONNECT_RETRY_ATTEMPTS, + REDIS_RESPONSE_TIMEOUT, REDIS_TLS_ALLOW_INSECURE, REDIS_TLS_CA, REDIS_TLS_CLIENT_CERT, REDIS_TLS_CLIENT_KEY, + REDIS_TLS_POLICY, REDIS_URL, REDIS_USERNAME, RUSTFS_WEBHOOK_SKIP_TLS_VERIFY_DEFAULT, WEBHOOK_AUTH_TOKEN, WEBHOOK_CLIENT_CA, + WEBHOOK_CLIENT_CERT, WEBHOOK_CLIENT_KEY, WEBHOOK_ENDPOINT, WEBHOOK_QUEUE_DIR, WEBHOOK_QUEUE_LIMIT, WEBHOOK_SKIP_TLS_VERIFY, }; use std::path::Path; use std::time::Duration; @@ -315,6 +316,11 @@ pub fn build_nats_args(config: &KVS, default_queue_dir: &str, target_type: Targe .lookup(NATS_QUEUE_LIMIT) .and_then(|v| v.parse::().ok()) .unwrap_or(DEFAULT_LIMIT), + // Strict parse: an unparsable value is a configuration error, never a silent disable, so the + // durability guarantee the operator asked for cannot vanish over a typo. + jetstream_enable: parse_jetstream_enable(config)?, + jetstream_stream_name: config.lookup(NATS_JETSTREAM_STREAM_NAME).filter(|name| !name.is_empty()), + jetstream_ack_timeout_secs: parse_jetstream_ack_timeout_secs(config)?, target_type, }) } @@ -598,9 +604,9 @@ pub fn validate_mysql_config(config: &KVS, default_queue_dir: &str) -> Result<() #[cfg(test)] mod tests { use super::{ - build_amqp_args, build_kafka_args, build_mysql_args, build_postgres_args, build_redis_args, build_webhook_args, - validate_amqp_config, validate_kafka_config, validate_mysql_config, validate_postgres_config, validate_redis_config, - validate_webhook_config, + build_amqp_args, build_kafka_args, build_mysql_args, build_nats_args, build_postgres_args, build_redis_args, + build_webhook_args, validate_amqp_config, validate_kafka_config, validate_mysql_config, validate_postgres_config, + validate_redis_config, validate_webhook_config, }; use crate::target::{ TargetType, @@ -612,7 +618,8 @@ mod tests { AMQP_EXCHANGE, AMQP_MANDATORY, AMQP_PASSWORD, AMQP_PERSISTENT, AMQP_QUEUE_DIR, AMQP_ROUTING_KEY, AMQP_TLS_CLIENT_CERT, AMQP_TLS_CLIENT_KEY, AMQP_URL, AMQP_USERNAME, KAFKA_ACKS, KAFKA_BROKERS, KAFKA_SASL_ENABLE, KAFKA_SASL_MECHANISM, KAFKA_SASL_PASSWORD, KAFKA_SASL_USERNAME, KAFKA_TLS_ENABLE, KAFKA_TOPIC, MYSQL_DSN_STRING, MYSQL_MAX_OPEN_CONNECTIONS, - MYSQL_QUEUE_DIR, MYSQL_TABLE, MYSQL_TLS_CA, MYSQL_TLS_CLIENT_CERT, MYSQL_TLS_CLIENT_KEY, POSTGRES_DSN_STRING, + MYSQL_QUEUE_DIR, MYSQL_TABLE, MYSQL_TLS_CA, MYSQL_TLS_CLIENT_CERT, MYSQL_TLS_CLIENT_KEY, NATS_ADDRESS, + NATS_JETSTREAM_ACK_TIMEOUT_SECS, NATS_JETSTREAM_ENABLE, NATS_JETSTREAM_STREAM_NAME, NATS_SUBJECT, POSTGRES_DSN_STRING, POSTGRES_FORMAT, POSTGRES_QUEUE_DIR, POSTGRES_TABLE, POSTGRES_TLS_CA, POSTGRES_TLS_CLIENT_CERT, POSTGRES_TLS_CLIENT_KEY, REDIS_CHANNEL, REDIS_CONNECTION_TIMEOUT, REDIS_MAX_RETRY_DELAY, REDIS_MIN_RETRY_DELAY, REDIS_PIPELINE_BUFFER_SIZE, REDIS_RECONNECT_RETRY_ATTEMPTS, REDIS_RESPONSE_TIMEOUT, REDIS_TLS_ALLOW_INSECURE, REDIS_URL, WEBHOOK_ENDPOINT, @@ -1157,4 +1164,68 @@ mod tests { let err = validate_postgres_config(&config, "").expect_err("relative tls_client_cert should fail"); assert!(err.to_string().contains("must be an absolute path")); } + + fn nats_base_config() -> KVS { + let mut config = KVS::new(); + config.insert(NATS_ADDRESS.to_string(), "nats://127.0.0.1:4222".to_string()); + config.insert(NATS_SUBJECT.to_string(), "rustfs.events".to_string()); + config + } + + // Absolute on Linux, macOS, and Windows. temp_dir needs no filesystem to exist for a + // validation-only test, and Path::is_absolute stays true across platforms. + fn nats_queue_dir() -> String { + std::env::temp_dir().join("rustfs-nats-queue").to_string_lossy().into_owned() + } + + #[test] + fn build_nats_args_parses_jetstream_keys() { + let mut config = nats_base_config(); + config.insert(NATS_JETSTREAM_ENABLE.to_string(), "on".to_string()); + config.insert(NATS_JETSTREAM_STREAM_NAME.to_string(), "RUSTFS_EVENTS".to_string()); + config.insert(NATS_JETSTREAM_ACK_TIMEOUT_SECS.to_string(), "45".to_string()); + + for target_type in [TargetType::NotifyEvent, TargetType::AuditLog] { + let args = build_nats_args(&config, &nats_queue_dir(), target_type).expect("jetstream keys parse"); + assert_eq!(args.jetstream_enable, Some(true)); + assert_eq!(args.jetstream_stream_name.as_deref(), Some("RUSTFS_EVENTS")); + assert_eq!(args.jetstream_ack_timeout_secs, Some(45)); + } + } + + #[test] + fn build_nats_args_rejects_an_invalid_jetstream_enable_value() { + // A typo in the enable flag is a configuration error naming the key, never a silent + // disable that drops the durability guarantee the operator asked for. + let mut config = nats_base_config(); + config.insert(NATS_JETSTREAM_ENABLE.to_string(), "t".to_string()); + + let err = build_nats_args(&config, &nats_queue_dir(), TargetType::NotifyEvent) + .expect_err("an unparsable enable value fails the build"); + assert!(err.to_string().contains(NATS_JETSTREAM_ENABLE), "the error names the key: {err}"); + } + + #[test] + fn build_nats_args_without_jetstream_keys_is_flag_off() { + let config = nats_base_config(); + let args = build_nats_args(&config, &nats_queue_dir(), TargetType::NotifyEvent).expect("baseline config parses"); + + assert_eq!(args.jetstream_enable, None); + assert_eq!(args.jetstream_stream_name, None); + assert_eq!(args.jetstream_ack_timeout_secs, None); + assert_eq!(args.address, "nats://127.0.0.1:4222"); + assert_eq!(args.subject, "rustfs.events"); + assert_eq!(args.queue_dir, nats_queue_dir()); + args.validate().expect("flag-off config validates"); + } + + #[test] + fn build_nats_args_rejects_non_integer_ack_timeout() { + let mut config = nats_base_config(); + config.insert(NATS_JETSTREAM_ACK_TIMEOUT_SECS.to_string(), "soon".to_string()); + + let err = build_nats_args(&config, &nats_queue_dir(), TargetType::NotifyEvent) + .expect_err("non-integer ack timeout should be rejected"); + assert!(err.to_string().contains("whole number of seconds")); + } } diff --git a/crates/targets/src/error.rs b/crates/targets/src/error.rs index c054cd3b2..bd29f02ca 100644 --- a/crates/targets/src/error.rs +++ b/crates/targets/src/error.rs @@ -52,6 +52,9 @@ pub enum TargetError { #[error("Request error: {0}")] Request(String), + #[error("JetStream publish error ({}): {detail}", if *.retryable { "retryable" } else { "terminal" })] + JetStreamPublish { retryable: bool, detail: String }, + #[error("Timeout error: {0}")] Timeout(String), diff --git a/crates/targets/src/runtime/mod.rs b/crates/targets/src/runtime/mod.rs index 4db68527f..4f83740eb 100644 --- a/crates/targets/src/runtime/mod.rs +++ b/crates/targets/src/runtime/mod.rs @@ -33,6 +33,37 @@ use std::{future::Future, pin::Pin, time::Duration}; use tokio::sync::{Semaphore, mpsc}; use tokio::task::JoinHandle; +/// Maximum number of replay attempts before a stored entry is exhausted. Each attempt runs one full +/// send (one ack wait at the configured timeout for a JetStream entry), then a backoff sleep before +/// the next. The JetStream duplicate-window validation derives its worst-case retry lifetime from the +/// attempt count and REPLAY_BASE_RETRY_DELAY through inter_attempt_backoff_sum, the same source the +/// sleep schedule builds on. Pinning tests hold each layer of the coupling: the shared per-attempt +/// term, the sum over the schedule, retry_lifetime against that sum, and the realized sleep in +/// replay_backoff_sleep under a paused clock. +pub(crate) const REPLAY_MAX_RETRIES: usize = 5; + +/// Base unit of the exponential replay backoff. The sleep before the retry at shift n is this +/// multiplied by 2^n. +pub(crate) const REPLAY_BASE_RETRY_DELAY: Duration = Duration::from_secs(2); + +/// Backoff sleep before the retry attempt at `shift`: REPLAY_BASE_RETRY_DELAY doubled `shift` times. +/// The single per-attempt term the replay sleep and the duplicate-window sum both derive from. +pub(crate) fn replay_backoff_term(shift: u32) -> Duration { + REPLAY_BASE_RETRY_DELAY.saturating_mul(1u32 << shift) +} + +/// Sum of the backoff sleeps that run between `attempts` replay sends. The retry at shift k is +/// preceded by replay_backoff_term(k) for k in 1..attempts, so attempts minus one terms contribute +/// and no sleep follows the final attempt. retry_lifetime derives its worst-case span from the same +/// sum, so the sleep schedule and the stream duplicate-window requirement move together. +pub(crate) fn inter_attempt_backoff_sum(attempts: usize) -> Duration { + let mut total = Duration::ZERO; + for shift in 1..attempts as u32 { + total = total.saturating_add(replay_backoff_term(shift)); + } + total +} + /// Shared target trait object used by the runtime manager. pub type SharedTarget = Arc + Send + Sync>; type ReplayHook = Arc) -> Pin + Send>> + Send + Sync>; @@ -163,6 +194,7 @@ pub struct RuntimeStatusSnapshot { #[derive(Debug, Clone, Default, PartialEq, Eq)] pub struct RuntimeTargetSnapshot { pub failed_messages: u64, + pub failed_store_length: u64, pub queue_length: u64, pub target_id: String, pub target_type: String, @@ -211,6 +243,7 @@ where target: SharedTarget, }, RetryExhausted { + detail: String, key: Key, target: SharedTarget, }, @@ -302,11 +335,9 @@ where self.remove_and_close(&target_id.to_string()).await } - /// Closes and removes every target, returning the id/error of each target - /// whose `close()` failed. Previously these flush/close errors were logged - /// and dropped, so a shutdown that failed to flush a target reported success. - /// Callers can now surface them (e.g. fail an explicit shutdown) while still - /// tearing down the rest of the runtime. + /// Closes and removes every target, returning the id and error of each target whose close failed. + /// Surfacing them lets a caller fail an explicit shutdown while still tearing down the rest of the + /// runtime. pub async fn clear_and_close(&mut self) -> Vec<(String, TargetError)> { let target_ids: Vec = self.targets.keys().cloned().collect(); let mut errors = Vec::new(); @@ -388,6 +419,7 @@ where fn snapshot_from_delivery(target_id: TargetID, delivery: TargetDeliverySnapshot) -> RuntimeTargetSnapshot { RuntimeTargetSnapshot { failed_messages: delivery.failed_messages, + failed_store_length: delivery.failed_store_length, queue_length: delivery.queue_length, target_id: target_id.to_string(), target_type: target_id.name, @@ -485,10 +517,8 @@ where (cancel_tx, join) } -/// Number of readable entries accumulated before a replay batch is flushed under -/// a single semaphore permit. The previous `!batch_keys.is_empty()` flush -/// condition was always true, so this effectively defaulted to 1 (a permit per -/// entry) and made `batch_timeout` dead code. +/// Number of readable entries accumulated before a replay batch is flushed under a single semaphore +/// permit. const REPLAY_BATCH_SIZE: usize = 16; /// Sleeps for `dur` unless a cancel signal arrives first. Returns `true` if @@ -503,6 +533,14 @@ async fn sleep_or_cancelled(dur: Duration, cancel_rx: &mut mpsc::Receiver<()>) - } } +/// Seeds an interval tracker one interval in the past so the first eligible tick runs the action +/// immediately. The monotonic clock starts at host boot, so a process started within one interval +/// of boot cannot represent an instant that far back. The seed falls back to now in that case and +/// the first run waits one full interval. +fn seed_interval_start(now: tokio::time::Instant, interval: Duration) -> tokio::time::Instant { + now.checked_sub(interval).unwrap_or(now) +} + async fn stream_replay_worker( store: &mut (dyn Store + Send), target: SharedTarget, @@ -514,18 +552,49 @@ async fn stream_replay_worker( ) where E: PluginEvent, { + // Lower bound between two failed-store expired-entry removal scans. Far below the retention TTL, + // so retention is unaffected, while keeping the read_dir scan off every idle tick. + const FAILED_STORE_PRUNE_INTERVAL: Duration = Duration::from_secs(60); + let mut batch_keys = Vec::with_capacity(REPLAY_BATCH_SIZE); let mut last_flush = tokio::time::Instant::now(); + // Seeded one interval back so the first eligible tick runs the removal. Near host boot the + // seed is now and the first removal waits one interval. + let mut last_prune = seed_interval_start(tokio::time::Instant::now(), FAILED_STORE_PRUNE_INTERVAL); loop { if cancel_rx.try_recv().is_ok() { return; } + // Remove expired failed-store entries on the replay tick rather than a separate timer, at most + // once per interval so the idle path does not scan the directory on every tick. Only a target + // that records terminal failures carries a failed store, so a target without one skips the scan. + if let Some(failed_store) = target.failed_store() + && last_prune.elapsed() >= FAILED_STORE_PRUNE_INTERVAL + { + // Run the stat-and-sort scan off the async worker thread. The owned handle shares the same + // directory and cached-count state through its Arc handles, so the scan reconciles the + // count the live handles read. + let maintenance_failed = failed_store.boxed_clone_failed(); + let outcome = tokio::task::spawn_blocking(move || maintenance_failed.prune_failed_store()).await; + match outcome { + Ok(Err(err)) => { + tracing::warn!(target_id = %target.id(), error = %err, "Failed to prune the failed-events store"); + } + Ok(Ok(_)) => {} + Err(join_err) => { + tracing::warn!(target_id = %target.id(), error = %join_err, "The failed-events maintenance task failed to join"); + } + } + last_prune = tokio::time::Instant::now(); + } + let keys = store.list(); if keys.is_empty() { if !batch_keys.is_empty() && last_flush.elapsed() >= batch_timeout { - if process_replay_batch(&mut batch_keys, target.clone(), &hook, semaphore.clone(), &mut cancel_rx).await { + if process_replay_batch(&*store, &mut batch_keys, target.clone(), &hook, semaphore.clone(), &mut cancel_rx).await + { return; } last_flush = tokio::time::Instant::now(); @@ -539,7 +608,8 @@ async fn stream_replay_worker( for key in keys { if cancel_rx.try_recv().is_ok() { if !batch_keys.is_empty() { - process_replay_batch(&mut batch_keys, target.clone(), &hook, semaphore.clone(), &mut cancel_rx).await; + process_replay_batch(&*store, &mut batch_keys, target.clone(), &hook, semaphore.clone(), &mut cancel_rx) + .await; } return; } @@ -573,25 +643,37 @@ async fn stream_replay_worker( // Flush once a full batch has accumulated or the batch has aged past // batch_timeout — real size/time-based batching, not once-per-entry. if batch_keys.len() >= REPLAY_BATCH_SIZE || last_flush.elapsed() >= batch_timeout { - if process_replay_batch(&mut batch_keys, target.clone(), &hook, semaphore.clone(), &mut cancel_rx).await { + if process_replay_batch(&*store, &mut batch_keys, target.clone(), &hook, semaphore.clone(), &mut cancel_rx).await + { return; } last_flush = tokio::time::Instant::now(); } } + // Flush a partial batch that has aged past batch_timeout, so a lone or slow-arriving entry is + // delivered rather than stranded waiting for a full batch to accumulate. + if !batch_keys.is_empty() && last_flush.elapsed() >= batch_timeout { + if process_replay_batch(&*store, &mut batch_keys, target.clone(), &hook, semaphore.clone(), &mut cancel_rx).await { + return; + } + last_flush = tokio::time::Instant::now(); + } + if sleep_or_cancelled(Duration::from_millis(100), &mut cancel_rx).await { return; } } } -/// Delivers a batch of queued entries under a single semaphore permit. +/// Delivers a batch of queued entries, each send bounded by a freshly acquired semaphore permit held +/// across the send and released before any backoff sleep or failed-store move. /// /// Returns `true` if a cancel signal was observed while processing (e.g. during /// retry backoff), so the caller can stop promptly instead of continuing to /// drain a store that a replacement worker may already own. async fn process_replay_batch( + store: &(dyn Store + Send), batch_keys: &mut Vec, target: SharedTarget, hook: &ReplayHook, @@ -601,32 +683,41 @@ async fn process_replay_batch( where E: PluginEvent, { - const MAX_RETRIES: usize = 5; - const BASE_RETRY_DELAY: Duration = Duration::from_secs(2); + const MAX_RETRIES: usize = REPLAY_MAX_RETRIES; if batch_keys.is_empty() { return false; } - let _permit = match semaphore { - Some(ref semaphore) => match semaphore.clone().acquire_owned().await { - Ok(permit) => Some(permit), - Err(err) => { - tracing::error!(error = %err, "Failed to acquire replay semaphore permit"); - batch_keys.clear(); - return false; - } - }, - None => None, - }; - let mut cancelled = false; 'keys: for key in batch_keys.iter() { let mut retry_count = 0usize; let mut success = false; + let mut last_retryable_detail = String::new(); while retry_count < MAX_RETRIES && !success { - match target.send_from_store(key.clone()).await { + // The permit bounds how many replay sends run concurrently across targets sharing the + // semaphore. It is held across the send await so the cap is real, then released before + // any backoff sleep or failed-store move, because the sleep blocks for the backoff and + // holding the shared permit across it would throttle the whole replay path. An absent + // semaphore means no bound (the audit path). + let permit = match &semaphore { + None => None, + Some(sem) => match sem.clone().acquire_owned().await { + Ok(permit) => Some(permit), + Err(err) => { + tracing::error!(error = %err, "Failed to acquire replay semaphore permit"); + // Drop the batch so its keys do not strand under the dedup guard. + batch_keys.clear(); + return cancelled; + } + }, + }; + + let result = target.send_from_store(key.clone()).await; + drop(permit); + + match result { Ok(_) => { hook(ReplayEvent::Delivered { key: key.clone(), @@ -636,8 +727,11 @@ where success = true; } Err(err) => match err { - TargetError::NotConnected | TargetError::Timeout(_) => { + TargetError::NotConnected + | TargetError::Timeout(_) + | TargetError::JetStreamPublish { retryable: true, .. } => { retry_count += 1; + last_retryable_detail = err.to_string(); hook(ReplayEvent::RetryableError { error: err, key: key.clone(), @@ -645,16 +739,29 @@ where target: target.clone(), }) .await; - - let jitter = Duration::from_millis(key.to_string().len() as u64 % 500); - let backoff = 1u32 << retry_count as u32; - // Observe cancellation during backoff so shutdown/reload is - // not blocked for the full (potentially many-second) delay. - if sleep_or_cancelled(BASE_RETRY_DELAY * backoff + jitter, cancel_rx).await { + // The backoff runs only when another attempt follows, so the final failure + // proceeds straight to the exhaustion handling. Cancellation is observed + // during the sleep so shutdown/reload is not blocked for the full + // (potentially many-second) delay. + if retry_count < MAX_RETRIES && replay_backoff_sleep(key, retry_count, cancel_rx).await { cancelled = true; break 'keys; } } + TargetError::JetStreamPublish { retryable: false, .. } => { + // The hook fires only on a completed move. On a failed move the entry stays + // live, the next scan retries the move, and a repaired failed-store + // directory heals without a restart, with the hook then firing exactly once. + if move_failed_entry(store, &target, key, &err, retry_count as u32).await { + hook(ReplayEvent::PermanentFailure { + error: err, + key: key.clone(), + target: target.clone(), + }) + .await; + } + break; + } TargetError::Dropped(reason) => { hook(ReplayEvent::Dropped { key: key.clone(), @@ -678,7 +785,10 @@ where } if retry_count >= MAX_RETRIES && !success { + // A retryable error that never succeeded within the bound leaves the entry on the live + // queue for the next scan. The exhaustion is reported through the hook for metrics. hook(ReplayEvent::RetryExhausted { + detail: last_retryable_detail, key: key.clone(), target: target.clone(), }) @@ -690,6 +800,30 @@ where cancelled } +/// Sleeps the exponential backoff for a retry attempt, with key-derived jitter. Returns `true` if a +/// cancel signal arrives first, so the caller can stop promptly instead of blocking for the full delay. +async fn replay_backoff_sleep(key: &Key, retry_count: usize, cancel_rx: &mut mpsc::Receiver<()>) -> bool { + let jitter = Duration::from_millis(key.to_string().len() as u64 % 500); + sleep_or_cancelled(replay_backoff_term(retry_count as u32) + jitter, cancel_rx).await +} + +/// Reacts to a terminal classification by delegating to the target's terminal-failure handling and +/// reports whether the entry was handled. The target moves the entry to its failed-events store and +/// reports true, or declines and reports false so the caller keeps the entry live and skips the +/// final-failure hook. A target without a terminal-failure store always declines. +async fn move_failed_entry( + store: &(dyn Store + Send), + target: &SharedTarget, + key: &Key, + error: &TargetError, + retry_count: u32, +) -> bool +where + E: PluginEvent, +{ + target.handle_terminal_failure(store, key, error, retry_count).await +} + #[cfg(test)] mod tests { use super::TargetRuntimeManager; @@ -703,6 +837,66 @@ mod tests { use std::sync::Arc; use std::sync::atomic::{AtomicUsize, Ordering}; + #[tokio::test(start_paused = true)] + async fn seed_interval_start_backdates_by_one_interval() { + // With one interval of clock history available, the seed is exactly one interval in the + // past, so the first eligible tick runs the action immediately. + let origin = tokio::time::Instant::now(); + tokio::time::advance(std::time::Duration::from_secs(60)).await; + let now = tokio::time::Instant::now(); + assert_eq!(super::seed_interval_start(now, std::time::Duration::from_secs(60)), origin); + } + + #[test] + fn seed_interval_start_falls_back_to_now_when_the_clock_is_too_young() { + // Duration::MAX reaches past the monotonic clock origin on any host, so the checked + // subtraction yields no instant and the seed falls back to now instead of panicking. + let now = tokio::time::Instant::now(); + assert_eq!(super::seed_interval_start(now, std::time::Duration::MAX), now); + } + + #[test] + fn inter_attempt_backoff_sum_matches_the_realized_sleep_schedule() { + // Sums the shared per-attempt term over the schedule shifts (1..REPLAY_MAX_RETRIES, no + // trailing sleep after the last attempt) and asserts it equals inter_attempt_backoff_sum. + // The sleep's own use of the term is pinned by the paused-clock test below. + let mut realized = std::time::Duration::ZERO; + for shift in 1..super::REPLAY_MAX_RETRIES as u32 { + realized += super::replay_backoff_term(shift); + } + assert_eq!(super::inter_attempt_backoff_sum(super::REPLAY_MAX_RETRIES), realized); + // 2s * (2 + 4 + 8 + 16) at the default base. + assert_eq!(realized, std::time::Duration::from_secs(60)); + } + + #[tokio::test(start_paused = true)] + async fn replay_backoff_sleep_sleeps_the_term_for_the_retry_plus_key_jitter() { + // Drives the sleep itself, so a shifted argument at its replay_backoff_term call (for + // example retry_count minus 1) fails here even though the term and sum tests still pass. + // The jitter is deterministic, the key string length modulo 500 milliseconds, so the + // virtual elapsed time is exact under the paused clock. + let key = Key { + name: "0198c0de-0000-7000-8000-000000000000".to_string(), + extension: ".event".to_string(), + item_count: 1, + compress: false, + }; + let jitter = std::time::Duration::from_millis(key.to_string().len() as u64 % 500); + let retry_count = 3usize; + + // The held sender keeps the cancel channel open so the sleep runs to its deadline. + let (_cancel_tx, mut cancel_rx) = tokio::sync::mpsc::channel::<()>(1); + let start = tokio::time::Instant::now(); + let cancelled = super::replay_backoff_sleep(&key, retry_count, &mut cancel_rx).await; + + assert!(!cancelled, "no cancel signal was sent"); + assert_eq!( + start.elapsed(), + super::replay_backoff_term(retry_count as u32) + jitter, + "the sleep is the term for this retry count plus the key-derived jitter" + ); + } + #[derive(Clone)] struct TestTarget { id: TargetID, @@ -835,4 +1029,1003 @@ mod tests { assert!(manager.is_empty()); assert!(exited.load(Ordering::SeqCst), "stop_all must await the worker to completion"); } + + mod classifier { + use super::super::{ReplayEvent, stream_replay_worker}; + use crate::arn::TargetID; + use crate::plugin::PluginEvent; + use crate::store::{FailedEventStore, Key, QueueStore, Store}; + use crate::target::{EntityTarget, QueuedPayload, QueuedPayloadMeta}; + use crate::{StoreError, Target, TargetError}; + use async_trait::async_trait; + use rustfs_s3_types::EventName; + use std::sync::Arc; + use std::sync::atomic::{AtomicUsize, Ordering}; + use std::time::Duration; + use tokio::sync::{Semaphore, mpsc}; + + /// Records each ReplayEvent class the worker emitted, so a test can assert no event class was + /// silently skipped. + #[derive(Default)] + struct ReplayTally { + delivered: AtomicUsize, + retryable: AtomicUsize, + dropped: AtomicUsize, + permanent_failure: AtomicUsize, + retry_exhausted: AtomicUsize, + unreadable: AtomicUsize, + } + + /// A target whose send_from_store returns a programmed error each call, so the replay + /// classifier can be exercised per error class against a real on-disk store. + #[derive(Clone)] + struct ProgrammedTarget { + id: TargetID, + error: Option>, + send_calls: Arc, + in_flight: Arc, + max_in_flight: Arc, + send_gate: Arc, + // A clone of the worker's store so the mock can delete on the Dropped or success path, as + // the real send_from_store does, rather than leaving the entry to be re-listed forever. + store: QueueStore, + // The failed-events handle the terminal move writes to. Defaults to the store itself and is + // overridden with a rejecting handle to model an unwritable failed store. + failed_handle: Arc, + } + + impl ProgrammedTarget { + fn new(error: Option, store: QueueStore) -> Self { + let failed_handle: Arc = Arc::new(store.clone()); + Self::new_with_failed_store(error, store, failed_handle) + } + + fn new_with_failed_store( + error: Option, + store: QueueStore, + failed_handle: Arc, + ) -> Self { + let send_gate = Arc::new(Semaphore::new(Semaphore::MAX_PERMITS)); + Self { + id: TargetID::new("target-a".to_string(), "nats".to_string()), + error: error.map(Arc::new), + send_calls: Arc::new(AtomicUsize::new(0)), + in_flight: Arc::new(AtomicUsize::new(0)), + max_in_flight: Arc::new(AtomicUsize::new(0)), + send_gate, + store, + failed_handle, + } + } + } + + /// Models the NATS target's terminal move for the classifier tests: reads the entry, encodes it + /// as a terminal failed entry, writes it to the failed-events store, then clears the live entry. + /// A failed write leaves the live entry in place for the next scan. The delete outcome mirrors + /// the production move, so a delete failure returns false rather than reporting the entry handled. + fn move_terminal_entry_for_test( + store: &(dyn Store + Send), + failed_store: &dyn FailedEventStore, + key: &Key, + error: &TargetError, + retry_count: u32, + ) -> bool { + let raw = match store.get_raw(key) { + Ok(raw) => raw, + Err(StoreError::NotFound) => return true, + Err(_) => return false, + }; + let Ok(queued) = QueuedPayload::decode(&raw) else { + return false; + }; + let resolved = crate::target::nats::resolve_dedup_id(&queued.meta.dedup_id, key); + let Ok(encoded) = crate::target::encode_failed_entry( + queued, + crate::target::FailedErrorClass::Terminal, + error, + retry_count, + &resolved, + ) else { + return false; + }; + if failed_store.put_failed_raw(&key.name, &encoded).is_err() { + return false; + } + // A missing entry is not a failure, any other delete error declines the move, matching the + // production delete which treats NotFound as done and propagates every other error. + matches!(store.del(key), Ok(()) | Err(StoreError::NotFound)) + } + + #[async_trait] + impl Target for ProgrammedTarget + where + E: PluginEvent, + { + fn id(&self) -> TargetID { + self.id.clone() + } + async fn is_active(&self) -> Result { + Ok(true) + } + async fn save(&self, _event: Arc>) -> Result<(), TargetError> { + Ok(()) + } + async fn send_raw_from_store(&self, _key: Key, _body: Vec, _meta: QueuedPayloadMeta) -> Result<(), TargetError> { + Ok(()) + } + async fn send_from_store(&self, key: Key) -> Result<(), TargetError> { + self.send_calls.fetch_add(1, Ordering::SeqCst); + let live = self.in_flight.fetch_add(1, Ordering::SeqCst) + 1; + self.max_in_flight.fetch_max(live, Ordering::SeqCst); + // The gate lets a test hold a send in flight to observe permit behaviour during the + // await. It is open by default. + let permit = self.send_gate.clone().acquire_owned().await; + drop(permit); + self.in_flight.fetch_sub(1, Ordering::SeqCst); + match self.error.as_deref() { + // A Dropped error means send_from_store already removed the invalid entry before + // returning, so the mock deletes it too. A retryable or terminal publish error + // leaves the entry in place, matching the real path where the entry is cleared only + // on Ok or by the failed-store move. + Some(TargetError::Dropped(reason)) => { + let _ = self.store.del(&key); + Err(TargetError::Dropped(reason.clone())) + } + Some(error) => Err(clone_error(error)), + None => { + // Success clears the entry, as the real default implementation does. + let _ = self.store.del(&key); + Ok(()) + } + } + } + async fn close(&self) -> Result<(), TargetError> { + Ok(()) + } + fn store(&self) -> Option<&(dyn Store + Send + Sync)> { + None + } + async fn handle_terminal_failure( + &self, + store: &(dyn Store + Send), + key: &Key, + error: &TargetError, + retry_count: u32, + ) -> bool { + move_terminal_entry_for_test(store, self.failed_handle.as_ref(), key, error, retry_count) + } + fn clone_dyn(&self) -> Box + Send + Sync> { + Box::new(self.clone()) + } + fn is_enabled(&self) -> bool { + true + } + } + + /// A store that delegates every operation to a real on-disk QueueStore but fails + /// put_failed_raw or del while the matching flag is set, modeling an unwritable failed-store + /// directory or a live queue whose delete fails, either of which an operator later repairs. + #[derive(Clone)] + struct FailingFailedStore { + inner: QueueStore, + fail_put_failed_raw: Arc, + fail_del: Arc, + } + + impl Store for FailingFailedStore { + type Error = StoreError; + type Key = Key; + + fn open(&self) -> Result<(), Self::Error> { + self.inner.open() + } + fn put(&self, item: Arc) -> Result { + self.inner.put(item) + } + fn put_multiple(&self, items: Vec) -> Result { + self.inner.put_multiple(items) + } + fn put_raw(&self, data: &[u8]) -> Result { + self.inner.put_raw(data) + } + fn get(&self, key: &Self::Key) -> Result { + self.inner.get(key) + } + fn get_multiple(&self, key: &Self::Key) -> Result, Self::Error> { + self.inner.get_multiple(key) + } + fn get_raw(&self, key: &Self::Key) -> Result, Self::Error> { + self.inner.get_raw(key) + } + fn del(&self, key: &Self::Key) -> Result<(), Self::Error> { + if self.fail_del.load(Ordering::SeqCst) { + return Err(StoreError::Internal("injected del failure".to_string())); + } + self.inner.del(key) + } + fn delete(&self) -> Result<(), Self::Error> { + self.inner.delete() + } + fn list(&self) -> Vec { + self.inner.list() + } + fn len(&self) -> usize { + self.inner.len() + } + fn is_empty(&self) -> bool { + self.inner.is_empty() + } + fn boxed_clone(&self) -> Box + Send + Sync> { + Box::new(self.clone()) + } + } + + impl FailedEventStore for FailingFailedStore { + fn put_failed_raw(&self, entry_name: &str, data: &[u8]) -> Result { + if self.fail_put_failed_raw.load(Ordering::SeqCst) { + return Err(StoreError::Internal("injected put_failed_raw failure".to_string())); + } + self.inner.put_failed_raw(entry_name, data) + } + fn prune_failed_store(&self) -> Result { + self.inner.prune_failed_store() + } + fn failed_len(&self) -> usize { + self.inner.failed_len() + } + fn boxed_clone_failed(&self) -> Box { + Box::new(self.clone()) + } + } + + fn clone_error(error: &TargetError) -> TargetError { + match error { + TargetError::NotConnected => TargetError::NotConnected, + TargetError::Timeout(value) => TargetError::Timeout(value.clone()), + TargetError::Dropped(value) => TargetError::Dropped(value.clone()), + TargetError::JetStreamPublish { retryable, detail } => TargetError::JetStreamPublish { + retryable: *retryable, + detail: detail.clone(), + }, + TargetError::Network(value) => TargetError::Network(value.clone()), + other => TargetError::Unknown(other.to_string()), + } + } + + fn temp_dir(name: &str) -> std::path::PathBuf { + std::env::temp_dir().join(format!("rustfs-runtime-{name}-{}", uuid::Uuid::new_v4())) + } + + fn seed_entry(store: &QueueStore, dedup_id: &str) -> Key { + let mut meta = QueuedPayloadMeta::new( + EventName::ObjectCreatedPut, + "bucket-a".to_string(), + "obj.txt".to_string(), + "application/json", + 7, + ); + meta.dedup_id = dedup_id.to_string(); + let payload = QueuedPayload::new(meta, br#"{"x":1}"#.to_vec()); + store.put_raw(&payload.encode().unwrap()).unwrap() + } + + /// Runs the replay worker against the seeded store for up to 600 virtual seconds, then cancels. + /// + /// The worker runs on a store clone (the QueueStore shares its state through an Arc), so the + /// caller's handle observes drain while the worker holds its own mutable borrow. Tests run with + /// a paused clock, so the production backoff sleeps cost no real time and the loop drains fast. + async fn run_worker(store: &mut QueueStore, target: Arc, tally: Arc) { + run_worker_until(store, target, tally, |_| false).await; + } + + /// Runs the replay worker like run_worker, polling the tally once per virtual second and + /// cancelling as soon as the stop condition holds, so a test over an entry that stays live + /// can observe exactly one retry cycle instead of every cycle the full window admits. + async fn run_worker_until( + store: &mut QueueStore, + target: Arc, + tally: Arc, + stop: impl Fn(&ReplayTally) -> bool, + ) { + let shared: Arc + Send + Sync> = target.clone(); + let (cancel_tx, cancel_rx) = mpsc::channel(1); + let hook = { + let tally = tally.clone(); + Arc::new(move |event: ReplayEvent| { + let tally = tally.clone(); + Box::pin(async move { + match event { + ReplayEvent::Delivered { .. } => tally.delivered.fetch_add(1, Ordering::SeqCst), + ReplayEvent::RetryableError { .. } => tally.retryable.fetch_add(1, Ordering::SeqCst), + ReplayEvent::Dropped { .. } => tally.dropped.fetch_add(1, Ordering::SeqCst), + ReplayEvent::PermanentFailure { .. } => tally.permanent_failure.fetch_add(1, Ordering::SeqCst), + ReplayEvent::RetryExhausted { .. } => tally.retry_exhausted.fetch_add(1, Ordering::SeqCst), + ReplayEvent::UnreadableEntry { .. } => tally.unreadable.fetch_add(1, Ordering::SeqCst), + }; + }) as std::pin::Pin + Send>> + }) as super::super::ReplayHook + }; + + let semaphore = Some(Arc::new(Semaphore::new(1))); + let mut worker_store = store.clone(); + let worker = tokio::spawn(async move { + stream_replay_worker( + &mut worker_store, + shared, + cancel_rx, + hook, + semaphore, + Duration::from_millis(10), + Duration::from_millis(10), + ) + .await; + }); + + // The paused clock makes the production backoff sleeps free, so the virtual window lets + // the worker fully process the seeded entry (a full retry exhaustion is about two + // virtual minutes) before the worker is cancelled. The once-per-virtual-second poll + // cancels within one second of the stop condition, well inside a retry cycle. + for _ in 0..600 { + if stop(&tally) { + break; + } + tokio::time::sleep(Duration::from_secs(1)).await; + } + let _ = cancel_tx.send(()).await; + let _ = worker.await; + } + + // A retryable JetStream publish error retries within the bound and, on exhaustion, leaves the + // entry on the live queue rather than moving it to the failed store. Covers TimedOut, + // BrokenPipe, MaxAckPending, and the bounded StreamNotFound path, all of which the classifier + // carries as a retryable publish error. + #[tokio::test(start_paused = true)] + async fn retryable_publish_error_retries_and_keeps_the_entry_live() { + for detail in ["timed out", "broken pipe", "max ack pending", "stream not found"] { + let dir = temp_dir(&format!("retryable-{}", detail.replace(' ', "-"))); + let mut store = QueueStore::::new_with_compression(&dir, 64, ".test", false); + store.open().unwrap(); + seed_entry(&store, "minted-id"); + + let target = Arc::new(ProgrammedTarget::new( + Some(TargetError::JetStreamPublish { + retryable: true, + detail: detail.to_string(), + }), + store.clone(), + )); + let tally = Arc::new(ReplayTally::default()); + // The entry stays live and is rescanned after exhaustion, so the run stops at the + // first exhaustion to observe one full retry cycle. + run_worker_until(&mut store, target.clone(), tally.clone(), |t| { + t.retry_exhausted.load(Ordering::SeqCst) >= 1 + }) + .await; + + assert!(tally.retryable.load(Ordering::SeqCst) >= 1, "{detail} retries at least once"); + assert_eq!(tally.retry_exhausted.load(Ordering::SeqCst), 1, "{detail} exhausts the bound once"); + assert_eq!(store.len(), 1, "{detail} keeps the entry on the live queue"); + assert_eq!(store.failed_len(), 0, "{detail} writes no failed entry"); + assert!(!dir.join("failed").exists(), "{detail} creates no failed directory"); + + let _ = store.delete(); + } + } + + // A terminal JetStream publish error writes a terminal failed entry on the first attempt with + // no retry, and the live entry is cleared. Covers MaxPayloadExceeded, WrongLastMessageId, + // WrongLastSequence, and the terminal Other code, all carried as a non-retryable publish error. + #[tokio::test(start_paused = true)] + async fn terminal_publish_error_writes_terminal_entry_without_retry() { + for detail in [ + "max payload exceeded", + "wrong last message id", + "wrong last sequence", + "stream sealed", + ] { + let dir = temp_dir(&format!("terminal-{}", detail.replace(' ', "-"))); + let mut store = QueueStore::::new_with_compression(&dir, 64, ".test", false); + store.open().unwrap(); + seed_entry(&store, "minted-id"); + + let target = Arc::new(ProgrammedTarget::new( + Some(TargetError::JetStreamPublish { + retryable: false, + detail: detail.to_string(), + }), + store.clone(), + )); + let tally = Arc::new(ReplayTally::default()); + run_worker(&mut store, target.clone(), tally.clone()).await; + + assert_eq!(tally.permanent_failure.load(Ordering::SeqCst), 1, "{detail} fails permanently once"); + assert_eq!(tally.retryable.load(Ordering::SeqCst), 0, "{detail} does not retry"); + assert_eq!(tally.retry_exhausted.load(Ordering::SeqCst), 0, "{detail} does not reach exhaustion"); + assert_eq!(store.len(), 0, "{detail} clears the live entry"); + assert_eq!(store.failed_len(), 1, "{detail} writes one terminal failed entry"); + + // The failed entry carries the terminal class and the full meta. + let failed_dir = dir.join("failed"); + let failed_file = std::fs::read_dir(&failed_dir).unwrap().next().unwrap().unwrap().path(); + let decoded = QueuedPayload::decode(&std::fs::read(&failed_file).unwrap()).unwrap(); + let failure = decoded.meta.failure.unwrap(); + assert_eq!(failure.error_class, crate::target::FailedErrorClass::Terminal); + assert_eq!(failure.nats_msg_id, "minted-id"); + assert_eq!(decoded.meta.bucket_name, "bucket-a"); + + let _ = store.delete(); + } + } + + // A Dropped entry is reported through the drop hook rather than silently discarded, and it + // writes no failed entry. + #[tokio::test(start_paused = true)] + async fn no_replay_path_silently_drops_an_event() { + // A Dropped error is recorded through the drop hook and writes no failed entry, because a + // dropped payload is an invalid entry the store itself removed, not a deliverable event. + let dir = temp_dir("nodrop-dropped"); + let mut store = QueueStore::::new_with_compression(&dir, 64, ".test", false); + store.open().unwrap(); + seed_entry(&store, "id"); + let target = Arc::new(ProgrammedTarget::new( + Some(TargetError::Dropped("invalid entry".to_string())), + store.clone(), + )); + let tally = Arc::new(ReplayTally::default()); + run_worker(&mut store, target.clone(), tally.clone()).await; + assert_eq!(tally.dropped.load(Ordering::SeqCst), 1, "a dropped entry is reported, not silent"); + assert_eq!(store.failed_len(), 0, "a dropped invalid entry writes no failed entry"); + + let _ = store.delete(); + } + + // The audit path and the notify path monomorphize the same generic replay worker, so the + // classifier and the failed-store write run identically for an audit entry type. Driving the + // worker over a non-String event type exercises the audit monomorphization. + #[tokio::test(start_paused = true)] + async fn audit_event_type_runs_the_identical_classifier_and_failed_store_logic() { + let dir = temp_dir("audit-parity"); + let mut store = QueueStore::::new_with_compression(&dir, 64, ".test", false); + store.open().unwrap(); + seed_entry(&store, "audit-id"); + + let shared: Arc + Send + Sync> = Arc::new(ProgrammedTarget::new( + Some(TargetError::JetStreamPublish { + retryable: false, + detail: "max payload exceeded".to_string(), + }), + store.clone(), + )); + let (cancel_tx, cancel_rx) = mpsc::channel(1); + let hook = Arc::new(move |_event: ReplayEvent| { + Box::pin(async move {}) as std::pin::Pin + Send>> + }) as super::super::ReplayHook; + let semaphore = Some(Arc::new(Semaphore::new(1))); + let worker = stream_replay_worker( + &mut store, + shared, + cancel_rx, + hook, + semaphore, + Duration::from_millis(10), + Duration::from_millis(10), + ); + let driver = async move { + tokio::time::sleep(Duration::from_secs(5)).await; + let _ = cancel_tx.send(()).await; + }; + tokio::join!(worker, driver); + + assert_eq!(store.len(), 0, "the audit live entry is cleared after the move"); + assert_eq!(store.failed_len(), 1, "the audit path writes a failed entry through the shared worker"); + + let _ = store.delete(); + } + + // The replay admission permit is held across the send await, so under a single-permit + // semaphore only one worker enters its send at a time. Two workers share a single-permit + // semaphore and a closed gate that holds every send in flight. The first worker admits and + // enters its gated send. The second worker blocks on the permit and never enters its send, so + // at most one send is in flight. The permit is still released before the backoff sleep, so a + // send that fails does not hold the shared permit across the backoff. + #[tokio::test] + async fn admission_permit_is_held_across_the_send_bounding_concurrent_sends() { + // A shared gate held closed keeps every send blocked inside its await, and a shared + // in-flight counter observes how many sends are concurrently in their await. + let shared_gate = Arc::new(Semaphore::new(Semaphore::MAX_PERMITS)); + shared_gate.forget_permits(Semaphore::MAX_PERMITS); + let in_flight = Arc::new(AtomicUsize::new(0)); + let max_in_flight = Arc::new(AtomicUsize::new(0)); + let semaphore = Some(Arc::new(Semaphore::new(1))); + + let mut stores = Vec::new(); + let mut workers = Vec::new(); + let mut cancels = Vec::new(); + for index in 0..2 { + let dir = temp_dir(&format!("permit-{index}")); + let store = QueueStore::::new_with_compression(&dir, 64, ".test", false); + store.open().unwrap(); + seed_entry(&store, &format!("id-{index}")); + + let mut target = ProgrammedTarget::new( + Some(TargetError::JetStreamPublish { + retryable: true, + detail: "timed out".to_string(), + }), + store.clone(), + ); + target.send_gate = shared_gate.clone(); + target.in_flight = in_flight.clone(); + target.max_in_flight = max_in_flight.clone(); + let shared: Arc + Send + Sync> = Arc::new(target); + + let (cancel_tx, cancel_rx) = mpsc::channel(1); + let hook = Arc::new(move |_event: ReplayEvent| { + Box::pin(async move {}) as std::pin::Pin + Send>> + }) as super::super::ReplayHook; + let semaphore = semaphore.clone(); + let mut worker_store = store.clone(); + let worker = tokio::spawn(async move { + stream_replay_worker( + &mut worker_store, + shared, + cancel_rx, + hook, + semaphore, + Duration::from_millis(10), + Duration::from_millis(10), + ) + .await; + }); + stores.push(store); + workers.push(worker); + cancels.push(cancel_tx); + } + + // Wait until the first worker has admitted and entered its gated send. + for _ in 0..200 { + if in_flight.load(Ordering::SeqCst) >= 1 { + break; + } + tokio::time::sleep(Duration::from_millis(5)).await; + } + // Give the second worker ample time to try to admit. It must block on the held permit and + // never enter its send. + tokio::time::sleep(Duration::from_millis(100)).await; + + assert_eq!( + max_in_flight.load(Ordering::SeqCst), + 1, + "the single permit is held across the send, so only one send is in flight at a time" + ); + + // The observation is complete. Abort the workers rather than draining them, so the test + // does not wait out the production backoff after the gated sends resume. + drop(cancels); + for worker in workers { + worker.abort(); + let _ = worker.await; + } + for store in stores { + let _ = store.delete(); + } + } + + // With the flag off the NATS target never produces a JetStream publish error, so the worker + // never writes the failed store. A non-JetStream connectivity error retries and exhausts + // through the existing path without creating the failed directory. + #[tokio::test(start_paused = true)] + async fn flag_off_path_creates_no_failed_directory() { + let dir = temp_dir("flag-off"); + let mut store = QueueStore::::new_with_compression(&dir, 64, ".test", false); + store.open().unwrap(); + seed_entry(&store, ""); + + // A flag-off target surfaces connectivity errors, not JetStreamPublish, so the failed-store + // move is never triggered. + let target = Arc::new(ProgrammedTarget::new(Some(TargetError::NotConnected), store.clone())); + let tally = Arc::new(ReplayTally::default()); + run_worker(&mut store, target.clone(), tally.clone()).await; + + assert!( + tally.retry_exhausted.load(Ordering::SeqCst) >= 1, + "the connectivity error exhausts its retries" + ); + assert!(!dir.join("failed").exists(), "no failed directory is created on the flag-off path"); + assert_eq!(store.failed_len(), 0); + + let _ = store.delete(); + } + + // A connect-level failure on a JetStream target surfaces as NotConnected, mapped at the + // publish path. The worker retries it with backoff and leaves the entry live at exhaustion, + // so a broker outage writes no failed-store entry and the event delivers when the connection + // recovers, matching the Core path. + #[tokio::test(start_paused = true)] + async fn jetstream_connect_failure_retries_and_keeps_the_entry_live() { + let dir = temp_dir("connect-failure"); + let mut store = QueueStore::::new_with_compression(&dir, 64, ".test", false); + store.open().unwrap(); + seed_entry(&store, "minted-id"); + + let target = Arc::new(ProgrammedTarget::new(Some(TargetError::NotConnected), store.clone())); + let tally = Arc::new(ReplayTally::default()); + // The live entry is rescanned after exhaustion, so the run stops at the first + // exhaustion to observe one full retry cycle. + run_worker_until(&mut store, target.clone(), tally.clone(), |t| { + t.retry_exhausted.load(Ordering::SeqCst) >= 1 + }) + .await; + + assert!( + tally.retryable.load(Ordering::SeqCst) >= 1, + "the connect failure retries within the bound" + ); + assert_eq!(tally.retry_exhausted.load(Ordering::SeqCst), 1, "the exhaustion hook fires exactly once"); + assert_eq!(store.len(), 1, "the entry stays live through the outage"); + assert_eq!(store.failed_len(), 0, "an outage writes no failed entry"); + assert!(!dir.join("failed").exists(), "no failed-store write occurs for a connect failure"); + + let _ = store.delete(); + } + + /// Spawns the replay worker over a FailingFailedStore and returns the cancel channel and + /// join handle, so a test can advance virtual time, repair the failed store mid-run, and + /// observe the hook tally across the repair. + fn spawn_failing_store_worker( + store: &FailingFailedStore, + target: Arc, + tally: Arc, + ) -> (mpsc::Sender<()>, tokio::task::JoinHandle<()>) { + let shared: Arc + Send + Sync> = target; + let (cancel_tx, cancel_rx) = mpsc::channel(1); + let hook = Arc::new(move |event: ReplayEvent| { + let tally = tally.clone(); + Box::pin(async move { + match event { + ReplayEvent::Delivered { .. } => tally.delivered.fetch_add(1, Ordering::SeqCst), + ReplayEvent::RetryableError { .. } => tally.retryable.fetch_add(1, Ordering::SeqCst), + ReplayEvent::Dropped { .. } => tally.dropped.fetch_add(1, Ordering::SeqCst), + ReplayEvent::PermanentFailure { .. } => tally.permanent_failure.fetch_add(1, Ordering::SeqCst), + ReplayEvent::RetryExhausted { .. } => tally.retry_exhausted.fetch_add(1, Ordering::SeqCst), + ReplayEvent::UnreadableEntry { .. } => tally.unreadable.fetch_add(1, Ordering::SeqCst), + }; + }) as std::pin::Pin + Send>> + }) as super::super::ReplayHook; + let semaphore = Some(Arc::new(Semaphore::new(1))); + let mut worker_store = store.clone(); + let worker = tokio::spawn(async move { + stream_replay_worker( + &mut worker_store, + shared, + cancel_rx, + hook, + semaphore, + Duration::from_millis(10), + Duration::from_millis(10), + ) + .await; + }); + (cancel_tx, worker) + } + + // A terminal failure whose failed-store move fails must not count as a final failure. No + // PermanentFailure hook fires, no failed entry lands, and the live entry stays for the next + // scan. Once the failed store is repaired, the move completes and the hook fires exactly + // once, so one lost event counts once rather than zero or once per scan. + #[tokio::test(start_paused = true)] + async fn a_failed_terminal_move_emits_no_hook_and_heals_once_repaired() { + let dir = temp_dir("move-fail-terminal"); + let inner = QueueStore::::new_with_compression(&dir, 64, ".test", false); + inner.open().unwrap(); + seed_entry(&inner, "minted-id"); + + let fail_flag = Arc::new(std::sync::atomic::AtomicBool::new(true)); + let store = FailingFailedStore { + inner: inner.clone(), + fail_put_failed_raw: fail_flag.clone(), + fail_del: Arc::new(std::sync::atomic::AtomicBool::new(false)), + }; + // The target's terminal move writes through the rejecting failed handle, so the injected + // failure is what the move hits. + let target = Arc::new(ProgrammedTarget::new_with_failed_store( + Some(TargetError::JetStreamPublish { + retryable: false, + detail: "max payload exceeded".to_string(), + }), + inner.clone(), + Arc::new(store.clone()), + )); + let tally = Arc::new(ReplayTally::default()); + let (cancel_tx, worker) = spawn_failing_store_worker(&store, target, tally.clone()); + + // Many scans run while the failed store rejects the move. + tokio::time::sleep(Duration::from_secs(300)).await; + assert_eq!( + tally.permanent_failure.load(Ordering::SeqCst), + 0, + "no final-failure hook fires while the move keeps failing" + ); + assert_eq!(inner.len(), 1, "the live entry stays for the next scan"); + assert_eq!(inner.failed_len(), 0, "no failed entry landed"); + + // The operator repairs the failed store. The next scan completes the move. + fail_flag.store(false, Ordering::SeqCst); + tokio::time::sleep(Duration::from_secs(300)).await; + assert_eq!( + tally.permanent_failure.load(Ordering::SeqCst), + 1, + "the hook fires exactly once for the completed move" + ); + assert_eq!(inner.len(), 0, "the live entry is cleared by the completed move"); + assert_eq!(inner.failed_len(), 1, "the failed entry landed once"); + + let _ = cancel_tx.send(()).await; + let _ = worker.await; + let _ = inner.delete(); + } + + // A terminal move whose failed-record write succeeds but whose live delete fails is not + // handled: no final-failure hook fires and the live entry stays for the next scan, while the + // durable failed copy is already in place and stays single through the idempotent overwrite. + #[tokio::test(start_paused = true)] + async fn a_move_with_a_failing_delete_reports_unhandled_and_keeps_the_entry() { + let dir = temp_dir("move-del-fail"); + let inner = QueueStore::::new_with_compression(&dir, 64, ".test", false); + inner.open().unwrap(); + seed_entry(&inner, "minted-id"); + + // The worker's live store rejects deletes while the failed handle (the plain inner store) + // accepts the failed-record write, so only the delete half of the move fails. + let store = FailingFailedStore { + inner: inner.clone(), + fail_put_failed_raw: Arc::new(std::sync::atomic::AtomicBool::new(false)), + fail_del: Arc::new(std::sync::atomic::AtomicBool::new(true)), + }; + let target = Arc::new(ProgrammedTarget::new( + Some(TargetError::JetStreamPublish { + retryable: false, + detail: "max payload exceeded".to_string(), + }), + inner.clone(), + )); + let tally = Arc::new(ReplayTally::default()); + let (cancel_tx, worker) = spawn_failing_store_worker(&store, target, tally.clone()); + + // Many scans run while the live delete keeps failing. + tokio::time::sleep(Duration::from_secs(300)).await; + assert_eq!( + tally.permanent_failure.load(Ordering::SeqCst), + 0, + "no final-failure hook fires while the delete fails" + ); + assert_eq!(inner.len(), 1, "the live entry stays for the next scan"); + assert_eq!(inner.failed_len(), 1, "the durable failed copy landed exactly once"); + + let _ = cancel_tx.send(()).await; + let _ = worker.await; + let _ = inner.delete(); + } + + // The fifth failure is the last attempt, so no backoff follows it. The exhaustion is reported + // after the four inter-attempt backoffs, 2s * (2 + 4 + 8 + 16) = 60s plus sub-second jitter, + // well inside the 100s bound asserted here. A trailing backoff after the final failure would + // add 64s more and push the report past the bound, failing the assertion. + #[tokio::test(start_paused = true)] + async fn exhaustion_reports_without_a_trailing_backoff() { + let dir = temp_dir("no-trailing-backoff"); + let store = QueueStore::::new_with_compression(&dir, 64, ".test", false); + store.open().unwrap(); + seed_entry(&store, "minted-id"); + + // The delegating wrapper with both failure flags off behaves as the plain on-disk store. + let wrapper = FailingFailedStore { + inner: store.clone(), + fail_put_failed_raw: Arc::new(std::sync::atomic::AtomicBool::new(false)), + fail_del: Arc::new(std::sync::atomic::AtomicBool::new(false)), + }; + let target = Arc::new(ProgrammedTarget::new( + Some(TargetError::JetStreamPublish { + retryable: true, + detail: "timed out".to_string(), + }), + store.clone(), + )); + let tally = Arc::new(ReplayTally::default()); + + let start = tokio::time::Instant::now(); + let (cancel_tx, worker) = spawn_failing_store_worker(&wrapper, target, tally.clone()); + + // Poll the virtual clock until the first exhaustion is reported. + while tally.retry_exhausted.load(Ordering::SeqCst) == 0 { + assert!(start.elapsed() < Duration::from_secs(600), "the exhaustion never reported"); + tokio::time::sleep(Duration::from_millis(100)).await; + } + let elapsed = start.elapsed(); + assert!( + elapsed < Duration::from_secs(100), + "the exhaustion reports after the inter-attempt backoffs only, got {elapsed:?}" + ); + assert_eq!(store.len(), 1, "the entry stays on the live queue"); + assert_eq!(store.failed_len(), 0, "a retryable exhaustion writes no failed entry"); + + let _ = cancel_tx.send(()).await; + let _ = worker.await; + let _ = store.delete(); + } + + // A raw Network error models a non-JetStream core target, whose connect failure keeps the core + // handling: a permanent failure left in the live queue, never diverted into the JetStream + // retryable path or the failed store. + #[tokio::test(start_paused = true)] + async fn core_network_error_is_not_diverted_to_the_failed_store() { + let dir = temp_dir("core-network"); + let mut store = QueueStore::::new_with_compression(&dir, 64, ".test", false); + store.open().unwrap(); + seed_entry(&store, ""); + + let target = Arc::new(ProgrammedTarget::new( + Some(TargetError::Network("connection refused".to_string())), + store.clone(), + )); + let tally = Arc::new(ReplayTally::default()); + run_worker(&mut store, target.clone(), tally.clone()).await; + + assert!( + tally.permanent_failure.load(Ordering::SeqCst) >= 1, + "a core Network error is reported permanent, not silently dropped" + ); + assert_eq!(tally.retryable.load(Ordering::SeqCst), 0, "a core Network error is not retried"); + assert_eq!( + tally.retry_exhausted.load(Ordering::SeqCst), + 0, + "a core Network error does not reach exhaustion" + ); + assert_eq!(store.failed_len(), 0, "a core Network error writes no failed entry"); + assert_eq!(store.len(), 1, "a core Network error is left queued, its existing behaviour"); + assert!(!dir.join("failed").exists(), "no failed directory is created for a core Network error"); + + let _ = store.delete(); + } + + // The single failed-store invariant: only a non-retryable error ever produces a failed-store + // entry. One retryable entry driven past the budget stays on the live queue while one terminal + // entry moves, so after both are processed the failed store holds exactly the terminal entry. + #[tokio::test(start_paused = true)] + async fn only_a_non_retryable_error_produces_a_failed_store_entry() { + // A target that reads each entry and returns a terminal error for the entry whose dedup id + // marks it terminal, and a retryable error for every other entry. + #[derive(Clone)] + struct ClassifyingTarget { + id: TargetID, + store: QueueStore, + } + #[async_trait] + impl Target for ClassifyingTarget { + fn id(&self) -> TargetID { + self.id.clone() + } + async fn is_active(&self) -> Result { + Ok(true) + } + async fn save(&self, _event: Arc>) -> Result<(), TargetError> { + Ok(()) + } + async fn send_raw_from_store( + &self, + _key: Key, + _body: Vec, + _meta: QueuedPayloadMeta, + ) -> Result<(), TargetError> { + Ok(()) + } + async fn send_from_store(&self, key: Key) -> Result<(), TargetError> { + let raw = self + .store + .get_raw(&key) + .map_err(|err| TargetError::Unknown(err.to_string()))?; + let decoded = QueuedPayload::decode(&raw).map_err(|err| TargetError::Unknown(err.to_string()))?; + if decoded.meta.dedup_id == "terminal-id" { + Err(TargetError::JetStreamPublish { + retryable: false, + detail: "max payload exceeded".to_string(), + }) + } else { + Err(TargetError::JetStreamPublish { + retryable: true, + detail: "timed out".to_string(), + }) + } + } + async fn close(&self) -> Result<(), TargetError> { + Ok(()) + } + fn store(&self) -> Option<&(dyn Store + Send + Sync)> { + None + } + async fn handle_terminal_failure( + &self, + store: &(dyn Store + Send), + key: &Key, + error: &TargetError, + retry_count: u32, + ) -> bool { + move_terminal_entry_for_test(store, &self.store, key, error, retry_count) + } + fn clone_dyn(&self) -> Box + Send + Sync> { + Box::new(self.clone()) + } + fn is_enabled(&self) -> bool { + true + } + } + + let dir = temp_dir("terminals-only-invariant"); + let store = QueueStore::::new_with_compression(&dir, 64, ".test", false); + store.open().unwrap(); + seed_entry(&store, "retryable-id"); + seed_entry(&store, "terminal-id"); + + let shared: Arc + Send + Sync> = Arc::new(ClassifyingTarget { + id: TargetID::new("target-a".to_string(), "nats".to_string()), + store: store.clone(), + }); + let tally = Arc::new(ReplayTally::default()); + let (cancel_tx, cancel_rx) = mpsc::channel(1); + let hook = { + let tally = tally.clone(); + Arc::new(move |event: ReplayEvent| { + let tally = tally.clone(); + Box::pin(async move { + match event { + ReplayEvent::Delivered { .. } => tally.delivered.fetch_add(1, Ordering::SeqCst), + ReplayEvent::RetryableError { .. } => tally.retryable.fetch_add(1, Ordering::SeqCst), + ReplayEvent::Dropped { .. } => tally.dropped.fetch_add(1, Ordering::SeqCst), + ReplayEvent::PermanentFailure { .. } => tally.permanent_failure.fetch_add(1, Ordering::SeqCst), + ReplayEvent::RetryExhausted { .. } => tally.retry_exhausted.fetch_add(1, Ordering::SeqCst), + ReplayEvent::UnreadableEntry { .. } => tally.unreadable.fetch_add(1, Ordering::SeqCst), + }; + }) as std::pin::Pin + Send>> + }) as super::super::ReplayHook + }; + let semaphore = Some(Arc::new(Semaphore::new(1))); + let mut worker_store = store.clone(); + let worker = tokio::spawn(async move { + stream_replay_worker( + &mut worker_store, + shared, + cancel_rx, + hook, + semaphore, + Duration::from_millis(10), + Duration::from_millis(10), + ) + .await; + }); + + // Drive until the terminal entry has moved and the retryable entry has exhausted at least + // once. Both share one batch, so one pass processes each. + for _ in 0..600 { + if store.failed_len() >= 1 && tally.retry_exhausted.load(Ordering::SeqCst) >= 1 { + break; + } + tokio::time::sleep(Duration::from_secs(1)).await; + } + let _ = cancel_tx.send(()).await; + let _ = worker.await; + + assert_eq!(store.failed_len(), 1, "the failed store holds exactly one entry"); + assert_eq!(store.len(), 1, "the retryable entry stays on the live queue"); + + // The one failed entry is the terminal one. + let failed_dir = dir.join("failed"); + let failed_file = std::fs::read_dir(&failed_dir).unwrap().next().unwrap().unwrap().path(); + let decoded = QueuedPayload::decode(&std::fs::read(&failed_file).unwrap()).unwrap(); + let failure = decoded.meta.failure.unwrap(); + assert_eq!(failure.error_class, crate::target::FailedErrorClass::Terminal); + assert_eq!(failure.nats_msg_id, "terminal-id"); + + let _ = store.delete(); + } + } } diff --git a/crates/targets/src/store.rs b/crates/targets/src/store.rs index e1e42353c..e890e5517 100644 --- a/crates/targets/src/store.rs +++ b/crates/targets/src/store.rs @@ -24,10 +24,10 @@ use std::{ marker::PhantomData, path::{Path, PathBuf}, sync::{ - Arc, RwLock, + Arc, Mutex, RwLock, atomic::{AtomicU64, Ordering}, }, - time::{SystemTime, UNIX_EPOCH}, + time::{Duration, SystemTime, UNIX_EPOCH}, }; use tracing::{debug, warn}; use uuid::Uuid; @@ -61,6 +61,59 @@ fn is_queue_file_name(file_name: &str, file_ext: &str) -> bool { base.ends_with(file_ext) } +/// Separator between the batch item count and the entry name in an on-disk batch filename. It must +/// stay outside the Windows reserved filename set so File::create succeeds on NTFS, and outside the +/// UUID alphabet so the count splits unambiguously when the name is parsed back. +const BATCH_COUNT_SEPARATOR: char = '_'; + +/// Legacy separator for the batch item count. Entries written with it are read back for +/// compatibility, but it is reserved on NTFS so it is never written. +const LEGACY_BATCH_COUNT_SEPARATOR: char = ':'; + +/// Name of the child directory inside a target queue directory that holds events which failed +/// terminally. It is created lazily on the first failed write, so a target that never fails +/// terminally creates no such directory. +const FAILED_STORE_SUBDIR: &str = "failed"; + +/// Maximum number of entries retained in the failed store per target, bounded independently of the +/// live queue so an accumulation of terminal failures cannot starve live events. At the bound the +/// oldest failed entry is dropped to admit the newer one, so a newer failure is never lost in favour +/// of an older one. +const FAILED_STORE_MAX_ENTRIES: usize = 10_000; + +/// Maximum age of a failed-store entry before it is removed as expired, measured from the file write +/// time, which is the instant the entry entered the failed store. +const FAILED_STORE_TTL: Duration = Duration::from_secs(72 * 60 * 60); + +/// Writes payload to a temp file in the same directory, flushes the file to disk, then atomically +/// renames it onto final_path. +/// +/// rename within a single directory is atomic on the supported filesystems, so a crash leaves +/// either final_path absent, the prior final_path, or the complete payload, never a partial file. +/// On failure the temp file is removed so an interrupted write leaves nothing behind. +fn write_temp_then_rename(temp_path: &Path, final_path: &Path, payload: &[u8]) -> Result<(), StoreError> { + if let Err(err) = write_and_sync_temp(temp_path, payload) { + let _ = std::fs::remove_file(temp_path); + return Err(err); + } + + if let Err(err) = std::fs::rename(temp_path, final_path) { + let _ = std::fs::remove_file(temp_path); + return Err(StoreError::Io(err)); + } + + Ok(()) +} + +/// Writes payload to path and flushes the file to disk before returning, so the bytes are durable +/// before the caller renames the file into place. +fn write_and_sync_temp(path: &Path, payload: &[u8]) -> Result<(), StoreError> { + let mut file = std::fs::File::create(path).map_err(StoreError::Io)?; + file.write_all(payload).map_err(StoreError::Io)?; + file.sync_all().map_err(StoreError::Io)?; + Ok(()) +} + fn resolve_queue_store_compression_from_env_value(value: Option<&str>) -> bool { value .and_then(|value| value.parse::().ok().map(|state| state.is_enabled())) @@ -88,8 +141,13 @@ pub struct Key { impl Key { /// Converts the key to a string (filename) pub fn to_key_string(&self) -> String { + self.to_key_string_with(BATCH_COUNT_SEPARATOR) + } + + /// Builds the filename using the given batch count separator. + fn to_key_string_with(&self, separator: char) -> String { let name_part = if self.item_count > 1 { - format!("{}:{}", self.item_count, self.name) + format!("{}{separator}{}", self.item_count, self.name) } else { self.name.clone() }; @@ -134,12 +192,16 @@ pub fn parse_key(s: &str) -> Key { name = name[..name.len() - COMPRESS_EXT.len()].to_string(); } - // Number of batch items parsed - if let Some(colon_pos) = name.find(':') - && let Ok(count) = name[..colon_pos].parse::() + // Number of batch items parsed. The current separator and the legacy one are both accepted so an + // entry written before the switch to the Windows-safe separator still reads back. The count is + // stripped only when it exceeds one, mirroring the render side which prefixes the count only for a + // real batch, so a hostile 1_ or 0_ prefix on a single-item name stays part of the name. + if let Some(separator_pos) = name.find([BATCH_COUNT_SEPARATOR, LEGACY_BATCH_COUNT_SEPARATOR]) + && let Ok(count) = name[..separator_pos].parse::() + && count > 1 { item_count = count; - name = name[colon_pos + 1..].to_string(); + name = name[separator_pos + 1..].to_string(); } // Resolve extension @@ -241,6 +303,33 @@ where fn boxed_clone(&self) -> Box + Send + Sync>; } +/// A failed-events store nested inside the queue directory for terminal entries, kept separate from the generic queue interface +/// so only the paths that record terminal failures carry the surface. +/// +/// The failed store holds events that could not be handed off and is bounded independently of the +/// live queue. At the count bound the oldest failed entry is dropped before a new entry is written, so +/// a newer failure is never lost in favour of an older one. +pub trait FailedEventStore: Send + Sync { + /// Writes pre-encoded bytes to the failed-events store under the given entry name and returns the + /// written entry id. + /// + /// The write is atomic. The entry name derives from the live entry, so a repeated move of the same + /// entry replaces its earlier failed file instead of accumulating duplicates. + fn put_failed_raw(&self, entry_name: &str, data: &[u8]) -> Result; + + /// Removes failed-store entries older than the retention bound and returns how many were removed. + /// + /// Runs on the replay maintenance tick, not a separate timer. + fn prune_failed_store(&self) -> Result; + + /// Returns the number of entries currently in the failed store. + fn failed_len(&self) -> usize; + + /// Clones the capability into an owned boxed handle sharing the same backing state, so the + /// maintenance scan can run inside a blocking task without borrowing the target. + fn boxed_clone_failed(&self) -> Box; +} + /// A store that uses the filesystem to persist events in a queue pub struct QueueStore { entry_limit: u64, @@ -249,6 +338,15 @@ pub struct QueueStore { compress: bool, entries: Arc>>, // key -> modtime as unix nano pending_entries: Arc, + /// Cached count of complete failed-store entries, shared across clones. Initialized by one + /// directory read in open() and kept current by the failed-store write, the capacity trim, and + /// the expired-entry removal, so failed_len() is a plain atomic load rather than a directory scan. + failed_count: Arc, + /// Serializes the failed-store writers and the maintenance scan among themselves. put_failed_raw, + /// the capacity trim it runs, and the expiry-plus-reconcile scan take this guard, so the at-bound + /// check and the write stay atomic and the scan reconcile cannot overwrite a concurrent write's + /// count update. The live queue path never takes it. Shared across clones through the Arc. + failed_store_guard: Arc>, fs_guard: Arc>, _phantom: PhantomData, } @@ -262,6 +360,8 @@ impl Clone for QueueStore { compress: self.compress, entries: Arc::clone(&self.entries), pending_entries: Arc::clone(&self.pending_entries), + failed_count: Arc::clone(&self.failed_count), + failed_store_guard: Arc::clone(&self.failed_store_guard), fs_guard: Arc::clone(&self.fs_guard), _phantom: PhantomData, } @@ -296,14 +396,26 @@ impl QueueStore { compress, entries: Arc::new(RwLock::new(HashMap::with_capacity((entry_limit as usize).min(MAX_PREALLOC_CAPACITY)))), pending_entries: Arc::new(AtomicU64::new(0)), + failed_count: Arc::new(AtomicU64::new(0)), + failed_store_guard: Arc::new(Mutex::new(())), fs_guard: Arc::new(RwLock::new(())), _phantom: PhantomData, } } - /// Returns the full path for a key + /// Returns the full path for a key. A batch entry (item_count > 1) may exist on disk under the + /// legacy separator, so when the current-separator path is absent and the legacy-separator path + /// exists, the legacy path is returned. Reads and deletes then resolve a legacy file while fresh + /// writes keep the current separator. fn file_path(&self, key: &Key) -> PathBuf { - self.directory.join(key.to_key_string()) + let path = self.directory.join(key.to_key_string()); + if key.item_count > 1 && !path.exists() { + let legacy = self.directory.join(key.to_key_string_with(LEGACY_BATCH_COUNT_SEPARATOR)); + if legacy.exists() { + return legacy; + } + } + path } fn build_key(&self, item_count: usize) -> Key { @@ -427,10 +539,10 @@ impl QueueStore { /// lost and no ghost/truncated entry is ever indexed. fn write_file(&self, key: &Key, data: &[u8]) -> Result { let path = self.file_path(key); - // Create directory if it doesn't exist - if let Some(parent) = path.parent() { - std::fs::create_dir_all(parent).map_err(StoreError::Io)?; - } + let parent = path + .parent() + .ok_or_else(|| StoreError::Internal(format!("store entry path {} has no parent directory", path.display())))?; + std::fs::create_dir_all(parent).map_err(StoreError::Io)?; let payload: std::borrow::Cow<'_, [u8]> = if key.compress { let mut encoder = Encoder::new(); @@ -448,26 +560,9 @@ impl QueueStore { self.directory.join(file_name) }; - // Write + fsync the full payload into the temp file, then atomically - // rename it into place. Any error path removes the temp file so a failed - // write cannot leave residue that would otherwise be cleaned only on the - // next open(). - let write_result = (|| -> Result<(), StoreError> { - let mut file = File::create(&tmp_path).map_err(StoreError::Io)?; - file.write_all(&payload).map_err(StoreError::Io)?; - file.sync_all().map_err(StoreError::Io)?; - Ok(()) - })(); - - if let Err(err) = write_result { - let _ = std::fs::remove_file(&tmp_path); - return Err(err); - } - - if let Err(err) = std::fs::rename(&tmp_path, &path) { - let _ = std::fs::remove_file(&tmp_path); - return Err(StoreError::Io(err)); - } + // One implementation of the write-fsync-rename invariant, shared with the failed-store write. + // The temp file is removed on any error path so a failed write leaves no residue. + write_temp_then_rename(&tmp_path, &path, &payload)?; // Best-effort: persist the new directory entry created by rename. Self::fsync_dir(&self.directory); @@ -512,6 +607,126 @@ impl QueueStore { } Ok(()) } + + /// Path of the failed-events child directory inside the live queue directory. + fn failed_dir(&self) -> PathBuf { + self.directory.join(FAILED_STORE_SUBDIR) + } + + /// Reads the failed directory once and counts complete entries, skipping non-file entries and + /// residual temp files from an interrupted write. Off the hot path, so it seeds the cached count + /// at open and reconciles it on the maintenance scan rather than serving failed_len. + fn count_failed_entries_on_disk(&self) -> u64 { + let read_dir = match std::fs::read_dir(self.failed_dir()) { + Ok(read_dir) => read_dir, + Err(_) => return 0, + }; + + let mut count = 0u64; + for entry in read_dir.flatten() { + if entry.file_name().to_string_lossy().ends_with(TMP_SUFFIX) { + continue; + } + // file_type reads the directory entry kind without a full stat where the platform records it. + if matches!(entry.file_type(), Ok(file_type) if file_type.is_file()) { + count += 1; + } + } + count + } + + /// Lowers the cached failed count by one removed entry, clamped at zero so a decrement can never + /// wrap. The maintenance scan reconciles any residual drift each interval. + fn decrement_failed_count(&self) { + // The closure always returns Some, so the update never fails and the Result is discarded. + let _ = self + .failed_count + .fetch_update(Ordering::SeqCst, Ordering::SeqCst, |current| Some(current.saturating_sub(1))); + } + + /// Maps a per-entry stat outcome inside the ordered failed scan. A NotFound error means the file + /// was removed between the directory listing and the stat, by the capacity trim or the + /// expired-entry removal on another handle, so the scan skips that entry (None) instead of + /// failing as a whole. Any other error still fails the scan. + fn failed_scan_entry_metadata(outcome: std::io::Result) -> Result, StoreError> { + match outcome { + Ok(metadata) => Ok(Some(metadata)), + Err(err) if err.kind() == std::io::ErrorKind::NotFound => Ok(None), + Err(err) => Err(StoreError::Io(err)), + } + } + + /// Filenames of current failed-store entries paired with their write time, oldest first. + /// + /// The directory is read fresh each call rather than indexed in memory, because the failed store + /// is a low-traffic operator surface and a fresh scan keeps it free of the live-queue bookkeeping. + /// A missing directory yields an empty list, so a target that never failed reports nothing. An + /// entry removed concurrently mid-scan is skipped. + fn failed_entries_oldest_first(&self) -> Result, StoreError> { + let dir = self.failed_dir(); + let read_dir = match std::fs::read_dir(&dir) { + Ok(read_dir) => read_dir, + Err(err) if err.kind() == std::io::ErrorKind::NotFound => return Ok(Vec::new()), + Err(err) => return Err(StoreError::Io(err)), + }; + + let mut entries = Vec::new(); + for entry in read_dir { + let entry = entry.map_err(StoreError::Io)?; + let Some(metadata) = Self::failed_scan_entry_metadata(entry.metadata())? else { + continue; + }; + if !metadata.is_file() { + continue; + } + let name = entry.file_name().to_string_lossy().to_string(); + // A residual temp file from an interrupted failed-store write is not a complete entry. + if name.ends_with(TMP_SUFFIX) { + let _ = std::fs::remove_file(entry.path()); + continue; + } + let written_at = metadata.modified().unwrap_or(UNIX_EPOCH); + entries.push((entry.path(), written_at)); + } + + entries.sort_by_key(|(_, written_at)| *written_at); + Ok(entries) + } + + /// Drops the oldest failed entry to admit a new one when the count bound is reached. The warn log + /// naming the trimmed entry fires after its unlink succeeds, so the log names exactly the entries + /// the capacity trim removed. + fn evict_oldest_failed_if_full(&self, current: &[(PathBuf, SystemTime)]) -> Result<(), StoreError> { + if current.len() < FAILED_STORE_MAX_ENTRIES { + return Ok(()); + } + + let drop_count = current.len() - FAILED_STORE_MAX_ENTRIES + 1; + for (path, _) in current.iter().take(drop_count) { + let evicted_id = path + .file_name() + .map(|name| name.to_string_lossy().to_string()) + .unwrap_or_default(); + match std::fs::remove_file(path) { + Ok(()) => { + self.decrement_failed_count(); + warn!( + event = EVENT_TARGET_STORE_STATE, + component = LOG_COMPONENT_TARGETS, + subsystem = LOG_SUBSYSTEM_STORE, + action = "failed_store_evict", + evicted_entry = %evicted_id, + reason = "capacity", + "target store state" + ); + } + // Already removed by another handle, so nothing was trimmed here and no log fires. + Err(err) if err.kind() == std::io::ErrorKind::NotFound => {} + Err(err) => return Err(StoreError::Io(err)), + } + } + Ok(()) + } } impl Store for QueueStore @@ -551,7 +766,7 @@ where continue; } - // Ignore foreign files that do not match our queue file extension so + // Ignore foreign files that do not match the queue file extension so // externally-dropped files cannot pollute the queue. if !is_queue_file_name(&file_name, &self.file_ext) { continue; @@ -570,6 +785,10 @@ where entries_map.insert(file_name, unix_nano); } + // Seed the cached failed count from one read of the failed directory, under the write guard + // already held, so failed_len() serves an atomic load rather than a directory scan. + self.failed_count.store(self.count_failed_entries_on_disk(), Ordering::SeqCst); + debug!( event = EVENT_TARGET_STORE_STATE, component = LOG_COMPONENT_TARGETS, @@ -675,10 +894,10 @@ where return Err(StoreError::Deserialization(format!("Failed to deserialize item in batch: {e}"))); } None => { - // Reached end of stream before deserializing `item_count` items: the + // Reached end of stream before deserializing item_count items: the // batch file was truncated or corrupted. This MUST be surfaced as an - // error rather than silently returning the partial set. If we returned - // `Ok(partial)`, the caller would treat the batch as fully delivered, + // error rather than silently returning the partial set. Returning + // Ok(partial) would let the caller treat the batch as fully delivered, // delete the store entry, and permanently lose the missing events. warn!( event = EVENT_TARGET_STORE_STATE, @@ -740,7 +959,13 @@ where .write() .map_err(|_| StoreError::Internal("Failed to acquire write lock on entries".to_string()))?; - if entries.remove(&key.to_key_string()).is_none() { + // A batch entry indexed under the legacy separator is removed under both renderings so its slot is reclaimed. + let removed_current = entries.remove(&key.to_key_string()).is_some(); + let removed_legacy = key.item_count > 1 + && entries + .remove(&key.to_key_string_with(LEGACY_BATCH_COUNT_SEPARATOR)) + .is_some(); + if !removed_current && !removed_legacy { debug!( event = EVENT_TARGET_STORE_STATE, component = LOG_COMPONENT_TARGETS, @@ -774,6 +999,7 @@ where .map_err(|_| StoreError::Internal("Failed to acquire write lock on entries".to_string()))?; entries.clear(); self.pending_entries.store(0, Ordering::SeqCst); + self.failed_count.store(0, Ordering::SeqCst); match std::fs::remove_dir_all(&self.directory) { Ok(()) => Ok(()), @@ -840,6 +1066,142 @@ where } } +impl FailedEventStore for QueueStore +where + T: Serialize + DeserializeOwned + Clone + Send + Sync + 'static, +{ + fn put_failed_raw(&self, entry_name: &str, data: &[u8]) -> Result { + // The name becomes a filename inside the failed directory, so a separator or a dot-only name + // is rejected before it can address a path outside it. + if entry_name.is_empty() || entry_name.contains(['/', '\\']) || entry_name == "." || entry_name == ".." { + return Err(StoreError::Internal(format!("invalid failed-store entry name: {entry_name}"))); + } + + // The seam holds across the at-bound check, the capacity trim, and the count update, so a + // second writer or the scan cannot interleave. It nests outside fs_guard, the order the scan + // uses too, so the two never invert. + let _failed_guard = self + .failed_store_guard + .lock() + .map_err(|_| StoreError::Internal("Failed to acquire the failed-store guard".to_string()))?; + let _fs_guard = self + .fs_guard + .read() + .map_err(|_| StoreError::Internal("Failed to acquire read lock on store filesystem".to_string()))?; + + let failed_dir = self.failed_dir(); + if !failed_dir.exists() { + std::fs::create_dir_all(&failed_dir).map_err(StoreError::Io)?; + // Best-effort: persist the lazily created directory entry in the queue directory, the + // same durability discipline the live write path applies after its rename. + Self::fsync_dir(&self.directory); + } + + let entry_id = entry_name.to_string(); + let final_path = failed_dir.join(&entry_id); + let is_new_entry = !final_path.exists(); + + // A repeated move of the same live entry lands on the same filename and replaces the earlier + // file. The count bound is enforced only when the write creates a new entry, since an + // overwrite does not grow the count and a capacity trim for it would drop a genuine entry. The + // cached count gates the ordered capacity scan with a plain atomic load, so a write below the + // bound skips the stat-and-sort entirely. The ordered scan runs only at the bound, oldest + // first, so the new failure is recorded even at capacity. + if is_new_entry && self.failed_len() >= FAILED_STORE_MAX_ENTRIES { + let existing = self.failed_entries_oldest_first()?; + self.evict_oldest_failed_if_full(&existing)?; + } + + let temp_path = failed_dir.join(format!("{entry_id}.{}{}", Uuid::new_v4(), TMP_SUFFIX)); + // rename replaces an existing destination file on the supported platforms, which the + // idempotent overwrite relies on. + write_temp_then_rename(&temp_path, &final_path, data)?; + + // A new failed file grows the cached count. An overwrite of an existing failed file leaves it + // unchanged. + if is_new_entry { + self.failed_count.fetch_add(1, Ordering::SeqCst); + } + + // Best-effort: persist the directory entry created by the rename, matching the live write + // path. + Self::fsync_dir(&failed_dir); + + debug!( + event = EVENT_TARGET_STORE_STATE, + component = LOG_COMPONENT_TARGETS, + subsystem = LOG_SUBSYSTEM_STORE, + action = "failed_store_write", + failed_entry = %entry_id, + "target store state" + ); + Ok(entry_id) + } + + fn prune_failed_store(&self) -> Result { + // Same seam and nesting order as put_failed_raw, so the reconcile store below rests on a + // listing no concurrent writer can shift, and the count it writes cannot overwrite a write's + // increment. + let _failed_guard = self + .failed_store_guard + .lock() + .map_err(|_| StoreError::Internal("Failed to acquire the failed-store guard".to_string()))?; + let _fs_guard = self + .fs_guard + .read() + .map_err(|_| StoreError::Internal("Failed to acquire read lock on store filesystem".to_string()))?; + + let entries = self.failed_entries_oldest_first()?; + let materialized_len = entries.len(); + let now = SystemTime::now(); + let mut pruned = 0usize; + for (path, written_at) in entries { + let age = now.duration_since(written_at).unwrap_or_default(); + if age < FAILED_STORE_TTL { + // Entries are oldest first, so the first within the retention bound ends the scan. + break; + } + let pruned_id = path + .file_name() + .map(|name| name.to_string_lossy().to_string()) + .unwrap_or_default(); + warn!( + event = EVENT_TARGET_STORE_STATE, + component = LOG_COMPONENT_TARGETS, + subsystem = LOG_SUBSYSTEM_STORE, + action = "failed_store_prune", + pruned_entry = %pruned_id, + reason = "ttl", + "target store state" + ); + match std::fs::remove_file(&path) { + Ok(()) => { + pruned += 1; + self.decrement_failed_count(); + } + Err(err) if err.kind() == std::io::ErrorKind::NotFound => {} + Err(err) => return Err(StoreError::Io(err)), + } + } + // Reconcile the cached count to this scan's materialized listing under the seam, so a change + // made outside the store drifts the value by at most one maintenance interval and no concurrent + // write is overwritten. + self.failed_count.store((materialized_len - pruned) as u64, Ordering::SeqCst); + Ok(pruned) + } + + fn failed_len(&self) -> usize { + // A plain load of the cached count seeded at open and kept current by the write, the capacity + // trim, and the expired-entry removal. The maintenance scan reconciles it to the directory + // each interval, so external drift is corrected within one interval. + self.failed_count.load(Ordering::SeqCst) as usize + } + + fn boxed_clone_failed(&self) -> Box { + Box::new(self.clone()) + } +} + #[cfg(test)] mod tests { use super::*; @@ -894,6 +1256,54 @@ mod tests { assert_eq!(parsed.compress, key.compress); } + // A batch filename must stay outside the Windows reserved character set so File::create succeeds + // on NTFS. The name carries a UUID, matching what build_key produces for a real entry. + #[test] + fn batch_key_filename_has_no_windows_reserved_characters() { + let key = Key { + name: Uuid::new_v4().to_string(), + extension: ".json".to_string(), + item_count: 7, + compress: true, + }; + + let file_name = key.to_key_string(); + + // The characters NTFS rejects in a filename, plus the path separators. + const WINDOWS_RESERVED: &[char] = &['<', '>', ':', '"', '/', '\\', '|', '?', '*']; + for reserved in WINDOWS_RESERVED { + assert!( + !file_name.contains(*reserved), + "batch filename {file_name} must not contain the reserved character {reserved}" + ); + } + assert!(file_name.contains(BATCH_COUNT_SEPARATOR), "the batch count uses the safe separator"); + } + + // An entry written with the legacy separator still reads back, so a queue populated before the + // switch to the Windows-safe separator is not misparsed after an upgrade. + #[test] + fn parse_key_reads_legacy_batch_separator() { + let legacy = format!("5{LEGACY_BATCH_COUNT_SEPARATOR}{}.json", "event-id"); + + let parsed = parse_key(&legacy); + + assert_eq!(parsed.item_count, 5); + assert_eq!(parsed.name, "event-id"); + assert_eq!(parsed.extension, ".json"); + } + + // The render side prefixes the count only when it exceeds one, so the parse side strips a count + // prefix only for the same range. A hostile 1_ or 0_ prefix stays part of the name and round-trips. + #[test] + fn parse_key_treats_a_count_of_one_or_zero_as_a_plain_name() { + for hostile in ["1_name.event", "0_name.event"] { + let parsed = parse_key(hostile); + assert_eq!(parsed.item_count, 1, "{hostile} is a single-item entry"); + assert_eq!(parsed.to_key_string(), hostile, "{hostile} round-trips as a plain name"); + } + } + #[test] fn put_raw_and_get_raw_round_trip_bytes() { let dir = temp_store_dir("raw-roundtrip"); @@ -947,8 +1357,8 @@ mod tests { assert_eq!(key.item_count, 3); // Truncate the batch file at a clean boundary after the first two serialized - // items (`"aa""bb"`), so the read of the third item hits end-of-stream — the - // exact "partial read" condition that previously returned Ok silently. + // items, so the read of the third item hits end-of-stream. This is the + // partial-read condition the batch reader must surface as an error. let prefix_len = serde_json::to_vec(&"aa".to_string()).unwrap().len() + serde_json::to_vec(&"bb".to_string()).unwrap().len(); let path = store.file_path(&key); @@ -1048,7 +1458,7 @@ mod tests { let key = store.put(Arc::new("payload".to_string())).unwrap(); // A non-empty file with an unrelated extension must not be indexed and - // must be left untouched (we only clean up our own residue). + // must be left untouched. Only residue from this store's own writes is cleaned up. let foreign_path = dir.join("intruder.txt"); std::fs::write(&foreign_path, b"not ours").unwrap(); @@ -1089,8 +1499,8 @@ mod tests { #[test] fn new_with_huge_limit_does_not_panic() { - // Previously `HashMap::with_capacity(entry_limit as usize)` would attempt - // a giant allocation / capacity overflow for an absurd configured limit. + // An absurd configured entry_limit must not drive a giant up-front HashMap + // allocation or a capacity-overflow panic. let dir = temp_store_dir("huge-limit"); let store = QueueStore::::new_with_compression(&dir, u64::MAX, ".test", false); store.open().unwrap(); @@ -1128,4 +1538,544 @@ mod tests { let _ = store.delete(); } + + fn count_temp_files(dir: &Path) -> usize { + match std::fs::read_dir(dir) { + Ok(read_dir) => read_dir + .filter_map(|entry| entry.ok()) + .filter(|entry| entry.file_name().to_string_lossy().ends_with(TMP_SUFFIX)) + .count(), + Err(_) => 0, + } + } + + /// Directory truth for the failed store: the number of complete failed files on disk, skipping + /// residual temp files, so a test can assert the cached failed_len matches the filesystem. + fn count_failed_files_on_disk(dir: &Path) -> usize { + match std::fs::read_dir(dir.join(FAILED_STORE_SUBDIR)) { + Ok(read_dir) => read_dir + .filter_map(|entry| entry.ok()) + .filter(|entry| { + let name = entry.file_name().to_string_lossy().into_owned(); + !name.ends_with(TMP_SUFFIX) && entry.file_type().map(|file_type| file_type.is_file()).unwrap_or(false) + }) + .count(), + Err(_) => 0, + } + } + + // A completed write leaves the full payload at the final path, readable, and no temp file. The + // atomic path renames the temp into place rather than writing the final path directly. + #[test] + fn write_leaves_complete_file_and_no_temp_residue() { + let dir = temp_store_dir("atomic-complete"); + let store = QueueStore::::new_with_compression(&dir, 8, ".test", false); + store.open().unwrap(); + + let payload = br#"{"kind":"notify","bucket":"demo","key":"complete.txt"}"#; + let key = store.put_raw(payload).unwrap(); + + let final_path = store.file_path(&key); + assert!(final_path.exists(), "final entry must exist after a complete write"); + assert_eq!(std::fs::read(&final_path).unwrap(), payload); + assert_eq!(store.get_raw(&key).unwrap(), payload); + assert_eq!(count_temp_files(&dir), 0, "a complete write leaves no temp file"); + + let _ = store.delete(); + } + + // A temp file from an interrupted write (one that never reached its rename) is not a valid + // entry. The open-time rescan removes it and never indexes it, so a partial write is absent + // rather than read as a corrupt entry. + #[test] + fn open_discards_residual_temp_file() { + let dir = temp_store_dir("atomic-residue"); + let store = QueueStore::::new_with_compression(&dir, 8, ".test", false); + store.open().unwrap(); + + let key = store.put_raw(br#"{"complete":true}"#).unwrap(); + + // Simulate a write interrupted before its rename. A half-written temp file is left in the + // store directory under the same naming scheme write_file uses. + let residual_temp = dir.join(format!("{}.{}{}", "2_orphan.test", Uuid::new_v4(), TMP_SUFFIX)); + std::fs::write(&residual_temp, b"partial payload, never renamed").unwrap(); + assert!(residual_temp.exists()); + + // Reopen to drive the rescan. + store.open().unwrap(); + + assert!(!residual_temp.exists(), "open must remove a residual temp file"); + assert_eq!(count_temp_files(&dir), 0); + assert_eq!(store.len(), 1, "only the complete entry is indexed"); + assert_eq!(store.get_raw(&key).unwrap(), br#"{"complete":true}"#); + + let _ = store.delete(); + } + + // Flag-independent. The queue round-trip (write, read, delete) is unchanged by the atomic write, + // for both the uncompressed and the snappy-compressed path. The atomic write changes how bytes + // land, not what round-trips. + #[test] + fn queue_round_trip_unchanged_uncompressed_and_compressed() { + for compress in [false, true] { + let dir = temp_store_dir("round-trip"); + let store = QueueStore::::new_with_compression(&dir, 8, ".test", compress); + store.open().unwrap(); + + let payload = br#"{"kind":"notify","bucket":"demo","key":"round-trip.txt"}"#; + let key = store.put_raw(payload).unwrap(); + assert_eq!(key.compress, compress); + + assert_eq!(store.get_raw(&key).unwrap(), payload, "read returns the written bytes"); + assert_eq!(store.len(), 1); + + store.del(&key).unwrap(); + assert!(matches!(store.get_raw(&key), Err(StoreError::NotFound))); + assert_eq!(store.len(), 0); + + let _ = store.delete(); + } + } + + // The temp file lives in the same directory as the final path, so the rename stays within one + // filesystem and is atomic. A cross-directory temp would break that guarantee. + #[test] + fn temp_file_shares_directory_with_final_path() { + let dir = temp_store_dir("same-dir"); + let final_path = dir.join("entry.test"); + let temp_path = dir.join(format!("entry.test.{}{}", Uuid::new_v4(), TMP_SUFFIX)); + + assert_eq!(temp_path.parent(), final_path.parent()); + } + + // The failed-events store is a child directory inside the live queue directory and is created only on the + // first failed write, so a target that never fails terminally leaves no failed directory. + #[test] + fn failed_store_directory_is_lazy_and_sibling_of_the_queue() { + let dir = temp_store_dir("failed-lazy"); + let store = QueueStore::::new_with_compression(&dir, 8, ".test", false); + store.open().unwrap(); + + let failed_dir = dir.join("failed"); + assert!(!failed_dir.exists(), "no failed directory before any failed write"); + assert_eq!(store.failed_len(), 0); + + let first_id = store.put_failed_raw("failed-entry-1", b"failed-entry-1").unwrap(); + assert!(failed_dir.exists(), "the failed directory is created on the first failed write"); + assert_eq!(store.failed_len(), 1); + assert_eq!( + store.failed_len(), + count_failed_files_on_disk(&dir), + "the cached count matches the directory truth" + ); + assert_eq!( + std::fs::read(failed_dir.join(&first_id)).unwrap(), + b"failed-entry-1", + "the lazy-create write lands the full payload" + ); + + // A later write finds the directory already present and takes the non-create branch. + let second_id = store.put_failed_raw("failed-entry-2", b"failed-entry-2").unwrap(); + assert_eq!(store.failed_len(), 2); + assert_eq!( + store.failed_len(), + count_failed_files_on_disk(&dir), + "the cached count matches the directory truth" + ); + assert_eq!( + std::fs::read(failed_dir.join(&second_id)).unwrap(), + b"failed-entry-2", + "a write into the existing directory lands the full payload" + ); + + let _ = store.delete(); + } + + // A repeated move of the same live entry lands on the same failed filename, so a persistent + // delete failure, or a crash that left the entry in both stores followed by another terminal + // failure, replaces the earlier failed file instead of accumulating duplicates. + #[test] + fn failed_store_write_is_idempotent_per_entry() { + let dir = temp_store_dir("failed-idempotent"); + let store = QueueStore::::new_with_compression(&dir, 8, ".test", false); + store.open().unwrap(); + + let first = store.put_failed_raw("entry-a", b"first-write").unwrap(); + let second = store.put_failed_raw("entry-a", b"second-write").unwrap(); + assert_eq!(first, second, "the same entry keeps the same failed filename"); + assert_eq!(store.failed_len(), 1, "a repeated move yields exactly one failed file"); + assert_eq!( + store.failed_len(), + count_failed_files_on_disk(&dir), + "an overwrite leaves the cached count matching the directory truth" + ); + assert_eq!( + std::fs::read(dir.join("failed").join(&second)).unwrap(), + b"second-write", + "the re-move replaces the earlier file" + ); + + let _ = store.delete(); + } + + // The failed filename is used inside the failed directory, so a name that could address a path + // outside it is rejected before any write. + #[test] + fn failed_store_write_rejects_a_path_escaping_entry_name() { + let dir = temp_store_dir("failed-name-guard"); + let store = QueueStore::::new_with_compression(&dir, 8, ".test", false); + store.open().unwrap(); + + for name in ["", ".", "..", "a/b", "a\\b"] { + assert!(store.put_failed_raw(name, b"payload").is_err(), "name {name:?} is rejected"); + } + assert_eq!(store.failed_len(), 0, "no rejected name produced a file"); + + let _ = store.delete(); + } + + // A per-entry NotFound stat inside the ordered failed scan means the file was removed between + // the directory listing and the stat, so that entry is skipped and the scan completes. Any other + // stat error still fails the scan. + #[test] + fn failed_scan_skips_an_entry_removed_mid_scan() { + let dir = temp_store_dir("failed-scan-skip"); + std::fs::create_dir_all(&dir).unwrap(); + let probe = dir.join("probe"); + std::fs::write(&probe, b"probe").unwrap(); + let metadata = std::fs::metadata(&probe).unwrap(); + + let present = QueueStore::::failed_scan_entry_metadata(Ok(metadata)).unwrap(); + assert!(present.is_some(), "a present entry passes its metadata through"); + + let removed = + QueueStore::::failed_scan_entry_metadata(Err(std::io::Error::from(std::io::ErrorKind::NotFound))).unwrap(); + assert!(removed.is_none(), "a concurrently removed entry is skipped, not an error"); + + let denied = + QueueStore::::failed_scan_entry_metadata(Err(std::io::Error::from(std::io::ErrorKind::PermissionDenied))); + assert!(denied.is_err(), "a non-NotFound stat error still fails the scan"); + + let _ = std::fs::remove_dir_all(&dir); + } + + // The failed store is bounded independently of the live queue. A live queue at its limit does not + // consume failed-store capacity and the reverse holds. + #[test] + fn failed_store_is_separate_from_the_live_queue_limit() { + let dir = temp_store_dir("failed-separate"); + let store = QueueStore::::new_with_compression(&dir, 2, ".test", false); + store.open().unwrap(); + + store.put_raw(b"live-1").unwrap(); + store.put_raw(b"live-2").unwrap(); + assert!(matches!(store.put_raw(b"live-3"), Err(StoreError::LimitExceeded))); + + // The live queue is full, yet failed writes still succeed and grow the separate failed store. + store.put_failed_raw("failed-1", b"failed-1").unwrap(); + store.put_failed_raw("failed-2", b"failed-2").unwrap(); + store.put_failed_raw("failed-3", b"failed-3").unwrap(); + assert_eq!(store.len(), 2, "the live queue stays at its own limit"); + assert_eq!(store.failed_len(), 3, "the failed store grows past the live limit"); + + let _ = store.delete(); + } + + // At the count bound the oldest failed entry is dropped to admit the newer one, so a newer + // terminal failure is never lost in preference to an older one. The directory is filled to the + // bound with plain files at distinct write times, then one real failed write triggers the + // capacity trim of the unambiguous oldest. + #[test] + fn failed_store_drops_oldest_at_the_count_bound() { + let dir = temp_store_dir("failed-evict"); + let store = QueueStore::::new_with_compression(&dir, 8, ".test", false); + store.open().unwrap(); + + let failed_dir = dir.join("failed"); + std::fs::create_dir_all(&failed_dir).unwrap(); + + let oldest_name = "0000-oldest"; + for index in 0..FAILED_STORE_MAX_ENTRIES { + let name = if index == 0 { + oldest_name.to_string() + } else { + format!("{index:06}-{}", Uuid::new_v4()) + }; + let path = failed_dir.join(&name); + std::fs::write(&path, b"prefilled").unwrap(); + let written_at = UNIX_EPOCH + Duration::from_secs(index as u64); + std::fs::OpenOptions::new() + .write(true) + .open(&path) + .unwrap() + .set_modified(written_at) + .unwrap(); + } + // The prefill lands entries directly on disk, so re-open to seed the cached count from the + // failed directory, the same path a restart with existing failed entries takes. + store.open().unwrap(); + assert_eq!(store.failed_len(), FAILED_STORE_MAX_ENTRIES); + assert_eq!( + store.failed_len(), + count_failed_files_on_disk(&dir), + "the seeded count matches the directory truth" + ); + + store.put_failed_raw("failed-newest", b"failed-newest").unwrap(); + assert_eq!(store.failed_len(), FAILED_STORE_MAX_ENTRIES, "the bound holds after the capacity trim"); + assert_eq!( + store.failed_len(), + count_failed_files_on_disk(&dir), + "the cached count matches the directory truth after the trim" + ); + assert!( + !failed_dir.join(oldest_name).exists(), + "the oldest failed entry is trimmed to admit the newer one" + ); + + let _ = store.delete(); + } + + // Entries older than the retention bound are removed as expired on the maintenance tick, while + // younger entries survive. The age is measured from the file write time. + #[test] + fn failed_store_prunes_entries_past_the_ttl() { + let dir = temp_store_dir("failed-ttl"); + let store = QueueStore::::new_with_compression(&dir, 8, ".test", false); + store.open().unwrap(); + + let failed_dir = dir.join("failed"); + + let stale_id = store.put_failed_raw("stale", b"stale").unwrap(); + let fresh_id = store.put_failed_raw("fresh", b"fresh").unwrap(); + + // Age the stale entry past the bound, keep the fresh entry recent. + let stale_time = SystemTime::now() - (FAILED_STORE_TTL + Duration::from_secs(60)); + std::fs::OpenOptions::new() + .write(true) + .open(failed_dir.join(&stale_id)) + .unwrap() + .set_modified(stale_time) + .unwrap(); + + let pruned = store.prune_failed_store().unwrap(); + assert_eq!(pruned, 1, "one entry is past the retention bound"); + assert!(!failed_dir.join(&stale_id).exists(), "the stale entry is removed as expired"); + assert!(failed_dir.join(&fresh_id).exists(), "the fresh entry survives"); + assert_eq!(store.failed_len(), 1, "the cached count drops with the expired removal"); + assert_eq!( + store.failed_len(), + count_failed_files_on_disk(&dir), + "the cached count matches the directory truth after expiry" + ); + + let _ = store.delete(); + } + + // Expired-entry removal on a store with no failed directory is a no-op, so a target that never + // failed terminally is not charged a maintenance error on the tick. + #[test] + fn failed_store_prune_is_a_noop_without_a_failed_directory() { + let dir = temp_store_dir("failed-prune-noop"); + let store = QueueStore::::new_with_compression(&dir, 8, ".test", false); + store.open().unwrap(); + + assert_eq!(store.prune_failed_store().unwrap(), 0); + assert!(!dir.join("failed").exists()); + + let _ = store.delete(); + } + + // failed_len counts complete failed entries and skips a residual temp file from an interrupted + // write. The count does not remove the temp file. Removal stays on the ordered scan run by the + // expired-entry removal. + #[test] + fn failed_len_counts_real_entries_and_ignores_temp_files() { + let dir = temp_store_dir("failed-len"); + let store = QueueStore::::new_with_compression(&dir, 8, ".test", false); + store.open().unwrap(); + + store.put_failed_raw("complete-one", b"complete-one").unwrap(); + store.put_failed_raw("complete-two", b"complete-two").unwrap(); + let failed_dir = dir.join("failed"); + let residual_temp = failed_dir.join(format!("orphan.{}{}", Uuid::new_v4(), TMP_SUFFIX)); + std::fs::write(&residual_temp, b"partial, never renamed").unwrap(); + + assert_eq!(store.failed_len(), 2, "only complete entries are counted"); + assert!(residual_temp.exists(), "the count does not remove the residual temp file"); + + // The ordered scan run by the expired-entry removal drops the residual temp file. + store.prune_failed_store().unwrap(); + assert!(!residual_temp.exists(), "the ordered scan removes the residual temp file"); + + let _ = store.delete(); + } + + // The maintenance scan reconciles the cached count to the directory truth, so a hand-planted + // drift is corrected on the next expired-entry removal, bounding external-mutation drift to one + // interval. + #[test] + fn maintenance_reconciles_a_hand_planted_count_drift() { + let dir = temp_store_dir("failed-reconcile"); + let store = QueueStore::::new_with_compression(&dir, 8, ".test", false); + store.open().unwrap(); + + store.put_failed_raw("entry-a", b"entry-a").unwrap(); + store.put_failed_raw("entry-b", b"entry-b").unwrap(); + assert_eq!(store.failed_len(), 2); + + // Plant a drift so the cached count disagrees with the directory truth. + store.failed_count.store(99, Ordering::SeqCst); + assert_eq!(store.failed_len(), 99, "the planted drift is observed before the reconcile"); + + // Both entries are within the retention bound, so the scan removes nothing and reconciles the + // count to the materialized listing. + let pruned = store.prune_failed_store().unwrap(); + assert_eq!(pruned, 0, "no entry is past the retention bound"); + assert_eq!(store.failed_len(), 2, "the reconcile restores the directory truth"); + assert_eq!(store.failed_len(), count_failed_files_on_disk(&dir)); + + let _ = store.delete(); + } + + // The maintenance scan holds the seam across its listing and reconcile, so a racing writer waits + // and its increment lands after the reconcile rather than being overwritten by it. The scan + // thread is parked inside the seam by the filesystem write guard, the writer is released against + // the held seam, and the final count must include the write. A scan that does not take the seam + // fails the held-seam wait below. + #[test] + fn maintenance_reconcile_does_not_overwrite_a_concurrent_failed_write() { + let dir = temp_store_dir("failed-reconcile-race"); + let store = Arc::new(QueueStore::::new_with_compression(&dir, 8, ".test", false)); + store.open().unwrap(); + + store.put_failed_raw("existing", b"existing").unwrap(); + // Plant a drift so the final value proves the reconcile ran and the write survived it. A + // reconcile that overwrites the write ends at 1, a write with no reconcile ends at 100, the + // serialized pair ends at 2. + store.failed_count.store(99, Ordering::SeqCst); + + // Hold the filesystem guard so the scan thread stops inside the seam, before its listing. + let fs_block = store.fs_guard.write().unwrap(); + + let scan_store = Arc::clone(&store); + let scan = thread::spawn(move || scan_store.prune_failed_store()); + + // Wait for the scan to take the seam. The deadline turns a scan that never takes the seam + // into a failure rather than a hang. + let deadline = std::time::Instant::now() + Duration::from_secs(5); + while store.failed_store_guard.try_lock().is_ok() { + assert!( + std::time::Instant::now() < deadline, + "the maintenance scan never took the failed-store seam" + ); + thread::yield_now(); + } + + // The writer is released while the scan holds the seam, so its write and increment wait for + // the reconcile. + let release = Arc::new(Barrier::new(2)); + let writer_store = Arc::clone(&store); + let writer_release = Arc::clone(&release); + let writer = thread::spawn(move || { + writer_release.wait(); + writer_store.put_failed_raw("racing", b"racing").unwrap(); + }); + release.wait(); + + drop(fs_block); + assert_eq!(scan.join().unwrap().unwrap(), 0, "both entries are within the retention bound"); + writer.join().unwrap(); + + assert_eq!(store.failed_len(), 2, "the reconcile does not overwrite the racing write"); + assert_eq!( + store.failed_len(), + count_failed_files_on_disk(&dir), + "the cached count matches the directory truth after the race" + ); + + let _ = store.delete(); + } + + // Two writers racing new failed entries at the count bound stay atomic through the seam, so the + // bound holds and the cached count matches the directory truth. Without the seam the at-bound + // check and the write interleave and the count overshoots the bound. + #[test] + fn concurrent_failed_writes_at_the_bound_hold_the_count() { + let dir = temp_store_dir("failed-race-bound"); + let store = Arc::new(QueueStore::::new_with_compression(&dir, 8, ".test", false)); + store.open().unwrap(); + + let failed_dir = dir.join("failed"); + std::fs::create_dir_all(&failed_dir).unwrap(); + for index in 0..FAILED_STORE_MAX_ENTRIES { + let path = failed_dir.join(format!("{index:06}-{}", Uuid::new_v4())); + std::fs::write(&path, b"prefilled").unwrap(); + let written_at = UNIX_EPOCH + Duration::from_secs(index as u64); + std::fs::OpenOptions::new() + .write(true) + .open(&path) + .unwrap() + .set_modified(written_at) + .unwrap(); + } + // Seed the cached count from the prefilled directory, the path a restart with existing entries takes. + store.open().unwrap(); + assert_eq!(store.failed_len(), FAILED_STORE_MAX_ENTRIES); + + let start = Arc::new(Barrier::new(2)); + let mut handles = Vec::new(); + for writer in 0..2 { + let store = Arc::clone(&store); + let start = Arc::clone(&start); + handles.push(thread::spawn(move || { + start.wait(); + store.put_failed_raw(&format!("racing-newest-{writer}"), b"newest").unwrap(); + })); + } + for handle in handles { + handle.join().unwrap(); + } + + assert_eq!(store.failed_len(), FAILED_STORE_MAX_ENTRIES, "the bound holds after two racing writes"); + assert_eq!( + store.failed_len(), + count_failed_files_on_disk(&dir), + "the cached count matches the directory truth after the race" + ); + + let _ = store.delete(); + } + + // Two writers racing the same failed entry name produce one failed file and one count increment, + // so a concurrent same-key move cannot double-count. + #[test] + fn concurrent_same_key_failed_writes_count_once() { + let dir = temp_store_dir("failed-race-same-key"); + let store = Arc::new(QueueStore::::new_with_compression(&dir, 8, ".test", false)); + store.open().unwrap(); + + let start = Arc::new(Barrier::new(2)); + let mut handles = Vec::new(); + for _ in 0..2 { + let store = Arc::clone(&store); + let start = Arc::clone(&start); + handles.push(thread::spawn(move || { + start.wait(); + store.put_failed_raw("same-entry", b"payload").unwrap(); + })); + } + for handle in handles { + handle.join().unwrap(); + } + + assert_eq!(store.failed_len(), 1, "two racing writes of one name count once"); + assert_eq!( + store.failed_len(), + count_failed_files_on_disk(&dir), + "the cached count matches the directory truth" + ); + + let _ = store.delete(); + } } diff --git a/crates/targets/src/target/amqp.rs b/crates/targets/src/target/amqp.rs index 2a3d096b2..1af39301a 100644 --- a/crates/targets/src/target/amqp.rs +++ b/crates/targets/src/target/amqp.rs @@ -657,8 +657,11 @@ where } fn delivery_snapshot(&self) -> TargetDeliverySnapshot { - self.delivery_counters - .snapshot(self.store.as_deref().map_or(0, |store| store.len() as u64)) + self.delivery_counters.snapshot( + self.store.as_deref().map_or(0, |store| store.len() as u64), + // AMQP targets record no terminal failures and keep no failed store. + 0, + ) } fn record_final_failure(&self) { diff --git a/crates/targets/src/target/kafka.rs b/crates/targets/src/target/kafka.rs index 928bce7cd..9b087f349 100644 --- a/crates/targets/src/target/kafka.rs +++ b/crates/targets/src/target/kafka.rs @@ -503,8 +503,11 @@ where } fn delivery_snapshot(&self) -> TargetDeliverySnapshot { - self.delivery_counters - .snapshot(self.store.as_deref().map_or(0, |store| store.len() as u64)) + self.delivery_counters.snapshot( + self.store.as_deref().map_or(0, |store| store.len() as u64), + // Kafka targets record no terminal failures and keep no failed store. + 0, + ) } fn record_final_failure(&self) { diff --git a/crates/targets/src/target/mod.rs b/crates/targets/src/target/mod.rs index e34bd7125..d3635b666 100644 --- a/crates/targets/src/target/mod.rs +++ b/crates/targets/src/target/mod.rs @@ -14,7 +14,7 @@ use crate::arn::TargetID; use crate::plugin::PluginEvent; -use crate::store::{Key, QueueStore, Store}; +use crate::store::{FailedEventStore, Key, QueueStore, Store}; use crate::{StoreError, TargetError, TargetLog}; use async_trait::async_trait; use rustfs_s3_types::EventName; @@ -58,6 +58,7 @@ pub(crate) fn redacted_optional_secret(value: Option<&str>) -> &'static str { #[derive(Debug, Clone, Default, PartialEq, Eq)] pub struct TargetDeliverySnapshot { pub failed_messages: u64, + pub failed_store_length: u64, pub queue_length: u64, pub total_messages: u64, } @@ -83,9 +84,10 @@ impl TargetDeliveryCounters { } #[inline] - pub fn snapshot(&self, queue_length: u64) -> TargetDeliverySnapshot { + pub fn snapshot(&self, queue_length: u64, failed_store_length: u64) -> TargetDeliverySnapshot { TargetDeliverySnapshot { failed_messages: self.failed_messages.load(Ordering::Relaxed), + failed_store_length, queue_length, total_messages: self.total_messages.load(Ordering::Relaxed), } @@ -159,6 +161,25 @@ where /// Returns the store associated with the target (if any) fn store(&self) -> Option<&(dyn Store + Send + Sync)>; + /// Returns the failed-events store capability when the target records terminal failures. + /// + /// The default is no capability, so a target that never parks a terminal entry runs no failed-store + /// maintenance and reports zero failed-store depth. + fn failed_store(&self) -> Option<&dyn FailedEventStore> { + None + } + + /// Moves a terminally failed entry to the target's failed-events store, returning true when the entry was handled. A target without a terminal-failure store declines the move and the entry stays live. + async fn handle_terminal_failure( + &self, + _store: &(dyn Store + Send), + _key: &Key, + _error: &TargetError, + _retry_count: u32, + ) -> bool { + false + } + /// Returns the type of the target fn clone_dyn(&self) -> Box + Send + Sync>; @@ -174,6 +195,7 @@ where /// Returns a read-only delivery snapshot for metrics collection. fn delivery_snapshot(&self) -> TargetDeliverySnapshot { TargetDeliverySnapshot { + failed_store_length: self.failed_store().map_or(0, |failed_store| failed_store.failed_len() as u64), queue_length: self.store().map_or(0, |store| store.len() as u64), ..TargetDeliverySnapshot::default() } @@ -202,6 +224,55 @@ pub struct QueuedPayloadMeta { pub content_type: String, pub queued_at_unix_ms: u64, pub payload_len: usize, + /// Stable per-entry deduplication identifier sent as the NATS JetStream Nats-Msg-Id header. + /// Empty for every non-JetStream target and for entries queued before JetStream was enabled, so + /// it is skipped on serialization and absent stored entries decode to an empty value. This keeps + /// the stored bytes identical to entries written without the field. + #[serde(default, skip_serializing_if = "String::is_empty")] + pub dedup_id: String, + + /// Set only on an entry written to the failed-events store. None on every live-queue entry, so it + /// is skipped on serialization and a live entry decodes with no failure metadata, keeping live + /// stored bytes identical to entries written without the field. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub failure: Option, +} + +/// The error class recorded on a failed-store entry. Only a non-retryable publish error reaches the +/// failed store, so the class confirms a terminal cause for an operator. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +pub enum FailedErrorClass { + /// A non-retryable publish error. Replaying it without fixing the underlying configuration repeats + /// the failure. + Terminal, +} + +impl FailedErrorClass { + /// Stable lowercase tag used in structured logs and operator tooling. + pub fn as_str(&self) -> &'static str { + match self { + FailedErrorClass::Terminal => "terminal", + } + } +} + +/// Failure metadata added to a queued payload when it is moved to the failed-events store, carried +/// inside the existing QueuedPayload meta so the failed entry decodes through the same reader. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct FailedEntryMeta { + /// The failure class recorded for operator triage. A failed-store entry is always terminal. + pub error_class: FailedErrorClass, + /// An allowlisted, credential-free summary of the failure for operator diagnosis. Never the raw + /// error rendering. + pub error_detail: String, + /// The stored dedup identifier, recording the id the publish attempted. A failed record is never + /// republished. + pub nats_msg_id: String, + /// The instant the entry entered the failed store, a diagnostic field for operator triage. Expiry + /// uses the filesystem modification time, not this value. + pub failed_at_unix_ms: u64, + /// The replay attempt count recorded at the terminal failure. + pub retry_count: u32, } impl QueuedPayloadMeta { @@ -219,6 +290,8 @@ impl QueuedPayloadMeta { content_type: content_type.into(), queued_at_unix_ms: SystemTime::now().duration_since(UNIX_EPOCH).unwrap_or_default().as_millis() as u64, payload_len, + dedup_id: String::new(), + failure: None, } } @@ -420,14 +493,13 @@ fn fnv1a_hash(bytes: &[u8]) -> u64 { /// Maps a target-id component to a filesystem-safe queue directory name. /// -/// Path-unsafe characters are replaced with `_`. Because that replacement is -/// lossy, two distinct ids (e.g. `a/b` and `a_b`) could otherwise collapse to -/// the same directory and interleave their persisted events. To prevent that, -/// whenever any character had to be replaced we append a short hash of the -/// original component, guaranteeing distinct ids map to distinct directories. +/// Path-unsafe characters are replaced with an underscore. That replacement is lossy, so two +/// distinct ids (for example a/b and a_b) could otherwise collapse to the same directory and +/// interleave their persisted events. A short hash of the original component is appended whenever any +/// character was replaced, so distinct ids map to distinct directories. /// -/// Ids that are already path-safe are returned unchanged, preserving the -/// on-disk directory layout for existing deployments (no queue migration). +/// Ids that are already path-safe are returned unchanged, preserving the on-disk directory layout for +/// existing deployments. pub(crate) fn sanitize_queue_dir_component(component: &str) -> String { let mut sanitized = String::with_capacity(component.len()); let mut lossy = false; @@ -441,7 +513,7 @@ pub(crate) fn sanitize_queue_dir_component(component: &str) -> String { } if sanitized.is_empty() { - // An entirely non-safe id would otherwise all collapse to "_"; key it by + // An entirely non-safe id would otherwise collapse to a single underscore. Key it by // the original bytes so distinct ids stay distinct. return format!("_{:016x}", fnv1a_hash(component.as_bytes())); } @@ -529,10 +601,20 @@ pub(crate) fn open_target_queue_store( target_id: &TargetID, open_context: &str, ) -> Result, TargetError> { - fn boxed_queue_store(store: QueueStore) -> BoxedQueuedStore { - Box::new(store) - } + let store = open_target_queue_store_typed(queue_dir, queue_limit, target_type, target_type_label, target_id, open_context)?; + Ok(store.map(|store| Box::new(store) as BoxedQueuedStore)) +} +/// Opens the queue store and returns the concrete QueueStore, so a target that needs its typed +/// failed-store capability holds it directly rather than through the type-erased Store handle. +pub(crate) fn open_target_queue_store_typed( + queue_dir: &str, + queue_limit: u64, + target_type: TargetType, + target_type_label: &str, + target_id: &TargetID, + open_context: &str, +) -> Result>, TargetError> { if queue_dir.is_empty() { return Ok(None); } @@ -547,7 +629,7 @@ pub(crate) fn open_target_queue_store( .open() .map_err(|err| TargetError::Storage(format!("{open_context}: {err}")))?; - Ok(Some(boxed_queue_store(store))) + Ok(Some(store)) } pub(crate) fn persist_queued_payload_to_store( @@ -584,7 +666,7 @@ pub(crate) fn mark_target_disconnected_on_connectivity_error(connected: &AtomicB } pub(crate) fn delete_stored_payload( - store: &(dyn Store + Send + Sync), + store: &(dyn Store + Send), key: &Key, ) -> Result<(), TargetError> { match store.del(key) { @@ -593,6 +675,96 @@ pub(crate) fn delete_stored_payload( } } +/// Upper bound on the characters retained from a classified error so a failed-store entry and its +/// alarm carry a diagnosable summary without an unbounded message. +const FAILED_ERROR_DETAIL_MAX_LEN: usize = 256; + +/// Fallback label substituted for a JetStreamPublish detail that falls outside the fixed vocabulary. +const UNRECOGNIZED_DETAIL_LABEL: &str = "unrecognized detail"; + +/// Restricts a JetStreamPublish detail to the fixed vocabulary at the persistence boundary. A detail +/// of lowercase letters, digits, space, underscore, and colon passes verbatim, any other content is +/// replaced with a fixed fallback label. +fn sanitize_failed_detail(detail: &str) -> &str { + let in_vocabulary = !detail.is_empty() + && detail.chars().all(|character| { + character.is_ascii_lowercase() || character.is_ascii_digit() || matches!(character, ' ' | '_' | ':') + }); + if in_vocabulary { detail } else { UNRECOGNIZED_DETAIL_LABEL } +} + +/// Builds a credential-free diagnostic string for a failed entry from a classified error. The +/// publish-error detail passes through the persistence-boundary sanitizer, a non-publish error +/// contributes only its variant category, and any embedded value is redacted. The result is +/// length-bounded at a character boundary. +pub(crate) fn build_failed_error_detail(error: &TargetError) -> String { + let summary = match error { + // The publish detail is sanitized to the fixed vocabulary at the persistence boundary. + TargetError::JetStreamPublish { detail, .. } => format!("jetstream_publish: {}", sanitize_failed_detail(detail)), + TargetError::Dropped(reason) => format!("dropped: {}", redacted_secret(reason)), + // Every other variant carries a free-form message that may name a host, path, or credential. + // Only the variant category is recorded, with any embedded value redacted. + TargetError::Network(value) => format!("network: {}", redacted_secret(value)), + TargetError::Request(value) => format!("request: {}", redacted_secret(value)), + TargetError::Timeout(value) => format!("timeout: {}", redacted_secret(value)), + TargetError::Storage(value) => format!("storage: {}", redacted_secret(value)), + TargetError::Authentication(_) => "authentication".to_string(), + TargetError::Configuration(_) => "configuration".to_string(), + other => format!("error: {}", redacted_secret(&other_error_category(other))), + }; + + let mut detail = summary; + truncate_to_char_boundary(&mut detail, FAILED_ERROR_DETAIL_MAX_LEN); + detail +} + +/// Truncates the string to at most max_len bytes, stepping the cut down to the nearest character +/// boundary so a multi-byte character straddling the cap is dropped whole rather than split. +fn truncate_to_char_boundary(value: &mut String, max_len: usize) { + if value.len() <= max_len { + return; + } + let mut cut = max_len; + while !value.is_char_boundary(cut) { + cut -= 1; + } + value.truncate(cut); +} + +/// Names the category of an otherwise free-form error without revealing its message. +fn other_error_category(error: &TargetError) -> String { + match error { + TargetError::Encoding(_) => "encoding".to_string(), + TargetError::Serialization(_) => "serialization".to_string(), + TargetError::Initialization(_) => "initialization".to_string(), + TargetError::Unknown(_) => "unknown".to_string(), + _ => "other".to_string(), + } +} + +/// Encodes a queued payload as a failed-store entry, extending its meta with the failure fields. +/// +/// Reuses the QueuedPayload format so the failed entry decodes through the same reader, preserving the +/// routing meta. The error_detail is the credential-free summary. The nats_msg_id is the resolved +/// dedup id, so an operator sees the id the server saw even for a pre-enable entry. +pub(crate) fn encode_failed_entry( + mut queued: QueuedPayload, + error_class: FailedErrorClass, + error: &TargetError, + retry_count: u32, + resolved_dedup_id: &str, +) -> Result, TargetError> { + let failed_at_unix_ms = SystemTime::now().duration_since(UNIX_EPOCH).unwrap_or_default().as_millis() as u64; + queued.meta.failure = Some(FailedEntryMeta { + error_class, + error_detail: build_failed_error_detail(error), + nats_msg_id: resolved_dedup_id.to_string(), + failed_at_unix_ms, + retry_count, + }); + queued.encode() +} + /// Ensures a rustls crypto provider is installed before any TLS operation. /// /// Multiple target modules (MySQL, Redis, Postgres, MQTT) need this because @@ -607,6 +779,97 @@ pub(crate) fn ensure_rustls_provider_installed() { } } +#[cfg(test)] +pub(crate) mod test_support { + use super::{EntityTarget, QueuedPayload, QueuedPayloadMeta}; + use crate::arn::TargetID; + use crate::store::{FailedEventStore, Key, QueueStore, Store}; + use crate::{StoreError, Target, TargetError}; + use async_trait::async_trait; + use rustfs_s3_types::EventName; + use std::path::PathBuf; + use std::sync::Arc; + use std::sync::atomic::{AtomicU64, Ordering}; + use uuid::Uuid; + + /// A minimal target for failed-store move tests: every delivery method succeeds, the optional + /// store backs the store and failed-store accessors, and final failures land on a shared counter. + #[derive(Clone)] + pub(crate) struct MoveTestTarget { + pub(crate) id: TargetID, + pub(crate) store: Option>>, + pub(crate) failed: Arc, + } + + #[async_trait] + impl Target for MoveTestTarget { + fn id(&self) -> TargetID { + self.id.clone() + } + async fn is_active(&self) -> Result { + Ok(true) + } + async fn save(&self, _event: Arc>) -> Result<(), TargetError> { + Ok(()) + } + async fn send_raw_from_store(&self, _key: Key, _body: Vec, _meta: QueuedPayloadMeta) -> Result<(), TargetError> { + Ok(()) + } + async fn close(&self) -> Result<(), TargetError> { + Ok(()) + } + fn store(&self) -> Option<&(dyn Store + Send + Sync)> { + self.store + .as_deref() + .map(|store| store as &(dyn Store<_, Error = StoreError, Key = Key> + Send + Sync)) + } + fn failed_store(&self) -> Option<&dyn FailedEventStore> { + self.store.as_deref().map(|store| store as &dyn FailedEventStore) + } + fn clone_dyn(&self) -> Box + Send + Sync> { + Box::new(self.clone()) + } + fn is_enabled(&self) -> bool { + true + } + fn record_final_failure(&self) { + self.failed.fetch_add(1, Ordering::Relaxed); + } + } + + pub(crate) fn move_test_target() -> Arc + Send + Sync> { + Arc::new(MoveTestTarget { + id: TargetID::new("target-a".to_string(), "nats".to_string()), + store: None, + failed: Arc::new(AtomicU64::new(0)), + }) + } + + pub(crate) fn move_test_target_with_store(store: Arc>) -> Arc + Send + Sync> { + Arc::new(MoveTestTarget { + id: TargetID::new("target-a".to_string(), "nats".to_string()), + store: Some(store), + failed: Arc::new(AtomicU64::new(0)), + }) + } + + pub(crate) fn failed_store_dir(name: &str) -> PathBuf { + std::env::temp_dir().join(format!("rustfs-failed-{name}-{}", Uuid::new_v4())) + } + + pub(crate) fn sample_queued(dedup_id: &str) -> QueuedPayload { + let mut meta = QueuedPayloadMeta::new( + EventName::ObjectCreatedPut, + "bucket-a".to_string(), + "obj.txt".to_string(), + "application/json", + 7, + ); + meta.dedup_id = dedup_id.to_string(); + QueuedPayload::new(meta, br#"{"x":1}"#.to_vec()) + } +} + #[cfg(test)] mod tls_state_tests { use super::{TargetTlsFingerprintState, TargetTlsGeneration, TargetTlsState}; @@ -743,6 +1006,45 @@ mod tests { assert_eq!(ChannelTargetType::Amqp.to_string(), "amqp"); } + #[test] + fn queued_payload_meta_omits_empty_dedup_id_on_serialization() { + let meta = QueuedPayloadMeta::new( + EventName::ObjectCreatedPut, + "bucket-a".to_string(), + "obj.txt".to_string(), + "application/json", + 7, + ); + assert!(meta.dedup_id.is_empty()); + + let json = serde_json::to_string(&meta).unwrap(); + assert!( + !json.contains("dedup_id"), + "an empty dedup id is skipped so stored bytes match the pre-feature format" + ); + + // An entry written without the field decodes to an empty dedup id. + let decoded: QueuedPayloadMeta = serde_json::from_str(&json).unwrap(); + assert!(decoded.dedup_id.is_empty()); + } + + #[test] + fn queued_payload_meta_round_trips_a_populated_dedup_id() { + let mut meta = QueuedPayloadMeta::new( + EventName::ObjectCreatedPut, + "bucket-a".to_string(), + "obj.txt".to_string(), + "application/json", + 7, + ); + meta.dedup_id = "minted-id".to_string(); + + let json = serde_json::to_string(&meta).unwrap(); + assert!(json.contains("dedup_id")); + let decoded: QueuedPayloadMeta = serde_json::from_str(&json).unwrap(); + assert_eq!(decoded.dedup_id, "minted-id"); + } + #[test] fn queued_payload_round_trips_meta_and_body() { let body = br#"{"ok":true}"#.to_vec(); @@ -1056,4 +1358,96 @@ mod tests { let err = QueuedPayload::decode(&encoded).unwrap_err(); assert!(err.to_string().contains("body length mismatch"), "unexpected error: {err}"); } + + use super::test_support::sample_queued; + + // The error_detail recorded on a failed entry is built from an allowlist of error categories and + // the fixed publish-error vocabulary, never the raw error rendering, so a credential embedded in + // a free-form error message is absent from the detail. + #[test] + fn build_failed_error_detail_redacts_credential_bearing_messages() { + let secret = "nats://user:supersecret@broker:4222"; + let cases = [ + TargetError::Network(secret.to_string()), + TargetError::Request(secret.to_string()), + TargetError::Timeout(secret.to_string()), + TargetError::Storage(secret.to_string()), + TargetError::Authentication(secret.to_string()), + TargetError::Configuration(secret.to_string()), + TargetError::Dropped(secret.to_string()), + TargetError::Unknown(secret.to_string()), + ]; + for error in cases { + let detail = build_failed_error_detail(&error); + assert!(!detail.contains("supersecret"), "detail leaked a credential: {detail}"); + assert!(!detail.contains("broker:4222"), "detail leaked a connection string: {detail}"); + } + } + + // The publish-error classifier sets the detail from its fixed vocabulary of kind labels and + // numeric codes, so the failed-entry detail names the cause without leaking a server-side + // message. + #[test] + fn build_failed_error_detail_uses_the_classified_publish_kind() { + let error = TargetError::JetStreamPublish { + retryable: false, + detail: "max payload exceeded".to_string(), + }; + let detail = build_failed_error_detail(&error); + assert_eq!(detail, "jetstream_publish: max payload exceeded"); + } + + // The persistence-boundary sanitizer replaces a JetStreamPublish detail that carries anything + // outside the fixed vocabulary with the fallback label, while a vocabulary detail passes through + // verbatim. + #[test] + fn build_failed_error_detail_sanitizes_a_hostile_jetstream_detail() { + let hostile = TargetError::JetStreamPublish { + retryable: false, + detail: "Boom! nats://user:pass@host/DROP".to_string(), + }; + assert_eq!(build_failed_error_detail(&hostile), "jetstream_publish: unrecognized detail"); + + let vocabulary = TargetError::JetStreamPublish { + retryable: false, + detail: "wrong last sequence".to_string(), + }; + assert_eq!(build_failed_error_detail(&vocabulary), "jetstream_publish: wrong last sequence"); + } + + // A summary whose cap lands inside a multi-byte character truncates at the preceding character + // boundary instead of panicking on a split character. A three-byte character does not divide + // the 256-byte cap, so one character straddles the cap by construction. + #[test] + fn truncate_to_char_boundary_handles_multi_byte_characters_at_the_cap() { + let mut value = "\u{4e2d}".repeat(FAILED_ERROR_DETAIL_MAX_LEN); + assert!(!value.is_char_boundary(FAILED_ERROR_DETAIL_MAX_LEN), "a character straddles the cap"); + truncate_to_char_boundary(&mut value, FAILED_ERROR_DETAIL_MAX_LEN); + assert!(value.len() <= FAILED_ERROR_DETAIL_MAX_LEN, "the result stays within the cap"); + assert!(value.is_char_boundary(value.len()), "the result ends on a character boundary"); + } + + // A terminal move writes a failed entry carrying the full failure meta extension and preserves the + // original routing metadata and dedup id through the reused QueuedPayload format. + #[test] + fn encode_failed_entry_carries_full_failure_meta() { + let queued = sample_queued("minted-id"); + let error = TargetError::JetStreamPublish { + retryable: false, + detail: "wrong last sequence".to_string(), + }; + let encoded = encode_failed_entry(queued, FailedErrorClass::Terminal, &error, 0, "minted-id").unwrap(); + let decoded = QueuedPayload::decode(&encoded).unwrap(); + + let failure = decoded.meta.failure.expect("a failed entry carries failure meta"); + assert_eq!(failure.error_class, FailedErrorClass::Terminal); + assert_eq!(failure.error_detail, "jetstream_publish: wrong last sequence"); + assert_eq!(failure.nats_msg_id, "minted-id"); + assert_eq!(failure.retry_count, 0); + assert!(failure.failed_at_unix_ms > 0); + // The original routing metadata survives the move. + assert_eq!(decoded.meta.bucket_name, "bucket-a"); + assert_eq!(decoded.meta.object_name, "obj.txt"); + assert_eq!(decoded.body, br#"{"x":1}"#); + } } diff --git a/crates/targets/src/target/mqtt.rs b/crates/targets/src/target/mqtt.rs index 1aeb102fa..bc4bf26a0 100644 --- a/crates/targets/src/target/mqtt.rs +++ b/crates/targets/src/target/mqtt.rs @@ -1693,8 +1693,11 @@ where } fn delivery_snapshot(&self) -> TargetDeliverySnapshot { - self.delivery_counters - .snapshot(self.store.as_deref().map_or(0, |store| store.len() as u64)) + self.delivery_counters.snapshot( + self.store.as_deref().map_or(0, |store| store.len() as u64), + // MQTT targets record no terminal failures and keep no failed store. + 0, + ) } fn record_final_failure(&self) { diff --git a/crates/targets/src/target/mysql.rs b/crates/targets/src/target/mysql.rs index 3a1508088..b14e252e8 100644 --- a/crates/targets/src/target/mysql.rs +++ b/crates/targets/src/target/mysql.rs @@ -1082,8 +1082,11 @@ where } fn delivery_snapshot(&self) -> TargetDeliverySnapshot { - self.delivery_counters - .snapshot(self.store.as_deref().map_or(0, |store| store.len() as u64)) + self.delivery_counters.snapshot( + self.store.as_deref().map_or(0, |store| store.len() as u64), + // MySQL targets record no terminal failures and keep no failed store. + 0, + ) } fn record_final_failure(&self) { diff --git a/crates/targets/src/target/nats/jetstream.rs b/crates/targets/src/target/nats/jetstream.rs new file mode 100644 index 000000000..836f248e5 --- /dev/null +++ b/crates/targets/src/target/nats/jetstream.rs @@ -0,0 +1,1236 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use super::NATSTarget; +use super::publish_error::classify_publish_error; +use super::validation::{gate_publish_on_stream_validation, validate_jetstream_stream}; +use crate::StoreError; +use crate::arn::TargetID; +use crate::error::TargetError; +use crate::plugin::PluginEvent; +use crate::runtime::{REPLAY_MAX_RETRIES, inter_attempt_backoff_sum, replay_backoff_term}; +use crate::store::{FailedEventStore, Key, Store}; +use crate::target::{ + FailedErrorClass, QueuedPayload, build_failed_error_detail, delete_stored_payload, encode_failed_entry, + truncate_to_char_boundary, +}; +use async_nats::header::{self, HeaderMap}; +use async_nats::jetstream::{self, context::PublishErrorKind}; +use sha2::{Digest, Sha256}; +use std::fmt::Write as _; +use std::sync::Arc; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::time::{Duration, Instant}; +use tracing::{debug, error, warn}; + +/// A cached JetStream context paired with its stream-validation verdict. The verdict starts +/// unvalidated for every built context, so a context rebuilt after a TLS rotation re-validates +/// before its first publish. A separate flag records whether the first passing validation has been +/// logged, so revalidations after a verdict reset stay at debug. Both start fresh with a rebuilt +/// context, and clones share them through the Arc. +#[derive(Clone)] +pub(crate) struct CachedJetStreamContext { + pub(crate) context: jetstream::Context, + pub(crate) stream_validated: Arc, + pub(crate) validation_logged: Arc, +} + +impl CachedJetStreamContext { + pub(crate) fn new(context: jetstream::Context) -> Self { + Self { + context, + validation_logged: Arc::new(AtomicBool::new(false)), + stream_validated: Arc::new(AtomicBool::new(false)), + } + } +} + +impl NATSTarget +where + E: PluginEvent, +{ + /// Returns the cached JetStream context, building it from the connected client on first use. The + /// context owns one background acker and one MaxAckPending semaphore, with its timeout set to the + /// ack timeout. + pub(crate) async fn jetstream_context(&self) -> Result { + // get_or_connect runs before the cache read so a detected TLS rotation swaps in a fresh + // context bound to the new client rather than one bound to the invalidated client. + let client = self.get_or_connect().await?; + + { + let guard = self.jetstream_context.lock().await; + if let Some(cached) = guard.as_ref() { + return Ok(cached.clone()); + } + } + + let ack_timeout = self.ack_timeout(); + + // Build under the lock so concurrent first callers do not each spawn an acker. new and + // set_timeout are synchronous, so no await holds the lock. + let mut guard = self.jetstream_context.lock().await; + let cached = guard.get_or_insert_with(|| { + let mut context = jetstream::new(client); + context.set_timeout(ack_timeout); + CachedJetStreamContext::new(context) + }); + Ok(cached.clone()) + } + + /// Returns the cached JetStream context after its configured stream has passed validation on the + /// bound connection. The verdict is cached with the context, so the steady-state cost is one + /// atomic load and a rebuilt context re-validates before its first publish. A connection-phase + /// failure classifies as NotConnected and a validation failure as a retryable JetStream publish + /// error, so either leaves the entry queued. + pub(crate) async fn validated_jetstream_context(&self) -> Result { + let cached = self.jetstream_context().await.map_err(jetstream_connect_error)?; + if !cached.stream_validated.load(Ordering::Acquire) { + let verdict = + validate_jetstream_stream(&cached.context, &self.args, &self.id.to_string(), Some(&cached.validation_logged)) + .await; + gate_publish_on_stream_validation(verdict, &cached.stream_validated)?; + } + Ok(cached) + } + + /// Publishes the body with the Nats-Msg-Id dedup header and awaits the real PublishAck. The whole + /// attempt, including connection establishment, stream validation, and the ack await, is bounded + /// by a single deadline equal to the ack timeout. Dropping a timed-out attempt releases its + /// resources, the broker suite exercises the drop path. + pub(crate) async fn publish_jetstream(&self, body: Vec, dedup_id: &str) -> Result<(), TargetError> { + attempt_within_deadline(self.ack_timeout(), self.publish_jetstream_attempt(body, dedup_id)).await + } + + /// Runs one publish attempt with no deadline of its own: acquires the validated context, + /// publishes with the dedup header, and awaits the PublishAck. + async fn publish_jetstream_attempt(&self, body: Vec, dedup_id: &str) -> Result<(), TargetError> { + let cached = self.validated_jetstream_context().await?; + + let mut headers = HeaderMap::new(); + headers.insert(header::NATS_MESSAGE_ID, dedup_id); + + // A stream-not-found outcome on either await resets the verdict before classification, so + // the next attempt re-validates instead of publishing blind into a missing stream. + let started = Instant::now(); + let ack_future = cached + .context + .publish_with_headers(self.args.subject.clone(), headers, body.into()) + .await + .inspect_err(|err| reset_verdict_on_stream_not_found(err, &cached.stream_validated)) + .map_err(|err| classify_publish_error(&err))?; + + let ack = ack_future + .await + .inspect_err(|err| reset_verdict_on_stream_not_found(err, &cached.stream_validated)) + .map_err(|err| classify_publish_error(&err))?; + let latency_ms = started.elapsed().as_millis() as u64; + + self.finish_acknowledged_publish(&ack.stream, ack.sequence, ack.duplicate, latency_ms, dedup_id, &cached.stream_validated) + } + + /// Completes a publish from the returned acknowledgment: verifies the acknowledging stream + /// matches the configured name, records the delivery, and logs the outcome. A mismatch resets + /// the verdict and returns a retryable error, so the entry stays queued and the next attempt + /// re-validates. + fn finish_acknowledged_publish( + &self, + ack_stream: &str, + sequence: u64, + duplicate: bool, + ack_latency_ms: u64, + dedup_id: &str, + stream_validated: &AtomicBool, + ) -> Result<(), TargetError> { + let configured_stream = self.args.jetstream_stream_name.as_deref().unwrap_or_default(); + if ack_stream != configured_stream { + stream_validated.store(false, Ordering::Release); + // Server-controlled text, length-bounded before logging. + let mut acknowledged_stream = ack_stream.to_string(); + truncate_to_char_boundary(&mut acknowledged_stream, ACK_STREAM_LOG_MAX_LEN); + warn!( + target_id = %self.id, + expected_stream = %configured_stream, + acknowledged_stream = %acknowledged_stream, + dedup_id = %dedup_id, + "JetStream publish acknowledged by an unexpected stream, verdict reset for re-validation" + ); + return Err(TargetError::JetStreamPublish { + retryable: true, + detail: ACK_STREAM_MISMATCH_DETAIL.to_string(), + }); + } + + // A first-attempt duplicate would mean two events collided on one id, which the minted uuid + // prevents, so the flag is logged rather than treated as an error. + debug!( + target_id = %self.id, + stream = %ack_stream, + sequence, + duplicate, + ack_latency_ms, + "JetStream publish acknowledged" + ); + + self.delivery_counters.record_success(); + Ok(()) + } +} + +/// Resolves the dedup id for a stored entry: the minted id when present, otherwise a hash of the +/// entry's on-disk key. Deriving from the key rather than the body keeps two entries with identical +/// bodies on separate ids, so the server does not drop the second as a duplicate. +pub(crate) fn resolve_dedup_id(meta_dedup_id: &str, key: &Key) -> String { + if !meta_dedup_id.is_empty() { + return meta_dedup_id.to_string(); + } + let digest = Sha256::digest(key.to_key_string().as_bytes()); + let mut id = String::with_capacity(digest.len() * 2); + for byte in digest { + // Writing a formatted byte into a String cannot fail. + let _ = write!(id, "{byte:02x}"); + } + id +} + +/// Moves a live queue entry that failed terminally to the failed-events store, then removes it from +/// the live queue. The failed entry is written first and the live entry deleted only after, so a +/// crash between the two leaves a recoverable duplicate rather than a loss. +pub(crate) async fn move_entry_to_failed_store( + store: &(dyn Store + Send), + failed_store: &dyn FailedEventStore, + target_id: &TargetID, + key: &Key, + error: &TargetError, + retry_count: u32, +) -> Result<(), TargetError> { + let raw = match store.get_raw(key) { + Ok(raw) => raw, + // A definitive ack from a concurrent path already cleared the entry. Nothing to move. + Err(StoreError::NotFound) => return Ok(()), + Err(err) => { + return Err(TargetError::Storage(format!( + "Failed to read queued payload before failed-store move: {err}" + ))); + } + }; + + let queued = QueuedPayload::decode(&raw) + .map_err(|err| TargetError::Storage(format!("Failed to decode queued payload before failed-store move: {err}")))?; + let meta = queued.meta.clone(); + // Same resolution the publish path applies, so the recorded id matches what the server saw even + // for a pre-enable entry with an empty stored id. + let resolved_dedup_id = resolve_dedup_id(&meta.dedup_id, key); + let encoded = encode_failed_entry(queued, FailedErrorClass::Terminal, error, retry_count, &resolved_dedup_id)?; + + failed_store + .put_failed_raw(&key.name, &encoded) + .map_err(|err| TargetError::Storage(format!("Failed to write failed-store entry: {err}")))?; + + // Error level: a terminal failure means a lost notification an operator must act on. Names + // routing metadata and the redacted detail only, never the event body or a credential. + error!( + target_id = %target_id, + error_class = FailedErrorClass::Terminal.as_str(), + error_detail = %build_failed_error_detail(error), + bucket = %meta.bucket_name, + object = %meta.object_name, + event_name = %meta.event_name.as_str(), + nats_msg_id = %resolved_dedup_id, + retry_count, + "event moved to the failed-events store" + ); + + delete_stored_payload(store, key)?; + // The failure count is recorded by the replay hook, not here, so it counts once. + Ok(()) +} + +/// Drains a JetStream context and drops it so its background acker exits. async-nats keeps the acker +/// task alive with no Drop on Context, so wait_for_acks drains every in-flight publish before the +/// final drop closes the ack channel and ends the task. None is a no-op. +pub(crate) async fn drain_jetstream_context(context: Option) { + if let Some(context) = context { + context.wait_for_acks().await; + } +} + +/// Maps a connectivity or configuration failure from establishing the JetStream connection to +/// NotConnected, so the replay worker retries it and leaves the entry queued rather than parking it +/// in the failed store. Errors already classified at the publish level pass through unchanged. +fn jetstream_connect_error(err: TargetError) -> TargetError { + match err { + TargetError::Network(_) | TargetError::Configuration(_) => TargetError::NotConnected, + other => other, + } +} + +/// Detail carried by the retryable publish error raised when a publish attempt exceeds the ack +/// timeout. +pub(crate) const ATTEMPT_DEADLINE_DETAIL: &str = "attempt deadline exceeded"; + +/// Bounds one publish attempt by the ack timeout. An attempt that does not resolve within the +/// deadline is dropped and classified as a retryable JetStream publish error, so the replay loop +/// counts it toward the bounded budget. An inner client timeout can resolve first with its own +/// timed-out error, and both paths classify retryable. +async fn attempt_within_deadline(ack_timeout: Duration, attempt: F) -> Result<(), TargetError> +where + F: std::future::Future>, +{ + match tokio::time::timeout(ack_timeout, attempt).await { + Ok(result) => result, + Err(_elapsed) => Err(TargetError::JetStreamPublish { + retryable: true, + detail: ATTEMPT_DEADLINE_DETAIL.to_string(), + }), + } +} + +/// Detail carried by the retryable publish error raised when an acknowledgment names a stream other +/// than the configured one. +pub(crate) const ACK_STREAM_MISMATCH_DETAIL: &str = "acknowledged by an unexpected stream"; + +/// Byte bound applied to the acknowledged stream name before it is logged. +const ACK_STREAM_LOG_MAX_LEN: usize = 64; + +/// Detail carried by the retryable publish error raised when the server reports the configured +/// stream does not exist. +pub(crate) const STREAM_NOT_FOUND_DETAIL: &str = "stream not found"; + +/// Resets the validation verdict when a publish outcome reports the configured stream does not +/// exist, so the next attempt re-validates. Keys off the raw error kind before classification, +/// keeping the classifier pure. +fn reset_verdict_on_stream_not_found(err: &async_nats::jetstream::context::PublishError, stream_validated: &AtomicBool) { + if matches!(err.kind(), PublishErrorKind::StreamNotFound) { + stream_validated.store(false, Ordering::Release); + } +} + +/// Conservative upper bound on the wall-clock retry span of a stored entry, used as the minimum +/// acceptable stream duplicate_window. Sums the per-attempt ack ceiling across REPLAY_MAX_RETRIES +/// with the realized backoff schedule plus a headroom term. The headroom is a deliberate +/// conservatism margin that holds the required duplicate_window above the realized retry span. A +/// duplicate_window at or above this keeps a late same-id retry inside the dedup window. Saturating +/// arithmetic keeps it panic-free. +pub(crate) fn retry_lifetime(ack_timeout: Duration) -> Duration { + let ack_span = ack_timeout.saturating_mul(REPLAY_MAX_RETRIES as u32); + let realized_backoff = inter_attempt_backoff_sum(REPLAY_MAX_RETRIES); + let headroom = replay_backoff_term(REPLAY_MAX_RETRIES as u32); + ack_span.saturating_add(realized_backoff).saturating_add(headroom) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::Target; + use crate::arn::TargetID; + use crate::store::{FailedEventStore, QueueStore}; + use crate::target::TargetType; + use crate::target::nats::test_support::*; + use crate::target::nats::validation::STREAM_VALIDATION_FAILED_DETAIL; + use crate::target::test_support::{ + MoveTestTarget, failed_store_dir, move_test_target, move_test_target_with_store, sample_queued, + }; + use crate::target::{build_target_tls_fingerprint, persist_queued_payload_to_store}; + use async_nats::jetstream::context::PublishError; + use rustfs_config::NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS; + use std::sync::atomic::AtomicU64; + use uuid::Uuid; + + #[test] + fn enabled_target_mints_dedup_id_stored_in_meta() { + let target = nats_target(jetstream_args(TargetType::NotifyEvent)); + let queued = target.build_queued_payload(&sample_event()).expect("payload builds"); + assert!(!queued.meta.dedup_id.is_empty(), "an enabled target mints a dedup id"); + // The minted id is a uuid string, distinct from any event field. + assert!(Uuid::parse_str(&queued.meta.dedup_id).is_ok(), "the dedup id is a uuid"); + } + + #[test] + fn disabled_target_mints_no_dedup_id() { + let mut args = jetstream_args(TargetType::NotifyEvent); + args.jetstream_enable = Some(false); + let target = nats_target(args); + let queued = target.build_queued_payload(&sample_event()).expect("payload builds"); + assert!(queued.meta.dedup_id.is_empty(), "a disabled target mints no dedup id"); + } + + #[test] + fn dedup_id_is_stable_across_reads_of_the_same_stored_entry() { + for target_type in [TargetType::NotifyEvent, TargetType::AuditLog] { + let target = nats_target(jetstream_args(target_type)); + let queued = target.build_queued_payload(&sample_event()).expect("payload builds"); + + let encoded = queued.encode().expect("payload encodes"); + let first = QueuedPayload::decode(&encoded).expect("payload decodes"); + let second = QueuedPayload::decode(&encoded).expect("payload decodes"); + + assert_eq!( + first.meta.dedup_id, second.meta.dedup_id, + "reading the same stored entry twice yields the same id for {target_type}" + ); + assert_eq!( + first.meta.dedup_id, queued.meta.dedup_id, + "the stored id is the minted id for {target_type}" + ); + } + } + + #[test] + fn dedup_id_is_distinct_across_entries() { + for target_type in [TargetType::NotifyEvent, TargetType::AuditLog] { + let target = nats_target(jetstream_args(target_type)); + let first = target.build_queued_payload(&sample_event()).expect("payload builds"); + let second = target.build_queued_payload(&sample_event()).expect("payload builds"); + assert_ne!( + first.meta.dedup_id, second.meta.dedup_id, + "two distinct entries get distinct ids for {target_type}" + ); + } + } + + #[test] + fn resolve_dedup_id_prefers_the_stored_minted_id() { + let id = resolve_dedup_id("minted-uuid", &sample_stored_key("entry-one")); + assert_eq!(id, "minted-uuid"); + } + + #[test] + fn resolve_dedup_id_derives_distinct_ids_for_entries_with_equal_bodies() { + // Distinct on-disk keys yield distinct fallback ids even for identical bodies. + let first = resolve_dedup_id("", &sample_stored_key("entry-one")); + let second = resolve_dedup_id("", &sample_stored_key("entry-two")); + assert_ne!(first, second, "distinct entries derive distinct ids"); + assert_eq!(first.len(), 64, "the derived id is a sha256 hex digest"); + } + + #[test] + fn resolve_dedup_id_is_stable_across_replays_of_one_entry() { + // The same entry keeps its key across replays, so the derived id is identical each time. + let key = sample_stored_key("entry-one"); + assert_eq!(resolve_dedup_id("", &key), resolve_dedup_id("", &key)); + } + + #[test] + fn jetstream_connect_error_maps_network_to_not_connected() { + // A connect-phase connectivity failure classifies as NotConnected so the entry stays queued. + let mapped = jetstream_connect_error(TargetError::Network("Failed to connect to NATS server: refused".to_string())); + assert!( + matches!(mapped, TargetError::NotConnected), + "a connect-phase connectivity failure classifies as NotConnected, got {mapped:?}" + ); + } + + #[test] + fn jetstream_connect_error_maps_configuration_to_not_connected() { + // A connect-phase configuration failure (unreadable TLS material after start) classifies as NotConnected. + let mapped = jetstream_connect_error(TargetError::Configuration("Failed to read TLS material".to_string())); + assert!( + matches!(mapped, TargetError::NotConnected), + "a connect-phase configuration failure classifies as NotConnected, got {mapped:?}" + ); + } + + #[test] + fn jetstream_connect_error_passes_through_classified_errors() { + // An already-classified publish error passes through, so the mapping cannot mask a terminal cause. + let terminal = jetstream_connect_error(TargetError::JetStreamPublish { + retryable: false, + detail: "stream sealed".to_string(), + }); + match terminal { + TargetError::JetStreamPublish { retryable, .. } => assert!(!retryable, "a terminal publish error stays terminal"), + other => panic!("expected JetStreamPublish, got {other:?}"), + } + } + + #[tokio::test] + async fn publish_jetstream_maps_a_connect_failure_to_not_connected() { + // A connect-level failure on the publish path surfaces as NotConnected. Port 1 refuses without a server. + let queue_dir = tempfile::tempdir().expect("queue dir"); + let mut args = jetstream_args(TargetType::NotifyEvent); + args.address = "nats://127.0.0.1:1".to_string(); + args.queue_dir = queue_dir.path().to_str().unwrap().to_string(); + let target = nats_target(args); + + let err = target + .publish_jetstream(b"body".to_vec(), "dedup-connect-fail") + .await + .expect_err("an unreachable broker fails the publish"); + assert!( + matches!(err, TargetError::NotConnected), + "a connect-level failure on the publish path surfaces as NotConnected, got {err:?}" + ); + } + + #[test] + fn a_wrong_stream_ack_resets_the_verdict_and_classifies_retryable() { + // A wrong-stream ack resets the verdict and returns the retryable mismatch error. A matching ack completes the publish and keeps the verdict. + let target = nats_target(jetstream_args(TargetType::NotifyEvent)); + let stream_validated = AtomicBool::new(true); + let err = target + .finish_acknowledged_publish("SOME_OTHER_STREAM", 7, false, 12, "dedup-mismatch", &stream_validated) + .expect_err("an unexpected acknowledging stream is rejected"); + match err { + TargetError::JetStreamPublish { retryable, detail } => { + assert!(retryable, "a wrong-stream ack is retryable so the entry stays queued"); + assert_eq!(detail, ACK_STREAM_MISMATCH_DETAIL, "the fixed mismatch label is the detail"); + } + other => panic!("expected a retryable JetStreamPublish, got {other:?}"), + } + assert!( + !stream_validated.load(Ordering::SeqCst), + "the verdict resets so the next attempt re-validates the stream" + ); + + // A matching acknowledging stream completes the publish and keeps the recorded verdict. + let validated = AtomicBool::new(true); + target + .finish_acknowledged_publish("RUSTFS_EVENTS", 8, false, 12, "dedup-match", &validated) + .expect("a matching stream ack completes the publish"); + assert!(validated.load(Ordering::SeqCst), "a matching ack keeps the recorded verdict"); + } + + #[test] + fn stream_not_found_publish_outcome_resets_the_verdict() { + // A StreamNotFound outcome resets the verdict. Any other kind leaves it untouched. + let stream_validated = AtomicBool::new(true); + reset_verdict_on_stream_not_found(&PublishError::new(PublishErrorKind::StreamNotFound), &stream_validated); + assert!(!stream_validated.load(Ordering::SeqCst), "a stream-not-found outcome resets the verdict"); + + for kind in [PublishErrorKind::TimedOut, PublishErrorKind::MaxPayloadExceeded] { + let verdict = AtomicBool::new(true); + reset_verdict_on_stream_not_found(&PublishError::new(kind), &verdict); + assert!(verdict.load(Ordering::SeqCst), "{kind} leaves the verdict intact"); + } + } + + #[tokio::test] + async fn a_failed_publish_attempt_leaves_the_queued_entry_intact() { + // A publish that fails before any send leaves the queued entry, since only a returned PublishAck clears it. + let queue_dir = tempfile::tempdir().expect("queue dir"); + let mut args = jetstream_args(TargetType::NotifyEvent); + args.address = "nats://127.0.0.1:1".to_string(); + let target = nats_target_with_store(args, queue_dir.path().to_str().unwrap()); + + let queued = target.build_queued_payload(&sample_event()).expect("payload builds"); + let store = target.store().expect("a store is configured"); + persist_queued_payload_to_store(store, &queued).expect("entry persists"); + let key = store.list().pop().expect("the entry is listed"); + + let err = target.send_from_store(key.clone()).await.expect_err("the attempt fails"); + assert!( + matches!(err, TargetError::NotConnected), + "the failed connect surfaces as a connectivity error, got {err:?}" + ); + assert_eq!(store.len(), 1, "the queued entry survives the failed attempt"); + assert!(store.get_raw(&key).is_ok(), "the surviving entry is still readable"); + } + + #[tokio::test(start_paused = true)] + async fn publish_path_consults_the_stream_gate_before_publishing() { + // The publish path runs stream validation first, so an unvalidated stream fails with the validation label rather than the publish call. + let client = async_nats::ConnectOptions::new() + .retry_on_initial_connect() + .connect("nats://127.0.0.1:1") + .await + .expect("retry_on_initial_connect returns without a live server"); + + let target = nats_target(jetstream_args(TargetType::NotifyEvent)); + let fingerprint = build_target_tls_fingerprint("", "", "").await.expect("fingerprint builds"); + target.tls_state.lock().refresh(fingerprint); + *target.client.lock().await = Some(client.clone()); + + let mut context = jetstream::new(client); + // Shorter than the 30s attempt deadline, so the validation lookup elapses first. + context.set_timeout(Duration::from_secs(5)); + let cached = CachedJetStreamContext::new(context); + let verdict = Arc::clone(&cached.stream_validated); + *target.jetstream_context.lock().await = Some(cached); + + let err = target + .publish_jetstream(b"body".to_vec(), "dedup-gate-wiring") + .await + .expect_err("the unvalidated stream blocks the publish"); + match err { + TargetError::JetStreamPublish { retryable, detail } => { + assert!(retryable, "the gate failure is retryable"); + assert_eq!( + detail, STREAM_VALIDATION_FAILED_DETAIL, + "the failure names the stream validation, not the publish" + ); + } + other => panic!("expected a retryable JetStreamPublish, got {other:?}"), + } + assert!(!verdict.load(Ordering::SeqCst), "the verdict stays closed after the failed validation"); + } + + #[tokio::test(start_paused = true)] + async fn is_active_reports_a_failing_stream_instead_of_healthy() { + // is_active consults the stream-validation gate, so a reachable broker with a failing stream reports the error, not healthy. + let client = async_nats::ConnectOptions::new() + .retry_on_initial_connect() + .connect("nats://127.0.0.1:1") + .await + .expect("retry_on_initial_connect returns without a live server"); + + let target = nats_target(jetstream_args(TargetType::NotifyEvent)); + let fingerprint = build_target_tls_fingerprint("", "", "").await.expect("fingerprint builds"); + target.tls_state.lock().refresh(fingerprint); + *target.client.lock().await = Some(client.clone()); + + let mut context = jetstream::new(client); + context.set_timeout(Duration::from_secs(5)); + let cached = CachedJetStreamContext::new(context); + let verdict = Arc::clone(&cached.stream_validated); + *target.jetstream_context.lock().await = Some(cached); + + let err = target.is_active().await.expect_err("a failing stream fails the health check"); + match err { + TargetError::JetStreamPublish { retryable, detail } => { + assert!(retryable, "the health failure carries the gate classification"); + assert_eq!( + detail, STREAM_VALIDATION_FAILED_DETAIL, + "the health check reports the stream validation failure" + ); + } + other => panic!("expected the gate classification, got {other:?}"), + } + + // A reset verdict forces the next check to re-run validation. + verdict.store(true, Ordering::SeqCst); + verdict.store(false, Ordering::SeqCst); + let err = target + .is_active() + .await + .expect_err("a freshly reset verdict re-runs validation"); + match err { + TargetError::JetStreamPublish { retryable, detail } => { + assert!(retryable, "the re-validation failure carries the gate classification"); + assert_eq!( + detail, STREAM_VALIDATION_FAILED_DETAIL, + "the reset verdict forces the health check to re-validate the stream" + ); + } + other => panic!("expected the gate classification after a reset, got {other:?}"), + } + } + + #[tokio::test(start_paused = true)] + async fn attempt_deadline_fires_against_a_pending_publish_and_classifies_retryable() { + // A publish that never resolves is cut off at the ack timeout and classified retryable. + let result = attempt_within_deadline(Duration::from_secs(30), std::future::pending::>()).await; + match result { + Err(TargetError::JetStreamPublish { retryable, detail }) => { + assert!(retryable, "an elapsed attempt deadline is retryable"); + assert_eq!(detail, ATTEMPT_DEADLINE_DETAIL); + } + other => panic!("expected a retryable JetStreamPublish, got {other:?}"), + } + } + + #[tokio::test(start_paused = true)] + async fn attempt_deadline_passes_through_a_result_within_the_deadline() { + attempt_within_deadline(Duration::from_secs(30), async { Ok(()) }) + .await + .expect("a resolved attempt passes its result through"); + + // An inner client timeout can resolve first and keeps its own retryable classification. + let inner = TargetError::JetStreamPublish { + retryable: true, + detail: "timed out".to_string(), + }; + let err = attempt_within_deadline(Duration::from_secs(30), async { Err(inner) }) + .await + .expect_err("an inner error passes through"); + match err { + TargetError::JetStreamPublish { retryable, detail } => { + assert!(retryable, "the inner timed-out classification is retryable"); + assert_eq!(detail, "timed out", "the inner detail is preserved"); + } + other => panic!("expected a retryable JetStreamPublish, got {other:?}"), + } + } + + #[test] + fn ack_timeout_uses_the_configured_value() { + let target = nats_target(jetstream_args(TargetType::NotifyEvent)); + assert_eq!(target.ack_timeout(), Duration::from_secs(30)); + + let mut args = jetstream_args(TargetType::NotifyEvent); + args.jetstream_ack_timeout_secs = None; + let target = nats_target(args); + assert_eq!( + target.ack_timeout(), + Duration::from_secs(NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS), + "an absent ack timeout uses the default" + ); + } + + #[test] + fn retry_lifetime_covers_every_attempt_ack_wait_plus_full_backoff() { + // The realized backoff schedule at the default base is 4 + 8 + 16 + 32 = 60s, plus a 64s headroom term = 124s. At the 30s ack timeout the worst case is 5 * 30 + 124 = 274s, the floor a duplicate_window must clear. + assert_eq!(retry_lifetime(Duration::from_secs(30)), Duration::from_secs(274)); + // With a zero ack timeout only the backoff schedule and the headroom term remain. + assert_eq!(retry_lifetime(Duration::from_secs(0)), Duration::from_secs(124)); + } + + #[test] + fn retry_lifetime_backoff_equals_the_realized_schedule_plus_headroom() { + // The duplicate-window backoff span must equal the worker's realized sleep schedule plus one final-shift headroom term. + let mut realized = Duration::ZERO; + for shift in 1..REPLAY_MAX_RETRIES as u32 { + realized += replay_backoff_term(shift); + } + assert_eq!( + realized, + inter_attempt_backoff_sum(REPLAY_MAX_RETRIES), + "the shared sum matches the worker sleep schedule" + ); + let headroom = replay_backoff_term(REPLAY_MAX_RETRIES as u32); + assert_eq!(retry_lifetime(Duration::ZERO), realized + headroom); + assert_eq!(retry_lifetime(Duration::ZERO), Duration::from_secs(124)); + } + + #[test] + fn clone_box_shares_one_cached_context_handle() { + // One acker and one backpressure semaphore serve every clone of the target. + let target = nats_target(jetstream_args(TargetType::NotifyEvent)); + let before = Arc::strong_count(&target.jetstream_context); + let _clone = target.clone_box(); + assert_eq!( + Arc::strong_count(&target.jetstream_context), + before + 1, + "clone_box shares the one context handle rather than building a new one" + ); + } + + #[tokio::test] + async fn close_keeps_a_queued_entry_for_replay() { + // Close releases cached handles but never durable entries, so a queued entry survives and replays. + let queue_dir = tempfile::tempdir().expect("queue dir"); + let target = nats_target_with_store(jetstream_args(TargetType::NotifyEvent), queue_dir.path().to_str().unwrap()); + let queued = target.build_queued_payload(&sample_event()).expect("payload builds"); + let store = target.store().expect("a store is configured"); + persist_queued_payload_to_store(store, &queued).expect("entry persists"); + assert_eq!(store.len(), 1, "the entry is queued before close"); + + target.close().await.expect("close succeeds"); + + let store_after = target.store().expect("the store handle is still held"); + assert_eq!(store_after.len(), 1, "the queued entry survives close"); + assert_eq!(store_after.list().len(), 1, "the surviving entry is listable for replay"); + } + + #[tokio::test] + async fn flag_off_close_matches_the_baseline() { + // With JetStream off no context is cached and close drains only the client. A queued entry still survives. + let queue_dir = tempfile::tempdir().expect("queue dir"); + let target = nats_target_with_store(base_args(), queue_dir.path().to_str().unwrap()); + assert!(!target.jetstream_enabled(), "the flag is off"); + assert!(target.jetstream_context.lock().await.is_none(), "no context is cached with the flag off"); + + let queued = target.build_queued_payload(&sample_event()).expect("payload builds"); + let store = target.store().expect("a store is configured"); + persist_queued_payload_to_store(store, &queued).expect("entry persists"); + + target.close().await.expect("flag-off close succeeds"); + + assert!( + target.jetstream_context.lock().await.is_none(), + "the flag-off close leaves the context field empty" + ); + assert_eq!(target.store().expect("store held").len(), 1, "the queued entry survives a flag-off close"); + } + + #[tokio::test] + async fn drain_jetstream_context_handles_none() { + // A None context (nothing cached) is a no-op. + drain_jetstream_context(None).await; + } + + #[tokio::test] + #[ignore] + async fn tls_change_rebuilds_the_context_and_drains_the_old_acker() { + // A TLS fingerprint change on the publish path rebuilds the cached context from the new client and drains the old acker. + let subject = format!("rustfs.tlsrebuild.{}", Uuid::new_v4().simple()); + let stream_name = format!("RUSTFS_TLSREBUILD_{}", Uuid::new_v4().simple()); + + let provision_client = async_nats::connect(broker_url()).await.expect("connect to provision"); + let provision_context = jetstream::new(provision_client); + provision_context + .create_stream(jetstream::stream::Config { + name: stream_name.clone(), + subjects: vec![subject.clone()], + duplicate_window: Duration::from_secs(300), + ..Default::default() + }) + .await + .expect("provision the stream"); + + let queue_dir = tempfile::tempdir().expect("queue dir"); + let mut args = jetstream_args(TargetType::NotifyEvent); + args.address = broker_url(); + args.subject = subject.clone(); + args.jetstream_stream_name = Some(stream_name.clone()); + args.queue_dir = queue_dir.path().to_str().unwrap().to_string(); + args.tls_required = false; + + let target = nats_target(args); + + // First publish builds and caches context A bound to the first connection. + target + .publish_jetstream(b"first".to_vec(), "dedup-first") + .await + .expect("publish A"); + let context_a = target.jetstream_context().await.expect("context A is cached").context; + // Read the client_id from the cache without get_or_connect, so the read never triggers a reconnect. + let cached_client_id = || async { + target + .client + .lock() + .await + .as_ref() + .expect("a client is cached") + .server_info() + .client_id + }; + let client_id_a = cached_client_id().await; + + // Reset the fingerprint so the next publish detects a change, as a rotated cert would. + target.tls_state.lock().reset(); + + // The next publish detects the change, reconnects, rebuilds the context, and drains context A. + target + .publish_jetstream(b"second".to_vec(), "dedup-second") + .await + .expect("publish B on the rebuilt context"); + let client_id_b = cached_client_id().await; + + // The cached client id changed, so only the publish path could have reconnected onto a new connection. + assert_ne!( + client_id_a, client_id_b, + "the production publish path rebuilt onto a fresh connection after the material change" + ); + + // A further publish on the rebuilt context acks. + target + .publish_jetstream(b"third".to_vec(), "dedup-third") + .await + .expect("third publish on the rebuilt context acks"); + + // Context A was drained in the swap, so a fresh wait returns promptly. + tokio::time::timeout(Duration::from_secs(2), context_a.wait_for_acks()) + .await + .expect("the old context is fully drained, its acker has nothing parked"); + + // Best-effort teardown. + let _ = provision_context.delete_stream(&stream_name).await; + } + + #[tokio::test] + #[ignore] + async fn tls_change_after_a_failed_reconnect_still_rebuilds_the_context() { + // A rotation detected while the broker is unreachable does not orphan the cached context: a failed reconnect followed by a successful one ends bound to the rebuilt context. + let subject = format!("rustfs.tlsfail.{}", Uuid::new_v4().simple()); + let stream_name = format!("RUSTFS_TLSFAIL_{}", Uuid::new_v4().simple()); + + let provision_client = async_nats::connect(broker_url()).await.expect("connect to provision"); + let provision_context = jetstream::new(provision_client); + provision_context + .create_stream(jetstream::stream::Config { + name: stream_name.clone(), + subjects: vec![subject.clone()], + duplicate_window: Duration::from_secs(300), + ..Default::default() + }) + .await + .expect("provision the stream"); + + let queue_dir = tempfile::tempdir().expect("queue dir"); + let mut args = jetstream_args(TargetType::NotifyEvent); + args.address = broker_url(); + args.subject = subject.clone(); + args.jetstream_stream_name = Some(stream_name.clone()); + args.queue_dir = queue_dir.path().to_str().unwrap().to_string(); + args.tls_required = false; + + let mut target = nats_target(args); + + // First production publish builds and caches context A bound to the first connection. + target + .publish_jetstream(b"first".to_vec(), "dedup-first") + .await + .expect("publish A"); + let context_a = target.jetstream_context().await.expect("context A is cached").context; + let client_id_a = context_a.client().server_info().client_id; + + // Signal a rotation, then point at a dead port so the first reconnect fails. + target.tls_state.lock().reset(); + let good_address = target.args.address.clone(); + target.args.address = "nats://127.0.0.1:1".to_string(); + + let failed = target.publish_jetstream(b"second".to_vec(), "dedup-second").await; + assert!( + failed.is_err(), + "the first reconnect after the change fails while the broker is unreachable" + ); + + // The broker returns. A later publish reconnects, rebuilds the context, and drains the old one. + target.args.address = good_address; + target + .publish_jetstream(b"third".to_vec(), "dedup-third") + .await + .expect("publish on the rebuilt context"); + + // The cached context is bound to a fresh connection, not the pre-rotation one. + let cached_context = target + .jetstream_context + .lock() + .await + .as_ref() + .expect("a context is cached") + .clone() + .context; + let cached_context_client_id = cached_context.client().server_info().client_id; + assert_ne!( + cached_context_client_id, client_id_a, + "the cached context was rebuilt onto a fresh connection after the failed-then-successful reconnect" + ); + + // Context A was drained in the swap, so a fresh wait returns promptly. + tokio::time::timeout(Duration::from_secs(2), context_a.wait_for_acks()) + .await + .expect("the pre-rotation context is drained, its acker has nothing parked"); + + let _ = provision_context.delete_stream(&stream_name).await; + } + + #[tokio::test] + #[ignore] + async fn publish_gate_rejects_an_unsafe_stream_and_heals_after_the_stream_is_fixed() { + // The gate rejects every publish while the stream's duplicate window is below the retry lifetime, and starts publishing once the operator widens it, without a restart. + let subject = format!("rustfs.gate.{}", Uuid::new_v4().simple()); + let stream_name = format!("RUSTFS_GATE_{}", Uuid::new_v4().simple()); + + let provision_client = async_nats::connect(broker_url()).await.expect("connect to provision"); + let provision_context = jetstream::new(provision_client); + provision_context + .create_stream(jetstream::stream::Config { + name: stream_name.clone(), + subjects: vec![subject.clone()], + // Below the 274s required at the default 30s ack timeout. + duplicate_window: Duration::from_secs(60), + ..Default::default() + }) + .await + .expect("provision the unsafe stream"); + + let queue_dir = tempfile::tempdir().expect("queue dir"); + let mut args = jetstream_args(TargetType::NotifyEvent); + args.address = broker_url(); + args.subject = subject.clone(); + args.jetstream_stream_name = Some(stream_name.clone()); + args.queue_dir = queue_dir.path().to_str().unwrap().to_string(); + args.tls_required = false; + + let target = nats_target(args); + + let err = target + .publish_jetstream(b"first".to_vec(), "dedup-gate-first") + .await + .expect_err("no publish proceeds against a stream that fails validation"); + assert!( + matches!(err, TargetError::JetStreamPublish { retryable: true, .. }), + "the gate failure is retryable, got {err:?}" + ); + + // The operator widens the window past the retry lifetime, and the next attempt re-validates and publishes. + provision_context + .update_stream(jetstream::stream::Config { + name: stream_name.clone(), + subjects: vec![subject.clone()], + duplicate_window: Duration::from_secs(300), + ..Default::default() + }) + .await + .expect("widen the duplicate window"); + + target + .publish_jetstream(b"second".to_vec(), "dedup-gate-second") + .await + .expect("the repaired stream accepts the publish without a restart"); + + let _ = provision_context.delete_stream(&stream_name).await; + } + + #[tokio::test] + #[ignore] + async fn a_remapped_subject_is_rejected_by_the_ack_stream_check_and_the_entry_stays_queued() { + // After a subject remap the takeover stream acknowledges, so the ack-stream check rejects it with the mismatch detail, keeps the entry queued, and resets the verdict for re-validation. + let subject = format!("rustfs.remap.{}", Uuid::new_v4().simple()); + let configured_stream = format!("RUSTFS_REMAP_A_{}", Uuid::new_v4().simple()); + let takeover_stream = format!("RUSTFS_REMAP_B_{}", Uuid::new_v4().simple()); + let parked_subject = format!("rustfs.remapparked.{}", Uuid::new_v4().simple()); + + let provision_client = async_nats::connect(broker_url()).await.expect("connect to provision"); + let provision_context = jetstream::new(provision_client); + provision_context + .create_stream(jetstream::stream::Config { + name: configured_stream.clone(), + subjects: vec![subject.clone()], + duplicate_window: Duration::from_secs(300), + ..Default::default() + }) + .await + .expect("provision the configured stream"); + + let queue_dir = tempfile::tempdir().expect("queue dir"); + let mut args = jetstream_args(TargetType::NotifyEvent); + args.address = broker_url(); + args.subject = subject.clone(); + args.jetstream_stream_name = Some(configured_stream.clone()); + let target = nats_target_with_store(args, queue_dir.path().to_str().unwrap()); + + // The pre-remap publish validates the stream and acks from it, recording a passing verdict. + target + .publish_jetstream(b"pre-remap".to_vec(), "dedup-remap-first") + .await + .expect("the configured stream acknowledges before the remap"); + + // The remap: the configured stream stops capturing the subject and the takeover stream starts. + provision_context + .update_stream(jetstream::stream::Config { + name: configured_stream.clone(), + subjects: vec![parked_subject], + duplicate_window: Duration::from_secs(300), + ..Default::default() + }) + .await + .expect("remap the configured stream away from the subject"); + provision_context + .create_stream(jetstream::stream::Config { + name: takeover_stream.clone(), + subjects: vec![subject.clone()], + duplicate_window: Duration::from_secs(300), + ..Default::default() + }) + .await + .expect("provision the takeover stream"); + + let queued = target.build_queued_payload(&sample_event()).expect("payload builds"); + let store = target.store().expect("a store is configured"); + persist_queued_payload_to_store(store, &queued).expect("entry persists"); + let key = store.list().pop().expect("the entry is listed"); + + let log = CapturedLog::default(); + let subscriber = tracing_subscriber::fmt() + .with_max_level(tracing::Level::WARN) + .with_writer(log.clone()) + .finish(); + let _log_guard = tracing::subscriber::set_default(subscriber); + + let err = target + .send_from_store(key.clone()) + .await + .expect_err("the takeover stream's acknowledgment is rejected"); + match err { + TargetError::JetStreamPublish { retryable, detail } => { + assert!(retryable, "the mismatch keeps the entry on the live queue"); + assert_eq!(detail, ACK_STREAM_MISMATCH_DETAIL, "the fixed mismatch label is the detail"); + } + other => panic!("expected the mismatch classification, got {other:?}"), + } + assert_eq!(store.len(), 1, "the queued entry survives the rejected acknowledgment"); + + let warn_line = log.contents(); + assert!( + warn_line.contains("acknowledged by an unexpected stream"), + "the mismatch warn is logged: {warn_line}" + ); + assert!( + warn_line.contains(&takeover_stream), + "the warn names the acknowledging stream: {warn_line}" + ); + + // The reset verdict forces re-validation on the next attempt, which reports the configured + // stream no longer captures the subject. The entry stays queued through that failure too. + let err = target + .send_from_store(key) + .await + .expect_err("re-validation fails while the subject is remapped away"); + match err { + TargetError::JetStreamPublish { detail, .. } => { + assert_eq!(detail, STREAM_VALIDATION_FAILED_DETAIL, "the retry re-validates the stream"); + } + other => panic!("expected the validation classification, got {other:?}"), + } + assert_eq!(store.len(), 1, "the entry stays queued while the remap persists"); + + let _ = provision_context.delete_stream(&configured_stream).await; + let _ = provision_context.delete_stream(&takeover_stream).await; + } + + // A successful move deletes the live entry and leaves exactly one failed entry. + #[tokio::test] + async fn move_entry_writes_failed_then_deletes_live() { + let dir = failed_store_dir("move-order"); + let store = QueueStore::::new_with_compression(&dir, 8, ".test", false); + store.open().unwrap(); + + let key = store.put_raw(&sample_queued("minted-id").encode().unwrap()).unwrap(); + assert_eq!(store.len(), 1); + + let target = move_test_target(); + let error = TargetError::JetStreamPublish { + retryable: false, + detail: "max payload exceeded".to_string(), + }; + move_entry_to_failed_store(&store, &store, &target.id(), &key, &error, 0) + .await + .unwrap(); + + assert_eq!(store.len(), 0, "the live entry is deleted after the failed write"); + assert_eq!(store.failed_len(), 1, "the failed entry is written"); + assert!(matches!(store.get_raw(&key), Err(StoreError::NotFound))); + + let _ = store.delete(); + } + + // A move of an entry the queue no longer holds is a no-op, not an error. + #[tokio::test] + async fn move_entry_is_a_noop_when_the_live_entry_is_already_cleared() { + let dir = failed_store_dir("move-missing"); + let store = QueueStore::::new_with_compression(&dir, 8, ".test", false); + store.open().unwrap(); + + let absent_key = Key { + name: "gone".to_string(), + extension: ".test".to_string(), + item_count: 1, + compress: false, + }; + let target = move_test_target(); + let error = TargetError::JetStreamPublish { + retryable: false, + detail: "max payload exceeded".to_string(), + }; + move_entry_to_failed_store(&store, &store, &target.id(), &absent_key, &error, 0) + .await + .unwrap(); + + assert_eq!(store.failed_len(), 0, "no failed entry is written for an absent live entry"); + + let _ = store.delete(); + } + + // The delivery snapshot reports the failed-store depth read from the store. + #[tokio::test] + async fn delivery_snapshot_reports_failed_store_depth() { + let dir = failed_store_dir("snapshot-depth"); + let store = Arc::new(QueueStore::::new_with_compression(&dir, 8, ".test", false)); + store.open().unwrap(); + + let target = move_test_target_with_store(store.clone()); + assert_eq!( + target.delivery_snapshot().failed_store_length, + 0, + "an empty failed store reports depth zero" + ); + + let key = store.put_raw(&sample_queued("minted-id").encode().unwrap()).unwrap(); + let error = TargetError::JetStreamPublish { + retryable: false, + detail: "max payload exceeded".to_string(), + }; + move_entry_to_failed_store(&*store, &*store, &target.id(), &key, &error, 0) + .await + .unwrap(); + + assert_eq!( + target.delivery_snapshot().failed_store_length, + 1, + "a terminal failure lands one entry and the snapshot reports depth one" + ); + + let _ = store.delete(); + } + + // For a pre-enable entry with an empty stored id, the failed record carries the id derived from the entry key. + #[tokio::test] + async fn move_entry_records_the_resolved_dedup_id_for_a_pre_enable_entry() { + let dir = failed_store_dir("resolved-id"); + let store = QueueStore::::new_with_compression(&dir, 8, ".test", false); + store.open().unwrap(); + + let key = store.put_raw(&sample_queued("").encode().unwrap()).unwrap(); + let expected = resolve_dedup_id("", &key); + assert!(!expected.is_empty(), "the resolved id is derived from the entry key"); + + let target = move_test_target(); + let error = TargetError::JetStreamPublish { + retryable: false, + detail: "max payload exceeded".to_string(), + }; + move_entry_to_failed_store(&store, &store, &target.id(), &key, &error, 0) + .await + .unwrap(); + + let failed_dir = dir.join("failed"); + let failed_file = std::fs::read_dir(&failed_dir) + .unwrap() + .filter_map(|entry| entry.ok()) + .map(|entry| entry.path()) + .find(|path| path.is_file()) + .expect("a failed entry is written"); + let decoded = QueuedPayload::decode(&std::fs::read(&failed_file).unwrap()).unwrap(); + assert_eq!( + decoded.meta.failure.unwrap().nats_msg_id, + expected, + "the failed record carries the resolved id, not the empty stored id" + ); + + let _ = store.delete(); + } + + // The move does not count the failure, so a move followed by one replay hook emit counts it exactly once. + #[tokio::test] + async fn a_single_failed_delivery_counts_once() { + let dir = failed_store_dir("count-once"); + let store = Arc::new(QueueStore::::new_with_compression(&dir, 8, ".test", false)); + store.open().unwrap(); + + let failed = Arc::new(AtomicU64::new(0)); + let target: Arc + Send + Sync> = Arc::new(MoveTestTarget { + id: TargetID::new("target-a".to_string(), "nats".to_string()), + store: Some(store.clone()), + failed: failed.clone(), + }); + + let key = store.put_raw(&sample_queued("minted-id").encode().unwrap()).unwrap(); + let error = TargetError::JetStreamPublish { + retryable: false, + detail: "max payload exceeded".to_string(), + }; + move_entry_to_failed_store(&*store, &*store, &target.id(), &key, &error, 0) + .await + .unwrap(); + assert_eq!(failed.load(Ordering::Relaxed), 0, "the move itself does not count the failure"); + + // The replay worker follows every move with one hook emit that records the failure. + target.record_final_failure(); + assert_eq!(failed.load(Ordering::Relaxed), 1, "one failed delivery counts exactly once"); + + let _ = store.delete(); + } +} diff --git a/crates/targets/src/target/nats.rs b/crates/targets/src/target/nats/mod.rs similarity index 57% rename from crates/targets/src/target/nats.rs rename to crates/targets/src/target/nats/mod.rs index 7b13fe3e0..81cc37c8f 100644 --- a/crates/targets/src/target/nats.rs +++ b/crates/targets/src/target/nats/mod.rs @@ -21,22 +21,36 @@ use crate::{ ReloadableTargetTls, TargetTlsGeneration, TargetTlsInputSet, TlsReloadAdapter, config::ReloadApplyMode, validate_tls_material, }, - store::{Key, Store}, + store::{FailedEventStore, Key, QueueStore, Store}, target::{ ChannelTargetType, EntityTarget, QueuedPayload, QueuedPayloadMeta, TargetDeliveryCounters, TargetDeliverySnapshot, TargetTlsState, TargetType, build_queued_payload_with_records, build_target_tls_fingerprint, is_connectivity_error, - open_target_queue_store, persist_queued_payload_to_store, redacted_secret, + open_target_queue_store_typed, persist_queued_payload_to_store, redacted_secret, }, }; use async_trait::async_trait; -use rustfs_config::{NATS_CREDENTIALS_FILE, NATS_TLS_CA, NATS_TLS_CLIENT_CERT, NATS_TLS_CLIENT_KEY}; +use rustfs_config::{ + NATS_CREDENTIALS_FILE, NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS, NATS_TLS_CA, NATS_TLS_CLIENT_CERT, NATS_TLS_CLIENT_KEY, +}; use std::fmt; use std::path::{Path, PathBuf}; use std::str::FromStr; use std::sync::Arc; use std::sync::atomic::{AtomicBool, Ordering}; +use std::time::Duration; use tokio::sync::Mutex; -use tracing::{info, instrument, warn}; +use tracing::{error, info, instrument, warn}; +use uuid::Uuid; + +mod jetstream; +mod publish_error; +mod validation; + +use jetstream::{CachedJetStreamContext, drain_jetstream_context}; +use publish_error::{classify_nats_flush_error, classify_nats_publish_error}; + +pub(crate) use jetstream::resolve_dedup_id; +pub(crate) use validation::{validate_jetstream_settings, validate_jetstream_stream}; #[derive(Clone)] pub struct NATSArgs { @@ -53,6 +67,10 @@ pub struct NATSArgs { pub tls_required: bool, pub queue_dir: String, pub queue_limit: u64, + // JetStream publish settings. Absent maps to off and the defaults. + pub jetstream_enable: Option, + pub jetstream_stream_name: Option, + pub jetstream_ack_timeout_secs: Option, pub target_type: TargetType, } @@ -72,6 +90,9 @@ impl fmt::Debug for NATSArgs { .field("tls_required", &self.tls_required) .field("queue_dir", &self.queue_dir) .field("queue_limit", &self.queue_limit) + .field("jetstream_enable", &self.jetstream_enable) + .field("jetstream_stream_name", &self.jetstream_stream_name) + .field("jetstream_ack_timeout_secs", &self.jetstream_ack_timeout_secs) .field("target_type", &self.target_type) .finish() } @@ -114,6 +135,13 @@ impl NATSArgs { return Err(TargetError::Configuration("NATS queue directory must be an absolute path".to_string())); } + validate_jetstream_settings( + self.jetstream_enable.unwrap_or(false), + self.jetstream_stream_name.as_deref().unwrap_or_default(), + &self.queue_dir, + self.jetstream_ack_timeout_secs, + )?; + Ok(()) } } @@ -160,12 +188,9 @@ fn validate_nats_auth(args: &NATSArgs) -> Result<(), TargetError> { Ok(()) } -/// Returns true when the target is configured to send credentials (token, -/// username/password, or a credentials file) over a connection that does not -/// require TLS, which would transmit the secrets in cleartext (backlog#983). -/// -/// TLS is considered active when `tls_required` is set or the address uses the -/// `tls://` scheme. +/// Returns true when the target sends credentials over a connection that does not require TLS, which +/// would transmit the secrets in cleartext (backlog#983). TLS is active when tls_required is set or +/// the address uses the tls:// scheme. fn nats_sends_credentials_without_tls(args: &NATSArgs) -> bool { let has_auth = !args.token.is_empty() || !args.credentials_file.is_empty() || !args.username.is_empty(); if !has_auth { @@ -211,12 +236,16 @@ where id: TargetID, args: NATSArgs, client: Arc>>, + /// Cached JetStream context, one per target shared across clones so a single acker and semaphore + /// serve every clone. Read out under the lock before each publish await, never held across it. + /// Carries the stream-validation verdict for the bound connection. + jetstream_context: Arc>>, tls_state: Arc>, - /// Adapter that bridges this target to the TLS reload coordinator. - /// When `Some`, the target uses coordinator-managed material; when `None`, - /// it falls back to inline fingerprint-based change detection. + /// When set, the coordinator drives TLS reload, otherwise inline fingerprint change detection. tls_adapter: Option>, - store: Option + Send + Sync>>, + /// The concrete queue store, held typed so the target projects both the generic Store handle and + /// its own failed-events capability from a single store. Shared across clones through the Arc. + store: Option>>, connected: AtomicBool, delivery_counters: Arc, _phantom: std::marker::PhantomData, @@ -231,9 +260,10 @@ where id: self.id.clone(), args: self.args.clone(), client: Arc::clone(&self.client), + jetstream_context: Arc::clone(&self.jetstream_context), tls_state: Arc::clone(&self.tls_state), tls_adapter: self.tls_adapter.clone(), - store: self.store.as_ref().map(|s| s.boxed_clone()), + store: self.store.clone(), connected: AtomicBool::new(self.connected.load(Ordering::SeqCst)), delivery_counters: Arc::clone(&self.delivery_counters), _phantom: std::marker::PhantomData, @@ -251,19 +281,21 @@ where ); } let target_id = TargetID::new(id, ChannelTargetType::Nats.as_str().to_string()); - let queue_store = open_target_queue_store( + let queue_store = open_target_queue_store_typed( &args.queue_dir, args.queue_limit, args.target_type, ChannelTargetType::Nats.as_str(), &target_id, "Failed to open store for NATS target", - )?; + )? + .map(Arc::new); Ok(Self { id: target_id, args, client: Arc::new(Mutex::new(None)), + jetstream_context: Arc::new(Mutex::new(None)), tls_state: Arc::new(parking_lot::Mutex::new(TargetTlsState::default())), tls_adapter: None, store: queue_store, @@ -299,7 +331,6 @@ where }; if tls_changed { self.invalidate_cached_client_connection().await; - self.tls_state.lock().refresh(next_fingerprint); } { @@ -316,13 +347,53 @@ where .map_err(|e| TargetError::Network(format!("Failed to flush NATS connection: {e}")))?; self.connected.store(true, Ordering::SeqCst); - let mut guard = self.client.lock().await; - let shared = guard.get_or_insert_with(|| client.clone()).clone(); + let mut client_guard = self.client.lock().await; + let shared = client_guard.get_or_insert_with(|| client.clone()).clone(); + + // Swap in a context built from the winning client while the client lock is held, so client + // and context swap as a unit and no clone reads a context bound to the old client. Only the + // JetStream path caches a context. The previous context is drained after the locks release. + let previous_context = if tls_changed && self.jetstream_enabled() { + let ack_timeout = self.ack_timeout(); + let mut context_guard = self.jetstream_context.lock().await; + let mut context = async_nats::jetstream::new(shared.clone()); + context.set_timeout(ack_timeout); + context_guard.replace(CachedJetStreamContext::new(context)) + } else { + None + }; + drop(client_guard); + + // Advance the recorded fingerprint only after the reconnect and flush succeed, so a rotation + // whose first reconnect fails stays detected and the next success still rebuilds the context. + if tls_changed { + self.tls_state.lock().refresh(next_fingerprint); + } + + drain_jetstream_context(previous_context.map(|cached| cached.context)).await; Ok(shared) } + fn jetstream_enabled(&self) -> bool { + self.args.jetstream_enable.unwrap_or(false) + } + + fn ack_timeout(&self) -> Duration { + Duration::from_secs( + self.args + .jetstream_ack_timeout_secs + .unwrap_or(NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS), + ) + } + fn build_queued_payload(&self, event: &EntityTarget) -> Result { - build_queued_payload_with_records(event, vec![event.clone()]) + let mut queued = build_queued_payload_with_records(event, vec![event.clone()])?; + if self.jetstream_enabled() { + // Mint the dedup id once at enqueue so it is identical across every retry and replay and + // distinct across entries. The body-hash fallback only covers entries queued before enable. + queued.meta.dedup_id = Uuid::new_v4().to_string(); + } + Ok(queued) } async fn send_body(&self, body: Vec) -> Result<(), TargetError> { @@ -336,10 +407,8 @@ where return Err(err); } - // `publish` only enqueues the message on the client's outbound channel; - // it does not guarantee the broker received it. Flush to confirm the - // message reached the server before the caller treats delivery as - // successful and deletes the durable copy (backlog#971). + // publish only enqueues the message on the client's outbound channel. Flush to confirm the + // message reached the server before delivery is treated as successful (backlog#971). if let Err(e) = client.flush().await { let err = classify_nats_flush_error(&e); self.invalidate_cached_client_connection().await; @@ -352,30 +421,6 @@ where } } -/// Classifies a NATS publish failure using the typed error kind rather than a -/// substring match. `Send` means the outbound channel is closed (the connection -/// is gone) and is retriable; payload/subject violations are permanent -/// request-level errors (backlog#971, backlog#973). -fn classify_nats_publish_error(err: &async_nats::PublishError) -> TargetError { - use async_nats::PublishErrorKind; - match err.kind() { - PublishErrorKind::Send => TargetError::NotConnected, - PublishErrorKind::MaxPayloadExceeded | PublishErrorKind::InvalidSubject => { - TargetError::Request(format!("Failed to publish NATS message: {err}")) - } - } -} - -/// Classifies a NATS flush failure. Both flush error kinds indicate the message -/// was not confirmed on the wire (a connectivity problem), so the event is kept -/// for replay (backlog#971, backlog#973). -fn classify_nats_flush_error(err: &async_nats::client::FlushError) -> TargetError { - use async_nats::client::FlushErrorKind; - match err.kind() { - FlushErrorKind::SendError | FlushErrorKind::FlushError => TargetError::NotConnected, - } -} - #[async_trait] impl Target for NATSTarget where @@ -386,6 +431,13 @@ where } async fn is_active(&self) -> Result { + // With JetStream enabled the health answer covers the stream as well as the connection, so a + // reachable broker with a failing stream reports the validation error. The verdict is cached + // and reset on a reconnect, TLS rotation, wrong-stream ack, or stream-not-found outcome, so a + // reset forces a live lookup on the next check. + if self.jetstream_enabled() { + self.validated_jetstream_context().await?; + } let client = self.get_or_connect().await?; client .flush() @@ -418,11 +470,31 @@ where } } - async fn send_raw_from_store(&self, _key: Key, body: Vec, _meta: QueuedPayloadMeta) -> Result<(), TargetError> { - self.send_body(body).await + async fn send_raw_from_store(&self, key: Key, body: Vec, meta: QueuedPayloadMeta) -> Result<(), TargetError> { + if self.jetstream_enabled() { + let dedup_id = resolve_dedup_id(&meta.dedup_id, &key); + self.publish_jetstream(body, &dedup_id).await + } else { + self.send_body(body).await + } } async fn close(&self) -> Result<(), TargetError> { + // Drain the cached context's in-flight acks before dropping it so the async-nats acker exits. + // Drain the context before the client, since the acker rides the client connection. + let context = { + let mut guard = self.jetstream_context.lock().await; + guard.take() + }; + // Bound the drain by the ack timeout so an unresponsive broker cannot stall close. On elapse + // the client drain below still runs. + if tokio::time::timeout(self.ack_timeout(), drain_jetstream_context(context.map(|cached| cached.context))) + .await + .is_err() + { + warn!(target_id = %self.id, "Timed out draining JetStream acks on close, proceeding to close the client"); + } + let client = { let mut guard = self.client.lock().await; guard.take() @@ -435,12 +507,49 @@ where .await .map_err(|e| TargetError::Network(format!("Failed to drain NATS client: {e}")))?; } + // The durable queue store stays on disk, so a queued entry survives close and replays. Close + // releases the cached handles, never the entries. info!(target_id = %self.id, "NATS target closed"); Ok(()) } fn store(&self) -> Option<&(dyn Store + Send + Sync)> { - self.store.as_deref() + self.store + .as_deref() + .map(|store| store as &(dyn Store + Send + Sync)) + } + + fn failed_store(&self) -> Option<&dyn FailedEventStore> { + self.store.as_deref().map(|store| store as &dyn FailedEventStore) + } + + async fn handle_terminal_failure( + &self, + store: &(dyn Store + Send), + key: &Key, + error: &TargetError, + retry_count: u32, + ) -> bool { + let Some(failed_store) = self.failed_store() else { + error!( + target_id = %self.id, + replay_key = %key, + "NATS target has no failed-events store for the terminal move" + ); + return false; + }; + match jetstream::move_entry_to_failed_store(store, failed_store, &self.id, key, error, retry_count).await { + Ok(()) => true, + Err(move_err) => { + error!( + target_id = %self.id, + error = %move_err, + replay_key = %key, + "Failed to move event to the failed-events store" + ); + false + } + } } fn clone_dyn(&self) -> Box + Send + Sync> { @@ -452,6 +561,13 @@ where return Ok(()); } let _ = self.get_or_connect().await?; + if self.jetstream_enabled() { + let cached = self.jetstream_context().await?; + validate_jetstream_stream(&cached.context, &self.args, &self.id.to_string(), Some(&cached.validation_logged)).await?; + // Record the verdict on the context so the first publish does not repeat the stream + // lookup that just passed. + cached.stream_validated.store(true, Ordering::Release); + } Ok(()) } @@ -460,8 +576,10 @@ where } fn delivery_snapshot(&self) -> TargetDeliverySnapshot { - self.delivery_counters - .snapshot(self.store.as_deref().map_or(0, |store| store.len() as u64)) + self.delivery_counters.snapshot( + self.store.as_deref().map_or(0, |store| store.len() as u64), + self.failed_store().map_or(0, |failed_store| failed_store.failed_len() as u64), + ) } fn record_final_failure(&self) { @@ -510,11 +628,18 @@ where } #[cfg(test)] -mod tests { +pub(crate) mod test_support { use super::*; - use crate::target::REDACTED_SECRET; + use async_nats::jetstream; + use rustfs_s3_types::EventName; - fn base_args() -> NATSArgs { + // Absolute on Linux, macOS, and Windows. temp_dir needs no filesystem to exist for a + // validation-only test, and Path::is_absolute stays true across platforms. + pub(crate) fn nats_queue_dir() -> String { + std::env::temp_dir().join("rustfs-nats-queue").to_string_lossy().into_owned() + } + + pub(crate) fn base_args() -> NATSArgs { NATSArgs { enable: true, address: "nats://127.0.0.1:4222".to_string(), @@ -529,10 +654,123 @@ mod tests { tls_required: false, queue_dir: String::new(), queue_limit: 0, + jetstream_enable: None, + jetstream_stream_name: None, + jetstream_ack_timeout_secs: None, target_type: TargetType::NotifyEvent, } } + pub(crate) fn jetstream_args(target_type: TargetType) -> NATSArgs { + NATSArgs { + jetstream_enable: Some(true), + jetstream_stream_name: Some("RUSTFS_EVENTS".to_string()), + jetstream_ack_timeout_secs: Some(30), + target_type, + ..base_args() + } + } + + pub(crate) fn nats_target(args: NATSArgs) -> NATSTarget { + NATSTarget:: { + id: TargetID::new("test-target".to_string(), ChannelTargetType::Nats.as_str().to_string()), + args, + client: Arc::new(Mutex::new(None)), + jetstream_context: Arc::new(Mutex::new(None)), + tls_state: Arc::new(parking_lot::Mutex::new(TargetTlsState::default())), + tls_adapter: None, + store: None, + connected: AtomicBool::new(false), + delivery_counters: Arc::new(TargetDeliveryCounters::default()), + _phantom: std::marker::PhantomData, + } + } + + pub(crate) fn sample_event() -> EntityTarget { + EntityTarget { + object_name: "folder/object.txt".to_string(), + bucket_name: "bucket-a".to_string(), + event_name: EventName::ObjectCreatedPut, + data: "payload-data".to_string(), + } + } + + pub(crate) fn sample_stored_key(name: &str) -> Key { + Key { + name: name.to_string(), + extension: ".event".to_string(), + item_count: 1, + compress: false, + } + } + + pub(crate) fn nats_target_with_store(args: NATSArgs, queue_dir: &str) -> NATSTarget { + let mut configured = args; + configured.queue_dir = queue_dir.to_string(); + NATSTarget::::new("store-target".to_string(), configured).expect("target with store builds") + } + + /// A stream configuration that passes every assertion: it captures the configured subject, returns + /// acks, accepts writes, and deduplicates well beyond the retry lifetime. + pub(crate) fn writable_stream_config(subject: &str) -> jetstream::stream::Config { + jetstream::stream::Config { + name: "RUSTFS_EVENTS".to_string(), + subjects: vec![subject.to_string()], + no_ack: false, + sealed: false, + duplicate_window: Duration::from_secs(600), + ..Default::default() + } + } + + // Live-broker behaviour tests. They require a NATS server with JetStream enabled and are ignored by + // default. To run locally: + // + // docker run -d --name rustfs-nats-test -p 4222:4222 nats:2 -js + // cargo test -p rustfs-targets --lib -- --ignored nats::tests::tls_change + // + // Override the server URL with RUSTFS_TEST_NATS_URL. + pub(crate) fn broker_url() -> String { + std::env::var("RUSTFS_TEST_NATS_URL").unwrap_or_else(|_| "nats://127.0.0.1:4222".to_string()) + } + + /// Log sink for asserting a specific line is written. The subscriber writes into a shared + /// buffer the assertion reads back. + #[derive(Clone, Default)] + pub(crate) struct CapturedLog(Arc>>); + + impl CapturedLog { + pub(crate) fn contents(&self) -> String { + String::from_utf8_lossy(&self.0.lock()).into_owned() + } + } + + impl std::io::Write for CapturedLog { + fn write(&mut self, buf: &[u8]) -> std::io::Result { + self.0.lock().extend_from_slice(buf); + Ok(buf.len()) + } + + fn flush(&mut self) -> std::io::Result<()> { + Ok(()) + } + } + + impl<'a> tracing_subscriber::fmt::MakeWriter<'a> for CapturedLog { + type Writer = CapturedLog; + + fn make_writer(&'a self) -> Self::Writer { + self.clone() + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::target::REDACTED_SECRET; + use crate::target::nats::test_support::*; + #[test] fn debug_redacts_nats_secret_fields() { let args = NATSArgs { @@ -575,8 +813,7 @@ mod tests { #[test] fn nats_credentials_without_tls_is_detected() { - // Token auth over a plaintext nats:// address without tls_required leaks - // the credential in cleartext (backlog#983). + // Token auth over a plaintext nats:// address without tls_required leaks the credential (backlog#983). let insecure = NATSArgs { token: "secret-token".to_string(), tls_required: false, @@ -603,25 +840,17 @@ mod tests { assert!(!nats_sends_credentials_without_tls(&no_auth)); } - #[test] - fn nats_publish_error_classification() { - use async_nats::PublishErrorKind; - // `Send` means the outbound channel is gone: retriable connectivity error. - let send_err = async_nats::PublishError::new(PublishErrorKind::Send); - assert!(matches!(classify_nats_publish_error(&send_err), TargetError::NotConnected)); + #[tokio::test] + async fn flag_off_stored_meta_is_byte_identical_to_pre_feature() { + // With the flag off the payload carries no dedup id, so its encoded meta matches a pre-feature entry. + let disabled = nats_target(base_args()); + let off_payload = disabled.build_queued_payload(&sample_event()).expect("payload builds"); + assert!(off_payload.meta.dedup_id.is_empty(), "no dedup id is minted with the flag off"); - // Payload/subject violations are permanent request-level errors. - let payload_err = async_nats::PublishError::new(PublishErrorKind::MaxPayloadExceeded); - assert!(matches!(classify_nats_publish_error(&payload_err), TargetError::Request(_))); - let subject_err = async_nats::PublishError::new(PublishErrorKind::InvalidSubject); - assert!(matches!(classify_nats_publish_error(&subject_err), TargetError::Request(_))); - } + let meta_json = serde_json::to_string(&off_payload.meta).expect("meta serializes"); + assert!(!meta_json.contains("dedup_id"), "the absent dedup id is skipped on serialization"); - #[test] - fn nats_flush_error_classification() { - use async_nats::client::{FlushError, FlushErrorKind}; - for kind in [FlushErrorKind::SendError, FlushErrorKind::FlushError] { - assert!(matches!(classify_nats_flush_error(&FlushError::new(kind)), TargetError::NotConnected)); - } + // No JetStream context is built on the flag-off path. + assert!(disabled.jetstream_context.lock().await.is_none(), "no context is built with the flag off"); } } diff --git a/crates/targets/src/target/nats/publish_error.rs b/crates/targets/src/target/nats/publish_error.rs new file mode 100644 index 000000000..cb2ec7c9e --- /dev/null +++ b/crates/targets/src/target/nats/publish_error.rs @@ -0,0 +1,346 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use super::jetstream::STREAM_NOT_FOUND_DETAIL; +use crate::error::TargetError; +use async_nats::jetstream::context::PublishErrorKind; + +/// Maps an async-nats publish error to a typed TargetError with the retryable-versus-terminal +/// classification, so the replay loop retries recoverable errors instead of dropping them. The +/// detail comes from the fixed vocabulary in publish_error_detail, never from raw error text. +pub(crate) fn classify_publish_error(err: &async_nats::jetstream::context::PublishError) -> TargetError { + let retryable = match err.kind() { + PublishErrorKind::TimedOut + | PublishErrorKind::BrokenPipe + | PublishErrorKind::MaxAckPending + | PublishErrorKind::StreamNotFound => true, + PublishErrorKind::MaxPayloadExceeded | PublishErrorKind::WrongLastMessageId | PublishErrorKind::WrongLastSequence => { + false + } + PublishErrorKind::Other => !other_error_is_terminal(err), + }; + TargetError::JetStreamPublish { + retryable, + detail: publish_error_detail(err), + } +} + +/// Fixed diagnostic detail for a publish failure. Each error kind maps to a constant label, and a +/// server rejection carries the numeric JetStream error code and status. Never the library rendering +/// or any server-returned text, so the detail is safe to persist in a failed-store entry. +fn publish_error_detail(err: &async_nats::jetstream::context::PublishError) -> String { + use async_nats::jetstream::Error as JetStreamApiError; + match err.kind() { + PublishErrorKind::StreamNotFound => STREAM_NOT_FOUND_DETAIL.to_string(), + PublishErrorKind::TimedOut => "publish timed out".to_string(), + PublishErrorKind::BrokenPipe => "broken pipe".to_string(), + PublishErrorKind::MaxAckPending => "max ack pending reached".to_string(), + PublishErrorKind::MaxPayloadExceeded => "max payload exceeded".to_string(), + PublishErrorKind::WrongLastMessageId => "wrong last message id".to_string(), + PublishErrorKind::WrongLastSequence => "wrong last sequence".to_string(), + PublishErrorKind::Other => { + match std::error::Error::source(err).and_then(|source| source.downcast_ref::()) { + Some(api_error) => format!("server error code {} status {}", api_error.error_code().0, api_error.code()), + None => "publish failed".to_string(), + } + } + } +} + +/// HTTP-style 5xx status range. A rejection reporting a 5xx status is a server-side condition and +/// stays retryable regardless of its error code. +const SERVER_ERROR_STATUS_RANGE: std::ops::Range = 500..600; + +/// Classifies an Other publish error against a terminal allowlist. Terminal only for an explicit +/// permanent error code, STREAM_SEALED being the sole current member. Everything else is retryable, +/// so only a recognized permanent rejection enters the failed store. +fn other_error_is_terminal(err: &async_nats::jetstream::context::PublishError) -> bool { + use async_nats::jetstream::{Error as JetStreamApiError, ErrorCode}; + + let Some(api_error) = std::error::Error::source(err).and_then(|source| source.downcast_ref::()) else { + return false; + }; + + // A 5xx status is a server-side condition, so it stays retryable even when the error code is a + // permanent one. + if SERVER_ERROR_STATUS_RANGE.contains(&api_error.code()) { + return false; + } + + matches!(api_error.error_code(), ErrorCode::STREAM_SEALED) +} + +/// Classifies a NATS publish failure by its typed error kind. Send means the outbound channel is +/// closed and is retriable. Payload and subject violations are permanent request-level errors +/// (backlog#971, backlog#973). +pub(crate) fn classify_nats_publish_error(err: &async_nats::PublishError) -> TargetError { + use async_nats::PublishErrorKind; + match err.kind() { + PublishErrorKind::Send => TargetError::NotConnected, + PublishErrorKind::MaxPayloadExceeded | PublishErrorKind::InvalidSubject => { + TargetError::Request(format!("Failed to publish NATS message: {err}")) + } + } +} + +/// Classifies a NATS flush failure. Both kinds mean the message was not confirmed on the wire, so +/// the event is kept for replay (backlog#971, backlog#973). +pub(crate) fn classify_nats_flush_error(err: &async_nats::client::FlushError) -> TargetError { + use async_nats::client::FlushErrorKind; + match err.kind() { + FlushErrorKind::SendError | FlushErrorKind::FlushError => TargetError::NotConnected, + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::target::nats::jetstream::{ACK_STREAM_MISMATCH_DETAIL, ATTEMPT_DEADLINE_DETAIL}; + use crate::target::nats::validation::gate_publish_on_stream_validation; + use async_nats::jetstream::context::{PublishError, PublishErrorKind}; + use std::sync::atomic::AtomicBool; + + #[test] + fn nats_publish_error_classification() { + use async_nats::PublishErrorKind; + // Send means the outbound channel is gone: retriable connectivity error. + let send_err = async_nats::PublishError::new(PublishErrorKind::Send); + assert!(matches!(classify_nats_publish_error(&send_err), TargetError::NotConnected)); + + // Payload and subject violations are permanent request-level errors. + let payload_err = async_nats::PublishError::new(PublishErrorKind::MaxPayloadExceeded); + assert!(matches!(classify_nats_publish_error(&payload_err), TargetError::Request(_))); + let subject_err = async_nats::PublishError::new(PublishErrorKind::InvalidSubject); + assert!(matches!(classify_nats_publish_error(&subject_err), TargetError::Request(_))); + } + + #[test] + fn nats_flush_error_classification() { + use async_nats::client::{FlushError, FlushErrorKind}; + for kind in [FlushErrorKind::SendError, FlushErrorKind::FlushError] { + assert!(matches!(classify_nats_flush_error(&FlushError::new(kind)), TargetError::NotConnected)); + } + } + + #[test] + fn classify_publish_error_marks_recoverable_kinds_retryable() { + for kind in [ + PublishErrorKind::TimedOut, + PublishErrorKind::BrokenPipe, + PublishErrorKind::MaxAckPending, + PublishErrorKind::StreamNotFound, + ] { + let classified = classify_publish_error(&PublishError::new(kind)); + match classified { + TargetError::JetStreamPublish { retryable, .. } => assert!(retryable, "{kind} is retryable"), + other => panic!("expected JetStreamPublish, got {other:?}"), + } + } + } + + #[test] + fn classify_publish_error_marks_terminal_kinds_terminal() { + for kind in [ + PublishErrorKind::MaxPayloadExceeded, + PublishErrorKind::WrongLastMessageId, + PublishErrorKind::WrongLastSequence, + ] { + let classified = classify_publish_error(&PublishError::new(kind)); + match classified { + TargetError::JetStreamPublish { retryable, .. } => assert!(!retryable, "{kind} is terminal"), + other => panic!("expected JetStreamPublish, got {other:?}"), + } + } + } + + #[test] + fn classify_publish_error_other_without_source_is_retryable() { + // An undecodable Other carries no code to recognize as permanent, so it stays on the retry path. + let classified = classify_publish_error(&PublishError::new(PublishErrorKind::Other)); + match classified { + TargetError::JetStreamPublish { retryable, .. } => assert!(retryable, "an undecodable Other is retryable"), + other => panic!("expected JetStreamPublish, got {other:?}"), + } + } + + #[test] + fn classify_publish_error_unknown_code_is_retryable() { + // A decoded rejection carrying a code outside the terminal allowlist is retryable. + let api_error: async_nats::jetstream::Error = + serde_json::from_value(serde_json::json!({"code": 400, "err_code": 10999, "description": "an unrecognized code"})) + .expect("api error deserializes"); + let publish_error = PublishError::with_source(PublishErrorKind::Other, api_error); + match classify_publish_error(&publish_error) { + TargetError::JetStreamPublish { retryable, .. } => assert!(retryable, "an unknown error code is retryable"), + other => panic!("expected JetStreamPublish, got {other:?}"), + } + } + + #[test] + fn classify_publish_error_other_with_transient_code_is_retryable() { + // Server conditions that each clear on their own. None is in the terminal allowlist, so each + // is retryable. The status is 400 in every case, so retryability comes from the allowlist + // policy rather than the 5xx status range. + for (err_code, condition) in [ + (10002, "account resources exceeded"), + (10008, "cluster not available"), + (10009, "cluster not leader"), + (10028, "memory resources exceeded"), + (10039, "jetstream not enabled for account"), + (10040, "cluster peer not a member"), + (10041, "raft general error"), + (10076, "jetstream not enabled"), + (10118, "stream offline"), + (10194, "stream offline with a reason"), + (10202, "server member change in flight"), + ] { + let api_error: async_nats::jetstream::Error = + serde_json::from_value(serde_json::json!({"code": 400, "err_code": err_code, "description": condition})) + .expect("api error deserializes"); + let publish_error = PublishError::with_source(PublishErrorKind::Other, api_error); + let classified = classify_publish_error(&publish_error); + match classified { + TargetError::JetStreamPublish { retryable, .. } => { + assert!(retryable, "transient code {err_code} ({condition}) is retryable") + } + other => panic!("expected JetStreamPublish for code {err_code}, got {other:?}"), + } + } + } + + #[test] + fn classify_publish_error_uncoded_server_error_status_is_retryable() { + // A rejection with no err_code and a 5xx status is a server-side condition, kept on the retry path. + for status in [500, 503] { + let api_error: async_nats::jetstream::Error = + serde_json::from_value(serde_json::json!({"code": status, "description": "insufficient resources"})) + .expect("api error deserializes"); + let publish_error = PublishError::with_source(PublishErrorKind::Other, api_error); + let classified = classify_publish_error(&publish_error); + match classified { + TargetError::JetStreamPublish { retryable, .. } => { + assert!(retryable, "an uncoded {status} status is retryable") + } + other => panic!("expected JetStreamPublish for status {status}, got {other:?}"), + } + } + } + + #[test] + fn classify_publish_error_other_with_terminal_code_is_terminal() { + // STREAM_SEALED (10109) is the sole terminal-allowlist member and its 400 status keeps it terminal. + let api_error: async_nats::jetstream::Error = + serde_json::from_value(serde_json::json!({"code": 400, "err_code": 10109, "description": "stream sealed"})) + .expect("api error deserializes"); + let publish_error = PublishError::with_source(PublishErrorKind::Other, api_error); + match classify_publish_error(&publish_error) { + TargetError::JetStreamPublish { retryable, .. } => assert!(!retryable, "a sealed stream is terminal"), + other => panic!("expected JetStreamPublish, got {other:?}"), + } + } + + #[test] + fn classify_publish_error_uncoded_4xx_is_retryable() { + // An uncoded 4xx rejection carries no permanent code, so the allowlist policy leaves it on the retry path. + let api_error: async_nats::jetstream::Error = + serde_json::from_value(serde_json::json!({"code": 400, "description": "bad request without an error code"})) + .expect("api error deserializes"); + let publish_error = PublishError::with_source(PublishErrorKind::Other, api_error); + match classify_publish_error(&publish_error) { + TargetError::JetStreamPublish { retryable, .. } => assert!(retryable, "an uncoded 4xx rejection is retryable"), + other => panic!("expected JetStreamPublish, got {other:?}"), + } + } + + #[test] + fn classify_publish_error_terminal_code_with_server_error_status_is_retryable() { + // Deliberate policy: a 5xx status classifies retryable even when the error code is terminal. + let api_error: async_nats::jetstream::Error = + serde_json::from_value(serde_json::json!({"code": 503, "err_code": 10109, "description": "stream sealed"})) + .expect("api error deserializes"); + let publish_error = PublishError::with_source(PublishErrorKind::Other, api_error); + let classified = classify_publish_error(&publish_error); + match classified { + TargetError::JetStreamPublish { retryable, .. } => { + assert!(retryable, "a 5xx status keeps the STREAM_SEALED code on the retry path") + } + other => panic!("expected JetStreamPublish, got {other:?}"), + } + } + + #[test] + fn classify_publish_error_detail_omits_server_body() { + // A server rejection contributes its numeric code and status, never the server-returned description. + let api_error: async_nats::jetstream::Error = serde_json::from_value( + serde_json::json!({"code": 503, "err_code": 10008, "description": "secret-token-abc123 in description"}), + ) + .expect("api error deserializes"); + let classified = classify_publish_error(&PublishError::with_source(PublishErrorKind::Other, api_error)); + match classified { + TargetError::JetStreamPublish { detail, .. } => { + assert_eq!(detail, "server error code 10008 status 503"); + } + other => panic!("expected JetStreamPublish, got {other:?}"), + } + } + + #[test] + fn classified_publish_details_use_the_fixed_vocabulary() { + // Every classification path draws its detail from fixed labels and numeric fields. The pattern + // (lowercase letters, digits, spaces, underscores, colons) rejects raw error text. + let is_fixed = |detail: &str| { + !detail.is_empty() + && detail + .chars() + .all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || matches!(c, ' ' | '_' | ':')) + }; + + let mut details = vec![ATTEMPT_DEADLINE_DETAIL.to_string(), ACK_STREAM_MISMATCH_DETAIL.to_string()]; + for kind in [ + PublishErrorKind::StreamNotFound, + PublishErrorKind::TimedOut, + PublishErrorKind::BrokenPipe, + PublishErrorKind::MaxAckPending, + PublishErrorKind::MaxPayloadExceeded, + PublishErrorKind::WrongLastMessageId, + PublishErrorKind::WrongLastSequence, + PublishErrorKind::Other, + ] { + match classify_publish_error(&PublishError::new(kind)) { + TargetError::JetStreamPublish { detail, .. } => details.push(detail), + other => panic!("expected JetStreamPublish, got {other:?}"), + } + } + + let api_error: async_nats::jetstream::Error = + serde_json::from_value(serde_json::json!({"code": 503, "err_code": 10077, "description": "raw server text"})) + .expect("api error deserializes"); + match classify_publish_error(&PublishError::with_source(PublishErrorKind::Other, api_error)) { + TargetError::JetStreamPublish { detail, .. } => details.push(detail), + other => panic!("expected JetStreamPublish, got {other:?}"), + } + + // The validation gate detail must stay fixed even when the verdict error carries text outside the vocabulary. + let stream_validated = AtomicBool::new(false); + let verdict = Err(TargetError::Configuration("Raw Display TEXT with (punctuation)".to_string())); + match gate_publish_on_stream_validation(verdict, &stream_validated) { + Err(TargetError::JetStreamPublish { detail, .. }) => details.push(detail), + other => panic!("expected a JetStreamPublish gate error, got {other:?}"), + } + + for detail in details { + assert!(is_fixed(&detail), "detail falls outside the fixed vocabulary: {detail:?}"); + } + } +} diff --git a/crates/targets/src/target/nats/validation.rs b/crates/targets/src/target/nats/validation.rs new file mode 100644 index 000000000..1879798a6 --- /dev/null +++ b/crates/targets/src/target/nats/validation.rs @@ -0,0 +1,405 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use super::NATSArgs; +use super::jetstream::retry_lifetime; +use crate::error::TargetError; +use async_nats::jetstream; +use rustfs_config::{ + NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS, NATS_JETSTREAM_ACK_TIMEOUT_MAX_SECS, NATS_JETSTREAM_ACK_TIMEOUT_MIN_SECS, + NATS_JETSTREAM_ACK_TIMEOUT_SECS, NATS_JETSTREAM_ENABLE, NATS_JETSTREAM_STREAM_NAME, NATS_QUEUE_DIR, +}; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::time::Duration; +use tracing::{debug, error, info}; + +/// Validates the JetStream publish settings shared by the config-time and connect-time validators, +/// so both reject the same combinations. +pub(crate) fn validate_jetstream_settings( + enable: bool, + stream_name: &str, + queue_dir: &str, + ack_timeout_secs: Option, +) -> Result<(), TargetError> { + if let Some(secs) = ack_timeout_secs + && !(NATS_JETSTREAM_ACK_TIMEOUT_MIN_SECS..=NATS_JETSTREAM_ACK_TIMEOUT_MAX_SECS).contains(&secs) + { + return Err(TargetError::Configuration(format!( + "{NATS_JETSTREAM_ACK_TIMEOUT_SECS} must be between {NATS_JETSTREAM_ACK_TIMEOUT_MIN_SECS} and {NATS_JETSTREAM_ACK_TIMEOUT_MAX_SECS} seconds" + ))); + } + + if !enable { + return Ok(()); + } + + if stream_name.trim().is_empty() { + return Err(TargetError::Configuration(format!( + "{NATS_JETSTREAM_STREAM_NAME} is required when {NATS_JETSTREAM_ENABLE} is on" + ))); + } + + if queue_dir.trim().is_empty() { + return Err(TargetError::Configuration(format!( + "{NATS_QUEUE_DIR} is required when {NATS_JETSTREAM_ENABLE} is on" + ))); + } + + Ok(()) +} + +/// Detail carried by the retryable publish error raised when the stream-validation gate closes. The +/// full validation cause is logged at error level inside the validator. +pub(crate) const STREAM_VALIDATION_FAILED_DETAIL: &str = "stream validation failed"; + +/// Applies a stream-validation verdict on the publish path. A pass records the verdict so later +/// publishes skip re-validation. A failure returns a retryable JetStream publish error carrying the +/// fixed label and leaves the verdict unvalidated. +pub(crate) fn gate_publish_on_stream_validation( + verdict: Result<(), TargetError>, + stream_validated: &AtomicBool, +) -> Result<(), TargetError> { + match verdict { + Ok(()) => { + stream_validated.store(true, Ordering::Release); + Ok(()) + } + Err(_) => Err(TargetError::JetStreamPublish { + retryable: true, + detail: STREAM_VALIDATION_FAILED_DETAIL.to_string(), + }), + } +} + +/// Reports whether a publish subject is captured by a stream subject filter, honouring the NATS +/// wildcard tokens: a single-token wildcard matches one subject token, a trailing multi-token +/// wildcard matches one or more remaining tokens, any other token matches literally. +fn subject_filter_captures(filter: &str, subject: &str) -> bool { + let mut filter_tokens = filter.split('.'); + let mut subject_tokens = subject.split('.'); + + loop { + match (filter_tokens.next(), subject_tokens.next()) { + (Some(">"), Some(_)) => return true, + (Some("*"), Some(_)) => continue, + (Some(filter_token), Some(subject_token)) if filter_token == subject_token => continue, + (None, None) => return true, + _ => return false, + } + } +} + +/// Validates the configured JetStream stream from a single get_stream lookup when enabled. Asserts +/// the stream exists, captures the subject, returns acks, accepts writes, and keeps a duplicate +/// window that covers one retry cycle. The client-facing error is generic and every concrete cause +/// is logged server-side. The stream is never created, a missing stream is an error. +/// +/// When validation_logged is set, the first passing validation logs at info and later revalidations +/// of the same context log at debug, so a persistent wrong-stream loop does not flood the log. +pub(crate) async fn validate_jetstream_stream( + context: &jetstream::Context, + args: &NATSArgs, + target_label: &str, + validation_logged: Option<&AtomicBool>, +) -> Result<(), TargetError> { + if !args.jetstream_enable.unwrap_or(false) { + return Ok(()); + } + + let stream_name = args.jetstream_stream_name.as_deref().unwrap_or_default(); + if stream_name.is_empty() { + error!(target = %target_label, "JetStream enabled without a configured stream name"); + return Err(TargetError::Configuration("JetStream stream is not configured".to_string())); + } + + let stream = context.get_stream(stream_name).await.map_err(|err| { + error!( + target = %target_label, + stream = %stream_name, + error = %err, + "JetStream stream lookup failed, the stream must be pre-provisioned" + ); + TargetError::Configuration("configured JetStream stream is unavailable".to_string()) + })?; + + assert_stream_writable(&stream.cached_info().config, args, target_label)?; + // Info on a context's first passing validation, debug on later revalidations of the same context. + let first_pass = validation_logged + .map(|logged| !logged.swap(true, Ordering::AcqRel)) + .unwrap_or(true); + if first_pass { + info!( + target = %target_label, + stream = %stream_name, + "JetStream stream validation succeeded" + ); + } else { + debug!( + target = %target_label, + stream = %stream_name, + "JetStream stream revalidation succeeded" + ); + } + Ok(()) +} + +/// Asserts the retrieved stream configuration captures the subject, returns acks, accepts writes, and +/// keeps a duplicate window that covers one retry cycle. The client-facing error is generic, the +/// concrete cause logged server-side. +fn assert_stream_writable(config: &jetstream::stream::Config, args: &NATSArgs, target_label: &str) -> Result<(), TargetError> { + let stream_name = args.jetstream_stream_name.as_deref().unwrap_or_default(); + let subject = &args.subject; + let subject_bound = config.subjects.iter().any(|filter| subject_filter_captures(filter, subject)); + if !subject_bound { + error!( + target = %target_label, + stream = %stream_name, + subject = %subject, + "configured subject is not captured by the JetStream stream subjects" + ); + return Err(TargetError::Configuration( + "configured JetStream stream does not capture the subject".to_string(), + )); + } + + if config.no_ack { + error!( + target = %target_label, + stream = %stream_name, + "JetStream stream has no_ack set, publishes would never acknowledge" + ); + return Err(TargetError::Configuration( + "configured JetStream stream does not acknowledge writes".to_string(), + )); + } + + if config.sealed { + error!( + target = %target_label, + stream = %stream_name, + "JetStream stream is sealed, writes are rejected" + ); + return Err(TargetError::Configuration("configured JetStream stream is sealed".to_string())); + } + + let ack_timeout = Duration::from_secs( + args.jetstream_ack_timeout_secs + .unwrap_or(NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS), + ); + let required_window = retry_lifetime(ack_timeout); + if config.duplicate_window < required_window { + error!( + target = %target_label, + stream = %stream_name, + configured_window_secs = config.duplicate_window.as_secs(), + required_window_secs = required_window.as_secs(), + "JetStream stream duplicate_window is below the retry lifetime, a late retry would be delivered twice" + ); + return Err(TargetError::Configuration( + "configured JetStream stream duplicate window is too small for the retry lifetime".to_string(), + )); + } + + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::target::TargetType; + use crate::target::nats::test_support::*; + + #[test] + fn jetstream_settings_disabled_passes() { + validate_jetstream_settings(false, "", "", None).expect("disabled jetstream validates"); + validate_jetstream_settings(false, "", "", Some(30)).expect("disabled jetstream ignores stream and queue"); + } + + #[test] + fn jetstream_settings_ack_timeout_range() { + for secs in [ + NATS_JETSTREAM_ACK_TIMEOUT_MIN_SECS, + NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS, + NATS_JETSTREAM_ACK_TIMEOUT_MAX_SECS, + ] { + validate_jetstream_settings(false, "", "", Some(secs)).expect("in-range ack timeout accepted"); + } + for secs in [ + NATS_JETSTREAM_ACK_TIMEOUT_MIN_SECS - 1, + NATS_JETSTREAM_ACK_TIMEOUT_MAX_SECS + 1, + ] { + let err = validate_jetstream_settings(false, "", "", Some(secs)).expect_err("out-of-range ack timeout rejected"); + assert!(err.to_string().contains("between")); + } + } + + #[test] + fn jetstream_settings_requires_stream_name() { + let err = validate_jetstream_settings(true, " ", &nats_queue_dir(), Some(30)) + .expect_err("empty stream name rejected when enabled"); + assert!(err.to_string().contains(NATS_JETSTREAM_STREAM_NAME)); + } + + #[test] + fn jetstream_settings_requires_queue_dir() { + let err = + validate_jetstream_settings(true, "RUSTFS_EVENTS", "", Some(30)).expect_err("empty queue_dir rejected when enabled"); + assert!(err.to_string().contains(NATS_QUEUE_DIR)); + } + + #[test] + fn jetstream_settings_enabled_with_stream_and_queue_passes() { + validate_jetstream_settings(true, "RUSTFS_EVENTS", &nats_queue_dir(), Some(30)) + .expect("a stream name and queue dir accept enabling jetstream"); + } + + #[test] + fn natsargs_validate_rejects_enabled_jetstream_without_stream() { + let args = NATSArgs { + queue_dir: nats_queue_dir(), + jetstream_enable: Some(true), + jetstream_stream_name: None, + ..base_args() + }; + let err = args.validate().expect_err("enabled jetstream without a stream is rejected"); + assert!(err.to_string().contains(NATS_JETSTREAM_STREAM_NAME)); + } + + #[test] + fn subject_filter_matches_literal_and_wildcards() { + assert!(subject_filter_captures("rustfs.events", "rustfs.events")); + assert!(subject_filter_captures("rustfs.*", "rustfs.events")); + assert!(subject_filter_captures("rustfs.>", "rustfs.events.created")); + assert!(subject_filter_captures("rustfs.*.created", "rustfs.events.created")); + assert!(!subject_filter_captures("rustfs.events", "rustfs.audit")); + assert!(!subject_filter_captures("rustfs.*", "rustfs.events.created")); + assert!(!subject_filter_captures("rustfs.events", "rustfs.events.created")); + assert!(!subject_filter_captures("rustfs.events.created", "rustfs.events")); + } + + #[test] + fn assert_stream_writable_enforces_the_worst_case_window_at_the_default_ack_timeout() { + // The default 30s ack timeout requires a 274s window. One second below is rejected, exactly at it is accepted. + let args = jetstream_args(TargetType::NotifyEvent); + assert_eq!(retry_lifetime(Duration::from_secs(30)), Duration::from_secs(274)); + + let mut below = writable_stream_config(&args.subject); + below.duplicate_window = Duration::from_secs(273); + let err = assert_stream_writable(&below, &args, "test").expect_err("a window below the worst case is rejected"); + assert!(matches!(err, TargetError::Configuration(_))); + + let mut at = writable_stream_config(&args.subject); + at.duplicate_window = Duration::from_secs(274); + assert_stream_writable(&at, &args, "test").expect("a window at the worst case is accepted"); + } + + #[test] + fn assert_stream_writable_accepts_a_valid_stream() { + for target_type in [TargetType::NotifyEvent, TargetType::AuditLog] { + let args = jetstream_args(target_type); + let config = writable_stream_config(&args.subject); + assert_stream_writable(&config, &args, "test") + .unwrap_or_else(|err| panic!("a writable stream passes for {target_type:?}: {err}")); + } + } + + #[test] + fn assert_stream_writable_passes_when_subject_is_bound_by_wildcard() { + let args = jetstream_args(TargetType::NotifyEvent); + let mut config = writable_stream_config(&args.subject); + config.subjects = vec!["rustfs.>".to_string()]; + // The configured subject rustfs.events is captured by the rustfs.> filter. + assert_stream_writable(&config, &args, "test").expect("a wildcard-bound subject passes"); + } + + #[test] + fn assert_stream_writable_rejects_a_subject_not_bound() { + let args = jetstream_args(TargetType::NotifyEvent); + let mut config = writable_stream_config(&args.subject); + config.subjects = vec!["some.other.subject".to_string()]; + let err = assert_stream_writable(&config, &args, "test").expect_err("an unbound subject is rejected"); + assert!(matches!(err, TargetError::Configuration(_)), "the error is a generic configuration error"); + assert!( + !err.to_string().contains("some.other.subject"), + "the client message omits the stream subjects" + ); + } + + #[test] + fn assert_stream_writable_rejects_no_ack() { + let args = jetstream_args(TargetType::NotifyEvent); + let mut config = writable_stream_config(&args.subject); + config.no_ack = true; + let err = assert_stream_writable(&config, &args, "test").expect_err("a no_ack stream is rejected"); + assert!(matches!(err, TargetError::Configuration(_))); + } + + #[test] + fn assert_stream_writable_rejects_sealed() { + let args = jetstream_args(TargetType::NotifyEvent); + let mut config = writable_stream_config(&args.subject); + config.sealed = true; + let err = assert_stream_writable(&config, &args, "test").expect_err("a sealed stream is rejected"); + assert!(matches!(err, TargetError::Configuration(_))); + } + + #[test] + fn assert_stream_writable_rejects_a_too_small_duplicate_window() { + let args = jetstream_args(TargetType::NotifyEvent); + let mut config = writable_stream_config(&args.subject); + // Below the worst-case retry lifetime at the default ack timeout, so a late retry would deliver twice. + config.duplicate_window = Duration::from_secs(90); + let err = assert_stream_writable(&config, &args, "test").expect_err("a too-small window is rejected"); + assert!(matches!(err, TargetError::Configuration(_))); + } + + #[test] + fn assert_stream_writable_rejects_a_zero_duplicate_window() { + let args = jetstream_args(TargetType::NotifyEvent); + let mut config = writable_stream_config(&args.subject); + // The window defaults to zero (dedup disabled) when the operator leaves it unset. + config.duplicate_window = Duration::ZERO; + let err = assert_stream_writable(&config, &args, "test").expect_err("a zero window is rejected"); + assert!(matches!(err, TargetError::Configuration(_))); + } + + #[test] + fn stream_validation_gate_blocks_the_publish_and_classifies_retryable() { + // A failed validation returns before any publish call, classified retryable so the entry is never lost. + let stream_validated = AtomicBool::new(false); + let verdict = Err(TargetError::Configuration("configured JetStream stream is unavailable".to_string())); + let err = gate_publish_on_stream_validation(verdict, &stream_validated).expect_err("a failed validation closes the gate"); + match err { + TargetError::JetStreamPublish { retryable, detail } => { + assert!(retryable, "a validation failure is retryable"); + assert_eq!(detail, STREAM_VALIDATION_FAILED_DETAIL, "the fixed label is the detail"); + } + other => panic!("expected a retryable JetStreamPublish, got {other:?}"), + } + assert!( + !stream_validated.load(Ordering::SeqCst), + "the verdict stays closed so the next attempt re-validates" + ); + } + + #[test] + fn stream_validation_gate_pass_is_recorded_with_the_context() { + let stream_validated = AtomicBool::new(false); + gate_publish_on_stream_validation(Ok(()), &stream_validated).expect("a passing validation opens the gate"); + assert!( + stream_validated.load(Ordering::SeqCst), + "the recorded verdict lets later publishes skip re-validation" + ); + } +} diff --git a/crates/targets/src/target/postgres.rs b/crates/targets/src/target/postgres.rs index fd4dcf6f0..6469c20a0 100644 --- a/crates/targets/src/target/postgres.rs +++ b/crates/targets/src/target/postgres.rs @@ -904,8 +904,11 @@ where } fn delivery_snapshot(&self) -> TargetDeliverySnapshot { - self.delivery_counters - .snapshot(self.store.as_deref().map_or(0, |store| store.len() as u64)) + self.delivery_counters.snapshot( + self.store.as_deref().map_or(0, |store| store.len() as u64), + // Postgres targets record no terminal failures and keep no failed store. + 0, + ) } fn record_final_failure(&self) { diff --git a/crates/targets/src/target/pulsar.rs b/crates/targets/src/target/pulsar.rs index a12604691..15c8122f3 100644 --- a/crates/targets/src/target/pulsar.rs +++ b/crates/targets/src/target/pulsar.rs @@ -481,8 +481,11 @@ where } fn delivery_snapshot(&self) -> TargetDeliverySnapshot { - self.delivery_counters - .snapshot(self.store.as_deref().map_or(0, |store| store.len() as u64)) + self.delivery_counters.snapshot( + self.store.as_deref().map_or(0, |store| store.len() as u64), + // Pulsar targets record no terminal failures and keep no failed store. + 0, + ) } fn record_final_failure(&self) { diff --git a/crates/targets/src/target/redis.rs b/crates/targets/src/target/redis.rs index b62470379..8d30823cc 100644 --- a/crates/targets/src/target/redis.rs +++ b/crates/targets/src/target/redis.rs @@ -658,8 +658,11 @@ where } fn delivery_snapshot(&self) -> TargetDeliverySnapshot { - self.delivery_counters - .snapshot(self.store.as_deref().map_or(0, |store| store.len() as u64)) + self.delivery_counters.snapshot( + self.store.as_deref().map_or(0, |store| store.len() as u64), + // Redis targets record no terminal failures and keep no failed store. + 0, + ) } fn record_final_failure(&self) { diff --git a/crates/targets/src/target/webhook.rs b/crates/targets/src/target/webhook.rs index a02e9e88b..cfd29866b 100644 --- a/crates/targets/src/target/webhook.rs +++ b/crates/targets/src/target/webhook.rs @@ -644,8 +644,11 @@ where } fn delivery_snapshot(&self) -> TargetDeliverySnapshot { - self.delivery_counters - .snapshot(self.store.as_deref().map_or(0, |store| store.len() as u64)) + self.delivery_counters.snapshot( + self.store.as_deref().map_or(0, |store| store.len() as u64), + // Webhook targets record no terminal failures and keep no failed store. + 0, + ) } fn record_final_failure(&self) { diff --git a/crates/targets/tests/nats_jetstream_regression_guards.rs b/crates/targets/tests/nats_jetstream_regression_guards.rs new file mode 100644 index 000000000..016b107cf --- /dev/null +++ b/crates/targets/tests/nats_jetstream_regression_guards.rs @@ -0,0 +1,259 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Regression guards for the NATS JetStream publish feature. +//! +//! The JetStream surface can be removed and the build still succeed unless something fails when it +//! disappears. These layered guards make a silent removal or a silent classification drift a build or +//! test failure: +//! +//! - a compile-time reference to the JetStream config surface and the target entry point, +//! - a module-presence test naming the public entry type and the publish-error variant, +//! - completeness tests over the publish-error-class enums with no wildcard arm, +//! - a server-backed end-to-end publish-and-ack test, ignored by default so the offline gate stays +//! green without a NATS server. +//! +//! The end-to-end guard requires a running NATS server with JetStream enabled. To run it locally: +//! +//! docker run -d --name rustfs-nats-test -p 4222:4222 nats:2 -js +//! cargo test -p rustfs-targets --test nats_jetstream_regression_guards -- --ignored +//! +//! Override the server URL with RUSTFS_TEST_NATS_URL. + +use async_nats::jetstream::context::PublishErrorKind; +use rustfs_targets::check_nats_server_available; +use rustfs_targets::error::TargetError; +use rustfs_targets::target::nats::{NATSArgs, NATSTarget}; +use rustfs_targets::target::{FailedErrorClass, TargetType}; + +// Compile-time module-presence assertion. +// +// A const initializer names the JetStream config fields on the public target args type and the public +// target constructor by path, referencing the JetStream wiring at compile time without running +// anything. Removing the jetstream_enable, jetstream_stream_name, or jetstream_ack_timeout_secs fields +// from NATSArgs, or removing the NATSTarget constructor, while the config flag keys remain, fails to +// compile here. A static reference rather than a runtime check, so it fires at build time even when no +// test is run. +const _: fn(&NATSArgs) = |args: &NATSArgs| { + // Each field is read by name, so removing any of the three from NATSArgs fails to compile here. + let _enable: Option = args.jetstream_enable; + let _stream_name: &Option = &args.jetstream_stream_name; + let _ack_timeout: Option = args.jetstream_ack_timeout_secs; + + // The publish target entry point. Naming it by path ties this assertion to the constructor that + // wires the JetStream context, so deleting the entry point breaks the build. + let _entry: fn(String, NATSArgs) -> Result, TargetError> = NATSTarget::::new; +}; + +/// Module-presence test naming the public JetStream entry type, the health-check entry, and the +/// publish-error variant by path. Deleting the NATSTarget constructor, the check_nats_server_available +/// entry, or the JetStreamPublish error variant fails to compile this test. +#[test] +fn jetstream_public_entry_points_are_present() { + let _entry: fn(String, NATSArgs) -> Result, TargetError> = NATSTarget::::new; + + let _health_check = check_nats_server_available; + + // The publish path classifies a publish failure into this variant. Constructing it by name binds + // the guard to the variant, so removing it from TargetError breaks the build. + let publish_error = TargetError::JetStreamPublish { + retryable: true, + detail: "guard".to_string(), + }; + match publish_error { + TargetError::JetStreamPublish { retryable, detail } => { + assert!(retryable); + assert_eq!(detail, "guard"); + } + other => panic!("expected the JetStream publish variant, found {other:?}"), + } +} + +/// Cross-module completeness over the publish-error class. +/// +/// Lists every async-nats PublishErrorKind variant by name with no wildcard arm. The publish path +/// classifies each kind into retryable or terminal. A new variant added upstream, or a renamed or +/// removed one, fails to compile this match, forcing the classification to be revisited rather than +/// silently folding the new kind into a catch-all. +#[test] +fn publish_error_kind_is_exhaustively_enumerated() { + fn name_of(kind: PublishErrorKind) -> &'static str { + match kind { + PublishErrorKind::StreamNotFound => "StreamNotFound", + PublishErrorKind::WrongLastMessageId => "WrongLastMessageId", + PublishErrorKind::WrongLastSequence => "WrongLastSequence", + PublishErrorKind::TimedOut => "TimedOut", + PublishErrorKind::BrokenPipe => "BrokenPipe", + PublishErrorKind::MaxAckPending => "MaxAckPending", + PublishErrorKind::MaxPayloadExceeded => "MaxPayloadExceeded", + PublishErrorKind::Other => "Other", + } + } + + let all = [ + PublishErrorKind::StreamNotFound, + PublishErrorKind::WrongLastMessageId, + PublishErrorKind::WrongLastSequence, + PublishErrorKind::TimedOut, + PublishErrorKind::BrokenPipe, + PublishErrorKind::MaxAckPending, + PublishErrorKind::MaxPayloadExceeded, + PublishErrorKind::Other, + ]; + assert_eq!( + all.len(), + 8, + "the publish-error class has eight kinds, update the classification on a change" + ); + for kind in all { + assert!(!name_of(kind).is_empty()); + } +} + +/// Cross-module completeness over the failed-store error class. +/// +/// Lists every FailedErrorClass variant by name with no wildcard arm. The replay worker tags a failed +/// entry with one of these classes. Adding or removing a class fails to compile this match, so the +/// tagging and the operator-facing tag stay in step. +#[test] +fn failed_error_class_is_exhaustively_enumerated() { + fn tag_of(class: FailedErrorClass) -> &'static str { + match class { + FailedErrorClass::Terminal => "terminal", + } + } + + let all = [FailedErrorClass::Terminal]; + assert_eq!( + all.len(), + 1, + "the failed-store error class has one variant, update the tagging on a change" + ); + for class in all { + assert_eq!(tag_of(class), class.as_str(), "the guard tag matches the operator-facing tag"); + } +} + +fn server_url() -> String { + std::env::var("RUSTFS_TEST_NATS_URL").unwrap_or_else(|_| "nats://127.0.0.1:4222".to_string()) +} + +fn jetstream_args(subject: &str, stream_name: &str, queue_dir: &str) -> NATSArgs { + NATSArgs { + enable: true, + address: server_url(), + subject: subject.to_string(), + username: String::new(), + password: String::new(), + token: String::new(), + credentials_file: String::new(), + tls_ca: String::new(), + tls_client_cert: String::new(), + tls_client_key: String::new(), + tls_required: false, + queue_dir: queue_dir.to_string(), + queue_limit: 0, + jetstream_enable: Some(true), + jetstream_stream_name: Some(stream_name.to_string()), + jetstream_ack_timeout_secs: Some(30), + target_type: TargetType::NotifyEvent, + } +} + +/// Server-backed end-to-end publish-and-ack guard. +/// +/// Provisions a writable stream as an operator would, then drives the real target publish path with +/// the JetStream flag on: init connects and validates the stream, save persists the event to the +/// on-disk queue, send_from_store publishes and awaits the real PublishAck and clears the entry on it. +/// The stream message count rising and the queue draining proves the message was published and +/// acknowledged through the production code, the user-visible signal a build-only check cannot see. +/// +/// Ignored by default because it needs a running NATS server with JetStream. +#[tokio::test] +#[ignore] +async fn end_to_end_publish_is_acked_on_the_stream() { + use rustfs_targets::EventName; + use rustfs_targets::Target; + use rustfs_targets::target::EntityTarget; + use std::sync::Arc; + use uuid::Uuid; + + let suffix = Uuid::new_v4().simple().to_string(); + let stream_name = format!("RUSTFS_TEST_{suffix}"); + let subject = format!("rustfs.guard.{suffix}"); + let queue_dir = std::env::temp_dir().join(format!("rustfs-nats-guard-{suffix}")); + let queue_dir = queue_dir.to_string_lossy().to_string(); + + // Provision the stream as the operator would. RustFS validates it, never creates it. + let client = async_nats::connect(server_url()).await.expect("connect to NATS"); + let context = async_nats::jetstream::new(client); + let _ = context.delete_stream(&stream_name).await; + context + .create_stream(async_nats::jetstream::stream::Config { + name: stream_name.clone(), + subjects: vec![subject.clone()], + duplicate_window: std::time::Duration::from_secs(300), + ..Default::default() + }) + .await + .expect("provision the stream"); + + let args = jetstream_args(&subject, &stream_name, &queue_dir); + let target = NATSTarget::::new("guard-target".to_string(), args).expect("build the flag-on target"); + + target + .init() + .await + .expect("init connects and validates the operator-provisioned stream"); + + let count_before = stream_message_count(&context, &stream_name).await; + + let event = EntityTarget { + object_name: "guard/object.txt".to_string(), + bucket_name: "guard-bucket".to_string(), + event_name: EventName::ObjectCreatedPut, + data: "guard-payload".to_string(), + }; + target + .save(Arc::new(event)) + .await + .expect("save persists the event to the queue"); + + let keys = target.store().expect("the flag-on target has a queue store").list(); + assert_eq!(keys.len(), 1, "the saved event is queued exactly once"); + target + .send_from_store(keys[0].clone()) + .await + .expect("the queued event publishes and is acked"); + + let count_after = stream_message_count(&context, &stream_name).await; + assert_eq!( + count_after, + count_before + 1, + "the published event is durably on the stream after its ack" + ); + + let remaining = target.store().expect("the flag-on target has a queue store").list(); + assert!(remaining.is_empty(), "the queue entry is cleared on the ack"); + + target.close().await.expect("close drains the target"); + let _ = context.delete_stream(&stream_name).await; + let _ = std::fs::remove_dir_all(&queue_dir); +} + +async fn stream_message_count(context: &async_nats::jetstream::Context, stream_name: &str) -> u64 { + let mut stream = context.get_stream(stream_name).await.expect("look up the stream"); + let info = stream.info().await.expect("read the stream info"); + info.state.messages +} diff --git a/crates/targets/tests/nats_jetstream_validation_integration.rs b/crates/targets/tests/nats_jetstream_validation_integration.rs new file mode 100644 index 000000000..3981b5e53 --- /dev/null +++ b/crates/targets/tests/nats_jetstream_validation_integration.rs @@ -0,0 +1,143 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! JetStream stream validation integration tests for the NATS target. +//! +//! These tests are ignored because they require a running NATS server with JetStream enabled. To +//! run locally: +//! +//! docker run -d --name rustfs-nats-test -p 4222:4222 nats:2 -js +//! cargo test -p rustfs-targets --test nats_jetstream_validation_integration -- --ignored +//! +//! Override the server URL with RUSTFS_TEST_NATS_URL. Each test provisions and removes its own +//! stream. RustFS never provisions the stream, the test harness does, mirroring the operator role. + +use async_nats::jetstream::stream::Config; +use rustfs_targets::check_nats_server_available; +use rustfs_targets::target::TargetType; +use rustfs_targets::target::nats::NATSArgs; +use std::time::Duration; +use uuid::Uuid; + +fn server_url() -> String { + std::env::var("RUSTFS_TEST_NATS_URL").unwrap_or_else(|_| "nats://127.0.0.1:4222".to_string()) +} + +fn jetstream_args(subject: &str, stream_name: &str) -> NATSArgs { + NATSArgs { + enable: true, + address: server_url(), + subject: subject.to_string(), + username: String::new(), + password: String::new(), + token: String::new(), + credentials_file: String::new(), + tls_ca: String::new(), + tls_client_cert: String::new(), + tls_client_key: String::new(), + tls_required: false, + queue_dir: "/tmp/rustfs-nats-jetstream-test".to_string(), + queue_limit: 0, + jetstream_enable: Some(true), + jetstream_stream_name: Some(stream_name.to_string()), + jetstream_ack_timeout_secs: Some(30), + target_type: TargetType::NotifyEvent, + } +} + +/// A pre-provisioned writable stream: it captures the subject, returns acks, accepts writes, and its +/// duplicate window exceeds the retry lifetime. The test harness creates it, never RustFS. +async fn provision_writable_stream(stream_name: &str, subject: &str) { + let client = async_nats::connect(server_url()).await.expect("connect to NATS"); + let context = async_nats::jetstream::new(client); + // Clear any leftover from a prior aborted run. Absent is fine, so the result is dropped. + let _ = context.delete_stream(stream_name).await; + context + .create_stream(Config { + name: stream_name.to_string(), + subjects: vec![subject.to_string()], + duplicate_window: Duration::from_secs(300), + ..Default::default() + }) + .await + .expect("provision the stream"); +} + +async fn remove_stream(stream_name: &str) { + let client = async_nats::connect(server_url()).await.expect("connect to NATS"); + let context = async_nats::jetstream::new(client); + // Best-effort teardown. A failure to delete does not fail the test, so the result is dropped. + let _ = context.delete_stream(stream_name).await; +} + +#[tokio::test] +#[ignore] +async fn missing_stream_fails_the_health_check() { + let stream_name = format!("RUSTFS_TEST_{}", Uuid::new_v4().simple()); + let args = jetstream_args("rustfs.events", &stream_name); + // No stream is provisioned, so the lookup must fail. + let err = check_nats_server_available(&args) + .await + .expect_err("a missing stream fails the health check"); + assert!(!err.to_string().is_empty(), "the failure carries a generic message"); +} + +#[tokio::test] +#[ignore] +async fn valid_stream_passes_the_health_check() { + let stream_name = format!("RUSTFS_TEST_{}", Uuid::new_v4().simple()); + let subject = "rustfs.events"; + provision_writable_stream(&stream_name, subject).await; + let args = jetstream_args(subject, &stream_name); + let result = check_nats_server_available(&args).await; + remove_stream(&stream_name).await; + result.expect("a pre-provisioned writable stream passes the health check"); +} + +#[tokio::test] +#[ignore] +async fn stream_not_capturing_the_subject_fails_the_health_check() { + let stream_name = format!("RUSTFS_TEST_{}", Uuid::new_v4().simple()); + // The stream binds a different subject than the target publishes to. + provision_writable_stream(&stream_name, "some.other.subject").await; + let args = jetstream_args("rustfs.events", &stream_name); + let result = check_nats_server_available(&args).await; + remove_stream(&stream_name).await; + result.expect_err("a stream that does not capture the subject fails the health check"); +} + +#[tokio::test] +#[ignore] +async fn too_small_duplicate_window_fails_the_health_check() { + let stream_name = format!("RUSTFS_TEST_{}", Uuid::new_v4().simple()); + let subject = "rustfs.events"; + let client = async_nats::connect(server_url()).await.expect("connect to NATS"); + let context = async_nats::jetstream::new(client); + // Clear any leftover from a prior aborted run. Absent is fine, so the result is dropped. + let _ = context.delete_stream(&stream_name).await; + context + .create_stream(Config { + name: stream_name.clone(), + subjects: vec![subject.to_string()], + // Below the worst-case retry lifetime at the default ack timeout. + duplicate_window: Duration::from_secs(60), + ..Default::default() + }) + .await + .expect("provision the stream"); + let args = jetstream_args(subject, &stream_name); + let result = check_nats_server_available(&args).await; + remove_stream(&stream_name).await; + result.expect_err("a too-small duplicate window fails the health check"); +} diff --git a/docs/operations/nats-jetstream.md b/docs/operations/nats-jetstream.md new file mode 100644 index 000000000..1116a031e --- /dev/null +++ b/docs/operations/nats-jetstream.md @@ -0,0 +1,356 @@ +# NATS JetStream Operations Guide + +The guide covers enabling and operating the JetStream publish path for the NATS +notify and audit targets in RustFS. It is written for operators who need +at-least-once delivery of events to NATS, must pre-provision and size the stream +the targets publish to, and need to diagnose delivery failures. + +## What the JetStream Path Does + +By default the NATS notify and audit targets publish with NATS Core, which +returns once the message is written to the socket. A process or server failure +in the gap between the socket write and the server persisting the message loses +the event. The JetStream path closes that gap. With it enabled, each event is +published to a JetStream stream and the local store-and-forward queue entry +clears only after the server returns a publish acknowledgement, which means the +stream leader has accepted and sequenced the message, or after a terminal +rejection has been recorded in the failed-events store. A server restart +mid-flight loses nothing, because an unacknowledged event stays in the local +queue and replays after the server returns. + +The path is opt-in and off by default. With it off, behaviour is the NATS Core +path. It applies to both the notify NATS target and the audit NATS target. + +RustFS gets each event to the server without losing it before the +acknowledgement. The operator owns the stream and its policy. RustFS validates +that the stream exists and is writable and reports a validation failure +otherwise. A store-backed target keeps running through a failed validation, +holding queued events until the stream is repaired. RustFS never creates the +stream. Stream retention, storage, and replication policy stay under operator +control. + +## Enabling JetStream: Recommended Configuration + +Three keys turn the path on and tune it. Each is available on both the notify +NATS target and the audit NATS target, and each has a configuration-key form and +an environment-variable form per target. + +```bash +RUSTFS_NOTIFY_NATS_JETSTREAM_ENABLE=true +RUSTFS_NOTIFY_NATS_JETSTREAM_STREAM_NAME=RUSTFS_EVENTS +RUSTFS_NOTIFY_NATS_JETSTREAM_ACK_TIMEOUT_SECS=30 +RUSTFS_NOTIFY_NATS_QUEUE_DIR=/var/lib/rustfs/notify-nats +``` + +The audit target takes the same keys under the RUSTFS_AUDIT_NATS_ prefix. + +- jetstream_enable turns the JetStream path on for the target. Off by default. + Environment forms RUSTFS_NOTIFY_NATS_JETSTREAM_ENABLE and + RUSTFS_AUDIT_NATS_JETSTREAM_ENABLE. +- jetstream_stream_name is the name of the pre-provisioned stream to publish to. + Required when enable is on, with no default. Environment forms + RUSTFS_NOTIFY_NATS_JETSTREAM_STREAM_NAME and + RUSTFS_AUDIT_NATS_JETSTREAM_STREAM_NAME. +- jetstream_ack_timeout_secs is how long a publish waits for an acknowledgement + before it is treated as timed out and retried. Range 10 to 120, default 30. + Environment forms RUSTFS_NOTIFY_NATS_JETSTREAM_ACK_TIMEOUT_SECS and + RUSTFS_AUDIT_NATS_JETSTREAM_ACK_TIMEOUT_SECS. +- queue_dir is the local store-and-forward queue directory and is required when + enable is on. Durability is not achievable without a local store to replay + from. Environment forms RUSTFS_NOTIFY_NATS_QUEUE_DIR and + RUSTFS_AUDIT_NATS_QUEUE_DIR. + +A configuration that enables the path without a stream name or without a queue +directory is rejected at startup and in the admin validation path. An +acknowledgement timeout outside the 10 to 120 second range is rejected the same +way. + +The 30 second acknowledgement timeout suits production. A server replicating to +multiple replicas acknowledges only after the replication and the deferred fsync, +which legitimately takes well over 100 milliseconds at the tail, so a short +timeout would spuriously fail a publish mid-replication. + +After enabling, confirm the target is on the JetStream path before relying on it. +Value errors, an out-of-range acknowledgement timeout or a missing stream name +while enable is on, are already rejected loudly at startup and in the admin +validation path. A mistyped configuration key name is not a value error, so it +escapes that check: the key reads as absent, and the target stays on the NATS +Core path with no durability. Verify the enable took effect by confirming the +target reports its JetStream fields on startup and logs the stream-validation +success line naming the configured stream. Absence of that line for a target that +was meant to be enabled indicates a key name the server did not recognise. + +## How Delivery Works + +An event is queued to the local store-and-forward queue first, then published +through JetStream. The publish carries a stable Nats-Msg-Id header and the path +awaits the real publish acknowledgement. + +- The queue entry clears only after the durable acknowledgement returns, or + after a terminal rejection has been recorded in the failed-events store. A + timeout, or any retryable error, retains the entry for a retry. The entry is + never cleared without one of those two outcomes and never silently dropped. A + process interruption between the failed-record write and the live-entry removal + can leave a record whose event a later replay still delivers, a diagnostic + residue and not a lost event. +- The Nats-Msg-Id is minted once when the event is queued and stored with it. It + is identical across every retry and replay of that entry, and unique per + entry, so the server collapses retries and replays of the same event within + the stream duplicate window. A retry after a slow acknowledgement reuses the + same identifier and is not delivered twice. +- A crash before the acknowledgement leaves the entry in the queue. The entry + replays after restart. Closing the target releases the cached connection and + context but never the queued entries, so an entry queued before close survives + on disk and replays on the next start. +- A broker outage detected at connection establishment keeps queued events on + disk. Replay retries with backoff while the broker is unreachable, and the + queued events deliver when the connection recovers, keeping entries queued like + the NATS Core path rather than replicating its mechanics byte for byte. Every + retryable publish failure on an established connection is treated the same way, + kept on the live queue and retried until it delivers, which includes a + connection dropping during the stream-validation lookup. Only a non-retryable + rejection is moved to the failed-events store. + +For durability across a node failure, provision the stream with a replica count +of at least 3. An acknowledgement returns after the stream leader commits. A +leader that acknowledges and then fails before the message replicates to a quorum +can lose that message on failover, so a single-replica stream is durable only +against a clean restart, not against the loss of the node holding the data. + +## Pre-provisioning the Stream + +The stream is provisioned by the operator before the path is enabled. RustFS +reads the stream once at target init and in the admin validation path, asserts +the requirements below, and reports a validation failure rather than publishing +into a stream where writes would silently fail. A failed validation does not +stop a store-backed target. Events keep queueing to the local store. Revalidation +runs while the validation verdict is unset and stops once one validation passes, +resuming only after the verdict is reset, and the queued events deliver once the +stream is repaired. The stream is not auto-created. A missing stream is an +error. + +The stream must: + +- Exist and be writable. A missing or unreachable stream fails validation. +- Capture the configured publish subject in its subject filter, by literal match + or by a NATS wildcard. A stream that does not capture the subject fails + validation. +- Acknowledge writes (no_ack false). A stream with no_ack set never returns an + acknowledgement, so the queue would never clear. It fails validation. +- Not be sealed. A sealed stream rejects writes and fails validation. +- Set a duplicate window of at least the retry span (see the next section). A + window below that span fails validation. + +Choose retention, storage type, and replica count to match the durability the +deployment needs. Use file storage if a server restart must preserve +already-persisted events. These are operator decisions and RustFS does not set +them. If the stream is provisioned shortly after the path is enabled, the queued +events deliver on the next replay rather than being lost, because a +stream-not-found error is retryable and keeps the events on the live queue until +the stream exists. + +## Setting the Duplicate Window + +The acknowledgement timeout and the stream duplicate window are linked. The +duplicate window must cover the worst-case retry span, or a late retry of an +event the server already persisted is delivered a second time instead of being +recognised as a duplicate. + +The worst-case retry span is the retry count times the acknowledgement timeout, +plus the realized backoff sleeps across the retries and a final headroom term: + + duplicate_window >= (retry_count * ack_timeout_secs) + realized_backoff + headroom + +There are 5 retry attempts. The realized backoff sleeps sum to 60 seconds +(4 + 8 + 16 + 32), and no sleep follows the last attempt, so a 64 second headroom +term is added above the realized span. At the default 30 second acknowledgement +timeout the required window is: + + (5 * 30) + 60 + 64 = 274 seconds + +Set the stream duplicate window to at least 274 seconds when the acknowledgement +timeout is at the 30 second default. A configuration whose duplicate window is +below this span is rejected at validation, with the configured and required +window named in the log line. + +The span scales with the acknowledgement timeout. Raising +jetstream_ack_timeout_secs requires raising the stream duplicate window in step +using the formula above. The 60 second realized backoff and the 64 second headroom +stay fixed, so each extra second of acknowledgement timeout adds 5 seconds to the +required window. At the 120 second maximum timeout the required window is +(5 * 120) + 60 + 64 = 724 seconds. Each publish attempt, including connection +establishment, is bounded by a single deadline equal to the acknowledgement +timeout. The formula is a conservative upper bound on the retry span, and the +final headroom term holds the required window above the realized span. + +The validated window covers one retry cycle: the worst-case span from the first +publish attempt of a stored entry to its last within a single replay cycle. An +entry that exhausts a cycle without delivering stays on the live queue and is +retried on a later cycle, so an entry surviving across cycles can spend longer in +the queue than the duplicate window and be delivered again, consistent with +at-least-once delivery. + +## The Failed-events Store + +An event that cannot be delivered is recorded, not silently dropped. Only one +case produces a failed event: + +- A terminal rejection. A message that exceeds the maximum payload size, a wrong + expected last message identifier or last sequence, or a sealed stream. These do + not improve with a retry, so they fail fast. + +Every other rejection is retryable and never produces a failed event. A timeout, +backpressure, no responders, a missing stream, a subject the stream does not +capture, or an authorization failure keeps the event on the live queue and +retries it until it delivers or an operator intervenes. A wrong subject and wrong +credentials surface through the validation and connection path as retryable, so +they hold the events on the live queue rather than moving them to the failed +store. + +A failed event is moved to an on-disk failed store that sits in a failed +child directory inside the queue directory, one per target. The directory is created +lazily on the first failed write, so a target that never fails terminally leaves +no failed directory. The entry preserves the event body, its routing metadata, the +deduplication identifier, an error class tag of terminal, the failure time, and +the retry count. An error-level log line is written for each failed event, naming +the bucket, object, event name, and the error, so a broken integration is visible +rather than hidden. A record in the failed store is diagnostic only and is never +republished, so a condition repaired later, for example a raised broker payload +limit, delivers only events still on the live queue. The NATS Core path without +JetStream instead retries such an event until the limit allows it. + +The failed store is bounded by count (10000 entries per target) and by age (a 72 +hour retention), and is kept separate from the live queue limit so an +accumulation of failures cannot crowd out new events. The count is maintained as a +cached value, seeded at startup and reconciled to the directory on each +maintenance interval, so it stays accurate without a directory scan on the hot +path. Failed-store writes and the maintenance scan run under one exclusive guard, +so the at-bound check and the write stay atomic and the bound holds against +concurrent writers. A change made to the directory outside the store can drift the +cached count until the next maintenance interval reconciles it. When the count +bound is reached the oldest failed entry is dropped, with a warning naming the +trimmed entry, so a newer failure is never lost in favour of an older one. Entries +past the retention bound are removed as expired on the replay maintenance tick. + +Because a retryable failure keeps events on the live queue, a long outage grows +the queue toward its configured queue_limit bound. Once the queue reaches that +bound, new events are rejected at ingest with a logged error rather than +overwriting queued events, so the backlog is bounded and visible. + +Size queue_limit for the longest outage the deployment must survive without +rejecting new events. Multiply the peak event rate in events per second by the +outage window in seconds. A target that averages 50 events per second and must +ride out a one hour broker outage needs a queue_limit of at least 50 * 3600 = +180000 entries. Add headroom above the calculated figure, and provision the +queue directory storage for the resulting entry count. + +## Publish Outcome Handling + +Every publish outcome falls into one of two families. Transient conditions are +retried until they deliver and the entry stays on the live queue, so a transient +condition never reaches the failed store. Permanent conditions move to the failed +store immediately. Transient conditions classify toward retry because a terminal +misclassification risks losing an entry once the failed store retention lapses. +Permanent conditions move immediately so a poison message cannot block the queue. + +Both families cover publish outcomes on an established connection. A failure to +establish the connection at all, a refused connection or unreadable TLS +material, is handled before either family applies: the entry retries with +backoff and stays queued until the connection recovers. A publish-level failure +on an established connection, including a connection that drops during the +stream-validation lookup, is retried on the live queue when it is transient and +moved to the failed store only when it is a permanent rejection. + +| Outcome family | Examples | Handling | +| --- | --- | --- | +| Connectivity | connection lost mid-publish, broken pipe | Retried until delivered, live queue | +| Timeouts | no acknowledgement within the timeout, attempt deadline reached | Retried until delivered, live queue | +| Cluster in transition | no leader elected, peer membership changing | Retried until delivered, live queue | +| Stream offline | stream or JetStream subsystem temporarily offline, stream not found | Retried until delivered, live queue | +| Resource and quota exhaustion | insufficient server resources, storage, memory, or account quota reached | Retried until delivered, live queue | +| Server errors | any rejection reporting a 5xx status, with or without a specific error code | Retried until delivered, live queue | +| Permanent rejections | payload too large, wrong expected sequence or message identifier, sealed stream | Failed store immediately | + +A sealed stream is classified at three points, with three outcomes. Startup validation and the admin +validation path reject a sealed stream before the target serves traffic, so init fails and no event +is queued against it. A stream sealed while the target runs is caught by the next validation on the +publish path, which classifies it retryable and keeps the entry on the live queue while validation +keeps failing. A publish that reaches an already-cached validation pass and is then rejected with the +sealed-stream code is terminal and moves to the failed store immediately. The permanent-rejections +row lists that last outcome. + +## Observability + +The failed_store_length gauge reports the number of entries in the failed-events +store per target, next to the existing failed_messages and queue_length gauges, +on both the notify and audit metric paths. A rising failed_store_length points to +terminal rejections accumulating for a target. An exhaustion warning marks each +cycle where an entry spends its full retry budget without delivering, after which +the entry stays queued and retries on the next scan. The warning repeats once per +retry cycle while the entry stays queued, so a persistent delivery problem is +visible at warn level without per-attempt noise. The failed_messages count +advances only on terminal and dropped events, never on a retry or an exhaustion, +so it counts entries that left the queue for good rather than entries still +retrying. + +## Troubleshooting + +- Validation fails at startup with a missing-stream error. The stream named in + jetstream_stream_name does not exist on the server, or is not reachable. + Provision it, or correct the name. +- Validation fails with a subject, no_ack, sealed, or duplicate-window error. The + pre-provisioned stream does not capture the publish subject, has no_ack set, is + sealed, or has a duplicate window below the worst-case retry span. Correct the + stream configuration to satisfy the pre-provisioning requirements above. +- Repeated stream-not-found errors in the log. The stream disappeared or was + never created. The publish is retryable, so the events stay on the live queue + and retry. Provision the stream so the queued events deliver on the next + replay. +- A warning that a publish was acknowledged by an unexpected stream. The + configured stream no longer captures the publish subject and another stream + does. The mismatched publish is rejected with a retryable error, the entry + stays on the live queue, and the mismatch resets the stream-validation + verdict. Later retries then fail validation before publishing, so the log + shows the validation failure rather than a repeating mismatch warning. The + mismatch warning fires again only after a validation pass lets another + publish through. The acknowledging stream named in the warning has persisted + a copy of each mismatched publish. Inspect that stream and remove the stray + copies. Delivery resumes once the configured stream captures the subject + again and validation passes. +- Health checks slow while a stream fails validation. Health reflects the last + stream-validation verdict rather than a live lookup on every check. The verdict + resets on a reconnect, a TLS rotation, an acknowledgment naming an unexpected + stream, or a stream-not-found publish outcome, and a reset forces a live stream + lookup on the next check or publish, bounded by the acknowledgement timeout. A + health snapshot taken right after a reset can take up to that timeout per + affected target until the stream is repaired. After a broker connection heals + on its own, publishes and health rely on the last validation verdict until an + error outcome resets it, a wrong-stream acknowledgment or a stream-not-found + publish outcome, so a stream reconfigured during a silent reconnect is + detected on the first publish evidence rather than immediately. +- Events stay in the queue and do not clear. The server is not acknowledging. + Check connectivity, that the stream subject filter captures the publish + subject, and that the server has capacity. Unacknowledged events are retained, + not lost. +- Duplicate deliveries observed. The duplicate window is shorter than the + worst-case retry span. Raise the stream duplicate window using the formula + above, especially after raising the acknowledgement timeout. An outage or a + restart that keeps an unacknowledged entry queued longer than the duplicate + window can also produce a duplicate on replay, consistent with at-least-once + delivery. Throttling RUSTFS_NOTIFY_TARGET_STREAM_CONCURRENCY below the number + of concurrently backlogged targets inserts untimed waits between the retries + of one entry and can push a late retry past the window. Keep the default or + add window margin when lowering it. +- Failed-store entries accumulating. A terminal rejection is recurring: a + message over the maximum payload size, a wrong expected last message identifier + or last sequence, or a sealed stream. Failed entries are on-disk diagnostic + records carrying the error class and a fixed diagnostic detail for inspection. + They are not re-published, and they are removed after the retention period. A + wrong subject or wrong credentials do not land here. They surface as retryable + and hold the events on the live queue until the configuration is fixed. + +## Disabling the Path + +Set jetstream_enable off. The target reverts to the NATS Core path. Events +already in the queue are delivered by the standard replay. No failed-store +entries are created while the path is off.