From a80699b6dd6c4317caf2acd14e6f8a7d495341fe Mon Sep 17 00:00:00 2001 From: escapecode <148354978+simon-escapecode@users.noreply.github.com> Date: Tue, 14 Jul 2026 08:36:14 +0100 Subject: [PATCH] feat: add an opt-in NATS JetStream publish path for the notify and audit targets (#4634) feat(targets): add an opt-in NATS JetStream publish path for the notify and audit targets The NATS notify and audit targets publish through NATS Core, which returns before the server has durably accepted the message. A broker restart or a connection drop between the publish and the flush loses the event, even though the send queue has already cleared it, and no acknowledgement gates that clear. An opt-in JetStream publish path clears a queued event only after the server returns a durable PublishAck, so delivery is at-least-once across a broker restart or a reconnect. It applies to both the notify and audit NATS targets, is off by default, and is byte-identical to the NATS Core path when disabled. The path includes durable store-and-forward, a stable dedup id sent as the Nats-Msg-Id header so a replayed event is collapsed by the stream duplicate window, pre-flight stream validation, and a bounded failed-events store for terminally-failed and retry-exhausted events. Three configuration keys per target select it: JETSTREAM_ENABLE, JETSTREAM_STREAM_NAME, and JETSTREAM_ACK_TIMEOUT_SECS, under the RUSTFS_NOTIFY_NATS_ and RUSTFS_AUDIT_NATS_ prefixes. The on-disk batch filename separator changes from colon to underscore so batch names are valid on Windows filesystems, with transparent read-back of files written under the previous separator. The migration affects the shared queue store for every target type and lands with this feature because the store gains its first Windows-exercised paths here. Co-authored-by: houseme --- CHANGELOG.md | 5 + Cargo.lock | 2 + crates/audit/src/pipeline.rs | 12 +- crates/audit/src/system.rs | 1 + crates/config/src/audit/nats.rs | 11 +- crates/config/src/constants/targets.rs | 7 + crates/config/src/notify/nats.rs | 11 +- crates/ecstore/src/config/audit.rs | 39 +- crates/ecstore/src/config/notify.rs | 37 +- crates/notify/src/integration.rs | 1 + crates/notify/src/runtime_facade.rs | 18 +- crates/notify/src/runtime_view.rs | 13 +- crates/obs/src/metrics/collectors/audit.rs | 20 +- .../metrics/collectors/notification_target.rs | 24 +- crates/obs/src/metrics/scheduler.rs | 2 + crates/obs/src/metrics/schema/audit.rs | 9 + .../src/metrics/schema/entry/metric_name.rs | 4 + .../src/metrics/schema/notification_target.rs | 9 + crates/targets/Cargo.toml | 2 + crates/targets/src/check.rs | 18 +- crates/targets/src/config/common.rs | 182 ++- crates/targets/src/config/target_args.rs | 103 +- crates/targets/src/error.rs | 3 + crates/targets/src/runtime/mod.rs | 1263 ++++++++++++++++- crates/targets/src/store.rs | 1036 +++++++++++++- crates/targets/src/target/amqp.rs | 7 +- crates/targets/src/target/kafka.rs | 7 +- crates/targets/src/target/mod.rs | 424 +++++- crates/targets/src/target/mqtt.rs | 7 +- crates/targets/src/target/mysql.rs | 7 +- crates/targets/src/target/nats/jetstream.rs | 1236 ++++++++++++++++ .../src/target/{nats.rs => nats/mod.rs} | 383 ++++- .../targets/src/target/nats/publish_error.rs | 346 +++++ crates/targets/src/target/nats/validation.rs | 405 ++++++ crates/targets/src/target/postgres.rs | 7 +- crates/targets/src/target/pulsar.rs | 7 +- crates/targets/src/target/redis.rs | 7 +- crates/targets/src/target/webhook.rs | 7 +- .../tests/nats_jetstream_regression_guards.rs | 259 ++++ .../nats_jetstream_validation_integration.rs | 143 ++ docs/operations/nats-jetstream.md | 356 +++++ 41 files changed, 6191 insertions(+), 249 deletions(-) create mode 100644 crates/targets/src/target/nats/jetstream.rs rename crates/targets/src/target/{nats.rs => nats/mod.rs} (57%) create mode 100644 crates/targets/src/target/nats/publish_error.rs create mode 100644 crates/targets/src/target/nats/validation.rs create mode 100644 crates/targets/tests/nats_jetstream_regression_guards.rs create mode 100644 crates/targets/tests/nats_jetstream_validation_integration.rs create mode 100644 docs/operations/nats-jetstream.md diff --git a/CHANGELOG.md b/CHANGELOG.md index b1cfeec37..05b19e177 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **Helm Ingress**: `customAnnotations` are now merged with class-specific annotations (nginx/traefik) instead of being ignored when `ingress.className` is set. ### Added +- **NATS JetStream Publish Path**: Opt-in at-least-once delivery for the NATS notify and audit targets. A NATS Core publish flushes to the connection without awaiting a broker acknowledgement, so an event can be lost across a broker restart or a reconnect after the send queue has already cleared it. A queued event now clears only after the JetStream `PublishAck`, so bucket notifications survive those interruptions. Off by default and byte-identical to the NATS Core path when disabled. + - Three configuration keys per target: `JETSTREAM_ENABLE`, `JETSTREAM_STREAM_NAME`, and `JETSTREAM_ACK_TIMEOUT_SECS`, under the `RUSTFS_NOTIFY_NATS_` and `RUSTFS_AUDIT_NATS_` prefixes + - Durable store-and-forward with a stable dedup id sent as the `Nats-Msg-Id` header, so a replay after a crash is collapsed by the server duplicate window + - Pre-flight stream validation, and a bounded failed-events store (count and TTL). Only a non-retryable rejection is recorded in the failed-events store. A retryable condition keeps the entry on the live queue until it is delivered + - Operator guide at `docs/operations/nats-jetstream.md` - **OpenStack Keystone Authentication Integration**: Full support for OpenStack Keystone authentication via X-Auth-Token headers - Tower-based middleware (`KeystoneAuthLayer`) self-contained within `rustfs-keystone` crate - Task-local storage for async-safe credential passing between middleware and auth handlers diff --git a/Cargo.lock b/Cargo.lock index e1a680218..c7fed349a 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -9971,6 +9971,7 @@ dependencies = [ "s3s", "serde", "serde_json", + "sha2 0.11.0", "snap", "sysinfo", "tempfile", @@ -9979,6 +9980,7 @@ dependencies = [ "tokio-postgres", "tokio-postgres-rustls", "tracing", + "tracing-subscriber", "url", "urlencoding", "uuid", diff --git a/crates/audit/src/pipeline.rs b/crates/audit/src/pipeline.rs index 9325e0cc4..6a83317c8 100644 --- a/crates/audit/src/pipeline.rs +++ b/crates/audit/src/pipeline.rs @@ -32,6 +32,7 @@ const EVENT_AUDIT_BATCH_DISPATCH_COMPLETED: &str = "audit_batch_dispatch_complet const EVENT_AUDIT_TARGET_STATE_CHANGED: &str = "audit_target_state_changed"; const EVENT_AUDIT_REPLAY_DELIVERED: &str = "audit_replay_delivered"; const EVENT_AUDIT_REPLAY_RETRY_SCHEDULED: &str = "audit_replay_retry_scheduled"; +const EVENT_AUDIT_REPLAY_RETRY_EXHAUSTED: &str = "audit_replay_retry_exhausted"; const EVENT_AUDIT_REPLAY_DROPPED: &str = "audit_replay_dropped"; const EVENT_AUDIT_REPLAY_STREAM_STATUS: &str = "audit_replay_stream_status"; @@ -281,6 +282,7 @@ impl AuditPipeline { let delivery = target.delivery_snapshot(); AuditTargetMetricSnapshot { failed_messages: delivery.failed_messages, + failed_store_length: delivery.failed_store_length, queue_length: delivery.queue_length, target_id: target.id().to_string(), total_messages: delivery.total_messages, @@ -463,18 +465,16 @@ impl AuditRuntimeFacade { target.record_final_failure(); observability::record_target_failure(); } - ReplayEvent::RetryExhausted { key, target } => { + ReplayEvent::RetryExhausted { detail, key, target } => { warn!( - event = EVENT_AUDIT_REPLAY_DROPPED, + event = EVENT_AUDIT_REPLAY_RETRY_EXHAUSTED, component = LOG_COMPONENT_AUDIT, subsystem = LOG_SUBSYSTEM_PIPELINE, target_id = %target.id(), replay_key = %key, - reason = "retry_exhausted", - "audit replay delivery" + error = %detail, + "audit replay retry budget exhausted, entry stays queued and retries" ); - target.record_final_failure(); - observability::record_target_failure(); } ReplayEvent::UnreadableEntry { key, error, target } => { warn!( diff --git a/crates/audit/src/system.rs b/crates/audit/src/system.rs index ead1f5157..4d79a04e6 100644 --- a/crates/audit/src/system.rs +++ b/crates/audit/src/system.rs @@ -30,6 +30,7 @@ const EVENT_AUDIT_CONFIG_RELOADED: &str = "audit_config_reloaded"; #[derive(Debug, Clone, Default, PartialEq, Eq)] pub struct AuditTargetMetricSnapshot { pub failed_messages: u64, + pub failed_store_length: u64, pub queue_length: u64, pub target_id: String, pub total_messages: u64, diff --git a/crates/config/src/audit/nats.rs b/crates/config/src/audit/nats.rs index c9657c62b..eccde6125 100644 --- a/crates/config/src/audit/nats.rs +++ b/crates/config/src/audit/nats.rs @@ -25,8 +25,11 @@ pub const ENV_AUDIT_NATS_TLS_CLIENT_KEY: &str = "RUSTFS_AUDIT_NATS_TLS_CLIENT_KE pub const ENV_AUDIT_NATS_TLS_REQUIRED: &str = "RUSTFS_AUDIT_NATS_TLS_REQUIRED"; pub const ENV_AUDIT_NATS_QUEUE_DIR: &str = "RUSTFS_AUDIT_NATS_QUEUE_DIR"; pub const ENV_AUDIT_NATS_QUEUE_LIMIT: &str = "RUSTFS_AUDIT_NATS_QUEUE_LIMIT"; +pub const ENV_AUDIT_NATS_JETSTREAM_ENABLE: &str = "RUSTFS_AUDIT_NATS_JETSTREAM_ENABLE"; +pub const ENV_AUDIT_NATS_JETSTREAM_STREAM_NAME: &str = "RUSTFS_AUDIT_NATS_JETSTREAM_STREAM_NAME"; +pub const ENV_AUDIT_NATS_JETSTREAM_ACK_TIMEOUT_SECS: &str = "RUSTFS_AUDIT_NATS_JETSTREAM_ACK_TIMEOUT_SECS"; -pub const ENV_AUDIT_NATS_KEYS: &[&str; 13] = &[ +pub const ENV_AUDIT_NATS_KEYS: &[&str; 16] = &[ ENV_AUDIT_NATS_ENABLE, ENV_AUDIT_NATS_ADDRESS, ENV_AUDIT_NATS_SUBJECT, @@ -40,6 +43,9 @@ pub const ENV_AUDIT_NATS_KEYS: &[&str; 13] = &[ ENV_AUDIT_NATS_TLS_REQUIRED, ENV_AUDIT_NATS_QUEUE_DIR, ENV_AUDIT_NATS_QUEUE_LIMIT, + ENV_AUDIT_NATS_JETSTREAM_ENABLE, + ENV_AUDIT_NATS_JETSTREAM_STREAM_NAME, + ENV_AUDIT_NATS_JETSTREAM_ACK_TIMEOUT_SECS, ]; pub const AUDIT_NATS_KEYS: &[&str] = &[ @@ -56,5 +62,8 @@ pub const AUDIT_NATS_KEYS: &[&str] = &[ crate::NATS_TLS_REQUIRED, crate::NATS_QUEUE_DIR, crate::NATS_QUEUE_LIMIT, + crate::NATS_JETSTREAM_ENABLE, + crate::NATS_JETSTREAM_STREAM_NAME, + crate::NATS_JETSTREAM_ACK_TIMEOUT_SECS, crate::COMMENT_KEY, ]; diff --git a/crates/config/src/constants/targets.rs b/crates/config/src/constants/targets.rs index 0d3e14a32..58e92067b 100644 --- a/crates/config/src/constants/targets.rs +++ b/crates/config/src/constants/targets.rs @@ -79,6 +79,13 @@ pub const NATS_TLS_CLIENT_KEY: &str = "tls_client_key"; pub const NATS_TLS_REQUIRED: &str = "tls_required"; pub const NATS_QUEUE_DIR: &str = "queue_dir"; pub const NATS_QUEUE_LIMIT: &str = "queue_limit"; +pub const NATS_JETSTREAM_ENABLE: &str = "jetstream_enable"; +pub const NATS_JETSTREAM_STREAM_NAME: &str = "jetstream_stream_name"; +pub const NATS_JETSTREAM_ACK_TIMEOUT_SECS: &str = "jetstream_ack_timeout_secs"; + +pub const NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS: u64 = 30; +pub const NATS_JETSTREAM_ACK_TIMEOUT_MIN_SECS: u64 = 10; +pub const NATS_JETSTREAM_ACK_TIMEOUT_MAX_SECS: u64 = 120; pub const PULSAR_BROKER: &str = "broker"; pub const PULSAR_TOPIC: &str = "topic"; diff --git a/crates/config/src/notify/nats.rs b/crates/config/src/notify/nats.rs index d9464128d..092d1ab9c 100644 --- a/crates/config/src/notify/nats.rs +++ b/crates/config/src/notify/nats.rs @@ -26,6 +26,9 @@ pub const NOTIFY_NATS_KEYS: &[&str] = &[ crate::NATS_TLS_REQUIRED, crate::NATS_QUEUE_DIR, crate::NATS_QUEUE_LIMIT, + crate::NATS_JETSTREAM_ENABLE, + crate::NATS_JETSTREAM_STREAM_NAME, + crate::NATS_JETSTREAM_ACK_TIMEOUT_SECS, crate::COMMENT_KEY, ]; @@ -42,8 +45,11 @@ pub const ENV_NOTIFY_NATS_TLS_CLIENT_KEY: &str = "RUSTFS_NOTIFY_NATS_TLS_CLIENT_ pub const ENV_NOTIFY_NATS_TLS_REQUIRED: &str = "RUSTFS_NOTIFY_NATS_TLS_REQUIRED"; pub const ENV_NOTIFY_NATS_QUEUE_DIR: &str = "RUSTFS_NOTIFY_NATS_QUEUE_DIR"; pub const ENV_NOTIFY_NATS_QUEUE_LIMIT: &str = "RUSTFS_NOTIFY_NATS_QUEUE_LIMIT"; +pub const ENV_NOTIFY_NATS_JETSTREAM_ENABLE: &str = "RUSTFS_NOTIFY_NATS_JETSTREAM_ENABLE"; +pub const ENV_NOTIFY_NATS_JETSTREAM_STREAM_NAME: &str = "RUSTFS_NOTIFY_NATS_JETSTREAM_STREAM_NAME"; +pub const ENV_NOTIFY_NATS_JETSTREAM_ACK_TIMEOUT_SECS: &str = "RUSTFS_NOTIFY_NATS_JETSTREAM_ACK_TIMEOUT_SECS"; -pub const ENV_NOTIFY_NATS_KEYS: &[&str; 13] = &[ +pub const ENV_NOTIFY_NATS_KEYS: &[&str; 16] = &[ ENV_NOTIFY_NATS_ENABLE, ENV_NOTIFY_NATS_ADDRESS, ENV_NOTIFY_NATS_SUBJECT, @@ -57,4 +63,7 @@ pub const ENV_NOTIFY_NATS_KEYS: &[&str; 13] = &[ ENV_NOTIFY_NATS_TLS_REQUIRED, ENV_NOTIFY_NATS_QUEUE_DIR, ENV_NOTIFY_NATS_QUEUE_LIMIT, + ENV_NOTIFY_NATS_JETSTREAM_ENABLE, + ENV_NOTIFY_NATS_JETSTREAM_STREAM_NAME, + ENV_NOTIFY_NATS_JETSTREAM_ACK_TIMEOUT_SECS, ]; diff --git a/crates/ecstore/src/config/audit.rs b/crates/ecstore/src/config/audit.rs index 3bb4ee545..b7ed83323 100644 --- a/crates/ecstore/src/config/audit.rs +++ b/crates/ecstore/src/config/audit.rs @@ -23,17 +23,19 @@ use rustfs_config::{ MQTT_QUEUE_LIMIT, MQTT_RECONNECT_INTERVAL, MQTT_TLS_CA, MQTT_TLS_CLIENT_CERT, MQTT_TLS_CLIENT_KEY, MQTT_TLS_POLICY, MQTT_TLS_TRUST_LEAF_AS_CA, MQTT_TOPIC, MQTT_USERNAME, MQTT_WS_PATH_ALLOWLIST, MYSQL_DSN_STRING, MYSQL_FORMAT, MYSQL_MAX_OPEN_CONNECTIONS, MYSQL_QUEUE_DIR, MYSQL_QUEUE_LIMIT, MYSQL_TABLE, MYSQL_TLS_CA, MYSQL_TLS_CLIENT_CERT, - MYSQL_TLS_CLIENT_KEY, NATS_ADDRESS, NATS_CREDENTIALS_FILE, NATS_PASSWORD, NATS_QUEUE_DIR, NATS_QUEUE_LIMIT, NATS_SUBJECT, - NATS_TLS_CA, NATS_TLS_CLIENT_CERT, NATS_TLS_CLIENT_KEY, NATS_TLS_REQUIRED, NATS_TOKEN, NATS_USERNAME, POSTGRES_DSN_STRING, - POSTGRES_FORMAT, POSTGRES_QUEUE_DIR, POSTGRES_QUEUE_LIMIT, POSTGRES_TABLE, POSTGRES_TLS_CA, POSTGRES_TLS_CLIENT_CERT, - POSTGRES_TLS_CLIENT_KEY, POSTGRES_TLS_REQUIRED, PULSAR_AUTH_TOKEN, PULSAR_BROKER, PULSAR_PASSWORD, PULSAR_QUEUE_DIR, - PULSAR_QUEUE_LIMIT, PULSAR_TLS_ALLOW_INSECURE, PULSAR_TLS_CA, PULSAR_TLS_HOSTNAME_VERIFICATION, PULSAR_TOPIC, - PULSAR_USERNAME, REDIS_CHANNEL, REDIS_CONNECTION_TIMEOUT, REDIS_KEEP_ALIVE_INTERVAL, REDIS_MAX_RETRY_ATTEMPTS, - REDIS_MAX_RETRY_DELAY, REDIS_MIN_RETRY_DELAY, REDIS_PASSWORD, REDIS_PIPELINE_BUFFER_SIZE, REDIS_QUEUE_DIR, REDIS_QUEUE_LIMIT, - REDIS_RECONNECT_RETRY_ATTEMPTS, REDIS_RESPONSE_TIMEOUT, REDIS_TLS_ALLOW_INSECURE, REDIS_TLS_CA, REDIS_TLS_CLIENT_CERT, - REDIS_TLS_CLIENT_KEY, REDIS_TLS_POLICY, REDIS_URL, REDIS_USERNAME, WEBHOOK_AUTH_TOKEN, WEBHOOK_BATCH_SIZE, WEBHOOK_CLIENT_CA, - WEBHOOK_CLIENT_CERT, WEBHOOK_CLIENT_KEY, WEBHOOK_ENDPOINT, WEBHOOK_HTTP_TIMEOUT, WEBHOOK_MAX_RETRY, WEBHOOK_QUEUE_DIR, - WEBHOOK_QUEUE_LIMIT, WEBHOOK_RETRY_INTERVAL, WEBHOOK_SKIP_TLS_VERIFY, + MYSQL_TLS_CLIENT_KEY, NATS_ADDRESS, NATS_CREDENTIALS_FILE, NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS, + NATS_JETSTREAM_ACK_TIMEOUT_SECS, NATS_JETSTREAM_ENABLE, NATS_JETSTREAM_STREAM_NAME, NATS_PASSWORD, NATS_QUEUE_DIR, + NATS_QUEUE_LIMIT, NATS_SUBJECT, NATS_TLS_CA, NATS_TLS_CLIENT_CERT, NATS_TLS_CLIENT_KEY, NATS_TLS_REQUIRED, NATS_TOKEN, + NATS_USERNAME, POSTGRES_DSN_STRING, POSTGRES_FORMAT, POSTGRES_QUEUE_DIR, POSTGRES_QUEUE_LIMIT, POSTGRES_TABLE, + POSTGRES_TLS_CA, POSTGRES_TLS_CLIENT_CERT, POSTGRES_TLS_CLIENT_KEY, POSTGRES_TLS_REQUIRED, PULSAR_AUTH_TOKEN, PULSAR_BROKER, + PULSAR_PASSWORD, PULSAR_QUEUE_DIR, PULSAR_QUEUE_LIMIT, PULSAR_TLS_ALLOW_INSECURE, PULSAR_TLS_CA, + PULSAR_TLS_HOSTNAME_VERIFICATION, PULSAR_TOPIC, PULSAR_USERNAME, REDIS_CHANNEL, REDIS_CONNECTION_TIMEOUT, + REDIS_KEEP_ALIVE_INTERVAL, REDIS_MAX_RETRY_ATTEMPTS, REDIS_MAX_RETRY_DELAY, REDIS_MIN_RETRY_DELAY, REDIS_PASSWORD, + REDIS_PIPELINE_BUFFER_SIZE, REDIS_QUEUE_DIR, REDIS_QUEUE_LIMIT, REDIS_RECONNECT_RETRY_ATTEMPTS, REDIS_RESPONSE_TIMEOUT, + REDIS_TLS_ALLOW_INSECURE, REDIS_TLS_CA, REDIS_TLS_CLIENT_CERT, REDIS_TLS_CLIENT_KEY, REDIS_TLS_POLICY, REDIS_URL, + REDIS_USERNAME, WEBHOOK_AUTH_TOKEN, WEBHOOK_BATCH_SIZE, WEBHOOK_CLIENT_CA, WEBHOOK_CLIENT_CERT, WEBHOOK_CLIENT_KEY, + WEBHOOK_ENDPOINT, WEBHOOK_HTTP_TIMEOUT, WEBHOOK_MAX_RETRY, WEBHOOK_QUEUE_DIR, WEBHOOK_QUEUE_LIMIT, WEBHOOK_RETRY_INTERVAL, + WEBHOOK_SKIP_TLS_VERIFY, }; use std::sync::LazyLock; @@ -350,6 +352,21 @@ pub static DEFAULT_AUDIT_NATS_KVS: LazyLock = LazyLock::new(|| { value: DEFAULT_LIMIT.to_string(), hidden_if_empty: false, }, + KV { + key: NATS_JETSTREAM_ENABLE.to_owned(), + value: EnableState::Off.to_string(), + hidden_if_empty: false, + }, + KV { + key: NATS_JETSTREAM_STREAM_NAME.to_owned(), + value: "".to_owned(), + hidden_if_empty: false, + }, + KV { + key: NATS_JETSTREAM_ACK_TIMEOUT_SECS.to_owned(), + value: NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS.to_string(), + hidden_if_empty: false, + }, KV { key: COMMENT_KEY.to_owned(), value: "".to_owned(), diff --git a/crates/ecstore/src/config/notify.rs b/crates/ecstore/src/config/notify.rs index a7130becd..4bb76e394 100644 --- a/crates/ecstore/src/config/notify.rs +++ b/crates/ecstore/src/config/notify.rs @@ -23,16 +23,18 @@ use rustfs_config::{ MQTT_QUEUE_LIMIT, MQTT_RECONNECT_INTERVAL, MQTT_TLS_CA, MQTT_TLS_CLIENT_CERT, MQTT_TLS_CLIENT_KEY, MQTT_TLS_POLICY, MQTT_TLS_TRUST_LEAF_AS_CA, MQTT_TOPIC, MQTT_USERNAME, MQTT_WS_PATH_ALLOWLIST, MYSQL_DSN_STRING, MYSQL_FORMAT, MYSQL_MAX_OPEN_CONNECTIONS, MYSQL_QUEUE_DIR, MYSQL_QUEUE_LIMIT, MYSQL_TABLE, MYSQL_TLS_CA, MYSQL_TLS_CLIENT_CERT, - MYSQL_TLS_CLIENT_KEY, NATS_ADDRESS, NATS_CREDENTIALS_FILE, NATS_PASSWORD, NATS_QUEUE_DIR, NATS_QUEUE_LIMIT, NATS_SUBJECT, - NATS_TLS_CA, NATS_TLS_CLIENT_CERT, NATS_TLS_CLIENT_KEY, NATS_TLS_REQUIRED, NATS_TOKEN, NATS_USERNAME, POSTGRES_DSN_STRING, - POSTGRES_FORMAT, POSTGRES_QUEUE_DIR, POSTGRES_QUEUE_LIMIT, POSTGRES_TABLE, POSTGRES_TLS_CA, POSTGRES_TLS_CLIENT_CERT, - POSTGRES_TLS_CLIENT_KEY, POSTGRES_TLS_REQUIRED, PULSAR_AUTH_TOKEN, PULSAR_BROKER, PULSAR_PASSWORD, PULSAR_QUEUE_DIR, - PULSAR_QUEUE_LIMIT, PULSAR_TLS_ALLOW_INSECURE, PULSAR_TLS_CA, PULSAR_TLS_HOSTNAME_VERIFICATION, PULSAR_TOPIC, - PULSAR_USERNAME, REDIS_CHANNEL, REDIS_CONNECTION_TIMEOUT, REDIS_KEEP_ALIVE_INTERVAL, REDIS_MAX_RETRY_ATTEMPTS, - REDIS_MAX_RETRY_DELAY, REDIS_MIN_RETRY_DELAY, REDIS_PASSWORD, REDIS_PIPELINE_BUFFER_SIZE, REDIS_QUEUE_DIR, REDIS_QUEUE_LIMIT, - REDIS_RECONNECT_RETRY_ATTEMPTS, REDIS_RESPONSE_TIMEOUT, REDIS_TLS_ALLOW_INSECURE, REDIS_TLS_CA, REDIS_TLS_CLIENT_CERT, - REDIS_TLS_CLIENT_KEY, REDIS_TLS_POLICY, REDIS_URL, REDIS_USERNAME, WEBHOOK_AUTH_TOKEN, WEBHOOK_CLIENT_CA, - WEBHOOK_CLIENT_CERT, WEBHOOK_CLIENT_KEY, WEBHOOK_ENDPOINT, WEBHOOK_QUEUE_DIR, WEBHOOK_QUEUE_LIMIT, WEBHOOK_SKIP_TLS_VERIFY, + MYSQL_TLS_CLIENT_KEY, NATS_ADDRESS, NATS_CREDENTIALS_FILE, NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS, + NATS_JETSTREAM_ACK_TIMEOUT_SECS, NATS_JETSTREAM_ENABLE, NATS_JETSTREAM_STREAM_NAME, NATS_PASSWORD, NATS_QUEUE_DIR, + NATS_QUEUE_LIMIT, NATS_SUBJECT, NATS_TLS_CA, NATS_TLS_CLIENT_CERT, NATS_TLS_CLIENT_KEY, NATS_TLS_REQUIRED, NATS_TOKEN, + NATS_USERNAME, POSTGRES_DSN_STRING, POSTGRES_FORMAT, POSTGRES_QUEUE_DIR, POSTGRES_QUEUE_LIMIT, POSTGRES_TABLE, + POSTGRES_TLS_CA, POSTGRES_TLS_CLIENT_CERT, POSTGRES_TLS_CLIENT_KEY, POSTGRES_TLS_REQUIRED, PULSAR_AUTH_TOKEN, PULSAR_BROKER, + PULSAR_PASSWORD, PULSAR_QUEUE_DIR, PULSAR_QUEUE_LIMIT, PULSAR_TLS_ALLOW_INSECURE, PULSAR_TLS_CA, + PULSAR_TLS_HOSTNAME_VERIFICATION, PULSAR_TOPIC, PULSAR_USERNAME, REDIS_CHANNEL, REDIS_CONNECTION_TIMEOUT, + REDIS_KEEP_ALIVE_INTERVAL, REDIS_MAX_RETRY_ATTEMPTS, REDIS_MAX_RETRY_DELAY, REDIS_MIN_RETRY_DELAY, REDIS_PASSWORD, + REDIS_PIPELINE_BUFFER_SIZE, REDIS_QUEUE_DIR, REDIS_QUEUE_LIMIT, REDIS_RECONNECT_RETRY_ATTEMPTS, REDIS_RESPONSE_TIMEOUT, + REDIS_TLS_ALLOW_INSECURE, REDIS_TLS_CA, REDIS_TLS_CLIENT_CERT, REDIS_TLS_CLIENT_KEY, REDIS_TLS_POLICY, REDIS_URL, + REDIS_USERNAME, WEBHOOK_AUTH_TOKEN, WEBHOOK_CLIENT_CA, WEBHOOK_CLIENT_CERT, WEBHOOK_CLIENT_KEY, WEBHOOK_ENDPOINT, + WEBHOOK_QUEUE_DIR, WEBHOOK_QUEUE_LIMIT, WEBHOOK_SKIP_TLS_VERIFY, }; use std::sync::LazyLock; @@ -328,6 +330,21 @@ pub static DEFAULT_NOTIFY_NATS_KVS: LazyLock = LazyLock::new(|| { value: DEFAULT_LIMIT.to_string(), hidden_if_empty: false, }, + KV { + key: NATS_JETSTREAM_ENABLE.to_owned(), + value: EnableState::Off.to_string(), + hidden_if_empty: false, + }, + KV { + key: NATS_JETSTREAM_STREAM_NAME.to_owned(), + value: "".to_owned(), + hidden_if_empty: false, + }, + KV { + key: NATS_JETSTREAM_ACK_TIMEOUT_SECS.to_owned(), + value: NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS.to_string(), + hidden_if_empty: false, + }, KV { key: COMMENT_KEY.to_owned(), value: "".to_owned(), diff --git a/crates/notify/src/integration.rs b/crates/notify/src/integration.rs index 1328f238e..a97f39cf7 100644 --- a/crates/notify/src/integration.rs +++ b/crates/notify/src/integration.rs @@ -80,6 +80,7 @@ pub struct NotificationMetricSnapshot { #[derive(Debug, Clone, Default, PartialEq, Eq)] pub struct NotificationTargetMetricSnapshot { pub failed_messages: u64, + pub failed_store_length: u64, pub queue_length: u64, pub target_id: String, pub target_type: String, diff --git a/crates/notify/src/runtime_facade.rs b/crates/notify/src/runtime_facade.rs index f59f20972..c6383844a 100644 --- a/crates/notify/src/runtime_facade.rs +++ b/crates/notify/src/runtime_facade.rs @@ -19,12 +19,13 @@ use rustfs_targets::{ use std::sync::Arc; use std::time::Duration; use tokio::sync::{RwLock, Semaphore}; -use tracing::{debug, info}; +use tracing::{debug, info, warn}; const LOG_COMPONENT_NOTIFY: &str = "notify"; const LOG_SUBSYSTEM_RUNTIME: &str = "runtime"; const EVENT_NOTIFY_RUNTIME_LIFECYCLE: &str = "notify_runtime_lifecycle"; const EVENT_NOTIFY_RUNTIME_SHUTDOWN_FAILED: &str = "notify_runtime_shutdown_failed"; +const EVENT_NOTIFY_REPLAY_RETRY_EXHAUSTED: &str = "notify_replay_retry_exhausted"; #[derive(Clone)] pub struct NotifyRuntimeFacade { @@ -48,9 +49,18 @@ impl NotifyRuntimeFacade { match event { ReplayEvent::Delivered { .. } => metrics.increment_processed(), ReplayEvent::RetryableError { .. } => {} - ReplayEvent::Dropped { target, .. } - | ReplayEvent::PermanentFailure { target, .. } - | ReplayEvent::RetryExhausted { target, .. } => { + ReplayEvent::RetryExhausted { detail, key, target } => { + warn!( + event = EVENT_NOTIFY_REPLAY_RETRY_EXHAUSTED, + component = LOG_COMPONENT_NOTIFY, + subsystem = LOG_SUBSYSTEM_RUNTIME, + target_id = %target.id(), + replay_key = %key, + error = %detail, + "notify replay retry budget exhausted, entry stays queued and retries" + ); + } + ReplayEvent::Dropped { target, .. } | ReplayEvent::PermanentFailure { target, .. } => { target.record_final_failure(); metrics.increment_failed(); } diff --git a/crates/notify/src/runtime_view.rs b/crates/notify/src/runtime_view.rs index 0ad80baa1..e8530b1b3 100644 --- a/crates/notify/src/runtime_view.rs +++ b/crates/notify/src/runtime_view.rs @@ -51,6 +51,7 @@ impl NotifyRuntimeView { .into_iter() .map(|snapshot| NotificationTargetMetricSnapshot { failed_messages: snapshot.failed_messages, + failed_store_length: snapshot.failed_store_length, queue_length: snapshot.queue_length, target_id: snapshot.target_id, target_type: snapshot.target_type, @@ -90,6 +91,7 @@ mod tests { active: bool, enabled: bool, failed_messages: Arc, + failed_store_length: u64, id: TargetID, total_messages: Arc, } @@ -100,6 +102,7 @@ mod tests { active: true, enabled: true, failed_messages: Arc::new(AtomicU64::new(0)), + failed_store_length: 0, id: TargetID::new(id.to_string(), name.to_string()), total_messages: Arc::new(AtomicU64::new(0)), } @@ -110,6 +113,11 @@ mod tests { self } + fn with_failed_store_length(mut self, failed_store_length: u64) -> Self { + self.failed_store_length = failed_store_length; + self + } + fn with_enabled(mut self, enabled: bool) -> Self { self.enabled = enabled; self @@ -168,6 +176,7 @@ mod tests { fn delivery_snapshot(&self) -> TargetDeliverySnapshot { TargetDeliverySnapshot { failed_messages: self.failed_messages.load(Ordering::Relaxed), + failed_store_length: self.failed_store_length, queue_length: 0, total_messages: self.total_messages.load(Ordering::Relaxed), } @@ -206,7 +215,7 @@ mod tests { let target_list = Arc::new(RwLock::new(TargetList::new())); let replay_workers = Arc::new(RwLock::new(ReplayWorkerManager::new())); - let online = Arc::new(TestTarget::new("primary", "webhook")); + let online = Arc::new(TestTarget::new("primary", "webhook").with_failed_store_length(7)); online.record_successes(3); online.record_failures(1); @@ -239,9 +248,11 @@ mod tests { assert_eq!(metric_snapshots.len(), 2); assert_eq!(metric_snapshots[0].target_id, "backup:mqtt"); assert_eq!(metric_snapshots[0].failed_messages, 0); + assert_eq!(metric_snapshots[0].failed_store_length, 0); assert_eq!(metric_snapshots[0].total_messages, 2); assert_eq!(metric_snapshots[1].target_id, "primary:webhook"); assert_eq!(metric_snapshots[1].failed_messages, 1); + assert_eq!(metric_snapshots[1].failed_store_length, 7); assert_eq!(metric_snapshots[1].total_messages, 3); let health_snapshots = runtime_view.snapshot_target_health().await; diff --git a/crates/obs/src/metrics/collectors/audit.rs b/crates/obs/src/metrics/collectors/audit.rs index 35a8129e2..0f7e1f600 100644 --- a/crates/obs/src/metrics/collectors/audit.rs +++ b/crates/obs/src/metrics/collectors/audit.rs @@ -33,6 +33,8 @@ pub struct AuditTargetStats { pub target_id: String, /// Number of messages that failed to send pub failed_messages: u64, + /// Number of messages held in the failed-events store + pub failed_store_length: u64, /// Number of unsent messages in queue pub queue_length: u64, /// Total number of messages sent @@ -48,7 +50,7 @@ pub fn collect_audit_metrics(stats: &[AuditTargetStats]) -> Vec = Cow::Owned(stat.target_id.clone()); @@ -56,6 +58,10 @@ pub fn collect_audit_metrics(stats: &[AuditTargetStats]) -> Vec return Vec::new(); } - let mut metrics = Vec::with_capacity(stats.len() * 3); + let mut metrics = Vec::with_capacity(stats.len() * 4); for stat in stats { let target_id: Cow<'static, str> = Cow::Owned(stat.target_id.clone()); let target_type: Cow<'static, str> = Cow::Owned(stat.target_type.clone()); @@ -45,6 +46,11 @@ pub fn collect_notification_target_metrics(stats: &[NotificationTargetStats]) -> .with_label(TARGET_ID, target_id.clone()) .with_label(TARGET_TYPE, target_type.clone()), ); + metrics.push( + PrometheusMetric::from_descriptor(&NOTIFICATION_TARGET_FAILED_STORE_LENGTH_MD, stat.failed_store_length as f64) + .with_label(TARGET_ID, target_id.clone()) + .with_label(TARGET_TYPE, target_type.clone()), + ); metrics.push( PrometheusMetric::from_descriptor(&NOTIFICATION_TARGET_QUEUE_LENGTH_MD, stat.queue_length as f64) .with_label(TARGET_ID, target_id.clone()) @@ -69,6 +75,7 @@ mod tests { fn test_collect_notification_target_metrics() { let stats = vec![NotificationTargetStats { failed_messages: 2, + failed_store_length: 3, queue_length: 4, target_id: "primary:webhook".to_string(), target_type: "webhook".to_string(), @@ -77,7 +84,15 @@ mod tests { let metrics = collect_notification_target_metrics(&stats); - assert_eq!(metrics.len(), 3); + assert_eq!(metrics.len(), 4); + assert!(metrics.iter().any(|metric| { + metric.value == 3.0 + && metric.name == NOTIFICATION_TARGET_FAILED_STORE_LENGTH_MD.get_full_metric_name() + && metric + .labels + .iter() + .any(|(key, value)| *key == TARGET_ID && value == "primary:webhook") + })); assert!(metrics.iter().any(|metric| { metric.value == 42.0 && metric @@ -94,6 +109,7 @@ mod tests { #[test] fn notification_target_totals_are_exported_as_gauges() { assert_eq!(NOTIFICATION_TARGET_FAILED_MESSAGES_MD.metric_type, MetricType::Gauge); + assert_eq!(NOTIFICATION_TARGET_FAILED_STORE_LENGTH_MD.metric_type, MetricType::Gauge); assert_eq!(NOTIFICATION_TARGET_QUEUE_LENGTH_MD.metric_type, MetricType::Gauge); assert_eq!(NOTIFICATION_TARGET_TOTAL_MESSAGES_MD.metric_type, MetricType::Gauge); } diff --git a/crates/obs/src/metrics/scheduler.rs b/crates/obs/src/metrics/scheduler.rs index 2fbfd7b55..dd306f1a7 100644 --- a/crates/obs/src/metrics/scheduler.rs +++ b/crates/obs/src/metrics/scheduler.rs @@ -1222,6 +1222,7 @@ pub fn init_metrics_runtime(token: CancellationToken) { .into_iter() .map(|snapshot| AuditTargetStats { failed_messages: snapshot.failed_messages, + failed_store_length: snapshot.failed_store_length, queue_length: snapshot.queue_length, target_id: snapshot.target_id, total_messages: snapshot.total_messages, @@ -1274,6 +1275,7 @@ pub fn init_metrics_runtime(token: CancellationToken) { .into_iter() .map(|snapshot| NotificationTargetStats { failed_messages: snapshot.failed_messages, + failed_store_length: snapshot.failed_store_length, queue_length: snapshot.queue_length, target_id: snapshot.target_id, target_type: snapshot.target_type, diff --git a/crates/obs/src/metrics/schema/audit.rs b/crates/obs/src/metrics/schema/audit.rs index a5d738c6a..ceeb9045e 100644 --- a/crates/obs/src/metrics/schema/audit.rs +++ b/crates/obs/src/metrics/schema/audit.rs @@ -33,6 +33,15 @@ pub static AUDIT_FAILED_MESSAGES_MD: LazyLock = LazyLock::new( ) }); +pub static AUDIT_FAILED_STORE_LENGTH_MD: LazyLock = LazyLock::new(|| { + new_gauge_md( + MetricName::AuditFailedStoreLength, + "Number of audit messages held in the failed-events store for target", + &[TARGET_ID], + subsystems::AUDIT, + ) +}); + pub static AUDIT_TARGET_QUEUE_LENGTH_MD: LazyLock = LazyLock::new(|| { new_gauge_md( MetricName::AuditTargetQueueLength, diff --git a/crates/obs/src/metrics/schema/entry/metric_name.rs b/crates/obs/src/metrics/schema/entry/metric_name.rs index 1eb890edb..9c0cbf2a2 100644 --- a/crates/obs/src/metrics/schema/entry/metric_name.rs +++ b/crates/obs/src/metrics/schema/entry/metric_name.rs @@ -175,6 +175,7 @@ pub enum MetricName { // Audit metrics AuditFailedMessages, + AuditFailedStoreLength, AuditTargetQueueLength, AuditTotalMessages, @@ -221,6 +222,7 @@ pub enum MetricName { NotificationEventsSentTotal, NotificationEventsSkippedTotal, NotificationTargetFailedMessages, + NotificationTargetFailedStoreLength, NotificationTargetQueueLength, NotificationTargetTotalMessages, @@ -581,6 +583,7 @@ impl MetricName { Self::ApiTrafficRecvBytes => "traffic_received_bytes".to_string(), Self::AuditFailedMessages => "failed_messages".to_string(), + Self::AuditFailedStoreLength => "failed_store_length".to_string(), Self::AuditTargetQueueLength => "target_queue_length".to_string(), Self::AuditTotalMessages => "total_messages".to_string(), @@ -627,6 +630,7 @@ impl MetricName { Self::NotificationEventsSentTotal => "events_sent_total".to_string(), Self::NotificationEventsSkippedTotal => "events_skipped_total".to_string(), Self::NotificationTargetFailedMessages => "failed_messages".to_string(), + Self::NotificationTargetFailedStoreLength => "failed_store_length".to_string(), Self::NotificationTargetQueueLength => "target_queue_length".to_string(), Self::NotificationTargetTotalMessages => "total_messages".to_string(), diff --git a/crates/obs/src/metrics/schema/notification_target.rs b/crates/obs/src/metrics/schema/notification_target.rs index ceb801c13..3a1c6e491 100644 --- a/crates/obs/src/metrics/schema/notification_target.rs +++ b/crates/obs/src/metrics/schema/notification_target.rs @@ -31,6 +31,15 @@ pub static NOTIFICATION_TARGET_FAILED_MESSAGES_MD: LazyLock = ) }); +pub static NOTIFICATION_TARGET_FAILED_STORE_LENGTH_MD: LazyLock = LazyLock::new(|| { + new_gauge_md( + MetricName::NotificationTargetFailedStoreLength, + "Number of notification messages held in the failed-events store for target", + &NOTIFICATION_TARGET_LABELS, + subsystems::NOTIFICATION, + ) +}); + pub static NOTIFICATION_TARGET_QUEUE_LENGTH_MD: LazyLock = LazyLock::new(|| { new_gauge_md( MetricName::NotificationTargetQueueLength, diff --git a/crates/targets/Cargo.toml b/crates/targets/Cargo.toml index 470362719..297fb591c 100644 --- a/crates/targets/Cargo.toml +++ b/crates/targets/Cargo.toml @@ -34,6 +34,7 @@ rustls-native-certs = { workspace = true } s3s = { workspace = true } serde = { workspace = true } serde_json = { workspace = true } +sha2 = { workspace = true } snap = { workspace = true } thiserror = { workspace = true } tokio = { workspace = true, features = ["fs", "rt-multi-thread", "sync", "time"] } @@ -56,6 +57,7 @@ metrics = { workspace = true } criterion = { workspace = true } tempfile = { workspace = true } rcgen = { workspace = true } +tracing-subscriber = { workspace = true } [[bench]] name = "queue_store_benchmark" diff --git a/crates/targets/src/check.rs b/crates/targets/src/check.rs index 7de161224..c56c14895 100644 --- a/crates/targets/src/check.rs +++ b/crates/targets/src/check.rs @@ -100,11 +100,27 @@ pub async fn check_nats_server_available(args: &crate::target::nats::NATSArgs) - .flush() .await .map_err(|e| crate::TargetError::Network(format!("NATS connection check failed: {e}")))?; + // Validate the configured stream on the live connection before the drain closes it, so a + // missing or non-writable stream is rejected here rather than at first publish. The lookup is + // read-only and never creates a stream. The drain runs regardless of the validation outcome so + // the check connection is closed gracefully, and the validation error then propagates. + let validation = if args.jetstream_enable.unwrap_or(false) { + let mut context = async_nats::jetstream::new(client.clone()); + // Match every other context construction site: the ack timeout is the per-operation await + // bound, so the validation get_stream lookup uses it rather than the library default. + context.set_timeout(std::time::Duration::from_secs( + args.jetstream_ack_timeout_secs + .unwrap_or(rustfs_config::NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS), + )); + crate::target::nats::validate_jetstream_stream(&context, args, "nats-check", None).await + } else { + Ok(()) + }; client .drain() .await .map_err(|e| crate::TargetError::Network(format!("Failed to close NATS check connection: {e}")))?; - Ok(()) + validation }) .await .unwrap_or_else(|_| Err(crate::TargetError::Timeout("NATS connection timed out".to_string()))) diff --git a/crates/targets/src/config/common.rs b/crates/targets/src/config/common.rs index ff3e850da..46f365410 100644 --- a/crates/targets/src/config/common.rs +++ b/crates/targets/src/config/common.rs @@ -13,13 +13,15 @@ // limitations under the License. use crate::TargetError; +use crate::target::nats::validate_jetstream_settings; use crate::target::pulsar::validate_pulsar_broker; use async_nats::ServerAddr; use rustfs_config::server_config::KVS; use rustfs_config::{ - DEFAULT_DELIMITER, ENABLE_KEY, EnableState, NATS_CREDENTIALS_FILE, NATS_PASSWORD, NATS_QUEUE_DIR, NATS_SUBJECT, NATS_TLS_CA, - NATS_TLS_CLIENT_CERT, NATS_TLS_CLIENT_KEY, NATS_TOKEN, NATS_USERNAME, PULSAR_AUTH_TOKEN, PULSAR_PASSWORD, PULSAR_QUEUE_DIR, - PULSAR_TLS_CA, PULSAR_TOPIC, PULSAR_USERNAME, + DEFAULT_DELIMITER, ENABLE_KEY, EnableState, NATS_CREDENTIALS_FILE, NATS_JETSTREAM_ACK_TIMEOUT_SECS, NATS_JETSTREAM_ENABLE, + NATS_JETSTREAM_STREAM_NAME, NATS_PASSWORD, NATS_QUEUE_DIR, NATS_SUBJECT, NATS_TLS_CA, NATS_TLS_CLIENT_CERT, + NATS_TLS_CLIENT_KEY, NATS_TOKEN, NATS_USERNAME, PULSAR_AUTH_TOKEN, PULSAR_PASSWORD, PULSAR_QUEUE_DIR, PULSAR_TLS_CA, + PULSAR_TOPIC, PULSAR_USERNAME, }; use rustfs_utils::egress::validate_outbound_url; use std::collections::HashSet; @@ -130,10 +132,42 @@ pub(super) fn validate_nats_server_config(server: &ServerAddr, config: &KVS, def return Err(TargetError::Configuration("NATS queue directory must be an absolute path".to_string())); } + let jetstream_enable = parse_jetstream_enable(config)?.unwrap_or(false); + let jetstream_stream_name = config.lookup(NATS_JETSTREAM_STREAM_NAME).unwrap_or_default(); + let jetstream_ack_timeout_secs = parse_jetstream_ack_timeout_secs(config)?; + validate_jetstream_settings(jetstream_enable, &jetstream_stream_name, &queue_dir, jetstream_ack_timeout_secs)?; + let _ = server; Ok(()) } +/// Parses the jetstream_enable flag, distinguishing an absent or blank key (None, disabled) from a +/// key that is present but carries an unparsable value, which is a configuration error rather than +/// a silent disable. Unwrapping an unparsable value to disabled would drop the durability guarantee +/// the operator asked for with no signal. +pub(super) fn parse_jetstream_enable(config: &KVS) -> Result, TargetError> { + match config.lookup(NATS_JETSTREAM_ENABLE) { + Some(value) if !value.trim().is_empty() => parse_target_bool(Some(value.as_str())).map(Some).ok_or_else(|| { + TargetError::Configuration(format!( + "Invalid {NATS_JETSTREAM_ENABLE} boolean value: {value} (accepted values include on, off, true, false, yes, no, 1, 0)" + )) + }), + _ => Ok(None), + } +} + +pub(super) fn parse_jetstream_ack_timeout_secs(config: &KVS) -> Result, TargetError> { + match config.lookup(NATS_JETSTREAM_ACK_TIMEOUT_SECS) { + Some(value) if !value.trim().is_empty() => { + let secs = value.trim().parse::().map_err(|_| { + TargetError::Configuration(format!("{NATS_JETSTREAM_ACK_TIMEOUT_SECS} must be a whole number of seconds")) + })?; + Ok(Some(secs)) + } + _ => Ok(None), + } +} + pub(super) fn validate_pulsar_broker_config(broker: &str, config: &KVS, default_queue_dir: &str) -> Result<(), TargetError> { let url = validate_pulsar_broker(broker)?; @@ -193,15 +227,26 @@ pub(super) fn validate_outbound_http_url(value: &Url, field_label: &str) -> Resu #[cfg(test)] mod tests { - use super::{validate_nats_server_config, validate_pulsar_broker_config}; + use super::{parse_jetstream_enable, validate_nats_server_config, validate_pulsar_broker_config}; use async_nats::ServerAddr; use rustfs_config::server_config::KVS; use rustfs_config::{ - NATS_PASSWORD, NATS_QUEUE_DIR, NATS_SUBJECT, NATS_TOKEN, NATS_USERNAME, PULSAR_TLS_ALLOW_INSECURE, PULSAR_TLS_CA, - PULSAR_TLS_HOSTNAME_VERIFICATION, PULSAR_TOPIC, + NATS_JETSTREAM_ACK_TIMEOUT_SECS, NATS_JETSTREAM_ENABLE, NATS_JETSTREAM_STREAM_NAME, NATS_PASSWORD, NATS_QUEUE_DIR, + NATS_SUBJECT, NATS_TOKEN, NATS_USERNAME, PULSAR_TLS_ALLOW_INSECURE, PULSAR_TLS_CA, PULSAR_TLS_HOSTNAME_VERIFICATION, + PULSAR_TOPIC, }; use std::str::FromStr; + fn nats_server() -> ServerAddr { + ServerAddr::from_str("nats://127.0.0.1:4222").expect("valid nats address") + } + + // Absolute on Linux, macOS, and Windows. temp_dir needs no filesystem to exist for a + // validation-only test, and Path::is_absolute stays true across platforms. + fn nats_queue_dir() -> String { + std::env::temp_dir().join("rustfs-nats-queue").to_string_lossy().into_owned() + } + #[test] fn validate_nats_server_config_rejects_multiple_auth_methods() { let server = ServerAddr::from_str("nats://127.0.0.1:4222").expect("valid nats address"); @@ -228,6 +273,131 @@ mod tests { assert!(err.to_string().contains("absolute path")); } + #[test] + fn validate_nats_server_config_rejects_jetstream_without_stream() { + let mut config = KVS::new(); + config.insert(NATS_SUBJECT.to_string(), "events".to_string()); + config.insert(NATS_QUEUE_DIR.to_string(), nats_queue_dir()); + config.insert(NATS_JETSTREAM_ENABLE.to_string(), "on".to_string()); + + let err = validate_nats_server_config(&nats_server(), &config, "") + .expect_err("enabled jetstream without a stream should be rejected"); + assert!(err.to_string().contains(NATS_JETSTREAM_STREAM_NAME)); + } + + #[test] + fn validate_nats_server_config_rejects_jetstream_without_queue_dir() { + let mut config = KVS::new(); + config.insert(NATS_SUBJECT.to_string(), "events".to_string()); + config.insert(NATS_JETSTREAM_ENABLE.to_string(), "on".to_string()); + config.insert(NATS_JETSTREAM_STREAM_NAME.to_string(), "RUSTFS_EVENTS".to_string()); + + let err = validate_nats_server_config(&nats_server(), &config, "") + .expect_err("enabled jetstream without a queue_dir should be rejected"); + assert!(err.to_string().contains(NATS_QUEUE_DIR)); + } + + #[test] + fn validate_nats_server_config_rejects_out_of_range_ack_timeout() { + let mut config = KVS::new(); + config.insert(NATS_SUBJECT.to_string(), "events".to_string()); + config.insert(NATS_JETSTREAM_ACK_TIMEOUT_SECS.to_string(), "5".to_string()); + + let err = + validate_nats_server_config(&nats_server(), &config, "").expect_err("out-of-range ack timeout should be rejected"); + assert!(err.to_string().contains("between")); + } + + #[test] + fn validate_nats_server_config_accepts_enabled_jetstream_with_stream_and_queue() { + let mut config = KVS::new(); + config.insert(NATS_SUBJECT.to_string(), "events".to_string()); + config.insert(NATS_QUEUE_DIR.to_string(), nats_queue_dir()); + config.insert(NATS_JETSTREAM_ENABLE.to_string(), "on".to_string()); + config.insert(NATS_JETSTREAM_STREAM_NAME.to_string(), "RUSTFS_EVENTS".to_string()); + + validate_nats_server_config(&nats_server(), &config, "").expect("a stream name and queue dir accept enabling jetstream"); + } + + #[test] + fn validate_nats_server_config_audit_parity_matches_notify() { + let mut config = KVS::new(); + config.insert(NATS_SUBJECT.to_string(), "events".to_string()); + config.insert(NATS_QUEUE_DIR.to_string(), nats_queue_dir()); + config.insert(NATS_JETSTREAM_ENABLE.to_string(), "on".to_string()); + + // The same invalid combination (enabled, stream name missing) is rejected identically on the + // notify and the audit path, proving one shared validator with no drift. + let notify = validate_nats_server_config(&nats_server(), &config, "/opt/rustfs/events").expect_err("notify rejects"); + let audit = validate_nats_server_config(&nats_server(), &config, "/opt/rustfs/audit").expect_err("audit rejects"); + assert_eq!(notify.to_string(), audit.to_string()); + } + + #[test] + fn validate_nats_server_config_accepts_jetstream_absent() { + let mut config = KVS::new(); + config.insert(NATS_SUBJECT.to_string(), "events".to_string()); + config.insert(NATS_QUEUE_DIR.to_string(), nats_queue_dir()); + + validate_nats_server_config(&nats_server(), &config, "").expect("baseline config validates"); + } + + #[test] + fn parse_jetstream_enable_accepts_the_standard_boolean_forms() { + // An absent or blank key is None, legitimately disabled. + assert_eq!(parse_jetstream_enable(&KVS::new()).unwrap(), None, "an absent key parses as unset"); + let mut blank = KVS::new(); + blank.insert(NATS_JETSTREAM_ENABLE.to_string(), " ".to_string()); + assert_eq!(parse_jetstream_enable(&blank).unwrap(), None, "a blank value parses as unset"); + + for (value, expected) in [ + ("on", true), + ("off", false), + ("true", true), + ("false", false), + ("yes", true), + ("no", false), + ("1", true), + ("0", false), + ("enabled", true), + ("disabled", false), + ] { + let mut config = KVS::new(); + config.insert(NATS_JETSTREAM_ENABLE.to_string(), value.to_string()); + assert_eq!( + parse_jetstream_enable(&config).unwrap(), + Some(expected), + "value {value} parses as {expected}" + ); + } + } + + #[test] + fn parse_jetstream_enable_rejects_an_unparsable_value_naming_the_key() { + // A typo must fail validation rather than silently disabling the durability guarantee. + for value in ["y", "t", "enabl", "10"] { + let mut config = KVS::new(); + config.insert(NATS_JETSTREAM_ENABLE.to_string(), value.to_string()); + let err = parse_jetstream_enable(&config).expect_err("an unparsable value is rejected"); + assert!( + err.to_string().contains(NATS_JETSTREAM_ENABLE), + "the error names the key for value {value}: {err}" + ); + } + } + + #[test] + fn validate_nats_server_config_rejects_an_invalid_jetstream_enable_value() { + let mut config = KVS::new(); + config.insert(NATS_SUBJECT.to_string(), "events".to_string()); + config.insert(NATS_QUEUE_DIR.to_string(), nats_queue_dir()); + config.insert(NATS_JETSTREAM_ENABLE.to_string(), "y".to_string()); + + let err = validate_nats_server_config(&nats_server(), &config, "") + .expect_err("an unparsable enable value fails validation instead of passing as disabled"); + assert!(err.to_string().contains(NATS_JETSTREAM_ENABLE), "the error names the key: {err}"); + } + #[test] fn validate_pulsar_broker_config_rejects_tls_ca_without_tls_scheme() { let mut config = KVS::new(); diff --git a/crates/targets/src/config/target_args.rs b/crates/targets/src/config/target_args.rs index f32741e0e..6f50c5d6e 100644 --- a/crates/targets/src/config/target_args.rs +++ b/crates/targets/src/config/target_args.rs @@ -13,7 +13,8 @@ // limitations under the License. use super::common::{ - parse_target_bool, parse_url, validate_nats_server_config, validate_outbound_http_url, validate_pulsar_broker_config, + parse_jetstream_ack_timeout_secs, parse_jetstream_enable, parse_target_bool, parse_url, validate_nats_server_config, + validate_outbound_http_url, validate_pulsar_broker_config, }; use crate::error::TargetError; use crate::target::{ @@ -38,17 +39,17 @@ use rustfs_config::{ MQTT_KEEP_ALIVE_INTERVAL, MQTT_PASSWORD, MQTT_QOS, MQTT_QUEUE_DIR, MQTT_QUEUE_LIMIT, MQTT_RECONNECT_INTERVAL, MQTT_TLS_CA, MQTT_TLS_CLIENT_CERT, MQTT_TLS_CLIENT_KEY, MQTT_TLS_POLICY, MQTT_TLS_TRUST_LEAF_AS_CA, MQTT_TOPIC, MQTT_USERNAME, MQTT_WS_PATH_ALLOWLIST, MYSQL_DSN_STRING, MYSQL_FORMAT, MYSQL_MAX_OPEN_CONNECTIONS, MYSQL_QUEUE_DIR, MYSQL_QUEUE_LIMIT, - MYSQL_TABLE, MYSQL_TLS_CA, MYSQL_TLS_CLIENT_CERT, MYSQL_TLS_CLIENT_KEY, NATS_ADDRESS, NATS_CREDENTIALS_FILE, NATS_PASSWORD, - NATS_QUEUE_DIR, NATS_QUEUE_LIMIT, NATS_SUBJECT, NATS_TLS_CA, NATS_TLS_CLIENT_CERT, NATS_TLS_CLIENT_KEY, NATS_TLS_REQUIRED, - NATS_TOKEN, NATS_USERNAME, POSTGRES_DSN_STRING, POSTGRES_FORMAT, POSTGRES_QUEUE_DIR, POSTGRES_QUEUE_LIMIT, POSTGRES_TABLE, - POSTGRES_TLS_CA, POSTGRES_TLS_CLIENT_CERT, POSTGRES_TLS_CLIENT_KEY, POSTGRES_TLS_REQUIRED, PULSAR_AUTH_TOKEN, PULSAR_BROKER, - PULSAR_PASSWORD, PULSAR_QUEUE_DIR, PULSAR_QUEUE_LIMIT, PULSAR_TLS_ALLOW_INSECURE, PULSAR_TLS_CA, - PULSAR_TLS_HOSTNAME_VERIFICATION, PULSAR_TOPIC, PULSAR_USERNAME, REDIS_CHANNEL, REDIS_CONNECTION_TIMEOUT, - REDIS_KEEP_ALIVE_INTERVAL, REDIS_MAX_RETRY_ATTEMPTS, REDIS_MAX_RETRY_DELAY, REDIS_MIN_RETRY_DELAY, REDIS_PASSWORD, - REDIS_PIPELINE_BUFFER_SIZE, REDIS_QUEUE_DIR, REDIS_QUEUE_LIMIT, REDIS_RECONNECT_RETRY_ATTEMPTS, REDIS_RESPONSE_TIMEOUT, - REDIS_TLS_ALLOW_INSECURE, REDIS_TLS_CA, REDIS_TLS_CLIENT_CERT, REDIS_TLS_CLIENT_KEY, REDIS_TLS_POLICY, REDIS_URL, - REDIS_USERNAME, RUSTFS_WEBHOOK_SKIP_TLS_VERIFY_DEFAULT, WEBHOOK_AUTH_TOKEN, WEBHOOK_CLIENT_CA, WEBHOOK_CLIENT_CERT, - WEBHOOK_CLIENT_KEY, WEBHOOK_ENDPOINT, WEBHOOK_QUEUE_DIR, WEBHOOK_QUEUE_LIMIT, WEBHOOK_SKIP_TLS_VERIFY, + MYSQL_TABLE, MYSQL_TLS_CA, MYSQL_TLS_CLIENT_CERT, MYSQL_TLS_CLIENT_KEY, NATS_ADDRESS, NATS_CREDENTIALS_FILE, + NATS_JETSTREAM_STREAM_NAME, NATS_PASSWORD, NATS_QUEUE_DIR, NATS_QUEUE_LIMIT, NATS_SUBJECT, NATS_TLS_CA, NATS_TLS_CLIENT_CERT, + NATS_TLS_CLIENT_KEY, NATS_TLS_REQUIRED, NATS_TOKEN, NATS_USERNAME, POSTGRES_DSN_STRING, POSTGRES_FORMAT, POSTGRES_QUEUE_DIR, + POSTGRES_QUEUE_LIMIT, POSTGRES_TABLE, POSTGRES_TLS_CA, POSTGRES_TLS_CLIENT_CERT, POSTGRES_TLS_CLIENT_KEY, + POSTGRES_TLS_REQUIRED, PULSAR_AUTH_TOKEN, PULSAR_BROKER, PULSAR_PASSWORD, PULSAR_QUEUE_DIR, PULSAR_QUEUE_LIMIT, + PULSAR_TLS_ALLOW_INSECURE, PULSAR_TLS_CA, PULSAR_TLS_HOSTNAME_VERIFICATION, PULSAR_TOPIC, PULSAR_USERNAME, REDIS_CHANNEL, + REDIS_CONNECTION_TIMEOUT, REDIS_KEEP_ALIVE_INTERVAL, REDIS_MAX_RETRY_ATTEMPTS, REDIS_MAX_RETRY_DELAY, REDIS_MIN_RETRY_DELAY, + REDIS_PASSWORD, REDIS_PIPELINE_BUFFER_SIZE, REDIS_QUEUE_DIR, REDIS_QUEUE_LIMIT, REDIS_RECONNECT_RETRY_ATTEMPTS, + REDIS_RESPONSE_TIMEOUT, REDIS_TLS_ALLOW_INSECURE, REDIS_TLS_CA, REDIS_TLS_CLIENT_CERT, REDIS_TLS_CLIENT_KEY, + REDIS_TLS_POLICY, REDIS_URL, REDIS_USERNAME, RUSTFS_WEBHOOK_SKIP_TLS_VERIFY_DEFAULT, WEBHOOK_AUTH_TOKEN, WEBHOOK_CLIENT_CA, + WEBHOOK_CLIENT_CERT, WEBHOOK_CLIENT_KEY, WEBHOOK_ENDPOINT, WEBHOOK_QUEUE_DIR, WEBHOOK_QUEUE_LIMIT, WEBHOOK_SKIP_TLS_VERIFY, }; use std::path::Path; use std::time::Duration; @@ -315,6 +316,11 @@ pub fn build_nats_args(config: &KVS, default_queue_dir: &str, target_type: Targe .lookup(NATS_QUEUE_LIMIT) .and_then(|v| v.parse::().ok()) .unwrap_or(DEFAULT_LIMIT), + // Strict parse: an unparsable value is a configuration error, never a silent disable, so the + // durability guarantee the operator asked for cannot vanish over a typo. + jetstream_enable: parse_jetstream_enable(config)?, + jetstream_stream_name: config.lookup(NATS_JETSTREAM_STREAM_NAME).filter(|name| !name.is_empty()), + jetstream_ack_timeout_secs: parse_jetstream_ack_timeout_secs(config)?, target_type, }) } @@ -598,9 +604,9 @@ pub fn validate_mysql_config(config: &KVS, default_queue_dir: &str) -> Result<() #[cfg(test)] mod tests { use super::{ - build_amqp_args, build_kafka_args, build_mysql_args, build_postgres_args, build_redis_args, build_webhook_args, - validate_amqp_config, validate_kafka_config, validate_mysql_config, validate_postgres_config, validate_redis_config, - validate_webhook_config, + build_amqp_args, build_kafka_args, build_mysql_args, build_nats_args, build_postgres_args, build_redis_args, + build_webhook_args, validate_amqp_config, validate_kafka_config, validate_mysql_config, validate_postgres_config, + validate_redis_config, validate_webhook_config, }; use crate::target::{ TargetType, @@ -612,7 +618,8 @@ mod tests { AMQP_EXCHANGE, AMQP_MANDATORY, AMQP_PASSWORD, AMQP_PERSISTENT, AMQP_QUEUE_DIR, AMQP_ROUTING_KEY, AMQP_TLS_CLIENT_CERT, AMQP_TLS_CLIENT_KEY, AMQP_URL, AMQP_USERNAME, KAFKA_ACKS, KAFKA_BROKERS, KAFKA_SASL_ENABLE, KAFKA_SASL_MECHANISM, KAFKA_SASL_PASSWORD, KAFKA_SASL_USERNAME, KAFKA_TLS_ENABLE, KAFKA_TOPIC, MYSQL_DSN_STRING, MYSQL_MAX_OPEN_CONNECTIONS, - MYSQL_QUEUE_DIR, MYSQL_TABLE, MYSQL_TLS_CA, MYSQL_TLS_CLIENT_CERT, MYSQL_TLS_CLIENT_KEY, POSTGRES_DSN_STRING, + MYSQL_QUEUE_DIR, MYSQL_TABLE, MYSQL_TLS_CA, MYSQL_TLS_CLIENT_CERT, MYSQL_TLS_CLIENT_KEY, NATS_ADDRESS, + NATS_JETSTREAM_ACK_TIMEOUT_SECS, NATS_JETSTREAM_ENABLE, NATS_JETSTREAM_STREAM_NAME, NATS_SUBJECT, POSTGRES_DSN_STRING, POSTGRES_FORMAT, POSTGRES_QUEUE_DIR, POSTGRES_TABLE, POSTGRES_TLS_CA, POSTGRES_TLS_CLIENT_CERT, POSTGRES_TLS_CLIENT_KEY, REDIS_CHANNEL, REDIS_CONNECTION_TIMEOUT, REDIS_MAX_RETRY_DELAY, REDIS_MIN_RETRY_DELAY, REDIS_PIPELINE_BUFFER_SIZE, REDIS_RECONNECT_RETRY_ATTEMPTS, REDIS_RESPONSE_TIMEOUT, REDIS_TLS_ALLOW_INSECURE, REDIS_URL, WEBHOOK_ENDPOINT, @@ -1157,4 +1164,68 @@ mod tests { let err = validate_postgres_config(&config, "").expect_err("relative tls_client_cert should fail"); assert!(err.to_string().contains("must be an absolute path")); } + + fn nats_base_config() -> KVS { + let mut config = KVS::new(); + config.insert(NATS_ADDRESS.to_string(), "nats://127.0.0.1:4222".to_string()); + config.insert(NATS_SUBJECT.to_string(), "rustfs.events".to_string()); + config + } + + // Absolute on Linux, macOS, and Windows. temp_dir needs no filesystem to exist for a + // validation-only test, and Path::is_absolute stays true across platforms. + fn nats_queue_dir() -> String { + std::env::temp_dir().join("rustfs-nats-queue").to_string_lossy().into_owned() + } + + #[test] + fn build_nats_args_parses_jetstream_keys() { + let mut config = nats_base_config(); + config.insert(NATS_JETSTREAM_ENABLE.to_string(), "on".to_string()); + config.insert(NATS_JETSTREAM_STREAM_NAME.to_string(), "RUSTFS_EVENTS".to_string()); + config.insert(NATS_JETSTREAM_ACK_TIMEOUT_SECS.to_string(), "45".to_string()); + + for target_type in [TargetType::NotifyEvent, TargetType::AuditLog] { + let args = build_nats_args(&config, &nats_queue_dir(), target_type).expect("jetstream keys parse"); + assert_eq!(args.jetstream_enable, Some(true)); + assert_eq!(args.jetstream_stream_name.as_deref(), Some("RUSTFS_EVENTS")); + assert_eq!(args.jetstream_ack_timeout_secs, Some(45)); + } + } + + #[test] + fn build_nats_args_rejects_an_invalid_jetstream_enable_value() { + // A typo in the enable flag is a configuration error naming the key, never a silent + // disable that drops the durability guarantee the operator asked for. + let mut config = nats_base_config(); + config.insert(NATS_JETSTREAM_ENABLE.to_string(), "t".to_string()); + + let err = build_nats_args(&config, &nats_queue_dir(), TargetType::NotifyEvent) + .expect_err("an unparsable enable value fails the build"); + assert!(err.to_string().contains(NATS_JETSTREAM_ENABLE), "the error names the key: {err}"); + } + + #[test] + fn build_nats_args_without_jetstream_keys_is_flag_off() { + let config = nats_base_config(); + let args = build_nats_args(&config, &nats_queue_dir(), TargetType::NotifyEvent).expect("baseline config parses"); + + assert_eq!(args.jetstream_enable, None); + assert_eq!(args.jetstream_stream_name, None); + assert_eq!(args.jetstream_ack_timeout_secs, None); + assert_eq!(args.address, "nats://127.0.0.1:4222"); + assert_eq!(args.subject, "rustfs.events"); + assert_eq!(args.queue_dir, nats_queue_dir()); + args.validate().expect("flag-off config validates"); + } + + #[test] + fn build_nats_args_rejects_non_integer_ack_timeout() { + let mut config = nats_base_config(); + config.insert(NATS_JETSTREAM_ACK_TIMEOUT_SECS.to_string(), "soon".to_string()); + + let err = build_nats_args(&config, &nats_queue_dir(), TargetType::NotifyEvent) + .expect_err("non-integer ack timeout should be rejected"); + assert!(err.to_string().contains("whole number of seconds")); + } } diff --git a/crates/targets/src/error.rs b/crates/targets/src/error.rs index c054cd3b2..bd29f02ca 100644 --- a/crates/targets/src/error.rs +++ b/crates/targets/src/error.rs @@ -52,6 +52,9 @@ pub enum TargetError { #[error("Request error: {0}")] Request(String), + #[error("JetStream publish error ({}): {detail}", if *.retryable { "retryable" } else { "terminal" })] + JetStreamPublish { retryable: bool, detail: String }, + #[error("Timeout error: {0}")] Timeout(String), diff --git a/crates/targets/src/runtime/mod.rs b/crates/targets/src/runtime/mod.rs index 4db68527f..4f83740eb 100644 --- a/crates/targets/src/runtime/mod.rs +++ b/crates/targets/src/runtime/mod.rs @@ -33,6 +33,37 @@ use std::{future::Future, pin::Pin, time::Duration}; use tokio::sync::{Semaphore, mpsc}; use tokio::task::JoinHandle; +/// Maximum number of replay attempts before a stored entry is exhausted. Each attempt runs one full +/// send (one ack wait at the configured timeout for a JetStream entry), then a backoff sleep before +/// the next. The JetStream duplicate-window validation derives its worst-case retry lifetime from the +/// attempt count and REPLAY_BASE_RETRY_DELAY through inter_attempt_backoff_sum, the same source the +/// sleep schedule builds on. Pinning tests hold each layer of the coupling: the shared per-attempt +/// term, the sum over the schedule, retry_lifetime against that sum, and the realized sleep in +/// replay_backoff_sleep under a paused clock. +pub(crate) const REPLAY_MAX_RETRIES: usize = 5; + +/// Base unit of the exponential replay backoff. The sleep before the retry at shift n is this +/// multiplied by 2^n. +pub(crate) const REPLAY_BASE_RETRY_DELAY: Duration = Duration::from_secs(2); + +/// Backoff sleep before the retry attempt at `shift`: REPLAY_BASE_RETRY_DELAY doubled `shift` times. +/// The single per-attempt term the replay sleep and the duplicate-window sum both derive from. +pub(crate) fn replay_backoff_term(shift: u32) -> Duration { + REPLAY_BASE_RETRY_DELAY.saturating_mul(1u32 << shift) +} + +/// Sum of the backoff sleeps that run between `attempts` replay sends. The retry at shift k is +/// preceded by replay_backoff_term(k) for k in 1..attempts, so attempts minus one terms contribute +/// and no sleep follows the final attempt. retry_lifetime derives its worst-case span from the same +/// sum, so the sleep schedule and the stream duplicate-window requirement move together. +pub(crate) fn inter_attempt_backoff_sum(attempts: usize) -> Duration { + let mut total = Duration::ZERO; + for shift in 1..attempts as u32 { + total = total.saturating_add(replay_backoff_term(shift)); + } + total +} + /// Shared target trait object used by the runtime manager. pub type SharedTarget = Arc + Send + Sync>; type ReplayHook = Arc) -> Pin + Send>> + Send + Sync>; @@ -163,6 +194,7 @@ pub struct RuntimeStatusSnapshot { #[derive(Debug, Clone, Default, PartialEq, Eq)] pub struct RuntimeTargetSnapshot { pub failed_messages: u64, + pub failed_store_length: u64, pub queue_length: u64, pub target_id: String, pub target_type: String, @@ -211,6 +243,7 @@ where target: SharedTarget, }, RetryExhausted { + detail: String, key: Key, target: SharedTarget, }, @@ -302,11 +335,9 @@ where self.remove_and_close(&target_id.to_string()).await } - /// Closes and removes every target, returning the id/error of each target - /// whose `close()` failed. Previously these flush/close errors were logged - /// and dropped, so a shutdown that failed to flush a target reported success. - /// Callers can now surface them (e.g. fail an explicit shutdown) while still - /// tearing down the rest of the runtime. + /// Closes and removes every target, returning the id and error of each target whose close failed. + /// Surfacing them lets a caller fail an explicit shutdown while still tearing down the rest of the + /// runtime. pub async fn clear_and_close(&mut self) -> Vec<(String, TargetError)> { let target_ids: Vec = self.targets.keys().cloned().collect(); let mut errors = Vec::new(); @@ -388,6 +419,7 @@ where fn snapshot_from_delivery(target_id: TargetID, delivery: TargetDeliverySnapshot) -> RuntimeTargetSnapshot { RuntimeTargetSnapshot { failed_messages: delivery.failed_messages, + failed_store_length: delivery.failed_store_length, queue_length: delivery.queue_length, target_id: target_id.to_string(), target_type: target_id.name, @@ -485,10 +517,8 @@ where (cancel_tx, join) } -/// Number of readable entries accumulated before a replay batch is flushed under -/// a single semaphore permit. The previous `!batch_keys.is_empty()` flush -/// condition was always true, so this effectively defaulted to 1 (a permit per -/// entry) and made `batch_timeout` dead code. +/// Number of readable entries accumulated before a replay batch is flushed under a single semaphore +/// permit. const REPLAY_BATCH_SIZE: usize = 16; /// Sleeps for `dur` unless a cancel signal arrives first. Returns `true` if @@ -503,6 +533,14 @@ async fn sleep_or_cancelled(dur: Duration, cancel_rx: &mut mpsc::Receiver<()>) - } } +/// Seeds an interval tracker one interval in the past so the first eligible tick runs the action +/// immediately. The monotonic clock starts at host boot, so a process started within one interval +/// of boot cannot represent an instant that far back. The seed falls back to now in that case and +/// the first run waits one full interval. +fn seed_interval_start(now: tokio::time::Instant, interval: Duration) -> tokio::time::Instant { + now.checked_sub(interval).unwrap_or(now) +} + async fn stream_replay_worker( store: &mut (dyn Store + Send), target: SharedTarget, @@ -514,18 +552,49 @@ async fn stream_replay_worker( ) where E: PluginEvent, { + // Lower bound between two failed-store expired-entry removal scans. Far below the retention TTL, + // so retention is unaffected, while keeping the read_dir scan off every idle tick. + const FAILED_STORE_PRUNE_INTERVAL: Duration = Duration::from_secs(60); + let mut batch_keys = Vec::with_capacity(REPLAY_BATCH_SIZE); let mut last_flush = tokio::time::Instant::now(); + // Seeded one interval back so the first eligible tick runs the removal. Near host boot the + // seed is now and the first removal waits one interval. + let mut last_prune = seed_interval_start(tokio::time::Instant::now(), FAILED_STORE_PRUNE_INTERVAL); loop { if cancel_rx.try_recv().is_ok() { return; } + // Remove expired failed-store entries on the replay tick rather than a separate timer, at most + // once per interval so the idle path does not scan the directory on every tick. Only a target + // that records terminal failures carries a failed store, so a target without one skips the scan. + if let Some(failed_store) = target.failed_store() + && last_prune.elapsed() >= FAILED_STORE_PRUNE_INTERVAL + { + // Run the stat-and-sort scan off the async worker thread. The owned handle shares the same + // directory and cached-count state through its Arc handles, so the scan reconciles the + // count the live handles read. + let maintenance_failed = failed_store.boxed_clone_failed(); + let outcome = tokio::task::spawn_blocking(move || maintenance_failed.prune_failed_store()).await; + match outcome { + Ok(Err(err)) => { + tracing::warn!(target_id = %target.id(), error = %err, "Failed to prune the failed-events store"); + } + Ok(Ok(_)) => {} + Err(join_err) => { + tracing::warn!(target_id = %target.id(), error = %join_err, "The failed-events maintenance task failed to join"); + } + } + last_prune = tokio::time::Instant::now(); + } + let keys = store.list(); if keys.is_empty() { if !batch_keys.is_empty() && last_flush.elapsed() >= batch_timeout { - if process_replay_batch(&mut batch_keys, target.clone(), &hook, semaphore.clone(), &mut cancel_rx).await { + if process_replay_batch(&*store, &mut batch_keys, target.clone(), &hook, semaphore.clone(), &mut cancel_rx).await + { return; } last_flush = tokio::time::Instant::now(); @@ -539,7 +608,8 @@ async fn stream_replay_worker( for key in keys { if cancel_rx.try_recv().is_ok() { if !batch_keys.is_empty() { - process_replay_batch(&mut batch_keys, target.clone(), &hook, semaphore.clone(), &mut cancel_rx).await; + process_replay_batch(&*store, &mut batch_keys, target.clone(), &hook, semaphore.clone(), &mut cancel_rx) + .await; } return; } @@ -573,25 +643,37 @@ async fn stream_replay_worker( // Flush once a full batch has accumulated or the batch has aged past // batch_timeout — real size/time-based batching, not once-per-entry. if batch_keys.len() >= REPLAY_BATCH_SIZE || last_flush.elapsed() >= batch_timeout { - if process_replay_batch(&mut batch_keys, target.clone(), &hook, semaphore.clone(), &mut cancel_rx).await { + if process_replay_batch(&*store, &mut batch_keys, target.clone(), &hook, semaphore.clone(), &mut cancel_rx).await + { return; } last_flush = tokio::time::Instant::now(); } } + // Flush a partial batch that has aged past batch_timeout, so a lone or slow-arriving entry is + // delivered rather than stranded waiting for a full batch to accumulate. + if !batch_keys.is_empty() && last_flush.elapsed() >= batch_timeout { + if process_replay_batch(&*store, &mut batch_keys, target.clone(), &hook, semaphore.clone(), &mut cancel_rx).await { + return; + } + last_flush = tokio::time::Instant::now(); + } + if sleep_or_cancelled(Duration::from_millis(100), &mut cancel_rx).await { return; } } } -/// Delivers a batch of queued entries under a single semaphore permit. +/// Delivers a batch of queued entries, each send bounded by a freshly acquired semaphore permit held +/// across the send and released before any backoff sleep or failed-store move. /// /// Returns `true` if a cancel signal was observed while processing (e.g. during /// retry backoff), so the caller can stop promptly instead of continuing to /// drain a store that a replacement worker may already own. async fn process_replay_batch( + store: &(dyn Store + Send), batch_keys: &mut Vec, target: SharedTarget, hook: &ReplayHook, @@ -601,32 +683,41 @@ async fn process_replay_batch( where E: PluginEvent, { - const MAX_RETRIES: usize = 5; - const BASE_RETRY_DELAY: Duration = Duration::from_secs(2); + const MAX_RETRIES: usize = REPLAY_MAX_RETRIES; if batch_keys.is_empty() { return false; } - let _permit = match semaphore { - Some(ref semaphore) => match semaphore.clone().acquire_owned().await { - Ok(permit) => Some(permit), - Err(err) => { - tracing::error!(error = %err, "Failed to acquire replay semaphore permit"); - batch_keys.clear(); - return false; - } - }, - None => None, - }; - let mut cancelled = false; 'keys: for key in batch_keys.iter() { let mut retry_count = 0usize; let mut success = false; + let mut last_retryable_detail = String::new(); while retry_count < MAX_RETRIES && !success { - match target.send_from_store(key.clone()).await { + // The permit bounds how many replay sends run concurrently across targets sharing the + // semaphore. It is held across the send await so the cap is real, then released before + // any backoff sleep or failed-store move, because the sleep blocks for the backoff and + // holding the shared permit across it would throttle the whole replay path. An absent + // semaphore means no bound (the audit path). + let permit = match &semaphore { + None => None, + Some(sem) => match sem.clone().acquire_owned().await { + Ok(permit) => Some(permit), + Err(err) => { + tracing::error!(error = %err, "Failed to acquire replay semaphore permit"); + // Drop the batch so its keys do not strand under the dedup guard. + batch_keys.clear(); + return cancelled; + } + }, + }; + + let result = target.send_from_store(key.clone()).await; + drop(permit); + + match result { Ok(_) => { hook(ReplayEvent::Delivered { key: key.clone(), @@ -636,8 +727,11 @@ where success = true; } Err(err) => match err { - TargetError::NotConnected | TargetError::Timeout(_) => { + TargetError::NotConnected + | TargetError::Timeout(_) + | TargetError::JetStreamPublish { retryable: true, .. } => { retry_count += 1; + last_retryable_detail = err.to_string(); hook(ReplayEvent::RetryableError { error: err, key: key.clone(), @@ -645,16 +739,29 @@ where target: target.clone(), }) .await; - - let jitter = Duration::from_millis(key.to_string().len() as u64 % 500); - let backoff = 1u32 << retry_count as u32; - // Observe cancellation during backoff so shutdown/reload is - // not blocked for the full (potentially many-second) delay. - if sleep_or_cancelled(BASE_RETRY_DELAY * backoff + jitter, cancel_rx).await { + // The backoff runs only when another attempt follows, so the final failure + // proceeds straight to the exhaustion handling. Cancellation is observed + // during the sleep so shutdown/reload is not blocked for the full + // (potentially many-second) delay. + if retry_count < MAX_RETRIES && replay_backoff_sleep(key, retry_count, cancel_rx).await { cancelled = true; break 'keys; } } + TargetError::JetStreamPublish { retryable: false, .. } => { + // The hook fires only on a completed move. On a failed move the entry stays + // live, the next scan retries the move, and a repaired failed-store + // directory heals without a restart, with the hook then firing exactly once. + if move_failed_entry(store, &target, key, &err, retry_count as u32).await { + hook(ReplayEvent::PermanentFailure { + error: err, + key: key.clone(), + target: target.clone(), + }) + .await; + } + break; + } TargetError::Dropped(reason) => { hook(ReplayEvent::Dropped { key: key.clone(), @@ -678,7 +785,10 @@ where } if retry_count >= MAX_RETRIES && !success { + // A retryable error that never succeeded within the bound leaves the entry on the live + // queue for the next scan. The exhaustion is reported through the hook for metrics. hook(ReplayEvent::RetryExhausted { + detail: last_retryable_detail, key: key.clone(), target: target.clone(), }) @@ -690,6 +800,30 @@ where cancelled } +/// Sleeps the exponential backoff for a retry attempt, with key-derived jitter. Returns `true` if a +/// cancel signal arrives first, so the caller can stop promptly instead of blocking for the full delay. +async fn replay_backoff_sleep(key: &Key, retry_count: usize, cancel_rx: &mut mpsc::Receiver<()>) -> bool { + let jitter = Duration::from_millis(key.to_string().len() as u64 % 500); + sleep_or_cancelled(replay_backoff_term(retry_count as u32) + jitter, cancel_rx).await +} + +/// Reacts to a terminal classification by delegating to the target's terminal-failure handling and +/// reports whether the entry was handled. The target moves the entry to its failed-events store and +/// reports true, or declines and reports false so the caller keeps the entry live and skips the +/// final-failure hook. A target without a terminal-failure store always declines. +async fn move_failed_entry( + store: &(dyn Store + Send), + target: &SharedTarget, + key: &Key, + error: &TargetError, + retry_count: u32, +) -> bool +where + E: PluginEvent, +{ + target.handle_terminal_failure(store, key, error, retry_count).await +} + #[cfg(test)] mod tests { use super::TargetRuntimeManager; @@ -703,6 +837,66 @@ mod tests { use std::sync::Arc; use std::sync::atomic::{AtomicUsize, Ordering}; + #[tokio::test(start_paused = true)] + async fn seed_interval_start_backdates_by_one_interval() { + // With one interval of clock history available, the seed is exactly one interval in the + // past, so the first eligible tick runs the action immediately. + let origin = tokio::time::Instant::now(); + tokio::time::advance(std::time::Duration::from_secs(60)).await; + let now = tokio::time::Instant::now(); + assert_eq!(super::seed_interval_start(now, std::time::Duration::from_secs(60)), origin); + } + + #[test] + fn seed_interval_start_falls_back_to_now_when_the_clock_is_too_young() { + // Duration::MAX reaches past the monotonic clock origin on any host, so the checked + // subtraction yields no instant and the seed falls back to now instead of panicking. + let now = tokio::time::Instant::now(); + assert_eq!(super::seed_interval_start(now, std::time::Duration::MAX), now); + } + + #[test] + fn inter_attempt_backoff_sum_matches_the_realized_sleep_schedule() { + // Sums the shared per-attempt term over the schedule shifts (1..REPLAY_MAX_RETRIES, no + // trailing sleep after the last attempt) and asserts it equals inter_attempt_backoff_sum. + // The sleep's own use of the term is pinned by the paused-clock test below. + let mut realized = std::time::Duration::ZERO; + for shift in 1..super::REPLAY_MAX_RETRIES as u32 { + realized += super::replay_backoff_term(shift); + } + assert_eq!(super::inter_attempt_backoff_sum(super::REPLAY_MAX_RETRIES), realized); + // 2s * (2 + 4 + 8 + 16) at the default base. + assert_eq!(realized, std::time::Duration::from_secs(60)); + } + + #[tokio::test(start_paused = true)] + async fn replay_backoff_sleep_sleeps_the_term_for_the_retry_plus_key_jitter() { + // Drives the sleep itself, so a shifted argument at its replay_backoff_term call (for + // example retry_count minus 1) fails here even though the term and sum tests still pass. + // The jitter is deterministic, the key string length modulo 500 milliseconds, so the + // virtual elapsed time is exact under the paused clock. + let key = Key { + name: "0198c0de-0000-7000-8000-000000000000".to_string(), + extension: ".event".to_string(), + item_count: 1, + compress: false, + }; + let jitter = std::time::Duration::from_millis(key.to_string().len() as u64 % 500); + let retry_count = 3usize; + + // The held sender keeps the cancel channel open so the sleep runs to its deadline. + let (_cancel_tx, mut cancel_rx) = tokio::sync::mpsc::channel::<()>(1); + let start = tokio::time::Instant::now(); + let cancelled = super::replay_backoff_sleep(&key, retry_count, &mut cancel_rx).await; + + assert!(!cancelled, "no cancel signal was sent"); + assert_eq!( + start.elapsed(), + super::replay_backoff_term(retry_count as u32) + jitter, + "the sleep is the term for this retry count plus the key-derived jitter" + ); + } + #[derive(Clone)] struct TestTarget { id: TargetID, @@ -835,4 +1029,1003 @@ mod tests { assert!(manager.is_empty()); assert!(exited.load(Ordering::SeqCst), "stop_all must await the worker to completion"); } + + mod classifier { + use super::super::{ReplayEvent, stream_replay_worker}; + use crate::arn::TargetID; + use crate::plugin::PluginEvent; + use crate::store::{FailedEventStore, Key, QueueStore, Store}; + use crate::target::{EntityTarget, QueuedPayload, QueuedPayloadMeta}; + use crate::{StoreError, Target, TargetError}; + use async_trait::async_trait; + use rustfs_s3_types::EventName; + use std::sync::Arc; + use std::sync::atomic::{AtomicUsize, Ordering}; + use std::time::Duration; + use tokio::sync::{Semaphore, mpsc}; + + /// Records each ReplayEvent class the worker emitted, so a test can assert no event class was + /// silently skipped. + #[derive(Default)] + struct ReplayTally { + delivered: AtomicUsize, + retryable: AtomicUsize, + dropped: AtomicUsize, + permanent_failure: AtomicUsize, + retry_exhausted: AtomicUsize, + unreadable: AtomicUsize, + } + + /// A target whose send_from_store returns a programmed error each call, so the replay + /// classifier can be exercised per error class against a real on-disk store. + #[derive(Clone)] + struct ProgrammedTarget { + id: TargetID, + error: Option>, + send_calls: Arc, + in_flight: Arc, + max_in_flight: Arc, + send_gate: Arc, + // A clone of the worker's store so the mock can delete on the Dropped or success path, as + // the real send_from_store does, rather than leaving the entry to be re-listed forever. + store: QueueStore, + // The failed-events handle the terminal move writes to. Defaults to the store itself and is + // overridden with a rejecting handle to model an unwritable failed store. + failed_handle: Arc, + } + + impl ProgrammedTarget { + fn new(error: Option, store: QueueStore) -> Self { + let failed_handle: Arc = Arc::new(store.clone()); + Self::new_with_failed_store(error, store, failed_handle) + } + + fn new_with_failed_store( + error: Option, + store: QueueStore, + failed_handle: Arc, + ) -> Self { + let send_gate = Arc::new(Semaphore::new(Semaphore::MAX_PERMITS)); + Self { + id: TargetID::new("target-a".to_string(), "nats".to_string()), + error: error.map(Arc::new), + send_calls: Arc::new(AtomicUsize::new(0)), + in_flight: Arc::new(AtomicUsize::new(0)), + max_in_flight: Arc::new(AtomicUsize::new(0)), + send_gate, + store, + failed_handle, + } + } + } + + /// Models the NATS target's terminal move for the classifier tests: reads the entry, encodes it + /// as a terminal failed entry, writes it to the failed-events store, then clears the live entry. + /// A failed write leaves the live entry in place for the next scan. The delete outcome mirrors + /// the production move, so a delete failure returns false rather than reporting the entry handled. + fn move_terminal_entry_for_test( + store: &(dyn Store + Send), + failed_store: &dyn FailedEventStore, + key: &Key, + error: &TargetError, + retry_count: u32, + ) -> bool { + let raw = match store.get_raw(key) { + Ok(raw) => raw, + Err(StoreError::NotFound) => return true, + Err(_) => return false, + }; + let Ok(queued) = QueuedPayload::decode(&raw) else { + return false; + }; + let resolved = crate::target::nats::resolve_dedup_id(&queued.meta.dedup_id, key); + let Ok(encoded) = crate::target::encode_failed_entry( + queued, + crate::target::FailedErrorClass::Terminal, + error, + retry_count, + &resolved, + ) else { + return false; + }; + if failed_store.put_failed_raw(&key.name, &encoded).is_err() { + return false; + } + // A missing entry is not a failure, any other delete error declines the move, matching the + // production delete which treats NotFound as done and propagates every other error. + matches!(store.del(key), Ok(()) | Err(StoreError::NotFound)) + } + + #[async_trait] + impl Target for ProgrammedTarget + where + E: PluginEvent, + { + fn id(&self) -> TargetID { + self.id.clone() + } + async fn is_active(&self) -> Result { + Ok(true) + } + async fn save(&self, _event: Arc>) -> Result<(), TargetError> { + Ok(()) + } + async fn send_raw_from_store(&self, _key: Key, _body: Vec, _meta: QueuedPayloadMeta) -> Result<(), TargetError> { + Ok(()) + } + async fn send_from_store(&self, key: Key) -> Result<(), TargetError> { + self.send_calls.fetch_add(1, Ordering::SeqCst); + let live = self.in_flight.fetch_add(1, Ordering::SeqCst) + 1; + self.max_in_flight.fetch_max(live, Ordering::SeqCst); + // The gate lets a test hold a send in flight to observe permit behaviour during the + // await. It is open by default. + let permit = self.send_gate.clone().acquire_owned().await; + drop(permit); + self.in_flight.fetch_sub(1, Ordering::SeqCst); + match self.error.as_deref() { + // A Dropped error means send_from_store already removed the invalid entry before + // returning, so the mock deletes it too. A retryable or terminal publish error + // leaves the entry in place, matching the real path where the entry is cleared only + // on Ok or by the failed-store move. + Some(TargetError::Dropped(reason)) => { + let _ = self.store.del(&key); + Err(TargetError::Dropped(reason.clone())) + } + Some(error) => Err(clone_error(error)), + None => { + // Success clears the entry, as the real default implementation does. + let _ = self.store.del(&key); + Ok(()) + } + } + } + async fn close(&self) -> Result<(), TargetError> { + Ok(()) + } + fn store(&self) -> Option<&(dyn Store + Send + Sync)> { + None + } + async fn handle_terminal_failure( + &self, + store: &(dyn Store + Send), + key: &Key, + error: &TargetError, + retry_count: u32, + ) -> bool { + move_terminal_entry_for_test(store, self.failed_handle.as_ref(), key, error, retry_count) + } + fn clone_dyn(&self) -> Box + Send + Sync> { + Box::new(self.clone()) + } + fn is_enabled(&self) -> bool { + true + } + } + + /// A store that delegates every operation to a real on-disk QueueStore but fails + /// put_failed_raw or del while the matching flag is set, modeling an unwritable failed-store + /// directory or a live queue whose delete fails, either of which an operator later repairs. + #[derive(Clone)] + struct FailingFailedStore { + inner: QueueStore, + fail_put_failed_raw: Arc, + fail_del: Arc, + } + + impl Store for FailingFailedStore { + type Error = StoreError; + type Key = Key; + + fn open(&self) -> Result<(), Self::Error> { + self.inner.open() + } + fn put(&self, item: Arc) -> Result { + self.inner.put(item) + } + fn put_multiple(&self, items: Vec) -> Result { + self.inner.put_multiple(items) + } + fn put_raw(&self, data: &[u8]) -> Result { + self.inner.put_raw(data) + } + fn get(&self, key: &Self::Key) -> Result { + self.inner.get(key) + } + fn get_multiple(&self, key: &Self::Key) -> Result, Self::Error> { + self.inner.get_multiple(key) + } + fn get_raw(&self, key: &Self::Key) -> Result, Self::Error> { + self.inner.get_raw(key) + } + fn del(&self, key: &Self::Key) -> Result<(), Self::Error> { + if self.fail_del.load(Ordering::SeqCst) { + return Err(StoreError::Internal("injected del failure".to_string())); + } + self.inner.del(key) + } + fn delete(&self) -> Result<(), Self::Error> { + self.inner.delete() + } + fn list(&self) -> Vec { + self.inner.list() + } + fn len(&self) -> usize { + self.inner.len() + } + fn is_empty(&self) -> bool { + self.inner.is_empty() + } + fn boxed_clone(&self) -> Box + Send + Sync> { + Box::new(self.clone()) + } + } + + impl FailedEventStore for FailingFailedStore { + fn put_failed_raw(&self, entry_name: &str, data: &[u8]) -> Result { + if self.fail_put_failed_raw.load(Ordering::SeqCst) { + return Err(StoreError::Internal("injected put_failed_raw failure".to_string())); + } + self.inner.put_failed_raw(entry_name, data) + } + fn prune_failed_store(&self) -> Result { + self.inner.prune_failed_store() + } + fn failed_len(&self) -> usize { + self.inner.failed_len() + } + fn boxed_clone_failed(&self) -> Box { + Box::new(self.clone()) + } + } + + fn clone_error(error: &TargetError) -> TargetError { + match error { + TargetError::NotConnected => TargetError::NotConnected, + TargetError::Timeout(value) => TargetError::Timeout(value.clone()), + TargetError::Dropped(value) => TargetError::Dropped(value.clone()), + TargetError::JetStreamPublish { retryable, detail } => TargetError::JetStreamPublish { + retryable: *retryable, + detail: detail.clone(), + }, + TargetError::Network(value) => TargetError::Network(value.clone()), + other => TargetError::Unknown(other.to_string()), + } + } + + fn temp_dir(name: &str) -> std::path::PathBuf { + std::env::temp_dir().join(format!("rustfs-runtime-{name}-{}", uuid::Uuid::new_v4())) + } + + fn seed_entry(store: &QueueStore, dedup_id: &str) -> Key { + let mut meta = QueuedPayloadMeta::new( + EventName::ObjectCreatedPut, + "bucket-a".to_string(), + "obj.txt".to_string(), + "application/json", + 7, + ); + meta.dedup_id = dedup_id.to_string(); + let payload = QueuedPayload::new(meta, br#"{"x":1}"#.to_vec()); + store.put_raw(&payload.encode().unwrap()).unwrap() + } + + /// Runs the replay worker against the seeded store for up to 600 virtual seconds, then cancels. + /// + /// The worker runs on a store clone (the QueueStore shares its state through an Arc), so the + /// caller's handle observes drain while the worker holds its own mutable borrow. Tests run with + /// a paused clock, so the production backoff sleeps cost no real time and the loop drains fast. + async fn run_worker(store: &mut QueueStore, target: Arc, tally: Arc) { + run_worker_until(store, target, tally, |_| false).await; + } + + /// Runs the replay worker like run_worker, polling the tally once per virtual second and + /// cancelling as soon as the stop condition holds, so a test over an entry that stays live + /// can observe exactly one retry cycle instead of every cycle the full window admits. + async fn run_worker_until( + store: &mut QueueStore, + target: Arc, + tally: Arc, + stop: impl Fn(&ReplayTally) -> bool, + ) { + let shared: Arc + Send + Sync> = target.clone(); + let (cancel_tx, cancel_rx) = mpsc::channel(1); + let hook = { + let tally = tally.clone(); + Arc::new(move |event: ReplayEvent| { + let tally = tally.clone(); + Box::pin(async move { + match event { + ReplayEvent::Delivered { .. } => tally.delivered.fetch_add(1, Ordering::SeqCst), + ReplayEvent::RetryableError { .. } => tally.retryable.fetch_add(1, Ordering::SeqCst), + ReplayEvent::Dropped { .. } => tally.dropped.fetch_add(1, Ordering::SeqCst), + ReplayEvent::PermanentFailure { .. } => tally.permanent_failure.fetch_add(1, Ordering::SeqCst), + ReplayEvent::RetryExhausted { .. } => tally.retry_exhausted.fetch_add(1, Ordering::SeqCst), + ReplayEvent::UnreadableEntry { .. } => tally.unreadable.fetch_add(1, Ordering::SeqCst), + }; + }) as std::pin::Pin + Send>> + }) as super::super::ReplayHook + }; + + let semaphore = Some(Arc::new(Semaphore::new(1))); + let mut worker_store = store.clone(); + let worker = tokio::spawn(async move { + stream_replay_worker( + &mut worker_store, + shared, + cancel_rx, + hook, + semaphore, + Duration::from_millis(10), + Duration::from_millis(10), + ) + .await; + }); + + // The paused clock makes the production backoff sleeps free, so the virtual window lets + // the worker fully process the seeded entry (a full retry exhaustion is about two + // virtual minutes) before the worker is cancelled. The once-per-virtual-second poll + // cancels within one second of the stop condition, well inside a retry cycle. + for _ in 0..600 { + if stop(&tally) { + break; + } + tokio::time::sleep(Duration::from_secs(1)).await; + } + let _ = cancel_tx.send(()).await; + let _ = worker.await; + } + + // A retryable JetStream publish error retries within the bound and, on exhaustion, leaves the + // entry on the live queue rather than moving it to the failed store. Covers TimedOut, + // BrokenPipe, MaxAckPending, and the bounded StreamNotFound path, all of which the classifier + // carries as a retryable publish error. + #[tokio::test(start_paused = true)] + async fn retryable_publish_error_retries_and_keeps_the_entry_live() { + for detail in ["timed out", "broken pipe", "max ack pending", "stream not found"] { + let dir = temp_dir(&format!("retryable-{}", detail.replace(' ', "-"))); + let mut store = QueueStore::::new_with_compression(&dir, 64, ".test", false); + store.open().unwrap(); + seed_entry(&store, "minted-id"); + + let target = Arc::new(ProgrammedTarget::new( + Some(TargetError::JetStreamPublish { + retryable: true, + detail: detail.to_string(), + }), + store.clone(), + )); + let tally = Arc::new(ReplayTally::default()); + // The entry stays live and is rescanned after exhaustion, so the run stops at the + // first exhaustion to observe one full retry cycle. + run_worker_until(&mut store, target.clone(), tally.clone(), |t| { + t.retry_exhausted.load(Ordering::SeqCst) >= 1 + }) + .await; + + assert!(tally.retryable.load(Ordering::SeqCst) >= 1, "{detail} retries at least once"); + assert_eq!(tally.retry_exhausted.load(Ordering::SeqCst), 1, "{detail} exhausts the bound once"); + assert_eq!(store.len(), 1, "{detail} keeps the entry on the live queue"); + assert_eq!(store.failed_len(), 0, "{detail} writes no failed entry"); + assert!(!dir.join("failed").exists(), "{detail} creates no failed directory"); + + let _ = store.delete(); + } + } + + // A terminal JetStream publish error writes a terminal failed entry on the first attempt with + // no retry, and the live entry is cleared. Covers MaxPayloadExceeded, WrongLastMessageId, + // WrongLastSequence, and the terminal Other code, all carried as a non-retryable publish error. + #[tokio::test(start_paused = true)] + async fn terminal_publish_error_writes_terminal_entry_without_retry() { + for detail in [ + "max payload exceeded", + "wrong last message id", + "wrong last sequence", + "stream sealed", + ] { + let dir = temp_dir(&format!("terminal-{}", detail.replace(' ', "-"))); + let mut store = QueueStore::::new_with_compression(&dir, 64, ".test", false); + store.open().unwrap(); + seed_entry(&store, "minted-id"); + + let target = Arc::new(ProgrammedTarget::new( + Some(TargetError::JetStreamPublish { + retryable: false, + detail: detail.to_string(), + }), + store.clone(), + )); + let tally = Arc::new(ReplayTally::default()); + run_worker(&mut store, target.clone(), tally.clone()).await; + + assert_eq!(tally.permanent_failure.load(Ordering::SeqCst), 1, "{detail} fails permanently once"); + assert_eq!(tally.retryable.load(Ordering::SeqCst), 0, "{detail} does not retry"); + assert_eq!(tally.retry_exhausted.load(Ordering::SeqCst), 0, "{detail} does not reach exhaustion"); + assert_eq!(store.len(), 0, "{detail} clears the live entry"); + assert_eq!(store.failed_len(), 1, "{detail} writes one terminal failed entry"); + + // The failed entry carries the terminal class and the full meta. + let failed_dir = dir.join("failed"); + let failed_file = std::fs::read_dir(&failed_dir).unwrap().next().unwrap().unwrap().path(); + let decoded = QueuedPayload::decode(&std::fs::read(&failed_file).unwrap()).unwrap(); + let failure = decoded.meta.failure.unwrap(); + assert_eq!(failure.error_class, crate::target::FailedErrorClass::Terminal); + assert_eq!(failure.nats_msg_id, "minted-id"); + assert_eq!(decoded.meta.bucket_name, "bucket-a"); + + let _ = store.delete(); + } + } + + // A Dropped entry is reported through the drop hook rather than silently discarded, and it + // writes no failed entry. + #[tokio::test(start_paused = true)] + async fn no_replay_path_silently_drops_an_event() { + // A Dropped error is recorded through the drop hook and writes no failed entry, because a + // dropped payload is an invalid entry the store itself removed, not a deliverable event. + let dir = temp_dir("nodrop-dropped"); + let mut store = QueueStore::::new_with_compression(&dir, 64, ".test", false); + store.open().unwrap(); + seed_entry(&store, "id"); + let target = Arc::new(ProgrammedTarget::new( + Some(TargetError::Dropped("invalid entry".to_string())), + store.clone(), + )); + let tally = Arc::new(ReplayTally::default()); + run_worker(&mut store, target.clone(), tally.clone()).await; + assert_eq!(tally.dropped.load(Ordering::SeqCst), 1, "a dropped entry is reported, not silent"); + assert_eq!(store.failed_len(), 0, "a dropped invalid entry writes no failed entry"); + + let _ = store.delete(); + } + + // The audit path and the notify path monomorphize the same generic replay worker, so the + // classifier and the failed-store write run identically for an audit entry type. Driving the + // worker over a non-String event type exercises the audit monomorphization. + #[tokio::test(start_paused = true)] + async fn audit_event_type_runs_the_identical_classifier_and_failed_store_logic() { + let dir = temp_dir("audit-parity"); + let mut store = QueueStore::::new_with_compression(&dir, 64, ".test", false); + store.open().unwrap(); + seed_entry(&store, "audit-id"); + + let shared: Arc + Send + Sync> = Arc::new(ProgrammedTarget::new( + Some(TargetError::JetStreamPublish { + retryable: false, + detail: "max payload exceeded".to_string(), + }), + store.clone(), + )); + let (cancel_tx, cancel_rx) = mpsc::channel(1); + let hook = Arc::new(move |_event: ReplayEvent| { + Box::pin(async move {}) as std::pin::Pin + Send>> + }) as super::super::ReplayHook; + let semaphore = Some(Arc::new(Semaphore::new(1))); + let worker = stream_replay_worker( + &mut store, + shared, + cancel_rx, + hook, + semaphore, + Duration::from_millis(10), + Duration::from_millis(10), + ); + let driver = async move { + tokio::time::sleep(Duration::from_secs(5)).await; + let _ = cancel_tx.send(()).await; + }; + tokio::join!(worker, driver); + + assert_eq!(store.len(), 0, "the audit live entry is cleared after the move"); + assert_eq!(store.failed_len(), 1, "the audit path writes a failed entry through the shared worker"); + + let _ = store.delete(); + } + + // The replay admission permit is held across the send await, so under a single-permit + // semaphore only one worker enters its send at a time. Two workers share a single-permit + // semaphore and a closed gate that holds every send in flight. The first worker admits and + // enters its gated send. The second worker blocks on the permit and never enters its send, so + // at most one send is in flight. The permit is still released before the backoff sleep, so a + // send that fails does not hold the shared permit across the backoff. + #[tokio::test] + async fn admission_permit_is_held_across_the_send_bounding_concurrent_sends() { + // A shared gate held closed keeps every send blocked inside its await, and a shared + // in-flight counter observes how many sends are concurrently in their await. + let shared_gate = Arc::new(Semaphore::new(Semaphore::MAX_PERMITS)); + shared_gate.forget_permits(Semaphore::MAX_PERMITS); + let in_flight = Arc::new(AtomicUsize::new(0)); + let max_in_flight = Arc::new(AtomicUsize::new(0)); + let semaphore = Some(Arc::new(Semaphore::new(1))); + + let mut stores = Vec::new(); + let mut workers = Vec::new(); + let mut cancels = Vec::new(); + for index in 0..2 { + let dir = temp_dir(&format!("permit-{index}")); + let store = QueueStore::::new_with_compression(&dir, 64, ".test", false); + store.open().unwrap(); + seed_entry(&store, &format!("id-{index}")); + + let mut target = ProgrammedTarget::new( + Some(TargetError::JetStreamPublish { + retryable: true, + detail: "timed out".to_string(), + }), + store.clone(), + ); + target.send_gate = shared_gate.clone(); + target.in_flight = in_flight.clone(); + target.max_in_flight = max_in_flight.clone(); + let shared: Arc + Send + Sync> = Arc::new(target); + + let (cancel_tx, cancel_rx) = mpsc::channel(1); + let hook = Arc::new(move |_event: ReplayEvent| { + Box::pin(async move {}) as std::pin::Pin + Send>> + }) as super::super::ReplayHook; + let semaphore = semaphore.clone(); + let mut worker_store = store.clone(); + let worker = tokio::spawn(async move { + stream_replay_worker( + &mut worker_store, + shared, + cancel_rx, + hook, + semaphore, + Duration::from_millis(10), + Duration::from_millis(10), + ) + .await; + }); + stores.push(store); + workers.push(worker); + cancels.push(cancel_tx); + } + + // Wait until the first worker has admitted and entered its gated send. + for _ in 0..200 { + if in_flight.load(Ordering::SeqCst) >= 1 { + break; + } + tokio::time::sleep(Duration::from_millis(5)).await; + } + // Give the second worker ample time to try to admit. It must block on the held permit and + // never enter its send. + tokio::time::sleep(Duration::from_millis(100)).await; + + assert_eq!( + max_in_flight.load(Ordering::SeqCst), + 1, + "the single permit is held across the send, so only one send is in flight at a time" + ); + + // The observation is complete. Abort the workers rather than draining them, so the test + // does not wait out the production backoff after the gated sends resume. + drop(cancels); + for worker in workers { + worker.abort(); + let _ = worker.await; + } + for store in stores { + let _ = store.delete(); + } + } + + // With the flag off the NATS target never produces a JetStream publish error, so the worker + // never writes the failed store. A non-JetStream connectivity error retries and exhausts + // through the existing path without creating the failed directory. + #[tokio::test(start_paused = true)] + async fn flag_off_path_creates_no_failed_directory() { + let dir = temp_dir("flag-off"); + let mut store = QueueStore::::new_with_compression(&dir, 64, ".test", false); + store.open().unwrap(); + seed_entry(&store, ""); + + // A flag-off target surfaces connectivity errors, not JetStreamPublish, so the failed-store + // move is never triggered. + let target = Arc::new(ProgrammedTarget::new(Some(TargetError::NotConnected), store.clone())); + let tally = Arc::new(ReplayTally::default()); + run_worker(&mut store, target.clone(), tally.clone()).await; + + assert!( + tally.retry_exhausted.load(Ordering::SeqCst) >= 1, + "the connectivity error exhausts its retries" + ); + assert!(!dir.join("failed").exists(), "no failed directory is created on the flag-off path"); + assert_eq!(store.failed_len(), 0); + + let _ = store.delete(); + } + + // A connect-level failure on a JetStream target surfaces as NotConnected, mapped at the + // publish path. The worker retries it with backoff and leaves the entry live at exhaustion, + // so a broker outage writes no failed-store entry and the event delivers when the connection + // recovers, matching the Core path. + #[tokio::test(start_paused = true)] + async fn jetstream_connect_failure_retries_and_keeps_the_entry_live() { + let dir = temp_dir("connect-failure"); + let mut store = QueueStore::::new_with_compression(&dir, 64, ".test", false); + store.open().unwrap(); + seed_entry(&store, "minted-id"); + + let target = Arc::new(ProgrammedTarget::new(Some(TargetError::NotConnected), store.clone())); + let tally = Arc::new(ReplayTally::default()); + // The live entry is rescanned after exhaustion, so the run stops at the first + // exhaustion to observe one full retry cycle. + run_worker_until(&mut store, target.clone(), tally.clone(), |t| { + t.retry_exhausted.load(Ordering::SeqCst) >= 1 + }) + .await; + + assert!( + tally.retryable.load(Ordering::SeqCst) >= 1, + "the connect failure retries within the bound" + ); + assert_eq!(tally.retry_exhausted.load(Ordering::SeqCst), 1, "the exhaustion hook fires exactly once"); + assert_eq!(store.len(), 1, "the entry stays live through the outage"); + assert_eq!(store.failed_len(), 0, "an outage writes no failed entry"); + assert!(!dir.join("failed").exists(), "no failed-store write occurs for a connect failure"); + + let _ = store.delete(); + } + + /// Spawns the replay worker over a FailingFailedStore and returns the cancel channel and + /// join handle, so a test can advance virtual time, repair the failed store mid-run, and + /// observe the hook tally across the repair. + fn spawn_failing_store_worker( + store: &FailingFailedStore, + target: Arc, + tally: Arc, + ) -> (mpsc::Sender<()>, tokio::task::JoinHandle<()>) { + let shared: Arc + Send + Sync> = target; + let (cancel_tx, cancel_rx) = mpsc::channel(1); + let hook = Arc::new(move |event: ReplayEvent| { + let tally = tally.clone(); + Box::pin(async move { + match event { + ReplayEvent::Delivered { .. } => tally.delivered.fetch_add(1, Ordering::SeqCst), + ReplayEvent::RetryableError { .. } => tally.retryable.fetch_add(1, Ordering::SeqCst), + ReplayEvent::Dropped { .. } => tally.dropped.fetch_add(1, Ordering::SeqCst), + ReplayEvent::PermanentFailure { .. } => tally.permanent_failure.fetch_add(1, Ordering::SeqCst), + ReplayEvent::RetryExhausted { .. } => tally.retry_exhausted.fetch_add(1, Ordering::SeqCst), + ReplayEvent::UnreadableEntry { .. } => tally.unreadable.fetch_add(1, Ordering::SeqCst), + }; + }) as std::pin::Pin + Send>> + }) as super::super::ReplayHook; + let semaphore = Some(Arc::new(Semaphore::new(1))); + let mut worker_store = store.clone(); + let worker = tokio::spawn(async move { + stream_replay_worker( + &mut worker_store, + shared, + cancel_rx, + hook, + semaphore, + Duration::from_millis(10), + Duration::from_millis(10), + ) + .await; + }); + (cancel_tx, worker) + } + + // A terminal failure whose failed-store move fails must not count as a final failure. No + // PermanentFailure hook fires, no failed entry lands, and the live entry stays for the next + // scan. Once the failed store is repaired, the move completes and the hook fires exactly + // once, so one lost event counts once rather than zero or once per scan. + #[tokio::test(start_paused = true)] + async fn a_failed_terminal_move_emits_no_hook_and_heals_once_repaired() { + let dir = temp_dir("move-fail-terminal"); + let inner = QueueStore::::new_with_compression(&dir, 64, ".test", false); + inner.open().unwrap(); + seed_entry(&inner, "minted-id"); + + let fail_flag = Arc::new(std::sync::atomic::AtomicBool::new(true)); + let store = FailingFailedStore { + inner: inner.clone(), + fail_put_failed_raw: fail_flag.clone(), + fail_del: Arc::new(std::sync::atomic::AtomicBool::new(false)), + }; + // The target's terminal move writes through the rejecting failed handle, so the injected + // failure is what the move hits. + let target = Arc::new(ProgrammedTarget::new_with_failed_store( + Some(TargetError::JetStreamPublish { + retryable: false, + detail: "max payload exceeded".to_string(), + }), + inner.clone(), + Arc::new(store.clone()), + )); + let tally = Arc::new(ReplayTally::default()); + let (cancel_tx, worker) = spawn_failing_store_worker(&store, target, tally.clone()); + + // Many scans run while the failed store rejects the move. + tokio::time::sleep(Duration::from_secs(300)).await; + assert_eq!( + tally.permanent_failure.load(Ordering::SeqCst), + 0, + "no final-failure hook fires while the move keeps failing" + ); + assert_eq!(inner.len(), 1, "the live entry stays for the next scan"); + assert_eq!(inner.failed_len(), 0, "no failed entry landed"); + + // The operator repairs the failed store. The next scan completes the move. + fail_flag.store(false, Ordering::SeqCst); + tokio::time::sleep(Duration::from_secs(300)).await; + assert_eq!( + tally.permanent_failure.load(Ordering::SeqCst), + 1, + "the hook fires exactly once for the completed move" + ); + assert_eq!(inner.len(), 0, "the live entry is cleared by the completed move"); + assert_eq!(inner.failed_len(), 1, "the failed entry landed once"); + + let _ = cancel_tx.send(()).await; + let _ = worker.await; + let _ = inner.delete(); + } + + // A terminal move whose failed-record write succeeds but whose live delete fails is not + // handled: no final-failure hook fires and the live entry stays for the next scan, while the + // durable failed copy is already in place and stays single through the idempotent overwrite. + #[tokio::test(start_paused = true)] + async fn a_move_with_a_failing_delete_reports_unhandled_and_keeps_the_entry() { + let dir = temp_dir("move-del-fail"); + let inner = QueueStore::::new_with_compression(&dir, 64, ".test", false); + inner.open().unwrap(); + seed_entry(&inner, "minted-id"); + + // The worker's live store rejects deletes while the failed handle (the plain inner store) + // accepts the failed-record write, so only the delete half of the move fails. + let store = FailingFailedStore { + inner: inner.clone(), + fail_put_failed_raw: Arc::new(std::sync::atomic::AtomicBool::new(false)), + fail_del: Arc::new(std::sync::atomic::AtomicBool::new(true)), + }; + let target = Arc::new(ProgrammedTarget::new( + Some(TargetError::JetStreamPublish { + retryable: false, + detail: "max payload exceeded".to_string(), + }), + inner.clone(), + )); + let tally = Arc::new(ReplayTally::default()); + let (cancel_tx, worker) = spawn_failing_store_worker(&store, target, tally.clone()); + + // Many scans run while the live delete keeps failing. + tokio::time::sleep(Duration::from_secs(300)).await; + assert_eq!( + tally.permanent_failure.load(Ordering::SeqCst), + 0, + "no final-failure hook fires while the delete fails" + ); + assert_eq!(inner.len(), 1, "the live entry stays for the next scan"); + assert_eq!(inner.failed_len(), 1, "the durable failed copy landed exactly once"); + + let _ = cancel_tx.send(()).await; + let _ = worker.await; + let _ = inner.delete(); + } + + // The fifth failure is the last attempt, so no backoff follows it. The exhaustion is reported + // after the four inter-attempt backoffs, 2s * (2 + 4 + 8 + 16) = 60s plus sub-second jitter, + // well inside the 100s bound asserted here. A trailing backoff after the final failure would + // add 64s more and push the report past the bound, failing the assertion. + #[tokio::test(start_paused = true)] + async fn exhaustion_reports_without_a_trailing_backoff() { + let dir = temp_dir("no-trailing-backoff"); + let store = QueueStore::::new_with_compression(&dir, 64, ".test", false); + store.open().unwrap(); + seed_entry(&store, "minted-id"); + + // The delegating wrapper with both failure flags off behaves as the plain on-disk store. + let wrapper = FailingFailedStore { + inner: store.clone(), + fail_put_failed_raw: Arc::new(std::sync::atomic::AtomicBool::new(false)), + fail_del: Arc::new(std::sync::atomic::AtomicBool::new(false)), + }; + let target = Arc::new(ProgrammedTarget::new( + Some(TargetError::JetStreamPublish { + retryable: true, + detail: "timed out".to_string(), + }), + store.clone(), + )); + let tally = Arc::new(ReplayTally::default()); + + let start = tokio::time::Instant::now(); + let (cancel_tx, worker) = spawn_failing_store_worker(&wrapper, target, tally.clone()); + + // Poll the virtual clock until the first exhaustion is reported. + while tally.retry_exhausted.load(Ordering::SeqCst) == 0 { + assert!(start.elapsed() < Duration::from_secs(600), "the exhaustion never reported"); + tokio::time::sleep(Duration::from_millis(100)).await; + } + let elapsed = start.elapsed(); + assert!( + elapsed < Duration::from_secs(100), + "the exhaustion reports after the inter-attempt backoffs only, got {elapsed:?}" + ); + assert_eq!(store.len(), 1, "the entry stays on the live queue"); + assert_eq!(store.failed_len(), 0, "a retryable exhaustion writes no failed entry"); + + let _ = cancel_tx.send(()).await; + let _ = worker.await; + let _ = store.delete(); + } + + // A raw Network error models a non-JetStream core target, whose connect failure keeps the core + // handling: a permanent failure left in the live queue, never diverted into the JetStream + // retryable path or the failed store. + #[tokio::test(start_paused = true)] + async fn core_network_error_is_not_diverted_to_the_failed_store() { + let dir = temp_dir("core-network"); + let mut store = QueueStore::::new_with_compression(&dir, 64, ".test", false); + store.open().unwrap(); + seed_entry(&store, ""); + + let target = Arc::new(ProgrammedTarget::new( + Some(TargetError::Network("connection refused".to_string())), + store.clone(), + )); + let tally = Arc::new(ReplayTally::default()); + run_worker(&mut store, target.clone(), tally.clone()).await; + + assert!( + tally.permanent_failure.load(Ordering::SeqCst) >= 1, + "a core Network error is reported permanent, not silently dropped" + ); + assert_eq!(tally.retryable.load(Ordering::SeqCst), 0, "a core Network error is not retried"); + assert_eq!( + tally.retry_exhausted.load(Ordering::SeqCst), + 0, + "a core Network error does not reach exhaustion" + ); + assert_eq!(store.failed_len(), 0, "a core Network error writes no failed entry"); + assert_eq!(store.len(), 1, "a core Network error is left queued, its existing behaviour"); + assert!(!dir.join("failed").exists(), "no failed directory is created for a core Network error"); + + let _ = store.delete(); + } + + // The single failed-store invariant: only a non-retryable error ever produces a failed-store + // entry. One retryable entry driven past the budget stays on the live queue while one terminal + // entry moves, so after both are processed the failed store holds exactly the terminal entry. + #[tokio::test(start_paused = true)] + async fn only_a_non_retryable_error_produces_a_failed_store_entry() { + // A target that reads each entry and returns a terminal error for the entry whose dedup id + // marks it terminal, and a retryable error for every other entry. + #[derive(Clone)] + struct ClassifyingTarget { + id: TargetID, + store: QueueStore, + } + #[async_trait] + impl Target for ClassifyingTarget { + fn id(&self) -> TargetID { + self.id.clone() + } + async fn is_active(&self) -> Result { + Ok(true) + } + async fn save(&self, _event: Arc>) -> Result<(), TargetError> { + Ok(()) + } + async fn send_raw_from_store( + &self, + _key: Key, + _body: Vec, + _meta: QueuedPayloadMeta, + ) -> Result<(), TargetError> { + Ok(()) + } + async fn send_from_store(&self, key: Key) -> Result<(), TargetError> { + let raw = self + .store + .get_raw(&key) + .map_err(|err| TargetError::Unknown(err.to_string()))?; + let decoded = QueuedPayload::decode(&raw).map_err(|err| TargetError::Unknown(err.to_string()))?; + if decoded.meta.dedup_id == "terminal-id" { + Err(TargetError::JetStreamPublish { + retryable: false, + detail: "max payload exceeded".to_string(), + }) + } else { + Err(TargetError::JetStreamPublish { + retryable: true, + detail: "timed out".to_string(), + }) + } + } + async fn close(&self) -> Result<(), TargetError> { + Ok(()) + } + fn store(&self) -> Option<&(dyn Store + Send + Sync)> { + None + } + async fn handle_terminal_failure( + &self, + store: &(dyn Store + Send), + key: &Key, + error: &TargetError, + retry_count: u32, + ) -> bool { + move_terminal_entry_for_test(store, &self.store, key, error, retry_count) + } + fn clone_dyn(&self) -> Box + Send + Sync> { + Box::new(self.clone()) + } + fn is_enabled(&self) -> bool { + true + } + } + + let dir = temp_dir("terminals-only-invariant"); + let store = QueueStore::::new_with_compression(&dir, 64, ".test", false); + store.open().unwrap(); + seed_entry(&store, "retryable-id"); + seed_entry(&store, "terminal-id"); + + let shared: Arc + Send + Sync> = Arc::new(ClassifyingTarget { + id: TargetID::new("target-a".to_string(), "nats".to_string()), + store: store.clone(), + }); + let tally = Arc::new(ReplayTally::default()); + let (cancel_tx, cancel_rx) = mpsc::channel(1); + let hook = { + let tally = tally.clone(); + Arc::new(move |event: ReplayEvent| { + let tally = tally.clone(); + Box::pin(async move { + match event { + ReplayEvent::Delivered { .. } => tally.delivered.fetch_add(1, Ordering::SeqCst), + ReplayEvent::RetryableError { .. } => tally.retryable.fetch_add(1, Ordering::SeqCst), + ReplayEvent::Dropped { .. } => tally.dropped.fetch_add(1, Ordering::SeqCst), + ReplayEvent::PermanentFailure { .. } => tally.permanent_failure.fetch_add(1, Ordering::SeqCst), + ReplayEvent::RetryExhausted { .. } => tally.retry_exhausted.fetch_add(1, Ordering::SeqCst), + ReplayEvent::UnreadableEntry { .. } => tally.unreadable.fetch_add(1, Ordering::SeqCst), + }; + }) as std::pin::Pin + Send>> + }) as super::super::ReplayHook + }; + let semaphore = Some(Arc::new(Semaphore::new(1))); + let mut worker_store = store.clone(); + let worker = tokio::spawn(async move { + stream_replay_worker( + &mut worker_store, + shared, + cancel_rx, + hook, + semaphore, + Duration::from_millis(10), + Duration::from_millis(10), + ) + .await; + }); + + // Drive until the terminal entry has moved and the retryable entry has exhausted at least + // once. Both share one batch, so one pass processes each. + for _ in 0..600 { + if store.failed_len() >= 1 && tally.retry_exhausted.load(Ordering::SeqCst) >= 1 { + break; + } + tokio::time::sleep(Duration::from_secs(1)).await; + } + let _ = cancel_tx.send(()).await; + let _ = worker.await; + + assert_eq!(store.failed_len(), 1, "the failed store holds exactly one entry"); + assert_eq!(store.len(), 1, "the retryable entry stays on the live queue"); + + // The one failed entry is the terminal one. + let failed_dir = dir.join("failed"); + let failed_file = std::fs::read_dir(&failed_dir).unwrap().next().unwrap().unwrap().path(); + let decoded = QueuedPayload::decode(&std::fs::read(&failed_file).unwrap()).unwrap(); + let failure = decoded.meta.failure.unwrap(); + assert_eq!(failure.error_class, crate::target::FailedErrorClass::Terminal); + assert_eq!(failure.nats_msg_id, "terminal-id"); + + let _ = store.delete(); + } + } } diff --git a/crates/targets/src/store.rs b/crates/targets/src/store.rs index e1e42353c..e890e5517 100644 --- a/crates/targets/src/store.rs +++ b/crates/targets/src/store.rs @@ -24,10 +24,10 @@ use std::{ marker::PhantomData, path::{Path, PathBuf}, sync::{ - Arc, RwLock, + Arc, Mutex, RwLock, atomic::{AtomicU64, Ordering}, }, - time::{SystemTime, UNIX_EPOCH}, + time::{Duration, SystemTime, UNIX_EPOCH}, }; use tracing::{debug, warn}; use uuid::Uuid; @@ -61,6 +61,59 @@ fn is_queue_file_name(file_name: &str, file_ext: &str) -> bool { base.ends_with(file_ext) } +/// Separator between the batch item count and the entry name in an on-disk batch filename. It must +/// stay outside the Windows reserved filename set so File::create succeeds on NTFS, and outside the +/// UUID alphabet so the count splits unambiguously when the name is parsed back. +const BATCH_COUNT_SEPARATOR: char = '_'; + +/// Legacy separator for the batch item count. Entries written with it are read back for +/// compatibility, but it is reserved on NTFS so it is never written. +const LEGACY_BATCH_COUNT_SEPARATOR: char = ':'; + +/// Name of the child directory inside a target queue directory that holds events which failed +/// terminally. It is created lazily on the first failed write, so a target that never fails +/// terminally creates no such directory. +const FAILED_STORE_SUBDIR: &str = "failed"; + +/// Maximum number of entries retained in the failed store per target, bounded independently of the +/// live queue so an accumulation of terminal failures cannot starve live events. At the bound the +/// oldest failed entry is dropped to admit the newer one, so a newer failure is never lost in favour +/// of an older one. +const FAILED_STORE_MAX_ENTRIES: usize = 10_000; + +/// Maximum age of a failed-store entry before it is removed as expired, measured from the file write +/// time, which is the instant the entry entered the failed store. +const FAILED_STORE_TTL: Duration = Duration::from_secs(72 * 60 * 60); + +/// Writes payload to a temp file in the same directory, flushes the file to disk, then atomically +/// renames it onto final_path. +/// +/// rename within a single directory is atomic on the supported filesystems, so a crash leaves +/// either final_path absent, the prior final_path, or the complete payload, never a partial file. +/// On failure the temp file is removed so an interrupted write leaves nothing behind. +fn write_temp_then_rename(temp_path: &Path, final_path: &Path, payload: &[u8]) -> Result<(), StoreError> { + if let Err(err) = write_and_sync_temp(temp_path, payload) { + let _ = std::fs::remove_file(temp_path); + return Err(err); + } + + if let Err(err) = std::fs::rename(temp_path, final_path) { + let _ = std::fs::remove_file(temp_path); + return Err(StoreError::Io(err)); + } + + Ok(()) +} + +/// Writes payload to path and flushes the file to disk before returning, so the bytes are durable +/// before the caller renames the file into place. +fn write_and_sync_temp(path: &Path, payload: &[u8]) -> Result<(), StoreError> { + let mut file = std::fs::File::create(path).map_err(StoreError::Io)?; + file.write_all(payload).map_err(StoreError::Io)?; + file.sync_all().map_err(StoreError::Io)?; + Ok(()) +} + fn resolve_queue_store_compression_from_env_value(value: Option<&str>) -> bool { value .and_then(|value| value.parse::().ok().map(|state| state.is_enabled())) @@ -88,8 +141,13 @@ pub struct Key { impl Key { /// Converts the key to a string (filename) pub fn to_key_string(&self) -> String { + self.to_key_string_with(BATCH_COUNT_SEPARATOR) + } + + /// Builds the filename using the given batch count separator. + fn to_key_string_with(&self, separator: char) -> String { let name_part = if self.item_count > 1 { - format!("{}:{}", self.item_count, self.name) + format!("{}{separator}{}", self.item_count, self.name) } else { self.name.clone() }; @@ -134,12 +192,16 @@ pub fn parse_key(s: &str) -> Key { name = name[..name.len() - COMPRESS_EXT.len()].to_string(); } - // Number of batch items parsed - if let Some(colon_pos) = name.find(':') - && let Ok(count) = name[..colon_pos].parse::() + // Number of batch items parsed. The current separator and the legacy one are both accepted so an + // entry written before the switch to the Windows-safe separator still reads back. The count is + // stripped only when it exceeds one, mirroring the render side which prefixes the count only for a + // real batch, so a hostile 1_ or 0_ prefix on a single-item name stays part of the name. + if let Some(separator_pos) = name.find([BATCH_COUNT_SEPARATOR, LEGACY_BATCH_COUNT_SEPARATOR]) + && let Ok(count) = name[..separator_pos].parse::() + && count > 1 { item_count = count; - name = name[colon_pos + 1..].to_string(); + name = name[separator_pos + 1..].to_string(); } // Resolve extension @@ -241,6 +303,33 @@ where fn boxed_clone(&self) -> Box + Send + Sync>; } +/// A failed-events store nested inside the queue directory for terminal entries, kept separate from the generic queue interface +/// so only the paths that record terminal failures carry the surface. +/// +/// The failed store holds events that could not be handed off and is bounded independently of the +/// live queue. At the count bound the oldest failed entry is dropped before a new entry is written, so +/// a newer failure is never lost in favour of an older one. +pub trait FailedEventStore: Send + Sync { + /// Writes pre-encoded bytes to the failed-events store under the given entry name and returns the + /// written entry id. + /// + /// The write is atomic. The entry name derives from the live entry, so a repeated move of the same + /// entry replaces its earlier failed file instead of accumulating duplicates. + fn put_failed_raw(&self, entry_name: &str, data: &[u8]) -> Result; + + /// Removes failed-store entries older than the retention bound and returns how many were removed. + /// + /// Runs on the replay maintenance tick, not a separate timer. + fn prune_failed_store(&self) -> Result; + + /// Returns the number of entries currently in the failed store. + fn failed_len(&self) -> usize; + + /// Clones the capability into an owned boxed handle sharing the same backing state, so the + /// maintenance scan can run inside a blocking task without borrowing the target. + fn boxed_clone_failed(&self) -> Box; +} + /// A store that uses the filesystem to persist events in a queue pub struct QueueStore { entry_limit: u64, @@ -249,6 +338,15 @@ pub struct QueueStore { compress: bool, entries: Arc>>, // key -> modtime as unix nano pending_entries: Arc, + /// Cached count of complete failed-store entries, shared across clones. Initialized by one + /// directory read in open() and kept current by the failed-store write, the capacity trim, and + /// the expired-entry removal, so failed_len() is a plain atomic load rather than a directory scan. + failed_count: Arc, + /// Serializes the failed-store writers and the maintenance scan among themselves. put_failed_raw, + /// the capacity trim it runs, and the expiry-plus-reconcile scan take this guard, so the at-bound + /// check and the write stay atomic and the scan reconcile cannot overwrite a concurrent write's + /// count update. The live queue path never takes it. Shared across clones through the Arc. + failed_store_guard: Arc>, fs_guard: Arc>, _phantom: PhantomData, } @@ -262,6 +360,8 @@ impl Clone for QueueStore { compress: self.compress, entries: Arc::clone(&self.entries), pending_entries: Arc::clone(&self.pending_entries), + failed_count: Arc::clone(&self.failed_count), + failed_store_guard: Arc::clone(&self.failed_store_guard), fs_guard: Arc::clone(&self.fs_guard), _phantom: PhantomData, } @@ -296,14 +396,26 @@ impl QueueStore { compress, entries: Arc::new(RwLock::new(HashMap::with_capacity((entry_limit as usize).min(MAX_PREALLOC_CAPACITY)))), pending_entries: Arc::new(AtomicU64::new(0)), + failed_count: Arc::new(AtomicU64::new(0)), + failed_store_guard: Arc::new(Mutex::new(())), fs_guard: Arc::new(RwLock::new(())), _phantom: PhantomData, } } - /// Returns the full path for a key + /// Returns the full path for a key. A batch entry (item_count > 1) may exist on disk under the + /// legacy separator, so when the current-separator path is absent and the legacy-separator path + /// exists, the legacy path is returned. Reads and deletes then resolve a legacy file while fresh + /// writes keep the current separator. fn file_path(&self, key: &Key) -> PathBuf { - self.directory.join(key.to_key_string()) + let path = self.directory.join(key.to_key_string()); + if key.item_count > 1 && !path.exists() { + let legacy = self.directory.join(key.to_key_string_with(LEGACY_BATCH_COUNT_SEPARATOR)); + if legacy.exists() { + return legacy; + } + } + path } fn build_key(&self, item_count: usize) -> Key { @@ -427,10 +539,10 @@ impl QueueStore { /// lost and no ghost/truncated entry is ever indexed. fn write_file(&self, key: &Key, data: &[u8]) -> Result { let path = self.file_path(key); - // Create directory if it doesn't exist - if let Some(parent) = path.parent() { - std::fs::create_dir_all(parent).map_err(StoreError::Io)?; - } + let parent = path + .parent() + .ok_or_else(|| StoreError::Internal(format!("store entry path {} has no parent directory", path.display())))?; + std::fs::create_dir_all(parent).map_err(StoreError::Io)?; let payload: std::borrow::Cow<'_, [u8]> = if key.compress { let mut encoder = Encoder::new(); @@ -448,26 +560,9 @@ impl QueueStore { self.directory.join(file_name) }; - // Write + fsync the full payload into the temp file, then atomically - // rename it into place. Any error path removes the temp file so a failed - // write cannot leave residue that would otherwise be cleaned only on the - // next open(). - let write_result = (|| -> Result<(), StoreError> { - let mut file = File::create(&tmp_path).map_err(StoreError::Io)?; - file.write_all(&payload).map_err(StoreError::Io)?; - file.sync_all().map_err(StoreError::Io)?; - Ok(()) - })(); - - if let Err(err) = write_result { - let _ = std::fs::remove_file(&tmp_path); - return Err(err); - } - - if let Err(err) = std::fs::rename(&tmp_path, &path) { - let _ = std::fs::remove_file(&tmp_path); - return Err(StoreError::Io(err)); - } + // One implementation of the write-fsync-rename invariant, shared with the failed-store write. + // The temp file is removed on any error path so a failed write leaves no residue. + write_temp_then_rename(&tmp_path, &path, &payload)?; // Best-effort: persist the new directory entry created by rename. Self::fsync_dir(&self.directory); @@ -512,6 +607,126 @@ impl QueueStore { } Ok(()) } + + /// Path of the failed-events child directory inside the live queue directory. + fn failed_dir(&self) -> PathBuf { + self.directory.join(FAILED_STORE_SUBDIR) + } + + /// Reads the failed directory once and counts complete entries, skipping non-file entries and + /// residual temp files from an interrupted write. Off the hot path, so it seeds the cached count + /// at open and reconciles it on the maintenance scan rather than serving failed_len. + fn count_failed_entries_on_disk(&self) -> u64 { + let read_dir = match std::fs::read_dir(self.failed_dir()) { + Ok(read_dir) => read_dir, + Err(_) => return 0, + }; + + let mut count = 0u64; + for entry in read_dir.flatten() { + if entry.file_name().to_string_lossy().ends_with(TMP_SUFFIX) { + continue; + } + // file_type reads the directory entry kind without a full stat where the platform records it. + if matches!(entry.file_type(), Ok(file_type) if file_type.is_file()) { + count += 1; + } + } + count + } + + /// Lowers the cached failed count by one removed entry, clamped at zero so a decrement can never + /// wrap. The maintenance scan reconciles any residual drift each interval. + fn decrement_failed_count(&self) { + // The closure always returns Some, so the update never fails and the Result is discarded. + let _ = self + .failed_count + .fetch_update(Ordering::SeqCst, Ordering::SeqCst, |current| Some(current.saturating_sub(1))); + } + + /// Maps a per-entry stat outcome inside the ordered failed scan. A NotFound error means the file + /// was removed between the directory listing and the stat, by the capacity trim or the + /// expired-entry removal on another handle, so the scan skips that entry (None) instead of + /// failing as a whole. Any other error still fails the scan. + fn failed_scan_entry_metadata(outcome: std::io::Result) -> Result, StoreError> { + match outcome { + Ok(metadata) => Ok(Some(metadata)), + Err(err) if err.kind() == std::io::ErrorKind::NotFound => Ok(None), + Err(err) => Err(StoreError::Io(err)), + } + } + + /// Filenames of current failed-store entries paired with their write time, oldest first. + /// + /// The directory is read fresh each call rather than indexed in memory, because the failed store + /// is a low-traffic operator surface and a fresh scan keeps it free of the live-queue bookkeeping. + /// A missing directory yields an empty list, so a target that never failed reports nothing. An + /// entry removed concurrently mid-scan is skipped. + fn failed_entries_oldest_first(&self) -> Result, StoreError> { + let dir = self.failed_dir(); + let read_dir = match std::fs::read_dir(&dir) { + Ok(read_dir) => read_dir, + Err(err) if err.kind() == std::io::ErrorKind::NotFound => return Ok(Vec::new()), + Err(err) => return Err(StoreError::Io(err)), + }; + + let mut entries = Vec::new(); + for entry in read_dir { + let entry = entry.map_err(StoreError::Io)?; + let Some(metadata) = Self::failed_scan_entry_metadata(entry.metadata())? else { + continue; + }; + if !metadata.is_file() { + continue; + } + let name = entry.file_name().to_string_lossy().to_string(); + // A residual temp file from an interrupted failed-store write is not a complete entry. + if name.ends_with(TMP_SUFFIX) { + let _ = std::fs::remove_file(entry.path()); + continue; + } + let written_at = metadata.modified().unwrap_or(UNIX_EPOCH); + entries.push((entry.path(), written_at)); + } + + entries.sort_by_key(|(_, written_at)| *written_at); + Ok(entries) + } + + /// Drops the oldest failed entry to admit a new one when the count bound is reached. The warn log + /// naming the trimmed entry fires after its unlink succeeds, so the log names exactly the entries + /// the capacity trim removed. + fn evict_oldest_failed_if_full(&self, current: &[(PathBuf, SystemTime)]) -> Result<(), StoreError> { + if current.len() < FAILED_STORE_MAX_ENTRIES { + return Ok(()); + } + + let drop_count = current.len() - FAILED_STORE_MAX_ENTRIES + 1; + for (path, _) in current.iter().take(drop_count) { + let evicted_id = path + .file_name() + .map(|name| name.to_string_lossy().to_string()) + .unwrap_or_default(); + match std::fs::remove_file(path) { + Ok(()) => { + self.decrement_failed_count(); + warn!( + event = EVENT_TARGET_STORE_STATE, + component = LOG_COMPONENT_TARGETS, + subsystem = LOG_SUBSYSTEM_STORE, + action = "failed_store_evict", + evicted_entry = %evicted_id, + reason = "capacity", + "target store state" + ); + } + // Already removed by another handle, so nothing was trimmed here and no log fires. + Err(err) if err.kind() == std::io::ErrorKind::NotFound => {} + Err(err) => return Err(StoreError::Io(err)), + } + } + Ok(()) + } } impl Store for QueueStore @@ -551,7 +766,7 @@ where continue; } - // Ignore foreign files that do not match our queue file extension so + // Ignore foreign files that do not match the queue file extension so // externally-dropped files cannot pollute the queue. if !is_queue_file_name(&file_name, &self.file_ext) { continue; @@ -570,6 +785,10 @@ where entries_map.insert(file_name, unix_nano); } + // Seed the cached failed count from one read of the failed directory, under the write guard + // already held, so failed_len() serves an atomic load rather than a directory scan. + self.failed_count.store(self.count_failed_entries_on_disk(), Ordering::SeqCst); + debug!( event = EVENT_TARGET_STORE_STATE, component = LOG_COMPONENT_TARGETS, @@ -675,10 +894,10 @@ where return Err(StoreError::Deserialization(format!("Failed to deserialize item in batch: {e}"))); } None => { - // Reached end of stream before deserializing `item_count` items: the + // Reached end of stream before deserializing item_count items: the // batch file was truncated or corrupted. This MUST be surfaced as an - // error rather than silently returning the partial set. If we returned - // `Ok(partial)`, the caller would treat the batch as fully delivered, + // error rather than silently returning the partial set. Returning + // Ok(partial) would let the caller treat the batch as fully delivered, // delete the store entry, and permanently lose the missing events. warn!( event = EVENT_TARGET_STORE_STATE, @@ -740,7 +959,13 @@ where .write() .map_err(|_| StoreError::Internal("Failed to acquire write lock on entries".to_string()))?; - if entries.remove(&key.to_key_string()).is_none() { + // A batch entry indexed under the legacy separator is removed under both renderings so its slot is reclaimed. + let removed_current = entries.remove(&key.to_key_string()).is_some(); + let removed_legacy = key.item_count > 1 + && entries + .remove(&key.to_key_string_with(LEGACY_BATCH_COUNT_SEPARATOR)) + .is_some(); + if !removed_current && !removed_legacy { debug!( event = EVENT_TARGET_STORE_STATE, component = LOG_COMPONENT_TARGETS, @@ -774,6 +999,7 @@ where .map_err(|_| StoreError::Internal("Failed to acquire write lock on entries".to_string()))?; entries.clear(); self.pending_entries.store(0, Ordering::SeqCst); + self.failed_count.store(0, Ordering::SeqCst); match std::fs::remove_dir_all(&self.directory) { Ok(()) => Ok(()), @@ -840,6 +1066,142 @@ where } } +impl FailedEventStore for QueueStore +where + T: Serialize + DeserializeOwned + Clone + Send + Sync + 'static, +{ + fn put_failed_raw(&self, entry_name: &str, data: &[u8]) -> Result { + // The name becomes a filename inside the failed directory, so a separator or a dot-only name + // is rejected before it can address a path outside it. + if entry_name.is_empty() || entry_name.contains(['/', '\\']) || entry_name == "." || entry_name == ".." { + return Err(StoreError::Internal(format!("invalid failed-store entry name: {entry_name}"))); + } + + // The seam holds across the at-bound check, the capacity trim, and the count update, so a + // second writer or the scan cannot interleave. It nests outside fs_guard, the order the scan + // uses too, so the two never invert. + let _failed_guard = self + .failed_store_guard + .lock() + .map_err(|_| StoreError::Internal("Failed to acquire the failed-store guard".to_string()))?; + let _fs_guard = self + .fs_guard + .read() + .map_err(|_| StoreError::Internal("Failed to acquire read lock on store filesystem".to_string()))?; + + let failed_dir = self.failed_dir(); + if !failed_dir.exists() { + std::fs::create_dir_all(&failed_dir).map_err(StoreError::Io)?; + // Best-effort: persist the lazily created directory entry in the queue directory, the + // same durability discipline the live write path applies after its rename. + Self::fsync_dir(&self.directory); + } + + let entry_id = entry_name.to_string(); + let final_path = failed_dir.join(&entry_id); + let is_new_entry = !final_path.exists(); + + // A repeated move of the same live entry lands on the same filename and replaces the earlier + // file. The count bound is enforced only when the write creates a new entry, since an + // overwrite does not grow the count and a capacity trim for it would drop a genuine entry. The + // cached count gates the ordered capacity scan with a plain atomic load, so a write below the + // bound skips the stat-and-sort entirely. The ordered scan runs only at the bound, oldest + // first, so the new failure is recorded even at capacity. + if is_new_entry && self.failed_len() >= FAILED_STORE_MAX_ENTRIES { + let existing = self.failed_entries_oldest_first()?; + self.evict_oldest_failed_if_full(&existing)?; + } + + let temp_path = failed_dir.join(format!("{entry_id}.{}{}", Uuid::new_v4(), TMP_SUFFIX)); + // rename replaces an existing destination file on the supported platforms, which the + // idempotent overwrite relies on. + write_temp_then_rename(&temp_path, &final_path, data)?; + + // A new failed file grows the cached count. An overwrite of an existing failed file leaves it + // unchanged. + if is_new_entry { + self.failed_count.fetch_add(1, Ordering::SeqCst); + } + + // Best-effort: persist the directory entry created by the rename, matching the live write + // path. + Self::fsync_dir(&failed_dir); + + debug!( + event = EVENT_TARGET_STORE_STATE, + component = LOG_COMPONENT_TARGETS, + subsystem = LOG_SUBSYSTEM_STORE, + action = "failed_store_write", + failed_entry = %entry_id, + "target store state" + ); + Ok(entry_id) + } + + fn prune_failed_store(&self) -> Result { + // Same seam and nesting order as put_failed_raw, so the reconcile store below rests on a + // listing no concurrent writer can shift, and the count it writes cannot overwrite a write's + // increment. + let _failed_guard = self + .failed_store_guard + .lock() + .map_err(|_| StoreError::Internal("Failed to acquire the failed-store guard".to_string()))?; + let _fs_guard = self + .fs_guard + .read() + .map_err(|_| StoreError::Internal("Failed to acquire read lock on store filesystem".to_string()))?; + + let entries = self.failed_entries_oldest_first()?; + let materialized_len = entries.len(); + let now = SystemTime::now(); + let mut pruned = 0usize; + for (path, written_at) in entries { + let age = now.duration_since(written_at).unwrap_or_default(); + if age < FAILED_STORE_TTL { + // Entries are oldest first, so the first within the retention bound ends the scan. + break; + } + let pruned_id = path + .file_name() + .map(|name| name.to_string_lossy().to_string()) + .unwrap_or_default(); + warn!( + event = EVENT_TARGET_STORE_STATE, + component = LOG_COMPONENT_TARGETS, + subsystem = LOG_SUBSYSTEM_STORE, + action = "failed_store_prune", + pruned_entry = %pruned_id, + reason = "ttl", + "target store state" + ); + match std::fs::remove_file(&path) { + Ok(()) => { + pruned += 1; + self.decrement_failed_count(); + } + Err(err) if err.kind() == std::io::ErrorKind::NotFound => {} + Err(err) => return Err(StoreError::Io(err)), + } + } + // Reconcile the cached count to this scan's materialized listing under the seam, so a change + // made outside the store drifts the value by at most one maintenance interval and no concurrent + // write is overwritten. + self.failed_count.store((materialized_len - pruned) as u64, Ordering::SeqCst); + Ok(pruned) + } + + fn failed_len(&self) -> usize { + // A plain load of the cached count seeded at open and kept current by the write, the capacity + // trim, and the expired-entry removal. The maintenance scan reconciles it to the directory + // each interval, so external drift is corrected within one interval. + self.failed_count.load(Ordering::SeqCst) as usize + } + + fn boxed_clone_failed(&self) -> Box { + Box::new(self.clone()) + } +} + #[cfg(test)] mod tests { use super::*; @@ -894,6 +1256,54 @@ mod tests { assert_eq!(parsed.compress, key.compress); } + // A batch filename must stay outside the Windows reserved character set so File::create succeeds + // on NTFS. The name carries a UUID, matching what build_key produces for a real entry. + #[test] + fn batch_key_filename_has_no_windows_reserved_characters() { + let key = Key { + name: Uuid::new_v4().to_string(), + extension: ".json".to_string(), + item_count: 7, + compress: true, + }; + + let file_name = key.to_key_string(); + + // The characters NTFS rejects in a filename, plus the path separators. + const WINDOWS_RESERVED: &[char] = &['<', '>', ':', '"', '/', '\\', '|', '?', '*']; + for reserved in WINDOWS_RESERVED { + assert!( + !file_name.contains(*reserved), + "batch filename {file_name} must not contain the reserved character {reserved}" + ); + } + assert!(file_name.contains(BATCH_COUNT_SEPARATOR), "the batch count uses the safe separator"); + } + + // An entry written with the legacy separator still reads back, so a queue populated before the + // switch to the Windows-safe separator is not misparsed after an upgrade. + #[test] + fn parse_key_reads_legacy_batch_separator() { + let legacy = format!("5{LEGACY_BATCH_COUNT_SEPARATOR}{}.json", "event-id"); + + let parsed = parse_key(&legacy); + + assert_eq!(parsed.item_count, 5); + assert_eq!(parsed.name, "event-id"); + assert_eq!(parsed.extension, ".json"); + } + + // The render side prefixes the count only when it exceeds one, so the parse side strips a count + // prefix only for the same range. A hostile 1_ or 0_ prefix stays part of the name and round-trips. + #[test] + fn parse_key_treats_a_count_of_one_or_zero_as_a_plain_name() { + for hostile in ["1_name.event", "0_name.event"] { + let parsed = parse_key(hostile); + assert_eq!(parsed.item_count, 1, "{hostile} is a single-item entry"); + assert_eq!(parsed.to_key_string(), hostile, "{hostile} round-trips as a plain name"); + } + } + #[test] fn put_raw_and_get_raw_round_trip_bytes() { let dir = temp_store_dir("raw-roundtrip"); @@ -947,8 +1357,8 @@ mod tests { assert_eq!(key.item_count, 3); // Truncate the batch file at a clean boundary after the first two serialized - // items (`"aa""bb"`), so the read of the third item hits end-of-stream — the - // exact "partial read" condition that previously returned Ok silently. + // items, so the read of the third item hits end-of-stream. This is the + // partial-read condition the batch reader must surface as an error. let prefix_len = serde_json::to_vec(&"aa".to_string()).unwrap().len() + serde_json::to_vec(&"bb".to_string()).unwrap().len(); let path = store.file_path(&key); @@ -1048,7 +1458,7 @@ mod tests { let key = store.put(Arc::new("payload".to_string())).unwrap(); // A non-empty file with an unrelated extension must not be indexed and - // must be left untouched (we only clean up our own residue). + // must be left untouched. Only residue from this store's own writes is cleaned up. let foreign_path = dir.join("intruder.txt"); std::fs::write(&foreign_path, b"not ours").unwrap(); @@ -1089,8 +1499,8 @@ mod tests { #[test] fn new_with_huge_limit_does_not_panic() { - // Previously `HashMap::with_capacity(entry_limit as usize)` would attempt - // a giant allocation / capacity overflow for an absurd configured limit. + // An absurd configured entry_limit must not drive a giant up-front HashMap + // allocation or a capacity-overflow panic. let dir = temp_store_dir("huge-limit"); let store = QueueStore::::new_with_compression(&dir, u64::MAX, ".test", false); store.open().unwrap(); @@ -1128,4 +1538,544 @@ mod tests { let _ = store.delete(); } + + fn count_temp_files(dir: &Path) -> usize { + match std::fs::read_dir(dir) { + Ok(read_dir) => read_dir + .filter_map(|entry| entry.ok()) + .filter(|entry| entry.file_name().to_string_lossy().ends_with(TMP_SUFFIX)) + .count(), + Err(_) => 0, + } + } + + /// Directory truth for the failed store: the number of complete failed files on disk, skipping + /// residual temp files, so a test can assert the cached failed_len matches the filesystem. + fn count_failed_files_on_disk(dir: &Path) -> usize { + match std::fs::read_dir(dir.join(FAILED_STORE_SUBDIR)) { + Ok(read_dir) => read_dir + .filter_map(|entry| entry.ok()) + .filter(|entry| { + let name = entry.file_name().to_string_lossy().into_owned(); + !name.ends_with(TMP_SUFFIX) && entry.file_type().map(|file_type| file_type.is_file()).unwrap_or(false) + }) + .count(), + Err(_) => 0, + } + } + + // A completed write leaves the full payload at the final path, readable, and no temp file. The + // atomic path renames the temp into place rather than writing the final path directly. + #[test] + fn write_leaves_complete_file_and_no_temp_residue() { + let dir = temp_store_dir("atomic-complete"); + let store = QueueStore::::new_with_compression(&dir, 8, ".test", false); + store.open().unwrap(); + + let payload = br#"{"kind":"notify","bucket":"demo","key":"complete.txt"}"#; + let key = store.put_raw(payload).unwrap(); + + let final_path = store.file_path(&key); + assert!(final_path.exists(), "final entry must exist after a complete write"); + assert_eq!(std::fs::read(&final_path).unwrap(), payload); + assert_eq!(store.get_raw(&key).unwrap(), payload); + assert_eq!(count_temp_files(&dir), 0, "a complete write leaves no temp file"); + + let _ = store.delete(); + } + + // A temp file from an interrupted write (one that never reached its rename) is not a valid + // entry. The open-time rescan removes it and never indexes it, so a partial write is absent + // rather than read as a corrupt entry. + #[test] + fn open_discards_residual_temp_file() { + let dir = temp_store_dir("atomic-residue"); + let store = QueueStore::::new_with_compression(&dir, 8, ".test", false); + store.open().unwrap(); + + let key = store.put_raw(br#"{"complete":true}"#).unwrap(); + + // Simulate a write interrupted before its rename. A half-written temp file is left in the + // store directory under the same naming scheme write_file uses. + let residual_temp = dir.join(format!("{}.{}{}", "2_orphan.test", Uuid::new_v4(), TMP_SUFFIX)); + std::fs::write(&residual_temp, b"partial payload, never renamed").unwrap(); + assert!(residual_temp.exists()); + + // Reopen to drive the rescan. + store.open().unwrap(); + + assert!(!residual_temp.exists(), "open must remove a residual temp file"); + assert_eq!(count_temp_files(&dir), 0); + assert_eq!(store.len(), 1, "only the complete entry is indexed"); + assert_eq!(store.get_raw(&key).unwrap(), br#"{"complete":true}"#); + + let _ = store.delete(); + } + + // Flag-independent. The queue round-trip (write, read, delete) is unchanged by the atomic write, + // for both the uncompressed and the snappy-compressed path. The atomic write changes how bytes + // land, not what round-trips. + #[test] + fn queue_round_trip_unchanged_uncompressed_and_compressed() { + for compress in [false, true] { + let dir = temp_store_dir("round-trip"); + let store = QueueStore::::new_with_compression(&dir, 8, ".test", compress); + store.open().unwrap(); + + let payload = br#"{"kind":"notify","bucket":"demo","key":"round-trip.txt"}"#; + let key = store.put_raw(payload).unwrap(); + assert_eq!(key.compress, compress); + + assert_eq!(store.get_raw(&key).unwrap(), payload, "read returns the written bytes"); + assert_eq!(store.len(), 1); + + store.del(&key).unwrap(); + assert!(matches!(store.get_raw(&key), Err(StoreError::NotFound))); + assert_eq!(store.len(), 0); + + let _ = store.delete(); + } + } + + // The temp file lives in the same directory as the final path, so the rename stays within one + // filesystem and is atomic. A cross-directory temp would break that guarantee. + #[test] + fn temp_file_shares_directory_with_final_path() { + let dir = temp_store_dir("same-dir"); + let final_path = dir.join("entry.test"); + let temp_path = dir.join(format!("entry.test.{}{}", Uuid::new_v4(), TMP_SUFFIX)); + + assert_eq!(temp_path.parent(), final_path.parent()); + } + + // The failed-events store is a child directory inside the live queue directory and is created only on the + // first failed write, so a target that never fails terminally leaves no failed directory. + #[test] + fn failed_store_directory_is_lazy_and_sibling_of_the_queue() { + let dir = temp_store_dir("failed-lazy"); + let store = QueueStore::::new_with_compression(&dir, 8, ".test", false); + store.open().unwrap(); + + let failed_dir = dir.join("failed"); + assert!(!failed_dir.exists(), "no failed directory before any failed write"); + assert_eq!(store.failed_len(), 0); + + let first_id = store.put_failed_raw("failed-entry-1", b"failed-entry-1").unwrap(); + assert!(failed_dir.exists(), "the failed directory is created on the first failed write"); + assert_eq!(store.failed_len(), 1); + assert_eq!( + store.failed_len(), + count_failed_files_on_disk(&dir), + "the cached count matches the directory truth" + ); + assert_eq!( + std::fs::read(failed_dir.join(&first_id)).unwrap(), + b"failed-entry-1", + "the lazy-create write lands the full payload" + ); + + // A later write finds the directory already present and takes the non-create branch. + let second_id = store.put_failed_raw("failed-entry-2", b"failed-entry-2").unwrap(); + assert_eq!(store.failed_len(), 2); + assert_eq!( + store.failed_len(), + count_failed_files_on_disk(&dir), + "the cached count matches the directory truth" + ); + assert_eq!( + std::fs::read(failed_dir.join(&second_id)).unwrap(), + b"failed-entry-2", + "a write into the existing directory lands the full payload" + ); + + let _ = store.delete(); + } + + // A repeated move of the same live entry lands on the same failed filename, so a persistent + // delete failure, or a crash that left the entry in both stores followed by another terminal + // failure, replaces the earlier failed file instead of accumulating duplicates. + #[test] + fn failed_store_write_is_idempotent_per_entry() { + let dir = temp_store_dir("failed-idempotent"); + let store = QueueStore::::new_with_compression(&dir, 8, ".test", false); + store.open().unwrap(); + + let first = store.put_failed_raw("entry-a", b"first-write").unwrap(); + let second = store.put_failed_raw("entry-a", b"second-write").unwrap(); + assert_eq!(first, second, "the same entry keeps the same failed filename"); + assert_eq!(store.failed_len(), 1, "a repeated move yields exactly one failed file"); + assert_eq!( + store.failed_len(), + count_failed_files_on_disk(&dir), + "an overwrite leaves the cached count matching the directory truth" + ); + assert_eq!( + std::fs::read(dir.join("failed").join(&second)).unwrap(), + b"second-write", + "the re-move replaces the earlier file" + ); + + let _ = store.delete(); + } + + // The failed filename is used inside the failed directory, so a name that could address a path + // outside it is rejected before any write. + #[test] + fn failed_store_write_rejects_a_path_escaping_entry_name() { + let dir = temp_store_dir("failed-name-guard"); + let store = QueueStore::::new_with_compression(&dir, 8, ".test", false); + store.open().unwrap(); + + for name in ["", ".", "..", "a/b", "a\\b"] { + assert!(store.put_failed_raw(name, b"payload").is_err(), "name {name:?} is rejected"); + } + assert_eq!(store.failed_len(), 0, "no rejected name produced a file"); + + let _ = store.delete(); + } + + // A per-entry NotFound stat inside the ordered failed scan means the file was removed between + // the directory listing and the stat, so that entry is skipped and the scan completes. Any other + // stat error still fails the scan. + #[test] + fn failed_scan_skips_an_entry_removed_mid_scan() { + let dir = temp_store_dir("failed-scan-skip"); + std::fs::create_dir_all(&dir).unwrap(); + let probe = dir.join("probe"); + std::fs::write(&probe, b"probe").unwrap(); + let metadata = std::fs::metadata(&probe).unwrap(); + + let present = QueueStore::::failed_scan_entry_metadata(Ok(metadata)).unwrap(); + assert!(present.is_some(), "a present entry passes its metadata through"); + + let removed = + QueueStore::::failed_scan_entry_metadata(Err(std::io::Error::from(std::io::ErrorKind::NotFound))).unwrap(); + assert!(removed.is_none(), "a concurrently removed entry is skipped, not an error"); + + let denied = + QueueStore::::failed_scan_entry_metadata(Err(std::io::Error::from(std::io::ErrorKind::PermissionDenied))); + assert!(denied.is_err(), "a non-NotFound stat error still fails the scan"); + + let _ = std::fs::remove_dir_all(&dir); + } + + // The failed store is bounded independently of the live queue. A live queue at its limit does not + // consume failed-store capacity and the reverse holds. + #[test] + fn failed_store_is_separate_from_the_live_queue_limit() { + let dir = temp_store_dir("failed-separate"); + let store = QueueStore::::new_with_compression(&dir, 2, ".test", false); + store.open().unwrap(); + + store.put_raw(b"live-1").unwrap(); + store.put_raw(b"live-2").unwrap(); + assert!(matches!(store.put_raw(b"live-3"), Err(StoreError::LimitExceeded))); + + // The live queue is full, yet failed writes still succeed and grow the separate failed store. + store.put_failed_raw("failed-1", b"failed-1").unwrap(); + store.put_failed_raw("failed-2", b"failed-2").unwrap(); + store.put_failed_raw("failed-3", b"failed-3").unwrap(); + assert_eq!(store.len(), 2, "the live queue stays at its own limit"); + assert_eq!(store.failed_len(), 3, "the failed store grows past the live limit"); + + let _ = store.delete(); + } + + // At the count bound the oldest failed entry is dropped to admit the newer one, so a newer + // terminal failure is never lost in preference to an older one. The directory is filled to the + // bound with plain files at distinct write times, then one real failed write triggers the + // capacity trim of the unambiguous oldest. + #[test] + fn failed_store_drops_oldest_at_the_count_bound() { + let dir = temp_store_dir("failed-evict"); + let store = QueueStore::::new_with_compression(&dir, 8, ".test", false); + store.open().unwrap(); + + let failed_dir = dir.join("failed"); + std::fs::create_dir_all(&failed_dir).unwrap(); + + let oldest_name = "0000-oldest"; + for index in 0..FAILED_STORE_MAX_ENTRIES { + let name = if index == 0 { + oldest_name.to_string() + } else { + format!("{index:06}-{}", Uuid::new_v4()) + }; + let path = failed_dir.join(&name); + std::fs::write(&path, b"prefilled").unwrap(); + let written_at = UNIX_EPOCH + Duration::from_secs(index as u64); + std::fs::OpenOptions::new() + .write(true) + .open(&path) + .unwrap() + .set_modified(written_at) + .unwrap(); + } + // The prefill lands entries directly on disk, so re-open to seed the cached count from the + // failed directory, the same path a restart with existing failed entries takes. + store.open().unwrap(); + assert_eq!(store.failed_len(), FAILED_STORE_MAX_ENTRIES); + assert_eq!( + store.failed_len(), + count_failed_files_on_disk(&dir), + "the seeded count matches the directory truth" + ); + + store.put_failed_raw("failed-newest", b"failed-newest").unwrap(); + assert_eq!(store.failed_len(), FAILED_STORE_MAX_ENTRIES, "the bound holds after the capacity trim"); + assert_eq!( + store.failed_len(), + count_failed_files_on_disk(&dir), + "the cached count matches the directory truth after the trim" + ); + assert!( + !failed_dir.join(oldest_name).exists(), + "the oldest failed entry is trimmed to admit the newer one" + ); + + let _ = store.delete(); + } + + // Entries older than the retention bound are removed as expired on the maintenance tick, while + // younger entries survive. The age is measured from the file write time. + #[test] + fn failed_store_prunes_entries_past_the_ttl() { + let dir = temp_store_dir("failed-ttl"); + let store = QueueStore::::new_with_compression(&dir, 8, ".test", false); + store.open().unwrap(); + + let failed_dir = dir.join("failed"); + + let stale_id = store.put_failed_raw("stale", b"stale").unwrap(); + let fresh_id = store.put_failed_raw("fresh", b"fresh").unwrap(); + + // Age the stale entry past the bound, keep the fresh entry recent. + let stale_time = SystemTime::now() - (FAILED_STORE_TTL + Duration::from_secs(60)); + std::fs::OpenOptions::new() + .write(true) + .open(failed_dir.join(&stale_id)) + .unwrap() + .set_modified(stale_time) + .unwrap(); + + let pruned = store.prune_failed_store().unwrap(); + assert_eq!(pruned, 1, "one entry is past the retention bound"); + assert!(!failed_dir.join(&stale_id).exists(), "the stale entry is removed as expired"); + assert!(failed_dir.join(&fresh_id).exists(), "the fresh entry survives"); + assert_eq!(store.failed_len(), 1, "the cached count drops with the expired removal"); + assert_eq!( + store.failed_len(), + count_failed_files_on_disk(&dir), + "the cached count matches the directory truth after expiry" + ); + + let _ = store.delete(); + } + + // Expired-entry removal on a store with no failed directory is a no-op, so a target that never + // failed terminally is not charged a maintenance error on the tick. + #[test] + fn failed_store_prune_is_a_noop_without_a_failed_directory() { + let dir = temp_store_dir("failed-prune-noop"); + let store = QueueStore::::new_with_compression(&dir, 8, ".test", false); + store.open().unwrap(); + + assert_eq!(store.prune_failed_store().unwrap(), 0); + assert!(!dir.join("failed").exists()); + + let _ = store.delete(); + } + + // failed_len counts complete failed entries and skips a residual temp file from an interrupted + // write. The count does not remove the temp file. Removal stays on the ordered scan run by the + // expired-entry removal. + #[test] + fn failed_len_counts_real_entries_and_ignores_temp_files() { + let dir = temp_store_dir("failed-len"); + let store = QueueStore::::new_with_compression(&dir, 8, ".test", false); + store.open().unwrap(); + + store.put_failed_raw("complete-one", b"complete-one").unwrap(); + store.put_failed_raw("complete-two", b"complete-two").unwrap(); + let failed_dir = dir.join("failed"); + let residual_temp = failed_dir.join(format!("orphan.{}{}", Uuid::new_v4(), TMP_SUFFIX)); + std::fs::write(&residual_temp, b"partial, never renamed").unwrap(); + + assert_eq!(store.failed_len(), 2, "only complete entries are counted"); + assert!(residual_temp.exists(), "the count does not remove the residual temp file"); + + // The ordered scan run by the expired-entry removal drops the residual temp file. + store.prune_failed_store().unwrap(); + assert!(!residual_temp.exists(), "the ordered scan removes the residual temp file"); + + let _ = store.delete(); + } + + // The maintenance scan reconciles the cached count to the directory truth, so a hand-planted + // drift is corrected on the next expired-entry removal, bounding external-mutation drift to one + // interval. + #[test] + fn maintenance_reconciles_a_hand_planted_count_drift() { + let dir = temp_store_dir("failed-reconcile"); + let store = QueueStore::::new_with_compression(&dir, 8, ".test", false); + store.open().unwrap(); + + store.put_failed_raw("entry-a", b"entry-a").unwrap(); + store.put_failed_raw("entry-b", b"entry-b").unwrap(); + assert_eq!(store.failed_len(), 2); + + // Plant a drift so the cached count disagrees with the directory truth. + store.failed_count.store(99, Ordering::SeqCst); + assert_eq!(store.failed_len(), 99, "the planted drift is observed before the reconcile"); + + // Both entries are within the retention bound, so the scan removes nothing and reconciles the + // count to the materialized listing. + let pruned = store.prune_failed_store().unwrap(); + assert_eq!(pruned, 0, "no entry is past the retention bound"); + assert_eq!(store.failed_len(), 2, "the reconcile restores the directory truth"); + assert_eq!(store.failed_len(), count_failed_files_on_disk(&dir)); + + let _ = store.delete(); + } + + // The maintenance scan holds the seam across its listing and reconcile, so a racing writer waits + // and its increment lands after the reconcile rather than being overwritten by it. The scan + // thread is parked inside the seam by the filesystem write guard, the writer is released against + // the held seam, and the final count must include the write. A scan that does not take the seam + // fails the held-seam wait below. + #[test] + fn maintenance_reconcile_does_not_overwrite_a_concurrent_failed_write() { + let dir = temp_store_dir("failed-reconcile-race"); + let store = Arc::new(QueueStore::::new_with_compression(&dir, 8, ".test", false)); + store.open().unwrap(); + + store.put_failed_raw("existing", b"existing").unwrap(); + // Plant a drift so the final value proves the reconcile ran and the write survived it. A + // reconcile that overwrites the write ends at 1, a write with no reconcile ends at 100, the + // serialized pair ends at 2. + store.failed_count.store(99, Ordering::SeqCst); + + // Hold the filesystem guard so the scan thread stops inside the seam, before its listing. + let fs_block = store.fs_guard.write().unwrap(); + + let scan_store = Arc::clone(&store); + let scan = thread::spawn(move || scan_store.prune_failed_store()); + + // Wait for the scan to take the seam. The deadline turns a scan that never takes the seam + // into a failure rather than a hang. + let deadline = std::time::Instant::now() + Duration::from_secs(5); + while store.failed_store_guard.try_lock().is_ok() { + assert!( + std::time::Instant::now() < deadline, + "the maintenance scan never took the failed-store seam" + ); + thread::yield_now(); + } + + // The writer is released while the scan holds the seam, so its write and increment wait for + // the reconcile. + let release = Arc::new(Barrier::new(2)); + let writer_store = Arc::clone(&store); + let writer_release = Arc::clone(&release); + let writer = thread::spawn(move || { + writer_release.wait(); + writer_store.put_failed_raw("racing", b"racing").unwrap(); + }); + release.wait(); + + drop(fs_block); + assert_eq!(scan.join().unwrap().unwrap(), 0, "both entries are within the retention bound"); + writer.join().unwrap(); + + assert_eq!(store.failed_len(), 2, "the reconcile does not overwrite the racing write"); + assert_eq!( + store.failed_len(), + count_failed_files_on_disk(&dir), + "the cached count matches the directory truth after the race" + ); + + let _ = store.delete(); + } + + // Two writers racing new failed entries at the count bound stay atomic through the seam, so the + // bound holds and the cached count matches the directory truth. Without the seam the at-bound + // check and the write interleave and the count overshoots the bound. + #[test] + fn concurrent_failed_writes_at_the_bound_hold_the_count() { + let dir = temp_store_dir("failed-race-bound"); + let store = Arc::new(QueueStore::::new_with_compression(&dir, 8, ".test", false)); + store.open().unwrap(); + + let failed_dir = dir.join("failed"); + std::fs::create_dir_all(&failed_dir).unwrap(); + for index in 0..FAILED_STORE_MAX_ENTRIES { + let path = failed_dir.join(format!("{index:06}-{}", Uuid::new_v4())); + std::fs::write(&path, b"prefilled").unwrap(); + let written_at = UNIX_EPOCH + Duration::from_secs(index as u64); + std::fs::OpenOptions::new() + .write(true) + .open(&path) + .unwrap() + .set_modified(written_at) + .unwrap(); + } + // Seed the cached count from the prefilled directory, the path a restart with existing entries takes. + store.open().unwrap(); + assert_eq!(store.failed_len(), FAILED_STORE_MAX_ENTRIES); + + let start = Arc::new(Barrier::new(2)); + let mut handles = Vec::new(); + for writer in 0..2 { + let store = Arc::clone(&store); + let start = Arc::clone(&start); + handles.push(thread::spawn(move || { + start.wait(); + store.put_failed_raw(&format!("racing-newest-{writer}"), b"newest").unwrap(); + })); + } + for handle in handles { + handle.join().unwrap(); + } + + assert_eq!(store.failed_len(), FAILED_STORE_MAX_ENTRIES, "the bound holds after two racing writes"); + assert_eq!( + store.failed_len(), + count_failed_files_on_disk(&dir), + "the cached count matches the directory truth after the race" + ); + + let _ = store.delete(); + } + + // Two writers racing the same failed entry name produce one failed file and one count increment, + // so a concurrent same-key move cannot double-count. + #[test] + fn concurrent_same_key_failed_writes_count_once() { + let dir = temp_store_dir("failed-race-same-key"); + let store = Arc::new(QueueStore::::new_with_compression(&dir, 8, ".test", false)); + store.open().unwrap(); + + let start = Arc::new(Barrier::new(2)); + let mut handles = Vec::new(); + for _ in 0..2 { + let store = Arc::clone(&store); + let start = Arc::clone(&start); + handles.push(thread::spawn(move || { + start.wait(); + store.put_failed_raw("same-entry", b"payload").unwrap(); + })); + } + for handle in handles { + handle.join().unwrap(); + } + + assert_eq!(store.failed_len(), 1, "two racing writes of one name count once"); + assert_eq!( + store.failed_len(), + count_failed_files_on_disk(&dir), + "the cached count matches the directory truth" + ); + + let _ = store.delete(); + } } diff --git a/crates/targets/src/target/amqp.rs b/crates/targets/src/target/amqp.rs index 2a3d096b2..1af39301a 100644 --- a/crates/targets/src/target/amqp.rs +++ b/crates/targets/src/target/amqp.rs @@ -657,8 +657,11 @@ where } fn delivery_snapshot(&self) -> TargetDeliverySnapshot { - self.delivery_counters - .snapshot(self.store.as_deref().map_or(0, |store| store.len() as u64)) + self.delivery_counters.snapshot( + self.store.as_deref().map_or(0, |store| store.len() as u64), + // AMQP targets record no terminal failures and keep no failed store. + 0, + ) } fn record_final_failure(&self) { diff --git a/crates/targets/src/target/kafka.rs b/crates/targets/src/target/kafka.rs index 928bce7cd..9b087f349 100644 --- a/crates/targets/src/target/kafka.rs +++ b/crates/targets/src/target/kafka.rs @@ -503,8 +503,11 @@ where } fn delivery_snapshot(&self) -> TargetDeliverySnapshot { - self.delivery_counters - .snapshot(self.store.as_deref().map_or(0, |store| store.len() as u64)) + self.delivery_counters.snapshot( + self.store.as_deref().map_or(0, |store| store.len() as u64), + // Kafka targets record no terminal failures and keep no failed store. + 0, + ) } fn record_final_failure(&self) { diff --git a/crates/targets/src/target/mod.rs b/crates/targets/src/target/mod.rs index e34bd7125..d3635b666 100644 --- a/crates/targets/src/target/mod.rs +++ b/crates/targets/src/target/mod.rs @@ -14,7 +14,7 @@ use crate::arn::TargetID; use crate::plugin::PluginEvent; -use crate::store::{Key, QueueStore, Store}; +use crate::store::{FailedEventStore, Key, QueueStore, Store}; use crate::{StoreError, TargetError, TargetLog}; use async_trait::async_trait; use rustfs_s3_types::EventName; @@ -58,6 +58,7 @@ pub(crate) fn redacted_optional_secret(value: Option<&str>) -> &'static str { #[derive(Debug, Clone, Default, PartialEq, Eq)] pub struct TargetDeliverySnapshot { pub failed_messages: u64, + pub failed_store_length: u64, pub queue_length: u64, pub total_messages: u64, } @@ -83,9 +84,10 @@ impl TargetDeliveryCounters { } #[inline] - pub fn snapshot(&self, queue_length: u64) -> TargetDeliverySnapshot { + pub fn snapshot(&self, queue_length: u64, failed_store_length: u64) -> TargetDeliverySnapshot { TargetDeliverySnapshot { failed_messages: self.failed_messages.load(Ordering::Relaxed), + failed_store_length, queue_length, total_messages: self.total_messages.load(Ordering::Relaxed), } @@ -159,6 +161,25 @@ where /// Returns the store associated with the target (if any) fn store(&self) -> Option<&(dyn Store + Send + Sync)>; + /// Returns the failed-events store capability when the target records terminal failures. + /// + /// The default is no capability, so a target that never parks a terminal entry runs no failed-store + /// maintenance and reports zero failed-store depth. + fn failed_store(&self) -> Option<&dyn FailedEventStore> { + None + } + + /// Moves a terminally failed entry to the target's failed-events store, returning true when the entry was handled. A target without a terminal-failure store declines the move and the entry stays live. + async fn handle_terminal_failure( + &self, + _store: &(dyn Store + Send), + _key: &Key, + _error: &TargetError, + _retry_count: u32, + ) -> bool { + false + } + /// Returns the type of the target fn clone_dyn(&self) -> Box + Send + Sync>; @@ -174,6 +195,7 @@ where /// Returns a read-only delivery snapshot for metrics collection. fn delivery_snapshot(&self) -> TargetDeliverySnapshot { TargetDeliverySnapshot { + failed_store_length: self.failed_store().map_or(0, |failed_store| failed_store.failed_len() as u64), queue_length: self.store().map_or(0, |store| store.len() as u64), ..TargetDeliverySnapshot::default() } @@ -202,6 +224,55 @@ pub struct QueuedPayloadMeta { pub content_type: String, pub queued_at_unix_ms: u64, pub payload_len: usize, + /// Stable per-entry deduplication identifier sent as the NATS JetStream Nats-Msg-Id header. + /// Empty for every non-JetStream target and for entries queued before JetStream was enabled, so + /// it is skipped on serialization and absent stored entries decode to an empty value. This keeps + /// the stored bytes identical to entries written without the field. + #[serde(default, skip_serializing_if = "String::is_empty")] + pub dedup_id: String, + + /// Set only on an entry written to the failed-events store. None on every live-queue entry, so it + /// is skipped on serialization and a live entry decodes with no failure metadata, keeping live + /// stored bytes identical to entries written without the field. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub failure: Option, +} + +/// The error class recorded on a failed-store entry. Only a non-retryable publish error reaches the +/// failed store, so the class confirms a terminal cause for an operator. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +pub enum FailedErrorClass { + /// A non-retryable publish error. Replaying it without fixing the underlying configuration repeats + /// the failure. + Terminal, +} + +impl FailedErrorClass { + /// Stable lowercase tag used in structured logs and operator tooling. + pub fn as_str(&self) -> &'static str { + match self { + FailedErrorClass::Terminal => "terminal", + } + } +} + +/// Failure metadata added to a queued payload when it is moved to the failed-events store, carried +/// inside the existing QueuedPayload meta so the failed entry decodes through the same reader. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct FailedEntryMeta { + /// The failure class recorded for operator triage. A failed-store entry is always terminal. + pub error_class: FailedErrorClass, + /// An allowlisted, credential-free summary of the failure for operator diagnosis. Never the raw + /// error rendering. + pub error_detail: String, + /// The stored dedup identifier, recording the id the publish attempted. A failed record is never + /// republished. + pub nats_msg_id: String, + /// The instant the entry entered the failed store, a diagnostic field for operator triage. Expiry + /// uses the filesystem modification time, not this value. + pub failed_at_unix_ms: u64, + /// The replay attempt count recorded at the terminal failure. + pub retry_count: u32, } impl QueuedPayloadMeta { @@ -219,6 +290,8 @@ impl QueuedPayloadMeta { content_type: content_type.into(), queued_at_unix_ms: SystemTime::now().duration_since(UNIX_EPOCH).unwrap_or_default().as_millis() as u64, payload_len, + dedup_id: String::new(), + failure: None, } } @@ -420,14 +493,13 @@ fn fnv1a_hash(bytes: &[u8]) -> u64 { /// Maps a target-id component to a filesystem-safe queue directory name. /// -/// Path-unsafe characters are replaced with `_`. Because that replacement is -/// lossy, two distinct ids (e.g. `a/b` and `a_b`) could otherwise collapse to -/// the same directory and interleave their persisted events. To prevent that, -/// whenever any character had to be replaced we append a short hash of the -/// original component, guaranteeing distinct ids map to distinct directories. +/// Path-unsafe characters are replaced with an underscore. That replacement is lossy, so two +/// distinct ids (for example a/b and a_b) could otherwise collapse to the same directory and +/// interleave their persisted events. A short hash of the original component is appended whenever any +/// character was replaced, so distinct ids map to distinct directories. /// -/// Ids that are already path-safe are returned unchanged, preserving the -/// on-disk directory layout for existing deployments (no queue migration). +/// Ids that are already path-safe are returned unchanged, preserving the on-disk directory layout for +/// existing deployments. pub(crate) fn sanitize_queue_dir_component(component: &str) -> String { let mut sanitized = String::with_capacity(component.len()); let mut lossy = false; @@ -441,7 +513,7 @@ pub(crate) fn sanitize_queue_dir_component(component: &str) -> String { } if sanitized.is_empty() { - // An entirely non-safe id would otherwise all collapse to "_"; key it by + // An entirely non-safe id would otherwise collapse to a single underscore. Key it by // the original bytes so distinct ids stay distinct. return format!("_{:016x}", fnv1a_hash(component.as_bytes())); } @@ -529,10 +601,20 @@ pub(crate) fn open_target_queue_store( target_id: &TargetID, open_context: &str, ) -> Result, TargetError> { - fn boxed_queue_store(store: QueueStore) -> BoxedQueuedStore { - Box::new(store) - } + let store = open_target_queue_store_typed(queue_dir, queue_limit, target_type, target_type_label, target_id, open_context)?; + Ok(store.map(|store| Box::new(store) as BoxedQueuedStore)) +} +/// Opens the queue store and returns the concrete QueueStore, so a target that needs its typed +/// failed-store capability holds it directly rather than through the type-erased Store handle. +pub(crate) fn open_target_queue_store_typed( + queue_dir: &str, + queue_limit: u64, + target_type: TargetType, + target_type_label: &str, + target_id: &TargetID, + open_context: &str, +) -> Result>, TargetError> { if queue_dir.is_empty() { return Ok(None); } @@ -547,7 +629,7 @@ pub(crate) fn open_target_queue_store( .open() .map_err(|err| TargetError::Storage(format!("{open_context}: {err}")))?; - Ok(Some(boxed_queue_store(store))) + Ok(Some(store)) } pub(crate) fn persist_queued_payload_to_store( @@ -584,7 +666,7 @@ pub(crate) fn mark_target_disconnected_on_connectivity_error(connected: &AtomicB } pub(crate) fn delete_stored_payload( - store: &(dyn Store + Send + Sync), + store: &(dyn Store + Send), key: &Key, ) -> Result<(), TargetError> { match store.del(key) { @@ -593,6 +675,96 @@ pub(crate) fn delete_stored_payload( } } +/// Upper bound on the characters retained from a classified error so a failed-store entry and its +/// alarm carry a diagnosable summary without an unbounded message. +const FAILED_ERROR_DETAIL_MAX_LEN: usize = 256; + +/// Fallback label substituted for a JetStreamPublish detail that falls outside the fixed vocabulary. +const UNRECOGNIZED_DETAIL_LABEL: &str = "unrecognized detail"; + +/// Restricts a JetStreamPublish detail to the fixed vocabulary at the persistence boundary. A detail +/// of lowercase letters, digits, space, underscore, and colon passes verbatim, any other content is +/// replaced with a fixed fallback label. +fn sanitize_failed_detail(detail: &str) -> &str { + let in_vocabulary = !detail.is_empty() + && detail.chars().all(|character| { + character.is_ascii_lowercase() || character.is_ascii_digit() || matches!(character, ' ' | '_' | ':') + }); + if in_vocabulary { detail } else { UNRECOGNIZED_DETAIL_LABEL } +} + +/// Builds a credential-free diagnostic string for a failed entry from a classified error. The +/// publish-error detail passes through the persistence-boundary sanitizer, a non-publish error +/// contributes only its variant category, and any embedded value is redacted. The result is +/// length-bounded at a character boundary. +pub(crate) fn build_failed_error_detail(error: &TargetError) -> String { + let summary = match error { + // The publish detail is sanitized to the fixed vocabulary at the persistence boundary. + TargetError::JetStreamPublish { detail, .. } => format!("jetstream_publish: {}", sanitize_failed_detail(detail)), + TargetError::Dropped(reason) => format!("dropped: {}", redacted_secret(reason)), + // Every other variant carries a free-form message that may name a host, path, or credential. + // Only the variant category is recorded, with any embedded value redacted. + TargetError::Network(value) => format!("network: {}", redacted_secret(value)), + TargetError::Request(value) => format!("request: {}", redacted_secret(value)), + TargetError::Timeout(value) => format!("timeout: {}", redacted_secret(value)), + TargetError::Storage(value) => format!("storage: {}", redacted_secret(value)), + TargetError::Authentication(_) => "authentication".to_string(), + TargetError::Configuration(_) => "configuration".to_string(), + other => format!("error: {}", redacted_secret(&other_error_category(other))), + }; + + let mut detail = summary; + truncate_to_char_boundary(&mut detail, FAILED_ERROR_DETAIL_MAX_LEN); + detail +} + +/// Truncates the string to at most max_len bytes, stepping the cut down to the nearest character +/// boundary so a multi-byte character straddling the cap is dropped whole rather than split. +fn truncate_to_char_boundary(value: &mut String, max_len: usize) { + if value.len() <= max_len { + return; + } + let mut cut = max_len; + while !value.is_char_boundary(cut) { + cut -= 1; + } + value.truncate(cut); +} + +/// Names the category of an otherwise free-form error without revealing its message. +fn other_error_category(error: &TargetError) -> String { + match error { + TargetError::Encoding(_) => "encoding".to_string(), + TargetError::Serialization(_) => "serialization".to_string(), + TargetError::Initialization(_) => "initialization".to_string(), + TargetError::Unknown(_) => "unknown".to_string(), + _ => "other".to_string(), + } +} + +/// Encodes a queued payload as a failed-store entry, extending its meta with the failure fields. +/// +/// Reuses the QueuedPayload format so the failed entry decodes through the same reader, preserving the +/// routing meta. The error_detail is the credential-free summary. The nats_msg_id is the resolved +/// dedup id, so an operator sees the id the server saw even for a pre-enable entry. +pub(crate) fn encode_failed_entry( + mut queued: QueuedPayload, + error_class: FailedErrorClass, + error: &TargetError, + retry_count: u32, + resolved_dedup_id: &str, +) -> Result, TargetError> { + let failed_at_unix_ms = SystemTime::now().duration_since(UNIX_EPOCH).unwrap_or_default().as_millis() as u64; + queued.meta.failure = Some(FailedEntryMeta { + error_class, + error_detail: build_failed_error_detail(error), + nats_msg_id: resolved_dedup_id.to_string(), + failed_at_unix_ms, + retry_count, + }); + queued.encode() +} + /// Ensures a rustls crypto provider is installed before any TLS operation. /// /// Multiple target modules (MySQL, Redis, Postgres, MQTT) need this because @@ -607,6 +779,97 @@ pub(crate) fn ensure_rustls_provider_installed() { } } +#[cfg(test)] +pub(crate) mod test_support { + use super::{EntityTarget, QueuedPayload, QueuedPayloadMeta}; + use crate::arn::TargetID; + use crate::store::{FailedEventStore, Key, QueueStore, Store}; + use crate::{StoreError, Target, TargetError}; + use async_trait::async_trait; + use rustfs_s3_types::EventName; + use std::path::PathBuf; + use std::sync::Arc; + use std::sync::atomic::{AtomicU64, Ordering}; + use uuid::Uuid; + + /// A minimal target for failed-store move tests: every delivery method succeeds, the optional + /// store backs the store and failed-store accessors, and final failures land on a shared counter. + #[derive(Clone)] + pub(crate) struct MoveTestTarget { + pub(crate) id: TargetID, + pub(crate) store: Option>>, + pub(crate) failed: Arc, + } + + #[async_trait] + impl Target for MoveTestTarget { + fn id(&self) -> TargetID { + self.id.clone() + } + async fn is_active(&self) -> Result { + Ok(true) + } + async fn save(&self, _event: Arc>) -> Result<(), TargetError> { + Ok(()) + } + async fn send_raw_from_store(&self, _key: Key, _body: Vec, _meta: QueuedPayloadMeta) -> Result<(), TargetError> { + Ok(()) + } + async fn close(&self) -> Result<(), TargetError> { + Ok(()) + } + fn store(&self) -> Option<&(dyn Store + Send + Sync)> { + self.store + .as_deref() + .map(|store| store as &(dyn Store<_, Error = StoreError, Key = Key> + Send + Sync)) + } + fn failed_store(&self) -> Option<&dyn FailedEventStore> { + self.store.as_deref().map(|store| store as &dyn FailedEventStore) + } + fn clone_dyn(&self) -> Box + Send + Sync> { + Box::new(self.clone()) + } + fn is_enabled(&self) -> bool { + true + } + fn record_final_failure(&self) { + self.failed.fetch_add(1, Ordering::Relaxed); + } + } + + pub(crate) fn move_test_target() -> Arc + Send + Sync> { + Arc::new(MoveTestTarget { + id: TargetID::new("target-a".to_string(), "nats".to_string()), + store: None, + failed: Arc::new(AtomicU64::new(0)), + }) + } + + pub(crate) fn move_test_target_with_store(store: Arc>) -> Arc + Send + Sync> { + Arc::new(MoveTestTarget { + id: TargetID::new("target-a".to_string(), "nats".to_string()), + store: Some(store), + failed: Arc::new(AtomicU64::new(0)), + }) + } + + pub(crate) fn failed_store_dir(name: &str) -> PathBuf { + std::env::temp_dir().join(format!("rustfs-failed-{name}-{}", Uuid::new_v4())) + } + + pub(crate) fn sample_queued(dedup_id: &str) -> QueuedPayload { + let mut meta = QueuedPayloadMeta::new( + EventName::ObjectCreatedPut, + "bucket-a".to_string(), + "obj.txt".to_string(), + "application/json", + 7, + ); + meta.dedup_id = dedup_id.to_string(); + QueuedPayload::new(meta, br#"{"x":1}"#.to_vec()) + } +} + #[cfg(test)] mod tls_state_tests { use super::{TargetTlsFingerprintState, TargetTlsGeneration, TargetTlsState}; @@ -743,6 +1006,45 @@ mod tests { assert_eq!(ChannelTargetType::Amqp.to_string(), "amqp"); } + #[test] + fn queued_payload_meta_omits_empty_dedup_id_on_serialization() { + let meta = QueuedPayloadMeta::new( + EventName::ObjectCreatedPut, + "bucket-a".to_string(), + "obj.txt".to_string(), + "application/json", + 7, + ); + assert!(meta.dedup_id.is_empty()); + + let json = serde_json::to_string(&meta).unwrap(); + assert!( + !json.contains("dedup_id"), + "an empty dedup id is skipped so stored bytes match the pre-feature format" + ); + + // An entry written without the field decodes to an empty dedup id. + let decoded: QueuedPayloadMeta = serde_json::from_str(&json).unwrap(); + assert!(decoded.dedup_id.is_empty()); + } + + #[test] + fn queued_payload_meta_round_trips_a_populated_dedup_id() { + let mut meta = QueuedPayloadMeta::new( + EventName::ObjectCreatedPut, + "bucket-a".to_string(), + "obj.txt".to_string(), + "application/json", + 7, + ); + meta.dedup_id = "minted-id".to_string(); + + let json = serde_json::to_string(&meta).unwrap(); + assert!(json.contains("dedup_id")); + let decoded: QueuedPayloadMeta = serde_json::from_str(&json).unwrap(); + assert_eq!(decoded.dedup_id, "minted-id"); + } + #[test] fn queued_payload_round_trips_meta_and_body() { let body = br#"{"ok":true}"#.to_vec(); @@ -1056,4 +1358,96 @@ mod tests { let err = QueuedPayload::decode(&encoded).unwrap_err(); assert!(err.to_string().contains("body length mismatch"), "unexpected error: {err}"); } + + use super::test_support::sample_queued; + + // The error_detail recorded on a failed entry is built from an allowlist of error categories and + // the fixed publish-error vocabulary, never the raw error rendering, so a credential embedded in + // a free-form error message is absent from the detail. + #[test] + fn build_failed_error_detail_redacts_credential_bearing_messages() { + let secret = "nats://user:supersecret@broker:4222"; + let cases = [ + TargetError::Network(secret.to_string()), + TargetError::Request(secret.to_string()), + TargetError::Timeout(secret.to_string()), + TargetError::Storage(secret.to_string()), + TargetError::Authentication(secret.to_string()), + TargetError::Configuration(secret.to_string()), + TargetError::Dropped(secret.to_string()), + TargetError::Unknown(secret.to_string()), + ]; + for error in cases { + let detail = build_failed_error_detail(&error); + assert!(!detail.contains("supersecret"), "detail leaked a credential: {detail}"); + assert!(!detail.contains("broker:4222"), "detail leaked a connection string: {detail}"); + } + } + + // The publish-error classifier sets the detail from its fixed vocabulary of kind labels and + // numeric codes, so the failed-entry detail names the cause without leaking a server-side + // message. + #[test] + fn build_failed_error_detail_uses_the_classified_publish_kind() { + let error = TargetError::JetStreamPublish { + retryable: false, + detail: "max payload exceeded".to_string(), + }; + let detail = build_failed_error_detail(&error); + assert_eq!(detail, "jetstream_publish: max payload exceeded"); + } + + // The persistence-boundary sanitizer replaces a JetStreamPublish detail that carries anything + // outside the fixed vocabulary with the fallback label, while a vocabulary detail passes through + // verbatim. + #[test] + fn build_failed_error_detail_sanitizes_a_hostile_jetstream_detail() { + let hostile = TargetError::JetStreamPublish { + retryable: false, + detail: "Boom! nats://user:pass@host/DROP".to_string(), + }; + assert_eq!(build_failed_error_detail(&hostile), "jetstream_publish: unrecognized detail"); + + let vocabulary = TargetError::JetStreamPublish { + retryable: false, + detail: "wrong last sequence".to_string(), + }; + assert_eq!(build_failed_error_detail(&vocabulary), "jetstream_publish: wrong last sequence"); + } + + // A summary whose cap lands inside a multi-byte character truncates at the preceding character + // boundary instead of panicking on a split character. A three-byte character does not divide + // the 256-byte cap, so one character straddles the cap by construction. + #[test] + fn truncate_to_char_boundary_handles_multi_byte_characters_at_the_cap() { + let mut value = "\u{4e2d}".repeat(FAILED_ERROR_DETAIL_MAX_LEN); + assert!(!value.is_char_boundary(FAILED_ERROR_DETAIL_MAX_LEN), "a character straddles the cap"); + truncate_to_char_boundary(&mut value, FAILED_ERROR_DETAIL_MAX_LEN); + assert!(value.len() <= FAILED_ERROR_DETAIL_MAX_LEN, "the result stays within the cap"); + assert!(value.is_char_boundary(value.len()), "the result ends on a character boundary"); + } + + // A terminal move writes a failed entry carrying the full failure meta extension and preserves the + // original routing metadata and dedup id through the reused QueuedPayload format. + #[test] + fn encode_failed_entry_carries_full_failure_meta() { + let queued = sample_queued("minted-id"); + let error = TargetError::JetStreamPublish { + retryable: false, + detail: "wrong last sequence".to_string(), + }; + let encoded = encode_failed_entry(queued, FailedErrorClass::Terminal, &error, 0, "minted-id").unwrap(); + let decoded = QueuedPayload::decode(&encoded).unwrap(); + + let failure = decoded.meta.failure.expect("a failed entry carries failure meta"); + assert_eq!(failure.error_class, FailedErrorClass::Terminal); + assert_eq!(failure.error_detail, "jetstream_publish: wrong last sequence"); + assert_eq!(failure.nats_msg_id, "minted-id"); + assert_eq!(failure.retry_count, 0); + assert!(failure.failed_at_unix_ms > 0); + // The original routing metadata survives the move. + assert_eq!(decoded.meta.bucket_name, "bucket-a"); + assert_eq!(decoded.meta.object_name, "obj.txt"); + assert_eq!(decoded.body, br#"{"x":1}"#); + } } diff --git a/crates/targets/src/target/mqtt.rs b/crates/targets/src/target/mqtt.rs index 1aeb102fa..bc4bf26a0 100644 --- a/crates/targets/src/target/mqtt.rs +++ b/crates/targets/src/target/mqtt.rs @@ -1693,8 +1693,11 @@ where } fn delivery_snapshot(&self) -> TargetDeliverySnapshot { - self.delivery_counters - .snapshot(self.store.as_deref().map_or(0, |store| store.len() as u64)) + self.delivery_counters.snapshot( + self.store.as_deref().map_or(0, |store| store.len() as u64), + // MQTT targets record no terminal failures and keep no failed store. + 0, + ) } fn record_final_failure(&self) { diff --git a/crates/targets/src/target/mysql.rs b/crates/targets/src/target/mysql.rs index 3a1508088..b14e252e8 100644 --- a/crates/targets/src/target/mysql.rs +++ b/crates/targets/src/target/mysql.rs @@ -1082,8 +1082,11 @@ where } fn delivery_snapshot(&self) -> TargetDeliverySnapshot { - self.delivery_counters - .snapshot(self.store.as_deref().map_or(0, |store| store.len() as u64)) + self.delivery_counters.snapshot( + self.store.as_deref().map_or(0, |store| store.len() as u64), + // MySQL targets record no terminal failures and keep no failed store. + 0, + ) } fn record_final_failure(&self) { diff --git a/crates/targets/src/target/nats/jetstream.rs b/crates/targets/src/target/nats/jetstream.rs new file mode 100644 index 000000000..836f248e5 --- /dev/null +++ b/crates/targets/src/target/nats/jetstream.rs @@ -0,0 +1,1236 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use super::NATSTarget; +use super::publish_error::classify_publish_error; +use super::validation::{gate_publish_on_stream_validation, validate_jetstream_stream}; +use crate::StoreError; +use crate::arn::TargetID; +use crate::error::TargetError; +use crate::plugin::PluginEvent; +use crate::runtime::{REPLAY_MAX_RETRIES, inter_attempt_backoff_sum, replay_backoff_term}; +use crate::store::{FailedEventStore, Key, Store}; +use crate::target::{ + FailedErrorClass, QueuedPayload, build_failed_error_detail, delete_stored_payload, encode_failed_entry, + truncate_to_char_boundary, +}; +use async_nats::header::{self, HeaderMap}; +use async_nats::jetstream::{self, context::PublishErrorKind}; +use sha2::{Digest, Sha256}; +use std::fmt::Write as _; +use std::sync::Arc; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::time::{Duration, Instant}; +use tracing::{debug, error, warn}; + +/// A cached JetStream context paired with its stream-validation verdict. The verdict starts +/// unvalidated for every built context, so a context rebuilt after a TLS rotation re-validates +/// before its first publish. A separate flag records whether the first passing validation has been +/// logged, so revalidations after a verdict reset stay at debug. Both start fresh with a rebuilt +/// context, and clones share them through the Arc. +#[derive(Clone)] +pub(crate) struct CachedJetStreamContext { + pub(crate) context: jetstream::Context, + pub(crate) stream_validated: Arc, + pub(crate) validation_logged: Arc, +} + +impl CachedJetStreamContext { + pub(crate) fn new(context: jetstream::Context) -> Self { + Self { + context, + validation_logged: Arc::new(AtomicBool::new(false)), + stream_validated: Arc::new(AtomicBool::new(false)), + } + } +} + +impl NATSTarget +where + E: PluginEvent, +{ + /// Returns the cached JetStream context, building it from the connected client on first use. The + /// context owns one background acker and one MaxAckPending semaphore, with its timeout set to the + /// ack timeout. + pub(crate) async fn jetstream_context(&self) -> Result { + // get_or_connect runs before the cache read so a detected TLS rotation swaps in a fresh + // context bound to the new client rather than one bound to the invalidated client. + let client = self.get_or_connect().await?; + + { + let guard = self.jetstream_context.lock().await; + if let Some(cached) = guard.as_ref() { + return Ok(cached.clone()); + } + } + + let ack_timeout = self.ack_timeout(); + + // Build under the lock so concurrent first callers do not each spawn an acker. new and + // set_timeout are synchronous, so no await holds the lock. + let mut guard = self.jetstream_context.lock().await; + let cached = guard.get_or_insert_with(|| { + let mut context = jetstream::new(client); + context.set_timeout(ack_timeout); + CachedJetStreamContext::new(context) + }); + Ok(cached.clone()) + } + + /// Returns the cached JetStream context after its configured stream has passed validation on the + /// bound connection. The verdict is cached with the context, so the steady-state cost is one + /// atomic load and a rebuilt context re-validates before its first publish. A connection-phase + /// failure classifies as NotConnected and a validation failure as a retryable JetStream publish + /// error, so either leaves the entry queued. + pub(crate) async fn validated_jetstream_context(&self) -> Result { + let cached = self.jetstream_context().await.map_err(jetstream_connect_error)?; + if !cached.stream_validated.load(Ordering::Acquire) { + let verdict = + validate_jetstream_stream(&cached.context, &self.args, &self.id.to_string(), Some(&cached.validation_logged)) + .await; + gate_publish_on_stream_validation(verdict, &cached.stream_validated)?; + } + Ok(cached) + } + + /// Publishes the body with the Nats-Msg-Id dedup header and awaits the real PublishAck. The whole + /// attempt, including connection establishment, stream validation, and the ack await, is bounded + /// by a single deadline equal to the ack timeout. Dropping a timed-out attempt releases its + /// resources, the broker suite exercises the drop path. + pub(crate) async fn publish_jetstream(&self, body: Vec, dedup_id: &str) -> Result<(), TargetError> { + attempt_within_deadline(self.ack_timeout(), self.publish_jetstream_attempt(body, dedup_id)).await + } + + /// Runs one publish attempt with no deadline of its own: acquires the validated context, + /// publishes with the dedup header, and awaits the PublishAck. + async fn publish_jetstream_attempt(&self, body: Vec, dedup_id: &str) -> Result<(), TargetError> { + let cached = self.validated_jetstream_context().await?; + + let mut headers = HeaderMap::new(); + headers.insert(header::NATS_MESSAGE_ID, dedup_id); + + // A stream-not-found outcome on either await resets the verdict before classification, so + // the next attempt re-validates instead of publishing blind into a missing stream. + let started = Instant::now(); + let ack_future = cached + .context + .publish_with_headers(self.args.subject.clone(), headers, body.into()) + .await + .inspect_err(|err| reset_verdict_on_stream_not_found(err, &cached.stream_validated)) + .map_err(|err| classify_publish_error(&err))?; + + let ack = ack_future + .await + .inspect_err(|err| reset_verdict_on_stream_not_found(err, &cached.stream_validated)) + .map_err(|err| classify_publish_error(&err))?; + let latency_ms = started.elapsed().as_millis() as u64; + + self.finish_acknowledged_publish(&ack.stream, ack.sequence, ack.duplicate, latency_ms, dedup_id, &cached.stream_validated) + } + + /// Completes a publish from the returned acknowledgment: verifies the acknowledging stream + /// matches the configured name, records the delivery, and logs the outcome. A mismatch resets + /// the verdict and returns a retryable error, so the entry stays queued and the next attempt + /// re-validates. + fn finish_acknowledged_publish( + &self, + ack_stream: &str, + sequence: u64, + duplicate: bool, + ack_latency_ms: u64, + dedup_id: &str, + stream_validated: &AtomicBool, + ) -> Result<(), TargetError> { + let configured_stream = self.args.jetstream_stream_name.as_deref().unwrap_or_default(); + if ack_stream != configured_stream { + stream_validated.store(false, Ordering::Release); + // Server-controlled text, length-bounded before logging. + let mut acknowledged_stream = ack_stream.to_string(); + truncate_to_char_boundary(&mut acknowledged_stream, ACK_STREAM_LOG_MAX_LEN); + warn!( + target_id = %self.id, + expected_stream = %configured_stream, + acknowledged_stream = %acknowledged_stream, + dedup_id = %dedup_id, + "JetStream publish acknowledged by an unexpected stream, verdict reset for re-validation" + ); + return Err(TargetError::JetStreamPublish { + retryable: true, + detail: ACK_STREAM_MISMATCH_DETAIL.to_string(), + }); + } + + // A first-attempt duplicate would mean two events collided on one id, which the minted uuid + // prevents, so the flag is logged rather than treated as an error. + debug!( + target_id = %self.id, + stream = %ack_stream, + sequence, + duplicate, + ack_latency_ms, + "JetStream publish acknowledged" + ); + + self.delivery_counters.record_success(); + Ok(()) + } +} + +/// Resolves the dedup id for a stored entry: the minted id when present, otherwise a hash of the +/// entry's on-disk key. Deriving from the key rather than the body keeps two entries with identical +/// bodies on separate ids, so the server does not drop the second as a duplicate. +pub(crate) fn resolve_dedup_id(meta_dedup_id: &str, key: &Key) -> String { + if !meta_dedup_id.is_empty() { + return meta_dedup_id.to_string(); + } + let digest = Sha256::digest(key.to_key_string().as_bytes()); + let mut id = String::with_capacity(digest.len() * 2); + for byte in digest { + // Writing a formatted byte into a String cannot fail. + let _ = write!(id, "{byte:02x}"); + } + id +} + +/// Moves a live queue entry that failed terminally to the failed-events store, then removes it from +/// the live queue. The failed entry is written first and the live entry deleted only after, so a +/// crash between the two leaves a recoverable duplicate rather than a loss. +pub(crate) async fn move_entry_to_failed_store( + store: &(dyn Store + Send), + failed_store: &dyn FailedEventStore, + target_id: &TargetID, + key: &Key, + error: &TargetError, + retry_count: u32, +) -> Result<(), TargetError> { + let raw = match store.get_raw(key) { + Ok(raw) => raw, + // A definitive ack from a concurrent path already cleared the entry. Nothing to move. + Err(StoreError::NotFound) => return Ok(()), + Err(err) => { + return Err(TargetError::Storage(format!( + "Failed to read queued payload before failed-store move: {err}" + ))); + } + }; + + let queued = QueuedPayload::decode(&raw) + .map_err(|err| TargetError::Storage(format!("Failed to decode queued payload before failed-store move: {err}")))?; + let meta = queued.meta.clone(); + // Same resolution the publish path applies, so the recorded id matches what the server saw even + // for a pre-enable entry with an empty stored id. + let resolved_dedup_id = resolve_dedup_id(&meta.dedup_id, key); + let encoded = encode_failed_entry(queued, FailedErrorClass::Terminal, error, retry_count, &resolved_dedup_id)?; + + failed_store + .put_failed_raw(&key.name, &encoded) + .map_err(|err| TargetError::Storage(format!("Failed to write failed-store entry: {err}")))?; + + // Error level: a terminal failure means a lost notification an operator must act on. Names + // routing metadata and the redacted detail only, never the event body or a credential. + error!( + target_id = %target_id, + error_class = FailedErrorClass::Terminal.as_str(), + error_detail = %build_failed_error_detail(error), + bucket = %meta.bucket_name, + object = %meta.object_name, + event_name = %meta.event_name.as_str(), + nats_msg_id = %resolved_dedup_id, + retry_count, + "event moved to the failed-events store" + ); + + delete_stored_payload(store, key)?; + // The failure count is recorded by the replay hook, not here, so it counts once. + Ok(()) +} + +/// Drains a JetStream context and drops it so its background acker exits. async-nats keeps the acker +/// task alive with no Drop on Context, so wait_for_acks drains every in-flight publish before the +/// final drop closes the ack channel and ends the task. None is a no-op. +pub(crate) async fn drain_jetstream_context(context: Option) { + if let Some(context) = context { + context.wait_for_acks().await; + } +} + +/// Maps a connectivity or configuration failure from establishing the JetStream connection to +/// NotConnected, so the replay worker retries it and leaves the entry queued rather than parking it +/// in the failed store. Errors already classified at the publish level pass through unchanged. +fn jetstream_connect_error(err: TargetError) -> TargetError { + match err { + TargetError::Network(_) | TargetError::Configuration(_) => TargetError::NotConnected, + other => other, + } +} + +/// Detail carried by the retryable publish error raised when a publish attempt exceeds the ack +/// timeout. +pub(crate) const ATTEMPT_DEADLINE_DETAIL: &str = "attempt deadline exceeded"; + +/// Bounds one publish attempt by the ack timeout. An attempt that does not resolve within the +/// deadline is dropped and classified as a retryable JetStream publish error, so the replay loop +/// counts it toward the bounded budget. An inner client timeout can resolve first with its own +/// timed-out error, and both paths classify retryable. +async fn attempt_within_deadline(ack_timeout: Duration, attempt: F) -> Result<(), TargetError> +where + F: std::future::Future>, +{ + match tokio::time::timeout(ack_timeout, attempt).await { + Ok(result) => result, + Err(_elapsed) => Err(TargetError::JetStreamPublish { + retryable: true, + detail: ATTEMPT_DEADLINE_DETAIL.to_string(), + }), + } +} + +/// Detail carried by the retryable publish error raised when an acknowledgment names a stream other +/// than the configured one. +pub(crate) const ACK_STREAM_MISMATCH_DETAIL: &str = "acknowledged by an unexpected stream"; + +/// Byte bound applied to the acknowledged stream name before it is logged. +const ACK_STREAM_LOG_MAX_LEN: usize = 64; + +/// Detail carried by the retryable publish error raised when the server reports the configured +/// stream does not exist. +pub(crate) const STREAM_NOT_FOUND_DETAIL: &str = "stream not found"; + +/// Resets the validation verdict when a publish outcome reports the configured stream does not +/// exist, so the next attempt re-validates. Keys off the raw error kind before classification, +/// keeping the classifier pure. +fn reset_verdict_on_stream_not_found(err: &async_nats::jetstream::context::PublishError, stream_validated: &AtomicBool) { + if matches!(err.kind(), PublishErrorKind::StreamNotFound) { + stream_validated.store(false, Ordering::Release); + } +} + +/// Conservative upper bound on the wall-clock retry span of a stored entry, used as the minimum +/// acceptable stream duplicate_window. Sums the per-attempt ack ceiling across REPLAY_MAX_RETRIES +/// with the realized backoff schedule plus a headroom term. The headroom is a deliberate +/// conservatism margin that holds the required duplicate_window above the realized retry span. A +/// duplicate_window at or above this keeps a late same-id retry inside the dedup window. Saturating +/// arithmetic keeps it panic-free. +pub(crate) fn retry_lifetime(ack_timeout: Duration) -> Duration { + let ack_span = ack_timeout.saturating_mul(REPLAY_MAX_RETRIES as u32); + let realized_backoff = inter_attempt_backoff_sum(REPLAY_MAX_RETRIES); + let headroom = replay_backoff_term(REPLAY_MAX_RETRIES as u32); + ack_span.saturating_add(realized_backoff).saturating_add(headroom) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::Target; + use crate::arn::TargetID; + use crate::store::{FailedEventStore, QueueStore}; + use crate::target::TargetType; + use crate::target::nats::test_support::*; + use crate::target::nats::validation::STREAM_VALIDATION_FAILED_DETAIL; + use crate::target::test_support::{ + MoveTestTarget, failed_store_dir, move_test_target, move_test_target_with_store, sample_queued, + }; + use crate::target::{build_target_tls_fingerprint, persist_queued_payload_to_store}; + use async_nats::jetstream::context::PublishError; + use rustfs_config::NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS; + use std::sync::atomic::AtomicU64; + use uuid::Uuid; + + #[test] + fn enabled_target_mints_dedup_id_stored_in_meta() { + let target = nats_target(jetstream_args(TargetType::NotifyEvent)); + let queued = target.build_queued_payload(&sample_event()).expect("payload builds"); + assert!(!queued.meta.dedup_id.is_empty(), "an enabled target mints a dedup id"); + // The minted id is a uuid string, distinct from any event field. + assert!(Uuid::parse_str(&queued.meta.dedup_id).is_ok(), "the dedup id is a uuid"); + } + + #[test] + fn disabled_target_mints_no_dedup_id() { + let mut args = jetstream_args(TargetType::NotifyEvent); + args.jetstream_enable = Some(false); + let target = nats_target(args); + let queued = target.build_queued_payload(&sample_event()).expect("payload builds"); + assert!(queued.meta.dedup_id.is_empty(), "a disabled target mints no dedup id"); + } + + #[test] + fn dedup_id_is_stable_across_reads_of_the_same_stored_entry() { + for target_type in [TargetType::NotifyEvent, TargetType::AuditLog] { + let target = nats_target(jetstream_args(target_type)); + let queued = target.build_queued_payload(&sample_event()).expect("payload builds"); + + let encoded = queued.encode().expect("payload encodes"); + let first = QueuedPayload::decode(&encoded).expect("payload decodes"); + let second = QueuedPayload::decode(&encoded).expect("payload decodes"); + + assert_eq!( + first.meta.dedup_id, second.meta.dedup_id, + "reading the same stored entry twice yields the same id for {target_type}" + ); + assert_eq!( + first.meta.dedup_id, queued.meta.dedup_id, + "the stored id is the minted id for {target_type}" + ); + } + } + + #[test] + fn dedup_id_is_distinct_across_entries() { + for target_type in [TargetType::NotifyEvent, TargetType::AuditLog] { + let target = nats_target(jetstream_args(target_type)); + let first = target.build_queued_payload(&sample_event()).expect("payload builds"); + let second = target.build_queued_payload(&sample_event()).expect("payload builds"); + assert_ne!( + first.meta.dedup_id, second.meta.dedup_id, + "two distinct entries get distinct ids for {target_type}" + ); + } + } + + #[test] + fn resolve_dedup_id_prefers_the_stored_minted_id() { + let id = resolve_dedup_id("minted-uuid", &sample_stored_key("entry-one")); + assert_eq!(id, "minted-uuid"); + } + + #[test] + fn resolve_dedup_id_derives_distinct_ids_for_entries_with_equal_bodies() { + // Distinct on-disk keys yield distinct fallback ids even for identical bodies. + let first = resolve_dedup_id("", &sample_stored_key("entry-one")); + let second = resolve_dedup_id("", &sample_stored_key("entry-two")); + assert_ne!(first, second, "distinct entries derive distinct ids"); + assert_eq!(first.len(), 64, "the derived id is a sha256 hex digest"); + } + + #[test] + fn resolve_dedup_id_is_stable_across_replays_of_one_entry() { + // The same entry keeps its key across replays, so the derived id is identical each time. + let key = sample_stored_key("entry-one"); + assert_eq!(resolve_dedup_id("", &key), resolve_dedup_id("", &key)); + } + + #[test] + fn jetstream_connect_error_maps_network_to_not_connected() { + // A connect-phase connectivity failure classifies as NotConnected so the entry stays queued. + let mapped = jetstream_connect_error(TargetError::Network("Failed to connect to NATS server: refused".to_string())); + assert!( + matches!(mapped, TargetError::NotConnected), + "a connect-phase connectivity failure classifies as NotConnected, got {mapped:?}" + ); + } + + #[test] + fn jetstream_connect_error_maps_configuration_to_not_connected() { + // A connect-phase configuration failure (unreadable TLS material after start) classifies as NotConnected. + let mapped = jetstream_connect_error(TargetError::Configuration("Failed to read TLS material".to_string())); + assert!( + matches!(mapped, TargetError::NotConnected), + "a connect-phase configuration failure classifies as NotConnected, got {mapped:?}" + ); + } + + #[test] + fn jetstream_connect_error_passes_through_classified_errors() { + // An already-classified publish error passes through, so the mapping cannot mask a terminal cause. + let terminal = jetstream_connect_error(TargetError::JetStreamPublish { + retryable: false, + detail: "stream sealed".to_string(), + }); + match terminal { + TargetError::JetStreamPublish { retryable, .. } => assert!(!retryable, "a terminal publish error stays terminal"), + other => panic!("expected JetStreamPublish, got {other:?}"), + } + } + + #[tokio::test] + async fn publish_jetstream_maps_a_connect_failure_to_not_connected() { + // A connect-level failure on the publish path surfaces as NotConnected. Port 1 refuses without a server. + let queue_dir = tempfile::tempdir().expect("queue dir"); + let mut args = jetstream_args(TargetType::NotifyEvent); + args.address = "nats://127.0.0.1:1".to_string(); + args.queue_dir = queue_dir.path().to_str().unwrap().to_string(); + let target = nats_target(args); + + let err = target + .publish_jetstream(b"body".to_vec(), "dedup-connect-fail") + .await + .expect_err("an unreachable broker fails the publish"); + assert!( + matches!(err, TargetError::NotConnected), + "a connect-level failure on the publish path surfaces as NotConnected, got {err:?}" + ); + } + + #[test] + fn a_wrong_stream_ack_resets_the_verdict_and_classifies_retryable() { + // A wrong-stream ack resets the verdict and returns the retryable mismatch error. A matching ack completes the publish and keeps the verdict. + let target = nats_target(jetstream_args(TargetType::NotifyEvent)); + let stream_validated = AtomicBool::new(true); + let err = target + .finish_acknowledged_publish("SOME_OTHER_STREAM", 7, false, 12, "dedup-mismatch", &stream_validated) + .expect_err("an unexpected acknowledging stream is rejected"); + match err { + TargetError::JetStreamPublish { retryable, detail } => { + assert!(retryable, "a wrong-stream ack is retryable so the entry stays queued"); + assert_eq!(detail, ACK_STREAM_MISMATCH_DETAIL, "the fixed mismatch label is the detail"); + } + other => panic!("expected a retryable JetStreamPublish, got {other:?}"), + } + assert!( + !stream_validated.load(Ordering::SeqCst), + "the verdict resets so the next attempt re-validates the stream" + ); + + // A matching acknowledging stream completes the publish and keeps the recorded verdict. + let validated = AtomicBool::new(true); + target + .finish_acknowledged_publish("RUSTFS_EVENTS", 8, false, 12, "dedup-match", &validated) + .expect("a matching stream ack completes the publish"); + assert!(validated.load(Ordering::SeqCst), "a matching ack keeps the recorded verdict"); + } + + #[test] + fn stream_not_found_publish_outcome_resets_the_verdict() { + // A StreamNotFound outcome resets the verdict. Any other kind leaves it untouched. + let stream_validated = AtomicBool::new(true); + reset_verdict_on_stream_not_found(&PublishError::new(PublishErrorKind::StreamNotFound), &stream_validated); + assert!(!stream_validated.load(Ordering::SeqCst), "a stream-not-found outcome resets the verdict"); + + for kind in [PublishErrorKind::TimedOut, PublishErrorKind::MaxPayloadExceeded] { + let verdict = AtomicBool::new(true); + reset_verdict_on_stream_not_found(&PublishError::new(kind), &verdict); + assert!(verdict.load(Ordering::SeqCst), "{kind} leaves the verdict intact"); + } + } + + #[tokio::test] + async fn a_failed_publish_attempt_leaves_the_queued_entry_intact() { + // A publish that fails before any send leaves the queued entry, since only a returned PublishAck clears it. + let queue_dir = tempfile::tempdir().expect("queue dir"); + let mut args = jetstream_args(TargetType::NotifyEvent); + args.address = "nats://127.0.0.1:1".to_string(); + let target = nats_target_with_store(args, queue_dir.path().to_str().unwrap()); + + let queued = target.build_queued_payload(&sample_event()).expect("payload builds"); + let store = target.store().expect("a store is configured"); + persist_queued_payload_to_store(store, &queued).expect("entry persists"); + let key = store.list().pop().expect("the entry is listed"); + + let err = target.send_from_store(key.clone()).await.expect_err("the attempt fails"); + assert!( + matches!(err, TargetError::NotConnected), + "the failed connect surfaces as a connectivity error, got {err:?}" + ); + assert_eq!(store.len(), 1, "the queued entry survives the failed attempt"); + assert!(store.get_raw(&key).is_ok(), "the surviving entry is still readable"); + } + + #[tokio::test(start_paused = true)] + async fn publish_path_consults_the_stream_gate_before_publishing() { + // The publish path runs stream validation first, so an unvalidated stream fails with the validation label rather than the publish call. + let client = async_nats::ConnectOptions::new() + .retry_on_initial_connect() + .connect("nats://127.0.0.1:1") + .await + .expect("retry_on_initial_connect returns without a live server"); + + let target = nats_target(jetstream_args(TargetType::NotifyEvent)); + let fingerprint = build_target_tls_fingerprint("", "", "").await.expect("fingerprint builds"); + target.tls_state.lock().refresh(fingerprint); + *target.client.lock().await = Some(client.clone()); + + let mut context = jetstream::new(client); + // Shorter than the 30s attempt deadline, so the validation lookup elapses first. + context.set_timeout(Duration::from_secs(5)); + let cached = CachedJetStreamContext::new(context); + let verdict = Arc::clone(&cached.stream_validated); + *target.jetstream_context.lock().await = Some(cached); + + let err = target + .publish_jetstream(b"body".to_vec(), "dedup-gate-wiring") + .await + .expect_err("the unvalidated stream blocks the publish"); + match err { + TargetError::JetStreamPublish { retryable, detail } => { + assert!(retryable, "the gate failure is retryable"); + assert_eq!( + detail, STREAM_VALIDATION_FAILED_DETAIL, + "the failure names the stream validation, not the publish" + ); + } + other => panic!("expected a retryable JetStreamPublish, got {other:?}"), + } + assert!(!verdict.load(Ordering::SeqCst), "the verdict stays closed after the failed validation"); + } + + #[tokio::test(start_paused = true)] + async fn is_active_reports_a_failing_stream_instead_of_healthy() { + // is_active consults the stream-validation gate, so a reachable broker with a failing stream reports the error, not healthy. + let client = async_nats::ConnectOptions::new() + .retry_on_initial_connect() + .connect("nats://127.0.0.1:1") + .await + .expect("retry_on_initial_connect returns without a live server"); + + let target = nats_target(jetstream_args(TargetType::NotifyEvent)); + let fingerprint = build_target_tls_fingerprint("", "", "").await.expect("fingerprint builds"); + target.tls_state.lock().refresh(fingerprint); + *target.client.lock().await = Some(client.clone()); + + let mut context = jetstream::new(client); + context.set_timeout(Duration::from_secs(5)); + let cached = CachedJetStreamContext::new(context); + let verdict = Arc::clone(&cached.stream_validated); + *target.jetstream_context.lock().await = Some(cached); + + let err = target.is_active().await.expect_err("a failing stream fails the health check"); + match err { + TargetError::JetStreamPublish { retryable, detail } => { + assert!(retryable, "the health failure carries the gate classification"); + assert_eq!( + detail, STREAM_VALIDATION_FAILED_DETAIL, + "the health check reports the stream validation failure" + ); + } + other => panic!("expected the gate classification, got {other:?}"), + } + + // A reset verdict forces the next check to re-run validation. + verdict.store(true, Ordering::SeqCst); + verdict.store(false, Ordering::SeqCst); + let err = target + .is_active() + .await + .expect_err("a freshly reset verdict re-runs validation"); + match err { + TargetError::JetStreamPublish { retryable, detail } => { + assert!(retryable, "the re-validation failure carries the gate classification"); + assert_eq!( + detail, STREAM_VALIDATION_FAILED_DETAIL, + "the reset verdict forces the health check to re-validate the stream" + ); + } + other => panic!("expected the gate classification after a reset, got {other:?}"), + } + } + + #[tokio::test(start_paused = true)] + async fn attempt_deadline_fires_against_a_pending_publish_and_classifies_retryable() { + // A publish that never resolves is cut off at the ack timeout and classified retryable. + let result = attempt_within_deadline(Duration::from_secs(30), std::future::pending::>()).await; + match result { + Err(TargetError::JetStreamPublish { retryable, detail }) => { + assert!(retryable, "an elapsed attempt deadline is retryable"); + assert_eq!(detail, ATTEMPT_DEADLINE_DETAIL); + } + other => panic!("expected a retryable JetStreamPublish, got {other:?}"), + } + } + + #[tokio::test(start_paused = true)] + async fn attempt_deadline_passes_through_a_result_within_the_deadline() { + attempt_within_deadline(Duration::from_secs(30), async { Ok(()) }) + .await + .expect("a resolved attempt passes its result through"); + + // An inner client timeout can resolve first and keeps its own retryable classification. + let inner = TargetError::JetStreamPublish { + retryable: true, + detail: "timed out".to_string(), + }; + let err = attempt_within_deadline(Duration::from_secs(30), async { Err(inner) }) + .await + .expect_err("an inner error passes through"); + match err { + TargetError::JetStreamPublish { retryable, detail } => { + assert!(retryable, "the inner timed-out classification is retryable"); + assert_eq!(detail, "timed out", "the inner detail is preserved"); + } + other => panic!("expected a retryable JetStreamPublish, got {other:?}"), + } + } + + #[test] + fn ack_timeout_uses_the_configured_value() { + let target = nats_target(jetstream_args(TargetType::NotifyEvent)); + assert_eq!(target.ack_timeout(), Duration::from_secs(30)); + + let mut args = jetstream_args(TargetType::NotifyEvent); + args.jetstream_ack_timeout_secs = None; + let target = nats_target(args); + assert_eq!( + target.ack_timeout(), + Duration::from_secs(NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS), + "an absent ack timeout uses the default" + ); + } + + #[test] + fn retry_lifetime_covers_every_attempt_ack_wait_plus_full_backoff() { + // The realized backoff schedule at the default base is 4 + 8 + 16 + 32 = 60s, plus a 64s headroom term = 124s. At the 30s ack timeout the worst case is 5 * 30 + 124 = 274s, the floor a duplicate_window must clear. + assert_eq!(retry_lifetime(Duration::from_secs(30)), Duration::from_secs(274)); + // With a zero ack timeout only the backoff schedule and the headroom term remain. + assert_eq!(retry_lifetime(Duration::from_secs(0)), Duration::from_secs(124)); + } + + #[test] + fn retry_lifetime_backoff_equals_the_realized_schedule_plus_headroom() { + // The duplicate-window backoff span must equal the worker's realized sleep schedule plus one final-shift headroom term. + let mut realized = Duration::ZERO; + for shift in 1..REPLAY_MAX_RETRIES as u32 { + realized += replay_backoff_term(shift); + } + assert_eq!( + realized, + inter_attempt_backoff_sum(REPLAY_MAX_RETRIES), + "the shared sum matches the worker sleep schedule" + ); + let headroom = replay_backoff_term(REPLAY_MAX_RETRIES as u32); + assert_eq!(retry_lifetime(Duration::ZERO), realized + headroom); + assert_eq!(retry_lifetime(Duration::ZERO), Duration::from_secs(124)); + } + + #[test] + fn clone_box_shares_one_cached_context_handle() { + // One acker and one backpressure semaphore serve every clone of the target. + let target = nats_target(jetstream_args(TargetType::NotifyEvent)); + let before = Arc::strong_count(&target.jetstream_context); + let _clone = target.clone_box(); + assert_eq!( + Arc::strong_count(&target.jetstream_context), + before + 1, + "clone_box shares the one context handle rather than building a new one" + ); + } + + #[tokio::test] + async fn close_keeps_a_queued_entry_for_replay() { + // Close releases cached handles but never durable entries, so a queued entry survives and replays. + let queue_dir = tempfile::tempdir().expect("queue dir"); + let target = nats_target_with_store(jetstream_args(TargetType::NotifyEvent), queue_dir.path().to_str().unwrap()); + let queued = target.build_queued_payload(&sample_event()).expect("payload builds"); + let store = target.store().expect("a store is configured"); + persist_queued_payload_to_store(store, &queued).expect("entry persists"); + assert_eq!(store.len(), 1, "the entry is queued before close"); + + target.close().await.expect("close succeeds"); + + let store_after = target.store().expect("the store handle is still held"); + assert_eq!(store_after.len(), 1, "the queued entry survives close"); + assert_eq!(store_after.list().len(), 1, "the surviving entry is listable for replay"); + } + + #[tokio::test] + async fn flag_off_close_matches_the_baseline() { + // With JetStream off no context is cached and close drains only the client. A queued entry still survives. + let queue_dir = tempfile::tempdir().expect("queue dir"); + let target = nats_target_with_store(base_args(), queue_dir.path().to_str().unwrap()); + assert!(!target.jetstream_enabled(), "the flag is off"); + assert!(target.jetstream_context.lock().await.is_none(), "no context is cached with the flag off"); + + let queued = target.build_queued_payload(&sample_event()).expect("payload builds"); + let store = target.store().expect("a store is configured"); + persist_queued_payload_to_store(store, &queued).expect("entry persists"); + + target.close().await.expect("flag-off close succeeds"); + + assert!( + target.jetstream_context.lock().await.is_none(), + "the flag-off close leaves the context field empty" + ); + assert_eq!(target.store().expect("store held").len(), 1, "the queued entry survives a flag-off close"); + } + + #[tokio::test] + async fn drain_jetstream_context_handles_none() { + // A None context (nothing cached) is a no-op. + drain_jetstream_context(None).await; + } + + #[tokio::test] + #[ignore] + async fn tls_change_rebuilds_the_context_and_drains_the_old_acker() { + // A TLS fingerprint change on the publish path rebuilds the cached context from the new client and drains the old acker. + let subject = format!("rustfs.tlsrebuild.{}", Uuid::new_v4().simple()); + let stream_name = format!("RUSTFS_TLSREBUILD_{}", Uuid::new_v4().simple()); + + let provision_client = async_nats::connect(broker_url()).await.expect("connect to provision"); + let provision_context = jetstream::new(provision_client); + provision_context + .create_stream(jetstream::stream::Config { + name: stream_name.clone(), + subjects: vec![subject.clone()], + duplicate_window: Duration::from_secs(300), + ..Default::default() + }) + .await + .expect("provision the stream"); + + let queue_dir = tempfile::tempdir().expect("queue dir"); + let mut args = jetstream_args(TargetType::NotifyEvent); + args.address = broker_url(); + args.subject = subject.clone(); + args.jetstream_stream_name = Some(stream_name.clone()); + args.queue_dir = queue_dir.path().to_str().unwrap().to_string(); + args.tls_required = false; + + let target = nats_target(args); + + // First publish builds and caches context A bound to the first connection. + target + .publish_jetstream(b"first".to_vec(), "dedup-first") + .await + .expect("publish A"); + let context_a = target.jetstream_context().await.expect("context A is cached").context; + // Read the client_id from the cache without get_or_connect, so the read never triggers a reconnect. + let cached_client_id = || async { + target + .client + .lock() + .await + .as_ref() + .expect("a client is cached") + .server_info() + .client_id + }; + let client_id_a = cached_client_id().await; + + // Reset the fingerprint so the next publish detects a change, as a rotated cert would. + target.tls_state.lock().reset(); + + // The next publish detects the change, reconnects, rebuilds the context, and drains context A. + target + .publish_jetstream(b"second".to_vec(), "dedup-second") + .await + .expect("publish B on the rebuilt context"); + let client_id_b = cached_client_id().await; + + // The cached client id changed, so only the publish path could have reconnected onto a new connection. + assert_ne!( + client_id_a, client_id_b, + "the production publish path rebuilt onto a fresh connection after the material change" + ); + + // A further publish on the rebuilt context acks. + target + .publish_jetstream(b"third".to_vec(), "dedup-third") + .await + .expect("third publish on the rebuilt context acks"); + + // Context A was drained in the swap, so a fresh wait returns promptly. + tokio::time::timeout(Duration::from_secs(2), context_a.wait_for_acks()) + .await + .expect("the old context is fully drained, its acker has nothing parked"); + + // Best-effort teardown. + let _ = provision_context.delete_stream(&stream_name).await; + } + + #[tokio::test] + #[ignore] + async fn tls_change_after_a_failed_reconnect_still_rebuilds_the_context() { + // A rotation detected while the broker is unreachable does not orphan the cached context: a failed reconnect followed by a successful one ends bound to the rebuilt context. + let subject = format!("rustfs.tlsfail.{}", Uuid::new_v4().simple()); + let stream_name = format!("RUSTFS_TLSFAIL_{}", Uuid::new_v4().simple()); + + let provision_client = async_nats::connect(broker_url()).await.expect("connect to provision"); + let provision_context = jetstream::new(provision_client); + provision_context + .create_stream(jetstream::stream::Config { + name: stream_name.clone(), + subjects: vec![subject.clone()], + duplicate_window: Duration::from_secs(300), + ..Default::default() + }) + .await + .expect("provision the stream"); + + let queue_dir = tempfile::tempdir().expect("queue dir"); + let mut args = jetstream_args(TargetType::NotifyEvent); + args.address = broker_url(); + args.subject = subject.clone(); + args.jetstream_stream_name = Some(stream_name.clone()); + args.queue_dir = queue_dir.path().to_str().unwrap().to_string(); + args.tls_required = false; + + let mut target = nats_target(args); + + // First production publish builds and caches context A bound to the first connection. + target + .publish_jetstream(b"first".to_vec(), "dedup-first") + .await + .expect("publish A"); + let context_a = target.jetstream_context().await.expect("context A is cached").context; + let client_id_a = context_a.client().server_info().client_id; + + // Signal a rotation, then point at a dead port so the first reconnect fails. + target.tls_state.lock().reset(); + let good_address = target.args.address.clone(); + target.args.address = "nats://127.0.0.1:1".to_string(); + + let failed = target.publish_jetstream(b"second".to_vec(), "dedup-second").await; + assert!( + failed.is_err(), + "the first reconnect after the change fails while the broker is unreachable" + ); + + // The broker returns. A later publish reconnects, rebuilds the context, and drains the old one. + target.args.address = good_address; + target + .publish_jetstream(b"third".to_vec(), "dedup-third") + .await + .expect("publish on the rebuilt context"); + + // The cached context is bound to a fresh connection, not the pre-rotation one. + let cached_context = target + .jetstream_context + .lock() + .await + .as_ref() + .expect("a context is cached") + .clone() + .context; + let cached_context_client_id = cached_context.client().server_info().client_id; + assert_ne!( + cached_context_client_id, client_id_a, + "the cached context was rebuilt onto a fresh connection after the failed-then-successful reconnect" + ); + + // Context A was drained in the swap, so a fresh wait returns promptly. + tokio::time::timeout(Duration::from_secs(2), context_a.wait_for_acks()) + .await + .expect("the pre-rotation context is drained, its acker has nothing parked"); + + let _ = provision_context.delete_stream(&stream_name).await; + } + + #[tokio::test] + #[ignore] + async fn publish_gate_rejects_an_unsafe_stream_and_heals_after_the_stream_is_fixed() { + // The gate rejects every publish while the stream's duplicate window is below the retry lifetime, and starts publishing once the operator widens it, without a restart. + let subject = format!("rustfs.gate.{}", Uuid::new_v4().simple()); + let stream_name = format!("RUSTFS_GATE_{}", Uuid::new_v4().simple()); + + let provision_client = async_nats::connect(broker_url()).await.expect("connect to provision"); + let provision_context = jetstream::new(provision_client); + provision_context + .create_stream(jetstream::stream::Config { + name: stream_name.clone(), + subjects: vec![subject.clone()], + // Below the 274s required at the default 30s ack timeout. + duplicate_window: Duration::from_secs(60), + ..Default::default() + }) + .await + .expect("provision the unsafe stream"); + + let queue_dir = tempfile::tempdir().expect("queue dir"); + let mut args = jetstream_args(TargetType::NotifyEvent); + args.address = broker_url(); + args.subject = subject.clone(); + args.jetstream_stream_name = Some(stream_name.clone()); + args.queue_dir = queue_dir.path().to_str().unwrap().to_string(); + args.tls_required = false; + + let target = nats_target(args); + + let err = target + .publish_jetstream(b"first".to_vec(), "dedup-gate-first") + .await + .expect_err("no publish proceeds against a stream that fails validation"); + assert!( + matches!(err, TargetError::JetStreamPublish { retryable: true, .. }), + "the gate failure is retryable, got {err:?}" + ); + + // The operator widens the window past the retry lifetime, and the next attempt re-validates and publishes. + provision_context + .update_stream(jetstream::stream::Config { + name: stream_name.clone(), + subjects: vec![subject.clone()], + duplicate_window: Duration::from_secs(300), + ..Default::default() + }) + .await + .expect("widen the duplicate window"); + + target + .publish_jetstream(b"second".to_vec(), "dedup-gate-second") + .await + .expect("the repaired stream accepts the publish without a restart"); + + let _ = provision_context.delete_stream(&stream_name).await; + } + + #[tokio::test] + #[ignore] + async fn a_remapped_subject_is_rejected_by_the_ack_stream_check_and_the_entry_stays_queued() { + // After a subject remap the takeover stream acknowledges, so the ack-stream check rejects it with the mismatch detail, keeps the entry queued, and resets the verdict for re-validation. + let subject = format!("rustfs.remap.{}", Uuid::new_v4().simple()); + let configured_stream = format!("RUSTFS_REMAP_A_{}", Uuid::new_v4().simple()); + let takeover_stream = format!("RUSTFS_REMAP_B_{}", Uuid::new_v4().simple()); + let parked_subject = format!("rustfs.remapparked.{}", Uuid::new_v4().simple()); + + let provision_client = async_nats::connect(broker_url()).await.expect("connect to provision"); + let provision_context = jetstream::new(provision_client); + provision_context + .create_stream(jetstream::stream::Config { + name: configured_stream.clone(), + subjects: vec![subject.clone()], + duplicate_window: Duration::from_secs(300), + ..Default::default() + }) + .await + .expect("provision the configured stream"); + + let queue_dir = tempfile::tempdir().expect("queue dir"); + let mut args = jetstream_args(TargetType::NotifyEvent); + args.address = broker_url(); + args.subject = subject.clone(); + args.jetstream_stream_name = Some(configured_stream.clone()); + let target = nats_target_with_store(args, queue_dir.path().to_str().unwrap()); + + // The pre-remap publish validates the stream and acks from it, recording a passing verdict. + target + .publish_jetstream(b"pre-remap".to_vec(), "dedup-remap-first") + .await + .expect("the configured stream acknowledges before the remap"); + + // The remap: the configured stream stops capturing the subject and the takeover stream starts. + provision_context + .update_stream(jetstream::stream::Config { + name: configured_stream.clone(), + subjects: vec![parked_subject], + duplicate_window: Duration::from_secs(300), + ..Default::default() + }) + .await + .expect("remap the configured stream away from the subject"); + provision_context + .create_stream(jetstream::stream::Config { + name: takeover_stream.clone(), + subjects: vec![subject.clone()], + duplicate_window: Duration::from_secs(300), + ..Default::default() + }) + .await + .expect("provision the takeover stream"); + + let queued = target.build_queued_payload(&sample_event()).expect("payload builds"); + let store = target.store().expect("a store is configured"); + persist_queued_payload_to_store(store, &queued).expect("entry persists"); + let key = store.list().pop().expect("the entry is listed"); + + let log = CapturedLog::default(); + let subscriber = tracing_subscriber::fmt() + .with_max_level(tracing::Level::WARN) + .with_writer(log.clone()) + .finish(); + let _log_guard = tracing::subscriber::set_default(subscriber); + + let err = target + .send_from_store(key.clone()) + .await + .expect_err("the takeover stream's acknowledgment is rejected"); + match err { + TargetError::JetStreamPublish { retryable, detail } => { + assert!(retryable, "the mismatch keeps the entry on the live queue"); + assert_eq!(detail, ACK_STREAM_MISMATCH_DETAIL, "the fixed mismatch label is the detail"); + } + other => panic!("expected the mismatch classification, got {other:?}"), + } + assert_eq!(store.len(), 1, "the queued entry survives the rejected acknowledgment"); + + let warn_line = log.contents(); + assert!( + warn_line.contains("acknowledged by an unexpected stream"), + "the mismatch warn is logged: {warn_line}" + ); + assert!( + warn_line.contains(&takeover_stream), + "the warn names the acknowledging stream: {warn_line}" + ); + + // The reset verdict forces re-validation on the next attempt, which reports the configured + // stream no longer captures the subject. The entry stays queued through that failure too. + let err = target + .send_from_store(key) + .await + .expect_err("re-validation fails while the subject is remapped away"); + match err { + TargetError::JetStreamPublish { detail, .. } => { + assert_eq!(detail, STREAM_VALIDATION_FAILED_DETAIL, "the retry re-validates the stream"); + } + other => panic!("expected the validation classification, got {other:?}"), + } + assert_eq!(store.len(), 1, "the entry stays queued while the remap persists"); + + let _ = provision_context.delete_stream(&configured_stream).await; + let _ = provision_context.delete_stream(&takeover_stream).await; + } + + // A successful move deletes the live entry and leaves exactly one failed entry. + #[tokio::test] + async fn move_entry_writes_failed_then_deletes_live() { + let dir = failed_store_dir("move-order"); + let store = QueueStore::::new_with_compression(&dir, 8, ".test", false); + store.open().unwrap(); + + let key = store.put_raw(&sample_queued("minted-id").encode().unwrap()).unwrap(); + assert_eq!(store.len(), 1); + + let target = move_test_target(); + let error = TargetError::JetStreamPublish { + retryable: false, + detail: "max payload exceeded".to_string(), + }; + move_entry_to_failed_store(&store, &store, &target.id(), &key, &error, 0) + .await + .unwrap(); + + assert_eq!(store.len(), 0, "the live entry is deleted after the failed write"); + assert_eq!(store.failed_len(), 1, "the failed entry is written"); + assert!(matches!(store.get_raw(&key), Err(StoreError::NotFound))); + + let _ = store.delete(); + } + + // A move of an entry the queue no longer holds is a no-op, not an error. + #[tokio::test] + async fn move_entry_is_a_noop_when_the_live_entry_is_already_cleared() { + let dir = failed_store_dir("move-missing"); + let store = QueueStore::::new_with_compression(&dir, 8, ".test", false); + store.open().unwrap(); + + let absent_key = Key { + name: "gone".to_string(), + extension: ".test".to_string(), + item_count: 1, + compress: false, + }; + let target = move_test_target(); + let error = TargetError::JetStreamPublish { + retryable: false, + detail: "max payload exceeded".to_string(), + }; + move_entry_to_failed_store(&store, &store, &target.id(), &absent_key, &error, 0) + .await + .unwrap(); + + assert_eq!(store.failed_len(), 0, "no failed entry is written for an absent live entry"); + + let _ = store.delete(); + } + + // The delivery snapshot reports the failed-store depth read from the store. + #[tokio::test] + async fn delivery_snapshot_reports_failed_store_depth() { + let dir = failed_store_dir("snapshot-depth"); + let store = Arc::new(QueueStore::::new_with_compression(&dir, 8, ".test", false)); + store.open().unwrap(); + + let target = move_test_target_with_store(store.clone()); + assert_eq!( + target.delivery_snapshot().failed_store_length, + 0, + "an empty failed store reports depth zero" + ); + + let key = store.put_raw(&sample_queued("minted-id").encode().unwrap()).unwrap(); + let error = TargetError::JetStreamPublish { + retryable: false, + detail: "max payload exceeded".to_string(), + }; + move_entry_to_failed_store(&*store, &*store, &target.id(), &key, &error, 0) + .await + .unwrap(); + + assert_eq!( + target.delivery_snapshot().failed_store_length, + 1, + "a terminal failure lands one entry and the snapshot reports depth one" + ); + + let _ = store.delete(); + } + + // For a pre-enable entry with an empty stored id, the failed record carries the id derived from the entry key. + #[tokio::test] + async fn move_entry_records_the_resolved_dedup_id_for_a_pre_enable_entry() { + let dir = failed_store_dir("resolved-id"); + let store = QueueStore::::new_with_compression(&dir, 8, ".test", false); + store.open().unwrap(); + + let key = store.put_raw(&sample_queued("").encode().unwrap()).unwrap(); + let expected = resolve_dedup_id("", &key); + assert!(!expected.is_empty(), "the resolved id is derived from the entry key"); + + let target = move_test_target(); + let error = TargetError::JetStreamPublish { + retryable: false, + detail: "max payload exceeded".to_string(), + }; + move_entry_to_failed_store(&store, &store, &target.id(), &key, &error, 0) + .await + .unwrap(); + + let failed_dir = dir.join("failed"); + let failed_file = std::fs::read_dir(&failed_dir) + .unwrap() + .filter_map(|entry| entry.ok()) + .map(|entry| entry.path()) + .find(|path| path.is_file()) + .expect("a failed entry is written"); + let decoded = QueuedPayload::decode(&std::fs::read(&failed_file).unwrap()).unwrap(); + assert_eq!( + decoded.meta.failure.unwrap().nats_msg_id, + expected, + "the failed record carries the resolved id, not the empty stored id" + ); + + let _ = store.delete(); + } + + // The move does not count the failure, so a move followed by one replay hook emit counts it exactly once. + #[tokio::test] + async fn a_single_failed_delivery_counts_once() { + let dir = failed_store_dir("count-once"); + let store = Arc::new(QueueStore::::new_with_compression(&dir, 8, ".test", false)); + store.open().unwrap(); + + let failed = Arc::new(AtomicU64::new(0)); + let target: Arc + Send + Sync> = Arc::new(MoveTestTarget { + id: TargetID::new("target-a".to_string(), "nats".to_string()), + store: Some(store.clone()), + failed: failed.clone(), + }); + + let key = store.put_raw(&sample_queued("minted-id").encode().unwrap()).unwrap(); + let error = TargetError::JetStreamPublish { + retryable: false, + detail: "max payload exceeded".to_string(), + }; + move_entry_to_failed_store(&*store, &*store, &target.id(), &key, &error, 0) + .await + .unwrap(); + assert_eq!(failed.load(Ordering::Relaxed), 0, "the move itself does not count the failure"); + + // The replay worker follows every move with one hook emit that records the failure. + target.record_final_failure(); + assert_eq!(failed.load(Ordering::Relaxed), 1, "one failed delivery counts exactly once"); + + let _ = store.delete(); + } +} diff --git a/crates/targets/src/target/nats.rs b/crates/targets/src/target/nats/mod.rs similarity index 57% rename from crates/targets/src/target/nats.rs rename to crates/targets/src/target/nats/mod.rs index 7b13fe3e0..81cc37c8f 100644 --- a/crates/targets/src/target/nats.rs +++ b/crates/targets/src/target/nats/mod.rs @@ -21,22 +21,36 @@ use crate::{ ReloadableTargetTls, TargetTlsGeneration, TargetTlsInputSet, TlsReloadAdapter, config::ReloadApplyMode, validate_tls_material, }, - store::{Key, Store}, + store::{FailedEventStore, Key, QueueStore, Store}, target::{ ChannelTargetType, EntityTarget, QueuedPayload, QueuedPayloadMeta, TargetDeliveryCounters, TargetDeliverySnapshot, TargetTlsState, TargetType, build_queued_payload_with_records, build_target_tls_fingerprint, is_connectivity_error, - open_target_queue_store, persist_queued_payload_to_store, redacted_secret, + open_target_queue_store_typed, persist_queued_payload_to_store, redacted_secret, }, }; use async_trait::async_trait; -use rustfs_config::{NATS_CREDENTIALS_FILE, NATS_TLS_CA, NATS_TLS_CLIENT_CERT, NATS_TLS_CLIENT_KEY}; +use rustfs_config::{ + NATS_CREDENTIALS_FILE, NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS, NATS_TLS_CA, NATS_TLS_CLIENT_CERT, NATS_TLS_CLIENT_KEY, +}; use std::fmt; use std::path::{Path, PathBuf}; use std::str::FromStr; use std::sync::Arc; use std::sync::atomic::{AtomicBool, Ordering}; +use std::time::Duration; use tokio::sync::Mutex; -use tracing::{info, instrument, warn}; +use tracing::{error, info, instrument, warn}; +use uuid::Uuid; + +mod jetstream; +mod publish_error; +mod validation; + +use jetstream::{CachedJetStreamContext, drain_jetstream_context}; +use publish_error::{classify_nats_flush_error, classify_nats_publish_error}; + +pub(crate) use jetstream::resolve_dedup_id; +pub(crate) use validation::{validate_jetstream_settings, validate_jetstream_stream}; #[derive(Clone)] pub struct NATSArgs { @@ -53,6 +67,10 @@ pub struct NATSArgs { pub tls_required: bool, pub queue_dir: String, pub queue_limit: u64, + // JetStream publish settings. Absent maps to off and the defaults. + pub jetstream_enable: Option, + pub jetstream_stream_name: Option, + pub jetstream_ack_timeout_secs: Option, pub target_type: TargetType, } @@ -72,6 +90,9 @@ impl fmt::Debug for NATSArgs { .field("tls_required", &self.tls_required) .field("queue_dir", &self.queue_dir) .field("queue_limit", &self.queue_limit) + .field("jetstream_enable", &self.jetstream_enable) + .field("jetstream_stream_name", &self.jetstream_stream_name) + .field("jetstream_ack_timeout_secs", &self.jetstream_ack_timeout_secs) .field("target_type", &self.target_type) .finish() } @@ -114,6 +135,13 @@ impl NATSArgs { return Err(TargetError::Configuration("NATS queue directory must be an absolute path".to_string())); } + validate_jetstream_settings( + self.jetstream_enable.unwrap_or(false), + self.jetstream_stream_name.as_deref().unwrap_or_default(), + &self.queue_dir, + self.jetstream_ack_timeout_secs, + )?; + Ok(()) } } @@ -160,12 +188,9 @@ fn validate_nats_auth(args: &NATSArgs) -> Result<(), TargetError> { Ok(()) } -/// Returns true when the target is configured to send credentials (token, -/// username/password, or a credentials file) over a connection that does not -/// require TLS, which would transmit the secrets in cleartext (backlog#983). -/// -/// TLS is considered active when `tls_required` is set or the address uses the -/// `tls://` scheme. +/// Returns true when the target sends credentials over a connection that does not require TLS, which +/// would transmit the secrets in cleartext (backlog#983). TLS is active when tls_required is set or +/// the address uses the tls:// scheme. fn nats_sends_credentials_without_tls(args: &NATSArgs) -> bool { let has_auth = !args.token.is_empty() || !args.credentials_file.is_empty() || !args.username.is_empty(); if !has_auth { @@ -211,12 +236,16 @@ where id: TargetID, args: NATSArgs, client: Arc>>, + /// Cached JetStream context, one per target shared across clones so a single acker and semaphore + /// serve every clone. Read out under the lock before each publish await, never held across it. + /// Carries the stream-validation verdict for the bound connection. + jetstream_context: Arc>>, tls_state: Arc>, - /// Adapter that bridges this target to the TLS reload coordinator. - /// When `Some`, the target uses coordinator-managed material; when `None`, - /// it falls back to inline fingerprint-based change detection. + /// When set, the coordinator drives TLS reload, otherwise inline fingerprint change detection. tls_adapter: Option>, - store: Option + Send + Sync>>, + /// The concrete queue store, held typed so the target projects both the generic Store handle and + /// its own failed-events capability from a single store. Shared across clones through the Arc. + store: Option>>, connected: AtomicBool, delivery_counters: Arc, _phantom: std::marker::PhantomData, @@ -231,9 +260,10 @@ where id: self.id.clone(), args: self.args.clone(), client: Arc::clone(&self.client), + jetstream_context: Arc::clone(&self.jetstream_context), tls_state: Arc::clone(&self.tls_state), tls_adapter: self.tls_adapter.clone(), - store: self.store.as_ref().map(|s| s.boxed_clone()), + store: self.store.clone(), connected: AtomicBool::new(self.connected.load(Ordering::SeqCst)), delivery_counters: Arc::clone(&self.delivery_counters), _phantom: std::marker::PhantomData, @@ -251,19 +281,21 @@ where ); } let target_id = TargetID::new(id, ChannelTargetType::Nats.as_str().to_string()); - let queue_store = open_target_queue_store( + let queue_store = open_target_queue_store_typed( &args.queue_dir, args.queue_limit, args.target_type, ChannelTargetType::Nats.as_str(), &target_id, "Failed to open store for NATS target", - )?; + )? + .map(Arc::new); Ok(Self { id: target_id, args, client: Arc::new(Mutex::new(None)), + jetstream_context: Arc::new(Mutex::new(None)), tls_state: Arc::new(parking_lot::Mutex::new(TargetTlsState::default())), tls_adapter: None, store: queue_store, @@ -299,7 +331,6 @@ where }; if tls_changed { self.invalidate_cached_client_connection().await; - self.tls_state.lock().refresh(next_fingerprint); } { @@ -316,13 +347,53 @@ where .map_err(|e| TargetError::Network(format!("Failed to flush NATS connection: {e}")))?; self.connected.store(true, Ordering::SeqCst); - let mut guard = self.client.lock().await; - let shared = guard.get_or_insert_with(|| client.clone()).clone(); + let mut client_guard = self.client.lock().await; + let shared = client_guard.get_or_insert_with(|| client.clone()).clone(); + + // Swap in a context built from the winning client while the client lock is held, so client + // and context swap as a unit and no clone reads a context bound to the old client. Only the + // JetStream path caches a context. The previous context is drained after the locks release. + let previous_context = if tls_changed && self.jetstream_enabled() { + let ack_timeout = self.ack_timeout(); + let mut context_guard = self.jetstream_context.lock().await; + let mut context = async_nats::jetstream::new(shared.clone()); + context.set_timeout(ack_timeout); + context_guard.replace(CachedJetStreamContext::new(context)) + } else { + None + }; + drop(client_guard); + + // Advance the recorded fingerprint only after the reconnect and flush succeed, so a rotation + // whose first reconnect fails stays detected and the next success still rebuilds the context. + if tls_changed { + self.tls_state.lock().refresh(next_fingerprint); + } + + drain_jetstream_context(previous_context.map(|cached| cached.context)).await; Ok(shared) } + fn jetstream_enabled(&self) -> bool { + self.args.jetstream_enable.unwrap_or(false) + } + + fn ack_timeout(&self) -> Duration { + Duration::from_secs( + self.args + .jetstream_ack_timeout_secs + .unwrap_or(NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS), + ) + } + fn build_queued_payload(&self, event: &EntityTarget) -> Result { - build_queued_payload_with_records(event, vec![event.clone()]) + let mut queued = build_queued_payload_with_records(event, vec![event.clone()])?; + if self.jetstream_enabled() { + // Mint the dedup id once at enqueue so it is identical across every retry and replay and + // distinct across entries. The body-hash fallback only covers entries queued before enable. + queued.meta.dedup_id = Uuid::new_v4().to_string(); + } + Ok(queued) } async fn send_body(&self, body: Vec) -> Result<(), TargetError> { @@ -336,10 +407,8 @@ where return Err(err); } - // `publish` only enqueues the message on the client's outbound channel; - // it does not guarantee the broker received it. Flush to confirm the - // message reached the server before the caller treats delivery as - // successful and deletes the durable copy (backlog#971). + // publish only enqueues the message on the client's outbound channel. Flush to confirm the + // message reached the server before delivery is treated as successful (backlog#971). if let Err(e) = client.flush().await { let err = classify_nats_flush_error(&e); self.invalidate_cached_client_connection().await; @@ -352,30 +421,6 @@ where } } -/// Classifies a NATS publish failure using the typed error kind rather than a -/// substring match. `Send` means the outbound channel is closed (the connection -/// is gone) and is retriable; payload/subject violations are permanent -/// request-level errors (backlog#971, backlog#973). -fn classify_nats_publish_error(err: &async_nats::PublishError) -> TargetError { - use async_nats::PublishErrorKind; - match err.kind() { - PublishErrorKind::Send => TargetError::NotConnected, - PublishErrorKind::MaxPayloadExceeded | PublishErrorKind::InvalidSubject => { - TargetError::Request(format!("Failed to publish NATS message: {err}")) - } - } -} - -/// Classifies a NATS flush failure. Both flush error kinds indicate the message -/// was not confirmed on the wire (a connectivity problem), so the event is kept -/// for replay (backlog#971, backlog#973). -fn classify_nats_flush_error(err: &async_nats::client::FlushError) -> TargetError { - use async_nats::client::FlushErrorKind; - match err.kind() { - FlushErrorKind::SendError | FlushErrorKind::FlushError => TargetError::NotConnected, - } -} - #[async_trait] impl Target for NATSTarget where @@ -386,6 +431,13 @@ where } async fn is_active(&self) -> Result { + // With JetStream enabled the health answer covers the stream as well as the connection, so a + // reachable broker with a failing stream reports the validation error. The verdict is cached + // and reset on a reconnect, TLS rotation, wrong-stream ack, or stream-not-found outcome, so a + // reset forces a live lookup on the next check. + if self.jetstream_enabled() { + self.validated_jetstream_context().await?; + } let client = self.get_or_connect().await?; client .flush() @@ -418,11 +470,31 @@ where } } - async fn send_raw_from_store(&self, _key: Key, body: Vec, _meta: QueuedPayloadMeta) -> Result<(), TargetError> { - self.send_body(body).await + async fn send_raw_from_store(&self, key: Key, body: Vec, meta: QueuedPayloadMeta) -> Result<(), TargetError> { + if self.jetstream_enabled() { + let dedup_id = resolve_dedup_id(&meta.dedup_id, &key); + self.publish_jetstream(body, &dedup_id).await + } else { + self.send_body(body).await + } } async fn close(&self) -> Result<(), TargetError> { + // Drain the cached context's in-flight acks before dropping it so the async-nats acker exits. + // Drain the context before the client, since the acker rides the client connection. + let context = { + let mut guard = self.jetstream_context.lock().await; + guard.take() + }; + // Bound the drain by the ack timeout so an unresponsive broker cannot stall close. On elapse + // the client drain below still runs. + if tokio::time::timeout(self.ack_timeout(), drain_jetstream_context(context.map(|cached| cached.context))) + .await + .is_err() + { + warn!(target_id = %self.id, "Timed out draining JetStream acks on close, proceeding to close the client"); + } + let client = { let mut guard = self.client.lock().await; guard.take() @@ -435,12 +507,49 @@ where .await .map_err(|e| TargetError::Network(format!("Failed to drain NATS client: {e}")))?; } + // The durable queue store stays on disk, so a queued entry survives close and replays. Close + // releases the cached handles, never the entries. info!(target_id = %self.id, "NATS target closed"); Ok(()) } fn store(&self) -> Option<&(dyn Store + Send + Sync)> { - self.store.as_deref() + self.store + .as_deref() + .map(|store| store as &(dyn Store + Send + Sync)) + } + + fn failed_store(&self) -> Option<&dyn FailedEventStore> { + self.store.as_deref().map(|store| store as &dyn FailedEventStore) + } + + async fn handle_terminal_failure( + &self, + store: &(dyn Store + Send), + key: &Key, + error: &TargetError, + retry_count: u32, + ) -> bool { + let Some(failed_store) = self.failed_store() else { + error!( + target_id = %self.id, + replay_key = %key, + "NATS target has no failed-events store for the terminal move" + ); + return false; + }; + match jetstream::move_entry_to_failed_store(store, failed_store, &self.id, key, error, retry_count).await { + Ok(()) => true, + Err(move_err) => { + error!( + target_id = %self.id, + error = %move_err, + replay_key = %key, + "Failed to move event to the failed-events store" + ); + false + } + } } fn clone_dyn(&self) -> Box + Send + Sync> { @@ -452,6 +561,13 @@ where return Ok(()); } let _ = self.get_or_connect().await?; + if self.jetstream_enabled() { + let cached = self.jetstream_context().await?; + validate_jetstream_stream(&cached.context, &self.args, &self.id.to_string(), Some(&cached.validation_logged)).await?; + // Record the verdict on the context so the first publish does not repeat the stream + // lookup that just passed. + cached.stream_validated.store(true, Ordering::Release); + } Ok(()) } @@ -460,8 +576,10 @@ where } fn delivery_snapshot(&self) -> TargetDeliverySnapshot { - self.delivery_counters - .snapshot(self.store.as_deref().map_or(0, |store| store.len() as u64)) + self.delivery_counters.snapshot( + self.store.as_deref().map_or(0, |store| store.len() as u64), + self.failed_store().map_or(0, |failed_store| failed_store.failed_len() as u64), + ) } fn record_final_failure(&self) { @@ -510,11 +628,18 @@ where } #[cfg(test)] -mod tests { +pub(crate) mod test_support { use super::*; - use crate::target::REDACTED_SECRET; + use async_nats::jetstream; + use rustfs_s3_types::EventName; - fn base_args() -> NATSArgs { + // Absolute on Linux, macOS, and Windows. temp_dir needs no filesystem to exist for a + // validation-only test, and Path::is_absolute stays true across platforms. + pub(crate) fn nats_queue_dir() -> String { + std::env::temp_dir().join("rustfs-nats-queue").to_string_lossy().into_owned() + } + + pub(crate) fn base_args() -> NATSArgs { NATSArgs { enable: true, address: "nats://127.0.0.1:4222".to_string(), @@ -529,10 +654,123 @@ mod tests { tls_required: false, queue_dir: String::new(), queue_limit: 0, + jetstream_enable: None, + jetstream_stream_name: None, + jetstream_ack_timeout_secs: None, target_type: TargetType::NotifyEvent, } } + pub(crate) fn jetstream_args(target_type: TargetType) -> NATSArgs { + NATSArgs { + jetstream_enable: Some(true), + jetstream_stream_name: Some("RUSTFS_EVENTS".to_string()), + jetstream_ack_timeout_secs: Some(30), + target_type, + ..base_args() + } + } + + pub(crate) fn nats_target(args: NATSArgs) -> NATSTarget { + NATSTarget:: { + id: TargetID::new("test-target".to_string(), ChannelTargetType::Nats.as_str().to_string()), + args, + client: Arc::new(Mutex::new(None)), + jetstream_context: Arc::new(Mutex::new(None)), + tls_state: Arc::new(parking_lot::Mutex::new(TargetTlsState::default())), + tls_adapter: None, + store: None, + connected: AtomicBool::new(false), + delivery_counters: Arc::new(TargetDeliveryCounters::default()), + _phantom: std::marker::PhantomData, + } + } + + pub(crate) fn sample_event() -> EntityTarget { + EntityTarget { + object_name: "folder/object.txt".to_string(), + bucket_name: "bucket-a".to_string(), + event_name: EventName::ObjectCreatedPut, + data: "payload-data".to_string(), + } + } + + pub(crate) fn sample_stored_key(name: &str) -> Key { + Key { + name: name.to_string(), + extension: ".event".to_string(), + item_count: 1, + compress: false, + } + } + + pub(crate) fn nats_target_with_store(args: NATSArgs, queue_dir: &str) -> NATSTarget { + let mut configured = args; + configured.queue_dir = queue_dir.to_string(); + NATSTarget::::new("store-target".to_string(), configured).expect("target with store builds") + } + + /// A stream configuration that passes every assertion: it captures the configured subject, returns + /// acks, accepts writes, and deduplicates well beyond the retry lifetime. + pub(crate) fn writable_stream_config(subject: &str) -> jetstream::stream::Config { + jetstream::stream::Config { + name: "RUSTFS_EVENTS".to_string(), + subjects: vec![subject.to_string()], + no_ack: false, + sealed: false, + duplicate_window: Duration::from_secs(600), + ..Default::default() + } + } + + // Live-broker behaviour tests. They require a NATS server with JetStream enabled and are ignored by + // default. To run locally: + // + // docker run -d --name rustfs-nats-test -p 4222:4222 nats:2 -js + // cargo test -p rustfs-targets --lib -- --ignored nats::tests::tls_change + // + // Override the server URL with RUSTFS_TEST_NATS_URL. + pub(crate) fn broker_url() -> String { + std::env::var("RUSTFS_TEST_NATS_URL").unwrap_or_else(|_| "nats://127.0.0.1:4222".to_string()) + } + + /// Log sink for asserting a specific line is written. The subscriber writes into a shared + /// buffer the assertion reads back. + #[derive(Clone, Default)] + pub(crate) struct CapturedLog(Arc>>); + + impl CapturedLog { + pub(crate) fn contents(&self) -> String { + String::from_utf8_lossy(&self.0.lock()).into_owned() + } + } + + impl std::io::Write for CapturedLog { + fn write(&mut self, buf: &[u8]) -> std::io::Result { + self.0.lock().extend_from_slice(buf); + Ok(buf.len()) + } + + fn flush(&mut self) -> std::io::Result<()> { + Ok(()) + } + } + + impl<'a> tracing_subscriber::fmt::MakeWriter<'a> for CapturedLog { + type Writer = CapturedLog; + + fn make_writer(&'a self) -> Self::Writer { + self.clone() + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::target::REDACTED_SECRET; + use crate::target::nats::test_support::*; + #[test] fn debug_redacts_nats_secret_fields() { let args = NATSArgs { @@ -575,8 +813,7 @@ mod tests { #[test] fn nats_credentials_without_tls_is_detected() { - // Token auth over a plaintext nats:// address without tls_required leaks - // the credential in cleartext (backlog#983). + // Token auth over a plaintext nats:// address without tls_required leaks the credential (backlog#983). let insecure = NATSArgs { token: "secret-token".to_string(), tls_required: false, @@ -603,25 +840,17 @@ mod tests { assert!(!nats_sends_credentials_without_tls(&no_auth)); } - #[test] - fn nats_publish_error_classification() { - use async_nats::PublishErrorKind; - // `Send` means the outbound channel is gone: retriable connectivity error. - let send_err = async_nats::PublishError::new(PublishErrorKind::Send); - assert!(matches!(classify_nats_publish_error(&send_err), TargetError::NotConnected)); + #[tokio::test] + async fn flag_off_stored_meta_is_byte_identical_to_pre_feature() { + // With the flag off the payload carries no dedup id, so its encoded meta matches a pre-feature entry. + let disabled = nats_target(base_args()); + let off_payload = disabled.build_queued_payload(&sample_event()).expect("payload builds"); + assert!(off_payload.meta.dedup_id.is_empty(), "no dedup id is minted with the flag off"); - // Payload/subject violations are permanent request-level errors. - let payload_err = async_nats::PublishError::new(PublishErrorKind::MaxPayloadExceeded); - assert!(matches!(classify_nats_publish_error(&payload_err), TargetError::Request(_))); - let subject_err = async_nats::PublishError::new(PublishErrorKind::InvalidSubject); - assert!(matches!(classify_nats_publish_error(&subject_err), TargetError::Request(_))); - } + let meta_json = serde_json::to_string(&off_payload.meta).expect("meta serializes"); + assert!(!meta_json.contains("dedup_id"), "the absent dedup id is skipped on serialization"); - #[test] - fn nats_flush_error_classification() { - use async_nats::client::{FlushError, FlushErrorKind}; - for kind in [FlushErrorKind::SendError, FlushErrorKind::FlushError] { - assert!(matches!(classify_nats_flush_error(&FlushError::new(kind)), TargetError::NotConnected)); - } + // No JetStream context is built on the flag-off path. + assert!(disabled.jetstream_context.lock().await.is_none(), "no context is built with the flag off"); } } diff --git a/crates/targets/src/target/nats/publish_error.rs b/crates/targets/src/target/nats/publish_error.rs new file mode 100644 index 000000000..cb2ec7c9e --- /dev/null +++ b/crates/targets/src/target/nats/publish_error.rs @@ -0,0 +1,346 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use super::jetstream::STREAM_NOT_FOUND_DETAIL; +use crate::error::TargetError; +use async_nats::jetstream::context::PublishErrorKind; + +/// Maps an async-nats publish error to a typed TargetError with the retryable-versus-terminal +/// classification, so the replay loop retries recoverable errors instead of dropping them. The +/// detail comes from the fixed vocabulary in publish_error_detail, never from raw error text. +pub(crate) fn classify_publish_error(err: &async_nats::jetstream::context::PublishError) -> TargetError { + let retryable = match err.kind() { + PublishErrorKind::TimedOut + | PublishErrorKind::BrokenPipe + | PublishErrorKind::MaxAckPending + | PublishErrorKind::StreamNotFound => true, + PublishErrorKind::MaxPayloadExceeded | PublishErrorKind::WrongLastMessageId | PublishErrorKind::WrongLastSequence => { + false + } + PublishErrorKind::Other => !other_error_is_terminal(err), + }; + TargetError::JetStreamPublish { + retryable, + detail: publish_error_detail(err), + } +} + +/// Fixed diagnostic detail for a publish failure. Each error kind maps to a constant label, and a +/// server rejection carries the numeric JetStream error code and status. Never the library rendering +/// or any server-returned text, so the detail is safe to persist in a failed-store entry. +fn publish_error_detail(err: &async_nats::jetstream::context::PublishError) -> String { + use async_nats::jetstream::Error as JetStreamApiError; + match err.kind() { + PublishErrorKind::StreamNotFound => STREAM_NOT_FOUND_DETAIL.to_string(), + PublishErrorKind::TimedOut => "publish timed out".to_string(), + PublishErrorKind::BrokenPipe => "broken pipe".to_string(), + PublishErrorKind::MaxAckPending => "max ack pending reached".to_string(), + PublishErrorKind::MaxPayloadExceeded => "max payload exceeded".to_string(), + PublishErrorKind::WrongLastMessageId => "wrong last message id".to_string(), + PublishErrorKind::WrongLastSequence => "wrong last sequence".to_string(), + PublishErrorKind::Other => { + match std::error::Error::source(err).and_then(|source| source.downcast_ref::()) { + Some(api_error) => format!("server error code {} status {}", api_error.error_code().0, api_error.code()), + None => "publish failed".to_string(), + } + } + } +} + +/// HTTP-style 5xx status range. A rejection reporting a 5xx status is a server-side condition and +/// stays retryable regardless of its error code. +const SERVER_ERROR_STATUS_RANGE: std::ops::Range = 500..600; + +/// Classifies an Other publish error against a terminal allowlist. Terminal only for an explicit +/// permanent error code, STREAM_SEALED being the sole current member. Everything else is retryable, +/// so only a recognized permanent rejection enters the failed store. +fn other_error_is_terminal(err: &async_nats::jetstream::context::PublishError) -> bool { + use async_nats::jetstream::{Error as JetStreamApiError, ErrorCode}; + + let Some(api_error) = std::error::Error::source(err).and_then(|source| source.downcast_ref::()) else { + return false; + }; + + // A 5xx status is a server-side condition, so it stays retryable even when the error code is a + // permanent one. + if SERVER_ERROR_STATUS_RANGE.contains(&api_error.code()) { + return false; + } + + matches!(api_error.error_code(), ErrorCode::STREAM_SEALED) +} + +/// Classifies a NATS publish failure by its typed error kind. Send means the outbound channel is +/// closed and is retriable. Payload and subject violations are permanent request-level errors +/// (backlog#971, backlog#973). +pub(crate) fn classify_nats_publish_error(err: &async_nats::PublishError) -> TargetError { + use async_nats::PublishErrorKind; + match err.kind() { + PublishErrorKind::Send => TargetError::NotConnected, + PublishErrorKind::MaxPayloadExceeded | PublishErrorKind::InvalidSubject => { + TargetError::Request(format!("Failed to publish NATS message: {err}")) + } + } +} + +/// Classifies a NATS flush failure. Both kinds mean the message was not confirmed on the wire, so +/// the event is kept for replay (backlog#971, backlog#973). +pub(crate) fn classify_nats_flush_error(err: &async_nats::client::FlushError) -> TargetError { + use async_nats::client::FlushErrorKind; + match err.kind() { + FlushErrorKind::SendError | FlushErrorKind::FlushError => TargetError::NotConnected, + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::target::nats::jetstream::{ACK_STREAM_MISMATCH_DETAIL, ATTEMPT_DEADLINE_DETAIL}; + use crate::target::nats::validation::gate_publish_on_stream_validation; + use async_nats::jetstream::context::{PublishError, PublishErrorKind}; + use std::sync::atomic::AtomicBool; + + #[test] + fn nats_publish_error_classification() { + use async_nats::PublishErrorKind; + // Send means the outbound channel is gone: retriable connectivity error. + let send_err = async_nats::PublishError::new(PublishErrorKind::Send); + assert!(matches!(classify_nats_publish_error(&send_err), TargetError::NotConnected)); + + // Payload and subject violations are permanent request-level errors. + let payload_err = async_nats::PublishError::new(PublishErrorKind::MaxPayloadExceeded); + assert!(matches!(classify_nats_publish_error(&payload_err), TargetError::Request(_))); + let subject_err = async_nats::PublishError::new(PublishErrorKind::InvalidSubject); + assert!(matches!(classify_nats_publish_error(&subject_err), TargetError::Request(_))); + } + + #[test] + fn nats_flush_error_classification() { + use async_nats::client::{FlushError, FlushErrorKind}; + for kind in [FlushErrorKind::SendError, FlushErrorKind::FlushError] { + assert!(matches!(classify_nats_flush_error(&FlushError::new(kind)), TargetError::NotConnected)); + } + } + + #[test] + fn classify_publish_error_marks_recoverable_kinds_retryable() { + for kind in [ + PublishErrorKind::TimedOut, + PublishErrorKind::BrokenPipe, + PublishErrorKind::MaxAckPending, + PublishErrorKind::StreamNotFound, + ] { + let classified = classify_publish_error(&PublishError::new(kind)); + match classified { + TargetError::JetStreamPublish { retryable, .. } => assert!(retryable, "{kind} is retryable"), + other => panic!("expected JetStreamPublish, got {other:?}"), + } + } + } + + #[test] + fn classify_publish_error_marks_terminal_kinds_terminal() { + for kind in [ + PublishErrorKind::MaxPayloadExceeded, + PublishErrorKind::WrongLastMessageId, + PublishErrorKind::WrongLastSequence, + ] { + let classified = classify_publish_error(&PublishError::new(kind)); + match classified { + TargetError::JetStreamPublish { retryable, .. } => assert!(!retryable, "{kind} is terminal"), + other => panic!("expected JetStreamPublish, got {other:?}"), + } + } + } + + #[test] + fn classify_publish_error_other_without_source_is_retryable() { + // An undecodable Other carries no code to recognize as permanent, so it stays on the retry path. + let classified = classify_publish_error(&PublishError::new(PublishErrorKind::Other)); + match classified { + TargetError::JetStreamPublish { retryable, .. } => assert!(retryable, "an undecodable Other is retryable"), + other => panic!("expected JetStreamPublish, got {other:?}"), + } + } + + #[test] + fn classify_publish_error_unknown_code_is_retryable() { + // A decoded rejection carrying a code outside the terminal allowlist is retryable. + let api_error: async_nats::jetstream::Error = + serde_json::from_value(serde_json::json!({"code": 400, "err_code": 10999, "description": "an unrecognized code"})) + .expect("api error deserializes"); + let publish_error = PublishError::with_source(PublishErrorKind::Other, api_error); + match classify_publish_error(&publish_error) { + TargetError::JetStreamPublish { retryable, .. } => assert!(retryable, "an unknown error code is retryable"), + other => panic!("expected JetStreamPublish, got {other:?}"), + } + } + + #[test] + fn classify_publish_error_other_with_transient_code_is_retryable() { + // Server conditions that each clear on their own. None is in the terminal allowlist, so each + // is retryable. The status is 400 in every case, so retryability comes from the allowlist + // policy rather than the 5xx status range. + for (err_code, condition) in [ + (10002, "account resources exceeded"), + (10008, "cluster not available"), + (10009, "cluster not leader"), + (10028, "memory resources exceeded"), + (10039, "jetstream not enabled for account"), + (10040, "cluster peer not a member"), + (10041, "raft general error"), + (10076, "jetstream not enabled"), + (10118, "stream offline"), + (10194, "stream offline with a reason"), + (10202, "server member change in flight"), + ] { + let api_error: async_nats::jetstream::Error = + serde_json::from_value(serde_json::json!({"code": 400, "err_code": err_code, "description": condition})) + .expect("api error deserializes"); + let publish_error = PublishError::with_source(PublishErrorKind::Other, api_error); + let classified = classify_publish_error(&publish_error); + match classified { + TargetError::JetStreamPublish { retryable, .. } => { + assert!(retryable, "transient code {err_code} ({condition}) is retryable") + } + other => panic!("expected JetStreamPublish for code {err_code}, got {other:?}"), + } + } + } + + #[test] + fn classify_publish_error_uncoded_server_error_status_is_retryable() { + // A rejection with no err_code and a 5xx status is a server-side condition, kept on the retry path. + for status in [500, 503] { + let api_error: async_nats::jetstream::Error = + serde_json::from_value(serde_json::json!({"code": status, "description": "insufficient resources"})) + .expect("api error deserializes"); + let publish_error = PublishError::with_source(PublishErrorKind::Other, api_error); + let classified = classify_publish_error(&publish_error); + match classified { + TargetError::JetStreamPublish { retryable, .. } => { + assert!(retryable, "an uncoded {status} status is retryable") + } + other => panic!("expected JetStreamPublish for status {status}, got {other:?}"), + } + } + } + + #[test] + fn classify_publish_error_other_with_terminal_code_is_terminal() { + // STREAM_SEALED (10109) is the sole terminal-allowlist member and its 400 status keeps it terminal. + let api_error: async_nats::jetstream::Error = + serde_json::from_value(serde_json::json!({"code": 400, "err_code": 10109, "description": "stream sealed"})) + .expect("api error deserializes"); + let publish_error = PublishError::with_source(PublishErrorKind::Other, api_error); + match classify_publish_error(&publish_error) { + TargetError::JetStreamPublish { retryable, .. } => assert!(!retryable, "a sealed stream is terminal"), + other => panic!("expected JetStreamPublish, got {other:?}"), + } + } + + #[test] + fn classify_publish_error_uncoded_4xx_is_retryable() { + // An uncoded 4xx rejection carries no permanent code, so the allowlist policy leaves it on the retry path. + let api_error: async_nats::jetstream::Error = + serde_json::from_value(serde_json::json!({"code": 400, "description": "bad request without an error code"})) + .expect("api error deserializes"); + let publish_error = PublishError::with_source(PublishErrorKind::Other, api_error); + match classify_publish_error(&publish_error) { + TargetError::JetStreamPublish { retryable, .. } => assert!(retryable, "an uncoded 4xx rejection is retryable"), + other => panic!("expected JetStreamPublish, got {other:?}"), + } + } + + #[test] + fn classify_publish_error_terminal_code_with_server_error_status_is_retryable() { + // Deliberate policy: a 5xx status classifies retryable even when the error code is terminal. + let api_error: async_nats::jetstream::Error = + serde_json::from_value(serde_json::json!({"code": 503, "err_code": 10109, "description": "stream sealed"})) + .expect("api error deserializes"); + let publish_error = PublishError::with_source(PublishErrorKind::Other, api_error); + let classified = classify_publish_error(&publish_error); + match classified { + TargetError::JetStreamPublish { retryable, .. } => { + assert!(retryable, "a 5xx status keeps the STREAM_SEALED code on the retry path") + } + other => panic!("expected JetStreamPublish, got {other:?}"), + } + } + + #[test] + fn classify_publish_error_detail_omits_server_body() { + // A server rejection contributes its numeric code and status, never the server-returned description. + let api_error: async_nats::jetstream::Error = serde_json::from_value( + serde_json::json!({"code": 503, "err_code": 10008, "description": "secret-token-abc123 in description"}), + ) + .expect("api error deserializes"); + let classified = classify_publish_error(&PublishError::with_source(PublishErrorKind::Other, api_error)); + match classified { + TargetError::JetStreamPublish { detail, .. } => { + assert_eq!(detail, "server error code 10008 status 503"); + } + other => panic!("expected JetStreamPublish, got {other:?}"), + } + } + + #[test] + fn classified_publish_details_use_the_fixed_vocabulary() { + // Every classification path draws its detail from fixed labels and numeric fields. The pattern + // (lowercase letters, digits, spaces, underscores, colons) rejects raw error text. + let is_fixed = |detail: &str| { + !detail.is_empty() + && detail + .chars() + .all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || matches!(c, ' ' | '_' | ':')) + }; + + let mut details = vec![ATTEMPT_DEADLINE_DETAIL.to_string(), ACK_STREAM_MISMATCH_DETAIL.to_string()]; + for kind in [ + PublishErrorKind::StreamNotFound, + PublishErrorKind::TimedOut, + PublishErrorKind::BrokenPipe, + PublishErrorKind::MaxAckPending, + PublishErrorKind::MaxPayloadExceeded, + PublishErrorKind::WrongLastMessageId, + PublishErrorKind::WrongLastSequence, + PublishErrorKind::Other, + ] { + match classify_publish_error(&PublishError::new(kind)) { + TargetError::JetStreamPublish { detail, .. } => details.push(detail), + other => panic!("expected JetStreamPublish, got {other:?}"), + } + } + + let api_error: async_nats::jetstream::Error = + serde_json::from_value(serde_json::json!({"code": 503, "err_code": 10077, "description": "raw server text"})) + .expect("api error deserializes"); + match classify_publish_error(&PublishError::with_source(PublishErrorKind::Other, api_error)) { + TargetError::JetStreamPublish { detail, .. } => details.push(detail), + other => panic!("expected JetStreamPublish, got {other:?}"), + } + + // The validation gate detail must stay fixed even when the verdict error carries text outside the vocabulary. + let stream_validated = AtomicBool::new(false); + let verdict = Err(TargetError::Configuration("Raw Display TEXT with (punctuation)".to_string())); + match gate_publish_on_stream_validation(verdict, &stream_validated) { + Err(TargetError::JetStreamPublish { detail, .. }) => details.push(detail), + other => panic!("expected a JetStreamPublish gate error, got {other:?}"), + } + + for detail in details { + assert!(is_fixed(&detail), "detail falls outside the fixed vocabulary: {detail:?}"); + } + } +} diff --git a/crates/targets/src/target/nats/validation.rs b/crates/targets/src/target/nats/validation.rs new file mode 100644 index 000000000..1879798a6 --- /dev/null +++ b/crates/targets/src/target/nats/validation.rs @@ -0,0 +1,405 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use super::NATSArgs; +use super::jetstream::retry_lifetime; +use crate::error::TargetError; +use async_nats::jetstream; +use rustfs_config::{ + NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS, NATS_JETSTREAM_ACK_TIMEOUT_MAX_SECS, NATS_JETSTREAM_ACK_TIMEOUT_MIN_SECS, + NATS_JETSTREAM_ACK_TIMEOUT_SECS, NATS_JETSTREAM_ENABLE, NATS_JETSTREAM_STREAM_NAME, NATS_QUEUE_DIR, +}; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::time::Duration; +use tracing::{debug, error, info}; + +/// Validates the JetStream publish settings shared by the config-time and connect-time validators, +/// so both reject the same combinations. +pub(crate) fn validate_jetstream_settings( + enable: bool, + stream_name: &str, + queue_dir: &str, + ack_timeout_secs: Option, +) -> Result<(), TargetError> { + if let Some(secs) = ack_timeout_secs + && !(NATS_JETSTREAM_ACK_TIMEOUT_MIN_SECS..=NATS_JETSTREAM_ACK_TIMEOUT_MAX_SECS).contains(&secs) + { + return Err(TargetError::Configuration(format!( + "{NATS_JETSTREAM_ACK_TIMEOUT_SECS} must be between {NATS_JETSTREAM_ACK_TIMEOUT_MIN_SECS} and {NATS_JETSTREAM_ACK_TIMEOUT_MAX_SECS} seconds" + ))); + } + + if !enable { + return Ok(()); + } + + if stream_name.trim().is_empty() { + return Err(TargetError::Configuration(format!( + "{NATS_JETSTREAM_STREAM_NAME} is required when {NATS_JETSTREAM_ENABLE} is on" + ))); + } + + if queue_dir.trim().is_empty() { + return Err(TargetError::Configuration(format!( + "{NATS_QUEUE_DIR} is required when {NATS_JETSTREAM_ENABLE} is on" + ))); + } + + Ok(()) +} + +/// Detail carried by the retryable publish error raised when the stream-validation gate closes. The +/// full validation cause is logged at error level inside the validator. +pub(crate) const STREAM_VALIDATION_FAILED_DETAIL: &str = "stream validation failed"; + +/// Applies a stream-validation verdict on the publish path. A pass records the verdict so later +/// publishes skip re-validation. A failure returns a retryable JetStream publish error carrying the +/// fixed label and leaves the verdict unvalidated. +pub(crate) fn gate_publish_on_stream_validation( + verdict: Result<(), TargetError>, + stream_validated: &AtomicBool, +) -> Result<(), TargetError> { + match verdict { + Ok(()) => { + stream_validated.store(true, Ordering::Release); + Ok(()) + } + Err(_) => Err(TargetError::JetStreamPublish { + retryable: true, + detail: STREAM_VALIDATION_FAILED_DETAIL.to_string(), + }), + } +} + +/// Reports whether a publish subject is captured by a stream subject filter, honouring the NATS +/// wildcard tokens: a single-token wildcard matches one subject token, a trailing multi-token +/// wildcard matches one or more remaining tokens, any other token matches literally. +fn subject_filter_captures(filter: &str, subject: &str) -> bool { + let mut filter_tokens = filter.split('.'); + let mut subject_tokens = subject.split('.'); + + loop { + match (filter_tokens.next(), subject_tokens.next()) { + (Some(">"), Some(_)) => return true, + (Some("*"), Some(_)) => continue, + (Some(filter_token), Some(subject_token)) if filter_token == subject_token => continue, + (None, None) => return true, + _ => return false, + } + } +} + +/// Validates the configured JetStream stream from a single get_stream lookup when enabled. Asserts +/// the stream exists, captures the subject, returns acks, accepts writes, and keeps a duplicate +/// window that covers one retry cycle. The client-facing error is generic and every concrete cause +/// is logged server-side. The stream is never created, a missing stream is an error. +/// +/// When validation_logged is set, the first passing validation logs at info and later revalidations +/// of the same context log at debug, so a persistent wrong-stream loop does not flood the log. +pub(crate) async fn validate_jetstream_stream( + context: &jetstream::Context, + args: &NATSArgs, + target_label: &str, + validation_logged: Option<&AtomicBool>, +) -> Result<(), TargetError> { + if !args.jetstream_enable.unwrap_or(false) { + return Ok(()); + } + + let stream_name = args.jetstream_stream_name.as_deref().unwrap_or_default(); + if stream_name.is_empty() { + error!(target = %target_label, "JetStream enabled without a configured stream name"); + return Err(TargetError::Configuration("JetStream stream is not configured".to_string())); + } + + let stream = context.get_stream(stream_name).await.map_err(|err| { + error!( + target = %target_label, + stream = %stream_name, + error = %err, + "JetStream stream lookup failed, the stream must be pre-provisioned" + ); + TargetError::Configuration("configured JetStream stream is unavailable".to_string()) + })?; + + assert_stream_writable(&stream.cached_info().config, args, target_label)?; + // Info on a context's first passing validation, debug on later revalidations of the same context. + let first_pass = validation_logged + .map(|logged| !logged.swap(true, Ordering::AcqRel)) + .unwrap_or(true); + if first_pass { + info!( + target = %target_label, + stream = %stream_name, + "JetStream stream validation succeeded" + ); + } else { + debug!( + target = %target_label, + stream = %stream_name, + "JetStream stream revalidation succeeded" + ); + } + Ok(()) +} + +/// Asserts the retrieved stream configuration captures the subject, returns acks, accepts writes, and +/// keeps a duplicate window that covers one retry cycle. The client-facing error is generic, the +/// concrete cause logged server-side. +fn assert_stream_writable(config: &jetstream::stream::Config, args: &NATSArgs, target_label: &str) -> Result<(), TargetError> { + let stream_name = args.jetstream_stream_name.as_deref().unwrap_or_default(); + let subject = &args.subject; + let subject_bound = config.subjects.iter().any(|filter| subject_filter_captures(filter, subject)); + if !subject_bound { + error!( + target = %target_label, + stream = %stream_name, + subject = %subject, + "configured subject is not captured by the JetStream stream subjects" + ); + return Err(TargetError::Configuration( + "configured JetStream stream does not capture the subject".to_string(), + )); + } + + if config.no_ack { + error!( + target = %target_label, + stream = %stream_name, + "JetStream stream has no_ack set, publishes would never acknowledge" + ); + return Err(TargetError::Configuration( + "configured JetStream stream does not acknowledge writes".to_string(), + )); + } + + if config.sealed { + error!( + target = %target_label, + stream = %stream_name, + "JetStream stream is sealed, writes are rejected" + ); + return Err(TargetError::Configuration("configured JetStream stream is sealed".to_string())); + } + + let ack_timeout = Duration::from_secs( + args.jetstream_ack_timeout_secs + .unwrap_or(NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS), + ); + let required_window = retry_lifetime(ack_timeout); + if config.duplicate_window < required_window { + error!( + target = %target_label, + stream = %stream_name, + configured_window_secs = config.duplicate_window.as_secs(), + required_window_secs = required_window.as_secs(), + "JetStream stream duplicate_window is below the retry lifetime, a late retry would be delivered twice" + ); + return Err(TargetError::Configuration( + "configured JetStream stream duplicate window is too small for the retry lifetime".to_string(), + )); + } + + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::target::TargetType; + use crate::target::nats::test_support::*; + + #[test] + fn jetstream_settings_disabled_passes() { + validate_jetstream_settings(false, "", "", None).expect("disabled jetstream validates"); + validate_jetstream_settings(false, "", "", Some(30)).expect("disabled jetstream ignores stream and queue"); + } + + #[test] + fn jetstream_settings_ack_timeout_range() { + for secs in [ + NATS_JETSTREAM_ACK_TIMEOUT_MIN_SECS, + NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS, + NATS_JETSTREAM_ACK_TIMEOUT_MAX_SECS, + ] { + validate_jetstream_settings(false, "", "", Some(secs)).expect("in-range ack timeout accepted"); + } + for secs in [ + NATS_JETSTREAM_ACK_TIMEOUT_MIN_SECS - 1, + NATS_JETSTREAM_ACK_TIMEOUT_MAX_SECS + 1, + ] { + let err = validate_jetstream_settings(false, "", "", Some(secs)).expect_err("out-of-range ack timeout rejected"); + assert!(err.to_string().contains("between")); + } + } + + #[test] + fn jetstream_settings_requires_stream_name() { + let err = validate_jetstream_settings(true, " ", &nats_queue_dir(), Some(30)) + .expect_err("empty stream name rejected when enabled"); + assert!(err.to_string().contains(NATS_JETSTREAM_STREAM_NAME)); + } + + #[test] + fn jetstream_settings_requires_queue_dir() { + let err = + validate_jetstream_settings(true, "RUSTFS_EVENTS", "", Some(30)).expect_err("empty queue_dir rejected when enabled"); + assert!(err.to_string().contains(NATS_QUEUE_DIR)); + } + + #[test] + fn jetstream_settings_enabled_with_stream_and_queue_passes() { + validate_jetstream_settings(true, "RUSTFS_EVENTS", &nats_queue_dir(), Some(30)) + .expect("a stream name and queue dir accept enabling jetstream"); + } + + #[test] + fn natsargs_validate_rejects_enabled_jetstream_without_stream() { + let args = NATSArgs { + queue_dir: nats_queue_dir(), + jetstream_enable: Some(true), + jetstream_stream_name: None, + ..base_args() + }; + let err = args.validate().expect_err("enabled jetstream without a stream is rejected"); + assert!(err.to_string().contains(NATS_JETSTREAM_STREAM_NAME)); + } + + #[test] + fn subject_filter_matches_literal_and_wildcards() { + assert!(subject_filter_captures("rustfs.events", "rustfs.events")); + assert!(subject_filter_captures("rustfs.*", "rustfs.events")); + assert!(subject_filter_captures("rustfs.>", "rustfs.events.created")); + assert!(subject_filter_captures("rustfs.*.created", "rustfs.events.created")); + assert!(!subject_filter_captures("rustfs.events", "rustfs.audit")); + assert!(!subject_filter_captures("rustfs.*", "rustfs.events.created")); + assert!(!subject_filter_captures("rustfs.events", "rustfs.events.created")); + assert!(!subject_filter_captures("rustfs.events.created", "rustfs.events")); + } + + #[test] + fn assert_stream_writable_enforces_the_worst_case_window_at_the_default_ack_timeout() { + // The default 30s ack timeout requires a 274s window. One second below is rejected, exactly at it is accepted. + let args = jetstream_args(TargetType::NotifyEvent); + assert_eq!(retry_lifetime(Duration::from_secs(30)), Duration::from_secs(274)); + + let mut below = writable_stream_config(&args.subject); + below.duplicate_window = Duration::from_secs(273); + let err = assert_stream_writable(&below, &args, "test").expect_err("a window below the worst case is rejected"); + assert!(matches!(err, TargetError::Configuration(_))); + + let mut at = writable_stream_config(&args.subject); + at.duplicate_window = Duration::from_secs(274); + assert_stream_writable(&at, &args, "test").expect("a window at the worst case is accepted"); + } + + #[test] + fn assert_stream_writable_accepts_a_valid_stream() { + for target_type in [TargetType::NotifyEvent, TargetType::AuditLog] { + let args = jetstream_args(target_type); + let config = writable_stream_config(&args.subject); + assert_stream_writable(&config, &args, "test") + .unwrap_or_else(|err| panic!("a writable stream passes for {target_type:?}: {err}")); + } + } + + #[test] + fn assert_stream_writable_passes_when_subject_is_bound_by_wildcard() { + let args = jetstream_args(TargetType::NotifyEvent); + let mut config = writable_stream_config(&args.subject); + config.subjects = vec!["rustfs.>".to_string()]; + // The configured subject rustfs.events is captured by the rustfs.> filter. + assert_stream_writable(&config, &args, "test").expect("a wildcard-bound subject passes"); + } + + #[test] + fn assert_stream_writable_rejects_a_subject_not_bound() { + let args = jetstream_args(TargetType::NotifyEvent); + let mut config = writable_stream_config(&args.subject); + config.subjects = vec!["some.other.subject".to_string()]; + let err = assert_stream_writable(&config, &args, "test").expect_err("an unbound subject is rejected"); + assert!(matches!(err, TargetError::Configuration(_)), "the error is a generic configuration error"); + assert!( + !err.to_string().contains("some.other.subject"), + "the client message omits the stream subjects" + ); + } + + #[test] + fn assert_stream_writable_rejects_no_ack() { + let args = jetstream_args(TargetType::NotifyEvent); + let mut config = writable_stream_config(&args.subject); + config.no_ack = true; + let err = assert_stream_writable(&config, &args, "test").expect_err("a no_ack stream is rejected"); + assert!(matches!(err, TargetError::Configuration(_))); + } + + #[test] + fn assert_stream_writable_rejects_sealed() { + let args = jetstream_args(TargetType::NotifyEvent); + let mut config = writable_stream_config(&args.subject); + config.sealed = true; + let err = assert_stream_writable(&config, &args, "test").expect_err("a sealed stream is rejected"); + assert!(matches!(err, TargetError::Configuration(_))); + } + + #[test] + fn assert_stream_writable_rejects_a_too_small_duplicate_window() { + let args = jetstream_args(TargetType::NotifyEvent); + let mut config = writable_stream_config(&args.subject); + // Below the worst-case retry lifetime at the default ack timeout, so a late retry would deliver twice. + config.duplicate_window = Duration::from_secs(90); + let err = assert_stream_writable(&config, &args, "test").expect_err("a too-small window is rejected"); + assert!(matches!(err, TargetError::Configuration(_))); + } + + #[test] + fn assert_stream_writable_rejects_a_zero_duplicate_window() { + let args = jetstream_args(TargetType::NotifyEvent); + let mut config = writable_stream_config(&args.subject); + // The window defaults to zero (dedup disabled) when the operator leaves it unset. + config.duplicate_window = Duration::ZERO; + let err = assert_stream_writable(&config, &args, "test").expect_err("a zero window is rejected"); + assert!(matches!(err, TargetError::Configuration(_))); + } + + #[test] + fn stream_validation_gate_blocks_the_publish_and_classifies_retryable() { + // A failed validation returns before any publish call, classified retryable so the entry is never lost. + let stream_validated = AtomicBool::new(false); + let verdict = Err(TargetError::Configuration("configured JetStream stream is unavailable".to_string())); + let err = gate_publish_on_stream_validation(verdict, &stream_validated).expect_err("a failed validation closes the gate"); + match err { + TargetError::JetStreamPublish { retryable, detail } => { + assert!(retryable, "a validation failure is retryable"); + assert_eq!(detail, STREAM_VALIDATION_FAILED_DETAIL, "the fixed label is the detail"); + } + other => panic!("expected a retryable JetStreamPublish, got {other:?}"), + } + assert!( + !stream_validated.load(Ordering::SeqCst), + "the verdict stays closed so the next attempt re-validates" + ); + } + + #[test] + fn stream_validation_gate_pass_is_recorded_with_the_context() { + let stream_validated = AtomicBool::new(false); + gate_publish_on_stream_validation(Ok(()), &stream_validated).expect("a passing validation opens the gate"); + assert!( + stream_validated.load(Ordering::SeqCst), + "the recorded verdict lets later publishes skip re-validation" + ); + } +} diff --git a/crates/targets/src/target/postgres.rs b/crates/targets/src/target/postgres.rs index fd4dcf6f0..6469c20a0 100644 --- a/crates/targets/src/target/postgres.rs +++ b/crates/targets/src/target/postgres.rs @@ -904,8 +904,11 @@ where } fn delivery_snapshot(&self) -> TargetDeliverySnapshot { - self.delivery_counters - .snapshot(self.store.as_deref().map_or(0, |store| store.len() as u64)) + self.delivery_counters.snapshot( + self.store.as_deref().map_or(0, |store| store.len() as u64), + // Postgres targets record no terminal failures and keep no failed store. + 0, + ) } fn record_final_failure(&self) { diff --git a/crates/targets/src/target/pulsar.rs b/crates/targets/src/target/pulsar.rs index a12604691..15c8122f3 100644 --- a/crates/targets/src/target/pulsar.rs +++ b/crates/targets/src/target/pulsar.rs @@ -481,8 +481,11 @@ where } fn delivery_snapshot(&self) -> TargetDeliverySnapshot { - self.delivery_counters - .snapshot(self.store.as_deref().map_or(0, |store| store.len() as u64)) + self.delivery_counters.snapshot( + self.store.as_deref().map_or(0, |store| store.len() as u64), + // Pulsar targets record no terminal failures and keep no failed store. + 0, + ) } fn record_final_failure(&self) { diff --git a/crates/targets/src/target/redis.rs b/crates/targets/src/target/redis.rs index b62470379..8d30823cc 100644 --- a/crates/targets/src/target/redis.rs +++ b/crates/targets/src/target/redis.rs @@ -658,8 +658,11 @@ where } fn delivery_snapshot(&self) -> TargetDeliverySnapshot { - self.delivery_counters - .snapshot(self.store.as_deref().map_or(0, |store| store.len() as u64)) + self.delivery_counters.snapshot( + self.store.as_deref().map_or(0, |store| store.len() as u64), + // Redis targets record no terminal failures and keep no failed store. + 0, + ) } fn record_final_failure(&self) { diff --git a/crates/targets/src/target/webhook.rs b/crates/targets/src/target/webhook.rs index a02e9e88b..cfd29866b 100644 --- a/crates/targets/src/target/webhook.rs +++ b/crates/targets/src/target/webhook.rs @@ -644,8 +644,11 @@ where } fn delivery_snapshot(&self) -> TargetDeliverySnapshot { - self.delivery_counters - .snapshot(self.store.as_deref().map_or(0, |store| store.len() as u64)) + self.delivery_counters.snapshot( + self.store.as_deref().map_or(0, |store| store.len() as u64), + // Webhook targets record no terminal failures and keep no failed store. + 0, + ) } fn record_final_failure(&self) { diff --git a/crates/targets/tests/nats_jetstream_regression_guards.rs b/crates/targets/tests/nats_jetstream_regression_guards.rs new file mode 100644 index 000000000..016b107cf --- /dev/null +++ b/crates/targets/tests/nats_jetstream_regression_guards.rs @@ -0,0 +1,259 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Regression guards for the NATS JetStream publish feature. +//! +//! The JetStream surface can be removed and the build still succeed unless something fails when it +//! disappears. These layered guards make a silent removal or a silent classification drift a build or +//! test failure: +//! +//! - a compile-time reference to the JetStream config surface and the target entry point, +//! - a module-presence test naming the public entry type and the publish-error variant, +//! - completeness tests over the publish-error-class enums with no wildcard arm, +//! - a server-backed end-to-end publish-and-ack test, ignored by default so the offline gate stays +//! green without a NATS server. +//! +//! The end-to-end guard requires a running NATS server with JetStream enabled. To run it locally: +//! +//! docker run -d --name rustfs-nats-test -p 4222:4222 nats:2 -js +//! cargo test -p rustfs-targets --test nats_jetstream_regression_guards -- --ignored +//! +//! Override the server URL with RUSTFS_TEST_NATS_URL. + +use async_nats::jetstream::context::PublishErrorKind; +use rustfs_targets::check_nats_server_available; +use rustfs_targets::error::TargetError; +use rustfs_targets::target::nats::{NATSArgs, NATSTarget}; +use rustfs_targets::target::{FailedErrorClass, TargetType}; + +// Compile-time module-presence assertion. +// +// A const initializer names the JetStream config fields on the public target args type and the public +// target constructor by path, referencing the JetStream wiring at compile time without running +// anything. Removing the jetstream_enable, jetstream_stream_name, or jetstream_ack_timeout_secs fields +// from NATSArgs, or removing the NATSTarget constructor, while the config flag keys remain, fails to +// compile here. A static reference rather than a runtime check, so it fires at build time even when no +// test is run. +const _: fn(&NATSArgs) = |args: &NATSArgs| { + // Each field is read by name, so removing any of the three from NATSArgs fails to compile here. + let _enable: Option = args.jetstream_enable; + let _stream_name: &Option = &args.jetstream_stream_name; + let _ack_timeout: Option = args.jetstream_ack_timeout_secs; + + // The publish target entry point. Naming it by path ties this assertion to the constructor that + // wires the JetStream context, so deleting the entry point breaks the build. + let _entry: fn(String, NATSArgs) -> Result, TargetError> = NATSTarget::::new; +}; + +/// Module-presence test naming the public JetStream entry type, the health-check entry, and the +/// publish-error variant by path. Deleting the NATSTarget constructor, the check_nats_server_available +/// entry, or the JetStreamPublish error variant fails to compile this test. +#[test] +fn jetstream_public_entry_points_are_present() { + let _entry: fn(String, NATSArgs) -> Result, TargetError> = NATSTarget::::new; + + let _health_check = check_nats_server_available; + + // The publish path classifies a publish failure into this variant. Constructing it by name binds + // the guard to the variant, so removing it from TargetError breaks the build. + let publish_error = TargetError::JetStreamPublish { + retryable: true, + detail: "guard".to_string(), + }; + match publish_error { + TargetError::JetStreamPublish { retryable, detail } => { + assert!(retryable); + assert_eq!(detail, "guard"); + } + other => panic!("expected the JetStream publish variant, found {other:?}"), + } +} + +/// Cross-module completeness over the publish-error class. +/// +/// Lists every async-nats PublishErrorKind variant by name with no wildcard arm. The publish path +/// classifies each kind into retryable or terminal. A new variant added upstream, or a renamed or +/// removed one, fails to compile this match, forcing the classification to be revisited rather than +/// silently folding the new kind into a catch-all. +#[test] +fn publish_error_kind_is_exhaustively_enumerated() { + fn name_of(kind: PublishErrorKind) -> &'static str { + match kind { + PublishErrorKind::StreamNotFound => "StreamNotFound", + PublishErrorKind::WrongLastMessageId => "WrongLastMessageId", + PublishErrorKind::WrongLastSequence => "WrongLastSequence", + PublishErrorKind::TimedOut => "TimedOut", + PublishErrorKind::BrokenPipe => "BrokenPipe", + PublishErrorKind::MaxAckPending => "MaxAckPending", + PublishErrorKind::MaxPayloadExceeded => "MaxPayloadExceeded", + PublishErrorKind::Other => "Other", + } + } + + let all = [ + PublishErrorKind::StreamNotFound, + PublishErrorKind::WrongLastMessageId, + PublishErrorKind::WrongLastSequence, + PublishErrorKind::TimedOut, + PublishErrorKind::BrokenPipe, + PublishErrorKind::MaxAckPending, + PublishErrorKind::MaxPayloadExceeded, + PublishErrorKind::Other, + ]; + assert_eq!( + all.len(), + 8, + "the publish-error class has eight kinds, update the classification on a change" + ); + for kind in all { + assert!(!name_of(kind).is_empty()); + } +} + +/// Cross-module completeness over the failed-store error class. +/// +/// Lists every FailedErrorClass variant by name with no wildcard arm. The replay worker tags a failed +/// entry with one of these classes. Adding or removing a class fails to compile this match, so the +/// tagging and the operator-facing tag stay in step. +#[test] +fn failed_error_class_is_exhaustively_enumerated() { + fn tag_of(class: FailedErrorClass) -> &'static str { + match class { + FailedErrorClass::Terminal => "terminal", + } + } + + let all = [FailedErrorClass::Terminal]; + assert_eq!( + all.len(), + 1, + "the failed-store error class has one variant, update the tagging on a change" + ); + for class in all { + assert_eq!(tag_of(class), class.as_str(), "the guard tag matches the operator-facing tag"); + } +} + +fn server_url() -> String { + std::env::var("RUSTFS_TEST_NATS_URL").unwrap_or_else(|_| "nats://127.0.0.1:4222".to_string()) +} + +fn jetstream_args(subject: &str, stream_name: &str, queue_dir: &str) -> NATSArgs { + NATSArgs { + enable: true, + address: server_url(), + subject: subject.to_string(), + username: String::new(), + password: String::new(), + token: String::new(), + credentials_file: String::new(), + tls_ca: String::new(), + tls_client_cert: String::new(), + tls_client_key: String::new(), + tls_required: false, + queue_dir: queue_dir.to_string(), + queue_limit: 0, + jetstream_enable: Some(true), + jetstream_stream_name: Some(stream_name.to_string()), + jetstream_ack_timeout_secs: Some(30), + target_type: TargetType::NotifyEvent, + } +} + +/// Server-backed end-to-end publish-and-ack guard. +/// +/// Provisions a writable stream as an operator would, then drives the real target publish path with +/// the JetStream flag on: init connects and validates the stream, save persists the event to the +/// on-disk queue, send_from_store publishes and awaits the real PublishAck and clears the entry on it. +/// The stream message count rising and the queue draining proves the message was published and +/// acknowledged through the production code, the user-visible signal a build-only check cannot see. +/// +/// Ignored by default because it needs a running NATS server with JetStream. +#[tokio::test] +#[ignore] +async fn end_to_end_publish_is_acked_on_the_stream() { + use rustfs_targets::EventName; + use rustfs_targets::Target; + use rustfs_targets::target::EntityTarget; + use std::sync::Arc; + use uuid::Uuid; + + let suffix = Uuid::new_v4().simple().to_string(); + let stream_name = format!("RUSTFS_TEST_{suffix}"); + let subject = format!("rustfs.guard.{suffix}"); + let queue_dir = std::env::temp_dir().join(format!("rustfs-nats-guard-{suffix}")); + let queue_dir = queue_dir.to_string_lossy().to_string(); + + // Provision the stream as the operator would. RustFS validates it, never creates it. + let client = async_nats::connect(server_url()).await.expect("connect to NATS"); + let context = async_nats::jetstream::new(client); + let _ = context.delete_stream(&stream_name).await; + context + .create_stream(async_nats::jetstream::stream::Config { + name: stream_name.clone(), + subjects: vec![subject.clone()], + duplicate_window: std::time::Duration::from_secs(300), + ..Default::default() + }) + .await + .expect("provision the stream"); + + let args = jetstream_args(&subject, &stream_name, &queue_dir); + let target = NATSTarget::::new("guard-target".to_string(), args).expect("build the flag-on target"); + + target + .init() + .await + .expect("init connects and validates the operator-provisioned stream"); + + let count_before = stream_message_count(&context, &stream_name).await; + + let event = EntityTarget { + object_name: "guard/object.txt".to_string(), + bucket_name: "guard-bucket".to_string(), + event_name: EventName::ObjectCreatedPut, + data: "guard-payload".to_string(), + }; + target + .save(Arc::new(event)) + .await + .expect("save persists the event to the queue"); + + let keys = target.store().expect("the flag-on target has a queue store").list(); + assert_eq!(keys.len(), 1, "the saved event is queued exactly once"); + target + .send_from_store(keys[0].clone()) + .await + .expect("the queued event publishes and is acked"); + + let count_after = stream_message_count(&context, &stream_name).await; + assert_eq!( + count_after, + count_before + 1, + "the published event is durably on the stream after its ack" + ); + + let remaining = target.store().expect("the flag-on target has a queue store").list(); + assert!(remaining.is_empty(), "the queue entry is cleared on the ack"); + + target.close().await.expect("close drains the target"); + let _ = context.delete_stream(&stream_name).await; + let _ = std::fs::remove_dir_all(&queue_dir); +} + +async fn stream_message_count(context: &async_nats::jetstream::Context, stream_name: &str) -> u64 { + let mut stream = context.get_stream(stream_name).await.expect("look up the stream"); + let info = stream.info().await.expect("read the stream info"); + info.state.messages +} diff --git a/crates/targets/tests/nats_jetstream_validation_integration.rs b/crates/targets/tests/nats_jetstream_validation_integration.rs new file mode 100644 index 000000000..3981b5e53 --- /dev/null +++ b/crates/targets/tests/nats_jetstream_validation_integration.rs @@ -0,0 +1,143 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! JetStream stream validation integration tests for the NATS target. +//! +//! These tests are ignored because they require a running NATS server with JetStream enabled. To +//! run locally: +//! +//! docker run -d --name rustfs-nats-test -p 4222:4222 nats:2 -js +//! cargo test -p rustfs-targets --test nats_jetstream_validation_integration -- --ignored +//! +//! Override the server URL with RUSTFS_TEST_NATS_URL. Each test provisions and removes its own +//! stream. RustFS never provisions the stream, the test harness does, mirroring the operator role. + +use async_nats::jetstream::stream::Config; +use rustfs_targets::check_nats_server_available; +use rustfs_targets::target::TargetType; +use rustfs_targets::target::nats::NATSArgs; +use std::time::Duration; +use uuid::Uuid; + +fn server_url() -> String { + std::env::var("RUSTFS_TEST_NATS_URL").unwrap_or_else(|_| "nats://127.0.0.1:4222".to_string()) +} + +fn jetstream_args(subject: &str, stream_name: &str) -> NATSArgs { + NATSArgs { + enable: true, + address: server_url(), + subject: subject.to_string(), + username: String::new(), + password: String::new(), + token: String::new(), + credentials_file: String::new(), + tls_ca: String::new(), + tls_client_cert: String::new(), + tls_client_key: String::new(), + tls_required: false, + queue_dir: "/tmp/rustfs-nats-jetstream-test".to_string(), + queue_limit: 0, + jetstream_enable: Some(true), + jetstream_stream_name: Some(stream_name.to_string()), + jetstream_ack_timeout_secs: Some(30), + target_type: TargetType::NotifyEvent, + } +} + +/// A pre-provisioned writable stream: it captures the subject, returns acks, accepts writes, and its +/// duplicate window exceeds the retry lifetime. The test harness creates it, never RustFS. +async fn provision_writable_stream(stream_name: &str, subject: &str) { + let client = async_nats::connect(server_url()).await.expect("connect to NATS"); + let context = async_nats::jetstream::new(client); + // Clear any leftover from a prior aborted run. Absent is fine, so the result is dropped. + let _ = context.delete_stream(stream_name).await; + context + .create_stream(Config { + name: stream_name.to_string(), + subjects: vec![subject.to_string()], + duplicate_window: Duration::from_secs(300), + ..Default::default() + }) + .await + .expect("provision the stream"); +} + +async fn remove_stream(stream_name: &str) { + let client = async_nats::connect(server_url()).await.expect("connect to NATS"); + let context = async_nats::jetstream::new(client); + // Best-effort teardown. A failure to delete does not fail the test, so the result is dropped. + let _ = context.delete_stream(stream_name).await; +} + +#[tokio::test] +#[ignore] +async fn missing_stream_fails_the_health_check() { + let stream_name = format!("RUSTFS_TEST_{}", Uuid::new_v4().simple()); + let args = jetstream_args("rustfs.events", &stream_name); + // No stream is provisioned, so the lookup must fail. + let err = check_nats_server_available(&args) + .await + .expect_err("a missing stream fails the health check"); + assert!(!err.to_string().is_empty(), "the failure carries a generic message"); +} + +#[tokio::test] +#[ignore] +async fn valid_stream_passes_the_health_check() { + let stream_name = format!("RUSTFS_TEST_{}", Uuid::new_v4().simple()); + let subject = "rustfs.events"; + provision_writable_stream(&stream_name, subject).await; + let args = jetstream_args(subject, &stream_name); + let result = check_nats_server_available(&args).await; + remove_stream(&stream_name).await; + result.expect("a pre-provisioned writable stream passes the health check"); +} + +#[tokio::test] +#[ignore] +async fn stream_not_capturing_the_subject_fails_the_health_check() { + let stream_name = format!("RUSTFS_TEST_{}", Uuid::new_v4().simple()); + // The stream binds a different subject than the target publishes to. + provision_writable_stream(&stream_name, "some.other.subject").await; + let args = jetstream_args("rustfs.events", &stream_name); + let result = check_nats_server_available(&args).await; + remove_stream(&stream_name).await; + result.expect_err("a stream that does not capture the subject fails the health check"); +} + +#[tokio::test] +#[ignore] +async fn too_small_duplicate_window_fails_the_health_check() { + let stream_name = format!("RUSTFS_TEST_{}", Uuid::new_v4().simple()); + let subject = "rustfs.events"; + let client = async_nats::connect(server_url()).await.expect("connect to NATS"); + let context = async_nats::jetstream::new(client); + // Clear any leftover from a prior aborted run. Absent is fine, so the result is dropped. + let _ = context.delete_stream(&stream_name).await; + context + .create_stream(Config { + name: stream_name.clone(), + subjects: vec![subject.to_string()], + // Below the worst-case retry lifetime at the default ack timeout. + duplicate_window: Duration::from_secs(60), + ..Default::default() + }) + .await + .expect("provision the stream"); + let args = jetstream_args(subject, &stream_name); + let result = check_nats_server_available(&args).await; + remove_stream(&stream_name).await; + result.expect_err("a too-small duplicate window fails the health check"); +} diff --git a/docs/operations/nats-jetstream.md b/docs/operations/nats-jetstream.md new file mode 100644 index 000000000..1116a031e --- /dev/null +++ b/docs/operations/nats-jetstream.md @@ -0,0 +1,356 @@ +# NATS JetStream Operations Guide + +The guide covers enabling and operating the JetStream publish path for the NATS +notify and audit targets in RustFS. It is written for operators who need +at-least-once delivery of events to NATS, must pre-provision and size the stream +the targets publish to, and need to diagnose delivery failures. + +## What the JetStream Path Does + +By default the NATS notify and audit targets publish with NATS Core, which +returns once the message is written to the socket. A process or server failure +in the gap between the socket write and the server persisting the message loses +the event. The JetStream path closes that gap. With it enabled, each event is +published to a JetStream stream and the local store-and-forward queue entry +clears only after the server returns a publish acknowledgement, which means the +stream leader has accepted and sequenced the message, or after a terminal +rejection has been recorded in the failed-events store. A server restart +mid-flight loses nothing, because an unacknowledged event stays in the local +queue and replays after the server returns. + +The path is opt-in and off by default. With it off, behaviour is the NATS Core +path. It applies to both the notify NATS target and the audit NATS target. + +RustFS gets each event to the server without losing it before the +acknowledgement. The operator owns the stream and its policy. RustFS validates +that the stream exists and is writable and reports a validation failure +otherwise. A store-backed target keeps running through a failed validation, +holding queued events until the stream is repaired. RustFS never creates the +stream. Stream retention, storage, and replication policy stay under operator +control. + +## Enabling JetStream: Recommended Configuration + +Three keys turn the path on and tune it. Each is available on both the notify +NATS target and the audit NATS target, and each has a configuration-key form and +an environment-variable form per target. + +```bash +RUSTFS_NOTIFY_NATS_JETSTREAM_ENABLE=true +RUSTFS_NOTIFY_NATS_JETSTREAM_STREAM_NAME=RUSTFS_EVENTS +RUSTFS_NOTIFY_NATS_JETSTREAM_ACK_TIMEOUT_SECS=30 +RUSTFS_NOTIFY_NATS_QUEUE_DIR=/var/lib/rustfs/notify-nats +``` + +The audit target takes the same keys under the RUSTFS_AUDIT_NATS_ prefix. + +- jetstream_enable turns the JetStream path on for the target. Off by default. + Environment forms RUSTFS_NOTIFY_NATS_JETSTREAM_ENABLE and + RUSTFS_AUDIT_NATS_JETSTREAM_ENABLE. +- jetstream_stream_name is the name of the pre-provisioned stream to publish to. + Required when enable is on, with no default. Environment forms + RUSTFS_NOTIFY_NATS_JETSTREAM_STREAM_NAME and + RUSTFS_AUDIT_NATS_JETSTREAM_STREAM_NAME. +- jetstream_ack_timeout_secs is how long a publish waits for an acknowledgement + before it is treated as timed out and retried. Range 10 to 120, default 30. + Environment forms RUSTFS_NOTIFY_NATS_JETSTREAM_ACK_TIMEOUT_SECS and + RUSTFS_AUDIT_NATS_JETSTREAM_ACK_TIMEOUT_SECS. +- queue_dir is the local store-and-forward queue directory and is required when + enable is on. Durability is not achievable without a local store to replay + from. Environment forms RUSTFS_NOTIFY_NATS_QUEUE_DIR and + RUSTFS_AUDIT_NATS_QUEUE_DIR. + +A configuration that enables the path without a stream name or without a queue +directory is rejected at startup and in the admin validation path. An +acknowledgement timeout outside the 10 to 120 second range is rejected the same +way. + +The 30 second acknowledgement timeout suits production. A server replicating to +multiple replicas acknowledges only after the replication and the deferred fsync, +which legitimately takes well over 100 milliseconds at the tail, so a short +timeout would spuriously fail a publish mid-replication. + +After enabling, confirm the target is on the JetStream path before relying on it. +Value errors, an out-of-range acknowledgement timeout or a missing stream name +while enable is on, are already rejected loudly at startup and in the admin +validation path. A mistyped configuration key name is not a value error, so it +escapes that check: the key reads as absent, and the target stays on the NATS +Core path with no durability. Verify the enable took effect by confirming the +target reports its JetStream fields on startup and logs the stream-validation +success line naming the configured stream. Absence of that line for a target that +was meant to be enabled indicates a key name the server did not recognise. + +## How Delivery Works + +An event is queued to the local store-and-forward queue first, then published +through JetStream. The publish carries a stable Nats-Msg-Id header and the path +awaits the real publish acknowledgement. + +- The queue entry clears only after the durable acknowledgement returns, or + after a terminal rejection has been recorded in the failed-events store. A + timeout, or any retryable error, retains the entry for a retry. The entry is + never cleared without one of those two outcomes and never silently dropped. A + process interruption between the failed-record write and the live-entry removal + can leave a record whose event a later replay still delivers, a diagnostic + residue and not a lost event. +- The Nats-Msg-Id is minted once when the event is queued and stored with it. It + is identical across every retry and replay of that entry, and unique per + entry, so the server collapses retries and replays of the same event within + the stream duplicate window. A retry after a slow acknowledgement reuses the + same identifier and is not delivered twice. +- A crash before the acknowledgement leaves the entry in the queue. The entry + replays after restart. Closing the target releases the cached connection and + context but never the queued entries, so an entry queued before close survives + on disk and replays on the next start. +- A broker outage detected at connection establishment keeps queued events on + disk. Replay retries with backoff while the broker is unreachable, and the + queued events deliver when the connection recovers, keeping entries queued like + the NATS Core path rather than replicating its mechanics byte for byte. Every + retryable publish failure on an established connection is treated the same way, + kept on the live queue and retried until it delivers, which includes a + connection dropping during the stream-validation lookup. Only a non-retryable + rejection is moved to the failed-events store. + +For durability across a node failure, provision the stream with a replica count +of at least 3. An acknowledgement returns after the stream leader commits. A +leader that acknowledges and then fails before the message replicates to a quorum +can lose that message on failover, so a single-replica stream is durable only +against a clean restart, not against the loss of the node holding the data. + +## Pre-provisioning the Stream + +The stream is provisioned by the operator before the path is enabled. RustFS +reads the stream once at target init and in the admin validation path, asserts +the requirements below, and reports a validation failure rather than publishing +into a stream where writes would silently fail. A failed validation does not +stop a store-backed target. Events keep queueing to the local store. Revalidation +runs while the validation verdict is unset and stops once one validation passes, +resuming only after the verdict is reset, and the queued events deliver once the +stream is repaired. The stream is not auto-created. A missing stream is an +error. + +The stream must: + +- Exist and be writable. A missing or unreachable stream fails validation. +- Capture the configured publish subject in its subject filter, by literal match + or by a NATS wildcard. A stream that does not capture the subject fails + validation. +- Acknowledge writes (no_ack false). A stream with no_ack set never returns an + acknowledgement, so the queue would never clear. It fails validation. +- Not be sealed. A sealed stream rejects writes and fails validation. +- Set a duplicate window of at least the retry span (see the next section). A + window below that span fails validation. + +Choose retention, storage type, and replica count to match the durability the +deployment needs. Use file storage if a server restart must preserve +already-persisted events. These are operator decisions and RustFS does not set +them. If the stream is provisioned shortly after the path is enabled, the queued +events deliver on the next replay rather than being lost, because a +stream-not-found error is retryable and keeps the events on the live queue until +the stream exists. + +## Setting the Duplicate Window + +The acknowledgement timeout and the stream duplicate window are linked. The +duplicate window must cover the worst-case retry span, or a late retry of an +event the server already persisted is delivered a second time instead of being +recognised as a duplicate. + +The worst-case retry span is the retry count times the acknowledgement timeout, +plus the realized backoff sleeps across the retries and a final headroom term: + + duplicate_window >= (retry_count * ack_timeout_secs) + realized_backoff + headroom + +There are 5 retry attempts. The realized backoff sleeps sum to 60 seconds +(4 + 8 + 16 + 32), and no sleep follows the last attempt, so a 64 second headroom +term is added above the realized span. At the default 30 second acknowledgement +timeout the required window is: + + (5 * 30) + 60 + 64 = 274 seconds + +Set the stream duplicate window to at least 274 seconds when the acknowledgement +timeout is at the 30 second default. A configuration whose duplicate window is +below this span is rejected at validation, with the configured and required +window named in the log line. + +The span scales with the acknowledgement timeout. Raising +jetstream_ack_timeout_secs requires raising the stream duplicate window in step +using the formula above. The 60 second realized backoff and the 64 second headroom +stay fixed, so each extra second of acknowledgement timeout adds 5 seconds to the +required window. At the 120 second maximum timeout the required window is +(5 * 120) + 60 + 64 = 724 seconds. Each publish attempt, including connection +establishment, is bounded by a single deadline equal to the acknowledgement +timeout. The formula is a conservative upper bound on the retry span, and the +final headroom term holds the required window above the realized span. + +The validated window covers one retry cycle: the worst-case span from the first +publish attempt of a stored entry to its last within a single replay cycle. An +entry that exhausts a cycle without delivering stays on the live queue and is +retried on a later cycle, so an entry surviving across cycles can spend longer in +the queue than the duplicate window and be delivered again, consistent with +at-least-once delivery. + +## The Failed-events Store + +An event that cannot be delivered is recorded, not silently dropped. Only one +case produces a failed event: + +- A terminal rejection. A message that exceeds the maximum payload size, a wrong + expected last message identifier or last sequence, or a sealed stream. These do + not improve with a retry, so they fail fast. + +Every other rejection is retryable and never produces a failed event. A timeout, +backpressure, no responders, a missing stream, a subject the stream does not +capture, or an authorization failure keeps the event on the live queue and +retries it until it delivers or an operator intervenes. A wrong subject and wrong +credentials surface through the validation and connection path as retryable, so +they hold the events on the live queue rather than moving them to the failed +store. + +A failed event is moved to an on-disk failed store that sits in a failed +child directory inside the queue directory, one per target. The directory is created +lazily on the first failed write, so a target that never fails terminally leaves +no failed directory. The entry preserves the event body, its routing metadata, the +deduplication identifier, an error class tag of terminal, the failure time, and +the retry count. An error-level log line is written for each failed event, naming +the bucket, object, event name, and the error, so a broken integration is visible +rather than hidden. A record in the failed store is diagnostic only and is never +republished, so a condition repaired later, for example a raised broker payload +limit, delivers only events still on the live queue. The NATS Core path without +JetStream instead retries such an event until the limit allows it. + +The failed store is bounded by count (10000 entries per target) and by age (a 72 +hour retention), and is kept separate from the live queue limit so an +accumulation of failures cannot crowd out new events. The count is maintained as a +cached value, seeded at startup and reconciled to the directory on each +maintenance interval, so it stays accurate without a directory scan on the hot +path. Failed-store writes and the maintenance scan run under one exclusive guard, +so the at-bound check and the write stay atomic and the bound holds against +concurrent writers. A change made to the directory outside the store can drift the +cached count until the next maintenance interval reconciles it. When the count +bound is reached the oldest failed entry is dropped, with a warning naming the +trimmed entry, so a newer failure is never lost in favour of an older one. Entries +past the retention bound are removed as expired on the replay maintenance tick. + +Because a retryable failure keeps events on the live queue, a long outage grows +the queue toward its configured queue_limit bound. Once the queue reaches that +bound, new events are rejected at ingest with a logged error rather than +overwriting queued events, so the backlog is bounded and visible. + +Size queue_limit for the longest outage the deployment must survive without +rejecting new events. Multiply the peak event rate in events per second by the +outage window in seconds. A target that averages 50 events per second and must +ride out a one hour broker outage needs a queue_limit of at least 50 * 3600 = +180000 entries. Add headroom above the calculated figure, and provision the +queue directory storage for the resulting entry count. + +## Publish Outcome Handling + +Every publish outcome falls into one of two families. Transient conditions are +retried until they deliver and the entry stays on the live queue, so a transient +condition never reaches the failed store. Permanent conditions move to the failed +store immediately. Transient conditions classify toward retry because a terminal +misclassification risks losing an entry once the failed store retention lapses. +Permanent conditions move immediately so a poison message cannot block the queue. + +Both families cover publish outcomes on an established connection. A failure to +establish the connection at all, a refused connection or unreadable TLS +material, is handled before either family applies: the entry retries with +backoff and stays queued until the connection recovers. A publish-level failure +on an established connection, including a connection that drops during the +stream-validation lookup, is retried on the live queue when it is transient and +moved to the failed store only when it is a permanent rejection. + +| Outcome family | Examples | Handling | +| --- | --- | --- | +| Connectivity | connection lost mid-publish, broken pipe | Retried until delivered, live queue | +| Timeouts | no acknowledgement within the timeout, attempt deadline reached | Retried until delivered, live queue | +| Cluster in transition | no leader elected, peer membership changing | Retried until delivered, live queue | +| Stream offline | stream or JetStream subsystem temporarily offline, stream not found | Retried until delivered, live queue | +| Resource and quota exhaustion | insufficient server resources, storage, memory, or account quota reached | Retried until delivered, live queue | +| Server errors | any rejection reporting a 5xx status, with or without a specific error code | Retried until delivered, live queue | +| Permanent rejections | payload too large, wrong expected sequence or message identifier, sealed stream | Failed store immediately | + +A sealed stream is classified at three points, with three outcomes. Startup validation and the admin +validation path reject a sealed stream before the target serves traffic, so init fails and no event +is queued against it. A stream sealed while the target runs is caught by the next validation on the +publish path, which classifies it retryable and keeps the entry on the live queue while validation +keeps failing. A publish that reaches an already-cached validation pass and is then rejected with the +sealed-stream code is terminal and moves to the failed store immediately. The permanent-rejections +row lists that last outcome. + +## Observability + +The failed_store_length gauge reports the number of entries in the failed-events +store per target, next to the existing failed_messages and queue_length gauges, +on both the notify and audit metric paths. A rising failed_store_length points to +terminal rejections accumulating for a target. An exhaustion warning marks each +cycle where an entry spends its full retry budget without delivering, after which +the entry stays queued and retries on the next scan. The warning repeats once per +retry cycle while the entry stays queued, so a persistent delivery problem is +visible at warn level without per-attempt noise. The failed_messages count +advances only on terminal and dropped events, never on a retry or an exhaustion, +so it counts entries that left the queue for good rather than entries still +retrying. + +## Troubleshooting + +- Validation fails at startup with a missing-stream error. The stream named in + jetstream_stream_name does not exist on the server, or is not reachable. + Provision it, or correct the name. +- Validation fails with a subject, no_ack, sealed, or duplicate-window error. The + pre-provisioned stream does not capture the publish subject, has no_ack set, is + sealed, or has a duplicate window below the worst-case retry span. Correct the + stream configuration to satisfy the pre-provisioning requirements above. +- Repeated stream-not-found errors in the log. The stream disappeared or was + never created. The publish is retryable, so the events stay on the live queue + and retry. Provision the stream so the queued events deliver on the next + replay. +- A warning that a publish was acknowledged by an unexpected stream. The + configured stream no longer captures the publish subject and another stream + does. The mismatched publish is rejected with a retryable error, the entry + stays on the live queue, and the mismatch resets the stream-validation + verdict. Later retries then fail validation before publishing, so the log + shows the validation failure rather than a repeating mismatch warning. The + mismatch warning fires again only after a validation pass lets another + publish through. The acknowledging stream named in the warning has persisted + a copy of each mismatched publish. Inspect that stream and remove the stray + copies. Delivery resumes once the configured stream captures the subject + again and validation passes. +- Health checks slow while a stream fails validation. Health reflects the last + stream-validation verdict rather than a live lookup on every check. The verdict + resets on a reconnect, a TLS rotation, an acknowledgment naming an unexpected + stream, or a stream-not-found publish outcome, and a reset forces a live stream + lookup on the next check or publish, bounded by the acknowledgement timeout. A + health snapshot taken right after a reset can take up to that timeout per + affected target until the stream is repaired. After a broker connection heals + on its own, publishes and health rely on the last validation verdict until an + error outcome resets it, a wrong-stream acknowledgment or a stream-not-found + publish outcome, so a stream reconfigured during a silent reconnect is + detected on the first publish evidence rather than immediately. +- Events stay in the queue and do not clear. The server is not acknowledging. + Check connectivity, that the stream subject filter captures the publish + subject, and that the server has capacity. Unacknowledged events are retained, + not lost. +- Duplicate deliveries observed. The duplicate window is shorter than the + worst-case retry span. Raise the stream duplicate window using the formula + above, especially after raising the acknowledgement timeout. An outage or a + restart that keeps an unacknowledged entry queued longer than the duplicate + window can also produce a duplicate on replay, consistent with at-least-once + delivery. Throttling RUSTFS_NOTIFY_TARGET_STREAM_CONCURRENCY below the number + of concurrently backlogged targets inserts untimed waits between the retries + of one entry and can push a late retry past the window. Keep the default or + add window margin when lowering it. +- Failed-store entries accumulating. A terminal rejection is recurring: a + message over the maximum payload size, a wrong expected last message identifier + or last sequence, or a sealed stream. Failed entries are on-disk diagnostic + records carrying the error class and a fixed diagnostic detail for inspection. + They are not re-published, and they are removed after the retention period. A + wrong subject or wrong credentials do not land here. They surface as retryable + and hold the events on the live queue until the configuration is fixed. + +## Disabling the Path + +Set jetstream_enable off. The target reverts to the NATS Core path. Events +already in the queue are delivered by the standard replay. No failed-store +entries are created while the path is off.