On macOS with DNS interception services, .invalid TLD domains resolve
to an interception address (e.g. 198.18.16.173) instead of failing DNS
resolution. The health probe then classifies the error as Unreachable
rather than DnsFailure. Accept both outcomes since both are correct
non-reachable error classifications.
Update selected workspace dependencies and lockfile entries.
Keep async-nats and rcgen on explicit feature sets while preserving the RustFS targets and TLS test surfaces.
Co-authored-by: heihutu <heihutu@gmail.com>
A single instance with a malformed `enable` value (e.g. the typo `enable`
instead of `enabled`/`on`) made every configured notification/audit target
of every type fail to load. `create_targets_from_config_with_store_mode`
collected instance configs through the strict `try_collect_target_configs`,
whose `.collect::<Result<Vec, _>>()` short-circuits on the first error and
drops all siblings; the surrounding `?` then aborted the loop over every
plugin type.
This contradicted the function's own documented contract ("Creation is
fault-isolated per instance") and was inconsistent: target construction
failures were already isolated into the `failures` accumulator, only config
collection failures were fatal.
Add `collect_target_config_results[_from_env]`, which returns the enabled
`(id, config)` pairs plus a per-instance failure summary, and use it in the
create path. Per-instance collection errors now flow into the same
`failures`/`creation_failures` accumulator as construction failures: healthy
siblings and other target types still load, while the malformed instance is
surfaced so the notify lifecycle stays non-converged/retryable and an Admin
write cannot report a false success. Audit likewise keeps logging through
its valid targets instead of going fully Stopped.
Follow-up to #5088.
Restore the workspace reqwest default feature stack for RustFS outbound HTTPS clients, while keeping per-crate extra APIs such as json, stream, and multipart explicit. Lazily initialize the notification runtime from admin target access when RUSTFS_NOTIFY_ENABLE=true is already effective, and add regression coverage for HTTPS webhook custom CA handling and target-list visibility.
Fixes#5052.
Co-authored-by: heihutu <heihutu@gmail.com>
Generate a unique Pulsar producer name for each producer instance while preserving the target type and target id context. This avoids reload-time collisions when an old producer is still connected while a new destination is initialized.
Co-authored-by: heihutu <heihutu@gmail.com>
* chore(deps): tighten crate dependency features
Narrow Tokio and dependency feature declarations for protocols, TLS runtime, utils, targets, and replication based on direct crate usage.
Co-Authored-By: heihutu <heihutu@gmail.com>
* chore(deps): trim hyper-rustls features
Keep direct hyper-rustls features aligned with the actual RustFS call sites. rustfs-targets only needs native root loading and the rustls provider/TLS policy features for MQTT TLS config construction, while rustfs-ecstore needs the HTTP connector protocol features but not webpki roots.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* chore(deps): remove redundant dependency features
Remove manifest feature entries that are implied by other requested features in the same dependency declaration.
Verified that the resolved Cargo feature graph is unchanged after the cleanup.
Co-Authored-By: heihutu <heihutu@gmail.com>
* chore(deps): narrow tokio and reqwest features
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Move workspace-level dependency feature lists into the member crates that consume each dependency while keeping required default-features flags at the workspace root.
Also refresh starshard to 2.2.2 via cargo update and cargo upgrade --exclude ratelimit.
Co-authored-by: heihutu <heihutu@gmail.com>
feat(targets): add an opt-in NATS JetStream publish path for the notify and audit targets
The NATS notify and audit targets publish through NATS Core, which returns
before the server has durably accepted the message. A broker restart or a
connection drop between the publish and the flush loses the event, even though
the send queue has already cleared it, and no acknowledgement gates that clear.
An opt-in JetStream publish path clears a queued event only after the server
returns a durable PublishAck, so delivery is at-least-once across a broker
restart or a reconnect. It applies to both the notify and audit NATS targets, is
off by default, and is byte-identical to the NATS Core path when disabled.
The path includes durable store-and-forward, a stable dedup id sent as the
Nats-Msg-Id header so a replayed event is collapsed by the stream duplicate
window, pre-flight stream validation, and a bounded failed-events store for
terminally-failed and retry-exhausted events. Three configuration keys per
target select it: JETSTREAM_ENABLE, JETSTREAM_STREAM_NAME, and
JETSTREAM_ACK_TIMEOUT_SECS, under the RUSTFS_NOTIFY_NATS_ and RUSTFS_AUDIT_NATS_
prefixes.
The on-disk batch filename separator changes from colon to underscore so
batch names are valid on Windows filesystems, with transparent read-back
of files written under the previous separator. The migration affects the
shared queue store for every target type and lands with this feature
because the store gains its first Windows-exercised paths here.
Co-authored-by: houseme <housemecn@gmail.com>
The pulsar config validation rejected any non-`pulsar+ssl` broker whenever
`tls_allow_insecure` was set or `tls_hostname_verification` was disabled.
Both toggles are inert on a plaintext `pulsar://` broker — the Pulsar client
only applies them for `pulsar+ssl` — so treating a non-default value as fatal
is wrong.
This surfaced as issue #4796: the console persists `tls_hostname_verification`
as `false` (the checkbox defaults to unchecked while the server default is
`on`). At creation time the value is not present in the unmerged request and
falls back to the safe default, so the connectivity check passes and the
target comes online. On restart the persisted config is merged and validated,
the `false` is read back, and the target is rejected — permanently offline
even though Pulsar is reachable.
Relax both the loader-side (`validate_pulsar_broker_config`) and runtime-side
(`PulsarArgs::validate`) checks so only a `tls_ca` bundle — real TLS trust
material that is never defaulted to a non-empty value — requires a
`pulsar+ssl` scheme. The inert toggles no longer fail the target. This also
heals configs already persisted with the bad value. Add regression coverage
for both paths.
Co-authored-by: heihutu <heihutu@gmail.com>
fix(targets): harden Postgres/MySQL SQL backends (pools, error class, idempotency)
Postgres (postgres.rs):
- Add deadpool wait/create/recycle timeouts + tokio-postgres connect_timeout,
and wrap every pool.get() in a Tokio hard-limit timeout so an unreachable
broker/DB can no longer block send_body/probe_table/is_active forever;
checkout timeouts map to TargetError::Timeout to trigger store replay.
- namespace format now DELETEs the row on s3:ObjectRemoved:* events instead of
UPSERTing, keeping the table consistent with the object lifecycle.
- map_pg_error: SQLSTATE class 40 (serialization_failure 40001,
deadlock_detected 40P01, transaction rollback) is now transient (Timeout,
retryable) instead of a permanent Request. Extracted map_pg_sqlstate for
unit-testable classification.
MySQL (mysql.rs):
- Wrap pool.get_conn() in a Tokio checkout timeout across insert/init/liveness
paths so an unreachable server cannot block the delivery thread.
- Add an event_id idempotency key: tables created by the target now carry an
event_id VARCHAR(255) PRIMARY KEY, inserts use ON DUPLICATE KEY UPDATE, and
replays reuse the stable store key so a lost-ack replay no longer appends a
duplicate audit row. Legacy two-column tables are detected and fall back to
the non-idempotent insert with a warning (backward compatible).
- redact_mysql_dsn splits on the last '@' so a password containing '@' no
longer leaks its tail into Debug output.
- Disconnect the stale pool before dropping it on inline TLS reload instead of
leaking its connections.
- Cache TLS file mtimes and only recompute the inline fingerprint when a cert
file changes, avoiding a 3-file read+hash on every checkout.
Adds unit tests for SQLSTATE classification, namespace delete SQL, removal-event
detection, DSN redaction with '@' in the password, and the MySQL insert/DDL
builders.
Relates to rustfs/backlog#976
Relates to rustfs/backlog#973
Relates to rustfs/backlog#983
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(targets): make persistent queue store crash-safe and replay lifecycle correct
Harden the target notification persistent queue (store.rs) and the replay
worker lifecycle (runtime) against data loss, silent truncation, ordering
drift, orphaned tasks, and a few low-risk robustness gaps.
store.rs
- Atomic, durable writes: write to a per-key temp file, fsync (sync_all),
then rename into place; best-effort parent-dir fsync. A crash mid-write
can no longer lose an acknowledged event or leave a half-written payload
that reads as a valid entry.
- open() now removes leftover .tmp residue and zero-byte files, and only
indexes files matching the queue extension, so ghosts/foreign files are
never replayed.
- FIFO ordering is derived from time-ordered UUIDv7 entry names instead of
coarse, clock-dependent file mtimes, so replay order is stable and
identical after a restart.
- Clamp HashMap/Vec pre-allocation derived from untrusted inputs
(entry_limit, batch item_count) to avoid capacity-overflow panics / giant
allocations.
target/mod.rs
- QueuedPayload::decode validates body length against the recorded
payload_len, rejecting torn/truncated writes instead of delivering a
silently truncated body.
- send_from_store purges a NotFound/empty entry (index + file) instead of
skipping it, so it cannot occupy a queue slot and be replayed forever.
- sanitize_queue_dir_component appends a stable hash suffix when the id was
lossy, so distinct target ids can no longer collapse onto the same queue
directory; path-safe ids are unchanged (no migration).
runtime
- Replay backoff, idle waits, and inter-scan pauses are now cancel-aware, so
reload/shutdown is not blocked for the full retry delay.
- ReplayWorkerManager::stop_all signals cancellation and then awaits each
worker's exit (bounded, with abort fallback), preventing orphaned tasks
and overlapping drain of the same store.
- Fix the always-true replay flush condition so batching is real
(size/timeout based, one semaphore permit per batch) rather than one
permit per entry; dedup keys already pending in the batch.
- clear_and_close aggregates and reports per-target close failures instead
of swallowing them; explicit shutdown surfaces them.
Relates to rustfs/backlog#966
Relates to rustfs/backlog#967
Relates to rustfs/backlog#975
Relates to rustfs/backlog#970
Relates to rustfs/backlog#983
Co-Authored-By: heihutu <heihutu@gmail.com>
* style(targets): apply rustfmt to replay batch dedup guard
Fixes the Quick Checks rustfmt failure on the multi-line `.iter().any(...)`
closure in the replay batch dedup guard.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Address a batch of correctness/security audit findings in the messaging
notification backends (mqtt/nats/kafka/amqp/redis/pulsar/webhook).
MQTT (backlog#971): wait for broker acknowledgement via publish_tracked +
wait_completion_async (PUBACK/PUBCOMP for QoS>=1, flush for QoS0) with a
30s timeout before reporting success, instead of treating the enqueue as
delivered. Replace substring error matching with typed ClientError /
PublishNoticeError classification.
NATS (backlog#971, #973, #983): flush() after publish to confirm the broker
received the message before the durable copy is deleted; classify publish
and flush failures by typed error kind; warn when credentials are sent
without TLS.
Kafka (backlog#973, #983): use Error::is_retriable() so transient broker
states (NotLeaderForPartition, LeaderNotAvailable, RequestTimedOut, ...) are
retried instead of dropped as permanent; add a bucket/object message key for
per-object partition ordering; build the producer without holding the cache
lock across the connect await.
AMQP (backlog#973, #980): classify permanent broker protocol errors
(404/403/406, NOT_ALLOWED, ...) as request-level errors rather than
connectivity errors to avoid reconnect storms; bound publish and
publisher-confirm waits with a timeout; check is_enabled in is_active; run
full close() cleanup even when the broker close fails; warn when
mandatory=false may silently drop unroutable messages.
Redis (backlog#982): warn when PUBLISH reaches 0 subscribers; reuse the
cached ConnectionManager for health probes instead of a fresh handshake;
warn when tls_allow_insecure disables certificate verification.
Pulsar (backlog#983): replace std::sync::Mutex + unwrap with parking_lot
Mutex so a panic while holding the guard cannot poison later accesses.
Webhook (backlog#983): drain the response body so the connection can be
reused (keep-alive). The #974 redirect-follow SSRF fix already landed.
Relates to rustfs/backlog#971
Relates to rustfs/backlog#973
Relates to rustfs/backlog#974
Relates to rustfs/backlog#980
Relates to rustfs/backlog#982
Relates to rustfs/backlog#983
Co-authored-by: heihutu <heihutu@gmail.com>
Addresses security/correctness audit findings in the target-plugin control
plane, TLS reload coordinator, and runtime extension registries.
control_plane (backlog#977):
- Install now preserves the currently installed revision as previous_revision
so Rollback actually restores the prior version instead of a no-op.
- Split the gate: circuit-breaker and runtime-activation checks apply only to
Install/Enable; Disable and Rollback stay available as break-glass
remediation while the breaker is open.
- Enforce the sidecar runtime protocol version for every external transport
(previously skippable by declaring a non-gRPC transport) and validate the
plugin api_compatibility_version at planning time.
- Download/signature/provenance host allowlisting now matches the full
host authority, so an allowlisted host never implicitly authorizes a
different host:port.
TLS reload coordinator/validate (backlog#981, #970-coordinator):
- Compute the fingerprint before building material in register() to remove the
TOCTOU that could permanently pin an old certificate; rotation self-heals.
- Always start a detection loop when reload is enabled; Watch mode no longer
returns success without any loop.
- validate_cert_key_pairing now verifies the private key matches the
certificate's public key instead of only parsing both files.
- Normalize a zero poll interval to a positive minimum to avoid a panic that
silently killed the poll loop.
- Serialize reload cycles with a per-target mutex; a first-step fingerprint
read failure now records last_error and a failure metric.
- Duplicate label registration stops and joins the previous loop before
publishing the replacement (stop-before-start), preventing orphaned loops.
runtime extension points (backlog#983 runtime subset):
- ops_profiler/ops_diagnostics authorize before probing the registry so an
unauthorized caller cannot learn whether a backend/surface exists.
- s3_hooks dispatch_post_auth actually traverses the registered hooks for the
point instead of unconditionally returning Continue.
- sidecar send_with_timeout redacts errors and uses the configurable failure
threshold via policy, and a successful send resets the breaker accounting.
Relates to rustfs/backlog#977
Relates to rustfs/backlog#981
Relates to rustfs/backlog#970
Relates to rustfs/backlog#983
Co-authored-by: heihutu <heihutu@gmail.com>
extract_params_header copied every request/response header verbatim into
the maps that feed audit entries (requestQuery/responseHeader) and
notification events (req_params). Sensitive headers such as
Authorization and X-Amz-Security-Token were serialized in plaintext and
forwarded to external sinks (webhook/kafka/file), leaking long-lived
credentials.
Redact credential-bearing headers at this single chokepoint: the header
name is kept for correlation, but its value is replaced with the shared
REDACTED_SECRET placeholder. A case-insensitive sensitive-header list
covers authorization, x-amz-security-token, x-amz-content-sha256,
cookie, and set-cookie. Non-sensitive headers keep their existing
behavior.
Refs: https://github.com/rustfs/backlog/issues/963
The MQTT target spawned its rumqttc event loop exactly once through a
`OnceCell`. When the loop exited on a fatal protocol error, the finished
`JoinHandle` stayed in the cell, so `init()` never respawned it. The target
then went permanently silent: `publish` calls kept enqueuing but nothing was
ever delivered, while the target still looked healthy.
Replace the one-shot spawn with a supervisor task. The `OnceCell` now guards
a single supervisor that runs one event-loop session at a time and, when a
session exits, rebuilds the client and event loop from the latest
`MqttOptions` after an exponential backoff (1s..30s). A session that connected
resets the backoff so a transient drop reconnects promptly; repeated immediate
failures back off to avoid a reconnect storm. Cancellation from `close()` is
handled by the supervisor dropping the in-flight session future, so no
per-session cancel channel is needed.
Rebuilding options per session also keeps TLS hot-reload working across
reconnects. The backoff policy is a pure function and the supervisor loop is
generic over the session runner, so both are unit tested without a broker:
one test asserts the session restarts repeatedly until cancelled, another that
a live session stops promptly on cancel.
Refs: https://github.com/rustfs/backlog/issues/972
fix(targets): disable HTTP redirects on webhook delivery client (backlog#974)
The webhook reqwest client used the default redirect policy (follows up to
10 redirects). Even though the configured endpoint is validated against
outbound-egress rules, a malicious or compromised endpoint could return a
3xx response to bounce the outbound request to an internal address (e.g. the
cloud metadata service at 169.254.169.254), bypassing that validation (SSRF).
- Set `.redirect(reqwest::redirect::Policy::none())` on the delivery client
so redirects are never followed.
- Treat a 3xx response as a delivery failure (error carries the status code)
instead of silently succeeding.
- Add a regression test that stands up a loopback server returning a 302 to
an internal metadata address and asserts the client surfaces the 3xx rather
than following it.
Refs: https://github.com/rustfs/backlog/issues/974
fix(targets): surface truncated store batches instead of silently returning partial entries (backlog#967)
get_multiple deserializes a persisted batch by pulling item_count items
from a concatenated JSON stream. When the stream ended early (a truncated
or corrupted batch file), the partial-read branch only logged a warn! and
broke out of the loop, returning Ok(partial). Callers then treated the
batch as fully delivered, deleted the store entry, and permanently lost
the remaining events with no error.
Return StoreError::Deserialization as soon as the stream ends before
item_count items are read, so the caller keeps the entry on disk and can
retry rather than silently dropping events. The empty-file case already
returned an error and remains covered by this path.
Add a regression test that writes a 3-item batch, truncates the file at a
clean boundary after two items, and asserts get_multiple returns Err and
leaves the batch file in place.
Refs: https://github.com/rustfs/backlog/issues/967
* feat(targets): support Kafka SASL auth
* fix(targets): infer Kafka SASL when enable is blank
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
* chore(deps): refresh workspace deps and linux fs_type gating
- refresh workspace dependency pins and lockfile updates
- remove now-unused crate dependency entries in multiple Cargo.toml files
- enable profiling export defaults in config and scripts/run.sh
- gate os::fs_type module/function/tests to Linux to avoid non-Linux dead_code warnings
* fix(utils): simplify fs_type linux gating
- keep fs_type module-level linux cfg in os::mod
- remove redundant linux cfg on get_fs_type and test module
* chore(deps): bump s3s git revision
- update workspace s3s dependency to rev 507e1312b211c3ddc214b03875d6fabd15d22ed5
- refresh Cargo.lock source entry for s3s
* chore(dev): allow mysql_async git source and env overrides
- allow mysql_async git source in deny.toml allow-git list
- make scripts/run.sh core env vars overrideable via existing shell env
* fix(utils): import get_fs_type in fs_type tests
- add explicit super::get_fs_type import in fs_type test module
- fix Linux E0425 unresolved function errors in unit tests
* chore(dev): tune run script observability defaults
- make profiling export env overrideable in scripts/run.sh
- set RUSTFS_OBS_SAMPLE_RATIO default from 2.0 to 1.0
- update allow-git review window comments in deny.toml
* test(obs): stabilize profiling env alias tests
cxymds