* chore(deps): remove redundant dependency features
Remove manifest feature entries that are implied by other requested features in the same dependency declaration.
Verified that the resolved Cargo feature graph is unchanged after the cleanup.
Co-Authored-By: heihutu <heihutu@gmail.com>
* chore(deps): narrow tokio and reqwest features
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Move workspace-level dependency feature lists into the member crates that consume each dependency while keeping required default-features flags at the workspace root.
Also refresh starshard to 2.2.2 via cargo update and cargo upgrade --exclude ratelimit.
Co-authored-by: heihutu <heihutu@gmail.com>
feat(targets): add an opt-in NATS JetStream publish path for the notify and audit targets
The NATS notify and audit targets publish through NATS Core, which returns
before the server has durably accepted the message. A broker restart or a
connection drop between the publish and the flush loses the event, even though
the send queue has already cleared it, and no acknowledgement gates that clear.
An opt-in JetStream publish path clears a queued event only after the server
returns a durable PublishAck, so delivery is at-least-once across a broker
restart or a reconnect. It applies to both the notify and audit NATS targets, is
off by default, and is byte-identical to the NATS Core path when disabled.
The path includes durable store-and-forward, a stable dedup id sent as the
Nats-Msg-Id header so a replayed event is collapsed by the stream duplicate
window, pre-flight stream validation, and a bounded failed-events store for
terminally-failed and retry-exhausted events. Three configuration keys per
target select it: JETSTREAM_ENABLE, JETSTREAM_STREAM_NAME, and
JETSTREAM_ACK_TIMEOUT_SECS, under the RUSTFS_NOTIFY_NATS_ and RUSTFS_AUDIT_NATS_
prefixes.
The on-disk batch filename separator changes from colon to underscore so
batch names are valid on Windows filesystems, with transparent read-back
of files written under the previous separator. The migration affects the
shared queue store for every target type and lands with this feature
because the store gains its first Windows-exercised paths here.
Co-authored-by: houseme <housemecn@gmail.com>
Follow-up hardening for the audit control plane. The ABBA deadlock
(backlog#961), dispatch failure propagation (backlog#962), and credential
header redaction (backlog#963) already landed on main; this change addresses
the remaining audit-side findings.
- backlog#970: reload/commit now shuts down existing replay workers and closes
old targets *before* activating the replacement set, so old and new workers
never drain the same store concurrently and re-deliver entries. Extracted a
state-neutral `shutdown_runtime_targets` helper (registry-then-cancellers
lock order preserved).
- backlog#978: `start()` claims the `Starting` transition atomically under the
state lock, closing the check-then-act race that could double-activate;
`dispatch()` no longer returns Ok while paused, it surfaces an explicit
`AuditError::Paused` so the audit trail is not silently corrupted.
- backlog#984 (audit part): `dispatch_audit_log` performs a single state read
and interprets not-running/paused as a deliberate skip while surfacing real
delivery failures; `dispatch_batch` records the same observability signals as
single dispatch; documented the unordered cross-target fan-out.
Adds unit tests: paused dispatch returns Err, concurrent start does not hang or
double-activate, commit closes old targets before installing new, and
dispatch/dispatch_batch delivery-outcome coverage (all-fail -> Err, partial ->
Ok, all-success -> Ok).
Relates to rustfs/backlog#970
Relates to rustfs/backlog#978
Relates to rustfs/backlog#984
Co-authored-by: heihutu <heihutu@gmail.com>
fix(audit): propagate dispatch delivery failures instead of swallowing them (backlog#962)
AuditPipeline::dispatch and dispatch_batch accumulated per-target save()
errors, logged them, and then unconditionally returned Ok(()). When every
configured target failed the audit entry was lost outright (for store-backed
targets a failed save() means the event was neither delivered nor persisted
for replay), yet callers such as dispatch_audit_log saw success and assumed
the audit trail was intact. Audit is a compliance-critical path, so a total
delivery failure that reports success is a silent data-loss bug.
Both methods now distinguish three outcomes: all targets succeeded (Ok),
partial failure where at least one target accepted the event (log a warning
and return Ok, since the entry is not lost), and total failure where no
delivery succeeded while errors were recorded (record the failure metric,
log an error, and return Err(AuditError::Target) carrying the first target
error). This lets the caller react instead of assuming success.
Adds regression tests with a FailingTarget mock asserting that dispatch and
dispatch_batch return Err on total failure and Ok on partial failure.
* fix(audit): resolve AB-BA lock-order deadlock between registry and stream_cancellers (backlog#961)
`AuditSystem::runtime_status_snapshot` acquired `stream_cancellers.read()`
before `registry.lock()`, while every other path that holds both locks
(`clear_runtime_targets`, `AuditRuntimeFacade::replace_targets`) acquires
`registry` first. Under concurrency (e.g. admin status polling racing a
config reload) this AB-BA ordering could deadlock the whole audit control
plane with no self-recovery.
Reorder `runtime_status_snapshot` to acquire `registry` before
`stream_cancellers`, unifying the global lock order to
`registry -> stream_cancellers` across all double-lock paths, and add a
multi-threaded regression test that hammers both critical sections with a
timeout guard to fail fast if the ordering regresses.
Refs: https://github.com/rustfs/backlog/issues/961
* docs(audit): reword AB-BA to ABBA to satisfy typos check (backlog#961)
* chore(deps): refresh workspace deps and linux fs_type gating
- refresh workspace dependency pins and lockfile updates
- remove now-unused crate dependency entries in multiple Cargo.toml files
- enable profiling export defaults in config and scripts/run.sh
- gate os::fs_type module/function/tests to Linux to avoid non-Linux dead_code warnings
* fix(utils): simplify fs_type linux gating
- keep fs_type module-level linux cfg in os::mod
- remove redundant linux cfg on get_fs_type and test module
* chore(deps): bump s3s git revision
- update workspace s3s dependency to rev 507e1312b211c3ddc214b03875d6fabd15d22ed5
- refresh Cargo.lock source entry for s3s
* chore(dev): allow mysql_async git source and env overrides
- allow mysql_async git source in deny.toml allow-git list
- make scripts/run.sh core env vars overrideable via existing shell env
* fix(utils): import get_fs_type in fs_type tests
- add explicit super::get_fs_type import in fs_type test module
- fix Linux E0425 unresolved function errors in unit tests
* chore(dev): tune run script observability defaults
- make profiling export env overrideable in scripts/run.sh
- set RUSTFS_OBS_SAMPLE_RATIO default from 2.0 to 1.0
- update allow-git review window comments in deny.toml
* test(obs): stabilize profiling env alias tests
Avoid setting AuditSystemState::Starting when target list is empty.
Now checks target availability before state transition, keeping the
system in Stopped state if no enabled targets are found.
- Check targets.is_empty() before setting Starting state
- Return early with Ok(()) when no targets exist
- Maintain consistent state machine behavior
- Prevent transient "Starting" state with no actual targets
Resolves issue where audit system would incorrectly enter Starting
state even when configuration contained no enabled targets.
* improve code for audit
* improve code ecfs.rs
* improve code
* improve code for ecfs.rs
* feat(storage): refactor audit and notification with OperationHelper
This commit introduces a significant refactoring of the audit logging and event notification mechanisms within `ecfs.rs`.
The core of this change is the new `OperationHelper` struct, which encapsulates and simplifies the logic for both concerns. It replaces the previous `AuditHelper` and manual event dispatching.
Key improvements include:
- **Unified Handling**: `OperationHelper` manages both audit and notification builders, providing a single, consistent entry point for S3 operations.
- **RAII for Automation**: By leveraging the `Drop` trait, the helper automatically dispatches logs and notifications when it goes out of scope. This simplifies S3 method implementations and ensures cleanup even on early returns.
- **Fluent API**: A builder-like pattern with methods such as `.object()`, `.version_id()`, and `.suppress_event()` makes the code more readable and expressive.
- **Context-Aware Logic**: The helper's `.complete()` method intelligently populates log details based on the operation's `S3Result` and only triggers notifications on success.
- **Modular Design**: All helper logic is now isolated in `rustfs/src/storage/helper.rs`, improving separation of concerns and making `ecfs.rs` cleaner.
This refactoring significantly enhances code clarity, reduces boilerplate, and improves the robustness of logging and notification handling across the storage layer.
* fix
* fix
* fix
* fix
* fix
* fix
* fix
* improve code for audit and notify
* fix
* fix
* fix
* improve code for metrics
* improve code for metrics
* fix
* fix
* Refactor telemetry initialization and environment functions ordering
- Reorder functions in envs.rs by type size (8-bit to 64-bit, signed before unsigned) and add missing variants like get_env_opt_u16.
- Optimize init_telemetry to support three modes: stdout logging (default error level with span tracing), file rolling logs (size-based with retention), and HTTP-based observability with sub-endpoints (trace, metric, log) falling back to unified endpoint.
- Fix stdout logging issue by retaining WorkerGuard in OtelGuard to prevent premature release of async writer threads.
- Enhance observability mode with HTTP protocol, compression, and proper resource management.
- Update OtelGuard to include tracing_guard for stdout and flexi_logger_handles for file logging.
- Improve error handling and configuration extraction in OtelConfig.
* fix
* up
* fix
* fix
* improve code for obs
* fix
* fix
cxymds