Land the remaining notify-crate audit fixes.
backlog#979(b): remove_target now enforces the same bucket-binding guard as
remove_target_config, refusing to delete a target still referenced by a bucket
rule so notification rules are not left orphaned.
backlog#984:
- event.rs: an unversioned object omits versionId entirely instead of
serializing versionId:"" (empty object/request versions treated as "no
version").
- notifier.rs: RUSTFS_NOTIFY_SEND_CONCURRENCY=0 coerces back to the default
instead of building a zero-permit semaphore that deadlocks every dispatch;
init_bucket_targets_shared closes the replaced targets instead of dropping
them without close() (connection leak).
- subscriber_index.rs: store_snapshot uses an atomic compute_if_absent upsert,
removing the get-then-insert TOCTOU that could clobber a concurrent
first-writer's snapshot cell.
- pipeline.rs: send_event assigns the history sequence and broadcasts to live
subscribers under one critical section so broadcast order matches recorded
sequence order.
- xml_config.rs: filter value length is bounded by character count, not byte
length, so valid multi-byte keys are no longer wrongly rejected.
- global.rs: a losing initialize() race shuts the just-initialized system down
instead of leaking its targets/replay workers.
backlog#970 (notify part): reload_config stops the running replay workers
before activating the new ones, so old and new workers do not concurrently
drain the same persisted stores. The full signal+join shutdown lives in the
targets crate under the same issue.
Tests: added regression coverage for each fix.
cargo build -p rustfs-notify, cargo test -p rustfs-notify --lib (98 passed),
cargo clippy -p rustfs-notify --all-targets (clean).
Relates to rustfs/backlog#979
Relates to rustfs/backlog#984
Relates to rustfs/backlog#970
Co-authored-by: heihutu <heihutu@gmail.com>
fix(notify): serialize persisted config read-modify-write to prevent lost updates (backlog#968)
The notify config write path performed a read -> modify -> write over the
persisted server config with no serialization: two concurrent updates could
both read the same base config, apply disjoint changes, and race their
full-config writes. The later write silently overwrote the earlier one,
losing updates (e.g. adding two targets concurrently could drop one) with no
error surfaced.
Guard the whole read -> modify -> write with a process-global tokio Mutex
(NOTIFY_CONFIG_RMW_LOCK). The persisted config is a single process-global
resource, so serializing the RMW makes concurrent updates apply in sequence
and every change is preserved. The lock is only acquired inside
update_server_config and is released before the in-memory reload, so it never
nests with the per-manager config RwLock and introduces no lock-ordering risk.
The RMW sequence is extracted into serialized_read_modify_write with injected
read/save operations so the exact production serialization path is exercised
by a regression test that concurrently adds 32 distinct targets and asserts
all of them survive.
Refs: https://github.com/rustfs/backlog/issues/968
cxymds