* feat(ecstore): add object lock diagnostics
Add configurable namespace lock diagnostics for object operations so production contention can be traced by operation, owner, and key.
Wrap object read/write lock acquisition in diagnostic guards across get, head, put, delete, copy, and multipart flows, and log slow acquisition and long hold durations behind new RUSTFS_OBJECT_LOCK_DIAG_* settings.
Verification:
- make pre-commit
* feat(obs): expose object lock diagnostics metrics
Add Prometheus metrics and Grafana panels for object namespace lock diagnostics, covering slow acquire counts, slow hold counts, acquire duration, hold duration, and the diagnostics-enabled state.
Adopt PR review feedback by keeping diagnostic guards alive through the guarded operation so long-hold warnings and metrics are emitted, reusing shared env helpers, and reducing default-path overhead when diagnostics are disabled.
Verification:
- cargo check -p rustfs-ecstore -p rustfs-io-metrics
- cargo test -p rustfs-ecstore store::object -- --nocapture
- make pre-commit
* perf(obs): reduce object lock diag overhead
Avoid repeated environment parsing on hot object-lock paths by caching the diagnostics-enabled flag, and stop allocating label strings for object lock metrics by recording static labels directly.
Strengthen io-metrics tests by using a local recorder and asserting that the expected object lock diagnostic counters, gauges, and histograms are emitted.
Verification:
- cargo check -p rustfs-ecstore -p rustfs-io-metrics
- cargo test -p rustfs-io-metrics -- --nocapture
- make pre-commit
* feat(admin): restore config admin compatibility
Co-authored-by: weisd <im@weisd.in>
* fix(admin): align config admin clean rebuild
Co-authored-by: weisd <im@weisd.in>
* fix(admin): align config history and peer signals
* fix(admin): harden config admin mutations
* fix(admin): tighten config review follow-ups
* perf(admin): reuse env snapshot in config render
* fix(ecstore): clean up config admin and listing error handling
Remove redundant is_all_volume_not_found check in list_merged, add
storage class encode/decode roundtrip tests, fresh boot integration
test, and config admin clean rebuild improvements.
Co-authored-by: hehutu <heihutu@gmail.com>
* fix(admin): sync global server config on mutation and reload
Change GLOBAL_SERVER_CONFIG from OnceLock to RwLock so config mutations
(set/del/restore/reload) are visible to readers without restart. Call
set_global_server_config after every store save and on snapshot reload.
Register storage_class as a dynamic config subsystem.
Co-authored-by: hehutu <heihutu@gmail.com>
* style: apply rustfmt to config and admin tests
Co-authored-by: hehutu <heihutu@gmail.com>
* fix(test): update signal_service test for storage_class dynamic subsystem
storage_class is now a valid dynamic config subsystem, so the
"requires object layer" test should expect "storage layer not initialized"
instead of "unsupported dynamic config subsystem".
Co-authored-by: hehutu <heihutu@gmail.com>
* fix(config): publish storage_class runtime config on dynamic reload
Change GLOBAL_STORAGE_CLASS from OnceLock to RwLock so runtime updates
are possible. apply_storage_class_runtime_config now actually publishes
the parsed config via set_global_storage_class instead of dropping it.
Addresses review feedback: storage_class was marked as dynamically
applied but the parsed result was discarded, so mc admin config set
returned config_applied=true while the runtime kept using stale parity
settings until restart.
Co-authored-by: hehutu <heihutu@gmail.com>
* fix(admin): harden config init, history ordering, and env redaction
- Change GLOBAL_SERVER_CONFIG from RwLock<Config> to RwLock<Option<Config>>
initialized with None, preserving "not initialized" detection via None
- Move save_server_config_history before save_server_config_to_store in
SetConfigKVHandler, DelConfigKVHandler, and SetConfigHandler so a
restore point exists before mutations are persisted
- Redact sensitive env override values with *redacted* instead of
silently omitting the line, improving admin visibility
- Add code comment explaining VolumeNotFound removal rationale in
list_merged for listing paths
Co-authored-by: hehutu <heihutu@gmail.com>
* fix(config): keep in-memory config in sync after set/restore/reload
GLOBAL_SERVER_CONFIG was a OnceLock set once at startup and never
updated. After mc admin config set writes to the store, any fallback
to get_global_server_config() returned stale init-time data. Similarly,
reload_runtime_config_snapshot read from the store but discarded the
result.
- Replace OnceLock with RwLock for GLOBAL_SERVER_CONFIG and
GLOBAL_STORAGE_CLASS so they can be updated at runtime
- Add set_global_server_config / set_global_storage_class setters
- Call set_global_server_config after every config save (set-kv,
del-kv, set-config, restore-history)
- Re-apply dynamic subsystems (storage_class, audit_webhook,
audit_mqtt) and signal peers in reload_runtime_config_snapshot
and full-config operations
- Fix render_selected_config scope boundary check: track per-scope
line count instead of checking global lines.is_empty()
- Include STORAGE_CLASS_SUB_SYS in is_dynamic_config_subsystem so
apply_storage_class_runtime_config is reachable
Co-authored-by: hehutu <heihutu@gmail.com>
* fix(storageclass): use CLASS_RRS key in lookup_config for RRS parity
lookup_config used kvs.get(RRS) where RRS="REDUCED_REDUNDANCY", but the
admin config path writes the key as CLASS_RRS="rrs". This caused RRS
values to never be read back, always falling back to default parity.
- Changed kvs.get(RRS) to kvs.get(CLASS_RRS) in lookup_config
- Added regression tests verifying RRS read/write consistency
Co-authored-by: hehutu <heihutu@gmail.com>
* fix(config): add peer-side logging and don't swallow apply errors
- Add tracing::warn! in reload_dynamic_config_runtime_state and
reload_runtime_config_snapshot when config read or subsystem apply
fails, so on-host diagnostics show which signal failed and why
- Change `let _ = apply_dynamic_config_for_subsystem(...)` to
`if let Err(err) = ... { warn!(...) }` in reload_runtime_config_snapshot
so per-subsystem failures are logged instead of silently swallowed
- Remove weak test global_server_config_returns_none_before_init that
had no meaningful assertion due to shared global state
Co-authored-by: hehutu <heihutu@gmail.com>
* style: apply rustfmt to config and storageclass tests
Co-authored-by: hehutu <heihutu@gmail.com>
---------
Co-authored-by: weisd <im@weisd.in>
Co-authored-by: hehutu <heihutu@gmail.com>
Henry Guo