Commit Graph

10 Commits

Author SHA1 Message Date
houseme 035ce5d784 feat(obs): add bounded metrics dimensions (#5645)
* feat(obs): add drive topology detail metrics

Expose additive drive info, topology, state, and per-drive API metrics while preserving the existing drive metric label sets.

Backlog: rustfs/backlog#1655

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(obs): preserve suspect drive runtime state

Keep suspect as a bounded drive runtime state and avoid all-zero runtime_state samples for that storage health state.

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(obs): skip unknown drive inode samples

Avoid exporting zero inode gauges for missing or stale drive snapshots and ignore zero-count API latency buckets.

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(obs): add scanner source work detail metrics

Expose additive scanner source and cycle work metrics with bounded server/source/state labels while leaving the existing aggregate scanner metrics unchanged.

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(obs): add ilm action detail metrics

Expose additive ILM action/state task metrics with a server label while preserving the existing aggregate ILM series.

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(obs): add delivery target server metrics

Expose additive audit and notification delivery target metrics with server labels and extend removed-target tombstones for the server-aware series.

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(obs): add replication target flow metrics

Expose additive bucket replication target sent and failed-flow metrics while preserving existing bucket aggregates and target backlog series.

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(obs): add request server metrics

Expose additive API request metrics with server labels while preserving the existing request and traffic metric label sets.

Co-Authored-By: heihutu <heihutu@gmail.com>

* style(obs): apply rustfmt to metrics changes

Apply rustfmt output to the metrics dimension changes without altering behavior.

Co-Authored-By: heihutu <heihutu@gmail.com>

* style(obs): reuse audit target label constant

Use the exported audit target_id label constant for legacy audit target metrics.

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(obs): populate drive disk metrics

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(obs): add scanner bucket drive result metrics

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(obs): add replication proxy server metrics

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(obs): address metric liveness review

Use checked division for drive API latency aggregation and keep recovered drive, scanner current-cycle, replication flow, audit target, and notification target series from retaining stale values.

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(obs): address metric dimension review

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(obs): address additional metric review

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(obs): count drive calls at start

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(obs): address metrics dimension review

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(metrics): address dimension review gaps

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(metrics): address scanner review follow-ups

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(metrics): address runtime review follow-ups

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(metrics): reduce disk metric contention

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(metrics): address runtime review follow-ups

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(metrics): retire stale dimension series

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-03 09:03:34 +08:00
Henry Guo d5df66ac4f fix(scanner): require authoritative usage snapshots (#5333) 2026-07-28 07:29:57 +08:00
houseme cb977c4c5a fix(admin): report unreachable info members as unknown with balanced drive counts (#4607)
fix(admin): report unreachable info members as `unknown` with balanced drive counts

`GET /rustfs/admin/v3/info` synthesizes an entry for every peer, but a peer
whose properties RPC failed below the offline threshold with no cached snapshot
was reported as `initializing` with an empty drive list. That drive list being
empty meant its drives disappeared from the backend summary entirely, so
`onlineDisks + offlineDisks` no longer equaled `totalDrivesPerSet` (an operator
sees `onlineDisks: 3, offlineDisks: 0` for a 4-drive set), and `initializing` is
a misleading state for a member that has been running for hours — it reads like
a node was ejected when nothing is wrong (upstream rustfs/rustfs#4566).

Changes:
- madmin: add `ItemState::Unknown` ("unknown") and an `unknownDisks` bucket on
  `ErasureBackend` (serde `default`, so older payloads still deserialize).
- ecstore diagnostics: `get_online_offline_disks_stats` now returns a third
  `unknown` bucket instead of folding those drives into `offline`, keeping
  `online + offline + unknown == totalDrivesPerSet`.
- ecstore notification_sys: a probe miss below the failure threshold with no
  cache is now reported as `unknown` (not `initializing`) and carries the host's
  drives from the pool topology (tagged `unknown`) so the summary stays balanced;
  drive synthesis is factored into `synthesized_disks(host, endpoints, state)`,
  shared by the offline and unknown paths.

Tracked in rustfs/backlog#1049 (P1-A + P0-A).

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-09 04:50:20 +00:00
cxymds ddd35badad fix(ecstore): avoid offline disks on admin timeout (#3263)
* fix(ecstore): avoid offline disks on admin timeout

* fix(ecstore): handle poisoned admin cache locks

* fix(ecstore): recover poisoned admin cache on success

---------

Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
2026-06-08 08:35:36 +00:00
houseme c90bfe2b23 fix(ecstore): harden runtime read-path quorum handling (#2872) 2026-05-08 09:56:39 +00:00
houseme 3dd0692917 refactor: unify credential env constants and deploy env usage (#2821)
Co-authored-by: Henry Guo <marshawcoco@gmail.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
2026-05-06 04:30:29 +00:00
houseme 1e9c75a201 feat(ecstore): enforce local disk topology guardrails and expose device ids (#2679) 2026-04-25 06:02:02 +00:00
houseme b7a945e453 fix(madmin): restore server_info msgpack compatibility across mixed nodes (#2621) 2026-04-21 01:09:25 +00:00
houseme 28edfd6190 fix(storage): harden offline drive fail-fast paths (#2564)
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
2026-04-16 09:21:45 +00:00
houseme 5826396cd0 refactor: Restructure project layout and clean up dependencies (#30)
This commit introduces a significant reorganization of the project structure to improve maintainability and clarity.

Key changes include:
- Adjusted the directory layout for a more logical module organization.
- Removed unused crate dependencies, reducing the overall project size and potentially speeding up build times.
- Updated import paths and configuration files to reflect the structural changes.
2025-07-02 19:33:12 +08:00