gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-08-22 20:36:38 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(kms): add observability dashboard, alert rules and runbook (#5517)

Browse Source 
Add a Grafana dashboard for the four KMS backend operation metrics
emitted at the policy choke point, Prometheus alert rules with
conservative default thresholds (pending staging baseline calibration),
and an operations runbook documenting the metric contract and the
response procedure for each alert. Register the new alert rules file in
the observability READMEs.

Refs rustfs/backlog#1584 (part of rustfs/backlog#1562)
This commit is contained in:
Zhengchao An
2026-08-01 05:28:10 +08:00
committed by GitHub
parent 74019845c4
commit 792f2ef204
6 changed files with 1133 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.docker/observability/README_ZH.md
+10
View File
@@ -60,6 +60,16 @@
| `CodecStreamingFallbackSpike` | 警告 | Codec streaming 回退 > 10x 基线，持续 10 分钟 |
| `IoQueueSaturation` | 警告 | IO 队列利用率 > 90%，持续 5 分钟 |
文件 `prometheus-rules/rustfs-kms-alerts.yml` 包含 KMS 后端操作指标的告警规则。阈值为保守默认值，待 staging 基线校准；响应流程见 `docs/operations/kms-observability-runbook.md`，配套仪表盘为 `deploy/observability/grafana/rustfs-kms-observability.json`。
| 告警 | 级别 | 条件 |
|------|------|------|
| `KmsBackendFatalErrors` | 严重 | fatal（不可重试）尝试失败 > 0，持续 5 分钟 |
| `KmsBackendHighErrorRate` | 严重 | 非 success 操作占比 > 5%，持续 10 分钟（含流量下限保护） |
| `KmsBackendP99LatencyHigh` | 警告 | 操作 p99 耗时（含重试）> 2s，持续 10 分钟 |
| `KmsBackendAttemptFailureSpike` | 警告 | 尝试失败率 > 0.5/s，持续 10 分钟 |
| `KmsBackendRetryBudgetExhausted` | 警告 | budget_exhausted / deadline_exceeded 结果 > 0.05/s，持续 10 分钟 |
### 启用告警规则
在 Prometheus 配置中添加告警规则文件：