feat(storage): extend PUT path tuning and observability (#3829)

* feat(storage): add multipart put stage metrics * feat(scripts): add multipart put focus runner * docs(operations): add multipart put server-path guides * chore(scripts): add local rustfs restart helper * docs(observability): add local metrics backend guide * docs(observability): add localized multipart guides * fix(ecstore): validate multipart batching path * feat(obs): add erasure encode overlap metrics * docs(ops): update overlap retest summary * docs(ops): add batchblocks retest matrix * docs(ops): extend overlap candidate summary * docs(ops): capture 8-run overlap summary * feat(storage): switch rename_data to msgpack map * test(storage): add rename_data payload checks * feat(object): add zero_copy_eager put path * docs(ops): add zero_copy_eager put guide * docs(ops): add deeper zero-copy next steps
2026-08-09 14:49:25 +00:00 · 2026-06-25 19:24:35 +08:00
parent 3942186f3d
commit a30357d21e
24 changed files with 3037 additions and 35 deletions
@@ -0,0 +1,232 @@
+# Issue #712 Local Queryable Metrics Backend Guide
+
+## 1. Purpose
+
+This guide is intended to unblock the third validation batch for `#712` by making multipart PUT stage metrics queryable from a local backend.
+
+The current problem is not that multipart stage metrics are missing from the code path. The actual problem is that the local environment does not expose a queryable metrics backend:
+
+1. `rustfs/admin/v3/metrics` is available, but it returns an admin-side JSON snapshot rather than the `metrics` crate histogram series
+2. there is no local Prometheus or equivalent queryable metrics endpoint listening by default
+3. therefore the following stage labels cannot be queried directly yet:
+   - `multipart_ingress_prepare`
+   - `multipart_set_disk_writer_setup`
+   - `multipart_set_disk_encode`
+   - `multipart_complete_tail`
+
+The goal of this guide is to close that gap.
+
+## 2. Recommended approach
+
+Reuse the repository's existing observability stack:
+
+1. `.docker/observability/docker-compose.yml`
+
+Why this is the preferred path:
+
+1. it is already maintained in-repo
+2. it includes OTEL Collector, Prometheus, and Grafana
+3. it can receive telemetry from RustFS through `RUSTFS_OBS_ENDPOINT`
+
+## 3. Expected data flow
+
+After startup, the intended flow is:
+
+1. RustFS
+   - `RUSTFS_OBS_ENDPOINT=http://host.docker.internal:4318`
+2. OTEL Collector
+   - receives OTLP/HTTP telemetry
+3. Prometheus
+   - scrapes collector-exported metrics
+4. Query surface
+   - `http://127.0.0.1:9090`
+
+## 4. Startup steps
+
+### 4.1 Start the observability stack
+
+From the repository root:
+
+```bash
+cd .docker/observability
+docker compose up -d
+```
+
+### 4.2 Wait for core services
+
+Recommended checks:
+
+```bash
+curl -fsS http://127.0.0.1:9090/-/ready
+curl -fsS http://127.0.0.1:3000/api/health
+```
+
+If you need container status:
+
+```bash
+docker compose ps
+```
+
+### 4.3 Point RustFS to the OTEL Collector
+
+For local single-node multi-disk validation:
+
+```bash
+export RUSTFS_OBS_ENDPOINT=http://host.docker.internal:4318
+```
+
+If you use the repository-local restart helper:
+
+```bash
+bash scripts/restart_local_single_node_multidisk_rustfs.sh
+```
+
+Make sure the final runtime environment really contains:
+
+```bash
+RUSTFS_OBS_ENDPOINT=http://host.docker.internal:4318
+```
+
+## 5. Minimal query validation
+
+### 5.1 Confirm Prometheus can see RustFS metrics
+
+```bash
+curl -fsS 'http://127.0.0.1:9090/api/v1/query?query=rustfs_s3_put_object_total'
+```
+
+### 5.2 Confirm stage labels exist
+
+```bash
+curl -fsS 'http://127.0.0.1:9090/api/v1/label/stage/values'
+```
+
+If the pipeline is working, the result should include:
+
+1. `multipart_ingress_prepare`
+2. `multipart_set_disk_writer_setup`
+3. `multipart_set_disk_encode`
+4. `multipart_complete_tail`
+
+### 5.3 Direct P95 query
+
+```bash
+curl -fsS 'http://127.0.0.1:9090/api/v1/query?query=histogram_quantile(0.95,sum by(stage,le)(rate(rustfs_s3_put_object_stage_duration_ms_bucket{stage=~"multipart_.*"}[5m])))'
+```
+
+## 6. Recommended silent validation order
+
+Once the backend is queryable, use this order for the third multipart validation batch:
+
+1. start the observability stack
+2. restart RustFS and confirm `RUSTFS_OBS_ENDPOINT` is active
+3. run one multipart baseline:
+   - `1g-64m-pc4`
+   - `2g-128m-pc4`
+4. ignore streaming benchmark logs and only keep:
+   - `summary.csv`
+   - Prometheus query outputs
+5. summarize:
+   - throughput / reqps / average latency
+   - P95 / P99 for the four multipart stages
+
+## 7. Recommended query set
+
+### 7.1 Multipart stage P95
+
+```promql
+histogram_quantile(
+  0.95,
+  sum by (stage, le) (
+    rate(rustfs_s3_put_object_stage_duration_ms_bucket{stage=~"multipart_.*"}[5m])
+  )
+)
+```
+
+### 7.2 Multipart stage P99
+
+```promql
+histogram_quantile(
+  0.99,
+  sum by (stage, le) (
+    rate(rustfs_s3_put_object_stage_duration_ms_bucket{stage=~"multipart_.*"}[5m])
+  )
+)
+```
+
+### 7.3 Complete tail focus
+
+```promql
+histogram_quantile(
+  0.95,
+  sum by (instance, le) (
+    rate(rustfs_s3_put_object_stage_duration_ms_bucket{stage="multipart_complete_tail"}[5m])
+  )
+)
+```
+
+### 7.4 Encode focus
+
+```promql
+histogram_quantile(
+  0.95,
+  sum by (instance, le) (
+    rate(rustfs_s3_put_object_stage_duration_ms_bucket{stage="multipart_set_disk_encode"}[5m])
+  )
+)
+```
+
+## 8. Suggested result layout
+
+Recommended layout:
+
+```text
+target/bench/
+  issue712-multipart-server-path-focus/
+    summary.csv
+    metrics-query.txt
+    promql/
+      multipart-stage-p95.txt
+      multipart-stage-p99.txt
+      multipart-complete-tail-p95.txt
+      multipart-encode-p95.txt
+```
+
+## 9. Troubleshooting
+
+### 9.1 Prometheus does not start
+
+Check:
+
+```bash
+cd .docker/observability
+docker compose logs prometheus
+```
+
+### 9.2 RustFS does not export telemetry
+
+Check:
+
+1. `RUSTFS_OBS_ENDPOINT` really points to `http://host.docker.internal:4318`
+2. the OTEL collector container is running
+3. RustFS was restarted after the environment variable changed
+
+### 9.3 Stage labels are missing
+
+Check:
+
+1. a multipart PUT workload really ran
+2. `put_stage_metrics_enabled()` was enabled at runtime
+3. the query window is not too short
+
+## 10. Recommendation
+
+When `#712` third-batch validation resumes, do not run the benchmark first and hunt for metrics later.
+
+Use this order instead:
+
+1. bring up a queryable backend
+2. run the multipart baseline
+3. read only:
+   - `summary.csv`
+   - Prometheus stage query outputs
@@ -0,0 +1,232 @@
+# Issue #712 本地可查询 metrics backend 启动手册
+
+## 1. 目的
+
+本文用于打通 `#712` 第三批验证所需的本地可查询 metrics backend。
+
+当前问题不是 multipart stage 指标没有打点，而是本地环境里没有可查询后端：
+
+1. `rustfs/admin/v3/metrics` 可用，但返回的是管理侧 JSON 快照，不包含 `metrics` crate 的 histogram 指标
+2. 本地默认没有 Prometheus / 可查询 metrics endpoint 在监听
+3. 因此无法直接查询：
+   - `multipart_ingress_prepare`
+   - `multipart_set_disk_writer_setup`
+   - `multipart_set_disk_encode`
+   - `multipart_complete_tail`
+
+本文的目标就是把这条链打通。
+
+## 2. 推荐方案
+
+推荐直接复用仓库内已有的 observability stack：
+
+1. `.docker/observability/docker-compose.yml`
+
+这套 stack 的优点：
+
+1. 已经是仓库现成维护的方案
+2. 包含 OTEL Collector、Prometheus、Grafana
+3. 可以直接承接 RustFS 的 `RUSTFS_OBS_ENDPOINT`
+
+## 3. 核心链路
+
+启动后，链路应该是：
+
+1. RustFS
+   - `RUSTFS_OBS_ENDPOINT=http://host.docker.internal:4318`
+2. OTEL Collector
+   - 接收 OTLP/HTTP
+3. Prometheus
+   - 抓取 collector 暴露的 metrics
+4. 查询
+   - `http://127.0.0.1:9090`
+
+## 4. 启动步骤
+
+### 4.1 启动 observability stack
+
+在仓库根目录执行：
+
+```bash
+cd .docker/observability
+docker compose up -d
+```
+
+### 4.2 等待组件就绪
+
+建议检查：
+
+```bash
+curl -fsS http://127.0.0.1:9090/-/ready
+curl -fsS http://127.0.0.1:3000/api/health
+```
+
+若需要查看容器：
+
+```bash
+docker compose ps
+```
+
+### 4.3 启动 RustFS 时显式指向 OTEL Collector
+
+本地单机多盘场景建议：
+
+```bash
+export RUSTFS_OBS_ENDPOINT=http://host.docker.internal:4318
+```
+
+如果使用我们现成的本地重启脚本：
+
+```bash
+bash scripts/restart_local_single_node_multidisk_rustfs.sh
+```
+
+请确保脚本最终生效的环境里包含：
+
+```bash
+RUSTFS_OBS_ENDPOINT=http://host.docker.internal:4318
+```
+
+## 5. 最小查询验证
+
+### 5.1 确认 Prometheus 能查询到 RustFS 指标
+
+```bash
+curl -fsS 'http://127.0.0.1:9090/api/v1/query?query=rustfs_s3_put_object_total'
+```
+
+### 5.2 查询 stage 指标是否存在
+
+```bash
+curl -fsS 'http://127.0.0.1:9090/api/v1/label/stage/values'
+```
+
+如果链路打通，返回结果中应能看到：
+
+1. `multipart_ingress_prepare`
+2. `multipart_set_disk_writer_setup`
+3. `multipart_set_disk_encode`
+4. `multipart_complete_tail`
+
+### 5.3 直接查询 P95
+
+```bash
+curl -fsS 'http://127.0.0.1:9090/api/v1/query?query=histogram_quantile(0.95,sum by(stage,le)(rate(rustfs_s3_put_object_stage_duration_ms_bucket{stage=~"multipart_.*"}[5m])))'
+```
+
+## 6. 建议的静默验证顺序
+
+打通 backend 后，建议按如下顺序做第三批：
+
+1. 启动 observability stack
+2. 重启 RustFS 并确认 `RUSTFS_OBS_ENDPOINT` 生效
+3. 先跑一轮 multipart baseline：
+   - `1g-64m-pc4`
+   - `2g-128m-pc4`
+4. 不看过程日志，只保留：
+   - `summary.csv`
+   - Prometheus 查询结果
+5. 最后整理：
+   - throughput / reqps / avg latency
+   - 4 个 multipart stage 的 P95 / P99
+
+## 7. 推荐查询集合
+
+### 7.1 multipart 阶段 P95
+
+```promql
+histogram_quantile(
+  0.95,
+  sum by (stage, le) (
+    rate(rustfs_s3_put_object_stage_duration_ms_bucket{stage=~"multipart_.*"}[5m])
+  )
+)
+```
+
+### 7.2 multipart 阶段 P99
+
+```promql
+histogram_quantile(
+  0.99,
+  sum by (stage, le) (
+    rate(rustfs_s3_put_object_stage_duration_ms_bucket{stage=~"multipart_.*"}[5m])
+  )
+)
+```
+
+### 7.3 complete tail 单独看
+
+```promql
+histogram_quantile(
+  0.95,
+  sum by (instance, le) (
+    rate(rustfs_s3_put_object_stage_duration_ms_bucket{stage="multipart_complete_tail"}[5m])
+  )
+)
+```
+
+### 7.4 encode 单独看
+
+```promql
+histogram_quantile(
+  0.95,
+  sum by (instance, le) (
+    rate(rustfs_s3_put_object_stage_duration_ms_bucket{stage="multipart_set_disk_encode"}[5m])
+  )
+)
+```
+
+## 8. 结果目录建议
+
+推荐目录：
+
+```text
+target/bench/
+  issue712-multipart-server-path-focus/
+    summary.csv
+    metrics-query.txt
+    promql/
+      multipart-stage-p95.txt
+      multipart-stage-p99.txt
+      multipart-complete-tail-p95.txt
+      multipart-encode-p95.txt
+```
+
+## 9. 失败排查
+
+### 9.1 Prometheus 起不来
+
+检查：
+
+```bash
+cd .docker/observability
+docker compose logs prometheus
+```
+
+### 9.2 RustFS 没有上报
+
+检查：
+
+1. `RUSTFS_OBS_ENDPOINT` 是否真的是 `http://host.docker.internal:4318`
+2. OTEL collector 是否运行
+3. RustFS 重启后是否带上了新环境变量
+
+### 9.3 查不到 stage label
+
+检查：
+
+1. 是否真的跑过 multipart PUT 请求
+2. `put_stage_metrics_enabled()` 是否在运行期被开启
+3. 查询窗口是否太短
+
+## 10. 建议
+
+下次推进 `#712` 第三批时，不要先跑 benchmark，再临时找 metrics。
+
+正确顺序应是：
+
+1. 先按本文把可查询 backend 起好
+2. 再跑 baseline
+3. 最后只读：
+   - `summary.csv`
+   - Prometheus stage 查询结果
@@ -0,0 +1,263 @@
+# Issue #712 multipart PUT 分阶段指标 Dashboard / PromQL 指南
+
+## 1. 目的
+
+本文给 `#712` 的第一批 server-path 观测增强配套一份可执行的 Dashboard / PromQL 指南。
+
+本批次新增的 multipart 阶段指标依然复用现有指标名：
+
+1. `rustfs_s3_put_object_stage_duration_ms`
+
+但新增了四个 stage label：
+
+1. `multipart_ingress_prepare`
+2. `multipart_set_disk_writer_setup`
+3. `multipart_set_disk_encode`
+4. `multipart_complete_tail`
+
+## 2. 使用前提
+
+这些阶段指标严格受全局开关控制：
+
+1. `rustfs_io_metrics::put_stage_metrics_enabled() == true`
+
+如果该开关没有开启：
+
+1. 不会上报这些阶段指标
+2. 也不会额外做阶段计时
+
+## 3. 推荐直接复用现有 Grafana Row
+
+当前 Dashboard 中已经有：
+
+1. `Large PUT Stage Breakdown`
+
+这意味着：
+
+1. 不需要重新设计一套全新 row
+2. 只需要在现有 row / stage 变量里选新的 multipart stage label 即可
+
+## 4. 推荐 PromQL
+
+### 4.1 multipart 阶段 P95
+
+```promql
+histogram_quantile(
+  0.95,
+  sum by (stage, le) (
+    rate(
+      rustfs_s3_put_object_stage_duration_ms_bucket{
+        job=~"$job",
+        stage=~"multipart_.*"
+      }[$__rate_interval]
+    )
+  )
+)
+```
+
+### 4.2 multipart 阶段 P99
+
+```promql
+histogram_quantile(
+  0.99,
+  sum by (stage, le) (
+    rate(
+      rustfs_s3_put_object_stage_duration_ms_bucket{
+        job=~"$job",
+        stage=~"multipart_.*"
+      }[$__rate_interval]
+    )
+  )
+)
+```
+
+### 4.3 单实例 multipart 阶段 P95
+
+```promql
+histogram_quantile(
+  0.95,
+  sum by (instance, stage, le) (
+    rate(
+      rustfs_s3_put_object_stage_duration_ms_bucket{
+        job=~"$job",
+        instance=~"$instance",
+        stage=~"multipart_.*"
+      }[$__rate_interval]
+    )
+  )
+)
+```
+
+### 4.4 multipart 与 ordinary PUT encode 对比
+
+```promql
+histogram_quantile(
+  0.95,
+  sum by (stage, le) (
+    rate(
+      rustfs_s3_put_object_stage_duration_ms_bucket{
+        job=~"$job",
+        stage=~"set_disk_encode|multipart_set_disk_encode"
+      }[$__rate_interval]
+    )
+  )
+)
+```
+
+### 4.5 multipart complete tail 重点盯盘
+
+```promql
+histogram_quantile(
+  0.95,
+  sum by (instance, le) (
+    rate(
+      rustfs_s3_put_object_stage_duration_ms_bucket{
+        job=~"$job",
+        stage="multipart_complete_tail",
+        instance=~"$instance"
+      }[$__rate_interval]
+    )
+  )
+)
+```
+
+### 4.6 multipart path 命中计数
+
+```promql
+sum by (path) (
+  rustfs_s3_put_object_path_total{
+    path=~"multipart_.*"
+  }
+)
+```
+
+用于回答：
+
+1. 当前 run 是否真的命中了 `multipart_write_pipeline_batched_large`
+2. batched gate 是否只是“代码存在”，还是“运行时实际生效”
+
+### 4.7 erasure encode 内部阶段均值
+
+```promql
+sum by (stage) (
+  increase(
+    rustfs_internal_stage_duration_ms_sum{
+      stage=~"erasure_encode.*"
+    }[$__rate_interval]
+  )
+)
+/
+sum by (stage) (
+  increase(
+    rustfs_internal_stage_duration_ms_count{
+      stage=~"erasure_encode.*"
+    }[$__rate_interval]
+  )
+)
+```
+
+用于回答：
+
+1. `multipart_set_disk_encode` 内部到底更偏 CPU encode，还是更偏 writer write
+2. producer / consumer 之间是 encoder 在等 writer，还是 writer 在等 encoder
+
+### 4.8 erasure encode 当前累计 counters
+
+当窗口查询容易受到 scrape 周期影响时，可以直接看当前累计值：
+
+```promql
+rustfs_internal_stage_duration_ms_count{
+  stage=~"erasure_encode.*"
+}
+```
+
+```promql
+rustfs_internal_stage_duration_ms_sum{
+  stage=~"erasure_encode.*"
+}
+```
+
+这在 focused 单 profile 验证里很有用，尤其适合“重启实例后只跑一轮”的场景。
+
+## 5. 推荐看板顺序
+
+当你在看 `>1GiB multipart PUT` 时，建议按下面顺序看：
+
+1. `multipart_ingress_prepare`
+2. `multipart_set_disk_writer_setup`
+3. `multipart_set_disk_encode`
+4. `multipart_complete_tail`
+5. `multipart_write_pipeline` vs `multipart_write_pipeline_batched_large`
+6. `erasure_encode_*` / `erasure_encode_batched_*`
+
+解释顺序：
+
+1. 如果 ingress 先高，先看 part ingress buffer / request-body handling
+2. 如果 writer setup 高，先看 bitrot writer / disk availability / shard_file_size path
+3. 如果 encode 高，先看 multipart 是否需要独立 encode strategy
+4. 如果 complete tail 高，优先看 `complete_multipart_upload()` 的 metadata / checksum / rename tail
+5. 如果 batched 预期已打开，但 path 仍然只有 `multipart_write_pipeline`，优先检查 size gate 是否真正被命中
+6. 如果 `erasure_encode_batched_send_wait` 很低、但 `erasure_encode_batched_recv_wait` 明显更高，优先怀疑当前 batch barrier 让 writer 侧在等下一批 encode 完成
+
+## 6. 推荐结合看的辅助指标
+
+建议和上面四个阶段一起看：
+
+1. `rustfs_io_put_object_concurrent_requests`
+2. `rustfs_ec_encode_inflight_bytes_current`
+3. host CPU
+4. per-instance disk write throughput
+5. readiness / write quorum 异常计数
+
+## 7. 典型解释模板
+
+### 7.1 ingress 高
+
+可能原因：
+
+1. part body stream buffering 不合适
+2. `part.size` 与 ingress buffer 不匹配
+
+### 7.2 writer setup 高
+
+可能原因：
+
+1. bitrot writer 构建成本偏高
+2. online disk / writer init 慢
+3. shard_file_size 相关路径有额外成本
+
+### 7.3 encode 高
+
+可能原因：
+
+1. multipart part 仍然借用了 ordinary PUT encode 行为
+2. `part.size` 太大，单 part encode CPU 时间过长
+3. batching / inflight 参数不合适
+
+### 7.4 complete tail 高
+
+可能原因：
+
+1. complete 阶段 part metadata 处理放大
+2. checksum combine 成本高
+3. rename / cleanup / commit tail 成本高
+
+## 8. 建议的截图 / 归档内容
+
+每次 `>1GiB multipart PUT` 复测，建议固定归档：
+
+1. multipart stage P95 截图
+2. multipart stage P99 截图
+3. `multipart_complete_tail` 单实例截图
+4. CPU / disk write 辅助图
+
+## 9. 当前阶段建议
+
+下一次进入 `#712` 继续推进时：
+
+1. 先开 `put_stage_metrics_enabled`
+2. 先跑推荐 baseline：
+   - `1GiB -> 64MiB / pc4`
+   - `2GiB -> 128MiB / pc4`
+3. 先看 `multipart_complete_tail` 是否明显高于其他阶段
+4. 再决定是先改 ingress / encode / writer setup / complete tail