mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-09 14:49:25 +00:00
feat(storage): optimize gt1g get read path (#3860)
* feat(storage): tune gt1g get read path * docs(ops): add gt1g get benchmark guide * feat(scripts): add gt1g get fallback runner * test(storage): trim gt1g get helper warnings * test(storage): reduce gt1g get helper type complexity
This commit is contained in:
houseme
@@ -0,0 +1,87 @@
|
||||
# Issue #713 `>1GiB GET` baseline v1 小结
|
||||
|
||||
## 1. 执行范围
|
||||
|
||||
本轮是 `#713` 的第一轮 plain baseline，先不碰 encrypted / compressed，只看：
|
||||
|
||||
1. `1GiB`
|
||||
2. `2GiB`
|
||||
3. `sequential`
|
||||
4. `ranged_parallel`
|
||||
5. `concurrency=1,4,8`
|
||||
6. `range_workers=4`
|
||||
7. `rounds=1`
|
||||
|
||||
结果目录：
|
||||
|
||||
1. `rustfs/target/bench/issue713-gt1g-get-baseline-v1-20260625T105149Z`
|
||||
|
||||
## 2. 结果总表
|
||||
|
||||
### 2.1 `1GiB`
|
||||
|
||||
`sequential`
|
||||
|
||||
1. `c1`: `99.36 MiB/s`, `10807.0ms`
|
||||
2. `c4`: `518.53 MiB/s`, `8267.5ms`
|
||||
3. `c8`: `708.04 MiB/s`, `11362.4ms`
|
||||
|
||||
`ranged_parallel`
|
||||
|
||||
1. `c1`: `481.71 MiB/s`, `2229.0ms`
|
||||
2. `c4`: `1536.66 MiB/s`, `2728.5ms`
|
||||
3. `c8`: `1548.29 MiB/s`, `5010.9ms`
|
||||
|
||||
### 2.2 `2GiB`
|
||||
|
||||
`sequential`
|
||||
|
||||
1. `c1`: `116.43 MiB/s`, `18444.0ms`
|
||||
2. `c4`: `522.06 MiB/s`, `16442.5ms`
|
||||
3. `c8`: `712.24 MiB/s`, `21842.1ms`
|
||||
|
||||
`ranged_parallel`
|
||||
|
||||
1. `c1`: `503.28 MiB/s`, `4267.0ms`
|
||||
2. `c4`: `1817.59 MiB/s`, `4707.3ms`
|
||||
3. `c8`: `1681.50 MiB/s`, `9502.1ms`
|
||||
|
||||
## 3. 当前第一结论
|
||||
|
||||
本轮 plain baseline 可以先收敛出三点：
|
||||
|
||||
1. `ranged_parallel` 在当前单机多盘场景下，明显优于单流 `sequential`
|
||||
2. 这种优势在 `1GiB` 和 `2GiB` 两个点上都成立
|
||||
3. 即使 `sequential` 已经过了当前这轮 `>1GiB GET` buffer / readahead 优化，当前仍然没有追平 `4-way ranged_parallel`
|
||||
|
||||
换句话说：
|
||||
|
||||
1. 这轮服务端优化是必要的
|
||||
2. 但对当前这组 plain `>1GiB GET` workload 来说，客户端并行 range 下载仍然是更强的模式
|
||||
|
||||
## 4. 当前推荐
|
||||
|
||||
在这轮 baseline 之后，最自然的下一步是：
|
||||
|
||||
1. 先补一轮更稳妥的确认性复测，只保留重点点位：
|
||||
- `1GiB c1`
|
||||
- `1GiB c4`
|
||||
- `2GiB c1`
|
||||
- `2GiB c4`
|
||||
2. 如果结论仍稳定，再继续补：
|
||||
- larger `range_workers` 对比
|
||||
- encrypted `>1GiB GET`
|
||||
- compressed `>1GiB GET`
|
||||
|
||||
## 5. 当前边界说明
|
||||
|
||||
这轮还不能直接下最终结论的地方：
|
||||
|
||||
1. 只有 `1 round`
|
||||
2. `ranged_parallel` 当前只测了 `range_workers=4`
|
||||
3. 还没有拆 plain / encrypted / compressed
|
||||
|
||||
因此当前更准确的表述是：
|
||||
|
||||
1. plain `>1GiB GET` 的第一轮 baseline 已显示 `ranged_parallel` 明显领先
|
||||
2. 但还需要更小面复测来确认这个结论的稳定性
|
||||
@@ -0,0 +1,258 @@
|
||||
# Issue #713 `>1GiB GET` 优化与验证手册
|
||||
|
||||
## 1. 目标
|
||||
|
||||
本文用于收口 `backlog#713` 当前这一轮 `>1GiB GET` 优化的实现点与验证方式。
|
||||
|
||||
当前这轮改动先解决三件事：
|
||||
|
||||
1. `sequential hint` 真正参与 GET I/O 策略计算
|
||||
2. storage media buffer cap 不再被统一的 `1MiB` hard clamp 二次截断
|
||||
3. `>1GiB`、非 range、允许 readahead 的 streaming GET，会走更大的 stream buffer 策略
|
||||
|
||||
## 2. 当前代码行为变化
|
||||
|
||||
### 2.1 多因素调度
|
||||
|
||||
涉及：
|
||||
|
||||
1. `rustfs/src/storage/concurrency/io_schedule.rs`
|
||||
|
||||
当前行为：
|
||||
|
||||
1. `IoSchedulingContext.is_sequential_hint=true` 时，会优先按 `Sequential` 模式参与 buffer / readahead 决策
|
||||
2. storage profile 提供的 media cap 现在可以真实生效
|
||||
3. 例如默认常量下：
|
||||
- `NVMe`: `2MiB`
|
||||
- `SSD`: `1MiB`
|
||||
- `HDD`: `512KiB`
|
||||
|
||||
### 2.2 GET body stream 策略
|
||||
|
||||
涉及：
|
||||
|
||||
1. `rustfs/src/app/object_usecase.rs`
|
||||
|
||||
当前行为：
|
||||
|
||||
1. `optimal_buffer_size` 不再被额外 `min(base_buffer_size)` 压回去
|
||||
2. `>1GiB`、非 range、`enable_readahead=true` 的 streaming GET，会命中：
|
||||
- `large_sequential_readahead`
|
||||
3. 该路径当前把 `ReaderStream` buffer 扩到：
|
||||
- `min(optimal_buffer_size * 2, 4MiB)`
|
||||
|
||||
### 2.3 GET observability
|
||||
|
||||
涉及：
|
||||
|
||||
1. `crates/io-metrics/src/lib.rs`
|
||||
|
||||
新增：
|
||||
|
||||
1. `rustfs_io_get_object_stream_strategy_total{strategy=...}`
|
||||
2. `rustfs_io_get_object_stream_buffer_size_bytes{strategy=...}`
|
||||
3. `rustfs_io_get_object_stream_response_size_bytes{strategy=...}`
|
||||
|
||||
当前策略标签：
|
||||
|
||||
1. `standard`
|
||||
2. `large_sequential_readahead`
|
||||
|
||||
## 3. 推荐验证顺序
|
||||
|
||||
按下面顺序做最稳妥：
|
||||
|
||||
1. 先准备 plain `1GiB` / `2GiB` 对象
|
||||
2. 先跑 `sequential` baseline
|
||||
3. 再跑 `ranged_parallel`
|
||||
4. 最后再决定是否继续补 encrypted / compressed
|
||||
|
||||
## 4. 准备测试对象
|
||||
|
||||
直接使用：
|
||||
|
||||
1. `scripts/prepare_gt1g_get_test_objects.sh`
|
||||
|
||||
示例：
|
||||
|
||||
```bash
|
||||
bash scripts/prepare_gt1g_get_test_objects.sh \
|
||||
--endpoint http://127.0.0.1:9000 \
|
||||
--access-key rustfsadmin \
|
||||
--secret-key rustfsadmin \
|
||||
--bucket rustfs-bench \
|
||||
--prefix bench/issue713/plain
|
||||
```
|
||||
|
||||
默认会准备：
|
||||
|
||||
1. `1GiB=plain-1g.bin`
|
||||
2. `2GiB=plain-2g.bin`
|
||||
|
||||
默认最终对象路径：
|
||||
|
||||
1. `s3://rustfs-bench/bench/issue713/plain/plain-1g.bin`
|
||||
2. `s3://rustfs-bench/bench/issue713/plain/plain-2g.bin`
|
||||
|
||||
## 5. 执行 GET 矩阵
|
||||
|
||||
直接使用：
|
||||
|
||||
1. `scripts/run_gt1g_get_http_matrix.sh`
|
||||
|
||||
这个脚本特点：
|
||||
|
||||
1. 不把响应体落到本地磁盘
|
||||
2. 直接把 body 丢到 `/dev/null`
|
||||
3. 用 `curl` 的 `size_download` 做字节校验和吞吐计算
|
||||
4. 同时支持：
|
||||
- `sequential`
|
||||
- `ranged_parallel`
|
||||
|
||||
补充说明：
|
||||
|
||||
1. 如果本机没有 `mc`，脚本会自动 fallback 到仓库内置的 `rustfs/tests/gt1g_get_benchmark_tool.rs`
|
||||
2. fallback 模式下，结果目录会落在：
|
||||
- `rustfs/target/bench/...`
|
||||
3. 非 fallback 模式下，结果目录仍按脚本参数落在：
|
||||
- `target/bench/...`
|
||||
|
||||
### 5.1 baseline：sequential + ranged_parallel
|
||||
|
||||
```bash
|
||||
bash scripts/run_gt1g_get_http_matrix.sh \
|
||||
--endpoint http://127.0.0.1:9000 \
|
||||
--access-key rustfsadmin \
|
||||
--secret-key rustfsadmin \
|
||||
--bucket rustfs-bench \
|
||||
--objects '1GiB=bench/issue713/plain/plain-1g.bin,2GiB=bench/issue713/plain/plain-2g.bin' \
|
||||
--modes sequential,ranged_parallel \
|
||||
--concurrencies 1,4,8 \
|
||||
--range-workers 4 \
|
||||
--rounds 3 \
|
||||
--cooldown-secs 20
|
||||
```
|
||||
|
||||
### 5.2 只跑 sequential
|
||||
|
||||
```bash
|
||||
bash scripts/run_gt1g_get_http_matrix.sh \
|
||||
--endpoint http://127.0.0.1:9000 \
|
||||
--access-key rustfsadmin \
|
||||
--secret-key rustfsadmin \
|
||||
--bucket rustfs-bench \
|
||||
--objects '1GiB=bench/issue713/plain/plain-1g.bin,2GiB=bench/issue713/plain/plain-2g.bin' \
|
||||
--modes sequential \
|
||||
--concurrencies 1,4,8 \
|
||||
--rounds 3 \
|
||||
--cooldown-secs 20
|
||||
```
|
||||
|
||||
### 5.3 只跑 ranged_parallel
|
||||
|
||||
```bash
|
||||
bash scripts/run_gt1g_get_http_matrix.sh \
|
||||
--endpoint http://127.0.0.1:9000 \
|
||||
--access-key rustfsadmin \
|
||||
--secret-key rustfsadmin \
|
||||
--bucket rustfs-bench \
|
||||
--objects '1GiB=bench/issue713/plain/plain-1g.bin,2GiB=bench/issue713/plain/plain-2g.bin' \
|
||||
--modes ranged_parallel \
|
||||
--concurrencies 1,4,8 \
|
||||
--range-workers 4,8 \
|
||||
--rounds 3 \
|
||||
--cooldown-secs 20
|
||||
```
|
||||
|
||||
## 6. 输出目录
|
||||
|
||||
默认输出：
|
||||
|
||||
```text
|
||||
target/bench/gt1g-get-http-<timestamp>/
|
||||
```
|
||||
|
||||
重点看：
|
||||
|
||||
1. `round_results.csv`
|
||||
2. `median_summary.csv`
|
||||
3. `logs/`
|
||||
|
||||
`round_results.csv` 字段：
|
||||
|
||||
1. `object_label`
|
||||
2. `object_key`
|
||||
3. `mode`
|
||||
4. `range_workers`
|
||||
5. `concurrency`
|
||||
6. `round`
|
||||
7. `status`
|
||||
8. `total_bytes`
|
||||
9. `elapsed_ms`
|
||||
10. `throughput_bps`
|
||||
11. `avg_client_latency_ms`
|
||||
|
||||
## 7. 当前推荐观察点
|
||||
|
||||
### 7.1 sequential 是否命中大对象策略
|
||||
|
||||
查询：
|
||||
|
||||
```promql
|
||||
sum by (strategy) (
|
||||
increase(rustfs_io_get_object_stream_strategy_total[5m])
|
||||
)
|
||||
```
|
||||
|
||||
预期：
|
||||
|
||||
1. `>1GiB` 顺序 GET 应出现 `large_sequential_readahead`
|
||||
|
||||
### 7.2 GET stream buffer 分布
|
||||
|
||||
```promql
|
||||
histogram_quantile(
|
||||
0.95,
|
||||
sum by (strategy, le) (
|
||||
rate(rustfs_io_get_object_stream_buffer_size_bytes_bucket[5m])
|
||||
)
|
||||
)
|
||||
```
|
||||
|
||||
### 7.3 GET 端到端完成时长
|
||||
|
||||
```promql
|
||||
histogram_quantile(
|
||||
0.95,
|
||||
sum by (le) (
|
||||
rate(rustfs_io_get_object_total_duration_seconds_bucket[5m])
|
||||
)
|
||||
)
|
||||
```
|
||||
|
||||
## 8. 当前判断建议
|
||||
|
||||
如果 sequential 在 `1GiB/2GiB`、`c1` 下仍然没有明显优于旧结果，优先检查：
|
||||
|
||||
1. `large_sequential_readahead` 是否真的命中
|
||||
2. buffer size 是否仍停在 `1MiB`
|
||||
3. 请求是否实际是 range / encrypted / compressed，从而没走当前优化路径
|
||||
|
||||
如果 ranged_parallel 明显优于 sequential，再继续补：
|
||||
|
||||
1. 客户端并行分片数建议
|
||||
2. plain 与 encrypted/compressed 的拆分对比
|
||||
|
||||
## 9. 当前边界
|
||||
|
||||
这轮实现还没有直接覆盖：
|
||||
|
||||
1. encrypted `>1GiB GET` 的单独 read-ahead 策略
|
||||
2. compressed `>1GiB GET` 的专门 index / range 成本优化
|
||||
3. server-side 主动预取缓存
|
||||
|
||||
因此当前更适合先做：
|
||||
|
||||
1. plain sequential vs ranged_parallel 基线
|
||||
2. 命中策略与吞吐结果收敛
|
||||
3. 再决定下一刀是否继续进 encrypted / compressed
|
||||
Reference in New Issue
Block a user