feat(storage): optimize gt1g get read path (#3860)

* feat(storage): tune gt1g get read path

* docs(ops): add gt1g get benchmark guide

* feat(scripts): add gt1g get fallback runner

* test(storage): trim gt1g get helper warnings

* test(storage): reduce gt1g get helper type complexity
This commit is contained in:
houseme
2026-06-25 20:12:08 +08:00
committed by GitHub
parent a30357d21e
commit 96b1f5c373
8 changed files with 1776 additions and 21 deletions
+10
View File
@@ -261,6 +261,16 @@ pub fn record_get_object_completion(total_duration_secs: f64, response_size_byte
histogram!("rustfs_io_get_object_buffer_size_bytes").record(buffer_size_bytes as f64);
}
/// Record the streaming strategy chosen for a GetObject response body.
#[inline(always)]
pub fn record_get_object_stream_strategy(strategy: &str, buffer_size_bytes: usize, response_size_bytes: i64) {
counter!("rustfs_io_get_object_stream_strategy_total", "strategy" => strategy.to_string()).increment(1);
histogram!("rustfs_io_get_object_stream_buffer_size_bytes", "strategy" => strategy.to_string())
.record(buffer_size_bytes as f64);
histogram!("rustfs_io_get_object_stream_response_size_bytes", "strategy" => strategy.to_string())
.record(response_size_bytes.max(0) as f64);
}
/// Record I/O queue congestion observation.
#[inline(always)]
pub fn record_io_queue_congestion() {
@@ -0,0 +1,87 @@
# Issue #713 `>1GiB GET` baseline v1 小结
## 1. 执行范围
本轮是 `#713` 的第一轮 plain baseline,先不碰 encrypted / compressed,只看:
1. `1GiB`
2. `2GiB`
3. `sequential`
4. `ranged_parallel`
5. `concurrency=1,4,8`
6. `range_workers=4`
7. `rounds=1`
结果目录:
1. `rustfs/target/bench/issue713-gt1g-get-baseline-v1-20260625T105149Z`
## 2. 结果总表
### 2.1 `1GiB`
`sequential`
1. `c1`: `99.36 MiB/s`, `10807.0ms`
2. `c4`: `518.53 MiB/s`, `8267.5ms`
3. `c8`: `708.04 MiB/s`, `11362.4ms`
`ranged_parallel`
1. `c1`: `481.71 MiB/s`, `2229.0ms`
2. `c4`: `1536.66 MiB/s`, `2728.5ms`
3. `c8`: `1548.29 MiB/s`, `5010.9ms`
### 2.2 `2GiB`
`sequential`
1. `c1`: `116.43 MiB/s`, `18444.0ms`
2. `c4`: `522.06 MiB/s`, `16442.5ms`
3. `c8`: `712.24 MiB/s`, `21842.1ms`
`ranged_parallel`
1. `c1`: `503.28 MiB/s`, `4267.0ms`
2. `c4`: `1817.59 MiB/s`, `4707.3ms`
3. `c8`: `1681.50 MiB/s`, `9502.1ms`
## 3. 当前第一结论
本轮 plain baseline 可以先收敛出三点:
1. `ranged_parallel` 在当前单机多盘场景下,明显优于单流 `sequential`
2. 这种优势在 `1GiB``2GiB` 两个点上都成立
3. 即使 `sequential` 已经过了当前这轮 `>1GiB GET` buffer / readahead 优化,当前仍然没有追平 `4-way ranged_parallel`
换句话说:
1. 这轮服务端优化是必要的
2. 但对当前这组 plain `>1GiB GET` workload 来说,客户端并行 range 下载仍然是更强的模式
## 4. 当前推荐
在这轮 baseline 之后,最自然的下一步是:
1. 先补一轮更稳妥的确认性复测,只保留重点点位:
- `1GiB c1`
- `1GiB c4`
- `2GiB c1`
- `2GiB c4`
2. 如果结论仍稳定,再继续补:
- larger `range_workers` 对比
- encrypted `>1GiB GET`
- compressed `>1GiB GET`
## 5. 当前边界说明
这轮还不能直接下最终结论的地方:
1. 只有 `1 round`
2. `ranged_parallel` 当前只测了 `range_workers=4`
3. 还没有拆 plain / encrypted / compressed
因此当前更准确的表述是:
1. plain `>1GiB GET` 的第一轮 baseline 已显示 `ranged_parallel` 明显领先
2. 但还需要更小面复测来确认这个结论的稳定性
@@ -0,0 +1,258 @@
# Issue #713 `>1GiB GET` 优化与验证手册
## 1. 目标
本文用于收口 `backlog#713` 当前这一轮 `>1GiB GET` 优化的实现点与验证方式。
当前这轮改动先解决三件事:
1. `sequential hint` 真正参与 GET I/O 策略计算
2. storage media buffer cap 不再被统一的 `1MiB` hard clamp 二次截断
3. `>1GiB`、非 range、允许 readahead 的 streaming GET,会走更大的 stream buffer 策略
## 2. 当前代码行为变化
### 2.1 多因素调度
涉及:
1. `rustfs/src/storage/concurrency/io_schedule.rs`
当前行为:
1. `IoSchedulingContext.is_sequential_hint=true` 时,会优先按 `Sequential` 模式参与 buffer / readahead 决策
2. storage profile 提供的 media cap 现在可以真实生效
3. 例如默认常量下:
- `NVMe`: `2MiB`
- `SSD`: `1MiB`
- `HDD`: `512KiB`
### 2.2 GET body stream 策略
涉及:
1. `rustfs/src/app/object_usecase.rs`
当前行为:
1. `optimal_buffer_size` 不再被额外 `min(base_buffer_size)` 压回去
2. `>1GiB`、非 range、`enable_readahead=true` 的 streaming GET,会命中:
- `large_sequential_readahead`
3. 该路径当前把 `ReaderStream` buffer 扩到:
- `min(optimal_buffer_size * 2, 4MiB)`
### 2.3 GET observability
涉及:
1. `crates/io-metrics/src/lib.rs`
新增:
1. `rustfs_io_get_object_stream_strategy_total{strategy=...}`
2. `rustfs_io_get_object_stream_buffer_size_bytes{strategy=...}`
3. `rustfs_io_get_object_stream_response_size_bytes{strategy=...}`
当前策略标签:
1. `standard`
2. `large_sequential_readahead`
## 3. 推荐验证顺序
按下面顺序做最稳妥:
1. 先准备 plain `1GiB` / `2GiB` 对象
2. 先跑 `sequential` baseline
3. 再跑 `ranged_parallel`
4. 最后再决定是否继续补 encrypted / compressed
## 4. 准备测试对象
直接使用:
1. `scripts/prepare_gt1g_get_test_objects.sh`
示例:
```bash
bash scripts/prepare_gt1g_get_test_objects.sh \
--endpoint http://127.0.0.1:9000 \
--access-key rustfsadmin \
--secret-key rustfsadmin \
--bucket rustfs-bench \
--prefix bench/issue713/plain
```
默认会准备:
1. `1GiB=plain-1g.bin`
2. `2GiB=plain-2g.bin`
默认最终对象路径:
1. `s3://rustfs-bench/bench/issue713/plain/plain-1g.bin`
2. `s3://rustfs-bench/bench/issue713/plain/plain-2g.bin`
## 5. 执行 GET 矩阵
直接使用:
1. `scripts/run_gt1g_get_http_matrix.sh`
这个脚本特点:
1. 不把响应体落到本地磁盘
2. 直接把 body 丢到 `/dev/null`
3.`curl``size_download` 做字节校验和吞吐计算
4. 同时支持:
- `sequential`
- `ranged_parallel`
补充说明:
1. 如果本机没有 `mc`,脚本会自动 fallback 到仓库内置的 `rustfs/tests/gt1g_get_benchmark_tool.rs`
2. fallback 模式下,结果目录会落在:
- `rustfs/target/bench/...`
3. 非 fallback 模式下,结果目录仍按脚本参数落在:
- `target/bench/...`
### 5.1 baselinesequential + ranged_parallel
```bash
bash scripts/run_gt1g_get_http_matrix.sh \
--endpoint http://127.0.0.1:9000 \
--access-key rustfsadmin \
--secret-key rustfsadmin \
--bucket rustfs-bench \
--objects '1GiB=bench/issue713/plain/plain-1g.bin,2GiB=bench/issue713/plain/plain-2g.bin' \
--modes sequential,ranged_parallel \
--concurrencies 1,4,8 \
--range-workers 4 \
--rounds 3 \
--cooldown-secs 20
```
### 5.2 只跑 sequential
```bash
bash scripts/run_gt1g_get_http_matrix.sh \
--endpoint http://127.0.0.1:9000 \
--access-key rustfsadmin \
--secret-key rustfsadmin \
--bucket rustfs-bench \
--objects '1GiB=bench/issue713/plain/plain-1g.bin,2GiB=bench/issue713/plain/plain-2g.bin' \
--modes sequential \
--concurrencies 1,4,8 \
--rounds 3 \
--cooldown-secs 20
```
### 5.3 只跑 ranged_parallel
```bash
bash scripts/run_gt1g_get_http_matrix.sh \
--endpoint http://127.0.0.1:9000 \
--access-key rustfsadmin \
--secret-key rustfsadmin \
--bucket rustfs-bench \
--objects '1GiB=bench/issue713/plain/plain-1g.bin,2GiB=bench/issue713/plain/plain-2g.bin' \
--modes ranged_parallel \
--concurrencies 1,4,8 \
--range-workers 4,8 \
--rounds 3 \
--cooldown-secs 20
```
## 6. 输出目录
默认输出:
```text
target/bench/gt1g-get-http-<timestamp>/
```
重点看:
1. `round_results.csv`
2. `median_summary.csv`
3. `logs/`
`round_results.csv` 字段:
1. `object_label`
2. `object_key`
3. `mode`
4. `range_workers`
5. `concurrency`
6. `round`
7. `status`
8. `total_bytes`
9. `elapsed_ms`
10. `throughput_bps`
11. `avg_client_latency_ms`
## 7. 当前推荐观察点
### 7.1 sequential 是否命中大对象策略
查询:
```promql
sum by (strategy) (
increase(rustfs_io_get_object_stream_strategy_total[5m])
)
```
预期:
1. `>1GiB` 顺序 GET 应出现 `large_sequential_readahead`
### 7.2 GET stream buffer 分布
```promql
histogram_quantile(
0.95,
sum by (strategy, le) (
rate(rustfs_io_get_object_stream_buffer_size_bytes_bucket[5m])
)
)
```
### 7.3 GET 端到端完成时长
```promql
histogram_quantile(
0.95,
sum by (le) (
rate(rustfs_io_get_object_total_duration_seconds_bucket[5m])
)
)
```
## 8. 当前判断建议
如果 sequential 在 `1GiB/2GiB``c1` 下仍然没有明显优于旧结果,优先检查:
1. `large_sequential_readahead` 是否真的命中
2. buffer size 是否仍停在 `1MiB`
3. 请求是否实际是 range / encrypted / compressed,从而没走当前优化路径
如果 ranged_parallel 明显优于 sequential,再继续补:
1. 客户端并行分片数建议
2. plain 与 encrypted/compressed 的拆分对比
## 9. 当前边界
这轮实现还没有直接覆盖:
1. encrypted `>1GiB GET` 的单独 read-ahead 策略
2. compressed `>1GiB GET` 的专门 index / range 成本优化
3. server-side 主动预取缓存
因此当前更适合先做:
1. plain sequential vs ranged_parallel 基线
2. 命中策略与吞吐结果收敛
3. 再决定下一刀是否继续进 encrypted / compressed
+102 -8
View File
@@ -87,6 +87,7 @@ use md5::Context as Md5Context;
use metrics::{counter, histogram};
use pin_project_lite::pin_project;
use rustfs_concurrency::GetObjectQueueSnapshot;
use rustfs_config::MI_B;
use rustfs_filemeta::{
REPLICATE_INCOMING_DELETE, ReplicateDecision, ReplicateTargetDecision, ReplicationState, ReplicationStatusType,
ReplicationType, RestoreStatusOps, VersionPurgeStatusType, parse_restore_obj_status, replication_statuses_map,
@@ -267,6 +268,7 @@ struct GetObjectStrategyContext {
#[allow(dead_code)]
io_strategy: concurrency::IoStrategy,
optimal_buffer_size: usize,
enable_readahead: bool,
}
struct GetObjectOutputContext {
@@ -282,6 +284,25 @@ enum GetObjectTimeoutStage {
BeforeRead,
}
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
enum GetObjectStreamStrategy {
Standard,
LargeSequentialReadahead,
}
impl GetObjectStreamStrategy {
fn as_str(self) -> &'static str {
match self {
Self::Standard => "standard",
Self::LargeSequentialReadahead => "large_sequential_readahead",
}
}
}
const LARGE_SEQUENTIAL_GET_THRESHOLD_BYTES: i64 = 1024 * 1024 * 1024;
const LARGE_SEQUENTIAL_GET_STREAM_BUFFER_CAP_BYTES: usize = 4 * MI_B;
const LARGE_SEQUENTIAL_GET_READAHEAD_MULTIPLIER: usize = 2;
async fn enqueue_transitioned_delete_cleanup(
store: Arc<ECStore>,
bucket: &str,
@@ -1530,15 +1551,42 @@ impl DefaultObjectUsecase {
)))
}
fn build_reader_blob<R>(reader: R, response_content_length: i64, optimal_buffer_size: usize) -> Option<StreamingBlob>
fn select_stream_buffer_strategy(
response_content_length: i64,
optimal_buffer_size: usize,
enable_readahead: bool,
has_range: bool,
) -> (usize, GetObjectStreamStrategy) {
if enable_readahead && !has_range && response_content_length >= LARGE_SEQUENTIAL_GET_THRESHOLD_BYTES {
let expanded_buffer_size = optimal_buffer_size
.saturating_mul(LARGE_SEQUENTIAL_GET_READAHEAD_MULTIPLIER)
.min(LARGE_SEQUENTIAL_GET_STREAM_BUFFER_CAP_BYTES)
.max(optimal_buffer_size);
return (expanded_buffer_size, GetObjectStreamStrategy::LargeSequentialReadahead);
}
(optimal_buffer_size, GetObjectStreamStrategy::Standard)
}
fn build_reader_blob<R>(
reader: R,
response_content_length: i64,
stream_buffer_size: usize,
stream_strategy: GetObjectStreamStrategy,
) -> Option<StreamingBlob>
where
R: AsyncRead + Send + Sync + 'static,
{
let expected = response_content_length.max(0) as usize;
rustfs_io_metrics::record_get_object_stream_strategy(
stream_strategy.as_str(),
stream_buffer_size,
response_content_length,
);
#[cfg(feature = "tracing-chunk-debug")]
let stream = {
let mut emitted = 0usize;
ReaderStream::with_capacity(reader, optimal_buffer_size).inspect(move |item| match item {
ReaderStream::with_capacity(reader, stream_buffer_size).inspect(move |item| match item {
Ok(bytes) => {
emitted += bytes.len();
tracing::debug!(emitted, expected, chunk_len = bytes.len(), "GetObject ReaderStream emitted bytes");
@@ -1554,7 +1602,7 @@ impl DefaultObjectUsecase {
})
};
#[cfg(not(feature = "tracing-chunk-debug"))]
let stream = ReaderStream::with_capacity(reader, optimal_buffer_size);
let stream = ReaderStream::with_capacity(reader, stream_buffer_size);
Some(StreamingBlob::wrap(bytes_stream(stream, expected)))
}
@@ -1864,7 +1912,11 @@ impl DefaultObjectUsecase {
queue_status: &concurrency::IoQueueStatus,
concurrent_requests: usize,
) -> GetObjectStrategyContext {
let base_buffer_size = self.base_buffer_size();
let base_buffer_size = if response_content_length > 0 {
get_buffer_size_opt_in(response_content_length)
} else {
self.base_buffer_size()
};
let is_sequential_hint = if rs.is_none() {
true
@@ -1932,9 +1984,8 @@ impl DefaultObjectUsecase {
"I/O priority finalized with actual request size"
);
let base_buffer_size = get_buffer_size_opt_in(response_content_length);
let optimal_buffer_size = if io_strategy.buffer_size > 0 {
io_strategy.buffer_size.min(base_buffer_size)
io_strategy.buffer_size
} else {
get_concurrency_aware_buffer_size(response_content_length, base_buffer_size)
};
@@ -1943,10 +1994,12 @@ impl DefaultObjectUsecase {
"GetObject buffer sizing: file_size={}, base={}, optimal={}, concurrent_requests={}, io_strategy={:?}",
response_content_length, base_buffer_size, optimal_buffer_size, concurrent_requests, io_strategy.load_level
);
let enable_readahead = io_strategy.enable_readahead;
GetObjectStrategyContext {
io_strategy,
optimal_buffer_size,
enable_readahead,
}
}
@@ -1993,6 +2046,7 @@ impl DefaultObjectUsecase {
info: &ObjectInfo,
response_content_length: i64,
optimal_buffer_size: usize,
enable_readahead: bool,
part_number: Option<usize>,
has_range: bool,
encryption_applied: bool,
@@ -2023,7 +2077,14 @@ impl DefaultObjectUsecase {
}
debug!(buffer_size = optimal_buffer_size, "Encrypted object uses streaming decrypt path");
return Ok(Self::build_reader_blob(final_stream, response_content_length, optimal_buffer_size));
let (stream_buffer_size, stream_strategy) =
Self::select_stream_buffer_strategy(response_content_length, optimal_buffer_size, enable_readahead, has_range);
return Ok(Self::build_reader_blob(
final_stream,
response_content_length,
stream_buffer_size,
stream_strategy,
));
}
let should_provide_seek_support =
@@ -2049,7 +2110,14 @@ impl DefaultObjectUsecase {
}
}
Ok(Self::build_reader_blob(final_stream, response_content_length, optimal_buffer_size))
let (stream_buffer_size, stream_strategy) =
Self::select_stream_buffer_strategy(response_content_length, optimal_buffer_size, enable_readahead, has_range);
Ok(Self::build_reader_blob(
final_stream,
response_content_length,
stream_buffer_size,
stream_strategy,
))
}
fn put_object_execution_context(req: &S3Request<PutObjectInput>) -> (EventName, QuotaOperation, &'static str) {
@@ -2754,6 +2822,7 @@ impl DefaultObjectUsecase {
let GetObjectStrategyContext {
io_strategy: _,
optimal_buffer_size,
enable_readahead,
} = strategy;
let body = Self::build_get_object_body(
@@ -2761,6 +2830,7 @@ impl DefaultObjectUsecase {
&info,
response_content_length,
optimal_buffer_size,
enable_readahead,
part_number,
rs.is_some(),
encryption_applied,
@@ -5487,6 +5557,7 @@ mod tests {
&info,
18_i64 * 1024 * 1024 * 1024,
128 * 1024,
true,
None,
false,
false,
@@ -5518,6 +5589,7 @@ mod tests {
&info,
18_i64 * 1024 * 1024 * 1024,
128 * 1024,
true,
None,
false,
true,
@@ -5533,6 +5605,28 @@ mod tests {
);
}
#[test]
fn select_stream_buffer_strategy_expands_large_sequential_gets() {
let (buffer_size, strategy) =
DefaultObjectUsecase::select_stream_buffer_strategy(2_i64 * 1024 * 1024 * 1024, 2 * MI_B, true, false);
assert_eq!(strategy, GetObjectStreamStrategy::LargeSequentialReadahead);
assert_eq!(buffer_size, 4 * MI_B);
}
#[test]
fn select_stream_buffer_strategy_keeps_ranges_and_small_gets_standard() {
let (range_buffer_size, range_strategy) =
DefaultObjectUsecase::select_stream_buffer_strategy(2_i64 * 1024 * 1024 * 1024, 2 * MI_B, true, true);
assert_eq!(range_strategy, GetObjectStreamStrategy::Standard);
assert_eq!(range_buffer_size, 2 * MI_B);
let (small_buffer_size, small_strategy) =
DefaultObjectUsecase::select_stream_buffer_strategy(64 * 1024 * 1024, 512 * 1024, true, false);
assert_eq!(small_strategy, GetObjectStreamStrategy::Standard);
assert_eq!(small_buffer_size, 512 * 1024);
}
#[test]
fn should_use_zero_copy_rejects_encrypted_requests_with_sse_customer_algorithm() {
let mut headers = HeaderMap::new();
+43 -13
View File
@@ -889,6 +889,11 @@ impl IoStrategy {
// Stage 1: Start with base buffer size
let mut buffer_size;
let mut buffer_multiplier = 1.0;
let effective_access_pattern = if context.is_sequential_hint {
AccessPattern::Sequential
} else {
context.access_pattern
};
// Stage 2: Apply load level reduction based on permit wait
let load_level = IoLoadLevel::from_wait_duration_with_thresholds(
@@ -924,7 +929,7 @@ impl IoStrategy {
);
// Stage 5: Apply access pattern adjustments
let pattern_multiplier = match context.access_pattern {
let pattern_multiplier = match effective_access_pattern {
AccessPattern::Sequential => storage_profile.sequential_boost_multiplier,
AccessPattern::Random => storage_profile.random_penalty_multiplier,
AccessPattern::Mixed => 1.0,
@@ -964,7 +969,7 @@ impl IoStrategy {
// Apply final clamp (safety bounds)
let clamp_min = 32 * KI_B;
let clamp_max = MI_B;
let clamp_max = buffer_cap.max(MI_B);
#[cfg(feature = "io-scheduler-debug")]
let clamp_min_applied = buffer_size < clamp_min;
#[cfg(feature = "io-scheduler-debug")]
@@ -982,7 +987,7 @@ impl IoStrategy {
};
// Apply access pattern override
let readahead_disabled_by_pattern = matches!(context.access_pattern, AccessPattern::Random);
let readahead_disabled_by_pattern = matches!(effective_access_pattern, AccessPattern::Random);
if readahead_disabled_by_pattern {
should_enable_readahead = false;
#[cfg(feature = "io-scheduler-debug")]
@@ -993,7 +998,7 @@ impl IoStrategy {
// Apply concurrency override
let readahead_disabled_by_concurrency = context.concurrent_requests >= config.random_readahead_disable_concurrency;
if readahead_disabled_by_concurrency && matches!(context.access_pattern, AccessPattern::Random) {
if readahead_disabled_by_concurrency && matches!(effective_access_pattern, AccessPattern::Random) {
should_enable_readahead = false;
#[cfg(feature = "io-scheduler-debug")]
{
@@ -1039,7 +1044,7 @@ impl IoStrategy {
let core = IoStrategyCore {
// ===== Basic Configuration =====
storage_media: context.storage_media,
access_pattern: context.access_pattern,
access_pattern: effective_access_pattern,
request_size: context.file_size,
base_buffer_size: context.base_buffer_size,
buffer_cap,
@@ -1055,7 +1060,7 @@ impl IoStrategy {
observed_bandwidth_bps: context.observed_bandwidth_bps,
bandwidth_tier,
bandwidth_limited,
sequential_detected: matches!(context.access_pattern, AccessPattern::Sequential),
sequential_detected: matches!(effective_access_pattern, AccessPattern::Sequential),
// ===== Decision Flags =====
storage_profile,
@@ -1065,8 +1070,8 @@ impl IoStrategy {
// ===== Tuning Multipliers =====
final_multiplier: buffer_multiplier,
should_throttle_random_io: matches!(context.access_pattern, AccessPattern::Random),
should_expand_for_sequential: matches!(context.access_pattern, AccessPattern::Sequential),
should_throttle_random_io: matches!(effective_access_pattern, AccessPattern::Random),
should_expand_for_sequential: matches!(effective_access_pattern, AccessPattern::Sequential),
should_reduce_for_concurrency: concurrency_multiplier < 1.0,
should_reduce_for_bandwidth: bandwidth_limited,
should_disable_readahead: !enable_readahead,
@@ -1112,7 +1117,7 @@ impl IoStrategy {
// ===== State Labels =====
load_level_label: load_level_clone.as_str(),
pattern_label: context.access_pattern.as_str(),
pattern_label: effective_access_pattern.as_str(),
media_label: match context.storage_media {
StorageMedia::Nvme => "nvme",
StorageMedia::Ssd => "ssd",
@@ -1143,8 +1148,8 @@ impl IoStrategy {
read_size_known: context.file_size > 0,
// ===== Decision Tracking =====
random_penalty_applied: matches!(context.access_pattern, AccessPattern::Random),
sequential_boost_applied: matches!(context.access_pattern, AccessPattern::Sequential),
random_penalty_applied: matches!(effective_access_pattern, AccessPattern::Random),
sequential_boost_applied: matches!(effective_access_pattern, AccessPattern::Sequential),
buffer_cap_applied,
clamp_min_applied,
clamp_max_applied,
@@ -2265,13 +2270,38 @@ mod tests {
let config = IoSchedulerConfig::default();
let strategy = IoStrategy::from_context_with_config(&context, &config);
// Should be capped at NVMe buffer cap and 1MB max
assert!(strategy.buffer_size <= MI_B, "Should be capped at 1MB max");
// Should be capped at the configured NVMe media cap.
assert_eq!(strategy.buffer_size, rustfs_config::DEFAULT_OBJECT_IO_NVME_BUFFER_CAP);
#[cfg(feature = "io-scheduler-debug")]
assert!(strategy.debug_info.buffer_cap_applied, "Buffer cap should be applied");
}
#[tokio::test]
#[serial]
async fn test_multi_factor_strategy_applies_sequential_hint_when_pattern_unknown() {
let context = IoSchedulingContext {
file_size: 2 * 1024 * 1024 * 1024, // 2GiB
base_buffer_size: 1024 * 1024, // 1MiB base
permit_wait_duration: Duration::from_millis(1), // Low load
is_sequential_hint: true,
access_pattern: AccessPattern::Unknown,
storage_media: StorageMedia::Nvme,
observed_bandwidth_bps: Some(1000 * 1024 * 1024),
concurrent_requests: 1,
};
let config = IoSchedulerConfig::default();
let strategy = IoStrategy::from_context_with_config(&context, &config);
assert_eq!(strategy.access_pattern, AccessPattern::Sequential);
assert!(strategy.enable_readahead, "Sequential hint should keep readahead enabled");
assert!(
strategy.buffer_size > context.base_buffer_size,
"Sequential hint should allow a larger buffer for large GETs"
);
}
#[tokio::test]
#[serial]
async fn test_multi_factor_strategy_bandwidth_low_reduces_buffer() {
+526
View File
@@ -0,0 +1,526 @@
use anyhow::{Context, Result, anyhow};
use aws_config::BehaviorVersion;
use aws_config::meta::region::RegionProviderChain;
use aws_sdk_s3::Client;
use aws_sdk_s3::config::{Builder as S3ConfigBuilder, Credentials, Region};
use aws_sdk_s3::primitives::ByteStream;
use std::env;
use std::path::PathBuf;
use std::time::Instant;
use tokio::fs::OpenOptions;
use tokio::io::AsyncWriteExt;
use tokio::time::{Duration, sleep};
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
enum ToolAction {
Prepare,
Bench,
}
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
enum BenchMode {
Sequential,
RangedParallel,
}
impl BenchMode {
fn as_str(self) -> &'static str {
match self {
Self::Sequential => "sequential",
Self::RangedParallel => "ranged_parallel",
}
}
}
#[derive(Clone, Debug)]
struct ObjectSpec {
label: String,
key: String,
size_bytes: i64,
}
#[derive(Clone, Debug)]
struct ToolSettings {
action: ToolAction,
endpoint: String,
access_key: String,
secret_key: String,
bucket: String,
region: String,
out_dir: PathBuf,
objects: Vec<ObjectSpec>,
modes: Vec<BenchMode>,
concurrencies: Vec<usize>,
range_workers: Vec<usize>,
rounds: usize,
cooldown_secs: u64,
force: bool,
}
type BenchGroupKey = (String, String, String, usize, usize);
type BenchGroupValue = (bool, u64, f64, f64, f64);
fn parse_action() -> Result<ToolAction> {
match env::var("GT1G_GET_ACTION")
.unwrap_or_else(|_| "bench".to_string())
.trim()
.to_ascii_lowercase()
.as_str()
{
"prepare" => Ok(ToolAction::Prepare),
"bench" => Ok(ToolAction::Bench),
other => Err(anyhow!("unsupported GT1G_GET_ACTION: {other}")),
}
}
fn parse_size_label_to_bytes(label: &str) -> Result<i64> {
let normalized = label.trim();
if let Some(raw) = normalized.strip_suffix("GiB") {
let value = raw.trim().parse::<i64>()?;
return Ok(value * 1024 * 1024 * 1024);
}
if let Some(raw) = normalized.strip_suffix("MiB") {
let value = raw.trim().parse::<i64>()?;
return Ok(value * 1024 * 1024);
}
if let Some(raw) = normalized.strip_suffix("KiB") {
let value = raw.trim().parse::<i64>()?;
return Ok(value * 1024);
}
Err(anyhow!("unsupported size label: {label}"))
}
fn parse_object_specs(raw: &str) -> Result<Vec<ObjectSpec>> {
raw.split(',')
.map(str::trim)
.filter(|entry| !entry.is_empty())
.map(|entry| {
let (label, key) = entry.split_once('=').ok_or_else(|| anyhow!("invalid object spec: {entry}"))?;
Ok(ObjectSpec {
label: label.trim().to_string(),
key: key.trim().to_string(),
size_bytes: parse_size_label_to_bytes(label.trim())?,
})
})
.collect()
}
fn parse_csv_usize(raw: &str, field: &str) -> Result<Vec<usize>> {
let values: Vec<usize> = raw
.split(',')
.map(str::trim)
.filter(|entry| !entry.is_empty())
.map(|entry| {
entry
.parse::<usize>()
.with_context(|| format!("invalid {field} value: {entry}"))
})
.collect::<Result<Vec<_>>>()?;
if values.is_empty() {
return Err(anyhow!("{field} cannot be empty"));
}
Ok(values)
}
fn parse_modes(raw: &str) -> Result<Vec<BenchMode>> {
let modes: Vec<BenchMode> = raw
.split(',')
.map(str::trim)
.filter(|entry| !entry.is_empty())
.map(|entry| match entry {
"sequential" => Ok(BenchMode::Sequential),
"ranged_parallel" => Ok(BenchMode::RangedParallel),
other => Err(anyhow!("unsupported GT1G_GET_MODES value: {other}")),
})
.collect::<Result<Vec<_>>>()?;
if modes.is_empty() {
return Err(anyhow!("GT1G_GET_MODES cannot be empty"));
}
Ok(modes)
}
impl ToolSettings {
fn from_env() -> Result<Self> {
let action = parse_action()?;
let endpoint = env::var("GT1G_GET_ENDPOINT").context("missing GT1G_GET_ENDPOINT")?;
let access_key = env::var("GT1G_GET_ACCESS_KEY").context("missing GT1G_GET_ACCESS_KEY")?;
let secret_key = env::var("GT1G_GET_SECRET_KEY").context("missing GT1G_GET_SECRET_KEY")?;
let bucket = env::var("GT1G_GET_BUCKET").context("missing GT1G_GET_BUCKET")?;
let region = env::var("GT1G_GET_REGION").unwrap_or_else(|_| "us-east-1".to_string());
let out_dir = env::var("GT1G_GET_OUT_DIR")
.map(PathBuf::from)
.unwrap_or_else(|_| PathBuf::from(format!("target/bench/gt1g-get-tool-{}", chrono_like_timestamp())));
let objects = parse_object_specs(&env::var("GT1G_GET_OBJECTS").context("missing GT1G_GET_OBJECTS")?)?;
let modes = parse_modes(&env::var("GT1G_GET_MODES").unwrap_or_else(|_| "sequential,ranged_parallel".to_string()))?;
let concurrencies = parse_csv_usize(
&env::var("GT1G_GET_CONCURRENCIES").unwrap_or_else(|_| "1,4,8".to_string()),
"GT1G_GET_CONCURRENCIES",
)?;
let range_workers = parse_csv_usize(
&env::var("GT1G_GET_RANGE_WORKERS").unwrap_or_else(|_| "4".to_string()),
"GT1G_GET_RANGE_WORKERS",
)?;
let rounds = env::var("GT1G_GET_ROUNDS")
.unwrap_or_else(|_| "3".to_string())
.parse::<usize>()
.context("invalid GT1G_GET_ROUNDS")?;
let cooldown_secs = env::var("GT1G_GET_COOLDOWN_SECS")
.unwrap_or_else(|_| "15".to_string())
.parse::<u64>()
.context("invalid GT1G_GET_COOLDOWN_SECS")?;
let force = env::var("GT1G_GET_FORCE")
.ok()
.map(|raw| matches!(raw.as_str(), "1" | "true" | "TRUE" | "yes" | "YES"))
.unwrap_or(false);
Ok(Self {
action,
endpoint,
access_key,
secret_key,
bucket,
region,
out_dir,
objects,
modes,
concurrencies,
range_workers,
rounds,
cooldown_secs,
force,
})
}
}
fn chrono_like_timestamp() -> String {
use std::time::{SystemTime, UNIX_EPOCH};
let now = SystemTime::now().duration_since(UNIX_EPOCH).unwrap_or_default().as_secs();
now.to_string()
}
async fn build_client(settings: &ToolSettings) -> Result<Client> {
let region = Region::new(settings.region.clone());
let region_provider = RegionProviderChain::first_try(Some(region.clone()));
let shared_config = aws_config::defaults(BehaviorVersion::latest())
.region(region_provider)
.credentials_provider(Credentials::new(
settings.access_key.clone(),
settings.secret_key.clone(),
None,
None,
"issue713-gt1g-get-tool",
))
.load()
.await;
let s3_config = S3ConfigBuilder::from(&shared_config)
.endpoint_url(settings.endpoint.clone())
.force_path_style(true)
.region(region)
.build();
Ok(Client::from_conf(s3_config))
}
async fn ensure_bucket(client: &Client, bucket: &str) -> Result<()> {
if client.head_bucket().bucket(bucket).send().await.is_ok() {
return Ok(());
}
client.create_bucket().bucket(bucket).send().await?;
Ok(())
}
async fn object_matches_size(client: &Client, bucket: &str, object: &ObjectSpec) -> Result<bool> {
let out = match client.head_object().bucket(bucket).key(&object.key).send().await {
Ok(out) => out,
Err(_) => return Ok(false),
};
Ok(out.content_length().unwrap_or_default() == object.size_bytes)
}
async fn create_sparse_file(path: &PathBuf, size_bytes: i64) -> Result<()> {
let file = OpenOptions::new()
.create(true)
.truncate(true)
.write(true)
.open(path)
.await
.with_context(|| format!("failed to open sparse file: {}", path.display()))?;
file.set_len(size_bytes as u64)
.await
.with_context(|| format!("failed to set sparse file length: {}", path.display()))?;
Ok(())
}
async fn prepare_objects(settings: &ToolSettings, client: &Client) -> Result<()> {
ensure_bucket(client, &settings.bucket).await?;
tokio::fs::create_dir_all(&settings.out_dir).await?;
let tmp_root = settings.out_dir.join("prepare-tmp");
tokio::fs::create_dir_all(&tmp_root).await?;
for object in &settings.objects {
if !settings.force && object_matches_size(client, &settings.bucket, object).await? {
println!("skip existing: s3://{}/{}", settings.bucket, object.key);
continue;
}
let local_path = tmp_root.join(object.key.replace('/', "_"));
create_sparse_file(&local_path, object.size_bytes).await?;
println!("uploading: s3://{}/{} ({})", settings.bucket, object.key, object.label);
client
.put_object()
.bucket(&settings.bucket)
.key(&object.key)
.body(ByteStream::from_path(&local_path).await?)
.send()
.await
.with_context(|| format!("failed to upload {}", object.key))?;
let _ = tokio::fs::remove_file(&local_path).await;
}
Ok(())
}
async fn drain_body(mut body: ByteStream) -> Result<u64> {
let mut total = 0u64;
while let Some(chunk) = body.try_next().await? {
total += chunk.len() as u64;
}
Ok(total)
}
async fn run_sequential_task(client: Client, bucket: String, key: String, expected_size: i64) -> Result<(u64, u128)> {
let started = Instant::now();
let output = client.get_object().bucket(bucket).key(key).send().await?;
let bytes = drain_body(output.body).await?;
if bytes != expected_size as u64 {
return Err(anyhow!("downloaded bytes mismatch: expected {}, got {}", expected_size, bytes));
}
Ok((bytes, started.elapsed().as_millis()))
}
async fn run_range_worker(client: Client, bucket: String, key: String, start: i64, end: i64) -> Result<u64> {
let range = format!("bytes={start}-{end}");
let output = client.get_object().bucket(bucket).key(key).range(range).send().await?;
let bytes = drain_body(output.body).await?;
Ok(bytes)
}
async fn run_ranged_parallel_task(
client: Client,
bucket: String,
key: String,
expected_size: i64,
range_workers: usize,
) -> Result<(u64, u128)> {
let started = Instant::now();
let chunk_size = (expected_size + range_workers as i64 - 1) / range_workers as i64;
let mut join_set = tokio::task::JoinSet::new();
for worker_index in 0..range_workers {
let start = worker_index as i64 * chunk_size;
if start >= expected_size {
continue;
}
let end = (start + chunk_size - 1).min(expected_size - 1);
join_set.spawn(run_range_worker(client.clone(), bucket.clone(), key.clone(), start, end));
}
let mut total = 0u64;
while let Some(result) = join_set.join_next().await {
total += result??;
}
if total != expected_size as u64 {
return Err(anyhow!("ranged download bytes mismatch: expected {}, got {}", expected_size, total));
}
Ok((total, started.elapsed().as_millis()))
}
async fn run_bench(settings: &ToolSettings, client: &Client) -> Result<()> {
tokio::fs::create_dir_all(&settings.out_dir).await?;
let round_csv = settings.out_dir.join("round_results.csv");
let median_csv = settings.out_dir.join("median_summary.csv");
tokio::fs::write(
&round_csv,
"object_label,object_key,mode,range_workers,concurrency,round,status,total_bytes,elapsed_ms,throughput_bps,avg_client_latency_ms\n",
)
.await?;
tokio::fs::write(
&median_csv,
"object_label,object_key,mode,range_workers,concurrency,successful_rounds,failed_rounds,median_total_bytes,median_elapsed_ms,median_throughput_bps,median_avg_client_latency_ms\n",
)
.await?;
let mut rows = Vec::new();
for object in &settings.objects {
for &mode in &settings.modes {
for &concurrency in &settings.concurrencies {
let worker_values = if mode == BenchMode::Sequential {
vec![1usize]
} else {
settings.range_workers.clone()
};
for range_workers in worker_values {
for round in 1..=settings.rounds {
let started = Instant::now();
let mut join_set = tokio::task::JoinSet::new();
for _ in 0..concurrency {
let task_client = client.clone();
let bucket = settings.bucket.clone();
let key = object.key.clone();
let size = object.size_bytes;
if mode == BenchMode::Sequential {
join_set.spawn(run_sequential_task(task_client, bucket, key, size));
} else {
join_set.spawn(run_ranged_parallel_task(task_client, bucket, key, size, range_workers));
}
}
let mut status = "ok".to_string();
let mut total_bytes = 0u64;
let mut latencies_ms = Vec::new();
while let Some(result) = join_set.join_next().await {
match result {
Ok(Ok((bytes, elapsed_ms))) => {
total_bytes += bytes;
latencies_ms.push(elapsed_ms);
}
Ok(Err(err)) => {
status = format!("failed:{err}");
}
Err(err) => {
status = format!("failed:{err}");
}
}
}
let elapsed_ms = started.elapsed().as_millis();
let avg_client_latency_ms = if latencies_ms.is_empty() {
0.0
} else {
latencies_ms.iter().sum::<u128>() as f64 / latencies_ms.len() as f64
};
let throughput_bps = if elapsed_ms == 0 {
0.0
} else {
total_bytes as f64 / (elapsed_ms as f64 / 1000.0)
};
let row = format!(
"{},{},{},{},{},{},{},{},{},{:.6},{:.3}\n",
object.label,
object.key,
mode.as_str(),
range_workers,
concurrency,
round,
status,
total_bytes,
elapsed_ms,
throughput_bps,
avg_client_latency_ms
);
tokio::fs::OpenOptions::new()
.append(true)
.open(&round_csv)
.await?
.write_all(row.as_bytes())
.await?;
rows.push((
object.label.clone(),
object.key.clone(),
mode.as_str().to_string(),
range_workers,
concurrency,
status,
total_bytes,
elapsed_ms as f64,
throughput_bps,
avg_client_latency_ms,
));
if settings.cooldown_secs > 0 && round < settings.rounds {
sleep(Duration::from_secs(settings.cooldown_secs)).await;
}
}
}
}
}
}
use std::collections::BTreeMap;
let mut grouped: BTreeMap<BenchGroupKey, Vec<BenchGroupValue>> = BTreeMap::new();
for (label, key, mode, range_workers, concurrency, status, total_bytes, elapsed_ms, throughput_bps, avg_latency_ms) in rows {
grouped
.entry((label, key, mode, range_workers, concurrency))
.or_default()
.push((status == "ok", total_bytes, elapsed_ms, throughput_bps, avg_latency_ms));
}
for ((label, key, mode, range_workers, concurrency), values) in grouped {
let successful_rounds = values.iter().filter(|v| v.0).count();
let failed_rounds = values.len().saturating_sub(successful_rounds);
let mut bytes = Vec::new();
let mut elapsed = Vec::new();
let mut throughput = Vec::new();
let mut latency = Vec::new();
for (ok, total_bytes, elapsed_ms, throughput_bps, avg_latency_ms) in values {
if ok {
bytes.push(total_bytes as f64);
elapsed.push(elapsed_ms);
throughput.push(throughput_bps);
latency.push(avg_latency_ms);
}
}
let median = |input: &mut Vec<f64>| -> String {
if input.is_empty() {
return "N/A".to_string();
}
input.sort_by(|a, b| a.partial_cmp(b).unwrap_or(std::cmp::Ordering::Equal));
let n = input.len();
if n % 2 == 1 {
format!("{:.6}", input[n / 2])
} else {
format!("{:.6}", (input[n / 2 - 1] + input[n / 2]) / 2.0)
}
};
let row = format!(
"{},{},{},{},{},{},{},{},{},{},{}\n",
label,
key,
mode,
range_workers,
concurrency,
successful_rounds,
failed_rounds,
median(&mut bytes),
median(&mut elapsed),
median(&mut throughput),
median(&mut latency)
);
tokio::fs::OpenOptions::new()
.append(true)
.open(&median_csv)
.await?
.write_all(row.as_bytes())
.await?;
}
Ok(())
}
#[tokio::test]
#[ignore]
async fn gt1g_get_benchmark_tool() -> Result<()> {
let settings = ToolSettings::from_env()?;
let client = build_client(&settings).await?;
match settings.action {
ToolAction::Prepare => prepare_objects(&settings, &client).await?,
ToolAction::Bench => run_bench(&settings, &client).await?,
}
Ok(())
}
+249
View File
@@ -0,0 +1,249 @@
#!/usr/bin/env bash
set -euo pipefail
ENDPOINT=""
ACCESS_KEY=""
SECRET_KEY=""
BUCKET=""
PREFIX="bench/issue713"
OBJECTS="1GiB=plain-1g.bin,2GiB=plain-2g.bin"
REGION="us-east-1"
MC_BIN="${MC_BIN:-mc}"
FORCE=false
INSECURE=false
DRY_RUN=false
usage() {
cat <<'USAGE'
Usage:
scripts/prepare_gt1g_get_test_objects.sh \
--endpoint <url> --access-key <ak> --secret-key <sk> --bucket <bucket> [options]
Required:
--endpoint <url>
--access-key <ak>
--secret-key <sk>
--bucket <bucket>
Options:
--prefix <path> Default: bench/issue713
--objects <csv> Default: 1GiB=plain-1g.bin,2GiB=plain-2g.bin
Format: size=object-name,size=object-name
--region <name> Default: us-east-1
--mc-bin <path> Default: mc
--force Re-upload even if object already exists
--insecure Allow insecure TLS
--dry-run
-h, --help
Examples:
scripts/prepare_gt1g_get_test_objects.sh \
--endpoint http://127.0.0.1:9000 \
--access-key rustfsadmin \
--secret-key rustfsadmin \
--bucket rustfs-bench \
--prefix bench/issue713/plain
USAGE
}
arg_value() {
local flag="$1"
local value="${2:-}"
if [[ -z "$value" || "$value" == --* ]]; then
echo "ERROR: missing value for $flag" >&2
exit 1
fi
printf '%s\n' "$value"
}
require_cmd() {
if ! command -v "$1" >/dev/null 2>&1; then
echo "ERROR: command not found: $1" >&2
exit 1
fi
}
run_rust_helper_fallback() {
local objects_arg=""
local helper_out_dir="$OUT_DIR"
local raw_spec
IFS=',' read -r -a object_specs <<< "$OBJECTS"
for raw_spec in "${object_specs[@]}"; do
local spec size object_name
spec="$(echo "$raw_spec" | awk '{$1=$1;print}')"
[[ -z "$spec" ]] && continue
size="${spec%%=*}"
object_name="${spec#*=}"
objects_arg+="${size}=${PREFIX%/}/${object_name},"
done
objects_arg="${objects_arg%,}"
if [[ -n "$helper_out_dir" && "$helper_out_dir" != /* ]]; then
helper_out_dir="$PWD/$helper_out_dir"
fi
echo "mc not found; falling back to rustfs/tests/gt1g_get_benchmark_tool.rs"
GT1G_GET_ACTION=prepare \
GT1G_GET_ENDPOINT="$ENDPOINT" \
GT1G_GET_ACCESS_KEY="$ACCESS_KEY" \
GT1G_GET_SECRET_KEY="$SECRET_KEY" \
GT1G_GET_BUCKET="$BUCKET" \
GT1G_GET_REGION="$REGION" \
GT1G_GET_OBJECTS="$objects_arg" \
GT1G_GET_OUT_DIR="${helper_out_dir:-$PWD/target/bench/issue713-prepare-helper}" \
GT1G_GET_FORCE="$([[ "$FORCE" == "true" ]] && echo true || echo false)" \
cargo test -p rustfs --test gt1g_get_benchmark_tool gt1g_get_benchmark_tool -- --ignored --nocapture
}
parse_args() {
while [[ $# -gt 0 ]]; do
case "$1" in
--endpoint) ENDPOINT="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--access-key) ACCESS_KEY="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--secret-key) SECRET_KEY="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--bucket) BUCKET="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--prefix) PREFIX="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--objects) OBJECTS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--region) REGION="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--mc-bin) MC_BIN="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--force) FORCE=true; shift ;;
--insecure) INSECURE=true; shift ;;
--dry-run) DRY_RUN=true; shift ;;
-h|--help) usage; exit 0 ;;
*)
echo "ERROR: unknown arg: $1" >&2
usage
exit 1
;;
esac
done
}
validate_args() {
if [[ -z "$ENDPOINT" || -z "$ACCESS_KEY" || -z "$SECRET_KEY" || -z "$BUCKET" ]]; then
echo "ERROR: --endpoint, --access-key, --secret-key, and --bucket are required" >&2
exit 1
fi
}
size_to_bytes() {
local size="$1"
case "$size" in
*GiB)
local n="${size%GiB}"
echo $((n * 1024 * 1024 * 1024))
;;
*MiB)
local n="${size%MiB}"
echo $((n * 1024 * 1024))
;;
*KiB)
local n="${size%KiB}"
echo $((n * 1024))
;;
*B)
echo "${size%B}"
;;
*)
echo "ERROR"
;;
esac
}
create_sparse_file() {
local file_path="$1"
local bytes="$2"
if command -v mkfile >/dev/null 2>&1; then
mkfile -n "$bytes" "$file_path"
else
truncate -s "$bytes" "$file_path"
fi
}
object_exists() {
local alias_path="$1"
local -a cmd=("$MC_BIN" stat "$alias_path")
if [[ "$INSECURE" == "true" ]]; then
cmd=("$MC_BIN" --insecure stat "$alias_path")
fi
"${cmd[@]}" >/dev/null 2>&1
}
main() {
parse_args "$@"
validate_args
if ! command -v "$MC_BIN" >/dev/null 2>&1; then
run_rust_helper_fallback
exit 0
fi
local tmp_root
tmp_root="$(mktemp -d "${TMPDIR:-/tmp}/issue713-gt1g-get.XXXXXX")"
trap 'rm -rf "$tmp_root"' EXIT
local mc_config_dir="$tmp_root/mc"
mkdir -p "$mc_config_dir"
local -a mc_alias_cmd=("$MC_BIN" --config-dir "$mc_config_dir")
if [[ "$INSECURE" == "true" ]]; then
mc_alias_cmd+=("--insecure")
fi
if [[ "$DRY_RUN" == "true" ]]; then
echo "[DRY-RUN] ${mc_alias_cmd[*]} alias set issue713 ${ENDPOINT} REDACTED REDACTED"
else
"${mc_alias_cmd[@]}" alias set issue713 "$ENDPOINT" "$ACCESS_KEY" "$SECRET_KEY" >/dev/null
fi
IFS=',' read -r -a object_specs <<< "$OBJECTS"
for raw_spec in "${object_specs[@]}"; do
local spec size object_name bytes local_file object_key alias_path
spec="$(echo "$raw_spec" | awk '{$1=$1;print}')"
[[ -z "$spec" ]] && continue
size="${spec%%=*}"
object_name="${spec#*=}"
if [[ -z "$size" || -z "$object_name" || "$size" == "$object_name" ]]; then
echo "ERROR: invalid object spec: $spec" >&2
exit 1
fi
bytes="$(size_to_bytes "$size")"
if [[ "$bytes" == "ERROR" ]]; then
echo "ERROR: unsupported size label: $size" >&2
exit 1
fi
object_key="${PREFIX%/}/${object_name}"
alias_path="issue713/${BUCKET}/${object_key}"
if [[ "$FORCE" != "true" && "$DRY_RUN" != "true" ]] && object_exists "$alias_path"; then
echo "skip existing: s3://${BUCKET}/${object_key}"
continue
fi
local_file="${tmp_root}/${object_name}"
create_sparse_file "$local_file" "$bytes"
if [[ "$DRY_RUN" == "true" ]]; then
echo "[DRY-RUN] create sparse file ${local_file} (${bytes} bytes)"
echo "[DRY-RUN] ${mc_alias_cmd[*]} cp ${local_file} ${alias_path}"
else
echo "uploading: s3://${BUCKET}/${object_key} (${size})"
"${mc_alias_cmd[@]}" cp "$local_file" "$alias_path" >/dev/null
fi
done
echo "prepared objects:"
for raw_spec in "${object_specs[@]}"; do
local spec object_name
spec="$(echo "$raw_spec" | awk '{$1=$1;print}')"
[[ -z "$spec" ]] && continue
object_name="${spec#*=}"
echo " s3://${BUCKET}/${PREFIX%/}/${object_name}"
done
}
main "$@"
+501
View File
@@ -0,0 +1,501 @@
#!/usr/bin/env bash
set -euo pipefail
ENDPOINT=""
ACCESS_KEY=""
SECRET_KEY=""
BUCKET=""
OBJECTS=""
REGION="us-east-1"
MODES="sequential,ranged_parallel"
CONCURRENCIES="1,4,8"
RANGE_WORKERS="4"
ROUNDS=3
COOLDOWN_SECS=15
PRESIGN_EXPIRY="2h"
OUT_DIR=""
MC_BIN="${MC_BIN:-mc}"
CURL_BIN="${CURL_BIN:-curl}"
JQ_BIN="${JQ_BIN:-jq}"
INSECURE=false
DRY_RUN=false
usage() {
cat <<'USAGE'
Usage:
scripts/run_gt1g_get_http_matrix.sh \
--endpoint <url> --access-key <ak> --secret-key <sk> \
--bucket <bucket> --objects <csv> [options]
Required:
--endpoint <url>
--access-key <ak>
--secret-key <sk>
--bucket <bucket>
--objects <csv> Format: label=object-key,label=object-key
Options:
--region <name> Default: us-east-1
--modes <csv> Default: sequential,ranged_parallel
--concurrencies <csv> Default: 1,4,8
--range-workers <csv> Default: 4
Used only by ranged_parallel mode
--rounds <n> Default: 3
--cooldown-secs <n> Default: 15
--presign-expiry <dur> Default: 2h
--out-dir <dir> Default: target/bench/gt1g-get-http-<timestamp>
--mc-bin <path> Default: mc
--curl-bin <path> Default: curl
--jq-bin <path> Default: jq
--insecure Allow insecure TLS
--dry-run
-h, --help
Examples:
scripts/run_gt1g_get_http_matrix.sh \
--endpoint http://127.0.0.1:9000 \
--access-key rustfsadmin \
--secret-key rustfsadmin \
--bucket rustfs-bench \
--objects '1GiB=bench/issue713/plain-1g.bin,2GiB=bench/issue713/plain-2g.bin'
USAGE
}
arg_value() {
local flag="$1"
local value="${2:-}"
if [[ -z "$value" || "$value" == --* ]]; then
echo "ERROR: missing value for $flag" >&2
exit 1
fi
printf '%s\n' "$value"
}
require_cmd() {
if ! command -v "$1" >/dev/null 2>&1; then
echo "ERROR: command not found: $1" >&2
exit 1
fi
}
run_rust_helper_fallback() {
local helper_out_dir="$OUT_DIR"
if [[ -n "$helper_out_dir" && "$helper_out_dir" != /* ]]; then
helper_out_dir="$PWD/$helper_out_dir"
fi
echo "mc not found; falling back to rustfs/tests/gt1g_get_benchmark_tool.rs"
GT1G_GET_ACTION=bench \
GT1G_GET_ENDPOINT="$ENDPOINT" \
GT1G_GET_ACCESS_KEY="$ACCESS_KEY" \
GT1G_GET_SECRET_KEY="$SECRET_KEY" \
GT1G_GET_BUCKET="$BUCKET" \
GT1G_GET_REGION="$REGION" \
GT1G_GET_OBJECTS="$OBJECTS" \
GT1G_GET_MODES="$MODES" \
GT1G_GET_CONCURRENCIES="$CONCURRENCIES" \
GT1G_GET_RANGE_WORKERS="$RANGE_WORKERS" \
GT1G_GET_ROUNDS="$ROUNDS" \
GT1G_GET_COOLDOWN_SECS="$COOLDOWN_SECS" \
GT1G_GET_OUT_DIR="$helper_out_dir" \
cargo test -p rustfs --test gt1g_get_benchmark_tool gt1g_get_benchmark_tool -- --ignored --nocapture
}
parse_args() {
while [[ $# -gt 0 ]]; do
case "$1" in
--endpoint) ENDPOINT="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--access-key) ACCESS_KEY="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--secret-key) SECRET_KEY="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--bucket) BUCKET="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--objects) OBJECTS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--region) REGION="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--modes) MODES="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--concurrencies) CONCURRENCIES="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--range-workers) RANGE_WORKERS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--rounds) ROUNDS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--cooldown-secs) COOLDOWN_SECS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--presign-expiry) PRESIGN_EXPIRY="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--out-dir) OUT_DIR="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--mc-bin) MC_BIN="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--curl-bin) CURL_BIN="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--jq-bin) JQ_BIN="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--insecure) INSECURE=true; shift ;;
--dry-run) DRY_RUN=true; shift ;;
-h|--help) usage; exit 0 ;;
*)
echo "ERROR: unknown arg: $1" >&2
usage
exit 1
;;
esac
done
}
validate_positive_int() {
local value="$1"
local label="$2"
if ! [[ "$value" =~ ^[0-9]+$ ]] || (( value <= 0 )); then
echo "ERROR: $label must be a positive integer, got: $value" >&2
exit 1
fi
}
validate_nonnegative_int() {
local value="$1"
local label="$2"
if ! [[ "$value" =~ ^[0-9]+$ ]]; then
echo "ERROR: $label must be a nonnegative integer, got: $value" >&2
exit 1
fi
}
trim() {
echo "$1" | awk '{$1=$1;print}'
}
setup_output() {
if [[ -z "$OUT_DIR" ]]; then
OUT_DIR="target/bench/gt1g-get-http-$(date -u +%Y%m%dT%H%M%SZ)"
fi
LOG_DIR="${OUT_DIR}/logs"
mkdir -p "$LOG_DIR"
ROUND_CSV="${OUT_DIR}/round_results.csv"
MEDIAN_CSV="${OUT_DIR}/median_summary.csv"
cat > "$ROUND_CSV" <<'EOF'
object_label,object_key,mode,range_workers,concurrency,round,status,total_bytes,elapsed_ms,throughput_bps,avg_client_latency_ms,log_file
EOF
cat > "$MEDIAN_CSV" <<'EOF'
object_label,object_key,mode,range_workers,concurrency,successful_rounds,failed_rounds,median_total_bytes,median_elapsed_ms,median_throughput_bps,median_avg_client_latency_ms
EOF
cat > "${OUT_DIR}/artifact_layout.txt" <<'EOF'
round_results.csv
median_summary.csv
artifact_layout.txt
logs/
- <label>_<mode>_rw<workers>_c<concurrency>_r<round>.log
EOF
}
now_ms() {
perl -MTime::HiRes=time -e 'printf "%.0f\n", time()*1000'
}
csv_to_lines() {
local csv="$1"
local raw item
IFS=',' read -r -a parts <<< "$csv"
for raw in "${parts[@]}"; do
item="$(trim "$raw")"
[[ -z "$item" ]] && continue
echo "$item"
done
}
cooldown_sleep() {
local secs="$1"
if (( secs <= 0 )); then
return 0
fi
if [[ "$DRY_RUN" == "true" ]]; then
echo "[DRY-RUN] sleep ${secs}s"
else
sleep "$secs"
fi
}
mc_cmd() {
if [[ "$INSECURE" == "true" ]]; then
"$MC_BIN" --config-dir "$MC_CONFIG_DIR" --insecure "$@"
else
"$MC_BIN" --config-dir "$MC_CONFIG_DIR" "$@"
fi
}
setup_mc_alias() {
MC_CONFIG_DIR="$(mktemp -d "${TMPDIR:-/tmp}/issue713-gt1g-get-mc.XXXXXX")"
trap 'rm -rf "$MC_CONFIG_DIR"' EXIT
if [[ "$DRY_RUN" == "true" ]]; then
echo "[DRY-RUN] mc alias set issue713 ${ENDPOINT} REDACTED REDACTED"
else
mc_cmd alias set issue713 "$ENDPOINT" "$ACCESS_KEY" "$SECRET_KEY" >/dev/null
fi
}
get_object_size_bytes() {
local object_key="$1"
if [[ "$DRY_RUN" == "true" ]]; then
echo 1073741824
return
fi
mc_cmd stat --json "issue713/${BUCKET}/${object_key}" | "$JQ_BIN" -r '.size'
}
presign_object_url() {
local object_key="$1"
if [[ "$DRY_RUN" == "true" ]]; then
echo "${ENDPOINT%/}/${BUCKET}/${object_key}"
return
fi
mc_cmd share download --json --expire "$PRESIGN_EXPIRY" "issue713/${BUCKET}/${object_key}" \
| "$JQ_BIN" -r '.share // .url // .download // empty'
}
sum_metric_files() {
local total=0 value
for metric_file in "$@"; do
[[ -f "$metric_file" ]] || continue
value="$(tr -d '\r\n' < "$metric_file")"
[[ -z "$value" ]] && continue
total="$(awk -v a="$total" -v b="$value" 'BEGIN { printf "%.0f\n", a + b }')"
done
echo "$total"
}
run_sequential_round() {
local url="$1"
local object_size="$2"
local concurrency="$3"
local log_file="$4"
local object_label="$5"
local round="$6"
local -a pids=()
local -a metric_files=()
local task_index metric_file
local curl_rc=0
local start_ms end_ms elapsed_ms total_bytes avg_latency_ms throughput_bps expected_total_bytes
start_ms="$(now_ms)"
for ((task_index = 1; task_index <= concurrency; task_index++)); do
metric_file="${LOG_DIR}/${object_label}_sequential_c${concurrency}_r${round}_t${task_index}.bytes"
metric_files+=("$metric_file")
(
if [[ "$INSECURE" == "true" ]]; then
"$CURL_BIN" -fsSL -k "$url" -o /dev/null -w '%{size_download}\n' > "$metric_file"
else
"$CURL_BIN" -fsSL "$url" -o /dev/null -w '%{size_download}\n' > "$metric_file"
fi
) >>"$log_file" 2>&1 &
pids+=("$!")
done
for pid in "${pids[@]}"; do
if ! wait "$pid"; then
curl_rc=1
fi
done
end_ms="$(now_ms)"
elapsed_ms=$((end_ms - start_ms))
total_bytes="$(sum_metric_files "${metric_files[@]}")"
expected_total_bytes=$((object_size * concurrency))
avg_latency_ms="$(awk -v elapsed="$elapsed_ms" -v conc="$concurrency" 'BEGIN { printf "%.3f\n", elapsed / conc }')"
throughput_bps="$(awk -v bytes="$total_bytes" -v elapsed="$elapsed_ms" 'BEGIN { if (elapsed <= 0) printf "0.000000\n"; else printf "%.6f\n", bytes / (elapsed / 1000.0) }')"
if [[ "$curl_rc" -ne 0 || "$total_bytes" -ne "$expected_total_bytes" ]]; then
echo "failed,${total_bytes},${elapsed_ms},${throughput_bps},${avg_latency_ms}"
else
echo "ok,${total_bytes},${elapsed_ms},${throughput_bps},${avg_latency_ms}"
fi
rm -f "${metric_files[@]}"
}
run_ranged_parallel_round() {
local url="$1"
local object_size="$2"
local concurrency="$3"
local range_workers="$4"
local log_file="$5"
local object_label="$6"
local round="$7"
local chunk_size=$(( (object_size + range_workers - 1) / range_workers ))
local -a pids=()
local -a metric_files=()
local curl_rc=0
local start_ms end_ms elapsed_ms total_bytes avg_latency_ms throughput_bps expected_total_bytes
local client_index worker_index range_start range_end metric_file
start_ms="$(now_ms)"
for ((client_index = 1; client_index <= concurrency; client_index++)); do
for ((worker_index = 0; worker_index < range_workers; worker_index++)); do
range_start=$((worker_index * chunk_size))
if (( range_start >= object_size )); then
continue
fi
range_end=$((range_start + chunk_size - 1))
if (( range_end >= object_size )); then
range_end=$((object_size - 1))
fi
metric_file="${LOG_DIR}/${object_label}_ranged_rw${range_workers}_c${concurrency}_r${round}_t${client_index}_w${worker_index}.bytes"
metric_files+=("$metric_file")
(
if [[ "$INSECURE" == "true" ]]; then
"$CURL_BIN" -fsSL -k -H "Range: bytes=${range_start}-${range_end}" "$url" -o /dev/null -w '%{size_download}\n' > "$metric_file"
else
"$CURL_BIN" -fsSL -H "Range: bytes=${range_start}-${range_end}" "$url" -o /dev/null -w '%{size_download}\n' > "$metric_file"
fi
) >>"$log_file" 2>&1 &
pids+=("$!")
done
done
for pid in "${pids[@]}"; do
if ! wait "$pid"; then
curl_rc=1
fi
done
end_ms="$(now_ms)"
elapsed_ms=$((end_ms - start_ms))
total_bytes="$(sum_metric_files "${metric_files[@]}")"
expected_total_bytes=$((object_size * concurrency))
avg_latency_ms="$(awk -v elapsed="$elapsed_ms" -v conc="$concurrency" 'BEGIN { printf "%.3f\n", elapsed / conc }')"
throughput_bps="$(awk -v bytes="$total_bytes" -v elapsed="$elapsed_ms" 'BEGIN { if (elapsed <= 0) printf "0.000000\n"; else printf "%.6f\n", bytes / (elapsed / 1000.0) }')"
if [[ "$curl_rc" -ne 0 || "$total_bytes" -ne "$expected_total_bytes" ]]; then
echo "failed,${total_bytes},${elapsed_ms},${throughput_bps},${avg_latency_ms}"
else
echo "ok,${total_bytes},${elapsed_ms},${throughput_bps},${avg_latency_ms}"
fi
rm -f "${metric_files[@]}"
}
median_from_numbers() {
local values="$1"
local count
count="$(printf '%s\n' "$values" | awk 'NF { c++ } END { print c + 0 }')"
if [[ "$count" -eq 0 ]]; then
echo "N/A"
return
fi
printf '%s\n' "$values" | awk 'NF' | sort -n | awk '
{ a[NR] = $1 }
END {
n = NR
if (n == 0) { print "N/A"; exit }
if (n % 2 == 1) {
printf "%.6f\n", a[(n + 1) / 2]
} else {
printf "%.6f\n", (a[n / 2] + a[n / 2 + 1]) / 2
}
}'
}
build_median_summary() {
local object_label object_key mode range_workers concurrency
while IFS=',' read -r object_label object_key mode range_workers concurrency; do
local ok_rounds fail_rounds bytes_vals elapsed_vals throughput_vals latency_vals
ok_rounds="$(awk -F',' -v l="$object_label" -v o="$object_key" -v m="$mode" -v rw="$range_workers" -v c="$concurrency" 'NR>1 && $1==l && $2==o && $3==m && $4==rw && $5==c && $7=="ok" {count++} END{print count+0}' "$ROUND_CSV")"
fail_rounds="$(awk -F',' -v l="$object_label" -v o="$object_key" -v m="$mode" -v rw="$range_workers" -v c="$concurrency" 'NR>1 && $1==l && $2==o && $3==m && $4==rw && $5==c && $7!="ok" {count++} END{print count+0}' "$ROUND_CSV")"
bytes_vals="$(awk -F',' -v l="$object_label" -v o="$object_key" -v m="$mode" -v rw="$range_workers" -v c="$concurrency" 'NR>1 && $1==l && $2==o && $3==m && $4==rw && $5==c && $7=="ok" {print $8}' "$ROUND_CSV")"
elapsed_vals="$(awk -F',' -v l="$object_label" -v o="$object_key" -v m="$mode" -v rw="$range_workers" -v c="$concurrency" 'NR>1 && $1==l && $2==o && $3==m && $4==rw && $5==c && $7=="ok" {print $9}' "$ROUND_CSV")"
throughput_vals="$(awk -F',' -v l="$object_label" -v o="$object_key" -v m="$mode" -v rw="$range_workers" -v c="$concurrency" 'NR>1 && $1==l && $2==o && $3==m && $4==rw && $5==c && $7=="ok" {print $10}' "$ROUND_CSV")"
latency_vals="$(awk -F',' -v l="$object_label" -v o="$object_key" -v m="$mode" -v rw="$range_workers" -v c="$concurrency" 'NR>1 && $1==l && $2==o && $3==m && $4==rw && $5==c && $7=="ok" {print $11}' "$ROUND_CSV")"
echo "${object_label},${object_key},${mode},${range_workers},${concurrency},${ok_rounds},${fail_rounds},$(median_from_numbers "$bytes_vals"),$(median_from_numbers "$elapsed_vals"),$(median_from_numbers "$throughput_vals"),$(median_from_numbers "$latency_vals")" \
>> "$MEDIAN_CSV"
done < <(
awk -F',' 'NR>1 { print $1 "," $2 "," $3 "," $4 "," $5 }' "$ROUND_CSV" | sort -u
)
}
main() {
parse_args "$@"
if [[ -z "$ENDPOINT" || -z "$ACCESS_KEY" || -z "$SECRET_KEY" || -z "$BUCKET" || -z "$OBJECTS" ]]; then
echo "ERROR: --endpoint, --access-key, --secret-key, --bucket, and --objects are required" >&2
exit 1
fi
validate_positive_int "$ROUNDS" "--rounds"
validate_nonnegative_int "$COOLDOWN_SECS" "--cooldown-secs"
if ! command -v "$MC_BIN" >/dev/null 2>&1; then
setup_output
run_rust_helper_fallback
exit 0
fi
require_cmd "$MC_BIN"
require_cmd "$CURL_BIN"
require_cmd "$JQ_BIN"
require_cmd awk
require_cmd sort
require_cmd perl
setup_output
setup_mc_alias
IFS=',' read -r -a object_specs <<< "$OBJECTS"
for raw_object_spec in "${object_specs[@]}"; do
local object_spec object_label object_key object_size presigned_url
object_spec="$(trim "$raw_object_spec")"
[[ -z "$object_spec" ]] && continue
object_label="${object_spec%%=*}"
object_key="${object_spec#*=}"
if [[ -z "$object_label" || -z "$object_key" || "$object_label" == "$object_key" ]]; then
echo "ERROR: invalid object spec: $object_spec" >&2
exit 1
fi
object_size="$(get_object_size_bytes "$object_key")"
presigned_url="$(presign_object_url "$object_key")"
if [[ -z "$presigned_url" ]]; then
echo "ERROR: failed to generate presigned URL for ${object_key}" >&2
exit 1
fi
for mode in $(csv_to_lines "$MODES"); do
for concurrency in $(csv_to_lines "$CONCURRENCIES"); do
validate_positive_int "$concurrency" "--concurrencies item"
if [[ "$mode" == "sequential" ]]; then
local round result status total_bytes elapsed_ms throughput_bps avg_latency_ms log_file
for ((round = 1; round <= ROUNDS; round++)); do
log_file="${LOG_DIR}/${object_label}_${mode}_rw1_c${concurrency}_r${round}.log"
result="$(run_sequential_round "$presigned_url" "$object_size" "$concurrency" "$log_file" "$object_label" "$round")"
IFS=',' read -r status total_bytes elapsed_ms throughput_bps avg_latency_ms <<< "$result"
echo "${object_label},${object_key},${mode},1,${concurrency},${round},${status},${total_bytes},${elapsed_ms},${throughput_bps},${avg_latency_ms},${log_file}" >> "$ROUND_CSV"
if (( COOLDOWN_SECS > 0 && round < ROUNDS )); then
cooldown_sleep "$COOLDOWN_SECS"
fi
done
continue
fi
if [[ "$mode" == "ranged_parallel" ]]; then
for range_workers in $(csv_to_lines "$RANGE_WORKERS"); do
validate_positive_int "$range_workers" "--range-workers item"
local round result status total_bytes elapsed_ms throughput_bps avg_latency_ms log_file
for ((round = 1; round <= ROUNDS; round++)); do
log_file="${LOG_DIR}/${object_label}_${mode}_rw${range_workers}_c${concurrency}_r${round}.log"
result="$(run_ranged_parallel_round "$presigned_url" "$object_size" "$concurrency" "$range_workers" "$log_file" "$object_label" "$round")"
IFS=',' read -r status total_bytes elapsed_ms throughput_bps avg_latency_ms <<< "$result"
echo "${object_label},${object_key},${mode},${range_workers},${concurrency},${round},${status},${total_bytes},${elapsed_ms},${throughput_bps},${avg_latency_ms},${log_file}" >> "$ROUND_CSV"
if (( COOLDOWN_SECS > 0 && round < ROUNDS )); then
cooldown_sleep "$COOLDOWN_SECS"
fi
done
done
continue
fi
echo "ERROR: unsupported mode: $mode" >&2
exit 1
done
done
done
build_median_summary
echo "Artifacts written to: $OUT_DIR"
echo "Summaries:"
echo " - $ROUND_CSV"
echo " - $MEDIAN_CSV"
}
main "$@"