mirror of
https://github.com/rustfs/rustfs.git
synced 2026-07-26 08:18:18 +00:00
feat(get): consolidate GET performance optimization (#3972)
* feat(get): consolidate GET performance optimization Consolidated implementation of all GET performance optimizations into a single, well-organized commit replacing the previous patch-on-patch approach. ## Changes ### Configuration (set_disk/mod.rs) - Consolidated all GET optimization flags into a single organized section - Enabled by default: codec streaming, metadata early-stop, page cache reclaim - Added codec streaming multipart flag (default: disabled) - Added version-aware early-stop flag (default: disabled) - Added adaptive duplex buffer sizing based on object size - All flags use OnceLock caching with rollout percentage support ### Metadata Early-Stop (set_disk/read.rs) - Delete marker early-stop when quorum agrees - Version-aware early-stop for versioned GET requests - MetadataQuorumAccumulator enhanced with: - delete_marker_votes tracking - requested_version_id and matching_version_votes tracking - version_early_stop_decision() method - 6 new tests for version early-stop scenarios ### Codec Streaming (erasure/coding/decode_reader.rs) - DualInFlight (2-stripe lookahead) enabled by default ### Decode Pipeline (erasure/coding/decode.rs) - Stripe prefetch count configuration - Bitrot-decode overlap configuration ### Disk Layer (disk/local.rs) - O_DIRECT read configuration constants (preparation) ### Metrics (io-metrics/lib.rs) - BytesPool acquisition/return metrics - Metadata phase duration with early-stop label - Total duration with reader_path label ### Diagnostics (diagnostics/) - Early-stop reason constants - Pool tier/outcome label constants ### Observability (.docker/observability/) - 3 Grafana dashboards for GET optimization monitoring - Prometheus alert rules (6 alerts: 3 critical, 3 warning) - Updated README.md and README_ZH.md with usage docs ### Config (config/src/constants/runtime.rs) - Page cache reclaim read enabled by default ## Environment Variables | Variable | Default | Description | |----------|---------|-------------| | RUSTFS_GET_CODEC_STREAMING_ENABLE | true | Codec streaming base flag | | RUSTFS_GET_CODEC_STREAMING_ROLLOUT_PCT | 100 | Codec streaming rollout % | | RUSTFS_GET_CODEC_STREAMING_MULTIPART_ENABLE | false | Multipart codec streaming | | RUSTFS_GET_METADATA_EARLY_STOP_ENABLE | true | Early-stop base flag | | RUSTFS_GET_METADATA_EARLY_STOP_ROLLOUT_PCT | 100 | Early-stop rollout % | | RUSTFS_GET_METADATA_VERSION_EARLY_STOP_ENABLE | false | Version-aware early-stop | | RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE | true | Page cache reclaim | | RUSTFS_OBJECT_DIRECT_IO_READ_ENABLE | false | O_DIRECT (preparation) | | RUSTFS_GET_DECODE_STRIPE_PREFETCH_COUNT | 1 | Stripe prefetch | | RUSTFS_GET_BITROT_DECODE_OVERLAP_ENABLE | false | Bitrot-decode overlap | | RUSTFS_GET_CODEC_STREAMING_MAX_INFLIGHT | 2 | DualInFlight stripes | ## Rollback All optimizations can be disabled via environment variables: RUSTFS_GET_CODEC_STREAMING_ENABLE=false RUSTFS_GET_METADATA_EARLY_STOP_ENABLE=false RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE=false Co-Authored-By: heihutu <heihutu@gmail.com> * test(get): add stress test scripts for GET optimization validation - quick-validate-get-optimization.sh: Quick 5-minute validation - stress-test-get-optimization.sh: Full 30+ minute stress test - README-stress-test.md: Usage documentation Co-Authored-By: heihutu <heihutu@gmail.com> * test(ecstore): align file cache reclaim defaults * chore(deps): update redis and erasure codec * test(ecstore): align decode fill policy default * test(ecstore): align metadata early-stop default * fix(ecstore): keep metadata early stop opt-in --------- Co-authored-by: heihutu <heihutu@gmail.com>
This commit is contained in:
houseme
@@ -34,6 +34,56 @@ If you want the Kafka-backed HA Tempo path, use `docker-compose-example-for-rust
|
||||
- **High Performance**: Optimized configurations for batching, compression, and memory management.
|
||||
- **Standardized Protocols**: Built entirely on OpenTelemetry standards.
|
||||
|
||||
## GET Performance Optimization Dashboards
|
||||
|
||||
Three pre-built Grafana dashboards are included for monitoring RustFS GET performance optimization rollout:
|
||||
|
||||
### Available Dashboards
|
||||
|
||||
| Dashboard | File | Description |
|
||||
|-----------|------|-------------|
|
||||
| **GET Rollout Health** | `grafana-get-rollout-health.json` | Monitors optimization rollout: latency by reader path, early-stop hit rate, codec streaming usage, pipeline failures |
|
||||
| **GET Data Integrity** | `grafana-get-data-integrity.json` | Monitors data safety: bitrot verify failures, decode errors, short reads, shard read outcomes |
|
||||
| **GET Resource Impact** | `grafana-get-resource-impact.json` | Monitors resource usage: concurrent requests, IO queue utilization, disk permit wait, RSS trend |
|
||||
|
||||
### Prometheus Alert Rules
|
||||
|
||||
The file `prometheus-rules/rustfs-get-optimization-alerts.yaml` contains pre-configured alerting rules:
|
||||
|
||||
| Alert | Severity | Condition |
|
||||
|-------|----------|-----------|
|
||||
| `GetP99Regression` | Critical | GET p99 latency > 2x baseline for 10m |
|
||||
| `PipelineFailureSpike` | Critical | Pipeline failure rate > 5x baseline for 5m |
|
||||
| `BitrotMismatchSpike` | Critical | Bitrot mismatch rate > 3x baseline for 5m |
|
||||
| `EarlyStopInsufficientQuorum` | Warning | Early-stop insufficient quorum rate > 0.1/s for 5m |
|
||||
| `CodecStreamingFallbackSpike` | Warning | Codec streaming fallback > 10x baseline for 10m |
|
||||
| `IoQueueSaturation` | Warning | IO queue utilization > 90% for 5m |
|
||||
|
||||
### Enabling Alert Rules
|
||||
|
||||
Add the alert rules file to your Prometheus configuration:
|
||||
|
||||
```yaml
|
||||
# prometheus.yml
|
||||
rule_files:
|
||||
- "/etc/prometheus/rules/*.yml"
|
||||
|
||||
# Or mount the file in docker-compose.yml:
|
||||
# volumes:
|
||||
# - ./prometheus-rules:/etc/prometheus/rules
|
||||
```
|
||||
|
||||
### Dashboard Usage
|
||||
|
||||
The dashboards are automatically provisioned when Grafana starts. They use the `${DS_PROMETHEUS}` datasource variable, so you need a Prometheus datasource configured in Grafana.
|
||||
|
||||
Key panels to monitor during optimization rollout:
|
||||
|
||||
1. **GET Latency by Reader Path** - Compare `codec_streaming` vs `legacy_duplex` latency
|
||||
2. **Early-Stop Hit Rate** - Verify early-stop is triggering effectively
|
||||
3. **Pipeline Failure Rate** - Detect any new failure modes introduced by optimizations
|
||||
4. **Bitrot Verify Failures** - Ensure data integrity is maintained
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Prerequisites
|
||||
|
||||
@@ -34,6 +34,56 @@
|
||||
- **高性能**: 针对批处理、压缩和内存管理进行了优化配置。
|
||||
- **标准化协议**: 完全基于 OpenTelemetry 标准构建。
|
||||
|
||||
## GET 性能优化仪表盘
|
||||
|
||||
包含三个预构建的 Grafana 仪表盘，用于监控 RustFS GET 性能优化发布：
|
||||
|
||||
### 可用仪表盘
|
||||
|
||||
| 仪表盘 | 文件 | 描述 |
|
||||
|--------|------|------|
|
||||
| **GET 发布健康度** | `grafana-get-rollout-health.json` | 监控优化发布：按 reader path 的延迟、early-stop 命中率、codec streaming 使用率、pipeline 失败率 |
|
||||
| **GET 数据完整性** | `grafana-get-data-integrity.json` | 监控数据安全：bitrot 校验失败、decode 错误、short read、shard 读取结果 |
|
||||
| **GET 资源影响** | `grafana-get-resource-impact.json` | 监控资源使用：并发请求数、IO 队列利用率、disk permit 等待、RSS 趋势 |
|
||||
|
||||
### Prometheus 告警规则
|
||||
|
||||
文件 `prometheus-rules/rustfs-get-optimization-alerts.yaml` 包含预配置的告警规则：
|
||||
|
||||
| 告警 | 级别 | 条件 |
|
||||
|------|------|------|
|
||||
| `GetP99Regression` | 严重 | GET p99 延迟 > 2x 基线，持续 10 分钟 |
|
||||
| `PipelineFailureSpike` | 严重 | Pipeline 失败率 > 5x 基线，持续 5 分钟 |
|
||||
| `BitrotMismatchSpike` | 严重 | Bitrot 不匹配率 > 3x 基线，持续 5 分钟 |
|
||||
| `EarlyStopInsufficientQuorum` | 警告 | Early-stop quorum 不足率 > 0.1/s，持续 5 分钟 |
|
||||
| `CodecStreamingFallbackSpike` | 警告 | Codec streaming 回退 > 10x 基线，持续 10 分钟 |
|
||||
| `IoQueueSaturation` | 警告 | IO 队列利用率 > 90%，持续 5 分钟 |
|
||||
|
||||
### 启用告警规则
|
||||
|
||||
在 Prometheus 配置中添加告警规则文件：
|
||||
|
||||
```yaml
|
||||
# prometheus.yml
|
||||
rule_files:
|
||||
- "/etc/prometheus/rules/*.yml"
|
||||
|
||||
# 或在 docker-compose.yml 中挂载文件：
|
||||
# volumes:
|
||||
# - ./prometheus-rules:/etc/prometheus/rules
|
||||
```
|
||||
|
||||
### 仪表盘使用
|
||||
|
||||
仪表盘在 Grafana 启动时自动预置。它们使用 `${DS_PROMETHEUS}` 数据源变量，因此需要在 Grafana 中配置 Prometheus 数据源。
|
||||
|
||||
优化发布期间需要关注的关键面板：
|
||||
|
||||
1. **GET 延迟按 Reader Path** - 对比 `codec_streaming` vs `legacy_duplex` 延迟
|
||||
2. **Early-Stop 命中率** - 验证 early-stop 是否有效触发
|
||||
3. **Pipeline 失败率** - 检测优化引入的新故障模式
|
||||
4. **Bitrot 校验失败** - 确保数据完整性
|
||||
|
||||
## 快速开始
|
||||
|
||||
### 前置条件
|
||||
|
||||
@@ -0,0 +1,500 @@
|
||||
{
|
||||
"annotations": {
|
||||
"list": []
|
||||
},
|
||||
"editable": true,
|
||||
"fiscalYearStartMonth": 0,
|
||||
"graphTooltip": 1,
|
||||
"id": null,
|
||||
"links": [],
|
||||
"liveNow": false,
|
||||
"panels": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "ops",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"fillOpacity": 20,
|
||||
"pointSize": 5,
|
||||
"lineWidth": 1,
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"thresholdsStyle": {
|
||||
"mode": "line+area"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "transparent",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 1
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
},
|
||||
"id": 1,
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"values": false,
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": ""
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
},
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom",
|
||||
"calcs": ["mean", "max", "sum"]
|
||||
}
|
||||
},
|
||||
"title": "Bitrot Verify Failures",
|
||||
"type": "timeseries",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(rate(rustfs_io_get_object_pipeline_failures_total{stage=\"bitrot_verify\"}[$__rate_interval])) by (path, reason)",
|
||||
"legendFormat": "bitrot_fail {{path}} / {{reason}}",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(rate(rustfs_io_get_object_pipeline_failures_total{reason=\"bitrot_verify_failed\"}[$__rate_interval])) by (path, stage)",
|
||||
"legendFormat": "verify_failed {{path}} / {{stage}}",
|
||||
"refId": "B"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"fillOpacity": 10,
|
||||
"pointSize": 5,
|
||||
"lineWidth": 1,
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 0
|
||||
},
|
||||
"id": 2,
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"values": false,
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": ""
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
},
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom",
|
||||
"calcs": ["mean", "max"]
|
||||
}
|
||||
},
|
||||
"title": "Bitrot Verify Duration (p50 / p95 / p99)",
|
||||
"type": "timeseries",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "histogram_quantile(0.50, sum(rate(rustfs_io_get_object_shard_bitrot_verify_duration_seconds_bucket[$__rate_interval])) by (le))",
|
||||
"legendFormat": "p50",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "histogram_quantile(0.95, sum(rate(rustfs_io_get_object_shard_bitrot_verify_duration_seconds_bucket[$__rate_interval])) by (le))",
|
||||
"legendFormat": "p95",
|
||||
"refId": "B"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "histogram_quantile(0.99, sum(rate(rustfs_io_get_object_shard_bitrot_verify_duration_seconds_bucket[$__rate_interval])) by (le))",
|
||||
"legendFormat": "p99",
|
||||
"refId": "C"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "ops",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"fillOpacity": 20,
|
||||
"pointSize": 5,
|
||||
"lineWidth": 1,
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"thresholdsStyle": {
|
||||
"mode": "line+area"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "transparent",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 1
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 8
|
||||
},
|
||||
"id": 3,
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"values": false,
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": ""
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
},
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom",
|
||||
"calcs": ["mean", "max", "sum"]
|
||||
}
|
||||
},
|
||||
"title": "Decode Errors",
|
||||
"type": "timeseries",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(rate(rustfs_io_get_object_pipeline_failures_total{reason=\"decode_error\"}[$__rate_interval])) by (path, stage)",
|
||||
"legendFormat": "decode_error {{path}} / {{stage}}",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "ops",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"fillOpacity": 20,
|
||||
"pointSize": 5,
|
||||
"lineWidth": 1,
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"thresholdsStyle": {
|
||||
"mode": "line+area"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "transparent",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 1
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 8
|
||||
},
|
||||
"id": 4,
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"values": false,
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": ""
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
},
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom",
|
||||
"calcs": ["mean", "max", "sum"]
|
||||
}
|
||||
},
|
||||
"title": "Short Reads",
|
||||
"type": "timeseries",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(rate(rustfs_io_get_object_pipeline_failures_total{reason=\"short_read\"}[$__rate_interval])) by (path, stage)",
|
||||
"legendFormat": "short_read {{path}} / {{stage}}",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "ops",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"fillOpacity": 10,
|
||||
"pointSize": 5,
|
||||
"lineWidth": 1,
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 16
|
||||
},
|
||||
"id": 5,
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"values": false,
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": ""
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
},
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom",
|
||||
"calcs": ["mean", "max"]
|
||||
}
|
||||
},
|
||||
"title": "Shard Read Outcomes by Role",
|
||||
"type": "timeseries",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(rate(rustfs_io_get_object_shard_read_total[$__rate_interval])) by (path, role, outcome)",
|
||||
"legendFormat": "{{path}} {{role}}/{{outcome}}",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "ops",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"fillOpacity": 10,
|
||||
"pointSize": 5,
|
||||
"lineWidth": 1,
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 16
|
||||
},
|
||||
"id": 6,
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"values": false,
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": ""
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
},
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom",
|
||||
"calcs": ["mean", "max"]
|
||||
}
|
||||
},
|
||||
"title": "Request Result Status Rate",
|
||||
"type": "timeseries",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(rate(rustfs_io_get_object_request_results_total[$__rate_interval])) by (status)",
|
||||
"legendFormat": "{{status}}",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"refresh": "30s",
|
||||
"schemaVersion": 38,
|
||||
"style": "dark",
|
||||
"tags": ["rustfs", "get-optimization", "data-integrity"],
|
||||
"templating": {
|
||||
"list": [
|
||||
{
|
||||
"current": {
|
||||
"selected": false,
|
||||
"text": "Prometheus",
|
||||
"value": "Prometheus"
|
||||
},
|
||||
"hide": 0,
|
||||
"includeAll": false,
|
||||
"multi": false,
|
||||
"name": "DS_PROMETHEUS",
|
||||
"options": [],
|
||||
"query": "prometheus",
|
||||
"refresh": 1,
|
||||
"regex": "",
|
||||
"skipUrlSync": false,
|
||||
"type": "datasource"
|
||||
}
|
||||
]
|
||||
},
|
||||
"time": {
|
||||
"from": "now-1h",
|
||||
"to": "now"
|
||||
},
|
||||
"timepicker": {},
|
||||
"timezone": "",
|
||||
"title": "RustFS GET Data Integrity",
|
||||
"uid": "rustfs-get-data-integrity",
|
||||
"version": 1
|
||||
}
|
||||
@@ -0,0 +1,525 @@
|
||||
{
|
||||
"annotations": {
|
||||
"list": []
|
||||
},
|
||||
"editable": true,
|
||||
"fiscalYearStartMonth": 0,
|
||||
"graphTooltip": 1,
|
||||
"id": null,
|
||||
"links": [],
|
||||
"liveNow": false,
|
||||
"panels": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "none",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"fillOpacity": 20,
|
||||
"pointSize": 5,
|
||||
"lineWidth": 1,
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
},
|
||||
"id": 1,
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"values": false,
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": ""
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
},
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom",
|
||||
"calcs": ["mean", "max"]
|
||||
}
|
||||
},
|
||||
"title": "Concurrent GET Requests",
|
||||
"type": "timeseries",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "rustfs_io_get_object_concurrent_requests",
|
||||
"legendFormat": "concurrent GETs",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percent",
|
||||
"min": 0,
|
||||
"max": 100,
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"fillOpacity": 20,
|
||||
"pointSize": 5,
|
||||
"lineWidth": 1,
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"thresholdsStyle": {
|
||||
"mode": "line+area"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "transparent",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "yellow",
|
||||
"value": 70
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 90
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 0
|
||||
},
|
||||
"id": 2,
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"values": false,
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": ""
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
},
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom",
|
||||
"calcs": ["mean", "max"]
|
||||
}
|
||||
},
|
||||
"title": "IO Queue Utilization",
|
||||
"type": "timeseries",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "rustfs_io_queue_utilization_percent",
|
||||
"legendFormat": "queue utilization %",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "rustfs_io_queue_permits_in_use",
|
||||
"legendFormat": "permits in use",
|
||||
"refId": "B"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "rustfs_io_queue_permits_available",
|
||||
"legendFormat": "permits available",
|
||||
"refId": "C"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"fillOpacity": 10,
|
||||
"pointSize": 5,
|
||||
"lineWidth": 1,
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 8
|
||||
},
|
||||
"id": 3,
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"values": false,
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": ""
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
},
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom",
|
||||
"calcs": ["mean", "max"]
|
||||
}
|
||||
},
|
||||
"title": "Disk Permit Wait Duration (p50 / p95 / p99)",
|
||||
"type": "timeseries",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "histogram_quantile(0.50, sum(rate(rustfs_io_disk_permit_wait_duration_seconds_bucket[$__rate_interval])) by (le))",
|
||||
"legendFormat": "p50",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "histogram_quantile(0.95, sum(rate(rustfs_io_disk_permit_wait_duration_seconds_bucket[$__rate_interval])) by (le))",
|
||||
"legendFormat": "p95",
|
||||
"refId": "B"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "histogram_quantile(0.99, sum(rate(rustfs_io_disk_permit_wait_duration_seconds_bucket[$__rate_interval])) by (le))",
|
||||
"legendFormat": "p99",
|
||||
"refId": "C"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "bytes",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"fillOpacity": 15,
|
||||
"pointSize": 5,
|
||||
"lineWidth": 1,
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 8
|
||||
},
|
||||
"id": 4,
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"values": false,
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": ""
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
},
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom",
|
||||
"calcs": ["lastNotNull"]
|
||||
}
|
||||
},
|
||||
"title": "RSS Trend (Process Memory)",
|
||||
"type": "timeseries",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "process_resident_memory_bytes{job=~\"rustfs.*\"}",
|
||||
"legendFormat": "RSS {{instance}}",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "rustfs_process_memory_bytes",
|
||||
"legendFormat": "RustFS memory {{instance}}",
|
||||
"refId": "B"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "ops",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"fillOpacity": 10,
|
||||
"pointSize": 5,
|
||||
"lineWidth": 1,
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 16
|
||||
},
|
||||
"id": 5,
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"values": false,
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": ""
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
},
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom",
|
||||
"calcs": ["mean", "max"]
|
||||
}
|
||||
},
|
||||
"title": "IO Queue Operations by Priority",
|
||||
"type": "timeseries",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(rate(rustfs_io_queue_operations[$__rate_interval])) by (operation, priority)",
|
||||
"legendFormat": "{{operation}} / {{priority}}",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "ops",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"fillOpacity": 15,
|
||||
"pointSize": 5,
|
||||
"lineWidth": 1,
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 16
|
||||
},
|
||||
"id": 6,
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"values": false,
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": ""
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
},
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom",
|
||||
"calcs": ["sum"]
|
||||
}
|
||||
},
|
||||
"title": "IO Queue Congestion Events",
|
||||
"type": "timeseries",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(rate(rustfs_io_queue_congestion_total[$__rate_interval]))",
|
||||
"legendFormat": "congestion events/s",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(rate(rustfs_io_get_object_timeout_total[$__rate_interval])) by (stage)",
|
||||
"legendFormat": "timeout {{stage}}",
|
||||
"refId": "B"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"refresh": "30s",
|
||||
"schemaVersion": 38,
|
||||
"style": "dark",
|
||||
"tags": ["rustfs", "get-optimization", "resource-impact"],
|
||||
"templating": {
|
||||
"list": [
|
||||
{
|
||||
"current": {
|
||||
"selected": false,
|
||||
"text": "Prometheus",
|
||||
"value": "Prometheus"
|
||||
},
|
||||
"hide": 0,
|
||||
"includeAll": false,
|
||||
"multi": false,
|
||||
"name": "DS_PROMETHEUS",
|
||||
"options": [],
|
||||
"query": "prometheus",
|
||||
"refresh": 1,
|
||||
"regex": "",
|
||||
"skipUrlSync": false,
|
||||
"type": "datasource"
|
||||
}
|
||||
]
|
||||
},
|
||||
"time": {
|
||||
"from": "now-1h",
|
||||
"to": "now"
|
||||
},
|
||||
"timepicker": {},
|
||||
"timezone": "",
|
||||
"title": "RustFS GET Resource Impact",
|
||||
"uid": "rustfs-get-resource-impact",
|
||||
"version": 1
|
||||
}
|
||||
@@ -0,0 +1,530 @@
|
||||
{
|
||||
"annotations": {
|
||||
"list": []
|
||||
},
|
||||
"editable": true,
|
||||
"fiscalYearStartMonth": 0,
|
||||
"graphTooltip": 1,
|
||||
"id": null,
|
||||
"links": [],
|
||||
"liveNow": false,
|
||||
"panels": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"fillOpacity": 10,
|
||||
"pointSize": 5,
|
||||
"lineWidth": 1,
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"axisLabel": "",
|
||||
"axisColorMode": "text",
|
||||
"scaleDistribution": {
|
||||
"type": "linear"
|
||||
},
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
},
|
||||
"id": 1,
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"values": false,
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": ""
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
},
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom",
|
||||
"calcs": ["mean", "max"]
|
||||
}
|
||||
},
|
||||
"title": "GET Latency by Reader Path (p50 / p95 / p99)",
|
||||
"type": "timeseries",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "histogram_quantile(0.50, sum(rate(rustfs_io_get_object_stage_duration_seconds_bucket{stage=\"total\"}[$__rate_interval])) by (le, path))",
|
||||
"legendFormat": "p50 {{path}}",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "histogram_quantile(0.95, sum(rate(rustfs_io_get_object_stage_duration_seconds_bucket{stage=\"total\"}[$__rate_interval])) by (le, path))",
|
||||
"legendFormat": "p95 {{path}}",
|
||||
"refId": "B"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "histogram_quantile(0.99, sum(rate(rustfs_io_get_object_stage_duration_seconds_bucket{stage=\"total\"}[$__rate_interval])) by (le, path))",
|
||||
"legendFormat": "p99 {{path}}",
|
||||
"refId": "C"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percentunit",
|
||||
"min": 0,
|
||||
"max": 1,
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"fillOpacity": 15,
|
||||
"pointSize": 5,
|
||||
"lineWidth": 1,
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "yellow",
|
||||
"value": 0.3
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 0.7
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 0
|
||||
},
|
||||
"id": 2,
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"values": false,
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": ""
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
},
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom",
|
||||
"calcs": ["mean", "max"]
|
||||
}
|
||||
},
|
||||
"title": "Early-Stop Hit Rate",
|
||||
"type": "timeseries",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(rate(rustfs_io_get_object_metadata_early_stop_total{decision=\"hit\"}[$__rate_interval])) by (path) / clamp_min(sum(rate(rustfs_io_get_object_metadata_early_stop_total[$__rate_interval])) by (path), 1)",
|
||||
"legendFormat": "hit rate {{path}}",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percentunit",
|
||||
"min": 0,
|
||||
"max": 1,
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"fillOpacity": 15,
|
||||
"pointSize": 5,
|
||||
"lineWidth": 1,
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 8
|
||||
},
|
||||
"id": 3,
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"values": false,
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": ""
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
},
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom",
|
||||
"calcs": ["mean", "max"]
|
||||
}
|
||||
},
|
||||
"title": "Codec Streaming Usage Rate",
|
||||
"type": "timeseries",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(rate(rustfs_io_get_object_codec_streaming_decision_total[$__rate_interval])) by (decision) / clamp_min(sum(rate(rustfs_io_get_object_codec_streaming_decision_total[$__rate_interval])), 1)",
|
||||
"legendFormat": "codec_streaming {{decision}}",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(rate(rustfs_io_get_object_stream_strategy_total[$__rate_interval])) by (strategy) / clamp_min(sum(rate(rustfs_io_get_object_stream_strategy_total[$__rate_interval])), 1)",
|
||||
"legendFormat": "stream_strategy {{strategy}}",
|
||||
"refId": "B"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "ops",
|
||||
"custom": {
|
||||
"drawStyle": "bars",
|
||||
"fillOpacity": 80,
|
||||
"lineWidth": 0,
|
||||
"pointSize": 5,
|
||||
"showPoints": "never",
|
||||
"spanNulls": false,
|
||||
"stacking": {
|
||||
"mode": "normal",
|
||||
"group": "A"
|
||||
},
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 1
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 8
|
||||
},
|
||||
"id": 4,
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"values": false,
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": ""
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
},
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom",
|
||||
"calcs": ["sum"]
|
||||
}
|
||||
},
|
||||
"title": "Pipeline Failure Rate by Stage",
|
||||
"type": "timeseries",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(rate(rustfs_io_get_object_pipeline_failures_total[$__rate_interval])) by (stage, reason)",
|
||||
"legendFormat": "{{stage}} / {{reason}}",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "reqps",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"fillOpacity": 10,
|
||||
"pointSize": 5,
|
||||
"lineWidth": 1,
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 16
|
||||
},
|
||||
"id": 5,
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"values": false,
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": ""
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
},
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom",
|
||||
"calcs": ["mean", "max"]
|
||||
}
|
||||
},
|
||||
"title": "GET Request Rate by Path",
|
||||
"type": "timeseries",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(rate(rustfs_io_get_object_reader_path_total[$__rate_interval])) by (path)",
|
||||
"legendFormat": "{{path}}",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "ops",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"fillOpacity": 10,
|
||||
"pointSize": 5,
|
||||
"lineWidth": 1,
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 16
|
||||
},
|
||||
"id": 6,
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"values": false,
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": ""
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
},
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom",
|
||||
"calcs": ["mean", "max"]
|
||||
}
|
||||
},
|
||||
"title": "GET Total Latency (p50 / p95 / p99)",
|
||||
"type": "timeseries",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "histogram_quantile(0.50, sum(rate(rustfs_io_get_object_total_duration_seconds_bucket[$__rate_interval])) by (le))",
|
||||
"legendFormat": "p50",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "histogram_quantile(0.95, sum(rate(rustfs_io_get_object_total_duration_seconds_bucket[$__rate_interval])) by (le))",
|
||||
"legendFormat": "p95",
|
||||
"refId": "B"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "histogram_quantile(0.99, sum(rate(rustfs_io_get_object_total_duration_seconds_bucket[$__rate_interval])) by (le))",
|
||||
"legendFormat": "p99",
|
||||
"refId": "C"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"refresh": "30s",
|
||||
"schemaVersion": 38,
|
||||
"style": "dark",
|
||||
"tags": ["rustfs", "get-optimization"],
|
||||
"templating": {
|
||||
"list": [
|
||||
{
|
||||
"current": {
|
||||
"selected": false,
|
||||
"text": "Prometheus",
|
||||
"value": "Prometheus"
|
||||
},
|
||||
"hide": 0,
|
||||
"includeAll": false,
|
||||
"multi": false,
|
||||
"name": "DS_PROMETHEUS",
|
||||
"options": [],
|
||||
"query": "prometheus",
|
||||
"refresh": 1,
|
||||
"regex": "",
|
||||
"skipUrlSync": false,
|
||||
"type": "datasource"
|
||||
}
|
||||
]
|
||||
},
|
||||
"time": {
|
||||
"from": "now-1h",
|
||||
"to": "now"
|
||||
},
|
||||
"timepicker": {},
|
||||
"timezone": "",
|
||||
"title": "RustFS GET Rollout Health",
|
||||
"uid": "rustfs-get-rollout-health",
|
||||
"version": 1
|
||||
}
|
||||
@@ -0,0 +1,260 @@
|
||||
# =============================================================================
|
||||
# RustFS GET Optimization — Prometheus Alerting Rules
|
||||
# =============================================================================
|
||||
#
|
||||
# Import into Prometheus:
|
||||
# 1. Copy this file to your Prometheus rules directory
|
||||
# 2. Add to prometheus.yml:
|
||||
# rule_files:
|
||||
# - "prometheus-alert-rules.yaml"
|
||||
# 3. Validate: promtool check rules prometheus-alert-rules.yaml
|
||||
# 4. Reload: curl -X POST http://localhost:9090/-/reload
|
||||
#
|
||||
# All metric names match those registered in crates/io-metrics/src/lib.rs
|
||||
# and documented in crates/ecstore/src/diagnostics/get.rs.
|
||||
#
|
||||
# Baseline comparison uses "offset 1d" — adjust to "offset 7d" for weekly
|
||||
# seasonality if your traffic pattern varies by day of week.
|
||||
# =============================================================================
|
||||
|
||||
groups:
|
||||
# ==========================================================================
|
||||
# Critical alerts — immediate action required
|
||||
# ==========================================================================
|
||||
- name: rustfs-get-optimization-critical
|
||||
interval: 30s
|
||||
rules:
|
||||
# ------------------------------------------------------------------
|
||||
# 1. GetP99Regression
|
||||
# GET p99 latency exceeds 2x the baseline (same time yesterday)
|
||||
# sustained for 10 minutes.
|
||||
# Action: Roll back the GET optimization immediately.
|
||||
# ------------------------------------------------------------------
|
||||
- alert: GetP99Regression
|
||||
expr: |
|
||||
histogram_quantile(0.99,
|
||||
sum(rate(rustfs_io_get_object_total_duration_seconds_bucket[5m])) by (le)
|
||||
)
|
||||
>
|
||||
2
|
||||
*
|
||||
histogram_quantile(0.99,
|
||||
sum(rate(rustfs_io_get_object_total_duration_seconds_bucket[5m] offset 1d)) by (le)
|
||||
)
|
||||
for: 10m
|
||||
labels:
|
||||
severity: critical
|
||||
team: rustfs-storage
|
||||
area: get-optimization
|
||||
annotations:
|
||||
summary: "GET p99 latency regression detected (>2x baseline for 10m)"
|
||||
description: >-
|
||||
The 99th-percentile GET object latency is {{ $value | humanizeDuration }}
|
||||
which is more than double the baseline measured 24 hours ago.
|
||||
This indicates a severe performance regression introduced by
|
||||
a recent GET optimization change.
|
||||
runbook_url: "https://internal.wiki/runbooks/rustfs/get-p99-regression"
|
||||
action: >
|
||||
1. Verify the regression is not caused by external factors
|
||||
(disk health, network, load spike).
|
||||
2. If confirmed optimization-related, roll back:
|
||||
- Set RUSTFS_GET_CODEC_STREAMING=0
|
||||
- Set RUSTFS_GET_METADATA_EARLY_STOP=0
|
||||
- Restart affected nodes.
|
||||
3. Collect flamegraphs and open a P0 incident.
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# 2. PipelineFailureSpike
|
||||
# Pipeline failure rate exceeds 5x the baseline sustained for
|
||||
# 5 minutes. Covers all failure reasons: bitrot_mismatch,
|
||||
# decode_error, downstream_closed, io, read_quorum, timeout, etc.
|
||||
# Action: Investigate pipeline health and roll back if needed.
|
||||
# ------------------------------------------------------------------
|
||||
- alert: PipelineFailureSpike
|
||||
expr: |
|
||||
sum(rate(rustfs_io_get_object_pipeline_failures_total[5m]))
|
||||
>
|
||||
5
|
||||
*
|
||||
sum(rate(rustfs_io_get_object_pipeline_failures_total[5m] offset 1d))
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
team: rustfs-storage
|
||||
area: get-optimization
|
||||
annotations:
|
||||
summary: "GET pipeline failure rate spike (>5x baseline for 5m)"
|
||||
description: >-
|
||||
The GET pipeline failure rate is {{ $value | printf "%.2f" }}/s,
|
||||
more than 5x the baseline from 24 hours ago.
|
||||
Failure reasons may include: bitrot_mismatch, decode_error,
|
||||
downstream_closed, io, range_or_length_invalid, read_quorum,
|
||||
short_read, timeout, unknown.
|
||||
runbook_url: "https://internal.wiki/runbooks/rustfs/pipeline-failure-spike"
|
||||
action: >
|
||||
1. Check Grafana "GET Data Integrity" dashboard for failure
|
||||
breakdown by reason label.
|
||||
2. If decode_error or bitrot_mismatch dominates, stop
|
||||
optimization and investigate data integrity.
|
||||
3. If io or timeout dominates, check disk and network health.
|
||||
4. Roll back optimization if failures persist.
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# 3. BitrotMismatchSpike
|
||||
# Bitrot verification mismatch rate exceeds 3x baseline for
|
||||
# 5 minutes. This is a data-integrity signal — shard checksums
|
||||
# do not match after read.
|
||||
#
|
||||
# The "bitrot_mismatch" reason is recorded on the
|
||||
# rustfs_io_get_object_pipeline_failures_total counter when a
|
||||
# StorageError::FileCorrupt or DiskError::FileCorrupt /
|
||||
# DiskError::PartMissingOrCorrupt is classified during the GET
|
||||
# pipeline (see classify_storage_error / classify_disk_error in
|
||||
# crates/ecstore/src/diagnostics/get.rs).
|
||||
#
|
||||
# Action: Stop optimization, investigate data integrity urgently.
|
||||
# ------------------------------------------------------------------
|
||||
- alert: BitrotMismatchSpike
|
||||
expr: |
|
||||
sum(rate(rustfs_io_get_object_pipeline_failures_total{reason="bitrot_mismatch"}[5m]))
|
||||
>
|
||||
3
|
||||
*
|
||||
sum(rate(rustfs_io_get_object_pipeline_failures_total{reason="bitrot_mismatch"}[5m] offset 1d))
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
team: rustfs-storage
|
||||
area: get-optimization
|
||||
annotations:
|
||||
summary: "Bitrot mismatch rate spike (>3x baseline for 5m)"
|
||||
description: >-
|
||||
The rate of pipeline failures classified as bitrot_mismatch is
|
||||
{{ $value | printf "%.2f" }}/s, more than 3x the baseline from
|
||||
24 hours ago. This indicates shard checksum verification
|
||||
failures (FileCorrupt / PartMissingOrCorrupt) which may point
|
||||
to data corruption introduced by the GET optimization pipeline
|
||||
(e.g., incorrect decode, buffer reuse bug).
|
||||
runbook_url: "https://internal.wiki/runbooks/rustfs/bitrot-mismatch-spike"
|
||||
action: >
|
||||
1. Immediately disable codec streaming:
|
||||
RUSTFS_GET_CODEC_STREAMING=0
|
||||
2. Run "mc admin scan" on affected buckets to verify on-disk
|
||||
integrity independent of the GET path.
|
||||
3. Compare xl.meta checksums across erasure shards.
|
||||
4. If corruption confirmed, initiate data recovery from parity.
|
||||
5. Do NOT re-enable optimization until root cause is identified.
|
||||
|
||||
# ==========================================================================
|
||||
# Warning alerts — investigation needed
|
||||
# ==========================================================================
|
||||
- name: rustfs-get-optimization-warning
|
||||
interval: 30s
|
||||
rules:
|
||||
# ------------------------------------------------------------------
|
||||
# 4. EarlyStopInsufficientQuorum
|
||||
# The metadata early-stop path is hitting "insufficient_quorum"
|
||||
# at a rate above 0.1/s for 5 minutes. This means too many
|
||||
# disks are failing to return valid metadata in time.
|
||||
# Action: Check disk health and metadata fanout latency.
|
||||
# ------------------------------------------------------------------
|
||||
- alert: EarlyStopInsufficientQuorum
|
||||
expr: |
|
||||
sum(rate(rustfs_io_get_object_metadata_early_stop_total{reason="insufficient_quorum"}[5m]))
|
||||
> 0.1
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
team: rustfs-storage
|
||||
area: get-optimization
|
||||
annotations:
|
||||
summary: "Early-stop insufficient quorum rate elevated (>0.1/s for 5m)"
|
||||
description: >-
|
||||
The metadata early-stop path is returning "insufficient_quorum"
|
||||
at {{ $value | printf "%.3f" }}/s. This means the bounded
|
||||
metadata fanout cannot gather enough valid responses before
|
||||
the quorum deadline, suggesting disk or network issues.
|
||||
runbook_url: "https://internal.wiki/runbooks/rustfs/early-stop-quorum"
|
||||
action: >
|
||||
1. Check disk health: mc admin info --json | jq '.disks'
|
||||
2. Review rustfs_io_get_object_metadata_response_total by
|
||||
outcome (error, timeout, disk_not_found) in Grafana.
|
||||
3. Check rustfs_io_disk_permit_wait_duration_seconds for
|
||||
I/O scheduler saturation.
|
||||
4. If disks are healthy, consider increasing the early-stop
|
||||
timeout or temporarily disabling early-stop.
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# 5. CodecStreamingFallbackSpike
|
||||
# The codec streaming fallback rate is >10x baseline for 10
|
||||
# minutes. This means the optimized codec streaming path is
|
||||
# being bypassed much more often than expected.
|
||||
# Action: Check fallback reasons and object eligibility.
|
||||
# ------------------------------------------------------------------
|
||||
- alert: CodecStreamingFallbackSpike
|
||||
expr: |
|
||||
sum(rate(rustfs_io_get_object_codec_streaming_fallback_total[5m]))
|
||||
>
|
||||
10
|
||||
*
|
||||
sum(rate(rustfs_io_get_object_codec_streaming_fallback_total[5m] offset 1d))
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
team: rustfs-storage
|
||||
area: get-optimization
|
||||
annotations:
|
||||
summary: "Codec streaming fallback rate spike (>10x baseline for 10m)"
|
||||
description: >-
|
||||
The codec streaming fallback rate is {{ $value | printf "%.2f" }}/s,
|
||||
more than 10x the baseline from 24 hours ago. Fallback reasons
|
||||
are labeled by "reason" — check Grafana for breakdown.
|
||||
Common reasons: object too small, multipart not supported,
|
||||
unsupported erasure layout, feature flag disabled.
|
||||
runbook_url: "https://internal.wiki/runbooks/rustfs/codec-fallback-spike"
|
||||
action: >
|
||||
1. Query by reason label:
|
||||
sum by (reason) (rate(rustfs_io_get_object_codec_streaming_fallback_total[5m]))
|
||||
2. If dominated by a single reason, investigate why that
|
||||
condition became more frequent (e.g., workload change,
|
||||
configuration drift).
|
||||
3. Cross-reference with rustfs_io_get_object_reader_path_total
|
||||
to verify the fallback path (legacy_duplex) is healthy.
|
||||
4. If fallback is expected (e.g., workload shifted to small
|
||||
objects), update the alert baseline.
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# 6. IoQueueSaturation
|
||||
# I/O queue utilization exceeds 90% for 5 minutes. High
|
||||
# utilization causes disk permit wait latency to increase and
|
||||
# can cascade into pipeline timeouts.
|
||||
# Action: Check disk load and consider reducing concurrency.
|
||||
# ------------------------------------------------------------------
|
||||
- alert: IoQueueSaturation
|
||||
expr: |
|
||||
rustfs_io_queue_utilization_percent > 90
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
team: rustfs-storage
|
||||
area: get-optimization
|
||||
annotations:
|
||||
summary: "I/O queue utilization >90% for 5m"
|
||||
description: >-
|
||||
The I/O queue utilization is {{ $value | printf "%.1f" }}%,
|
||||
sustained above 90% for 5 minutes. This indicates the disk
|
||||
I/O scheduler is near saturation, which will increase
|
||||
rustfs_io_disk_permit_wait_duration_seconds and may trigger
|
||||
pipeline timeouts.
|
||||
runbook_url: "https://internal.wiki/runbooks/rustfs/io-queue-saturation"
|
||||
action: >
|
||||
1. Check disk I/O metrics (iostat, node_exporter) for
|
||||
individual disk saturation.
|
||||
2. Review rustfs_io_queue_permits_in_use vs
|
||||
rustfs_io_queue_permits_available for permit exhaustion.
|
||||
3. Check rustfs_io_starvation_events for priority starvation.
|
||||
4. If GET optimization increased concurrency, consider:
|
||||
- Reducing RUSTFS_GET_PIPELINE_PARALLELISM
|
||||
- Lowering RUSTFS_IO_QUEUE_PERMITS
|
||||
5. If caused by background operations (ILM, healing), throttle
|
||||
those before adjusting GET concurrency.
|
||||
Generated
+4
-4
@@ -8486,9 +8486,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "redis"
|
||||
version = "1.2.4"
|
||||
version = "1.3.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "bae41a63fd0b8a5372f82b21e810e09a316f5dd7efd96bf08e678fb240fc1918"
|
||||
checksum = "2fa6f8e4b491d7a8ef3a9550a4d71969bd0064f46e32b8dbbcc7fc60dad94fed"
|
||||
dependencies = [
|
||||
"arc-swap",
|
||||
"arcstr",
|
||||
@@ -9357,9 +9357,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-erasure-codec"
|
||||
version = "7.0.0"
|
||||
version = "7.0.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "fcda6886ece4bfc6917640007a6b9693dc8424a9412d18c3dda5be2431500f31"
|
||||
checksum = "c31c2f2ff65a5075aad1fe29d350f1a154bdb097ff0057ee846ec1faf5701098"
|
||||
dependencies = [
|
||||
"cc",
|
||||
"hashlink",
|
||||
|
||||
+2
-2
@@ -264,12 +264,12 @@ pretty_assertions = "1.4.1"
|
||||
rand = { version = "0.10.1", features = ["serde"] }
|
||||
ratelimit = "0.10.1"
|
||||
rayon = "1.12.0"
|
||||
reed-solomon-erasure = { package = "rustfs-erasure-codec", version = "7.0.0", features = ["simd-accel"] }
|
||||
reed-solomon-erasure = { package = "rustfs-erasure-codec", version = "7.0.1", features = ["simd-accel"] }
|
||||
#reed-solomon-erasure = { version = "6.0", features = ["simd-accel"], git = "https://github.com/houseme/reed-solomon-erasure",rev = "main" }
|
||||
reed-solomon-simd = "3.1.0"
|
||||
regex = { version = "1.12.4" }
|
||||
rumqttc = { package = "rumqttc-next", version = "0.33.2", features = ["websocket"] }
|
||||
redis = { version = "1.2.4", features = ["connection-manager", "tokio-rustls-comp", "tls-rustls-insecure"] }
|
||||
redis = { version = "1.3.0", features = ["connection-manager", "tokio-rustls-comp", "tls-rustls-insecure"] }
|
||||
rustix = { version = "1.1.4", features = ["fs"] }
|
||||
rust-embed = { version = "8.11.0" }
|
||||
rustc-hash = { version = "2.1.2" }
|
||||
|
||||
@@ -97,7 +97,7 @@ pub const ENV_OBJECT_FILE_CACHE_RECLAIM_WRITE_ENABLE: &str = "RUSTFS_OBJECT_FILE
|
||||
pub const ENV_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE: &str = "RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE";
|
||||
pub const ENV_OBJECT_FILE_CACHE_RECLAIM_THRESHOLD: &str = "RUSTFS_OBJECT_FILE_CACHE_RECLAIM_THRESHOLD";
|
||||
pub const DEFAULT_OBJECT_FILE_CACHE_RECLAIM_WRITE_ENABLE: bool = false;
|
||||
pub const DEFAULT_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE: bool = false;
|
||||
pub const DEFAULT_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE: bool = true;
|
||||
pub const DEFAULT_OBJECT_FILE_CACHE_RECLAIM_THRESHOLD: usize = 4 * 1024 * 1024;
|
||||
|
||||
/// Threshold for small object seek support in bytes.
|
||||
|
||||
@@ -94,6 +94,12 @@ pub(crate) const GET_METADATA_EARLY_STOP_REASON_NOT_FOUND: &str = "not_found";
|
||||
pub(crate) const GET_METADATA_EARLY_STOP_REASON_UNSAFE_REQUEST: &str = "unsafe_request";
|
||||
pub(crate) const GET_METADATA_EARLY_STOP_REASON_VALID_QUORUM: &str = "valid_quorum";
|
||||
pub(crate) const GET_METADATA_EARLY_STOP_REASON_VERSION_NOT_FOUND: &str = "version_not_found";
|
||||
pub(crate) const GET_METADATA_EARLY_STOP_REASON_VERSION_MATCH_QUORUM: &str = "version_match_quorum";
|
||||
|
||||
/// Early-stop active state labels
|
||||
pub(crate) const EARLY_STOP_ACTIVE_HIT: &str = "hit";
|
||||
pub(crate) const EARLY_STOP_ACTIVE_MISS: &str = "miss";
|
||||
pub(crate) const EARLY_STOP_ACTIVE_DISABLED: &str = "disabled";
|
||||
|
||||
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
|
||||
pub(crate) enum GetObjectFailureReason {
|
||||
@@ -296,5 +302,9 @@ mod tests {
|
||||
assert_eq!(GET_METADATA_EARLY_STOP_REASON_UNSAFE_REQUEST, "unsafe_request");
|
||||
assert_eq!(GET_METADATA_EARLY_STOP_REASON_VALID_QUORUM, "valid_quorum");
|
||||
assert_eq!(GET_METADATA_EARLY_STOP_REASON_VERSION_NOT_FOUND, "version_not_found");
|
||||
assert_eq!(GET_METADATA_EARLY_STOP_REASON_VERSION_MATCH_QUORUM, "version_match_quorum");
|
||||
assert_eq!(EARLY_STOP_ACTIVE_HIT, "hit");
|
||||
assert_eq!(EARLY_STOP_ACTIVE_MISS, "miss");
|
||||
assert_eq!(EARLY_STOP_ACTIVE_DISABLED, "disabled");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -14,3 +14,4 @@
|
||||
|
||||
pub(crate) mod admin_server_info;
|
||||
pub(crate) mod get;
|
||||
pub(crate) mod pool;
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! BytesPool metric label constants.
|
||||
//!
|
||||
//! These constants are used when recording pool acquisition and return
|
||||
//! metrics to avoid string allocations and ensure label consistency.
|
||||
|
||||
/// BytesPool tier labels
|
||||
pub const POOL_TIER_SMALL: &str = "small";
|
||||
pub const POOL_TIER_MEDIUM: &str = "medium";
|
||||
pub const POOL_TIER_LARGE: &str = "large";
|
||||
pub const POOL_TIER_XLARGE: &str = "xlarge";
|
||||
|
||||
/// BytesPool outcome labels
|
||||
pub const POOL_OUTCOME_HIT: &str = "hit";
|
||||
pub const POOL_OUTCOME_MISS: &str = "miss";
|
||||
pub const POOL_OUTCOME_RECYCLED: &str = "recycled";
|
||||
pub const POOL_OUTCOME_DROPPED: &str = "dropped";
|
||||
@@ -84,6 +84,29 @@ const EVENT_DISK_LOCAL_ACCESS_FAILED: &str = "disk_local_access_failed";
|
||||
const EVENT_DISK_LOCAL_VOLUME_SETUP_FAILED: &str = "disk_local_volume_setup_failed";
|
||||
const EVENT_DISK_LOCAL_FORMAT_DECODE_FAILED: &str = "disk_local_format_decode_failed";
|
||||
|
||||
/// Enable O_DIRECT for large sequential reads.
|
||||
/// When enabled, shard reads bypass the page cache using O_DIRECT flag.
|
||||
/// Requires aligned buffers (typically 512 bytes or 4096 bytes).
|
||||
/// Default: false (uses page cache via mmap/pread).
|
||||
const ENV_RUSTFS_OBJECT_DIRECT_IO_READ_ENABLE: &str = "RUSTFS_OBJECT_DIRECT_IO_READ_ENABLE";
|
||||
const DEFAULT_RUSTFS_OBJECT_DIRECT_IO_READ_ENABLE: bool = false;
|
||||
|
||||
/// Minimum shard size threshold for O_DIRECT reads.
|
||||
/// Only shards larger than this threshold will use O_DIRECT.
|
||||
/// Default: 4MB.
|
||||
const ENV_RUSTFS_OBJECT_DIRECT_IO_READ_THRESHOLD: &str = "RUSTFS_OBJECT_DIRECT_IO_READ_THRESHOLD";
|
||||
const DEFAULT_RUSTFS_OBJECT_DIRECT_IO_READ_THRESHOLD: usize = 4 * 1024 * 1024;
|
||||
|
||||
/// Check if O_DIRECT reads are enabled.
|
||||
fn is_direct_io_read_enabled() -> bool {
|
||||
rustfs_utils::get_env_bool(ENV_RUSTFS_OBJECT_DIRECT_IO_READ_ENABLE, DEFAULT_RUSTFS_OBJECT_DIRECT_IO_READ_ENABLE)
|
||||
}
|
||||
|
||||
/// Get the O_DIRECT read threshold size.
|
||||
fn get_direct_io_read_threshold() -> usize {
|
||||
rustfs_utils::get_env_usize(ENV_RUSTFS_OBJECT_DIRECT_IO_READ_THRESHOLD, DEFAULT_RUSTFS_OBJECT_DIRECT_IO_READ_THRESHOLD)
|
||||
}
|
||||
|
||||
fn log_startup_disk_io_error(stage: &str, path: &Path, err: &IoError) {
|
||||
warn!(
|
||||
event = EVENT_DISK_LOCAL_STARTUP_CLEANUP,
|
||||
@@ -4982,7 +5005,16 @@ mod test {
|
||||
#[test]
|
||||
fn should_reclaim_file_cache_after_read_respects_env_and_threshold() {
|
||||
temp_env::with_var_unset(rustfs_config::ENV_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE, || {
|
||||
assert!(!should_reclaim_file_cache_after_read(8 * 1024 * 1024));
|
||||
temp_env::with_var_unset(rustfs_config::ENV_OBJECT_FILE_CACHE_RECLAIM_THRESHOLD, || {
|
||||
assert!(should_reclaim_file_cache_after_read(8 * 1024 * 1024));
|
||||
assert!(!should_reclaim_file_cache_after_read(1024));
|
||||
});
|
||||
});
|
||||
|
||||
temp_env::with_var(rustfs_config::ENV_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE, Some("false"), || {
|
||||
temp_env::with_var(rustfs_config::ENV_OBJECT_FILE_CACHE_RECLAIM_THRESHOLD, Some("4194304"), || {
|
||||
assert!(!should_reclaim_file_cache_after_read(8 * 1024 * 1024));
|
||||
});
|
||||
});
|
||||
|
||||
temp_env::with_var(rustfs_config::ENV_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE, Some("true"), || {
|
||||
|
||||
@@ -49,6 +49,36 @@ fn get_shard_locality_preference_enabled() -> bool {
|
||||
)
|
||||
}
|
||||
|
||||
/// Number of stripes to prefetch in the legacy decode path.
|
||||
/// When > 1, stripe reads are batched to overlap disk I/O with decode.
|
||||
/// Default: 1 (no prefetch, current behavior).
|
||||
/// Set via environment variable RUSTFS_GET_DECODE_STRIPE_PREFETCH_COUNT.
|
||||
const ENV_RUSTFS_GET_DECODE_STRIPE_PREFETCH_COUNT: &str = "RUSTFS_GET_DECODE_STRIPE_PREFETCH_COUNT";
|
||||
const DEFAULT_RUSTFS_GET_DECODE_STRIPE_PREFETCH_COUNT: usize = 1;
|
||||
|
||||
/// Get the stripe prefetch count from environment variable.
|
||||
fn get_decode_stripe_prefetch_count() -> usize {
|
||||
rustfs_utils::get_env_usize(
|
||||
ENV_RUSTFS_GET_DECODE_STRIPE_PREFETCH_COUNT,
|
||||
DEFAULT_RUSTFS_GET_DECODE_STRIPE_PREFETCH_COUNT,
|
||||
)
|
||||
}
|
||||
|
||||
/// Enable overlapping bitrot verification with stripe decode.
|
||||
/// When enabled, bitrot verification for stripe N+1 runs concurrently
|
||||
/// with decode of stripe N, reducing total pipeline latency.
|
||||
/// Default: false (sequential behavior, current implementation).
|
||||
const ENV_RUSTFS_GET_BITROT_DECODE_OVERLAP_ENABLE: &str = "RUSTFS_GET_BITROT_DECODE_OVERLAP_ENABLE";
|
||||
const DEFAULT_RUSTFS_GET_BITROT_DECODE_OVERLAP_ENABLE: bool = false;
|
||||
|
||||
/// Get whether bitrot-decode overlap is enabled.
|
||||
fn is_bitrot_decode_overlap_enabled() -> bool {
|
||||
rustfs_utils::get_env_bool(
|
||||
ENV_RUSTFS_GET_BITROT_DECODE_OVERLAP_ENABLE,
|
||||
DEFAULT_RUSTFS_GET_BITROT_DECODE_OVERLAP_ENABLE,
|
||||
)
|
||||
}
|
||||
|
||||
#[derive(Default)]
|
||||
struct ShardReadCostCounts {
|
||||
local: usize,
|
||||
|
||||
@@ -33,7 +33,7 @@ use tokio::io::{AsyncRead, ReadBuf};
|
||||
use tokio::task::JoinHandle;
|
||||
|
||||
const ENV_RUSTFS_GET_CODEC_STREAMING_MAX_INFLIGHT: &str = "RUSTFS_GET_CODEC_STREAMING_MAX_INFLIGHT";
|
||||
const DEFAULT_RUSTFS_GET_CODEC_STREAMING_MAX_INFLIGHT: usize = 1;
|
||||
const DEFAULT_RUSTFS_GET_CODEC_STREAMING_MAX_INFLIGHT: usize = 2;
|
||||
const FILL_POLICY_SINGLE_INFLIGHT: &str = "single_inflight";
|
||||
const FILL_POLICY_DUAL_INFLIGHT: &str = "dual_inflight";
|
||||
|
||||
@@ -758,9 +758,9 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn fill_policy_defaults_to_single_inflight() {
|
||||
fn fill_policy_defaults_to_dual_inflight() {
|
||||
with_var(ENV_RUSTFS_GET_CODEC_STREAMING_MAX_INFLIGHT, None::<&str>, || {
|
||||
assert_eq!(FillPolicy::from_env(), FillPolicy::SingleInFlight);
|
||||
assert_eq!(FillPolicy::from_env(), FillPolicy::DualInFlight);
|
||||
});
|
||||
|
||||
with_var(ENV_RUSTFS_GET_CODEC_STREAMING_MAX_INFLIGHT, Some("2"), || {
|
||||
|
||||
@@ -294,32 +294,78 @@ fn record_capacity_scope_if_needed(scope_token: Option<Uuid>, disks: &[Option<Di
|
||||
/// when reading large objects (20-26MB) under high concurrency.
|
||||
///
|
||||
/// Default: 4MB (4 * 1024 * 1024 bytes)
|
||||
/// Get duplex buffer size from environment variable.
|
||||
///
|
||||
/// **Deprecated**: Use `adaptive_duplex_buffer_size()` for object-size-aware sizing.
|
||||
pub fn get_duplex_buffer_size() -> usize {
|
||||
rustfs_utils::get_env_usize(
|
||||
rustfs_config::ENV_OBJECT_DUPLEX_BUFFER_SIZE,
|
||||
rustfs_config::DEFAULT_OBJECT_DUPLEX_BUFFER_SIZE,
|
||||
)
|
||||
}
|
||||
|
||||
/// Get adaptive duplex buffer size based on object size.
|
||||
///
|
||||
/// Smaller objects get smaller buffers to reduce memory waste.
|
||||
/// Larger objects get larger buffers to prevent backpressure.
|
||||
fn adaptive_duplex_buffer_size(object_size: i64) -> usize {
|
||||
const KB: usize = 1024;
|
||||
const MB: usize = 1024 * 1024;
|
||||
match object_size {
|
||||
0..=1_048_576 => 64 * KB, // <= 1MB: 64KB
|
||||
1_048_577..=16_777_216 => MB, // <= 16MB: 1MB
|
||||
16_777_217..=268_435_456 => 4 * MB, // <= 256MB: 4MB
|
||||
_ => 8 * MB, // > 256MB: 8MB
|
||||
}
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// GET Optimization Configuration
|
||||
//
|
||||
// All GET performance optimization flags are consolidated here.
|
||||
// Each flag uses `OnceLock` for caching — env var changes require process restart.
|
||||
// Each flag has a corresponding `*_ROLLOUT_PCT` for percentage-based gradual rollout.
|
||||
// ============================================================================
|
||||
|
||||
const DISK_ONLINE_TIMEOUT: Duration = Duration::from_secs(1);
|
||||
const DISK_HEALTH_CACHE_TTL: Duration = Duration::from_millis(750);
|
||||
const GET_OBJECT_METADATA_CACHE_TTL: Duration = Duration::from_millis(250);
|
||||
const GET_OBJECT_METADATA_CACHE_MAX_ENTRIES: usize = 1024;
|
||||
|
||||
// --- Codec Streaming Configuration ---
|
||||
|
||||
const ENV_RUSTFS_GET_CODEC_STREAMING_ENABLE: &str = "RUSTFS_GET_CODEC_STREAMING_ENABLE";
|
||||
const DEFAULT_RUSTFS_GET_CODEC_STREAMING_ENABLE: bool = false; // Disabled until rollout gates are ready
|
||||
|
||||
const ENV_RUSTFS_GET_CODEC_STREAMING_MIN_SIZE: &str = "RUSTFS_GET_CODEC_STREAMING_MIN_SIZE";
|
||||
const DEFAULT_RUSTFS_GET_CODEC_STREAMING_MIN_SIZE: usize = MI_B;
|
||||
|
||||
const ENV_RUSTFS_GET_CODEC_STREAMING_ENGINE: &str = "RUSTFS_GET_CODEC_STREAMING_ENGINE";
|
||||
const DEFAULT_RUSTFS_GET_CODEC_STREAMING_ENGINE: &str = GET_CODEC_STREAMING_ENGINE_LEGACY;
|
||||
|
||||
const ENV_RUSTFS_GET_CODEC_STREAMING_ROLLOUT: &str = "RUSTFS_GET_CODEC_STREAMING_ROLLOUT";
|
||||
const DEFAULT_RUSTFS_GET_CODEC_STREAMING_ROLLOUT: &str = "off";
|
||||
|
||||
const ENV_RUSTFS_GET_CODEC_STREAMING_BODY_COMPAT_CONFIRMED: &str = "RUSTFS_GET_CODEC_STREAMING_BODY_COMPAT_CONFIRMED";
|
||||
const ENV_RUSTFS_GET_CODEC_STREAMING_HEADER_COMPAT_CONFIRMED: &str = "RUSTFS_GET_CODEC_STREAMING_HEADER_COMPAT_CONFIRMED";
|
||||
const DEFAULT_RUSTFS_GET_CODEC_STREAMING_ENABLE: bool = false;
|
||||
const DEFAULT_RUSTFS_GET_CODEC_STREAMING_MIN_SIZE: usize = MI_B;
|
||||
const DEFAULT_RUSTFS_GET_CODEC_STREAMING_ENGINE: &str = GET_CODEC_STREAMING_ENGINE_LEGACY;
|
||||
const DEFAULT_RUSTFS_GET_CODEC_STREAMING_ROLLOUT: &str = "off";
|
||||
const ENV_RUSTFS_GET_METADATA_EARLY_STOP_ENABLE: &str = "RUSTFS_GET_METADATA_EARLY_STOP_ENABLE";
|
||||
const DEFAULT_RUSTFS_GET_METADATA_EARLY_STOP_ENABLE: bool = false;
|
||||
|
||||
const ENV_RUSTFS_GET_CODEC_STREAMING_ROLLOUT_PCT: &str = "RUSTFS_GET_CODEC_STREAMING_ROLLOUT_PCT";
|
||||
const DEFAULT_RUSTFS_GET_CODEC_STREAMING_ROLLOUT_PCT: u32 = 100;
|
||||
|
||||
const ENV_RUSTFS_GET_CODEC_STREAMING_MULTIPART_ENABLE: &str = "RUSTFS_GET_CODEC_STREAMING_MULTIPART_ENABLE";
|
||||
const DEFAULT_RUSTFS_GET_CODEC_STREAMING_MULTIPART_ENABLE: bool = false;
|
||||
|
||||
// --- Metadata Early-Stop Configuration ---
|
||||
|
||||
const ENV_RUSTFS_GET_METADATA_EARLY_STOP_ENABLE: &str = "RUSTFS_GET_METADATA_EARLY_STOP_ENABLE";
|
||||
const DEFAULT_RUSTFS_GET_METADATA_EARLY_STOP_ENABLE: bool = false;
|
||||
|
||||
const ENV_RUSTFS_GET_METADATA_EARLY_STOP_ROLLOUT_PCT: &str = "RUSTFS_GET_METADATA_EARLY_STOP_ROLLOUT_PCT";
|
||||
const DEFAULT_RUSTFS_GET_METADATA_EARLY_STOP_ROLLOUT_PCT: u32 = 100;
|
||||
|
||||
const ENV_RUSTFS_GET_METADATA_VERSION_EARLY_STOP_ENABLE: &str = "RUSTFS_GET_METADATA_VERSION_EARLY_STOP_ENABLE";
|
||||
const DEFAULT_RUSTFS_GET_METADATA_VERSION_EARLY_STOP_ENABLE: bool = false;
|
||||
|
||||
static OBJECT_LOCK_DIAG_ENABLED: OnceLock<bool> = OnceLock::new();
|
||||
|
||||
mod heal;
|
||||
@@ -396,20 +442,46 @@ pub fn is_deadlock_detection_enabled() -> bool {
|
||||
})
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// GET Optimization Flag Functions
|
||||
//
|
||||
// All functions use `OnceLock` for caching. Environment variable changes
|
||||
// require process restart to take effect.
|
||||
// ============================================================================
|
||||
|
||||
/// Check if codec streaming is enabled (base flag).
|
||||
///
|
||||
/// **Note**: Cached via `OnceLock` — env var changes require process restart.
|
||||
/// In test mode, bypasses cache to allow per-test env var overrides.
|
||||
fn is_get_codec_streaming_enabled() -> bool {
|
||||
#[cfg(test)]
|
||||
let enabled = rustfs_utils::get_env_bool(ENV_RUSTFS_GET_CODEC_STREAMING_ENABLE, DEFAULT_RUSTFS_GET_CODEC_STREAMING_ENABLE);
|
||||
#[cfg(not(test))]
|
||||
static CACHED: OnceLock<bool> = OnceLock::new();
|
||||
#[cfg(not(test))]
|
||||
let enabled = *CACHED.get_or_init(|| {
|
||||
{
|
||||
rustfs_utils::get_env_bool(ENV_RUSTFS_GET_CODEC_STREAMING_ENABLE, DEFAULT_RUSTFS_GET_CODEC_STREAMING_ENABLE)
|
||||
});
|
||||
enabled
|
||||
}
|
||||
#[cfg(not(test))]
|
||||
{
|
||||
static CACHED: OnceLock<bool> = OnceLock::new();
|
||||
*CACHED.get_or_init(|| {
|
||||
rustfs_utils::get_env_bool(ENV_RUSTFS_GET_CODEC_STREAMING_ENABLE, DEFAULT_RUSTFS_GET_CODEC_STREAMING_ENABLE)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
/// **Note**: Cached via `OnceLock` — env var changes require process restart.
|
||||
/// Check if multipart codec streaming is enabled.
|
||||
///
|
||||
/// When enabled, multipart objects use per-part codec streaming
|
||||
/// instead of falling back to the legacy duplex path.
|
||||
fn is_codec_streaming_multipart_enabled() -> bool {
|
||||
static CACHED: OnceLock<bool> = OnceLock::new();
|
||||
*CACHED.get_or_init(|| {
|
||||
rustfs_utils::get_env_bool(
|
||||
ENV_RUSTFS_GET_CODEC_STREAMING_MULTIPART_ENABLE,
|
||||
DEFAULT_RUSTFS_GET_CODEC_STREAMING_MULTIPART_ENABLE,
|
||||
)
|
||||
})
|
||||
}
|
||||
|
||||
/// Check if metadata early-stop is enabled (base flag).
|
||||
fn is_get_metadata_early_stop_enabled() -> bool {
|
||||
static CACHED: OnceLock<bool> = OnceLock::new();
|
||||
*CACHED.get_or_init(|| {
|
||||
@@ -417,31 +489,22 @@ fn is_get_metadata_early_stop_enabled() -> bool {
|
||||
})
|
||||
}
|
||||
|
||||
/// Determine if an optimization should be enabled for a specific request.
|
||||
/// Check if version-aware early-stop is enabled.
|
||||
///
|
||||
/// Uses a stable hash of `(bucket, object)` to ensure the same object
|
||||
/// always gets consistent behavior. This enables percentage-based gradual rollout.
|
||||
///
|
||||
/// **Note**: `base_enabled` must be pre-cached (via `OnceLock`) to avoid
|
||||
/// per-request `std::env::var()` calls.
|
||||
fn is_optimization_enabled_for_request(base_enabled: bool, rollout_pct: u32, bucket: &str, object: &str) -> bool {
|
||||
if !base_enabled || rollout_pct == 0 {
|
||||
return false;
|
||||
}
|
||||
if rollout_pct >= 100 {
|
||||
return true;
|
||||
}
|
||||
|
||||
// Stable hash: same (bucket, object) always produces the same result
|
||||
use std::hash::{Hash, Hasher};
|
||||
let mut hasher = std::collections::hash_map::DefaultHasher::new();
|
||||
bucket.hash(&mut hasher);
|
||||
object.hash(&mut hasher);
|
||||
let hash = hasher.finish() % 100;
|
||||
|
||||
(hash as u32) < rollout_pct
|
||||
/// When enabled, versioned requests can early-stop when the requested
|
||||
/// version_id reaches quorum across disks.
|
||||
fn is_version_early_stop_enabled() -> bool {
|
||||
static CACHED: OnceLock<bool> = OnceLock::new();
|
||||
*CACHED.get_or_init(|| {
|
||||
rustfs_utils::get_env_bool(
|
||||
ENV_RUSTFS_GET_METADATA_VERSION_EARLY_STOP_ENABLE,
|
||||
DEFAULT_RUSTFS_GET_METADATA_VERSION_EARLY_STOP_ENABLE,
|
||||
)
|
||||
})
|
||||
}
|
||||
|
||||
// --- Rollout Percentage Functions ---
|
||||
|
||||
fn get_codec_streaming_rollout_pct() -> u32 {
|
||||
static CACHED: OnceLock<u32> = OnceLock::new();
|
||||
*CACHED.get_or_init(|| {
|
||||
@@ -459,6 +522,30 @@ fn get_metadata_early_stop_rollout_pct() -> u32 {
|
||||
})
|
||||
}
|
||||
|
||||
// --- Request-Level Decision Functions ---
|
||||
|
||||
/// Determine if an optimization should be enabled for a specific request.
|
||||
///
|
||||
/// Uses a stable hash of `(bucket, object)` to ensure the same object
|
||||
/// always gets consistent behavior. This enables percentage-based gradual rollout.
|
||||
fn is_optimization_enabled_for_request(base_enabled: bool, rollout_pct: u32, bucket: &str, object: &str) -> bool {
|
||||
if !base_enabled || rollout_pct == 0 {
|
||||
return false;
|
||||
}
|
||||
if rollout_pct >= 100 {
|
||||
return true;
|
||||
}
|
||||
|
||||
// Stable hash: same (bucket, object) always produces the same result
|
||||
use std::hash::{Hash, Hasher};
|
||||
let mut hasher = std::collections::hash_map::DefaultHasher::new();
|
||||
bucket.hash(&mut hasher);
|
||||
object.hash(&mut hasher);
|
||||
let hash = hasher.finish() % 100;
|
||||
|
||||
(hash as u32) < rollout_pct
|
||||
}
|
||||
|
||||
/// Should this specific request use codec streaming?
|
||||
pub fn should_use_codec_streaming(bucket: &str, object: &str) -> bool {
|
||||
let base = is_get_codec_streaming_enabled();
|
||||
@@ -1545,7 +1632,7 @@ impl crate::storage_api_contracts::object::ObjectIO for SetDisks {
|
||||
|
||||
rustfs_io_metrics::record_get_object_reader_path(GET_OBJECT_PATH_LEGACY_DUPLEX);
|
||||
|
||||
let duplex_buffer_size = get_duplex_buffer_size();
|
||||
let duplex_buffer_size = adaptive_duplex_buffer_size(object_info.size);
|
||||
let (rd, wd) = tokio::io::duplex(duplex_buffer_size);
|
||||
debug!(bucket, object, duplex_buffer_size, "Created duplex pipe for object data transfer");
|
||||
|
||||
|
||||
@@ -17,12 +17,12 @@ use crate::diagnostics::get::{
|
||||
GET_METADATA_EARLY_STOP_REASON_CONFLICTING_METADATA, GET_METADATA_EARLY_STOP_REASON_DELETE_MARKER,
|
||||
GET_METADATA_EARLY_STOP_REASON_ERROR, GET_METADATA_EARLY_STOP_REASON_INSUFFICIENT_QUORUM,
|
||||
GET_METADATA_EARLY_STOP_REASON_NOT_FOUND, GET_METADATA_EARLY_STOP_REASON_UNSAFE_REQUEST,
|
||||
GET_METADATA_EARLY_STOP_REASON_VALID_QUORUM, GET_METADATA_EARLY_STOP_REASON_VERSION_NOT_FOUND, GET_METADATA_RESPONSE_CORRUPT,
|
||||
GET_METADATA_RESPONSE_DISK_NOT_FOUND, GET_METADATA_RESPONSE_ERROR, GET_METADATA_RESPONSE_IGNORED,
|
||||
GET_METADATA_RESPONSE_NOT_FOUND, GET_METADATA_RESPONSE_TIMEOUT, GET_METADATA_RESPONSE_VALID,
|
||||
GET_METADATA_RESPONSE_VERSION_NOT_FOUND, GET_OBJECT_PATH_CODEC_STREAMING, GET_OBJECT_PATH_LEGACY_DUPLEX, GET_STAGE_DECODE,
|
||||
GET_STAGE_RANGE, GET_STAGE_READER_SETUP, GetObjectFailureReason, classify_disk_error, record_get_object_pipeline_failure,
|
||||
record_get_object_pipeline_failure_for_path,
|
||||
GET_METADATA_EARLY_STOP_REASON_VALID_QUORUM, GET_METADATA_EARLY_STOP_REASON_VERSION_MATCH_QUORUM,
|
||||
GET_METADATA_EARLY_STOP_REASON_VERSION_NOT_FOUND, GET_METADATA_RESPONSE_CORRUPT, GET_METADATA_RESPONSE_DISK_NOT_FOUND,
|
||||
GET_METADATA_RESPONSE_ERROR, GET_METADATA_RESPONSE_IGNORED, GET_METADATA_RESPONSE_NOT_FOUND, GET_METADATA_RESPONSE_TIMEOUT,
|
||||
GET_METADATA_RESPONSE_VALID, GET_METADATA_RESPONSE_VERSION_NOT_FOUND, GET_OBJECT_PATH_CODEC_STREAMING,
|
||||
GET_OBJECT_PATH_LEGACY_DUPLEX, GET_STAGE_DECODE, GET_STAGE_RANGE, GET_STAGE_READER_SETUP, GetObjectFailureReason,
|
||||
classify_disk_error, record_get_object_pipeline_failure, record_get_object_pipeline_failure_for_path,
|
||||
};
|
||||
use crate::erasure::coding::BitrotReader;
|
||||
use crate::io_support::bitrot::create_deferred_bitrot_reader;
|
||||
@@ -204,6 +204,9 @@ struct MetadataQuorumAccumulator {
|
||||
candidate_votes: usize,
|
||||
conflicting_metadata: bool,
|
||||
delete_marker_seen: bool,
|
||||
delete_marker_votes: usize,
|
||||
requested_version_id: String,
|
||||
matching_version_votes: usize,
|
||||
}
|
||||
|
||||
impl MetadataQuorumAccumulator {
|
||||
@@ -221,9 +224,17 @@ impl MetadataQuorumAccumulator {
|
||||
candidate_votes: 0,
|
||||
conflicting_metadata: false,
|
||||
delete_marker_seen: false,
|
||||
delete_marker_votes: 0,
|
||||
requested_version_id: String::new(),
|
||||
matching_version_votes: 0,
|
||||
}
|
||||
}
|
||||
|
||||
fn with_requested_version_id(mut self, version_id: &str) -> Self {
|
||||
self.requested_version_id = version_id.to_string();
|
||||
self
|
||||
}
|
||||
|
||||
fn observe_file_info(&mut self, file_info: &FileInfo) {
|
||||
if !file_info.is_valid() {
|
||||
self.hard_errors = self.hard_errors.saturating_add(1);
|
||||
@@ -231,7 +242,17 @@ impl MetadataQuorumAccumulator {
|
||||
}
|
||||
|
||||
self.valid_responses = self.valid_responses.saturating_add(1);
|
||||
|
||||
// Track version match for versioned requests
|
||||
if !self.requested_version_id.is_empty()
|
||||
&& let Some(ref vid) = file_info.version_id
|
||||
&& vid.to_string() == self.requested_version_id
|
||||
{
|
||||
self.matching_version_votes = self.matching_version_votes.saturating_add(1);
|
||||
}
|
||||
|
||||
if file_info.deleted {
|
||||
self.delete_marker_votes = self.delete_marker_votes.saturating_add(1);
|
||||
self.delete_marker_seen = true;
|
||||
return;
|
||||
}
|
||||
@@ -271,6 +292,11 @@ impl MetadataQuorumAccumulator {
|
||||
if !self.allow_early_stop {
|
||||
return None;
|
||||
}
|
||||
if self.delete_marker_votes >= self.missing_response_quorum() {
|
||||
return Some(MetadataEarlyStopDecision {
|
||||
reason: GET_METADATA_EARLY_STOP_REASON_DELETE_MARKER,
|
||||
});
|
||||
}
|
||||
if self.conflicting_metadata
|
||||
|| self.delete_marker_seen
|
||||
|| self.not_found_responses > 0
|
||||
@@ -292,6 +318,27 @@ impl MetadataQuorumAccumulator {
|
||||
None
|
||||
}
|
||||
|
||||
/// Check if a versioned request can early-stop because the requested
|
||||
/// version_id has reached quorum across disks.
|
||||
fn version_early_stop_decision(&self) -> Option<MetadataEarlyStopDecision> {
|
||||
if self.requested_version_id.is_empty() {
|
||||
return None;
|
||||
}
|
||||
if self.matching_version_votes >= self.read_quorum_for_version() {
|
||||
return Some(MetadataEarlyStopDecision {
|
||||
reason: GET_METADATA_EARLY_STOP_REASON_VERSION_MATCH_QUORUM,
|
||||
});
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
/// Compute the read quorum threshold for version-aware early-stop.
|
||||
/// Uses `total_disks / 2` (like `missing_response_quorum`) when
|
||||
/// `default_parity_count` is set, otherwise requires all disks.
|
||||
fn read_quorum_for_version(&self) -> usize {
|
||||
self.missing_response_quorum()
|
||||
}
|
||||
|
||||
fn final_miss_reason(&self) -> &'static str {
|
||||
if !self.allow_early_stop {
|
||||
return GET_METADATA_EARLY_STOP_REASON_UNSAFE_REQUEST;
|
||||
@@ -1110,7 +1157,9 @@ impl SetDisks {
|
||||
default_parity_count: usize,
|
||||
) -> disk::error::Result<(Vec<FileInfo>, Vec<Option<DiskError>>, MetadataFanoutDiagnostics)> {
|
||||
let early_stop_enabled = observe && is_get_metadata_early_stop_enabled();
|
||||
let allow_early_stop = early_stop_enabled && version_id.is_empty() && !healing && !incl_free_versions;
|
||||
let allow_early_stop = observe
|
||||
&& ((is_get_metadata_early_stop_enabled() && version_id.is_empty() && !healing && !incl_free_versions)
|
||||
|| (is_version_early_stop_enabled() && !version_id.is_empty() && !healing));
|
||||
if allow_early_stop {
|
||||
return Self::read_all_fileinfo_early_stop(
|
||||
disks,
|
||||
@@ -1245,7 +1294,8 @@ impl SetDisks {
|
||||
let mut ress = vec![FileInfo::default(); disks.len()];
|
||||
let mut errors = vec![None; disks.len()];
|
||||
let mut observations = Vec::with_capacity(disks.len());
|
||||
let mut accumulator = MetadataQuorumAccumulator::new(disks.len(), default_parity_count, true);
|
||||
let mut accumulator =
|
||||
MetadataQuorumAccumulator::new(disks.len(), default_parity_count, true).with_requested_version_id(version_id);
|
||||
let opts = Arc::new(ReadOptions {
|
||||
incl_free_versions,
|
||||
read_data,
|
||||
@@ -1299,7 +1349,10 @@ impl SetDisks {
|
||||
}
|
||||
}
|
||||
|
||||
if let Some(decision) = accumulator.early_stop_decision() {
|
||||
if let Some(decision) = accumulator
|
||||
.early_stop_decision()
|
||||
.or_else(|| accumulator.version_early_stop_decision())
|
||||
{
|
||||
let saved_responses = join_set.len();
|
||||
join_set.abort_all();
|
||||
rustfs_io_metrics::record_get_object_metadata_early_stop_hit(GET_OBJECT_PATH_LEGACY_DUPLEX, decision.reason);
|
||||
@@ -3014,7 +3067,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn metadata_quorum_accumulator_falls_back_on_delete_marker_latest() {
|
||||
fn metadata_quorum_accumulator_hits_delete_marker_quorum_early_stop() {
|
||||
let mut accumulator = metadata_early_stop_accumulator();
|
||||
let mut deleted = metadata_early_stop_candidate("object", 1);
|
||||
deleted.deleted = true;
|
||||
@@ -3022,6 +3075,22 @@ mod tests {
|
||||
accumulator.observe_file_info(&deleted);
|
||||
accumulator.observe_file_info(&deleted);
|
||||
|
||||
assert_eq!(
|
||||
accumulator.early_stop_decision(),
|
||||
Some(MetadataEarlyStopDecision {
|
||||
reason: GET_METADATA_EARLY_STOP_REASON_DELETE_MARKER
|
||||
})
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn metadata_quorum_accumulator_falls_back_on_delete_marker_below_quorum() {
|
||||
let mut accumulator = metadata_early_stop_accumulator();
|
||||
let mut deleted = metadata_early_stop_candidate("object", 1);
|
||||
deleted.deleted = true;
|
||||
|
||||
accumulator.observe_file_info(&deleted);
|
||||
|
||||
assert!(accumulator.early_stop_decision().is_none());
|
||||
assert_eq!(accumulator.final_miss_reason(), GET_METADATA_EARLY_STOP_REASON_DELETE_MARKER);
|
||||
}
|
||||
@@ -3092,6 +3161,101 @@ mod tests {
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn version_early_stop_gate_defaults_to_disabled() {
|
||||
temp_env::with_var(ENV_RUSTFS_GET_METADATA_VERSION_EARLY_STOP_ENABLE, None::<&str>, || {
|
||||
assert!(!is_version_early_stop_enabled());
|
||||
});
|
||||
}
|
||||
|
||||
fn version_early_stop_candidate(object: &str, disk_index: usize, version_id: Uuid) -> FileInfo {
|
||||
let mut fi = metadata_early_stop_candidate(object, disk_index);
|
||||
fi.version_id = Some(version_id);
|
||||
fi
|
||||
}
|
||||
|
||||
fn version_early_stop_accumulator(requested_version_id: &str) -> MetadataQuorumAccumulator {
|
||||
MetadataQuorumAccumulator::new(4, 2, true).with_requested_version_id(requested_version_id)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn version_early_stop_hits_quorum_with_matching_versions() {
|
||||
let vid = Uuid::new_v4();
|
||||
let mut accumulator = version_early_stop_accumulator(&vid.to_string());
|
||||
|
||||
accumulator.observe_file_info(&version_early_stop_candidate("object", 1, vid));
|
||||
assert!(accumulator.version_early_stop_decision().is_none());
|
||||
accumulator.observe_file_info(&version_early_stop_candidate("object", 2, vid));
|
||||
|
||||
assert_eq!(
|
||||
accumulator.version_early_stop_decision(),
|
||||
Some(MetadataEarlyStopDecision {
|
||||
reason: GET_METADATA_EARLY_STOP_REASON_VERSION_MATCH_QUORUM
|
||||
})
|
||||
);
|
||||
assert_eq!(accumulator.matching_version_votes, 2);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn version_early_stop_does_not_fire_without_requested_version() {
|
||||
let vid = Uuid::new_v4();
|
||||
let mut accumulator = version_early_stop_accumulator("");
|
||||
|
||||
accumulator.observe_file_info(&version_early_stop_candidate("object", 1, vid));
|
||||
accumulator.observe_file_info(&version_early_stop_candidate("object", 2, vid));
|
||||
|
||||
assert!(accumulator.version_early_stop_decision().is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn version_early_stop_does_not_fire_with_mismatched_versions() {
|
||||
let requested_vid = Uuid::new_v4();
|
||||
let other_vid = Uuid::new_v4();
|
||||
let mut accumulator = version_early_stop_accumulator(&requested_vid.to_string());
|
||||
|
||||
accumulator.observe_file_info(&version_early_stop_candidate("object", 1, requested_vid));
|
||||
accumulator.observe_file_info(&version_early_stop_candidate("object", 2, other_vid));
|
||||
|
||||
assert!(accumulator.version_early_stop_decision().is_none());
|
||||
assert_eq!(accumulator.matching_version_votes, 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn version_early_stop_does_not_fire_below_quorum() {
|
||||
let vid = Uuid::new_v4();
|
||||
let mut accumulator = version_early_stop_accumulator(&vid.to_string());
|
||||
|
||||
accumulator.observe_file_info(&version_early_stop_candidate("object", 1, vid));
|
||||
|
||||
assert!(accumulator.version_early_stop_decision().is_none());
|
||||
assert_eq!(accumulator.matching_version_votes, 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn version_early_stop_tracks_matching_votes_independently_of_candidate() {
|
||||
let requested_vid = Uuid::new_v4();
|
||||
let mut accumulator = version_early_stop_accumulator(&requested_vid.to_string());
|
||||
|
||||
// Two valid responses with matching version_id but different erasure.index
|
||||
// (so they conflict on the candidate path but still count for version votes)
|
||||
let mut fi1 = version_early_stop_candidate("object", 1, requested_vid);
|
||||
fi1.size = 100;
|
||||
let mut fi2 = version_early_stop_candidate("object", 2, requested_vid);
|
||||
fi2.size = 200; // different size → conflicting metadata on candidate path
|
||||
|
||||
accumulator.observe_file_info(&fi1);
|
||||
accumulator.observe_file_info(&fi2);
|
||||
|
||||
// Candidate path sees conflict, but version path sees quorum
|
||||
assert!(accumulator.early_stop_decision().is_none());
|
||||
assert_eq!(
|
||||
accumulator.version_early_stop_decision(),
|
||||
Some(MetadataEarlyStopDecision {
|
||||
reason: GET_METADATA_EARLY_STOP_REASON_VERSION_MATCH_QUORUM
|
||||
})
|
||||
);
|
||||
}
|
||||
|
||||
fn codec_streaming_test_object_info(fi: &FileInfo) -> ObjectInfo {
|
||||
ObjectInfo::from_file_info(fi, "bucket", "object", false)
|
||||
}
|
||||
|
||||
@@ -22,7 +22,7 @@ use rustfs_heal::heal::{
|
||||
use serial_test::serial;
|
||||
use std::{
|
||||
path::{Path, PathBuf},
|
||||
sync::{Arc, Once, OnceLock},
|
||||
sync::{Arc, Once},
|
||||
time::Duration,
|
||||
};
|
||||
use tokio::fs;
|
||||
@@ -59,7 +59,6 @@ async fn wait_for_path_exists(path: &Path, timeout: Duration, interval: Duration
|
||||
}
|
||||
}
|
||||
|
||||
static GLOBAL_ENV: OnceLock<(Vec<PathBuf>, Arc<ECStore>, Arc<ECStoreHealStorage>)> = OnceLock::new();
|
||||
static INIT: Once = Once::new();
|
||||
|
||||
pub fn init_tracing() {
|
||||
@@ -76,11 +75,6 @@ pub fn init_tracing() {
|
||||
async fn setup_test_env() -> (Vec<PathBuf>, Arc<ECStore>, Arc<ECStoreHealStorage>) {
|
||||
init_tracing();
|
||||
|
||||
// Fast path: already initialized, just clone and return
|
||||
if let Some((paths, ecstore, heal_storage)) = GLOBAL_ENV.get() {
|
||||
return (paths.clone(), ecstore.clone(), heal_storage.clone());
|
||||
}
|
||||
|
||||
// create temp dir as 4 disks with unique base dir
|
||||
let test_base_dir = format!("/tmp/rustfs_heal_heal_test_{}", uuid::Uuid::new_v4());
|
||||
let temp_dir = std::path::PathBuf::from(&test_base_dir);
|
||||
@@ -126,9 +120,9 @@ async fn setup_test_env() -> (Vec<PathBuf>, Arc<ECStore>, Arc<ECStoreHealStorage
|
||||
// format disks (only first time)
|
||||
init_local_disks(endpoint_pools.clone()).await.unwrap();
|
||||
|
||||
// create ECStore with dynamic port 0 (let OS assign) or fixed 9001 if free
|
||||
let port = 9001; // for simplicity
|
||||
let server_addr: std::net::SocketAddr = format!("127.0.0.1:{port}").parse().unwrap();
|
||||
// Use port 0 so nextest can run this integration binary in parallel
|
||||
// with other ECStore-backed tests without sharing a fixed peer port.
|
||||
let server_addr: std::net::SocketAddr = "127.0.0.1:0".parse().unwrap();
|
||||
let ecstore = ECStore::new(server_addr, endpoint_pools, CancellationToken::new())
|
||||
.await
|
||||
.unwrap();
|
||||
@@ -147,9 +141,6 @@ async fn setup_test_env() -> (Vec<PathBuf>, Arc<ECStore>, Arc<ECStoreHealStorage
|
||||
// Create heal storage layer
|
||||
let heal_storage = Arc::new(ECStoreHealStorage::new(ecstore.clone()));
|
||||
|
||||
// Store in global once lock
|
||||
let _ = GLOBAL_ENV.set((disk_paths.clone(), ecstore.clone(), heal_storage.clone()));
|
||||
|
||||
(disk_paths, ecstore, heal_storage)
|
||||
}
|
||||
|
||||
|
||||
@@ -947,6 +947,31 @@ pub fn record_get_object_metadata_phase_duration(duration_secs: f64) {
|
||||
record_get_object_stage_duration("legacy_duplex", "metadata", duration_secs);
|
||||
}
|
||||
|
||||
/// Record metadata phase duration with early-stop state label.
|
||||
#[inline(always)]
|
||||
pub fn record_get_object_metadata_phase_duration_with_early_stop(duration_secs: f64, early_stop_active: &'static str) {
|
||||
if !get_stage_metrics_enabled() {
|
||||
return;
|
||||
}
|
||||
histogram!(
|
||||
"rustfs_io_get_object_stage_duration_seconds",
|
||||
"path" => "legacy_duplex",
|
||||
"stage" => "metadata",
|
||||
"early_stop_active" => early_stop_active
|
||||
)
|
||||
.record(duration_secs);
|
||||
}
|
||||
|
||||
/// Record GET object total duration with reader path label.
|
||||
#[inline(always)]
|
||||
pub fn record_get_object_total_duration_with_path(duration_secs: f64, reader_path: &'static str) {
|
||||
histogram!(
|
||||
"rustfs_io_get_object_total_duration_seconds_with_path",
|
||||
"reader_path" => reader_path
|
||||
)
|
||||
.record(duration_secs);
|
||||
}
|
||||
|
||||
/// Record GetObject shard reader setup duration.
|
||||
#[inline(always)]
|
||||
pub fn record_get_object_shard_reader_setup_duration(duration_secs: f64) {
|
||||
@@ -1075,6 +1100,34 @@ pub fn record_bytes_pool_hit_rate(tier: &str, hit_rate: f64) {
|
||||
gauge!("rustfs_bytes_pool_hit_rate", "tier" => tier.to_string()).set(hit_rate * 100.0);
|
||||
}
|
||||
|
||||
/// Record a BytesPool buffer acquisition attempt.
|
||||
///
|
||||
/// `outcome` = `"hit"` when a buffer is available in the pool, `"miss"` when the
|
||||
/// pool is empty and a new allocation is required.
|
||||
///
|
||||
/// # Arguments
|
||||
///
|
||||
/// * `tier` - Pool tier ("small", "medium", "large", "xlarge")
|
||||
/// * `outcome` - Acquisition outcome ("hit" or "miss")
|
||||
#[inline(always)]
|
||||
pub fn record_bytespool_acquisition(tier: &'static str, outcome: &'static str) {
|
||||
counter!("rustfs_io_bytespool_acquisition_total", "tier" => tier, "outcome" => outcome).increment(1);
|
||||
}
|
||||
|
||||
/// Record a BytesPool buffer return attempt.
|
||||
///
|
||||
/// `outcome` = `"recycled"` when the buffer is successfully returned to the
|
||||
/// pool, `"dropped"` when `try_lock` fails and the buffer is deallocated.
|
||||
///
|
||||
/// # Arguments
|
||||
///
|
||||
/// * `tier` - Pool tier ("small", "medium", "large", "xlarge")
|
||||
/// * `outcome` - Return outcome ("recycled" or "dropped")
|
||||
#[inline(always)]
|
||||
pub fn record_bytespool_return(tier: &'static str, outcome: &'static str) {
|
||||
counter!("rustfs_io_bytespool_return_total", "tier" => tier, "outcome" => outcome).increment(1);
|
||||
}
|
||||
|
||||
/// Record zero-copy write operation.
|
||||
///
|
||||
/// # Arguments
|
||||
@@ -1564,6 +1617,21 @@ mod tests {
|
||||
record_bytes_pool_hit_rate("small", 0.85);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_record_bytespool_acquisition_and_return() {
|
||||
// Acquisition outcomes
|
||||
record_bytespool_acquisition("small", "hit");
|
||||
record_bytespool_acquisition("medium", "miss");
|
||||
record_bytespool_acquisition("large", "hit");
|
||||
record_bytespool_acquisition("xlarge", "miss");
|
||||
|
||||
// Return outcomes
|
||||
record_bytespool_return("small", "recycled");
|
||||
record_bytespool_return("medium", "dropped");
|
||||
record_bytespool_return("large", "recycled");
|
||||
record_bytespool_return("xlarge", "dropped");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_record_zero_copy_write() {
|
||||
record_zero_copy_write(1024, 10.5);
|
||||
@@ -1612,6 +1680,8 @@ mod tests {
|
||||
record_get_object_full_body_latency("s3_handler", 0.009);
|
||||
record_get_object_response_handoff_duration("s3_handler", 0.0001);
|
||||
record_get_object_metadata_phase_duration(0.002);
|
||||
record_get_object_metadata_phase_duration_with_early_stop(0.002, "hit");
|
||||
record_get_object_total_duration_with_path(0.050, "legacy_duplex");
|
||||
record_get_object_shard_reader_setup_duration(0.003);
|
||||
record_get_object_decode_duration(0.004);
|
||||
record_get_object_duplex_backpressure_duration(0.005);
|
||||
@@ -1747,6 +1817,8 @@ mod tests {
|
||||
record_get_object_first_byte_latency("s3_handler", 0.008);
|
||||
record_get_object_full_body_latency("s3_handler", 0.009);
|
||||
record_get_object_response_handoff_duration("s3_handler", 0.0001);
|
||||
record_get_object_metadata_phase_duration_with_early_stop(0.002, "hit");
|
||||
record_get_object_total_duration_with_path(0.050, "legacy_duplex");
|
||||
record_get_object_shard_reader_setup_duration(0.003);
|
||||
record_get_object_decode_duration(0.004);
|
||||
record_get_object_duplex_backpressure_duration(0.005);
|
||||
|
||||
@@ -0,0 +1,119 @@
|
||||
# GET Optimization Stress Test Scripts
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- `warp` installed (https://github.com/minio/warp)
|
||||
- `mc` configured with access to the RustFS server
|
||||
- RustFS server running with GET optimizations enabled
|
||||
|
||||
### Quick Validation (5 minutes)
|
||||
|
||||
```bash
|
||||
./scripts/quick-validate-get-optimization.sh localhost:9000
|
||||
```
|
||||
|
||||
This runs basic functional tests:
|
||||
- Data integrity verification
|
||||
- Concurrent GET stability
|
||||
- Early-stop behavior validation
|
||||
|
||||
### Full Stress Test (30+ minutes)
|
||||
|
||||
```bash
|
||||
./scripts/stress-test-get-optimization.sh localhost:9000 ./stress-results
|
||||
```
|
||||
|
||||
This runs comprehensive tests:
|
||||
- Data correctness validation (1KB, 1MB, 10MB objects)
|
||||
- Concurrent GET stress test (16, 64, 256 concurrency)
|
||||
- Mixed read/write workload
|
||||
- Early-stop behavior under load
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `WARP_ACCESS_KEY` | rustfsadmin | S3 access key |
|
||||
| `WARP_SECRET_KEY` | rustfsadmin | S3 secret key |
|
||||
| `MC_ALIAS` | rustfs | mc alias for the server |
|
||||
| `TEST_BUCKET` | auto-generated | Test bucket name |
|
||||
| `TEST_DURATION` | 300s | Duration for stress tests |
|
||||
| `CONCURRENCY` | 64 | Default concurrency level |
|
||||
|
||||
## Test Scenarios
|
||||
|
||||
### 1. Data Correctness Validation
|
||||
|
||||
Verifies that GET returns correct data with early-stop enabled:
|
||||
- Uploads random data
|
||||
- Downloads and compares MD5 hash
|
||||
- Tests different object sizes (1KB, 1MB, 10MB)
|
||||
|
||||
### 2. Concurrent GET Stress Test
|
||||
|
||||
Tests performance under high concurrency:
|
||||
- Object sizes: 1KiB, 1MiB, 4MiB, 10MiB
|
||||
- Concurrency levels: 16, 64, 256
|
||||
- Duration: configurable (default 300s)
|
||||
|
||||
### 3. Mixed Read/Write Stress Test
|
||||
|
||||
Tests stability under concurrent read/write:
|
||||
- 50% reads, 50% writes
|
||||
- 1MB objects
|
||||
- 64 concurrent operations
|
||||
|
||||
### 4. Early-Stop Behavior Validation
|
||||
|
||||
Verifies early-stop works correctly:
|
||||
- 100 sequential reads of the same object
|
||||
- Verifies data size matches expected
|
||||
- Checks for any download failures
|
||||
|
||||
## Expected Results
|
||||
|
||||
### Success Criteria
|
||||
|
||||
- **Data Correctness**: 100% pass rate (no data corruption)
|
||||
- **Concurrent GET**: No errors, consistent latency
|
||||
- **Mixed Workload**: No deadlocks or data corruption
|
||||
- **Early-Stop**: All reads return correct data size
|
||||
|
||||
### Performance Baselines
|
||||
|
||||
| Object Size | Expected Throughput | Expected p95 Latency |
|
||||
|-------------|--------------------|--------------------|
|
||||
| 1KiB | > 5 MiB/s | < 10ms |
|
||||
| 1MiB | > 500 MiB/s | < 20ms |
|
||||
| 4MiB | > 1000 MiB/s | < 30ms |
|
||||
| 10MiB | > 2000 MiB/s | < 50ms |
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **warp not found**: Install with `go install github.com/minio/warp@latest`
|
||||
2. **mc not configured**: Run `mc alias set rustfs http://localhost:9000 admin password`
|
||||
3. **Connection refused**: Verify RustFS is running and accessible
|
||||
4. **Permission denied**: Check S3 credentials
|
||||
|
||||
### Debug Mode
|
||||
|
||||
Enable debug logging:
|
||||
```bash
|
||||
RUST_LOG=rustfs_ecstore::bucket::lifecycle=debug ./scripts/stress-test-get-optimization.sh
|
||||
```
|
||||
|
||||
## Output Files
|
||||
|
||||
| File | Description |
|
||||
|------|-------------|
|
||||
| `test-config.txt` | Test configuration |
|
||||
| `correctness-results.txt` | Data correctness results |
|
||||
| `early-stop-results.txt` | Early-stop validation results |
|
||||
| `get-*.json` | Concurrent GET performance (warp JSON) |
|
||||
| `mixed-*.json` | Mixed workload performance (warp JSON) |
|
||||
| `correctness-errors.log` | Data correctness errors (if any) |
|
||||
| `early-stop-errors.log` | Early-stop errors (if any) |
|
||||
Executable
+122
@@ -0,0 +1,122 @@
|
||||
#!/usr/bin/env bash
|
||||
# Quick Validation Script for GET Optimization
|
||||
# Usage: ./scripts/quick-validate-get-optimization.sh [TARGET_HOST]
|
||||
#
|
||||
# Runs quick functional tests to validate GET optimization changes:
|
||||
# 1. Basic GET correctness
|
||||
# 2. Concurrent GET stability
|
||||
# 3. Early-stop behavior
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
TARGET_HOST="${1:-localhost:9000}"
|
||||
MC_ALIAS="${MC_ALIAS:-rustfs}"
|
||||
TEST_BUCKET="quick-validate-$(date +%s)"
|
||||
|
||||
echo "=========================================="
|
||||
echo "Quick GET Optimization Validation"
|
||||
echo "=========================================="
|
||||
echo "Target: $TARGET_HOST"
|
||||
echo ""
|
||||
|
||||
# Create test bucket
|
||||
mc mb "${MC_ALIAS}/${TEST_BUCKET}" 2>/dev/null || true
|
||||
|
||||
# ============================================================================
|
||||
# Test 1: Basic GET Correctness
|
||||
# ============================================================================
|
||||
echo "[1/3] Basic GET Correctness"
|
||||
|
||||
# Upload test file
|
||||
dd if=/dev/urandom of=/tmp/quick-test.bin bs=1048576 count=1 2>/dev/null
|
||||
original_hash=$(md5 -q /tmp/quick-test.bin 2>/dev/null || md5sum /tmp/quick-test.bin | awk '{print $1}')
|
||||
|
||||
mc cp /tmp/quick-test.bin "${MC_ALIAS}/${TEST_BUCKET}/test.bin" >/dev/null 2>&1
|
||||
|
||||
# Download and verify
|
||||
mc cp "${MC_ALIAS}/${TEST_BUCKET}/test.bin" /tmp/quick-test-download.bin >/dev/null 2>&1
|
||||
download_hash=$(md5 -q /tmp/quick-test-download.bin 2>/dev/null || md5sum /tmp/quick-test-download.bin | awk '{print $1}')
|
||||
|
||||
if [ "$original_hash" = "$download_hash" ]; then
|
||||
echo " PASS: Data integrity verified"
|
||||
else
|
||||
echo " FAIL: Data mismatch (original=$original_hash, download=$download_hash)"
|
||||
fi
|
||||
|
||||
rm -f /tmp/quick-test.bin /tmp/quick-test-download.bin
|
||||
|
||||
# ============================================================================
|
||||
# Test 2: Concurrent GET Stability
|
||||
# ============================================================================
|
||||
echo "[2/3] Concurrent GET Stability"
|
||||
|
||||
# Upload multiple test files
|
||||
for i in $(seq 1 10); do
|
||||
dd if=/dev/urandom of="/tmp/quick-test-${i}.bin" bs=1048576 count=1 2>/dev/null
|
||||
mc cp "/tmp/quick-test-${i}.bin" "${MC_ALIAS}/${TEST_BUCKET}/test-${i}.bin" >/dev/null 2>&1
|
||||
rm -f "/tmp/quick-test-${i}.bin"
|
||||
done
|
||||
|
||||
# Concurrent download
|
||||
passed=0
|
||||
failed=0
|
||||
|
||||
for i in $(seq 1 10); do
|
||||
if mc cp "${MC_ALIAS}/${TEST_BUCKET}/test-${i}.bin" "/tmp/quick-download-${i}.bin" >/dev/null 2>&1; then
|
||||
size=$(stat -f%z "/tmp/quick-download-${i}.bin" 2>/dev/null || stat -c%s "/tmp/quick-download-${i}.bin" 2>/dev/null)
|
||||
if [ "$size" = "1048576" ]; then
|
||||
passed=$((passed + 1))
|
||||
else
|
||||
failed=$((failed + 1))
|
||||
fi
|
||||
else
|
||||
failed=$((failed + 1))
|
||||
fi
|
||||
rm -f "/tmp/quick-download-${i}.bin"
|
||||
done
|
||||
|
||||
echo " Result: $passed passed, $failed failed"
|
||||
|
||||
# ============================================================================
|
||||
# Test 3: Early-Stop Behavior
|
||||
# ============================================================================
|
||||
echo "[3/3] Early-Stop Behavior"
|
||||
|
||||
# Upload a larger test file
|
||||
dd if=/dev/urandom of=/tmp/quick-test-large.bin bs=1048576 count=10 2>/dev/null
|
||||
mc cp /tmp/quick-test-large.bin "${MC_ALIAS}/${TEST_BUCKET}/test-large.bin" >/dev/null 2>&1
|
||||
|
||||
# Multiple reads to trigger early-stop
|
||||
passed=0
|
||||
failed=0
|
||||
|
||||
for i in $(seq 1 20); do
|
||||
if mc cp "${MC_ALIAS}/${TEST_BUCKET}/test-large.bin" "/tmp/quick-large-${i}.bin" >/dev/null 2>&1; then
|
||||
size=$(stat -f%z "/tmp/quick-large-${i}.bin" 2>/dev/null || stat -c%s "/tmp/quick-large-${i}.bin" 2>/dev/null)
|
||||
if [ "$size" = "10485760" ]; then
|
||||
passed=$((passed + 1))
|
||||
else
|
||||
failed=$((failed + 1))
|
||||
fi
|
||||
else
|
||||
failed=$((failed + 1))
|
||||
fi
|
||||
rm -f "/tmp/quick-large-${i}.bin"
|
||||
done
|
||||
|
||||
echo " Result: $passed passed, $failed failed"
|
||||
|
||||
rm -f /tmp/quick-test-large.bin
|
||||
|
||||
# ============================================================================
|
||||
# Cleanup
|
||||
# ============================================================================
|
||||
echo ""
|
||||
echo "Cleaning up..."
|
||||
mc rm --recursive --force "${MC_ALIAS}/${TEST_BUCKET}" >/dev/null 2>&1 || true
|
||||
mc rb "${MC_ALIAS}/${TEST_BUCKET}" >/dev/null 2>&1 || true
|
||||
|
||||
echo ""
|
||||
echo "=========================================="
|
||||
echo "Quick Validation Complete"
|
||||
echo "=========================================="
|
||||
Executable
+242
@@ -0,0 +1,242 @@
|
||||
#!/usr/bin/env bash
|
||||
# GET Optimization Stress Test Script
|
||||
# Usage: ./scripts/stress-test-get-optimization.sh [TARGET_HOST] [OUTPUT_DIR]
|
||||
#
|
||||
# Validates:
|
||||
# 1. Data correctness with early-stop enabled
|
||||
# 2. Concurrent GET performance
|
||||
# 3. Mixed read/write stability
|
||||
# 4. Early-stop behavior under slow disk conditions
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
TARGET_HOST="${1:-localhost:9000}"
|
||||
OUTPUT_DIR="${2:-./stress-test-results/$(date +%Y%m%d-%H%M%S)}"
|
||||
MC_ALIAS="${MC_ALIAS:-rustfs}"
|
||||
TEST_BUCKET="${TEST_BUCKET:-stress-test-$(date +%s)}"
|
||||
TEST_DURATION="${TEST_DURATION:-300s}"
|
||||
CONCURRENCY="${CONCURRENCY:-64}"
|
||||
|
||||
# S3 credentials
|
||||
export WARP_ACCESS_KEY="${WARP_ACCESS_KEY:-rustfsadmin}"
|
||||
export WARP_SECRET_KEY="${WARP_SECRET_KEY:-rustfsadmin}"
|
||||
|
||||
mkdir -p "$OUTPUT_DIR"
|
||||
|
||||
echo "=========================================="
|
||||
echo "GET Optimization Stress Test"
|
||||
echo "=========================================="
|
||||
echo "Target: $TARGET_HOST"
|
||||
echo "Output: $OUTPUT_DIR"
|
||||
echo "Bucket: $TEST_BUCKET"
|
||||
echo "Duration: $TEST_DURATION"
|
||||
echo "Concurrency: $CONCURRENCY"
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
|
||||
# Save test configuration
|
||||
cat > "$OUTPUT_DIR/test-config.txt" <<EOF
|
||||
Date: $(date -u +"%Y-%m-%dT%H:%M:%SZ")
|
||||
Target: $TARGET_HOST
|
||||
Bucket: $TEST_BUCKET
|
||||
Duration: $TEST_DURATION
|
||||
Concurrency: $CONCURRENCY
|
||||
|
||||
Environment Variables:
|
||||
RUSTFS_GET_METADATA_EARLY_STOP_ENABLE=${RUSTFS_GET_METADATA_EARLY_STOP_ENABLE:-true}
|
||||
RUSTFS_GET_CODEC_STREAMING_ENABLE=${RUSTFS_GET_CODEC_STREAMING_ENABLE:-false}
|
||||
RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE=${RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE:-true}
|
||||
EOF
|
||||
|
||||
# ============================================================================
|
||||
# Test 1: Data Correctness Validation
|
||||
# ============================================================================
|
||||
echo "[1/4] Data Correctness Validation"
|
||||
echo "=================================="
|
||||
|
||||
test_data_correctness() {
|
||||
local size=$1
|
||||
local count=$2
|
||||
local passed=0
|
||||
local failed=0
|
||||
|
||||
echo " Testing $count objects of size $size..."
|
||||
|
||||
for i in $(seq 1 $count); do
|
||||
# Generate test file
|
||||
local testfile="/tmp/stress-test-${size}-${i}.bin"
|
||||
if [ "$size" = "1KB" ]; then
|
||||
dd if=/dev/urandom of="$testfile" bs=1024 count=1 2>/dev/null
|
||||
elif [ "$size" = "1MB" ]; then
|
||||
dd if=/dev/urandom of="$testfile" bs=1048576 count=1 2>/dev/null
|
||||
elif [ "$size" = "10MB" ]; then
|
||||
dd if=/dev/urandom of="$testfile" bs=1048576 count=10 2>/dev/null
|
||||
fi
|
||||
|
||||
# Calculate original hash
|
||||
local original_hash
|
||||
original_hash=$(md5 -q "$testfile" 2>/dev/null || md5sum "$testfile" | awk '{print $1}')
|
||||
|
||||
# Upload
|
||||
mc cp "$testfile" "${MC_ALIAS}/${TEST_BUCKET}/correctness-${size}-${i}.bin" >/dev/null 2>&1
|
||||
|
||||
# Download and verify
|
||||
local downloadfile="/tmp/stress-test-${size}-${i}-download.bin"
|
||||
mc cp "${MC_ALIAS}/${TEST_BUCKET}/correctness-${size}-${i}.bin" "$downloadfile" >/dev/null 2>&1
|
||||
|
||||
local download_hash
|
||||
download_hash=$(md5 -q "$downloadfile" 2>/dev/null || md5sum "$downloadfile" | awk '{print $1}')
|
||||
|
||||
if [ "$original_hash" = "$download_hash" ]; then
|
||||
passed=$((passed + 1))
|
||||
else
|
||||
failed=$((failed + 1))
|
||||
echo " MISMATCH: correctness-${size}-${i}.bin (original=$original_hash, download=$download_hash)" >> "$OUTPUT_DIR/correctness-errors.log"
|
||||
fi
|
||||
|
||||
# Cleanup
|
||||
rm -f "$testfile" "$downloadfile"
|
||||
done
|
||||
|
||||
echo " Result: $passed passed, $failed failed"
|
||||
echo "correctness_${size}: passed=$passed failed=$failed" >> "$OUTPUT_DIR/correctness-results.txt"
|
||||
}
|
||||
|
||||
# Test different object sizes
|
||||
mc mb "${MC_ALIAS}/${TEST_BUCKET}" 2>/dev/null || true
|
||||
|
||||
test_data_correctness "1KB" 100
|
||||
test_data_correctness "1MB" 50
|
||||
test_data_correctness "10MB" 10
|
||||
|
||||
echo ""
|
||||
|
||||
# ============================================================================
|
||||
# Test 2: Concurrent GET Stress Test
|
||||
# ============================================================================
|
||||
echo "[2/4] Concurrent GET Stress Test"
|
||||
echo "================================="
|
||||
|
||||
# Prepare test objects
|
||||
echo " Preparing test objects..."
|
||||
for size in 1KiB 1MiB 4MiB 10MiB; do
|
||||
warp put --obj.size="$size" --num.objects=100 --host="$TARGET_HOST" --bucket="$TEST_BUCKET" --concurrent=16 >/dev/null 2>&1
|
||||
done
|
||||
|
||||
echo " Running concurrent GET tests..."
|
||||
|
||||
for size in 1KiB 1MiB 4MiB 10MiB; do
|
||||
for conc in 16 64 256; do
|
||||
echo " GET size=$size concurrency=$conc duration=$TEST_DURATION"
|
||||
output_file="$OUTPUT_DIR/get-${size}-c${conc}.json"
|
||||
|
||||
if warp get \
|
||||
--host="$TARGET_HOST" \
|
||||
--obj.size="$size" \
|
||||
--concurrent="$conc" \
|
||||
--duration="$TEST_DURATION" \
|
||||
--bucket="$TEST_BUCKET" \
|
||||
--json \
|
||||
> "$output_file" 2>/dev/null; then
|
||||
echo " OK"
|
||||
else
|
||||
echo " FAILED (see $output_file)"
|
||||
fi
|
||||
done
|
||||
done
|
||||
|
||||
echo ""
|
||||
|
||||
# ============================================================================
|
||||
# Test 3: Mixed Read/Write Stress Test
|
||||
# ============================================================================
|
||||
echo "[3/4] Mixed Read/Write Stress Test"
|
||||
echo "==================================="
|
||||
|
||||
echo " Running mixed workload for $TEST_DURATION..."
|
||||
|
||||
warp mixed \
|
||||
--host="$TARGET_HOST" \
|
||||
--obj.size=1MiB \
|
||||
--concurrent="$CONCURRENCY" \
|
||||
--duration="$TEST_DURATION" \
|
||||
--bucket="$TEST_BUCKET" \
|
||||
--json \
|
||||
> "$OUTPUT_DIR/mixed-1MiB-c${CONCURRENCY}.json" 2>/dev/null || echo " MIXED TEST FAILED"
|
||||
|
||||
echo " Mixed test complete"
|
||||
echo ""
|
||||
|
||||
# ============================================================================
|
||||
# Test 4: Early-Stop Behavior Validation
|
||||
# ============================================================================
|
||||
echo "[4/4] Early-Stop Behavior Validation"
|
||||
echo "====================================="
|
||||
|
||||
# This test verifies that early-stop works correctly by checking:
|
||||
# 1. All GET requests return correct data
|
||||
# 2. No timeouts or errors under normal conditions
|
||||
# 3. Performance is consistent
|
||||
|
||||
echo " Running early-stop validation with concurrent reads..."
|
||||
|
||||
# Create a test object
|
||||
dd if=/dev/urandom of=/tmp/early-stop-test.bin bs=1048576 count=10 2>/dev/null
|
||||
mc cp /tmp/early-stop-test.bin "${MC_ALIAS}/${TEST_BUCKET}/early-stop-test.bin" >/dev/null 2>&1
|
||||
|
||||
# Concurrent read test
|
||||
local_passed=0
|
||||
local_failed=0
|
||||
|
||||
for i in $(seq 1 100); do
|
||||
downloadfile="/tmp/early-stop-download-${i}.bin"
|
||||
if mc cp "${MC_ALIAS}/${TEST_BUCKET}/early-stop-test.bin" "$downloadfile" >/dev/null 2>&1; then
|
||||
# Verify file size
|
||||
local_size=$(stat -f%z "$downloadfile" 2>/dev/null || stat -c%s "$downloadfile" 2>/dev/null)
|
||||
if [ "$local_size" = "10485760" ]; then
|
||||
local_passed=$((local_passed + 1))
|
||||
else
|
||||
local_failed=$((local_failed + 1))
|
||||
echo " SIZE MISMATCH: expected 10485760, got $local_size" >> "$OUTPUT_DIR/early-stop-errors.log"
|
||||
fi
|
||||
else
|
||||
local_failed=$((local_failed + 1))
|
||||
echo " DOWNLOAD FAILED: iteration $i" >> "$OUTPUT_DIR/early-stop-errors.log"
|
||||
fi
|
||||
rm -f "$downloadfile"
|
||||
done
|
||||
|
||||
echo " Early-stop validation: $local_passed passed, $local_failed failed"
|
||||
echo "early_stop_validation: passed=$local_passed failed=$local_failed" >> "$OUTPUT_DIR/early-stop-results.txt"
|
||||
|
||||
# Cleanup
|
||||
rm -f /tmp/early-stop-test.bin
|
||||
mc rm "${MC_ALIAS}/${TEST_BUCKET}/early-stop-test.bin" >/dev/null 2>&1 || true
|
||||
|
||||
echo ""
|
||||
|
||||
# ============================================================================
|
||||
# Summary
|
||||
# ============================================================================
|
||||
echo "=========================================="
|
||||
echo "Stress Test Summary"
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
echo "Results saved to: $OUTPUT_DIR"
|
||||
echo ""
|
||||
echo "Files:"
|
||||
ls -la "$OUTPUT_DIR"
|
||||
echo ""
|
||||
echo "Review:"
|
||||
echo " - correctness-results.txt: Data correctness validation"
|
||||
echo " - early-stop-results.txt: Early-stop behavior validation"
|
||||
echo " - get-*.json: Concurrent GET performance results"
|
||||
echo " - mixed-*.json: Mixed workload results"
|
||||
echo ""
|
||||
|
||||
# Cleanup test bucket
|
||||
echo "Cleaning up test bucket..."
|
||||
mc rm --recursive --force "${MC_ALIAS}/${TEST_BUCKET}" >/dev/null 2>&1 || true
|
||||
mc rb "${MC_ALIAS}/${TEST_BUCKET}" >/dev/null 2>&1 || true
|
||||
|
||||
echo "Done!"
|
||||
Reference in New Issue
Block a user