mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-07 05:43:14 +00:00
chore(scripts): add scripts index and archive one-shot scripts (#4822)
chore(scripts): index scripts/ and archive 29 one-shot scripts backlog#1153 infra-13. scripts/ had 80+ unlabelled top-level entries mixing CI gates with finished one-shot issue-validation scripts. - scripts/README.md — one index row per entry with status (ci-gate / dev-tool / archived), purpose, and wiring; subdirectories get one row each. run_scanner_benchmarks.sh is annotated "disposition owned by backlog perf-10" and deliberately untouched. - git mv 29 confirmed-stale one-shot entries to scripts/archive/: 11 issue-scoped validation/perf-capture scripts, the 5-script backlog#706 large-PUT breakdown family, the 4-file GET-optimization stress suite, 2 gt1g one-shots, and 7 other orphaned one-shots. Evidence: a whole-tree boundary-aware reference census showed zero references from CI/Makefiles/docs/code for every moved entry (or references only from other scripts inside the same archived set); re-run after the move shows zero dangling references. - docs/testing/README.md links the index. Moves only — no script content changed.
This commit is contained in:
@@ -0,0 +1,119 @@
|
||||
# GET Optimization Stress Test Scripts
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- `warp` installed (https://github.com/minio/warp)
|
||||
- `mc` configured with access to the RustFS server
|
||||
- RustFS server running with GET optimizations enabled
|
||||
|
||||
### Quick Validation (5 minutes)
|
||||
|
||||
```bash
|
||||
./scripts/quick-validate-get-optimization.sh localhost:9000
|
||||
```
|
||||
|
||||
This runs basic functional tests:
|
||||
- Data integrity verification
|
||||
- Concurrent GET stability
|
||||
- Early-stop behavior validation
|
||||
|
||||
### Full Stress Test (30+ minutes)
|
||||
|
||||
```bash
|
||||
./scripts/stress-test-get-optimization.sh localhost:9000 ./stress-results
|
||||
```
|
||||
|
||||
This runs comprehensive tests:
|
||||
- Data correctness validation (1KB, 1MB, 10MB objects)
|
||||
- Concurrent GET stress test (16, 64, 256 concurrency)
|
||||
- Mixed read/write workload
|
||||
- Early-stop behavior under load
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `WARP_ACCESS_KEY` | rustfsadmin | S3 access key |
|
||||
| `WARP_SECRET_KEY` | rustfsadmin | S3 secret key |
|
||||
| `MC_ALIAS` | rustfs | mc alias for the server |
|
||||
| `TEST_BUCKET` | auto-generated | Test bucket name |
|
||||
| `TEST_DURATION` | 300s | Duration for stress tests |
|
||||
| `CONCURRENCY` | 64 | Default concurrency level |
|
||||
|
||||
## Test Scenarios
|
||||
|
||||
### 1. Data Correctness Validation
|
||||
|
||||
Verifies that GET returns correct data with early-stop enabled:
|
||||
- Uploads random data
|
||||
- Downloads and compares MD5 hash
|
||||
- Tests different object sizes (1KB, 1MB, 10MB)
|
||||
|
||||
### 2. Concurrent GET Stress Test
|
||||
|
||||
Tests performance under high concurrency:
|
||||
- Object sizes: 1KiB, 1MiB, 4MiB, 10MiB
|
||||
- Concurrency levels: 16, 64, 256
|
||||
- Duration: configurable (default 300s)
|
||||
|
||||
### 3. Mixed Read/Write Stress Test
|
||||
|
||||
Tests stability under concurrent read/write:
|
||||
- 50% reads, 50% writes
|
||||
- 1MB objects
|
||||
- 64 concurrent operations
|
||||
|
||||
### 4. Early-Stop Behavior Validation
|
||||
|
||||
Verifies early-stop works correctly:
|
||||
- 100 sequential reads of the same object
|
||||
- Verifies data size matches expected
|
||||
- Checks for any download failures
|
||||
|
||||
## Expected Results
|
||||
|
||||
### Success Criteria
|
||||
|
||||
- **Data Correctness**: 100% pass rate (no data corruption)
|
||||
- **Concurrent GET**: No errors, consistent latency
|
||||
- **Mixed Workload**: No deadlocks or data corruption
|
||||
- **Early-Stop**: All reads return correct data size
|
||||
|
||||
### Performance Baselines
|
||||
|
||||
| Object Size | Expected Throughput | Expected p95 Latency |
|
||||
|-------------|--------------------|--------------------|
|
||||
| 1KiB | > 5 MiB/s | < 10ms |
|
||||
| 1MiB | > 500 MiB/s | < 20ms |
|
||||
| 4MiB | > 1000 MiB/s | < 30ms |
|
||||
| 10MiB | > 2000 MiB/s | < 50ms |
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **warp not found**: Install with `go install github.com/minio/warp@latest`
|
||||
2. **mc not configured**: Run `mc alias set rustfs http://localhost:9000 admin password`
|
||||
3. **Connection refused**: Verify RustFS is running and accessible
|
||||
4. **Permission denied**: Check S3 credentials
|
||||
|
||||
### Debug Mode
|
||||
|
||||
Enable debug logging:
|
||||
```bash
|
||||
RUST_LOG=rustfs_ecstore::bucket::lifecycle=debug ./scripts/stress-test-get-optimization.sh
|
||||
```
|
||||
|
||||
## Output Files
|
||||
|
||||
| File | Description |
|
||||
|------|-------------|
|
||||
| `test-config.txt` | Test configuration |
|
||||
| `correctness-results.txt` | Data correctness results |
|
||||
| `early-stop-results.txt` | Early-stop validation results |
|
||||
| `get-*.json` | Concurrent GET performance (warp JSON) |
|
||||
| `mixed-*.json` | Mixed workload performance (warp JSON) |
|
||||
| `correctness-errors.log` | Data correctness errors (if any) |
|
||||
| `early-stop-errors.log` | Early-stop errors (if any) |
|
||||
@@ -0,0 +1,322 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Summarize per-round PUT service metric deltas from before/after snapshots."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import csv
|
||||
import re
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
PROM_LINE_RE = re.compile(
|
||||
r"^([a-zA-Z_:][a-zA-Z0-9_:]*)(?:\{(.*)\})?\s+([-+]?(?:\d+(?:\.\d*)?|\.\d+)(?:[eE][-+]?\d+)?)"
|
||||
)
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--capture-csv", required=True, help="aggregate_service_metrics_captures.csv path")
|
||||
parser.add_argument("--delta-csv", required=True, help="Output generic metric delta CSV")
|
||||
parser.add_argument("--path-summary-csv", required=True, help="Output PUT path delta summary CSV")
|
||||
parser.add_argument("--stage-summary-csv", required=True, help="Output PUT stage duration summary CSV")
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def split_label_items(labels: str) -> list[str]:
|
||||
items: list[str] = []
|
||||
start = 0
|
||||
in_quotes = False
|
||||
escaped = False
|
||||
for index, char in enumerate(labels):
|
||||
if escaped:
|
||||
escaped = False
|
||||
continue
|
||||
if char == "\\":
|
||||
escaped = True
|
||||
continue
|
||||
if char == '"':
|
||||
in_quotes = not in_quotes
|
||||
continue
|
||||
if char == "," and not in_quotes:
|
||||
items.append(labels[start:index])
|
||||
start = index + 1
|
||||
items.append(labels[start:])
|
||||
return [item.strip() for item in items if item.strip()]
|
||||
|
||||
|
||||
def parse_labels(labels: str) -> dict[str, str]:
|
||||
parsed: dict[str, str] = {}
|
||||
for item in split_label_items(labels):
|
||||
key, sep, value = item.partition("=")
|
||||
if not sep:
|
||||
continue
|
||||
value = value.strip()
|
||||
if value.startswith('"') and value.endswith('"') and len(value) >= 2:
|
||||
value = value[1:-1]
|
||||
value = value.replace(r"\"", '"').replace(r"\\", "\\")
|
||||
parsed[key.strip()] = value
|
||||
return parsed
|
||||
|
||||
|
||||
def canonical_labels(labels: dict[str, str]) -> str:
|
||||
return ",".join(f'{key}="{escape_label_value(labels[key])}"' for key in sorted(labels))
|
||||
|
||||
|
||||
def escape_label_value(value: str) -> str:
|
||||
return value.replace("\\", r"\\").replace('"', r"\"").replace("\n", r"\n")
|
||||
|
||||
|
||||
def parse_prom(path: Path) -> dict[tuple[str, str], float]:
|
||||
metrics: dict[tuple[str, str], float] = {}
|
||||
with path.open("r", encoding="utf-8") as file:
|
||||
for line in file:
|
||||
line = line.strip()
|
||||
if not line or line.startswith("#"):
|
||||
continue
|
||||
match = PROM_LINE_RE.match(line)
|
||||
if not match:
|
||||
continue
|
||||
metric, raw_labels, raw_value = match.groups()
|
||||
labels = canonical_labels(parse_labels(raw_labels or ""))
|
||||
try:
|
||||
value = float(raw_value)
|
||||
except ValueError:
|
||||
continue
|
||||
metrics[(metric, labels)] = value
|
||||
return metrics
|
||||
|
||||
|
||||
def classify_metric(metric: str) -> str:
|
||||
if metric == "rustfs_s3_put_object_path_total":
|
||||
return "path_total"
|
||||
if metric == "rustfs_s3_put_object_diagnostics_total":
|
||||
return "diagnostics_total"
|
||||
if metric.endswith("_bucket"):
|
||||
return "histogram_bucket"
|
||||
if metric.endswith("_count"):
|
||||
return "histogram_count"
|
||||
if metric.endswith("_sum"):
|
||||
return "histogram_sum"
|
||||
if metric.endswith("_total"):
|
||||
return "counter_total"
|
||||
return "gauge_delta"
|
||||
|
||||
|
||||
def is_monotonic_metric(metric: str) -> bool:
|
||||
return (
|
||||
metric.endswith("_total")
|
||||
or metric.endswith("_bucket")
|
||||
or metric.endswith("_count")
|
||||
or metric.endswith("_sum")
|
||||
)
|
||||
|
||||
|
||||
def group_capture_rows(capture_csv: Path) -> dict[tuple[str, str, str, str, str, str], dict[str, dict[str, str]]]:
|
||||
groups: dict[tuple[str, str, str, str, str, str], dict[str, dict[str, str]]] = defaultdict(dict)
|
||||
with capture_csv.open("r", encoding="utf-8", newline="") as file:
|
||||
for row in csv.DictReader(file):
|
||||
phase = row.get("phase", "")
|
||||
if phase not in {"before", "after"}:
|
||||
continue
|
||||
key = (
|
||||
row.get("concurrency", ""),
|
||||
row.get("run_dir", ""),
|
||||
row.get("size", ""),
|
||||
row.get("tool", ""),
|
||||
row.get("round", ""),
|
||||
row.get("attempt", ""),
|
||||
)
|
||||
groups[key][phase] = row
|
||||
return groups
|
||||
|
||||
|
||||
def label_value(labels: str, name: str) -> str:
|
||||
return parse_labels(labels).get(name, "")
|
||||
|
||||
|
||||
def write_outputs(args: argparse.Namespace) -> None:
|
||||
capture_csv = Path(args.capture_csv)
|
||||
delta_csv = Path(args.delta_csv)
|
||||
path_summary_csv = Path(args.path_summary_csv)
|
||||
stage_summary_csv = Path(args.stage_summary_csv)
|
||||
for output in (delta_csv, path_summary_csv, stage_summary_csv):
|
||||
output.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
groups = group_capture_rows(capture_csv)
|
||||
|
||||
delta_fields = [
|
||||
"concurrency",
|
||||
"run_dir",
|
||||
"size",
|
||||
"tool",
|
||||
"round",
|
||||
"attempt",
|
||||
"source",
|
||||
"before_status",
|
||||
"after_status",
|
||||
"metric",
|
||||
"labels",
|
||||
"before",
|
||||
"after",
|
||||
"delta",
|
||||
"classification",
|
||||
]
|
||||
path_fields = [
|
||||
"concurrency",
|
||||
"run_dir",
|
||||
"size",
|
||||
"tool",
|
||||
"round",
|
||||
"attempt",
|
||||
"path",
|
||||
"delta",
|
||||
"before_status",
|
||||
"after_status",
|
||||
]
|
||||
stage_fields = [
|
||||
"concurrency",
|
||||
"run_dir",
|
||||
"size",
|
||||
"tool",
|
||||
"round",
|
||||
"attempt",
|
||||
"stage",
|
||||
"count_delta",
|
||||
"sum_delta",
|
||||
"avg_ms",
|
||||
"before_status",
|
||||
"after_status",
|
||||
]
|
||||
|
||||
with (
|
||||
delta_csv.open("w", encoding="utf-8", newline="") as delta_file,
|
||||
path_summary_csv.open("w", encoding="utf-8", newline="") as path_file,
|
||||
stage_summary_csv.open("w", encoding="utf-8", newline="") as stage_file,
|
||||
):
|
||||
delta_writer = csv.DictWriter(delta_file, fieldnames=delta_fields)
|
||||
path_writer = csv.DictWriter(path_file, fieldnames=path_fields)
|
||||
stage_writer = csv.DictWriter(stage_file, fieldnames=stage_fields)
|
||||
delta_writer.writeheader()
|
||||
path_writer.writeheader()
|
||||
stage_writer.writeheader()
|
||||
|
||||
for key in sorted(groups):
|
||||
phases = groups[key]
|
||||
before_row = phases.get("before")
|
||||
after_row = phases.get("after")
|
||||
if not before_row or not after_row:
|
||||
continue
|
||||
|
||||
before_path = Path(before_row.get("snapshot_file", ""))
|
||||
after_path = Path(after_row.get("snapshot_file", ""))
|
||||
if before_row.get("status") != "ok" or after_row.get("status") != "ok":
|
||||
continue
|
||||
if not before_path.is_file() or not after_path.is_file():
|
||||
continue
|
||||
|
||||
before = parse_prom(before_path)
|
||||
after = parse_prom(after_path)
|
||||
concurrency, run_dir, size, tool, round_no, attempt = key
|
||||
source = after_row.get("source", before_row.get("source", ""))
|
||||
before_status = before_row.get("status", "")
|
||||
after_status = after_row.get("status", "")
|
||||
|
||||
stage_counts: dict[str, float] = defaultdict(float)
|
||||
stage_sums: dict[str, float] = defaultdict(float)
|
||||
|
||||
for metric_key in sorted(after):
|
||||
metric, labels = metric_key
|
||||
before_value = before.get(metric_key, 0.0)
|
||||
after_value = after.get(metric_key, 0.0)
|
||||
delta = after_value - before_value
|
||||
if is_monotonic_metric(metric) and delta < 0:
|
||||
continue
|
||||
if delta == 0:
|
||||
continue
|
||||
|
||||
classification = classify_metric(metric)
|
||||
delta_writer.writerow(
|
||||
{
|
||||
"concurrency": concurrency,
|
||||
"run_dir": run_dir,
|
||||
"size": size,
|
||||
"tool": tool,
|
||||
"round": round_no,
|
||||
"attempt": attempt,
|
||||
"source": source,
|
||||
"before_status": before_status,
|
||||
"after_status": after_status,
|
||||
"metric": metric,
|
||||
"labels": labels,
|
||||
"before": f"{before_value:.12g}",
|
||||
"after": f"{after_value:.12g}",
|
||||
"delta": f"{delta:.12g}",
|
||||
"classification": classification,
|
||||
}
|
||||
)
|
||||
|
||||
if metric == "rustfs_s3_put_object_path_total":
|
||||
path_writer.writerow(
|
||||
{
|
||||
"concurrency": concurrency,
|
||||
"run_dir": run_dir,
|
||||
"size": size,
|
||||
"tool": tool,
|
||||
"round": round_no,
|
||||
"attempt": attempt,
|
||||
"path": label_value(labels, "path"),
|
||||
"delta": f"{delta:.12g}",
|
||||
"before_status": before_status,
|
||||
"after_status": after_status,
|
||||
}
|
||||
)
|
||||
|
||||
if metric.endswith("_stage_duration_ms_count"):
|
||||
stage = label_value(labels, "stage")
|
||||
if stage:
|
||||
stage_counts[stage] += delta
|
||||
elif metric.endswith("_stage_duration_ms_sum"):
|
||||
stage = label_value(labels, "stage")
|
||||
if stage:
|
||||
stage_sums[stage] += delta
|
||||
elif metric.endswith("_stage_duration_seconds_count"):
|
||||
stage = label_value(labels, "stage")
|
||||
if stage:
|
||||
stage_counts[stage] += delta
|
||||
elif metric.endswith("_stage_duration_seconds_sum"):
|
||||
stage = label_value(labels, "stage")
|
||||
if stage:
|
||||
stage_sums[stage] += delta * 1000.0
|
||||
|
||||
for stage in sorted(set(stage_counts) | set(stage_sums)):
|
||||
count_delta = stage_counts.get(stage, 0.0)
|
||||
if count_delta <= 0:
|
||||
continue
|
||||
sum_delta = stage_sums.get(stage, 0.0)
|
||||
avg_ms = sum_delta / count_delta if count_delta else 0.0
|
||||
stage_writer.writerow(
|
||||
{
|
||||
"concurrency": concurrency,
|
||||
"run_dir": run_dir,
|
||||
"size": size,
|
||||
"tool": tool,
|
||||
"round": round_no,
|
||||
"attempt": attempt,
|
||||
"stage": stage,
|
||||
"count_delta": f"{count_delta:.12g}",
|
||||
"sum_delta": f"{sum_delta:.12g}",
|
||||
"avg_ms": f"{avg_ms:.6f}",
|
||||
"before_status": before_status,
|
||||
"after_status": after_status,
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
write_outputs(parse_args())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Executable
+166
@@ -0,0 +1,166 @@
|
||||
#!/usr/bin/env bash
|
||||
# Benchmark script for GET small-file optimization (SF01-SF07)
|
||||
# Matches baseline parameters from issue714-local-single-machine-multidisk-get-2026-06-26.md
|
||||
set -euo pipefail
|
||||
|
||||
WARP_HOST="${WARP_HOST:-127.0.0.1:19031}"
|
||||
export WARP_ACCESS_KEY="${WARP_ACCESS_KEY:-rustfsadmin}"
|
||||
export WARP_SECRET_KEY="${WARP_SECRET_KEY:-rustfsadmin}"
|
||||
|
||||
SIZES="1KiB 4KiB 10KiB 100KiB 1MiB"
|
||||
CONCURRENCY=32
|
||||
DURATION=10s
|
||||
ROUNDS=3
|
||||
COOLDOWN=10
|
||||
OBJECTS=8
|
||||
OUT_DIR="target/bench/sf-optimization-$(date +%Y%m%d-%H%M%S)"
|
||||
|
||||
mkdir -p "$OUT_DIR"
|
||||
|
||||
echo "=========================================="
|
||||
echo "GET Small-File Optimization Benchmark"
|
||||
echo "=========================================="
|
||||
echo "Host: $WARP_HOST"
|
||||
echo "Output: $OUT_DIR"
|
||||
echo ""
|
||||
|
||||
# Save environment info
|
||||
cat > "$OUT_DIR/meta.env" <<EOF
|
||||
DATE=$(date -u +%Y-%m-%dT%H:%M:%SZ)
|
||||
HOST=$WARP_HOST
|
||||
CONCURRENCY=$CONCURRENCY
|
||||
DURATION=$DURATION
|
||||
ROUNDS=$ROUNDS
|
||||
OBJECTS=$OBJECTS
|
||||
BRANCH=$(git rev-parse --abbrev-ref HEAD)
|
||||
COMMIT=$(git rev-parse --short HEAD)
|
||||
RUST_VERSION=$(rustc --version)
|
||||
EOF
|
||||
|
||||
# CSV header for summary
|
||||
echo "size,round,throughput_mib_s,requests_per_s,p50_ms,total_bytes,errors" > "$OUT_DIR/summary.csv"
|
||||
|
||||
for size in $SIZES; do
|
||||
echo ""
|
||||
echo "=========================================="
|
||||
echo "Testing: $size"
|
||||
echo "=========================================="
|
||||
|
||||
for round in $(seq 1 $ROUNDS); do
|
||||
echo " Round $round/$ROUNDS..."
|
||||
|
||||
json_file="$OUT_DIR/get-${size}-round${round}.json"
|
||||
|
||||
warp get \
|
||||
--host="$WARP_HOST" \
|
||||
--obj.size="$size" \
|
||||
--concurrent=$CONCURRENCY \
|
||||
--duration=$DURATION \
|
||||
--objects=$OBJECTS \
|
||||
--noclear \
|
||||
--lookup=path \
|
||||
--analyze.out="$json_file" \
|
||||
2>/dev/null
|
||||
|
||||
# Extract key metrics from JSON
|
||||
if [ -f "$json_file" ]; then
|
||||
throughput=$(python3 -c "
|
||||
import json, sys
|
||||
with open('$json_file') as f:
|
||||
data = json.load(f)
|
||||
for op in data.get('operations', []):
|
||||
if op.get('operation') == 'GET':
|
||||
mb_s = op.get('mb_per_sec', 0)
|
||||
rps = op.get('requests_per_sec', 0)
|
||||
p50 = 0
|
||||
for t in op.get('throughput', []):
|
||||
pass
|
||||
# Get p50 from time_series_aggregated
|
||||
tsa = op.get('time_series_aggregated', {})
|
||||
if tsa:
|
||||
p50 = tsa.get('median_ms', 0)
|
||||
total = op.get('total_bytes', 0)
|
||||
errors = op.get('requests_errors', 0) or 0
|
||||
print(f'{mb_s:.2f},{rps:.2f},{p50:.1f},{total},{errors}')
|
||||
sys.exit(0)
|
||||
print('0,0,0,0,0')
|
||||
" 2>/dev/null || echo "0,0,0,0,0")
|
||||
|
||||
echo "$size,$round,$throughput" >> "$OUT_DIR/summary.csv"
|
||||
echo " -> $throughput"
|
||||
else
|
||||
echo "$size,$round,0,0,0,0,0" >> "$OUT_DIR/summary.csv"
|
||||
echo " -> FAILED (no output)"
|
||||
fi
|
||||
|
||||
echo " Cooling down ${COOLDOWN}s..."
|
||||
sleep $COOLDOWN
|
||||
done
|
||||
|
||||
# Extra cooldown between sizes
|
||||
echo " Extra cooldown ${COOLDOWN}s between sizes..."
|
||||
sleep $COOLDOWN
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "=========================================="
|
||||
echo "Benchmark Complete"
|
||||
echo "=========================================="
|
||||
echo "Results: $OUT_DIR/summary.csv"
|
||||
echo ""
|
||||
|
||||
# Print summary table
|
||||
echo "Summary (MiB/s):"
|
||||
echo "---------------------------------------------------"
|
||||
printf "%-10s" "Size"
|
||||
for r in $(seq 1 $ROUNDS); do
|
||||
printf "%-12s" "Round $r"
|
||||
done
|
||||
printf "%-12s\n" "Median"
|
||||
echo "---------------------------------------------------"
|
||||
|
||||
for size in $SIZES; do
|
||||
printf "%-10s" "$size"
|
||||
values=()
|
||||
for r in $(seq 1 $ROUNDS); do
|
||||
val=$(grep "^$size,$r," "$OUT_DIR/summary.csv" | cut -d',' -f3)
|
||||
values+=("$val")
|
||||
printf "%-12s" "$val"
|
||||
done
|
||||
# Calculate median
|
||||
median=$(printf '%s\n' "${values[@]}" | sort -n | sed -n "$(((${#values[@]}+1)/2))p")
|
||||
printf "%-12s\n" "$median"
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "Comparison with baseline (MiB/s):"
|
||||
echo "---------------------------------------------------"
|
||||
printf "%-10s %-12s %-12s %-12s %-12s\n" "Size" "MinIO" "RustFS main" "This branch" "vs main"
|
||||
echo "---------------------------------------------------"
|
||||
|
||||
# Baseline data from issue714
|
||||
declare -A MINIO_DATA=(
|
||||
["1KiB"]="21.15" ["4KiB"]="51.19" ["10KiB"]="201.23" ["100KiB"]="1142.89" ["1MiB"]="7264.10"
|
||||
)
|
||||
declare -A MAIN_DATA=(
|
||||
["1KiB"]="2.88" ["4KiB"]="11.30" ["10KiB"]="28.56" ["100KiB"]="277.49" ["1MiB"]="2270.27"
|
||||
)
|
||||
|
||||
for size in $SIZES; do
|
||||
values=()
|
||||
for r in $(seq 1 $ROUNDS); do
|
||||
val=$(grep "^$size,$r," "$OUT_DIR/summary.csv" | cut -d',' -f3)
|
||||
values+=("$val")
|
||||
done
|
||||
median=$(printf '%s\n' "${values[@]}" | sort -n | sed -n "$(((${#values[@]}+1)/2))p")
|
||||
main_val=${MAIN_DATA[$size]}
|
||||
minio_val=${MINIO_DATA[$size]}
|
||||
|
||||
if [ "$main_val" != "0" ] && [ "$main_val" != "" ]; then
|
||||
pct_change=$(python3 -c "print(f'{(($median - $main_val) / $main_val * 100):+.1f}%')" 2>/dev/null || echo "N/A")
|
||||
else
|
||||
pct_change="N/A"
|
||||
fi
|
||||
|
||||
printf "%-10s %-12s %-12s %-12s %-12s\n" "$size" "$minio_val" "$main_val" "$median" "$pct_change"
|
||||
done
|
||||
@@ -0,0 +1,491 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# Helper artifact collector for rustfs/backlog#706 large-object PUT stage-breakdown runs.
|
||||
# Designed to pair with scripts/run_put_large_stage_breakdown.sh and store
|
||||
# supporting evidence under:
|
||||
# <run-root>/captures/<label>/
|
||||
#
|
||||
# Captures:
|
||||
# - health and readiness snapshots
|
||||
# - signed admin metrics snapshots per endpoint
|
||||
# - optional plain Prometheus text snapshots
|
||||
# - optional host telemetry (pidstat/iostat/mpstat)
|
||||
# - per-sample process snapshots when a rustfs pid is available
|
||||
|
||||
RUN_ROOT=""
|
||||
LABEL=""
|
||||
ENDPOINT=""
|
||||
ACCESS_KEY="${RUSTFS_ACCESS_KEY:-}"
|
||||
SECRET_KEY=""
|
||||
SECRET_KEY_ENV="RUSTFS_SECRET_KEY"
|
||||
REGION="us-east-1"
|
||||
METRICS_ENDPOINTS=""
|
||||
PROM_METRICS_URLS=""
|
||||
DURATION_SECS=180
|
||||
INTERVAL_SECS=15
|
||||
RUSTFS_PID=""
|
||||
SKIP_HOST_TELEMETRY=false
|
||||
AWSCURL_BIN="awscurl"
|
||||
CURL_BIN="curl"
|
||||
JQ_BIN="jq"
|
||||
DRY_RUN=false
|
||||
AWSCURL_AVAILABLE=true
|
||||
|
||||
usage() {
|
||||
cat <<'USAGE'
|
||||
Usage:
|
||||
scripts/collect_put_large_stage_breakdown_artifacts.sh --run-root <dir> \
|
||||
--endpoint <url> [options]
|
||||
|
||||
Required:
|
||||
--run-root <dir> Benchmark run root, usually from
|
||||
scripts/run_put_large_stage_breakdown.sh
|
||||
--endpoint <url> RustFS endpoint, e.g. http://127.0.0.1:9000
|
||||
|
||||
Auth / signed admin metrics:
|
||||
--access-key <ak> Override RUSTFS_ACCESS_KEY
|
||||
--secret-key-env <var> Secret-key environment variable
|
||||
(default: RUSTFS_SECRET_KEY)
|
||||
--region <name> SigV4 region (default: us-east-1)
|
||||
|
||||
Capture options:
|
||||
--label <name> Capture label (default: capture-<timestamp>)
|
||||
--metrics-endpoints <csv> Signed admin metrics endpoints. If omitted,
|
||||
defaults to --endpoint.
|
||||
--prom-metrics-urls <csv> Optional plain Prometheus text endpoints
|
||||
captured with curl.
|
||||
--duration-secs <n> Total capture duration (default: 180)
|
||||
--interval-secs <n> Sample interval (default: 15)
|
||||
--rustfs-pid <pid> RustFS pid. Auto-detect with pidof rustfs if omitted.
|
||||
--skip-host-telemetry Skip pidstat/iostat/mpstat collection.
|
||||
--dry-run Print intended actions without live capture.
|
||||
|
||||
Output:
|
||||
<run-root>/captures/<label>/
|
||||
capture-meta.txt
|
||||
capture-layout.txt
|
||||
capture-report.md
|
||||
health/
|
||||
metrics/
|
||||
prom/
|
||||
host/
|
||||
USAGE
|
||||
}
|
||||
|
||||
require_cmd() {
|
||||
if ! command -v "$1" >/dev/null 2>&1; then
|
||||
echo "ERROR: command not found: $1" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
arg_value() {
|
||||
local flag="$1"
|
||||
local value="${2:-}"
|
||||
if [[ -z "$value" || "$value" == --* ]]; then
|
||||
echo "ERROR: missing value for $flag" >&2
|
||||
exit 1
|
||||
fi
|
||||
printf '%s\n' "$value"
|
||||
}
|
||||
|
||||
parse_args() {
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--run-root) RUN_ROOT="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--label) LABEL="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--endpoint) ENDPOINT="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--access-key) ACCESS_KEY="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--secret-key-env) SECRET_KEY_ENV="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--region) REGION="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--metrics-endpoints) METRICS_ENDPOINTS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--prom-metrics-urls) PROM_METRICS_URLS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--duration-secs) DURATION_SECS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--interval-secs) INTERVAL_SECS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--rustfs-pid) RUSTFS_PID="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--skip-host-telemetry) SKIP_HOST_TELEMETRY=true; shift ;;
|
||||
--dry-run) DRY_RUN=true; shift ;;
|
||||
-h|--help) usage; exit 0 ;;
|
||||
*)
|
||||
echo "ERROR: unknown arg: $1" >&2
|
||||
usage
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
}
|
||||
|
||||
is_nonnegative_integer() {
|
||||
[[ "$1" =~ ^[0-9]+$ ]]
|
||||
}
|
||||
|
||||
is_positive_integer() {
|
||||
[[ "$1" =~ ^[1-9][0-9]*$ ]]
|
||||
}
|
||||
|
||||
validate_args() {
|
||||
if [[ -z "$RUN_ROOT" || -z "$ENDPOINT" ]]; then
|
||||
echo "ERROR: --run-root and --endpoint are required" >&2
|
||||
exit 1
|
||||
fi
|
||||
if ! [[ "$SECRET_KEY_ENV" =~ ^[A-Za-z_][A-Za-z0-9_]*$ ]]; then
|
||||
echo "ERROR: --secret-key-env must be a valid environment variable name" >&2
|
||||
exit 1
|
||||
fi
|
||||
if ! is_nonnegative_integer "$DURATION_SECS" || ! is_positive_integer "$INTERVAL_SECS"; then
|
||||
echo "ERROR: --duration-secs must be >= 0 and --interval-secs must be > 0" >&2
|
||||
exit 1
|
||||
fi
|
||||
SECRET_KEY="${!SECRET_KEY_ENV:-}"
|
||||
if [[ -z "$LABEL" ]]; then
|
||||
LABEL="capture-$(date -u +%Y%m%dT%H%M%SZ)"
|
||||
fi
|
||||
}
|
||||
|
||||
setup_output() {
|
||||
CAPTURE_ROOT="${RUN_ROOT}/captures/${LABEL}"
|
||||
HEALTH_DIR="${CAPTURE_ROOT}/health"
|
||||
METRICS_DIR="${CAPTURE_ROOT}/metrics"
|
||||
PROM_DIR="${CAPTURE_ROOT}/prom"
|
||||
HOST_DIR="${CAPTURE_ROOT}/host"
|
||||
mkdir -p "$HEALTH_DIR" "$METRICS_DIR" "$PROM_DIR" "$HOST_DIR"
|
||||
}
|
||||
|
||||
endpoint_label() {
|
||||
local endpoint="$1"
|
||||
local label
|
||||
label="${endpoint#*://}"
|
||||
label="${label%%/*}"
|
||||
printf '%s' "$label" | tr -c 'A-Za-z0-9._-' '_'
|
||||
}
|
||||
|
||||
csv_to_lines() {
|
||||
local csv="$1"
|
||||
local raw item
|
||||
IFS=',' read -r -a arr <<< "$csv"
|
||||
for raw in "${arr[@]}"; do
|
||||
item="$(echo "$raw" | awk '{$1=$1;print}')"
|
||||
[[ -z "$item" ]] && continue
|
||||
echo "$item"
|
||||
done
|
||||
}
|
||||
|
||||
resolve_metrics_endpoints() {
|
||||
if [[ -n "$METRICS_ENDPOINTS" ]]; then
|
||||
csv_to_lines "$METRICS_ENDPOINTS"
|
||||
else
|
||||
printf '%s\n' "$ENDPOINT"
|
||||
fi
|
||||
}
|
||||
|
||||
resolve_pid() {
|
||||
if [[ -n "$RUSTFS_PID" ]]; then
|
||||
echo "$RUSTFS_PID"
|
||||
return
|
||||
fi
|
||||
if command -v pidof >/dev/null 2>&1; then
|
||||
pidof rustfs 2>/dev/null | awk '{print $1}' && return 0
|
||||
fi
|
||||
|
||||
if command -v pgrep >/dev/null 2>&1; then
|
||||
pgrep -f 'rustfs server' 2>/dev/null | head -n 1 && return 0
|
||||
fi
|
||||
|
||||
if command -v ps >/dev/null 2>&1; then
|
||||
ps -ef 2>/dev/null | awk '/[r]ustfs server/ {print $2; exit}' && return 0
|
||||
fi
|
||||
|
||||
true
|
||||
}
|
||||
|
||||
health_url() {
|
||||
printf '%s/health\n' "${1%/}"
|
||||
}
|
||||
|
||||
ready_url() {
|
||||
printf '%s/health/ready\n' "${1%/}"
|
||||
}
|
||||
|
||||
admin_metrics_url() {
|
||||
printf '%s/rustfs/admin/v3/metrics?types=1&by-host=true&n=1\n' "${1%/}"
|
||||
}
|
||||
|
||||
write_layout_file() {
|
||||
cat > "${CAPTURE_ROOT}/capture-layout.txt" <<'EOF'
|
||||
capture-meta.txt
|
||||
capture-layout.txt
|
||||
capture-report.md
|
||||
|
||||
health/
|
||||
- health.<endpoint>.<index>.<timestamp>.txt
|
||||
- ready.<endpoint>.<index>.<timestamp>.txt
|
||||
|
||||
metrics/
|
||||
- admin-metrics.<endpoint>.<index>.<timestamp>.ndjson
|
||||
|
||||
prom/
|
||||
- prometheus.<endpoint>.<index>.<timestamp>.prom
|
||||
|
||||
host/
|
||||
- pidstat.txt
|
||||
- iostat.txt
|
||||
- mpstat.txt
|
||||
- proc-status.<index>.<timestamp>.txt
|
||||
- proc-io.<index>.<timestamp>.txt
|
||||
- ps.<index>.<timestamp>.txt
|
||||
EOF
|
||||
}
|
||||
|
||||
write_meta_file() {
|
||||
local git_commit git_branch
|
||||
git_commit="$(git rev-parse HEAD 2>/dev/null || echo "unknown")"
|
||||
git_branch="$(git branch --show-current 2>/dev/null || echo "unknown")"
|
||||
cat > "${CAPTURE_ROOT}/capture-meta.txt" <<EOF
|
||||
created_utc=$(date -u +%Y-%m-%dT%H:%M:%SZ)
|
||||
run_root=${RUN_ROOT}
|
||||
label=${LABEL}
|
||||
endpoint=${ENDPOINT}
|
||||
metrics_endpoints=${METRICS_ENDPOINTS:-$ENDPOINT}
|
||||
prom_metrics_urls=${PROM_METRICS_URLS:-N/A}
|
||||
region=${REGION}
|
||||
duration_secs=${DURATION_SECS}
|
||||
interval_secs=${INTERVAL_SECS}
|
||||
rustfs_pid=${RUSTFS_PID:-auto}
|
||||
resolved_pid=${RESOLVED_PID:-N/A}
|
||||
skip_host_telemetry=${SKIP_HOST_TELEMETRY}
|
||||
dry_run=${DRY_RUN}
|
||||
git_branch=${git_branch}
|
||||
git_commit=${git_commit}
|
||||
access_key_present=$([[ -n "$ACCESS_KEY" ]] && echo true || echo false)
|
||||
secret_key_env=${SECRET_KEY_ENV}
|
||||
EOF
|
||||
}
|
||||
|
||||
capture_health_sample() {
|
||||
local index="$1"
|
||||
local ts="$2"
|
||||
local endpoint="$3"
|
||||
local label file
|
||||
label="$(endpoint_label "$endpoint")"
|
||||
|
||||
file="${HEALTH_DIR}/health.${label}.${index}.${ts}.txt"
|
||||
if [[ "$DRY_RUN" == "true" ]]; then
|
||||
echo "dry run" > "$file"
|
||||
else
|
||||
"$CURL_BIN" -fsS "$(health_url "$endpoint")" > "$file" 2>&1 || true
|
||||
fi
|
||||
|
||||
file="${HEALTH_DIR}/ready.${label}.${index}.${ts}.txt"
|
||||
if [[ "$DRY_RUN" == "true" ]]; then
|
||||
echo "dry run" > "$file"
|
||||
else
|
||||
"$CURL_BIN" -fsS "$(ready_url "$endpoint")" > "$file" 2>&1 || true
|
||||
fi
|
||||
}
|
||||
|
||||
capture_admin_metrics_sample() {
|
||||
local index="$1"
|
||||
local ts="$2"
|
||||
local endpoint="$3"
|
||||
local label file
|
||||
label="$(endpoint_label "$endpoint")"
|
||||
file="${METRICS_DIR}/admin-metrics.${label}.${index}.${ts}.ndjson"
|
||||
|
||||
if [[ "$DRY_RUN" == "true" ]]; then
|
||||
echo "dry run" > "$file"
|
||||
return
|
||||
fi
|
||||
|
||||
if [[ "$AWSCURL_AVAILABLE" != "true" ]]; then
|
||||
echo "awscurl unavailable; signed admin metrics skipped" > "$file"
|
||||
return
|
||||
fi
|
||||
|
||||
if [[ -z "$ACCESS_KEY" || -z "$SECRET_KEY" ]]; then
|
||||
echo "missing access/secret for signed metrics capture" > "$file"
|
||||
return
|
||||
fi
|
||||
|
||||
AWS_ACCESS_KEY_ID="$ACCESS_KEY" \
|
||||
AWS_SECRET_ACCESS_KEY="$SECRET_KEY" \
|
||||
AWS_DEFAULT_REGION="$REGION" \
|
||||
"$AWSCURL_BIN" \
|
||||
--service s3 \
|
||||
--region "$REGION" \
|
||||
--request GET \
|
||||
"$(admin_metrics_url "$endpoint")" \
|
||||
> "$file" 2>&1 || true
|
||||
}
|
||||
|
||||
capture_prometheus_sample() {
|
||||
local index="$1"
|
||||
local ts="$2"
|
||||
local endpoint="$3"
|
||||
local label file
|
||||
label="$(endpoint_label "$endpoint")"
|
||||
file="${PROM_DIR}/prometheus.${label}.${index}.${ts}.prom"
|
||||
|
||||
if [[ "$DRY_RUN" == "true" ]]; then
|
||||
echo "dry run" > "$file"
|
||||
else
|
||||
"$CURL_BIN" -fsS "$endpoint" > "$file" 2>&1 || true
|
||||
fi
|
||||
}
|
||||
|
||||
capture_proc_snapshot() {
|
||||
local index="$1"
|
||||
local ts="$2"
|
||||
local pid="$3"
|
||||
[[ -n "$pid" ]] || return 0
|
||||
|
||||
if [[ -r "/proc/${pid}/status" ]]; then
|
||||
cp "/proc/${pid}/status" "${HOST_DIR}/proc-status.${index}.${ts}.txt" || true
|
||||
fi
|
||||
if [[ -r "/proc/${pid}/io" ]]; then
|
||||
cp "/proc/${pid}/io" "${HOST_DIR}/proc-io.${index}.${ts}.txt" || true
|
||||
fi
|
||||
if command -v ps >/dev/null 2>&1; then
|
||||
ps -p "$pid" -o pid,ppid,stat,pcpu,pmem,rss,vsz,etime,args > "${HOST_DIR}/ps.${index}.${ts}.txt" 2>&1 || true
|
||||
fi
|
||||
}
|
||||
|
||||
start_host_telemetry() {
|
||||
HOST_TELEMETRY_PIDS=()
|
||||
if [[ "$SKIP_HOST_TELEMETRY" == "true" ]]; then
|
||||
return
|
||||
fi
|
||||
|
||||
local count
|
||||
count=$(( DURATION_SECS / INTERVAL_SECS + 1 ))
|
||||
(( count < 1 )) && count=1
|
||||
|
||||
if [[ -n "$RESOLVED_PID" ]] && command -v pidstat >/dev/null 2>&1; then
|
||||
pidstat -durwh -p "$RESOLVED_PID" "$INTERVAL_SECS" "$count" > "${HOST_DIR}/pidstat.txt" 2>&1 &
|
||||
HOST_TELEMETRY_PIDS+=("$!")
|
||||
fi
|
||||
|
||||
if command -v iostat >/dev/null 2>&1; then
|
||||
iostat -xz "$INTERVAL_SECS" "$count" > "${HOST_DIR}/iostat.txt" 2>&1 &
|
||||
HOST_TELEMETRY_PIDS+=("$!")
|
||||
fi
|
||||
|
||||
if command -v mpstat >/dev/null 2>&1; then
|
||||
mpstat "$INTERVAL_SECS" "$count" > "${HOST_DIR}/mpstat.txt" 2>&1 &
|
||||
HOST_TELEMETRY_PIDS+=("$!")
|
||||
fi
|
||||
}
|
||||
|
||||
wait_host_telemetry() {
|
||||
local pid
|
||||
for pid in "${HOST_TELEMETRY_PIDS[@]:-}"; do
|
||||
wait "$pid" || true
|
||||
done
|
||||
}
|
||||
|
||||
capture_one_sample() {
|
||||
local index="$1"
|
||||
local ts endpoint prom_url
|
||||
ts="$(date -u +%Y%m%dT%H%M%SZ)"
|
||||
|
||||
while IFS= read -r endpoint; do
|
||||
capture_health_sample "$index" "$ts" "$endpoint"
|
||||
capture_admin_metrics_sample "$index" "$ts" "$endpoint"
|
||||
done < <(resolve_metrics_endpoints)
|
||||
|
||||
if [[ -n "$PROM_METRICS_URLS" ]]; then
|
||||
while IFS= read -r prom_url; do
|
||||
capture_prometheus_sample "$index" "$ts" "$prom_url"
|
||||
done < <(csv_to_lines "$PROM_METRICS_URLS")
|
||||
fi
|
||||
|
||||
if [[ "$DRY_RUN" != "true" ]]; then
|
||||
capture_proc_snapshot "$index" "$ts" "$RESOLVED_PID"
|
||||
fi
|
||||
}
|
||||
|
||||
capture_series() {
|
||||
local total_samples index
|
||||
total_samples=$(( DURATION_SECS / INTERVAL_SECS + 1 ))
|
||||
(( total_samples < 1 )) && total_samples=1
|
||||
|
||||
for ((index = 1; index <= total_samples; index++)); do
|
||||
capture_one_sample "$index"
|
||||
if (( index < total_samples )); then
|
||||
sleep "$INTERVAL_SECS"
|
||||
fi
|
||||
done
|
||||
}
|
||||
|
||||
count_artifacts() {
|
||||
local dir="$1"
|
||||
local pattern="$2"
|
||||
find "$dir" -type f -name "$pattern" 2>/dev/null | wc -l | tr -d ' '
|
||||
}
|
||||
|
||||
write_report() {
|
||||
local health_count metrics_count prom_count proc_count
|
||||
health_count="$(count_artifacts "$HEALTH_DIR" 'health.*.txt')"
|
||||
metrics_count="$(count_artifacts "$METRICS_DIR" 'admin-metrics.*.ndjson')"
|
||||
prom_count="$(count_artifacts "$PROM_DIR" 'prometheus.*.prom')"
|
||||
proc_count="$(count_artifacts "$HOST_DIR" 'proc-status.*.txt')"
|
||||
|
||||
cat > "${CAPTURE_ROOT}/capture-report.md" <<EOF
|
||||
## Large PUT Stage-Breakdown Capture Report
|
||||
|
||||
Run root: ${RUN_ROOT}
|
||||
Label: ${LABEL}
|
||||
Endpoint: ${ENDPOINT}
|
||||
Duration seconds: ${DURATION_SECS}
|
||||
Interval seconds: ${INTERVAL_SECS}
|
||||
Resolved pid: ${RESOLVED_PID:-N/A}
|
||||
|
||||
## Artifact Summary
|
||||
|
||||
- Health snapshots: ${health_count}
|
||||
- Signed admin metrics snapshots: ${metrics_count}
|
||||
- Plain Prometheus snapshots: ${prom_count}
|
||||
- Process snapshots: ${proc_count}
|
||||
- Host telemetry present: $([[ -s "${HOST_DIR}/pidstat.txt" || -s "${HOST_DIR}/iostat.txt" || -s "${HOST_DIR}/mpstat.txt" ]] && echo yes || echo no)
|
||||
|
||||
## Review Checklist
|
||||
|
||||
- Align capture timestamps with benchmark windows in \`runs/cXX/\`.
|
||||
- Check whether health or readiness degraded during the benchmark window.
|
||||
- Compare admin metrics snapshots with the Prometheus / Grafana queries used for stage interpretation.
|
||||
- Attach pidstat/iostat/mpstat when evaluating CPU, RSS, and disk pressure.
|
||||
- Use the capture directory together with \`aggregate_median_summary.csv\` when filling the stage-breakdown report template.
|
||||
EOF
|
||||
}
|
||||
|
||||
main() {
|
||||
parse_args "$@"
|
||||
validate_args
|
||||
require_cmd bash
|
||||
require_cmd "$CURL_BIN"
|
||||
require_cmd awk
|
||||
if [[ "$DRY_RUN" != "true" ]]; then
|
||||
require_cmd date
|
||||
fi
|
||||
if [[ "$DRY_RUN" != "true" && -n "$ACCESS_KEY" && -n "$SECRET_KEY" ]]; then
|
||||
if command -v "$AWSCURL_BIN" >/dev/null 2>&1; then
|
||||
AWSCURL_AVAILABLE=true
|
||||
else
|
||||
AWSCURL_AVAILABLE=false
|
||||
echo "WARN: ${AWSCURL_BIN} not found; signed admin metrics snapshots will be skipped" >&2
|
||||
fi
|
||||
fi
|
||||
|
||||
setup_output
|
||||
RESOLVED_PID="$(resolve_pid)"
|
||||
write_layout_file
|
||||
write_meta_file
|
||||
start_host_telemetry
|
||||
capture_series
|
||||
wait_host_telemetry
|
||||
write_report
|
||||
|
||||
echo "Capture artifacts written to ${CAPTURE_ROOT}"
|
||||
}
|
||||
|
||||
main "$@"
|
||||
+249
@@ -0,0 +1,249 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
ENDPOINT=""
|
||||
ACCESS_KEY=""
|
||||
SECRET_KEY=""
|
||||
BUCKET=""
|
||||
PREFIX="bench/issue713"
|
||||
OBJECTS="1GiB=plain-1g.bin,2GiB=plain-2g.bin"
|
||||
REGION="us-east-1"
|
||||
MC_BIN="${MC_BIN:-mc}"
|
||||
FORCE=false
|
||||
INSECURE=false
|
||||
DRY_RUN=false
|
||||
|
||||
usage() {
|
||||
cat <<'USAGE'
|
||||
Usage:
|
||||
scripts/prepare_gt1g_get_test_objects.sh \
|
||||
--endpoint <url> --access-key <ak> --secret-key <sk> --bucket <bucket> [options]
|
||||
|
||||
Required:
|
||||
--endpoint <url>
|
||||
--access-key <ak>
|
||||
--secret-key <sk>
|
||||
--bucket <bucket>
|
||||
|
||||
Options:
|
||||
--prefix <path> Default: bench/issue713
|
||||
--objects <csv> Default: 1GiB=plain-1g.bin,2GiB=plain-2g.bin
|
||||
Format: size=object-name,size=object-name
|
||||
--region <name> Default: us-east-1
|
||||
--mc-bin <path> Default: mc
|
||||
--force Re-upload even if object already exists
|
||||
--insecure Allow insecure TLS
|
||||
--dry-run
|
||||
-h, --help
|
||||
|
||||
Examples:
|
||||
scripts/prepare_gt1g_get_test_objects.sh \
|
||||
--endpoint http://127.0.0.1:9000 \
|
||||
--access-key rustfsadmin \
|
||||
--secret-key rustfsadmin \
|
||||
--bucket rustfs-bench \
|
||||
--prefix bench/issue713/plain
|
||||
USAGE
|
||||
}
|
||||
|
||||
arg_value() {
|
||||
local flag="$1"
|
||||
local value="${2:-}"
|
||||
if [[ -z "$value" || "$value" == --* ]]; then
|
||||
echo "ERROR: missing value for $flag" >&2
|
||||
exit 1
|
||||
fi
|
||||
printf '%s\n' "$value"
|
||||
}
|
||||
|
||||
require_cmd() {
|
||||
if ! command -v "$1" >/dev/null 2>&1; then
|
||||
echo "ERROR: command not found: $1" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
run_rust_helper_fallback() {
|
||||
local objects_arg=""
|
||||
local helper_out_dir="$OUT_DIR"
|
||||
local raw_spec
|
||||
IFS=',' read -r -a object_specs <<< "$OBJECTS"
|
||||
for raw_spec in "${object_specs[@]}"; do
|
||||
local spec size object_name
|
||||
spec="$(echo "$raw_spec" | awk '{$1=$1;print}')"
|
||||
[[ -z "$spec" ]] && continue
|
||||
size="${spec%%=*}"
|
||||
object_name="${spec#*=}"
|
||||
objects_arg+="${size}=${PREFIX%/}/${object_name},"
|
||||
done
|
||||
objects_arg="${objects_arg%,}"
|
||||
|
||||
if [[ -n "$helper_out_dir" && "$helper_out_dir" != /* ]]; then
|
||||
helper_out_dir="$PWD/$helper_out_dir"
|
||||
fi
|
||||
|
||||
echo "mc not found; falling back to rustfs/tests/gt1g_get_benchmark_tool.rs"
|
||||
GT1G_GET_ACTION=prepare \
|
||||
GT1G_GET_ENDPOINT="$ENDPOINT" \
|
||||
GT1G_GET_ACCESS_KEY="$ACCESS_KEY" \
|
||||
GT1G_GET_SECRET_KEY="$SECRET_KEY" \
|
||||
GT1G_GET_BUCKET="$BUCKET" \
|
||||
GT1G_GET_REGION="$REGION" \
|
||||
GT1G_GET_OBJECTS="$objects_arg" \
|
||||
GT1G_GET_OUT_DIR="${helper_out_dir:-$PWD/target/bench/issue713-prepare-helper}" \
|
||||
GT1G_GET_FORCE="$([[ "$FORCE" == "true" ]] && echo true || echo false)" \
|
||||
cargo test -p rustfs --test gt1g_get_benchmark_tool gt1g_get_benchmark_tool -- --ignored --nocapture
|
||||
}
|
||||
|
||||
parse_args() {
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--endpoint) ENDPOINT="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--access-key) ACCESS_KEY="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--secret-key) SECRET_KEY="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--bucket) BUCKET="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--prefix) PREFIX="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--objects) OBJECTS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--region) REGION="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--mc-bin) MC_BIN="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--force) FORCE=true; shift ;;
|
||||
--insecure) INSECURE=true; shift ;;
|
||||
--dry-run) DRY_RUN=true; shift ;;
|
||||
-h|--help) usage; exit 0 ;;
|
||||
*)
|
||||
echo "ERROR: unknown arg: $1" >&2
|
||||
usage
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
}
|
||||
|
||||
validate_args() {
|
||||
if [[ -z "$ENDPOINT" || -z "$ACCESS_KEY" || -z "$SECRET_KEY" || -z "$BUCKET" ]]; then
|
||||
echo "ERROR: --endpoint, --access-key, --secret-key, and --bucket are required" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
size_to_bytes() {
|
||||
local size="$1"
|
||||
case "$size" in
|
||||
*GiB)
|
||||
local n="${size%GiB}"
|
||||
echo $((n * 1024 * 1024 * 1024))
|
||||
;;
|
||||
*MiB)
|
||||
local n="${size%MiB}"
|
||||
echo $((n * 1024 * 1024))
|
||||
;;
|
||||
*KiB)
|
||||
local n="${size%KiB}"
|
||||
echo $((n * 1024))
|
||||
;;
|
||||
*B)
|
||||
echo "${size%B}"
|
||||
;;
|
||||
*)
|
||||
echo "ERROR"
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
create_sparse_file() {
|
||||
local file_path="$1"
|
||||
local bytes="$2"
|
||||
if command -v mkfile >/dev/null 2>&1; then
|
||||
mkfile -n "$bytes" "$file_path"
|
||||
else
|
||||
truncate -s "$bytes" "$file_path"
|
||||
fi
|
||||
}
|
||||
|
||||
object_exists() {
|
||||
local alias_path="$1"
|
||||
local -a cmd=("$MC_BIN" stat "$alias_path")
|
||||
if [[ "$INSECURE" == "true" ]]; then
|
||||
cmd=("$MC_BIN" --insecure stat "$alias_path")
|
||||
fi
|
||||
|
||||
"${cmd[@]}" >/dev/null 2>&1
|
||||
}
|
||||
|
||||
main() {
|
||||
parse_args "$@"
|
||||
validate_args
|
||||
|
||||
if ! command -v "$MC_BIN" >/dev/null 2>&1; then
|
||||
run_rust_helper_fallback
|
||||
exit 0
|
||||
fi
|
||||
|
||||
local tmp_root
|
||||
tmp_root="$(mktemp -d "${TMPDIR:-/tmp}/issue713-gt1g-get.XXXXXX")"
|
||||
trap 'rm -rf "$tmp_root"' EXIT
|
||||
|
||||
local mc_config_dir="$tmp_root/mc"
|
||||
mkdir -p "$mc_config_dir"
|
||||
|
||||
local -a mc_alias_cmd=("$MC_BIN" --config-dir "$mc_config_dir")
|
||||
if [[ "$INSECURE" == "true" ]]; then
|
||||
mc_alias_cmd+=("--insecure")
|
||||
fi
|
||||
|
||||
if [[ "$DRY_RUN" == "true" ]]; then
|
||||
echo "[DRY-RUN] ${mc_alias_cmd[*]} alias set issue713 ${ENDPOINT} REDACTED REDACTED"
|
||||
else
|
||||
"${mc_alias_cmd[@]}" alias set issue713 "$ENDPOINT" "$ACCESS_KEY" "$SECRET_KEY" >/dev/null
|
||||
fi
|
||||
|
||||
IFS=',' read -r -a object_specs <<< "$OBJECTS"
|
||||
for raw_spec in "${object_specs[@]}"; do
|
||||
local spec size object_name bytes local_file object_key alias_path
|
||||
spec="$(echo "$raw_spec" | awk '{$1=$1;print}')"
|
||||
[[ -z "$spec" ]] && continue
|
||||
|
||||
size="${spec%%=*}"
|
||||
object_name="${spec#*=}"
|
||||
if [[ -z "$size" || -z "$object_name" || "$size" == "$object_name" ]]; then
|
||||
echo "ERROR: invalid object spec: $spec" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
bytes="$(size_to_bytes "$size")"
|
||||
if [[ "$bytes" == "ERROR" ]]; then
|
||||
echo "ERROR: unsupported size label: $size" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
object_key="${PREFIX%/}/${object_name}"
|
||||
alias_path="issue713/${BUCKET}/${object_key}"
|
||||
|
||||
if [[ "$FORCE" != "true" && "$DRY_RUN" != "true" ]] && object_exists "$alias_path"; then
|
||||
echo "skip existing: s3://${BUCKET}/${object_key}"
|
||||
continue
|
||||
fi
|
||||
|
||||
local_file="${tmp_root}/${object_name}"
|
||||
create_sparse_file "$local_file" "$bytes"
|
||||
|
||||
if [[ "$DRY_RUN" == "true" ]]; then
|
||||
echo "[DRY-RUN] create sparse file ${local_file} (${bytes} bytes)"
|
||||
echo "[DRY-RUN] ${mc_alias_cmd[*]} cp ${local_file} ${alias_path}"
|
||||
else
|
||||
echo "uploading: s3://${BUCKET}/${object_key} (${size})"
|
||||
"${mc_alias_cmd[@]}" cp "$local_file" "$alias_path" >/dev/null
|
||||
fi
|
||||
done
|
||||
|
||||
echo "prepared objects:"
|
||||
for raw_spec in "${object_specs[@]}"; do
|
||||
local spec object_name
|
||||
spec="$(echo "$raw_spec" | awk '{$1=$1;print}')"
|
||||
[[ -z "$spec" ]] && continue
|
||||
object_name="${spec#*=}"
|
||||
echo " s3://${BUCKET}/${PREFIX%/}/${object_name}"
|
||||
done
|
||||
}
|
||||
|
||||
main "$@"
|
||||
+122
@@ -0,0 +1,122 @@
|
||||
#!/usr/bin/env bash
|
||||
# Quick Validation Script for GET Optimization
|
||||
# Usage: ./scripts/quick-validate-get-optimization.sh [TARGET_HOST]
|
||||
#
|
||||
# Runs quick functional tests to validate GET optimization changes:
|
||||
# 1. Basic GET correctness
|
||||
# 2. Concurrent GET stability
|
||||
# 3. Early-stop behavior
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
TARGET_HOST="${1:-localhost:9000}"
|
||||
MC_ALIAS="${MC_ALIAS:-rustfs}"
|
||||
TEST_BUCKET="quick-validate-$(date +%s)"
|
||||
|
||||
echo "=========================================="
|
||||
echo "Quick GET Optimization Validation"
|
||||
echo "=========================================="
|
||||
echo "Target: $TARGET_HOST"
|
||||
echo ""
|
||||
|
||||
# Create test bucket
|
||||
mc mb "${MC_ALIAS}/${TEST_BUCKET}" 2>/dev/null || true
|
||||
|
||||
# ============================================================================
|
||||
# Test 1: Basic GET Correctness
|
||||
# ============================================================================
|
||||
echo "[1/3] Basic GET Correctness"
|
||||
|
||||
# Upload test file
|
||||
dd if=/dev/urandom of=/tmp/quick-test.bin bs=1048576 count=1 2>/dev/null
|
||||
original_hash=$(md5 -q /tmp/quick-test.bin 2>/dev/null || md5sum /tmp/quick-test.bin | awk '{print $1}')
|
||||
|
||||
mc cp /tmp/quick-test.bin "${MC_ALIAS}/${TEST_BUCKET}/test.bin" >/dev/null 2>&1
|
||||
|
||||
# Download and verify
|
||||
mc cp "${MC_ALIAS}/${TEST_BUCKET}/test.bin" /tmp/quick-test-download.bin >/dev/null 2>&1
|
||||
download_hash=$(md5 -q /tmp/quick-test-download.bin 2>/dev/null || md5sum /tmp/quick-test-download.bin | awk '{print $1}')
|
||||
|
||||
if [ "$original_hash" = "$download_hash" ]; then
|
||||
echo " PASS: Data integrity verified"
|
||||
else
|
||||
echo " FAIL: Data mismatch (original=$original_hash, download=$download_hash)"
|
||||
fi
|
||||
|
||||
rm -f /tmp/quick-test.bin /tmp/quick-test-download.bin
|
||||
|
||||
# ============================================================================
|
||||
# Test 2: Concurrent GET Stability
|
||||
# ============================================================================
|
||||
echo "[2/3] Concurrent GET Stability"
|
||||
|
||||
# Upload multiple test files
|
||||
for i in $(seq 1 10); do
|
||||
dd if=/dev/urandom of="/tmp/quick-test-${i}.bin" bs=1048576 count=1 2>/dev/null
|
||||
mc cp "/tmp/quick-test-${i}.bin" "${MC_ALIAS}/${TEST_BUCKET}/test-${i}.bin" >/dev/null 2>&1
|
||||
rm -f "/tmp/quick-test-${i}.bin"
|
||||
done
|
||||
|
||||
# Concurrent download
|
||||
passed=0
|
||||
failed=0
|
||||
|
||||
for i in $(seq 1 10); do
|
||||
if mc cp "${MC_ALIAS}/${TEST_BUCKET}/test-${i}.bin" "/tmp/quick-download-${i}.bin" >/dev/null 2>&1; then
|
||||
size=$(stat -f%z "/tmp/quick-download-${i}.bin" 2>/dev/null || stat -c%s "/tmp/quick-download-${i}.bin" 2>/dev/null)
|
||||
if [ "$size" = "1048576" ]; then
|
||||
passed=$((passed + 1))
|
||||
else
|
||||
failed=$((failed + 1))
|
||||
fi
|
||||
else
|
||||
failed=$((failed + 1))
|
||||
fi
|
||||
rm -f "/tmp/quick-download-${i}.bin"
|
||||
done
|
||||
|
||||
echo " Result: $passed passed, $failed failed"
|
||||
|
||||
# ============================================================================
|
||||
# Test 3: Early-Stop Behavior
|
||||
# ============================================================================
|
||||
echo "[3/3] Early-Stop Behavior"
|
||||
|
||||
# Upload a larger test file
|
||||
dd if=/dev/urandom of=/tmp/quick-test-large.bin bs=1048576 count=10 2>/dev/null
|
||||
mc cp /tmp/quick-test-large.bin "${MC_ALIAS}/${TEST_BUCKET}/test-large.bin" >/dev/null 2>&1
|
||||
|
||||
# Multiple reads to trigger early-stop
|
||||
passed=0
|
||||
failed=0
|
||||
|
||||
for i in $(seq 1 20); do
|
||||
if mc cp "${MC_ALIAS}/${TEST_BUCKET}/test-large.bin" "/tmp/quick-large-${i}.bin" >/dev/null 2>&1; then
|
||||
size=$(stat -f%z "/tmp/quick-large-${i}.bin" 2>/dev/null || stat -c%s "/tmp/quick-large-${i}.bin" 2>/dev/null)
|
||||
if [ "$size" = "10485760" ]; then
|
||||
passed=$((passed + 1))
|
||||
else
|
||||
failed=$((failed + 1))
|
||||
fi
|
||||
else
|
||||
failed=$((failed + 1))
|
||||
fi
|
||||
rm -f "/tmp/quick-large-${i}.bin"
|
||||
done
|
||||
|
||||
echo " Result: $passed passed, $failed failed"
|
||||
|
||||
rm -f /tmp/quick-test-large.bin
|
||||
|
||||
# ============================================================================
|
||||
# Cleanup
|
||||
# ============================================================================
|
||||
echo ""
|
||||
echo "Cleaning up..."
|
||||
mc rm --recursive --force "${MC_ALIAS}/${TEST_BUCKET}" >/dev/null 2>&1 || true
|
||||
mc rb "${MC_ALIAS}/${TEST_BUCKET}" >/dev/null 2>&1 || true
|
||||
|
||||
echo ""
|
||||
echo "=========================================="
|
||||
echo "Quick Validation Complete"
|
||||
echo "=========================================="
|
||||
+514
@@ -0,0 +1,514 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
|
||||
ENHANCED_BENCH="${PROJECT_ROOT}/scripts/run_object_batch_bench_enhanced.sh"
|
||||
|
||||
ADDRESS="127.0.0.1:19031"
|
||||
ACCESS_KEY="rustfsadmin"
|
||||
SECRET_KEY="rustfsadmin"
|
||||
REGION="us-east-1"
|
||||
SIZE="10MiB"
|
||||
CONCURRENCY=32
|
||||
DURATION="10s"
|
||||
ROUNDS=3
|
||||
ROUND_COOLDOWN_SECS=10
|
||||
RETRY_PER_ROUND=1
|
||||
SEED_DURATION="5s"
|
||||
SEED_CONCURRENCY=8
|
||||
HEALTH_TIMEOUT_SECS=60
|
||||
OUT_DIR=""
|
||||
DATA_ROOT=""
|
||||
BUCKET=""
|
||||
RUSTFS_BIN="${PROJECT_ROOT}/target/release/rustfs"
|
||||
WARP_BIN="warp"
|
||||
BASELINE_CSV=""
|
||||
DRY_RUN=false
|
||||
SKIP_BUILD=false
|
||||
|
||||
SERVER_PID=""
|
||||
|
||||
usage() {
|
||||
cat <<'USAGE'
|
||||
Usage:
|
||||
scripts/run_get_metrics_gate_smoke.sh [options]
|
||||
|
||||
Purpose:
|
||||
Start one local single-node multi-disk RustFS server with observability
|
||||
export disabled, seed 10MiB objects, and run a focused warp GET benchmark.
|
||||
|
||||
Options:
|
||||
--address <host:port> RustFS listen address (default: 127.0.0.1:19031)
|
||||
--access-key <value> Access key (default: rustfsadmin)
|
||||
--secret-key <value> Secret key (default: rustfsadmin)
|
||||
--region <value> Region (default: us-east-1)
|
||||
--bucket <name> Benchmark bucket (default: auto-generated)
|
||||
--size <label> Object size label (default: 10MiB)
|
||||
--concurrency <n> warp GET concurrency (default: 32)
|
||||
--duration <duration> warp GET duration per round (default: 10s)
|
||||
--rounds <n> Benchmark rounds (default: 3)
|
||||
--round-cooldown-secs <n> Cooldown after each round (default: 10)
|
||||
--retry-per-round <n> Failed-round retries (default: 1)
|
||||
--seed-duration <duration> warp PUT duration for object seeding (default: 5s)
|
||||
--seed-concurrency <n> warp PUT concurrency for object seeding (default: 8)
|
||||
--health-timeout-secs <n> Health wait timeout (default: 60)
|
||||
--out-dir <path> Output directory (default: target/bench/get-metrics-gate-<timestamp>)
|
||||
--data-root <path> Data root for d1..d4 (default: /private/tmp/get-metrics-gate-<timestamp>)
|
||||
--rustfs-bin <path> RustFS binary (default: target/release/rustfs)
|
||||
--warp-bin <path> warp binary (default: warp)
|
||||
--baseline-csv <path> Optional baseline median_summary.csv for delta output
|
||||
--skip-build Skip cargo build --release -p rustfs --bin rustfs
|
||||
--dry-run Print commands only
|
||||
-h, --help Show help
|
||||
USAGE
|
||||
}
|
||||
|
||||
die() {
|
||||
echo "ERROR: $*" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
require_cmd() {
|
||||
if ! command -v "$1" >/dev/null 2>&1; then
|
||||
die "command not found: $1"
|
||||
fi
|
||||
}
|
||||
|
||||
validate_positive_int() {
|
||||
local value="$1"
|
||||
local name="$2"
|
||||
if ! [[ "$value" =~ ^[0-9]+$ ]] || [[ "$value" -le 0 ]]; then
|
||||
die "$name must be a positive integer, got: $value"
|
||||
fi
|
||||
}
|
||||
|
||||
validate_non_negative_int() {
|
||||
local value="$1"
|
||||
local name="$2"
|
||||
if ! [[ "$value" =~ ^[0-9]+$ ]]; then
|
||||
die "$name must be a non-negative integer, got: $value"
|
||||
fi
|
||||
}
|
||||
|
||||
endpoint_url() {
|
||||
echo "http://${ADDRESS}"
|
||||
}
|
||||
|
||||
normalize_warp_host() {
|
||||
local raw="$1"
|
||||
raw="${raw#http://}"
|
||||
raw="${raw#https://}"
|
||||
raw="${raw%%/*}"
|
||||
raw="${raw%%\?*}"
|
||||
raw="${raw%%\#*}"
|
||||
echo "$raw"
|
||||
}
|
||||
|
||||
parse_args() {
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--address) ADDRESS="$2"; shift 2 ;;
|
||||
--access-key) ACCESS_KEY="$2"; shift 2 ;;
|
||||
--secret-key) SECRET_KEY="$2"; shift 2 ;;
|
||||
--region) REGION="$2"; shift 2 ;;
|
||||
--bucket) BUCKET="$2"; shift 2 ;;
|
||||
--size) SIZE="$2"; shift 2 ;;
|
||||
--concurrency) CONCURRENCY="$2"; shift 2 ;;
|
||||
--duration) DURATION="$2"; shift 2 ;;
|
||||
--rounds) ROUNDS="$2"; shift 2 ;;
|
||||
--round-cooldown-secs) ROUND_COOLDOWN_SECS="$2"; shift 2 ;;
|
||||
--retry-per-round) RETRY_PER_ROUND="$2"; shift 2 ;;
|
||||
--seed-duration) SEED_DURATION="$2"; shift 2 ;;
|
||||
--seed-concurrency) SEED_CONCURRENCY="$2"; shift 2 ;;
|
||||
--health-timeout-secs) HEALTH_TIMEOUT_SECS="$2"; shift 2 ;;
|
||||
--out-dir) OUT_DIR="$2"; shift 2 ;;
|
||||
--data-root) DATA_ROOT="$2"; shift 2 ;;
|
||||
--rustfs-bin) RUSTFS_BIN="$2"; shift 2 ;;
|
||||
--warp-bin) WARP_BIN="$2"; shift 2 ;;
|
||||
--baseline-csv) BASELINE_CSV="$2"; shift 2 ;;
|
||||
--skip-build) SKIP_BUILD=true; shift ;;
|
||||
--dry-run) DRY_RUN=true; shift ;;
|
||||
-h|--help) usage; exit 0 ;;
|
||||
*)
|
||||
usage >&2
|
||||
die "unknown arg: $1"
|
||||
;;
|
||||
esac
|
||||
done
|
||||
}
|
||||
|
||||
validate_args() {
|
||||
[[ -n "$ADDRESS" ]] || die "--address must not be empty"
|
||||
[[ -n "$ACCESS_KEY" ]] || die "--access-key must not be empty"
|
||||
[[ -n "$SECRET_KEY" ]] || die "--secret-key must not be empty"
|
||||
[[ -n "$REGION" ]] || die "--region must not be empty"
|
||||
[[ -n "$SIZE" ]] || die "--size must not be empty"
|
||||
validate_positive_int "$CONCURRENCY" "--concurrency"
|
||||
validate_positive_int "$ROUNDS" "--rounds"
|
||||
validate_positive_int "$RETRY_PER_ROUND" "--retry-per-round"
|
||||
validate_positive_int "$SEED_CONCURRENCY" "--seed-concurrency"
|
||||
validate_positive_int "$HEALTH_TIMEOUT_SECS" "--health-timeout-secs"
|
||||
validate_non_negative_int "$ROUND_COOLDOWN_SECS" "--round-cooldown-secs"
|
||||
|
||||
[[ -x "$ENHANCED_BENCH" ]] || die "benchmark script is not executable: $ENHANCED_BENCH"
|
||||
require_cmd curl
|
||||
require_cmd git
|
||||
require_cmd "$WARP_BIN"
|
||||
if [[ "$DRY_RUN" != "true" && "$SKIP_BUILD" != "true" ]]; then
|
||||
require_cmd cargo
|
||||
fi
|
||||
if [[ -n "$BASELINE_CSV" && ! -f "$BASELINE_CSV" ]]; then
|
||||
die "--baseline-csv does not exist: $BASELINE_CSV"
|
||||
fi
|
||||
}
|
||||
|
||||
setup_paths() {
|
||||
local timestamp
|
||||
timestamp="$(date +%Y%m%d-%H%M%S)"
|
||||
if [[ -z "$OUT_DIR" ]]; then
|
||||
OUT_DIR="${PROJECT_ROOT}/target/bench/get-metrics-gate-${timestamp}"
|
||||
fi
|
||||
if [[ -z "$DATA_ROOT" ]]; then
|
||||
DATA_ROOT="/private/tmp/get-metrics-gate-${timestamp}"
|
||||
fi
|
||||
if [[ -z "$BUCKET" ]]; then
|
||||
BUCKET="rustfs-get-metrics-${timestamp}"
|
||||
fi
|
||||
|
||||
mkdir -p "$OUT_DIR" "$DATA_ROOT"/d1 "$DATA_ROOT"/d2 "$DATA_ROOT"/d3 "$DATA_ROOT"/d4
|
||||
}
|
||||
|
||||
build_rustfs_if_needed() {
|
||||
if [[ "$DRY_RUN" == "true" || "$SKIP_BUILD" == "true" ]]; then
|
||||
return
|
||||
fi
|
||||
cargo build --release -p rustfs --bin rustfs
|
||||
}
|
||||
|
||||
write_manifest() {
|
||||
local git_head
|
||||
git_head="$(git -C "$PROJECT_ROOT" rev-parse HEAD)"
|
||||
|
||||
cat >"${OUT_DIR}/manifest.env" <<EOF
|
||||
git_head=${git_head}
|
||||
endpoint=$(endpoint_url)
|
||||
address=${ADDRESS}
|
||||
bucket=${BUCKET}
|
||||
region=${REGION}
|
||||
size=${SIZE}
|
||||
concurrency=${CONCURRENCY}
|
||||
duration=${DURATION}
|
||||
rounds=${ROUNDS}
|
||||
round_cooldown_secs=${ROUND_COOLDOWN_SECS}
|
||||
retry_per_round=${RETRY_PER_ROUND}
|
||||
seed_duration=${SEED_DURATION}
|
||||
seed_concurrency=${SEED_CONCURRENCY}
|
||||
rustfs_bin=${RUSTFS_BIN}
|
||||
warp_bin=${WARP_BIN}
|
||||
data_root=${DATA_ROOT}
|
||||
RUST_LOG=off
|
||||
RUSTFS_OBS_LOGGER_LEVEL=off
|
||||
RUSTFS_OBS_TRACES_EXPORT_ENABLED=false
|
||||
RUSTFS_OBS_METRICS_EXPORT_ENABLED=false
|
||||
RUSTFS_OBS_LOGS_EXPORT_ENABLED=false
|
||||
RUSTFS_OBS_PROFILING_EXPORT_ENABLED=false
|
||||
RUSTFS_OBS_USE_STDOUT=false
|
||||
RUSTFS_OBS_LOG_STDOUT_ENABLED=false
|
||||
RUSTFS_SCANNER_ENABLED=false
|
||||
RUSTFS_CONSOLE_ENABLE=false
|
||||
RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true
|
||||
EOF
|
||||
}
|
||||
|
||||
stop_server() {
|
||||
if [[ -n "$SERVER_PID" ]]; then
|
||||
if kill -0 "$SERVER_PID" >/dev/null 2>&1; then
|
||||
kill "$SERVER_PID" >/dev/null 2>&1 || true
|
||||
wait "$SERVER_PID" >/dev/null 2>&1 || true
|
||||
fi
|
||||
SERVER_PID=""
|
||||
fi
|
||||
}
|
||||
|
||||
wait_for_health() {
|
||||
local health_url
|
||||
health_url="$(endpoint_url)/health"
|
||||
|
||||
for ((attempt = 1; attempt <= HEALTH_TIMEOUT_SECS; attempt++)); do
|
||||
if curl -fsS --noproxy '*' --connect-timeout 2 --max-time 3 "$health_url" >/dev/null 2>&1; then
|
||||
return
|
||||
fi
|
||||
if [[ -n "$SERVER_PID" ]] && ! kill -0 "$SERVER_PID" >/dev/null 2>&1; then
|
||||
tail -n 80 "${OUT_DIR}/rustfs.log" >&2 || true
|
||||
die "RustFS exited before health check passed"
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
|
||||
tail -n 80 "${OUT_DIR}/rustfs.log" >&2 || true
|
||||
die "RustFS health check timed out"
|
||||
}
|
||||
|
||||
start_server() {
|
||||
if [[ "$DRY_RUN" == "true" ]]; then
|
||||
echo "[DRY-RUN] start RustFS at $(endpoint_url)"
|
||||
return
|
||||
fi
|
||||
|
||||
[[ -x "$RUSTFS_BIN" ]] || die "RustFS binary is not executable: $RUSTFS_BIN"
|
||||
|
||||
(
|
||||
export RUST_LOG=off
|
||||
export RUSTFS_OBS_LOGGER_LEVEL=off
|
||||
export RUSTFS_OBS_TRACES_EXPORT_ENABLED=false
|
||||
export RUSTFS_OBS_METRICS_EXPORT_ENABLED=false
|
||||
export RUSTFS_OBS_LOGS_EXPORT_ENABLED=false
|
||||
export RUSTFS_OBS_PROFILING_EXPORT_ENABLED=false
|
||||
export RUSTFS_OBS_USE_STDOUT=false
|
||||
export RUSTFS_OBS_LOG_STDOUT_ENABLED=false
|
||||
export RUSTFS_SCANNER_ENABLED=false
|
||||
export RUSTFS_CONSOLE_ENABLE=false
|
||||
export RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true
|
||||
export RUSTFS_ADDRESS="$ADDRESS"
|
||||
export RUSTFS_ACCESS_KEY="$ACCESS_KEY"
|
||||
export RUSTFS_SECRET_KEY="$SECRET_KEY"
|
||||
export RUSTFS_RPC_SECRET="rustfs-get-metrics-gate-rpc-secret"
|
||||
export RUSTFS_REGION="$REGION"
|
||||
exec "$RUSTFS_BIN" server \
|
||||
"${DATA_ROOT}/d1" \
|
||||
"${DATA_ROOT}/d2" \
|
||||
"${DATA_ROOT}/d3" \
|
||||
"${DATA_ROOT}/d4"
|
||||
) >"${OUT_DIR}/rustfs.log" 2>&1 &
|
||||
|
||||
SERVER_PID="$!"
|
||||
wait_for_health
|
||||
}
|
||||
|
||||
run_bench() {
|
||||
local cmd=(
|
||||
"$ENHANCED_BENCH"
|
||||
--tool warp
|
||||
--endpoint "$(endpoint_url)"
|
||||
--access-key "$ACCESS_KEY"
|
||||
--secret-key "$SECRET_KEY"
|
||||
--bucket "$BUCKET"
|
||||
--region "$REGION"
|
||||
--warp-bin "$WARP_BIN"
|
||||
--warp-mode get
|
||||
--sizes "$SIZE"
|
||||
--concurrency "$CONCURRENCY"
|
||||
--duration "$DURATION"
|
||||
--rounds "$ROUNDS"
|
||||
--retry-per-round "$RETRY_PER_ROUND"
|
||||
--round-cooldown-secs "$ROUND_COOLDOWN_SECS"
|
||||
--out-dir "${OUT_DIR}/warp"
|
||||
)
|
||||
|
||||
if [[ -n "$BASELINE_CSV" ]]; then
|
||||
cmd+=(--baseline-csv "$BASELINE_CSV")
|
||||
fi
|
||||
if [[ "$DRY_RUN" == "true" ]]; then
|
||||
cmd+=(--dry-run)
|
||||
fi
|
||||
|
||||
"${cmd[@]}"
|
||||
}
|
||||
|
||||
to_bps() {
|
||||
local human="$1"
|
||||
local number unit factor
|
||||
if [[ "$human" == "N/A" || -z "$human" ]]; then
|
||||
echo "N/A"
|
||||
return
|
||||
fi
|
||||
|
||||
number="$(echo "$human" | awk '{print $1}')"
|
||||
unit="$(echo "$human" | awk '{print $2}')"
|
||||
case "$unit" in
|
||||
GiB/s) factor=1073741824 ;;
|
||||
MiB/s) factor=1048576 ;;
|
||||
KiB/s) factor=1024 ;;
|
||||
GB/s) factor=1000000000 ;;
|
||||
MB/s) factor=1000000 ;;
|
||||
KB/s) factor=1000 ;;
|
||||
B/s) factor=1 ;;
|
||||
*)
|
||||
echo "N/A"
|
||||
return
|
||||
;;
|
||||
esac
|
||||
|
||||
awk -v n="$number" -v f="$factor" 'BEGIN { printf "%.6f\n", n * f }'
|
||||
}
|
||||
|
||||
to_ms() {
|
||||
local human="$1"
|
||||
local number unit factor
|
||||
if [[ "$human" == "N/A" || -z "$human" ]]; then
|
||||
echo "N/A"
|
||||
return
|
||||
fi
|
||||
|
||||
number="$(echo "$human" | awk '{print $1}')"
|
||||
unit="$(echo "$human" | awk '{print $2}')"
|
||||
case "$unit" in
|
||||
s) factor=1000 ;;
|
||||
ms) factor=1 ;;
|
||||
us|µs) factor=0.001 ;;
|
||||
*)
|
||||
echo "N/A"
|
||||
return
|
||||
;;
|
||||
esac
|
||||
|
||||
awk -v n="$number" -v f="$factor" 'BEGIN { printf "%.6f\n", n * f }'
|
||||
}
|
||||
|
||||
extract_final_get_metrics() {
|
||||
local log_file="$1"
|
||||
local avg_line reqs_line throughput_human reqps latency_human
|
||||
|
||||
avg_line="$(awk '/^[[:space:]]*\* Average:/ { line=$0 } END { print line }' "$log_file")"
|
||||
reqs_line="$(awk '/^[[:space:]]*\* Reqs: Avg:/ { line=$0 } END { print line }' "$log_file")"
|
||||
|
||||
throughput_human="$(
|
||||
echo "$avg_line" \
|
||||
| sed -nE 's/^[[:space:]]*\* Average: ([0-9]+(\.[0-9]+)? [A-Za-z\/]+), ([0-9]+(\.[0-9]+)?) obj\/s$/\1/p'
|
||||
)"
|
||||
reqps="$(
|
||||
echo "$avg_line" \
|
||||
| sed -nE 's/^[[:space:]]*\* Average: ([0-9]+(\.[0-9]+)? [A-Za-z\/]+), ([0-9]+(\.[0-9]+)?) obj\/s$/\3/p'
|
||||
)"
|
||||
latency_human="$(
|
||||
echo "$reqs_line" \
|
||||
| sed -nE 's/^[[:space:]]*\* Reqs: Avg: ([0-9]+(\.[0-9]+)?)(ms|us|µs|s),.*$/\1 \3/p'
|
||||
)"
|
||||
|
||||
echo "${throughput_human:-N/A},${reqps:-N/A},${latency_human:-N/A}"
|
||||
}
|
||||
|
||||
median_from_numbers() {
|
||||
local values="$1"
|
||||
local count
|
||||
|
||||
count="$(printf '%s\n' "$values" | awk 'NF{c++} END{print c+0}')"
|
||||
if [[ "$count" -eq 0 ]]; then
|
||||
echo "N/A"
|
||||
return
|
||||
fi
|
||||
|
||||
printf '%s\n' "$values" | awk 'NF' | sort -n | awk '
|
||||
{a[NR]=$1}
|
||||
END{
|
||||
n=NR
|
||||
if (n==0) { print "N/A"; exit }
|
||||
if (n%2==1) {
|
||||
printf "%.6f\n", a[(n+1)/2]
|
||||
} else {
|
||||
printf "%.6f\n", (a[n/2]+a[n/2+1])/2
|
||||
}
|
||||
}'
|
||||
}
|
||||
|
||||
rebuild_round_results() {
|
||||
local round_csv="${OUT_DIR}/warp/round_results.csv"
|
||||
local tmp_csv
|
||||
tmp_csv="$(mktemp)"
|
||||
|
||||
while IFS=',' read -r size tool round attempt concurrency status throughput_human throughput_bps reqps latency_human latency_ms log_file; do
|
||||
if [[ "$size" == "size" ]]; then
|
||||
echo "$size,$tool,$round,$attempt,$concurrency,$status,$throughput_human,$throughput_bps,$reqps,$latency_human,$latency_ms,$log_file" >> "$tmp_csv"
|
||||
continue
|
||||
fi
|
||||
|
||||
if [[ "$status" == "ok" ]]; then
|
||||
local metrics
|
||||
metrics="$(extract_final_get_metrics "$log_file")"
|
||||
throughput_human="$(echo "$metrics" | cut -d',' -f1)"
|
||||
reqps="$(echo "$metrics" | cut -d',' -f2)"
|
||||
latency_human="$(echo "$metrics" | cut -d',' -f3)"
|
||||
throughput_bps="$(to_bps "$throughput_human")"
|
||||
latency_ms="$(to_ms "$latency_human")"
|
||||
fi
|
||||
|
||||
echo "$size,$tool,$round,$attempt,$concurrency,$status,$throughput_human,$throughput_bps,$reqps,$latency_human,$latency_ms,$log_file" >> "$tmp_csv"
|
||||
done < "$round_csv"
|
||||
|
||||
mv "$tmp_csv" "$round_csv"
|
||||
}
|
||||
|
||||
rebuild_median_summary() {
|
||||
local round_csv="${OUT_DIR}/warp/round_results.csv"
|
||||
local median_csv="${OUT_DIR}/warp/median_summary.csv"
|
||||
local ok_rounds fail_rounds t_vals r_vals l_vals m_t m_r m_l
|
||||
|
||||
ok_rounds="$(awk -F',' 'NR>1 && $6=="ok" {c++} END{print c+0}' "$round_csv")"
|
||||
fail_rounds="$(awk -F',' 'NR>1 && $6!="ok" {c++} END{print c+0}' "$round_csv")"
|
||||
t_vals="$(awk -F',' 'NR>1 && $6=="ok" && $8!="N/A" {print $8}' "$round_csv")"
|
||||
r_vals="$(awk -F',' 'NR>1 && $6=="ok" && $9!="N/A" {print $9}' "$round_csv")"
|
||||
l_vals="$(awk -F',' 'NR>1 && $6=="ok" && $11!="N/A" {print $11}' "$round_csv")"
|
||||
|
||||
m_t="$(median_from_numbers "$t_vals")"
|
||||
m_r="$(median_from_numbers "$r_vals")"
|
||||
m_l="$(median_from_numbers "$l_vals")"
|
||||
|
||||
{
|
||||
echo "size,tool,concurrency,successful_rounds,failed_rounds,median_throughput_bps,median_reqps,median_latency_ms"
|
||||
echo "$SIZE,warp,$CONCURRENCY,$ok_rounds,$fail_rounds,$m_t,$m_r,$m_l"
|
||||
} > "$median_csv"
|
||||
}
|
||||
|
||||
rebuild_baseline_compare() {
|
||||
local compare_csv="${OUT_DIR}/warp/baseline_compare.csv"
|
||||
if [[ -z "$BASELINE_CSV" ]]; then
|
||||
return
|
||||
fi
|
||||
|
||||
echo "size,tool,concurrency,new_median_reqps,baseline_median_reqps,delta_reqps_pct,new_median_latency_ms,baseline_median_latency_ms,delta_latency_pct,new_median_throughput_bps,baseline_median_throughput_bps,delta_throughput_pct" > "$compare_csv"
|
||||
|
||||
awk -F',' '
|
||||
NR==FNR {
|
||||
if (FNR==1) next
|
||||
b_req=$7
|
||||
b_lat=$8
|
||||
b_thr=$6
|
||||
next
|
||||
}
|
||||
FNR==2 {
|
||||
n_thr=$6
|
||||
n_req=$7
|
||||
n_lat=$8
|
||||
dr="N/A"; dl="N/A"; dt="N/A"
|
||||
if (b_req!="N/A" && n_req!="N/A" && b_req+0!=0) dr=sprintf("%.2f", ((n_req-b_req)/b_req)*100)
|
||||
if (b_lat!="N/A" && n_lat!="N/A" && b_lat+0!=0) dl=sprintf("%.2f", ((n_lat-b_lat)/b_lat)*100)
|
||||
if (b_thr!="N/A" && n_thr!="N/A" && b_thr+0!=0) dt=sprintf("%.2f", ((n_thr-b_thr)/b_thr)*100)
|
||||
print $1 "," $2 "," $3 "," n_req "," b_req "," dr "," n_lat "," b_lat "," dl "," n_thr "," b_thr "," dt
|
||||
}
|
||||
' "$BASELINE_CSV" "${OUT_DIR}/warp/median_summary.csv" >> "$compare_csv"
|
||||
}
|
||||
|
||||
postprocess_results() {
|
||||
rebuild_round_results
|
||||
rebuild_median_summary
|
||||
rebuild_baseline_compare
|
||||
}
|
||||
|
||||
main() {
|
||||
parse_args "$@"
|
||||
validate_args
|
||||
setup_paths
|
||||
write_manifest
|
||||
build_rustfs_if_needed
|
||||
|
||||
trap stop_server EXIT INT TERM
|
||||
|
||||
echo "Output dir: $OUT_DIR"
|
||||
start_server
|
||||
run_bench
|
||||
postprocess_results
|
||||
stop_server
|
||||
}
|
||||
|
||||
main "$@"
|
||||
@@ -0,0 +1,222 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
|
||||
|
||||
HOST="${HOST:-127.0.0.1:9000}"
|
||||
ACCESS_KEY="${ACCESS_KEY:-}"
|
||||
SECRET_KEY="${SECRET_KEY:-}"
|
||||
BUCKET_PREFIX="${BUCKET_PREFIX:-issue712-gt1g-multipart-focus}"
|
||||
LOOKUP="${LOOKUP:-path}"
|
||||
DURATION="${DURATION:-10m}"
|
||||
WARP_BIN="${WARP_BIN:-warp}"
|
||||
OUT_DIR="${OUT_DIR:-${PROJECT_ROOT}/target/bench/gt1g-multipart-server-path-$(date +%Y%m%d-%H%M%S)}"
|
||||
DRY_RUN=false
|
||||
|
||||
DEFAULT_PROFILES="1g-64m-pc4,2g-128m-pc4"
|
||||
PROFILES="${PROFILES:-$DEFAULT_PROFILES}"
|
||||
|
||||
usage() {
|
||||
cat <<'USAGE'
|
||||
Usage:
|
||||
scripts/run_gt1g_multipart_put_server_path_focus.sh --access-key <ak> --secret-key <sk> [options]
|
||||
|
||||
Required:
|
||||
--access-key <ak>
|
||||
--secret-key <sk>
|
||||
|
||||
Options:
|
||||
--host <host:port> Default: 127.0.0.1:9000
|
||||
--bucket-prefix <prefix> Default: issue712-gt1g-multipart-focus
|
||||
--lookup <path|dns> Default: path
|
||||
--duration <dur> Default: 10m
|
||||
--profiles <csv> Default: 1g-64m-pc4,2g-128m-pc4
|
||||
--warp-bin <path> Default: warp
|
||||
--out-dir <dir> Default: target/bench/gt1g-multipart-server-path-<timestamp>
|
||||
--dry-run
|
||||
-h, --help
|
||||
|
||||
Profiles:
|
||||
1g-64m-pc4 concurrent=4 parts=16 part.size=64MiB part.concurrent=4
|
||||
2g-128m-pc4 concurrent=4 parts=16 part.size=128MiB part.concurrent=4
|
||||
2g-256m-pc4 concurrent=4 parts=8 part.size=256MiB part.concurrent=4
|
||||
USAGE
|
||||
}
|
||||
|
||||
require_cmd() {
|
||||
if ! command -v "$1" >/dev/null 2>&1; then
|
||||
echo "ERROR: command not found: $1" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
trim() {
|
||||
echo "$1" | awk '{$1=$1;print}'
|
||||
}
|
||||
|
||||
parse_args() {
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--host) HOST="$2"; shift 2 ;;
|
||||
--access-key) ACCESS_KEY="$2"; shift 2 ;;
|
||||
--secret-key) SECRET_KEY="$2"; shift 2 ;;
|
||||
--bucket-prefix) BUCKET_PREFIX="$2"; shift 2 ;;
|
||||
--lookup) LOOKUP="$2"; shift 2 ;;
|
||||
--duration) DURATION="$2"; shift 2 ;;
|
||||
--profiles) PROFILES="$2"; shift 2 ;;
|
||||
--warp-bin) WARP_BIN="$2"; shift 2 ;;
|
||||
--out-dir) OUT_DIR="$2"; shift 2 ;;
|
||||
--dry-run) DRY_RUN=true; shift ;;
|
||||
-h|--help) usage; exit 0 ;;
|
||||
*)
|
||||
echo "ERROR: unknown arg: $1" >&2
|
||||
usage
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
}
|
||||
|
||||
validate_args() {
|
||||
if [[ -z "$ACCESS_KEY" || -z "$SECRET_KEY" ]]; then
|
||||
echo "ERROR: --access-key and --secret-key are required" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
setup_output() {
|
||||
mkdir -p "$OUT_DIR/logs" "$OUT_DIR/benchdata"
|
||||
SUMMARY_CSV="$OUT_DIR/summary.csv"
|
||||
COMMANDS_TXT="$OUT_DIR/commands.txt"
|
||||
MANIFEST_TXT="$OUT_DIR/run_manifest.txt"
|
||||
|
||||
echo "profile,host,bucket,duration,concurrent,parts,part_size,part_concurrent,throughput_human,reqps,latency_human,log_file,benchdata_file,status" > "$SUMMARY_CSV"
|
||||
: > "$COMMANDS_TXT"
|
||||
}
|
||||
|
||||
write_manifest() {
|
||||
{
|
||||
echo "created_at=$(date +%Y-%m-%dT%H:%M:%S%z)"
|
||||
echo "host=${HOST}"
|
||||
echo "bucket_prefix=${BUCKET_PREFIX}"
|
||||
echo "lookup=${LOOKUP}"
|
||||
echo "duration=${DURATION}"
|
||||
echo "profiles=${PROFILES}"
|
||||
echo "warp_bin=${WARP_BIN}"
|
||||
echo "dry_run=${DRY_RUN}"
|
||||
echo "access_key=REDACTED"
|
||||
echo "secret_key=REDACTED"
|
||||
} > "$MANIFEST_TXT"
|
||||
}
|
||||
|
||||
profile_spec() {
|
||||
case "$1" in
|
||||
1g-64m-pc4) echo "4|16|64MiB|4" ;;
|
||||
2g-128m-pc4) echo "4|16|128MiB|4" ;;
|
||||
2g-256m-pc4) echo "4|8|256MiB|4" ;;
|
||||
*)
|
||||
echo "ERROR: unknown profile: $1" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
extract_first() {
|
||||
local regex="$1"
|
||||
local file="$2"
|
||||
rg -o "$regex" "$file" | head -n1 || true
|
||||
}
|
||||
|
||||
extract_metrics() {
|
||||
local log_file="$1"
|
||||
local throughput reqps latency
|
||||
|
||||
throughput="$(extract_first '[0-9]+(\.[0-9]+)?[[:space:]]*(GiB/s|MiB/s|KiB/s|GB/s|MB/s|KB/s|B/s)' "$log_file")"
|
||||
reqps="$(extract_first '[0-9]+(\.[0-9]+)?[[:space:]]*(obj/s|req/s|ops/s|requests/s)' "$log_file")"
|
||||
latency="$(rg -o 'Reqs:[[:space:]]+Avg:[[:space:]]+[0-9]+(\.[0-9]+)?(ms|us|µs|s)' "$log_file" | head -n1 | sed -E 's/^Reqs:[[:space:]]+Avg:[[:space:]]+//')"
|
||||
|
||||
throughput="$(trim "${throughput:-N/A}")"
|
||||
reqps="$(trim "${reqps:-N/A}")"
|
||||
latency="$(trim "${latency:-N/A}")"
|
||||
reqps="$(echo "$reqps" | awk '{print $1}')"
|
||||
|
||||
echo "${throughput},${reqps:-N/A},${latency}"
|
||||
}
|
||||
|
||||
run_profile() {
|
||||
local profile="$1"
|
||||
local spec concurrent parts part_size part_concurrent
|
||||
local bucket log_file benchdata_file status metrics throughput reqps latency
|
||||
|
||||
spec="$(profile_spec "$profile")"
|
||||
IFS='|' read -r concurrent parts part_size part_concurrent <<< "$spec"
|
||||
|
||||
bucket="$(echo "${BUCKET_PREFIX}-${profile}-$(date +%Y%m%d%H%M%S)" | tr '[:upper:]' '[:lower:]' | sed -E 's/[^a-z0-9-]+/-/g; s/-{2,}/-/g; s/^-+//; s/-+$//')"
|
||||
bucket="${bucket:0:63}"
|
||||
log_file="$OUT_DIR/logs/${profile}.log"
|
||||
benchdata_file="$OUT_DIR/benchdata/${profile}.csv.zst"
|
||||
|
||||
local -a cmd=(
|
||||
"$WARP_BIN" multipart-put
|
||||
--host "$HOST"
|
||||
--access-key "$ACCESS_KEY"
|
||||
--secret-key "$SECRET_KEY"
|
||||
--bucket "$bucket"
|
||||
--lookup "$LOOKUP"
|
||||
--duration "$DURATION"
|
||||
--concurrent "$concurrent"
|
||||
--parts "$parts"
|
||||
--part.size "$part_size"
|
||||
--part.concurrent "$part_concurrent"
|
||||
--benchdata "$benchdata_file"
|
||||
--analyze.v
|
||||
--no-color
|
||||
)
|
||||
|
||||
printf '%q ' "${cmd[@]}" >> "$COMMANDS_TXT"
|
||||
printf '\n' >> "$COMMANDS_TXT"
|
||||
|
||||
status="ok"
|
||||
if [[ "$DRY_RUN" == "true" ]]; then
|
||||
printf '[DRY-RUN] %q ' "${cmd[@]}"
|
||||
printf '\n'
|
||||
: > "$log_file"
|
||||
else
|
||||
if ! "${cmd[@]}" >"$log_file" 2>&1; then
|
||||
status="failed"
|
||||
fi
|
||||
fi
|
||||
|
||||
metrics="$(extract_metrics "$log_file")"
|
||||
throughput="$(echo "$metrics" | cut -d',' -f1)"
|
||||
reqps="$(echo "$metrics" | cut -d',' -f2)"
|
||||
latency="$(echo "$metrics" | cut -d',' -f3)"
|
||||
|
||||
echo "${profile},${HOST},${bucket},${DURATION},${concurrent},${parts},${part_size},${part_concurrent},${throughput},${reqps},${latency},${log_file},${benchdata_file},${status}" >> "$SUMMARY_CSV"
|
||||
}
|
||||
|
||||
main() {
|
||||
parse_args "$@"
|
||||
validate_args
|
||||
require_cmd "$WARP_BIN"
|
||||
require_cmd rg
|
||||
setup_output
|
||||
write_manifest
|
||||
|
||||
echo "Output dir: $OUT_DIR"
|
||||
echo "Profiles: $PROFILES"
|
||||
|
||||
IFS=',' read -r -a profiles <<< "$PROFILES"
|
||||
for raw in "${profiles[@]}"; do
|
||||
profile="$(trim "$raw")"
|
||||
[[ -z "$profile" ]] && continue
|
||||
run_profile "$profile"
|
||||
done
|
||||
|
||||
echo
|
||||
echo "Summary:"
|
||||
cat "$SUMMARY_CSV"
|
||||
}
|
||||
|
||||
main "$@"
|
||||
@@ -0,0 +1,141 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
|
||||
RUNNER_SCRIPT="${PROJECT_ROOT}/scripts/run_put_large_stage_breakdown_with_capture.sh"
|
||||
|
||||
ENDPOINT="${ENDPOINT:-http://127.0.0.1:9000}"
|
||||
ACCESS_KEY="${ACCESS_KEY:-}"
|
||||
SECRET_KEY="${SECRET_KEY:-}"
|
||||
REGION="${REGION:-us-east-1}"
|
||||
SIZES="${SIZES:-64MiB,128MiB,256MiB}"
|
||||
CONCURRENCIES="${CONCURRENCIES:-16}"
|
||||
DURATION="${DURATION:-60s}"
|
||||
ROUNDS="${ROUNDS:-1}"
|
||||
COOLDOWN_SECS="${COOLDOWN_SECS:-15}"
|
||||
OUT_DIR="${OUT_DIR:-target/bench/issue712-deeper-zero-copy-capture-$(date -u +%Y%m%dT%H%M%SZ)}"
|
||||
CAPTURE_INTERVAL_SECS="${CAPTURE_INTERVAL_SECS:-15}"
|
||||
CAPTURE_PROM_METRICS_URLS="${CAPTURE_PROM_METRICS_URLS:-http://127.0.0.1:8889/metrics}"
|
||||
CAPTURE_RUSTFS_PID="${CAPTURE_RUSTFS_PID:-}"
|
||||
WORKLOAD_LABEL="${WORKLOAD_LABEL:-issue-712-deeper-zero-copy}"
|
||||
DRY_RUN=false
|
||||
|
||||
usage() {
|
||||
cat <<'USAGE'
|
||||
Usage:
|
||||
scripts/run_issue712_deeper_zero_copy_put_with_capture.sh \
|
||||
--access-key <ak> --secret-key <sk> [options]
|
||||
|
||||
Required:
|
||||
--access-key <ak>
|
||||
--secret-key <sk>
|
||||
|
||||
Options:
|
||||
--endpoint <url> Default: http://127.0.0.1:9000
|
||||
--region <name> Default: us-east-1
|
||||
--sizes <csv> Default: 64MiB,128MiB,256MiB
|
||||
--concurrencies <csv> Default: 16
|
||||
--duration <dur> Default: 60s
|
||||
--rounds <n> Default: 1
|
||||
--cooldown-secs <n> Default: 15
|
||||
--out-dir <dir> Default: target/bench/issue712-deeper-zero-copy-capture-<timestamp>
|
||||
--capture-interval-secs <n> Default: 15
|
||||
--capture-prom-metrics-urls <csv>
|
||||
Default: http://127.0.0.1:8889/metrics
|
||||
--capture-rustfs-pid <pid> Optional explicit rustfs pid
|
||||
--dry-run
|
||||
-h, --help
|
||||
|
||||
Notes:
|
||||
- This wrapper assumes the local RustFS server is already running with:
|
||||
RUSTFS_ERASURE_ENCODE_BYTESMUT_INGEST=true
|
||||
- It reuses scripts/run_put_large_stage_breakdown_with_capture.sh
|
||||
and only narrows the matrix to the deeper-zero-copy focus area.
|
||||
USAGE
|
||||
}
|
||||
|
||||
arg_value() {
|
||||
local flag="$1"
|
||||
local value="${2:-}"
|
||||
if [[ -z "$value" || "$value" == --* ]]; then
|
||||
echo "ERROR: missing value for $flag" >&2
|
||||
exit 1
|
||||
fi
|
||||
printf '%s\n' "$value"
|
||||
}
|
||||
|
||||
parse_args() {
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--endpoint) ENDPOINT="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--access-key) ACCESS_KEY="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--secret-key) SECRET_KEY="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--region) REGION="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--sizes) SIZES="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--concurrencies) CONCURRENCIES="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--duration) DURATION="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--rounds) ROUNDS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--cooldown-secs) COOLDOWN_SECS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--out-dir) OUT_DIR="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--capture-interval-secs) CAPTURE_INTERVAL_SECS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--capture-prom-metrics-urls) CAPTURE_PROM_METRICS_URLS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--capture-rustfs-pid) CAPTURE_RUSTFS_PID="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--dry-run) DRY_RUN=true; shift ;;
|
||||
-h|--help) usage; exit 0 ;;
|
||||
*)
|
||||
echo "ERROR: unknown arg: $1" >&2
|
||||
usage
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
}
|
||||
|
||||
validate_args() {
|
||||
if [[ -z "$ACCESS_KEY" || -z "$SECRET_KEY" ]]; then
|
||||
echo "ERROR: --access-key and --secret-key are required" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
main() {
|
||||
parse_args "$@"
|
||||
validate_args
|
||||
|
||||
local -a cmd=(
|
||||
bash "$RUNNER_SCRIPT"
|
||||
--endpoint "$ENDPOINT"
|
||||
--access-key "$ACCESS_KEY"
|
||||
--secret-key "$SECRET_KEY"
|
||||
--region "$REGION"
|
||||
--sizes "$SIZES"
|
||||
--concurrencies "$CONCURRENCIES"
|
||||
--duration "$DURATION"
|
||||
--rounds "$ROUNDS"
|
||||
--retry-per-round 1
|
||||
--retry-sleep-secs 2
|
||||
--cooldown-secs "$COOLDOWN_SECS"
|
||||
--out-dir "$OUT_DIR"
|
||||
--workload-label "$WORKLOAD_LABEL"
|
||||
--capture-label deeper-zero-copy-window
|
||||
--capture-interval-secs "$CAPTURE_INTERVAL_SECS"
|
||||
--capture-prom-metrics-urls "$CAPTURE_PROM_METRICS_URLS"
|
||||
)
|
||||
|
||||
if [[ -n "$CAPTURE_RUSTFS_PID" ]]; then
|
||||
cmd+=(--capture-rustfs-pid "$CAPTURE_RUSTFS_PID")
|
||||
fi
|
||||
|
||||
if [[ "$DRY_RUN" == "true" ]]; then
|
||||
cmd+=(--dry-run)
|
||||
fi
|
||||
|
||||
printf 'Command:'
|
||||
printf ' %q' "${cmd[@]}"
|
||||
printf '\n'
|
||||
|
||||
"${cmd[@]}"
|
||||
}
|
||||
|
||||
main "$@"
|
||||
+582
@@ -0,0 +1,582 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# Local 4-node / 16-disk RustFS runner for rustfs/backlog#797.
|
||||
# It starts four local RustFS processes, runs warp workloads, and captures
|
||||
# health, logs, benchmark summaries, and optional signed admin metrics.
|
||||
|
||||
PROJECT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
|
||||
RUSTFS_BIN="${RUSTFS_BIN:-${PROJECT_ROOT}/target/debug/rustfs}"
|
||||
BUILD_BIN="${BUILD_BIN:-true}"
|
||||
BASE_PORT="${BASE_PORT:-19100}"
|
||||
NODE_COUNT="${NODE_COUNT:-4}"
|
||||
DISKS_PER_NODE="${DISKS_PER_NODE:-4}"
|
||||
ACCESS_KEY="${RUSTFS_ACCESS_KEY:-rustfsadmin}"
|
||||
SECRET_KEY="${RUSTFS_SECRET_KEY:-rustfsadmin}"
|
||||
REGION="${REGION:-us-east-1}"
|
||||
SIZES="${SIZES:-4KiB,1MiB}"
|
||||
CONCURRENCY="${CONCURRENCY:-8}"
|
||||
DURATION="${DURATION:-60s}"
|
||||
WARP_BIN="${WARP_BIN:-warp}"
|
||||
WARP_MODE="${WARP_MODE:-mixed}"
|
||||
WARP_EXTRA_ARGS="${WARP_EXTRA_ARGS:---noclear}"
|
||||
PROFILES="${PROFILES:-baseline,metrics_logging}"
|
||||
OUT_DIR="${OUT_DIR:-${PROJECT_ROOT}/target/bench/issue-797-local-4node-16disk-ab-$(date -u +%Y%m%dT%H%M%SZ)}"
|
||||
DATA_ROOT="${DATA_ROOT:-/tmp/issue797-local-4node-16disk-ab-$(date -u +%Y%m%dT%H%M%SZ)}"
|
||||
KEEP_DATA="${KEEP_DATA:-false}"
|
||||
WAIT_TIMEOUT_SECS="${WAIT_TIMEOUT_SECS:-180}"
|
||||
HEALTH_POLL_SECS="${HEALTH_POLL_SECS:-2}"
|
||||
CAPTURE_ADMIN_METRICS="${CAPTURE_ADMIN_METRICS:-true}"
|
||||
AWSCURL_BIN="${AWSCURL_BIN:-awscurl}"
|
||||
CURL_BIN="${CURL_BIN:-curl}"
|
||||
RG_BIN="${RG_BIN:-rg}"
|
||||
DRY_RUN=false
|
||||
|
||||
PIDS=()
|
||||
|
||||
usage() {
|
||||
cat <<'USAGE'
|
||||
Usage:
|
||||
scripts/run_issue797_local_4node_16disk_ab.sh [options]
|
||||
|
||||
Options:
|
||||
--rustfs-bin <path> RustFS binary (default: target/debug/rustfs)
|
||||
--skip-build Do not build rustfs before running
|
||||
--base-port <port> First node port; uses port..port+3
|
||||
--sizes <csv> Object sizes for warp (default: 4KiB,1MiB)
|
||||
--concurrency <n> Warp concurrency (default: 8)
|
||||
--duration <dur> Warp duration per size/profile (default: 60s)
|
||||
--profiles <csv> baseline,metrics_logging,locality_on
|
||||
--out-dir <dir> Output directory
|
||||
--data-root <dir> Temporary disk root
|
||||
--keep-data Keep data root after exit
|
||||
--warp-extra-args <args> Extra args appended to warp (default: --noclear)
|
||||
--skip-admin-metrics Do not attempt signed admin metrics capture
|
||||
--dry-run Print planned layout without running
|
||||
-h, --help Show help
|
||||
|
||||
Profiles:
|
||||
baseline Metrics exports, file logging, shard locality, and batch
|
||||
processor observation disabled.
|
||||
metrics_logging Enables metrics export gate, bounded warn-level file
|
||||
logging, shard locality observe mode, and batch processor
|
||||
observe mode.
|
||||
locality_on Same as metrics_logging, but uses shard locality on mode.
|
||||
|
||||
Notes:
|
||||
Admin metrics are captured from /rustfs/admin/v3/metrics when awscurl or
|
||||
curl --aws-sigv4 is available. If neither is available, benchmark still runs
|
||||
and records a skipped metrics status file.
|
||||
USAGE
|
||||
}
|
||||
|
||||
log() {
|
||||
printf '[INFO] %s\n' "$*"
|
||||
}
|
||||
|
||||
warn() {
|
||||
printf '[WARN] %s\n' "$*" >&2
|
||||
}
|
||||
|
||||
die() {
|
||||
printf '[ERROR] %s\n' "$*" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
arg_value() {
|
||||
local flag="$1"
|
||||
local value="${2:-}"
|
||||
if [[ -z "$value" || "$value" == --* ]]; then
|
||||
die "missing value for ${flag}"
|
||||
fi
|
||||
printf '%s\n' "$value"
|
||||
}
|
||||
|
||||
# Like arg_value, but accepts values that themselves start with `--`
|
||||
# (e.g. `--warp-extra-args --noclear`).
|
||||
arg_value_allow_dashes() {
|
||||
local flag="$1"
|
||||
local value="${2:-}"
|
||||
if [[ -z "$value" ]]; then
|
||||
die "missing value for ${flag}"
|
||||
fi
|
||||
printf '%s\n' "$value"
|
||||
}
|
||||
|
||||
parse_args() {
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--rustfs-bin) RUSTFS_BIN="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--skip-build) BUILD_BIN=false; shift ;;
|
||||
--base-port) BASE_PORT="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--sizes) SIZES="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--concurrency) CONCURRENCY="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--duration) DURATION="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--profiles) PROFILES="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--out-dir) OUT_DIR="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--data-root) DATA_ROOT="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--keep-data) KEEP_DATA=true; shift ;;
|
||||
--warp-extra-args) WARP_EXTRA_ARGS="$(arg_value_allow_dashes "$1" "${2:-}")"; shift 2 ;;
|
||||
--skip-admin-metrics) CAPTURE_ADMIN_METRICS=false; shift ;;
|
||||
--dry-run) DRY_RUN=true; shift ;;
|
||||
-h|--help) usage; exit 0 ;;
|
||||
*) die "unknown arg: $1" ;;
|
||||
esac
|
||||
done
|
||||
}
|
||||
|
||||
is_positive_integer() {
|
||||
[[ "$1" =~ ^[1-9][0-9]*$ ]]
|
||||
}
|
||||
|
||||
validate_args() {
|
||||
is_positive_integer "$BASE_PORT" || die "--base-port must be a positive integer"
|
||||
is_positive_integer "$CONCURRENCY" || die "--concurrency must be a positive integer"
|
||||
[[ "$NODE_COUNT" == "4" ]] || die "NODE_COUNT is fixed to 4 for this runner"
|
||||
[[ "$DISKS_PER_NODE" == "4" ]] || die "DISKS_PER_NODE is fixed to 4 for this runner"
|
||||
}
|
||||
|
||||
require_cmd() {
|
||||
if ! command -v "$1" >/dev/null 2>&1; then
|
||||
die "command not found: $1"
|
||||
fi
|
||||
}
|
||||
|
||||
print_redacted_command() {
|
||||
local redact_next=false
|
||||
local arg
|
||||
|
||||
printf 'Command:'
|
||||
for arg in "$@"; do
|
||||
if [[ "$redact_next" == "true" ]]; then
|
||||
printf ' %q' "REDACTED"
|
||||
redact_next=false
|
||||
continue
|
||||
fi
|
||||
|
||||
case "$arg" in
|
||||
--access-key|--secret-key)
|
||||
printf ' %q' "$arg"
|
||||
redact_next=true
|
||||
;;
|
||||
-accessKey=*|-secretKey=*)
|
||||
printf ' %q' "${arg%%=*}=REDACTED"
|
||||
;;
|
||||
*)
|
||||
printf ' %q' "$arg"
|
||||
;;
|
||||
esac
|
||||
done
|
||||
printf '\n'
|
||||
}
|
||||
|
||||
endpoint_for_node() {
|
||||
local endpoint_node_id="$1"
|
||||
printf 'http://127.0.0.1:%s' "$((BASE_PORT + endpoint_node_id - 1))"
|
||||
}
|
||||
|
||||
endpoint_label() {
|
||||
local endpoint="$1"
|
||||
local label
|
||||
label="${endpoint#*://}"
|
||||
label="${label%%/*}"
|
||||
printf '%s' "$label" | tr -c 'A-Za-z0-9._-' '_'
|
||||
}
|
||||
|
||||
all_endpoints_csv() {
|
||||
local endpoint_loop_id
|
||||
local endpoints=()
|
||||
for ((endpoint_loop_id = 1; endpoint_loop_id <= NODE_COUNT; endpoint_loop_id++)); do
|
||||
endpoints+=("$(endpoint_for_node "$endpoint_loop_id")")
|
||||
done
|
||||
local IFS=','
|
||||
printf '%s' "${endpoints[*]}"
|
||||
}
|
||||
|
||||
metrics_url() {
|
||||
# SCANNER(1) + NET(32) + RPC(256): enough for readiness context plus
|
||||
# internode aggregate traffic without collecting the heavier all-metrics set.
|
||||
printf '%s/rustfs/admin/v3/metrics?types=289&by-host=true&n=1\n' "${1%/}"
|
||||
}
|
||||
|
||||
curl_supports_aws_sigv4() {
|
||||
"$CURL_BIN" --help all 2>/dev/null | grep -q -- '--aws-sigv4'
|
||||
}
|
||||
|
||||
profile_is_supported() {
|
||||
case "$1" in
|
||||
baseline|metrics_logging|locality_on) return 0 ;;
|
||||
*) return 1 ;;
|
||||
esac
|
||||
}
|
||||
|
||||
profile_env_file() {
|
||||
local profile="$1"
|
||||
local env_file="$2"
|
||||
case "$profile" in
|
||||
baseline)
|
||||
cat >"$env_file" <<'EOF'
|
||||
RUSTFS_OBS_LOGGER_LEVEL=off
|
||||
RUSTFS_OBS_TRACES_EXPORT_ENABLED=false
|
||||
RUSTFS_OBS_METRICS_EXPORT_ENABLED=false
|
||||
RUSTFS_OBS_LOGS_EXPORT_ENABLED=false
|
||||
RUSTFS_OBS_PROFILING_EXPORT_ENABLED=false
|
||||
RUSTFS_OBS_USE_STDOUT=false
|
||||
RUSTFS_OBS_LOG_STDOUT_ENABLED=false
|
||||
RUSTFS_BATCH_PROCESSOR_ADAPTIVE=off
|
||||
RUSTFS_SHARD_LOCALITY_SCHEDULING=off
|
||||
EOF
|
||||
;;
|
||||
metrics_logging)
|
||||
cat >"$env_file" <<'EOF'
|
||||
RUSTFS_OBS_LOGGER_LEVEL=warn
|
||||
RUSTFS_OBS_ENDPOINT=http://127.0.0.1:4318
|
||||
RUSTFS_OBS_ENDPOINT_TIMEOUT_MILLIS=500
|
||||
RUSTFS_OBS_TRACES_EXPORT_ENABLED=false
|
||||
RUSTFS_OBS_METRICS_EXPORT_ENABLED=true
|
||||
RUSTFS_OBS_LOGS_EXPORT_ENABLED=false
|
||||
RUSTFS_OBS_PROFILING_EXPORT_ENABLED=false
|
||||
RUSTFS_OBS_USE_STDOUT=false
|
||||
RUSTFS_OBS_LOG_STDOUT_ENABLED=false
|
||||
RUSTFS_OBS_METER_INTERVAL=5
|
||||
RUSTFS_OBS_LOG_KEEP_FILES=2
|
||||
RUSTFS_OBS_LOG_MAX_TOTAL_SIZE_BYTES=268435456
|
||||
RUSTFS_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES=134217728
|
||||
RUSTFS_OBS_LOG_MIN_FILE_AGE_SECONDS=0
|
||||
RUSTFS_OBS_LOG_CLEANUP_INTERVAL_SECONDS=30
|
||||
RUSTFS_BATCH_PROCESSOR_ADAPTIVE=observe
|
||||
RUSTFS_SHARD_LOCALITY_SCHEDULING=observe
|
||||
EOF
|
||||
;;
|
||||
locality_on)
|
||||
cat >"$env_file" <<'EOF'
|
||||
RUSTFS_OBS_LOGGER_LEVEL=warn
|
||||
RUSTFS_OBS_ENDPOINT=http://127.0.0.1:4318
|
||||
RUSTFS_OBS_ENDPOINT_TIMEOUT_MILLIS=500
|
||||
RUSTFS_OBS_TRACES_EXPORT_ENABLED=false
|
||||
RUSTFS_OBS_METRICS_EXPORT_ENABLED=true
|
||||
RUSTFS_OBS_LOGS_EXPORT_ENABLED=false
|
||||
RUSTFS_OBS_PROFILING_EXPORT_ENABLED=false
|
||||
RUSTFS_OBS_USE_STDOUT=false
|
||||
RUSTFS_OBS_LOG_STDOUT_ENABLED=false
|
||||
RUSTFS_OBS_METER_INTERVAL=5
|
||||
RUSTFS_OBS_LOG_KEEP_FILES=2
|
||||
RUSTFS_OBS_LOG_MAX_TOTAL_SIZE_BYTES=268435456
|
||||
RUSTFS_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES=134217728
|
||||
RUSTFS_OBS_LOG_MIN_FILE_AGE_SECONDS=0
|
||||
RUSTFS_OBS_LOG_CLEANUP_INTERVAL_SECONDS=30
|
||||
RUSTFS_BATCH_PROCESSOR_ADAPTIVE=observe
|
||||
RUSTFS_SHARD_LOCALITY_SCHEDULING=on
|
||||
EOF
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
load_profile_env() {
|
||||
local env_file="$1"
|
||||
set -a
|
||||
# shellcheck disable=SC1090
|
||||
source "$env_file"
|
||||
set +a
|
||||
}
|
||||
|
||||
prepare_profile_layout() {
|
||||
local profile="$1"
|
||||
local profile_dir="$OUT_DIR/$profile"
|
||||
mkdir -p "$profile_dir"/{bench,health,logs,metrics,pids}
|
||||
profile_env_file "$profile" "$profile_dir/profile.env"
|
||||
}
|
||||
|
||||
build_volumes() {
|
||||
local profile="$1"
|
||||
local node_index disk_index
|
||||
local volumes=()
|
||||
for ((node_index = 1; node_index <= NODE_COUNT; node_index++)); do
|
||||
for ((disk_index = 1; disk_index <= DISKS_PER_NODE; disk_index++)); do
|
||||
volumes+=("http://127.0.0.1:$((BASE_PORT + node_index - 1))${DATA_ROOT}/${profile}/node${node_index}/disk${disk_index}")
|
||||
done
|
||||
done
|
||||
printf '%s ' "${volumes[@]}"
|
||||
}
|
||||
|
||||
prepare_data_dirs() {
|
||||
local profile="$1"
|
||||
local node_index disk_index
|
||||
for ((node_index = 1; node_index <= NODE_COUNT; node_index++)); do
|
||||
for ((disk_index = 1; disk_index <= DISKS_PER_NODE; disk_index++)); do
|
||||
mkdir -p "${DATA_ROOT}/${profile}/node${node_index}/disk${disk_index}"
|
||||
done
|
||||
done
|
||||
}
|
||||
|
||||
stop_nodes() {
|
||||
local pid
|
||||
if [[ ${#PIDS[@]} -eq 0 ]]; then
|
||||
return
|
||||
fi
|
||||
for pid in "${PIDS[@]}"; do
|
||||
if kill -0 "$pid" >/dev/null 2>&1; then
|
||||
kill "$pid" >/dev/null 2>&1 || true
|
||||
fi
|
||||
done
|
||||
for ((_ = 1; _ <= 5; _++)); do
|
||||
local remaining=0
|
||||
for pid in "${PIDS[@]}"; do
|
||||
if kill -0 "$pid" >/dev/null 2>&1; then
|
||||
remaining=$((remaining + 1))
|
||||
fi
|
||||
done
|
||||
[[ "$remaining" -eq 0 ]] && break
|
||||
sleep 1
|
||||
done
|
||||
for pid in "${PIDS[@]}"; do
|
||||
if kill -0 "$pid" >/dev/null 2>&1; then
|
||||
kill -9 "$pid" >/dev/null 2>&1 || true
|
||||
fi
|
||||
wait "$pid" >/dev/null 2>&1 || true
|
||||
done
|
||||
PIDS=()
|
||||
}
|
||||
|
||||
cleanup() {
|
||||
stop_nodes
|
||||
if [[ "$KEEP_DATA" != "true" && "$DRY_RUN" != "true" && -n "$DATA_ROOT" && -d "$DATA_ROOT" ]]; then
|
||||
rm -rf "$DATA_ROOT"
|
||||
fi
|
||||
}
|
||||
|
||||
trap cleanup EXIT
|
||||
|
||||
start_nodes() {
|
||||
local profile="$1"
|
||||
local profile_dir="$OUT_DIR/$profile"
|
||||
local volumes
|
||||
volumes="$(build_volumes "$profile")"
|
||||
|
||||
prepare_data_dirs "$profile"
|
||||
load_profile_env "$profile_dir/profile.env"
|
||||
|
||||
local node_index endpoint log_file pid_file
|
||||
for ((node_index = 1; node_index <= NODE_COUNT; node_index++)); do
|
||||
endpoint="$(endpoint_for_node "$node_index")"
|
||||
log_file="$profile_dir/logs/node${node_index}.log"
|
||||
pid_file="$profile_dir/pids/node${node_index}.pid"
|
||||
|
||||
(
|
||||
export RUSTFS_ACCESS_KEY="$ACCESS_KEY"
|
||||
export RUSTFS_SECRET_KEY="$SECRET_KEY"
|
||||
export RUSTFS_ADDRESS="127.0.0.1:$((BASE_PORT + node_index - 1))"
|
||||
export RUSTFS_CONSOLE_ENABLE=false
|
||||
export RUSTFS_SCANNER_ENABLED=false
|
||||
export RUSTFS_SCANNER_START_DELAY_SECS=3600
|
||||
export RUSTFS_SCANNER_CYCLE=3600
|
||||
export RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true
|
||||
export RUSTFS_VOLUMES="$volumes"
|
||||
export RUSTFS_OBS_SERVICE_NAME="rustfs-issue797-${profile}-node${node_index}"
|
||||
export RUSTFS_OBS_LOG_DIRECTORY="$profile_dir/logs/node${node_index}"
|
||||
mkdir -p "$RUSTFS_OBS_LOG_DIRECTORY"
|
||||
# Keep both env and positional volumes: current CLI still requires the
|
||||
# positional server volumes, while env makes the layout visible in logs.
|
||||
# shellcheck disable=SC2086
|
||||
exec "$RUSTFS_BIN" server $volumes
|
||||
) >"$log_file" 2>&1 &
|
||||
PIDS+=("$!")
|
||||
printf '%s\n' "$!" >"$pid_file"
|
||||
log "started ${profile} node${node_index} ${endpoint} pid=$!"
|
||||
done
|
||||
}
|
||||
|
||||
wait_for_health() {
|
||||
local profile="$1"
|
||||
local profile_dir="$OUT_DIR/$profile"
|
||||
local deadline node_index endpoint status_file
|
||||
deadline=$((SECONDS + WAIT_TIMEOUT_SECS))
|
||||
while (( SECONDS < deadline )); do
|
||||
local ready=0
|
||||
for ((node_index = 1; node_index <= NODE_COUNT; node_index++)); do
|
||||
endpoint="$(endpoint_for_node "$node_index")"
|
||||
status_file="$profile_dir/health/node${node_index}.live"
|
||||
if "$CURL_BIN" -fsS --max-time 2 "${endpoint}/health/live" >"$status_file" 2>"$status_file.err"; then
|
||||
ready=$((ready + 1))
|
||||
fi
|
||||
done
|
||||
if [[ "$ready" -eq "$NODE_COUNT" ]]; then
|
||||
log "${profile}: all nodes are live"
|
||||
return 0
|
||||
fi
|
||||
sleep "$HEALTH_POLL_SECS"
|
||||
done
|
||||
|
||||
for ((node_index = 1; node_index <= NODE_COUNT; node_index++)); do
|
||||
warn "${profile} node${node_index} log tail:"
|
||||
tail -n 40 "$profile_dir/logs/node${node_index}.log" >&2 || true
|
||||
done
|
||||
die "${profile}: timed out waiting for health"
|
||||
}
|
||||
|
||||
capture_admin_metrics() {
|
||||
local profile="$1"
|
||||
local phase="$2"
|
||||
local profile_dir="$OUT_DIR/$profile"
|
||||
local status_file="$profile_dir/metrics/${phase}.status"
|
||||
local endpoint label metrics_file endpoints
|
||||
|
||||
if [[ "$CAPTURE_ADMIN_METRICS" != "true" ]]; then
|
||||
echo "skipped: disabled" >"$status_file"
|
||||
return 0
|
||||
fi
|
||||
local capture_method=""
|
||||
if command -v "$AWSCURL_BIN" >/dev/null 2>&1; then
|
||||
capture_method="awscurl"
|
||||
elif curl_supports_aws_sigv4; then
|
||||
capture_method="curl-sigv4"
|
||||
else
|
||||
echo "skipped: awscurl not found and curl lacks --aws-sigv4" >"$status_file"
|
||||
return 0
|
||||
fi
|
||||
|
||||
echo "capturing: ${capture_method}" >"$status_file"
|
||||
IFS=',' read -r -a endpoints <<<"$(all_endpoints_csv)"
|
||||
for endpoint in "${endpoints[@]}"; do
|
||||
label="$(endpoint_label "$endpoint")"
|
||||
metrics_file="$profile_dir/metrics/${phase}.${label}.ndjson"
|
||||
if [[ "$capture_method" == "awscurl" ]]; then
|
||||
AWS_ACCESS_KEY_ID="$ACCESS_KEY" \
|
||||
AWS_SECRET_ACCESS_KEY="$SECRET_KEY" \
|
||||
AWS_DEFAULT_REGION="$REGION" \
|
||||
"$AWSCURL_BIN" \
|
||||
--service s3 \
|
||||
--region "$REGION" \
|
||||
--request GET \
|
||||
"$(metrics_url "$endpoint")" \
|
||||
>"$metrics_file" 2>"$metrics_file.err" || true
|
||||
else
|
||||
local curl_config="$profile_dir/metrics/.curl-sigv4-${phase}.${label}.conf"
|
||||
{
|
||||
printf 'aws-sigv4 = "aws:amz:%s:s3"\n' "$REGION"
|
||||
printf 'user = "%s:%s"\n' "$ACCESS_KEY" "$SECRET_KEY"
|
||||
printf 'request = "GET"\n'
|
||||
printf 'max-time = 10\n'
|
||||
printf 'fail\n'
|
||||
printf 'silent\n'
|
||||
printf 'show-error\n'
|
||||
} >"$curl_config"
|
||||
chmod 600 "$curl_config"
|
||||
"$CURL_BIN" --config "$curl_config" "$(metrics_url "$endpoint")" >"$metrics_file" 2>"$metrics_file.err" || true
|
||||
rm -f "$curl_config"
|
||||
fi
|
||||
done
|
||||
echo "done: ${capture_method}" >"$status_file"
|
||||
}
|
||||
|
||||
run_bench() {
|
||||
local profile="$1"
|
||||
local profile_dir="$OUT_DIR/$profile"
|
||||
local bucket_profile="${profile//_/-}"
|
||||
local -a cmd=(
|
||||
"${PROJECT_ROOT}/scripts/run_object_batch_bench.sh"
|
||||
--tool warp
|
||||
--endpoint "$(all_endpoints_csv)"
|
||||
--access-key "$ACCESS_KEY"
|
||||
--secret-key "$SECRET_KEY"
|
||||
--region "$REGION"
|
||||
--auto-new-bucket
|
||||
--bucket-prefix "issue797-${bucket_profile}"
|
||||
--sizes "$SIZES"
|
||||
--concurrency "$CONCURRENCY"
|
||||
--duration "$DURATION"
|
||||
--warp-bin "$WARP_BIN"
|
||||
--warp-mode "$WARP_MODE"
|
||||
--out-dir "$profile_dir/bench"
|
||||
)
|
||||
|
||||
if [[ -n "$WARP_EXTRA_ARGS" ]]; then
|
||||
cmd+=(--extra-args "$WARP_EXTRA_ARGS")
|
||||
fi
|
||||
|
||||
print_redacted_command "${cmd[@]}" >"$profile_dir/bench/command.txt"
|
||||
log "${profile}: running warp benchmark"
|
||||
"${cmd[@]}" 2>&1 | tee "$profile_dir/bench/run.log"
|
||||
}
|
||||
|
||||
extract_tail_summary() {
|
||||
local profile="$1"
|
||||
local profile_dir="$OUT_DIR/$profile"
|
||||
local output="$profile_dir/bench/tail_latency_summary.txt"
|
||||
: >"$output"
|
||||
if ! command -v "$RG_BIN" >/dev/null 2>&1; then
|
||||
echo "rg not found; skipped" >"$output"
|
||||
return 0
|
||||
fi
|
||||
"$RG_BIN" -n 'Average:|Median:|90th:|99th:|Fastest:|Slowest:|StdDev:|Total:|Throughput by host|warp: <ERROR>' \
|
||||
"$profile_dir/bench"/*.log >"$output" || true
|
||||
}
|
||||
|
||||
write_run_meta() {
|
||||
local meta="$OUT_DIR/run-meta.txt"
|
||||
mkdir -p "$OUT_DIR"
|
||||
{
|
||||
echo "created_utc=$(date -u +%Y-%m-%dT%H:%M:%SZ)"
|
||||
echo "rustfs_bin=$RUSTFS_BIN"
|
||||
echo "base_port=$BASE_PORT"
|
||||
echo "node_count=$NODE_COUNT"
|
||||
echo "disks_per_node=$DISKS_PER_NODE"
|
||||
echo "sizes=$SIZES"
|
||||
echo "concurrency=$CONCURRENCY"
|
||||
echo "duration=$DURATION"
|
||||
echo "profiles=$PROFILES"
|
||||
echo "warp_extra_args=$WARP_EXTRA_ARGS"
|
||||
echo "endpoints=$(all_endpoints_csv)"
|
||||
echo "data_root=$DATA_ROOT"
|
||||
echo "keep_data=$KEEP_DATA"
|
||||
echo "capture_admin_metrics=$CAPTURE_ADMIN_METRICS"
|
||||
} >"$meta"
|
||||
}
|
||||
|
||||
run_profile() {
|
||||
local profile="$1"
|
||||
profile_is_supported "$profile" || die "unsupported profile: $profile"
|
||||
prepare_profile_layout "$profile"
|
||||
start_nodes "$profile"
|
||||
wait_for_health "$profile"
|
||||
capture_admin_metrics "$profile" before
|
||||
run_bench "$profile"
|
||||
capture_admin_metrics "$profile" after
|
||||
extract_tail_summary "$profile"
|
||||
stop_nodes
|
||||
}
|
||||
|
||||
main() {
|
||||
parse_args "$@"
|
||||
validate_args
|
||||
|
||||
write_run_meta
|
||||
|
||||
if [[ "$DRY_RUN" == "true" ]]; then
|
||||
log "dry run only"
|
||||
cat "$OUT_DIR/run-meta.txt"
|
||||
return 0
|
||||
fi
|
||||
|
||||
require_cmd "$CURL_BIN"
|
||||
require_cmd "$WARP_BIN"
|
||||
|
||||
if [[ "$BUILD_BIN" == "true" ]]; then
|
||||
log "building rustfs binary"
|
||||
cargo build -p rustfs --bin rustfs
|
||||
fi
|
||||
[[ -x "$RUSTFS_BIN" ]] || die "rustfs binary is not executable: $RUSTFS_BIN"
|
||||
|
||||
local profile
|
||||
IFS=',' read -r -a profile_arr <<<"$PROFILES"
|
||||
for profile in "${profile_arr[@]}"; do
|
||||
profile="${profile//[[:space:]]/}"
|
||||
[[ -z "$profile" ]] && continue
|
||||
run_profile "$profile"
|
||||
done
|
||||
|
||||
log "done. Output dir: $OUT_DIR"
|
||||
}
|
||||
|
||||
main "$@"
|
||||
Executable
+257
@@ -0,0 +1,257 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# Issue 2573 acceptance runner
|
||||
# Runs the key workload profiles discussed in docs/tasks/issue-2573/05-benchmark-and-acceptance.md
|
||||
# and samples the RustFS process RSS during load and cooldown.
|
||||
|
||||
WARP_BIN="${WARP_BIN:-warp}"
|
||||
HOST="${HOST:-http://127.0.0.1:9000}"
|
||||
ACCESS_KEY="${ACCESS_KEY:-rustfsadmin}"
|
||||
SECRET_KEY="${SECRET_KEY:-rustfsadmin}"
|
||||
BUCKET="${BUCKET:-rustfs-issue-2573}"
|
||||
REGION="${REGION:-us-east-1}"
|
||||
CONCURRENCY="${CONCURRENCY:-30}"
|
||||
DURATION="${DURATION:-60s}"
|
||||
COOLDOWN_SECS="${COOLDOWN_SECS:-180}"
|
||||
SAMPLE_SECS="${SAMPLE_SECS:-1}"
|
||||
RUSTFS_PID="${RUSTFS_PID:-}"
|
||||
OUT_DIR="${OUT_DIR:-target/bench/issue-2573-acceptance-$(date +%Y%m%d-%H%M%S)}"
|
||||
INSECURE="${INSECURE:-false}"
|
||||
|
||||
usage() {
|
||||
cat <<'USAGE'
|
||||
Usage:
|
||||
scripts/run_issue_2573_acceptance.sh [options]
|
||||
|
||||
Options:
|
||||
--warp-bin <path> warp binary (default: warp)
|
||||
--host <url> S3 endpoint; accepts either URL or host:port (default: http://127.0.0.1:9000)
|
||||
--access-key <ak> access key (default: rustfsadmin)
|
||||
--secret-key <sk> secret key (default: rustfsadmin)
|
||||
--bucket <name> bucket name (default: rustfs-issue-2573)
|
||||
--region <name> region (default: us-east-1)
|
||||
--concurrency <n> warp concurrency (default: 30)
|
||||
--duration <dur> warp duration per profile (default: 60s)
|
||||
--cooldown-secs <n> cooldown sampling after each profile (default: 180)
|
||||
--sample-secs <n> RSS sample interval seconds (default: 1)
|
||||
--pid <pid> rustfs process pid (optional; auto-detect if omitted)
|
||||
--out-dir <dir> output directory
|
||||
--insecure pass --insecure to warp
|
||||
-h, --help show help
|
||||
|
||||
Profiles executed:
|
||||
1. 4KiB mixed
|
||||
2. 11MiB mixed
|
||||
3. 11MiB delete
|
||||
USAGE
|
||||
}
|
||||
|
||||
require_cmd() {
|
||||
if ! command -v "$1" >/dev/null 2>&1; then
|
||||
echo "ERROR: command not found: $1" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
parse_args() {
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--warp-bin) WARP_BIN="$2"; shift 2 ;;
|
||||
--host) HOST="$2"; shift 2 ;;
|
||||
--access-key) ACCESS_KEY="$2"; shift 2 ;;
|
||||
--secret-key) SECRET_KEY="$2"; shift 2 ;;
|
||||
--bucket) BUCKET="$2"; shift 2 ;;
|
||||
--region) REGION="$2"; shift 2 ;;
|
||||
--concurrency) CONCURRENCY="$2"; shift 2 ;;
|
||||
--duration) DURATION="$2"; shift 2 ;;
|
||||
--cooldown-secs) COOLDOWN_SECS="$2"; shift 2 ;;
|
||||
--sample-secs) SAMPLE_SECS="$2"; shift 2 ;;
|
||||
--pid) RUSTFS_PID="$2"; shift 2 ;;
|
||||
--out-dir) OUT_DIR="$2"; shift 2 ;;
|
||||
--insecure) INSECURE=true; shift ;;
|
||||
-h|--help) usage; exit 0 ;;
|
||||
*)
|
||||
echo "ERROR: unknown arg: $1" >&2
|
||||
usage
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
}
|
||||
|
||||
resolve_pid() {
|
||||
if [[ -n "$RUSTFS_PID" ]]; then
|
||||
echo "$RUSTFS_PID"
|
||||
return
|
||||
fi
|
||||
|
||||
local pid
|
||||
pid="$(pgrep -n rustfs || true)"
|
||||
if [[ -z "$pid" ]]; then
|
||||
echo ""
|
||||
return
|
||||
fi
|
||||
echo "$pid"
|
||||
}
|
||||
|
||||
normalize_warp_host() {
|
||||
local raw="$1"
|
||||
# Strip scheme when a URL is provided.
|
||||
raw="${raw#http://}"
|
||||
raw="${raw#https://}"
|
||||
# Remove any path/query/fragment to satisfy warp's --host requirements.
|
||||
raw="${raw%%/*}"
|
||||
raw="${raw%%\?*}"
|
||||
raw="${raw%%\#*}"
|
||||
echo "$raw"
|
||||
}
|
||||
|
||||
sample_rss_loop() {
|
||||
local pid="$1"
|
||||
local out_file="$2"
|
||||
if [[ -z "$pid" ]]; then
|
||||
return 0
|
||||
fi
|
||||
local started_at
|
||||
started_at="$(date +%s)"
|
||||
|
||||
echo "timestamp,elapsed_seconds,rss_kib,vsz_kib" > "$out_file"
|
||||
while kill -0 "$pid" >/dev/null 2>&1; do
|
||||
local now elapsed sample
|
||||
now="$(date +%s)"
|
||||
elapsed="$((now - started_at))"
|
||||
sample="$(ps -o rss=,vsz= -p "$pid" | awk 'NF>=2 {print $1","$2}')"
|
||||
if [[ -n "$sample" ]]; then
|
||||
echo "$(date +%Y-%m-%dT%H:%M:%S),${elapsed},${sample}" >> "$out_file"
|
||||
fi
|
||||
sleep "$SAMPLE_SECS"
|
||||
done
|
||||
}
|
||||
|
||||
sample_rss_window() {
|
||||
local pid="$1"
|
||||
local seconds="$2"
|
||||
local out_file="$3"
|
||||
if [[ -z "$pid" ]]; then
|
||||
return 0
|
||||
fi
|
||||
local started_at deadline
|
||||
started_at="$(date +%s)"
|
||||
deadline="$((started_at + seconds))"
|
||||
|
||||
echo "timestamp,elapsed_seconds,rss_kib,vsz_kib" > "$out_file"
|
||||
while true; do
|
||||
local now elapsed sample
|
||||
now="$(date +%s)"
|
||||
if (( now > deadline )); then
|
||||
break
|
||||
fi
|
||||
elapsed="$((now - started_at))"
|
||||
if ! kill -0 "$pid" >/dev/null 2>&1; then
|
||||
break
|
||||
fi
|
||||
sample="$(ps -o rss=,vsz= -p "$pid" | awk 'NF>=2 {print $1","$2}')"
|
||||
if [[ -n "$sample" ]]; then
|
||||
echo "$(date +%Y-%m-%dT%H:%M:%S),${elapsed},${sample}" >> "$out_file"
|
||||
fi
|
||||
sleep "$SAMPLE_SECS"
|
||||
done
|
||||
}
|
||||
|
||||
run_profile() {
|
||||
local profile_name="$1"
|
||||
local mode="$2"
|
||||
local obj_size="$3"
|
||||
local pid="$4"
|
||||
local warp_host
|
||||
warp_host="$(normalize_warp_host "$HOST")"
|
||||
local benchdata="$OUT_DIR/${profile_name// /-}"
|
||||
local warp_log="$OUT_DIR/${profile_name// /-}.warp.log"
|
||||
local rss_during="$OUT_DIR/${profile_name// /-}.rss_during.csv"
|
||||
local rss_cooldown="$OUT_DIR/${profile_name// /-}.rss_cooldown.csv"
|
||||
|
||||
local -a cmd=(
|
||||
"$WARP_BIN" "$mode"
|
||||
"--host" "$warp_host"
|
||||
"--access-key" "$ACCESS_KEY"
|
||||
"--secret-key" "$SECRET_KEY"
|
||||
"--bucket" "$BUCKET"
|
||||
"--region" "$REGION"
|
||||
"--obj.size" "$obj_size"
|
||||
"--concurrent" "$CONCURRENCY"
|
||||
"--duration" "$DURATION"
|
||||
"--benchdata" "$benchdata"
|
||||
)
|
||||
if [[ "$INSECURE" == "true" ]]; then
|
||||
cmd+=("--insecure")
|
||||
fi
|
||||
|
||||
echo "==== Running profile: $profile_name ===="
|
||||
printf 'Command:'
|
||||
printf ' %q' "${cmd[@]}"
|
||||
printf '\n'
|
||||
|
||||
local sampler_pid=""
|
||||
if [[ -n "$pid" ]]; then
|
||||
sample_rss_loop "$pid" "$rss_during" &
|
||||
sampler_pid=$!
|
||||
else
|
||||
echo "WARN: rustfs pid unavailable; skipping RSS sampling for $profile_name" >&2
|
||||
fi
|
||||
if ! "${cmd[@]}" 2>&1 | tee "$warp_log"; then
|
||||
echo "ERROR: profile failed: $profile_name" >&2
|
||||
if [[ -n "$sampler_pid" ]]; then
|
||||
kill "$sampler_pid" >/dev/null 2>&1 || true
|
||||
wait "$sampler_pid" >/dev/null 2>&1 || true
|
||||
fi
|
||||
exit 1
|
||||
fi
|
||||
if [[ -n "$sampler_pid" ]]; then
|
||||
kill "$sampler_pid" >/dev/null 2>&1 || true
|
||||
wait "$sampler_pid" >/dev/null 2>&1 || true
|
||||
fi
|
||||
|
||||
echo "==== Cooldown sampling: $profile_name ($COOLDOWN_SECS s) ===="
|
||||
sample_rss_window "$pid" "$COOLDOWN_SECS" "$rss_cooldown"
|
||||
}
|
||||
|
||||
main() {
|
||||
parse_args "$@"
|
||||
require_cmd "$WARP_BIN"
|
||||
require_cmd awk
|
||||
require_cmd ps
|
||||
require_cmd pgrep
|
||||
require_cmd tee
|
||||
mkdir -p "$OUT_DIR"
|
||||
|
||||
local pid
|
||||
pid="$(resolve_pid)"
|
||||
local warp_host
|
||||
warp_host="$(normalize_warp_host "$HOST")"
|
||||
|
||||
echo "Output dir: $OUT_DIR"
|
||||
if [[ -n "$pid" ]]; then
|
||||
echo "RustFS pid: $pid"
|
||||
else
|
||||
echo "RustFS pid: auto-detect failed (continuing without RSS sampling)"
|
||||
fi
|
||||
echo "Host: $HOST"
|
||||
echo "Warp host: $warp_host"
|
||||
echo "Bucket: $BUCKET"
|
||||
echo "Profiles:"
|
||||
echo " - 4KiB mixed"
|
||||
echo " - 11MiB mixed"
|
||||
echo " - 11MiB delete"
|
||||
|
||||
run_profile "4KiB mixed" "mixed" "4KiB" "$pid"
|
||||
run_profile "11MiB mixed" "mixed" "11MiB" "$pid"
|
||||
run_profile "11MiB delete" "delete" "11MiB" "$pid"
|
||||
|
||||
echo
|
||||
echo "Acceptance run finished."
|
||||
echo "Artifacts:"
|
||||
find "$OUT_DIR" -maxdepth 1 -type f | sort
|
||||
}
|
||||
|
||||
main "$@"
|
||||
+290
@@ -0,0 +1,290 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
PROJECT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
LABEL="${LABEL:-issue-2941}"
|
||||
DURATION_SECS="${DURATION_SECS:-60}"
|
||||
PERF_FREQ="${PERF_FREQ:-99}"
|
||||
OUT_DIR="${OUT_DIR:-}"
|
||||
RUSTFS_PID="${RUSTFS_PID:-}"
|
||||
CONTAINER_NAME="${CONTAINER_NAME:-}"
|
||||
ENDPOINT="${ENDPOINT:-http://127.0.0.1:9000}"
|
||||
PERF_MODE="${PERF_MODE:-auto}" # auto|on|off
|
||||
SUDO_CMD="${SUDO_CMD:-}" # example: sudo
|
||||
|
||||
usage() {
|
||||
cat <<'USAGE'
|
||||
Usage:
|
||||
scripts/run_issue_2941_perf_capture.sh [options]
|
||||
|
||||
Options:
|
||||
--label <name> artifact label prefix
|
||||
--duration <secs> sample duration in seconds (default: 60)
|
||||
--out-dir <dir> artifact output directory
|
||||
--pid <pid> rustfs pid; auto-detect if omitted
|
||||
--container <name> docker container name/id for extra stats
|
||||
--endpoint <url> rustfs endpoint for health probes (default: http://127.0.0.1:9000)
|
||||
--perf <auto|on|off> whether to run perf record (default: auto)
|
||||
--perf-freq <hz> perf sample frequency (default: 99)
|
||||
--sudo-cmd <cmd> optional prefix for privileged perf, e.g. "sudo"
|
||||
-h, --help show help
|
||||
|
||||
Environment:
|
||||
LABEL
|
||||
DURATION_SECS
|
||||
PERF_FREQ
|
||||
OUT_DIR
|
||||
RUSTFS_PID
|
||||
CONTAINER_NAME
|
||||
ENDPOINT
|
||||
PERF_MODE
|
||||
SUDO_CMD
|
||||
|
||||
Examples:
|
||||
scripts/run_issue_2941_perf_capture.sh --label musl-baseline --container rustfs
|
||||
scripts/run_issue_2941_perf_capture.sh --label glibc-test --pid 12345 --perf on --sudo-cmd sudo
|
||||
USAGE
|
||||
}
|
||||
|
||||
log() {
|
||||
printf '[INFO] %s\n' "$*"
|
||||
}
|
||||
|
||||
warn() {
|
||||
printf '[WARN] %s\n' "$*" >&2
|
||||
}
|
||||
|
||||
require_arg() {
|
||||
local option="$1"
|
||||
local value="${2-}"
|
||||
if [[ $# -lt 2 || -z "${value}" || "${value}" == --* ]]; then
|
||||
warn "missing value for ${option}"
|
||||
usage
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
parse_args() {
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--label) require_arg "$1" "${2-}"; LABEL="$2"; shift 2 ;;
|
||||
--duration) require_arg "$1" "${2-}"; DURATION_SECS="$2"; shift 2 ;;
|
||||
--out-dir) require_arg "$1" "${2-}"; OUT_DIR="$2"; shift 2 ;;
|
||||
--pid) require_arg "$1" "${2-}"; RUSTFS_PID="$2"; shift 2 ;;
|
||||
--container) require_arg "$1" "${2-}"; CONTAINER_NAME="$2"; shift 2 ;;
|
||||
--endpoint) require_arg "$1" "${2-}"; ENDPOINT="$2"; shift 2 ;;
|
||||
--perf) require_arg "$1" "${2-}"; PERF_MODE="$2"; shift 2 ;;
|
||||
--perf-freq) require_arg "$1" "${2-}"; PERF_FREQ="$2"; shift 2 ;;
|
||||
--sudo-cmd) require_arg "$1" "${2-}"; SUDO_CMD="$2"; shift 2 ;;
|
||||
-h|--help) usage; exit 0 ;;
|
||||
*)
|
||||
warn "unknown argument: $1"
|
||||
usage
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
}
|
||||
|
||||
finalize_defaults() {
|
||||
if [[ -z "${OUT_DIR}" ]]; then
|
||||
OUT_DIR="${PROJECT_ROOT}/target/perf/${LABEL}-$(date +%Y%m%d-%H%M%S)}"
|
||||
fi
|
||||
}
|
||||
|
||||
command_exists() {
|
||||
command -v "$1" >/dev/null 2>&1
|
||||
}
|
||||
|
||||
write_cmd_output() {
|
||||
local out_file="$1"
|
||||
shift
|
||||
if "$@" >"$out_file" 2>&1; then
|
||||
return 0
|
||||
fi
|
||||
warn "command failed, see ${out_file}"
|
||||
return 1
|
||||
}
|
||||
|
||||
resolve_pid() {
|
||||
if [[ -n "${RUSTFS_PID}" ]]; then
|
||||
printf '%s\n' "${RUSTFS_PID}"
|
||||
return
|
||||
fi
|
||||
|
||||
if [[ -n "${CONTAINER_NAME}" ]] && command_exists docker; then
|
||||
local pid
|
||||
pid="$(docker inspect --format '{{.State.Pid}}' "${CONTAINER_NAME}" 2>/dev/null || true)"
|
||||
if [[ -n "${pid}" && "${pid}" != "0" ]]; then
|
||||
printf '%s\n' "${pid}"
|
||||
return
|
||||
fi
|
||||
fi
|
||||
|
||||
pgrep -n rustfs || true
|
||||
}
|
||||
|
||||
snapshot_proc() {
|
||||
local pid="$1"
|
||||
local prefix="$2"
|
||||
[[ -n "${pid}" ]] || return 0
|
||||
|
||||
[[ -r "/proc/${pid}/status" ]] && cp "/proc/${pid}/status" "${OUT_DIR}/${prefix}.proc-status.txt" || true
|
||||
[[ -r "/proc/${pid}/io" ]] && cp "/proc/${pid}/io" "${OUT_DIR}/${prefix}.proc-io.txt" || true
|
||||
[[ -r "/proc/${pid}/sched" ]] && cp "/proc/${pid}/sched" "${OUT_DIR}/${prefix}.proc-sched.txt" || true
|
||||
[[ -r "/proc/${pid}/smaps_rollup" ]] && cp "/proc/${pid}/smaps_rollup" "${OUT_DIR}/${prefix}.proc-smaps-rollup.txt" || true
|
||||
[[ -r "/proc/${pid}/limits" ]] && cp "/proc/${pid}/limits" "${OUT_DIR}/${prefix}.proc-limits.txt" || true
|
||||
|
||||
if command_exists ps; then
|
||||
ps -p "${pid}" -o pid,ppid,stat,pcpu,pmem,rss,vsz,etime,args >"${OUT_DIR}/${prefix}.ps.txt" 2>&1 || true
|
||||
ps -L -p "${pid}" -o pid,tid,psr,pcpu,stat,wchan:32,comm >"${OUT_DIR}/${prefix}.threads.txt" 2>&1 || true
|
||||
fi
|
||||
|
||||
if command_exists top; then
|
||||
if [[ "$(uname -s)" == "Linux" ]]; then
|
||||
top -H -b -n 1 -p "${pid}" >"${OUT_DIR}/${prefix}.top.txt" 2>&1 || true
|
||||
else
|
||||
top -l 1 -pid "${pid}" >"${OUT_DIR}/${prefix}.top.txt" 2>&1 || true
|
||||
fi
|
||||
fi
|
||||
}
|
||||
|
||||
capture_host_info() {
|
||||
write_cmd_output "${OUT_DIR}/uname.txt" uname -a || true
|
||||
command_exists lscpu && write_cmd_output "${OUT_DIR}/lscpu.txt" lscpu || true
|
||||
command_exists free && write_cmd_output "${OUT_DIR}/free.txt" free -h || true
|
||||
command_exists df && write_cmd_output "${OUT_DIR}/df.txt" df -h || true
|
||||
command_exists mount && write_cmd_output "${OUT_DIR}/mount.txt" mount || true
|
||||
}
|
||||
|
||||
capture_endpoint_info() {
|
||||
if command_exists curl; then
|
||||
curl -fsS "${ENDPOINT}/health" >"${OUT_DIR}/health.txt" 2>&1 || true
|
||||
curl -fsS "${ENDPOINT}/health/ready" >"${OUT_DIR}/health-ready.txt" 2>&1 || true
|
||||
fi
|
||||
}
|
||||
|
||||
capture_container_info() {
|
||||
[[ -n "${CONTAINER_NAME}" ]] || return 0
|
||||
command_exists docker || return 0
|
||||
|
||||
docker inspect "${CONTAINER_NAME}" >"${OUT_DIR}/docker-inspect.json" 2>&1 || true
|
||||
docker logs --tail 500 "${CONTAINER_NAME}" >"${OUT_DIR}/docker-logs-tail.txt" 2>&1 || true
|
||||
docker stats --no-stream --format '{{json .}}' "${CONTAINER_NAME}" >"${OUT_DIR}/docker-stats-once.jsonl" 2>&1 || true
|
||||
}
|
||||
|
||||
sample_container_stats_loop() {
|
||||
[[ -n "${CONTAINER_NAME}" ]] || return 0
|
||||
command_exists docker || return 0
|
||||
|
||||
local out_file="${OUT_DIR}/docker-stats-loop.jsonl"
|
||||
: >"${out_file}"
|
||||
local end_ts=$((SECONDS + DURATION_SECS))
|
||||
while (( SECONDS < end_ts )); do
|
||||
docker stats --no-stream --format '{{json .}}' "${CONTAINER_NAME}" >>"${out_file}" 2>/dev/null || true
|
||||
sleep 1
|
||||
done
|
||||
}
|
||||
|
||||
sample_pidstat() {
|
||||
local pid="$1"
|
||||
[[ -n "${pid}" ]] || return 0
|
||||
command_exists pidstat || {
|
||||
echo "pidstat unavailable" >"${OUT_DIR}/pidstat.txt"
|
||||
return 0
|
||||
}
|
||||
|
||||
pidstat -durwh -p "${pid}" 1 "${DURATION_SECS}" >"${OUT_DIR}/pidstat.txt" 2>&1 || true
|
||||
}
|
||||
|
||||
sample_perf() {
|
||||
local pid="$1"
|
||||
[[ -n "${pid}" ]] || return 0
|
||||
[[ "${PERF_MODE}" == "off" ]] && return 0
|
||||
command_exists perf || {
|
||||
echo "perf unavailable" >"${OUT_DIR}/perf-record.log"
|
||||
[[ "${PERF_MODE}" == "on" ]] && warn "perf requested but not installed"
|
||||
return 0
|
||||
}
|
||||
|
||||
local perf_data="${OUT_DIR}/perf.data"
|
||||
local perf_log="${OUT_DIR}/perf-record.log"
|
||||
local perf_report="${OUT_DIR}/perf-report.txt"
|
||||
|
||||
local -a prefix=()
|
||||
if [[ -n "${SUDO_CMD}" ]]; then
|
||||
read -r -a prefix <<<"${SUDO_CMD}"
|
||||
fi
|
||||
|
||||
if "${prefix[@]}" perf record -F "${PERF_FREQ}" -g -p "${pid}" -o "${perf_data}" -- sleep "${DURATION_SECS}" \
|
||||
>"${perf_log}" 2>&1; then
|
||||
"${prefix[@]}" perf report --stdio -i "${perf_data}" >"${perf_report}" 2>&1 || true
|
||||
else
|
||||
if [[ "${PERF_MODE}" == "on" ]]; then
|
||||
warn "perf record failed; see ${perf_log}"
|
||||
fi
|
||||
fi
|
||||
}
|
||||
|
||||
capture_version_info() {
|
||||
local pid="$1"
|
||||
if [[ -n "${pid}" && -x "/proc/${pid}/exe" ]]; then
|
||||
readlink "/proc/${pid}/exe" >"${OUT_DIR}/binary-path.txt" 2>&1 || true
|
||||
"/proc/${pid}/exe" --help >"${OUT_DIR}/binary-help.txt" 2>&1 || true
|
||||
fi
|
||||
}
|
||||
|
||||
main() {
|
||||
parse_args "$@"
|
||||
finalize_defaults
|
||||
mkdir -p "${OUT_DIR}"
|
||||
|
||||
local pid
|
||||
pid="$(resolve_pid)"
|
||||
if [[ -z "${pid}" ]]; then
|
||||
warn "failed to detect rustfs pid automatically"
|
||||
else
|
||||
log "using rustfs pid=${pid}"
|
||||
fi
|
||||
|
||||
cat >"${OUT_DIR}/capture-meta.txt" <<EOF
|
||||
label=${LABEL}
|
||||
duration_secs=${DURATION_SECS}
|
||||
perf_freq=${PERF_FREQ}
|
||||
endpoint=${ENDPOINT}
|
||||
container_name=${CONTAINER_NAME}
|
||||
rustfs_pid=${pid}
|
||||
perf_mode=${PERF_MODE}
|
||||
sudo_cmd=${SUDO_CMD}
|
||||
started_at=$(date -u +%Y-%m-%dT%H:%M:%SZ)
|
||||
project_root=${PROJECT_ROOT}
|
||||
git_branch=$(git -C "${PROJECT_ROOT}" branch --show-current 2>/dev/null || true)
|
||||
git_head=$(git -C "${PROJECT_ROOT}" rev-parse HEAD 2>/dev/null || true)
|
||||
EOF
|
||||
|
||||
capture_host_info
|
||||
capture_endpoint_info
|
||||
capture_container_info
|
||||
capture_version_info "${pid}"
|
||||
snapshot_proc "${pid}" "start"
|
||||
|
||||
local bg_pids=()
|
||||
sample_pidstat "${pid}" &
|
||||
bg_pids+=($!)
|
||||
sample_container_stats_loop &
|
||||
bg_pids+=($!)
|
||||
sample_perf "${pid}" &
|
||||
bg_pids+=($!)
|
||||
|
||||
for bg_pid in "${bg_pids[@]}"; do
|
||||
wait "${bg_pid}" || true
|
||||
done
|
||||
|
||||
snapshot_proc "${pid}" "end"
|
||||
capture_endpoint_info
|
||||
|
||||
log "issue-2941 perf capture artifacts written to ${OUT_DIR}"
|
||||
find "${OUT_DIR}" -maxdepth 1 -type f | sort
|
||||
}
|
||||
|
||||
main "$@"
|
||||
+1324
File diff suppressed because it is too large
Load Diff
+403
@@ -0,0 +1,403 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# One-click controller:
|
||||
# - Switches RUSTFS_CAPACITY_* and RUSTFS_OBJECT_* by profile A/B/C
|
||||
# - Calls scripts/run_object_batch_bench_enhanced.sh for each profile
|
||||
# - Supports optional "apply command" hook to reload/restart RustFS per profile
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ENHANCED_SCRIPT="$SCRIPT_DIR/run_object_batch_bench_enhanced.sh"
|
||||
|
||||
GROUP="all" # all|A|B|C
|
||||
ENDPOINT=""
|
||||
ACCESS_KEY=""
|
||||
SECRET_KEY=""
|
||||
BUCKET="rustfs-bench"
|
||||
REGION="us-east-1"
|
||||
TOOL="warp"
|
||||
CONCURRENCY=128
|
||||
ROUNDS=3
|
||||
RETRY_PER_ROUND=2
|
||||
RETRY_SLEEP_SECS=2
|
||||
INSECURE=false
|
||||
DRY_RUN=false
|
||||
OUT_ROOT=""
|
||||
BASELINE_ROOT=""
|
||||
|
||||
# tool-specific
|
||||
WARP_BIN="warp"
|
||||
WARP_MODE="mixed"
|
||||
DURATION="60s"
|
||||
S3BENCH_BIN="s3bench"
|
||||
SAMPLES=20000
|
||||
|
||||
# optional hooks
|
||||
APPLY_CMD=""
|
||||
APPLY_CMD_ARR=()
|
||||
APPLY_WAIT_SECS=20
|
||||
|
||||
EXTRA_ARGS=()
|
||||
|
||||
usage() {
|
||||
cat <<'USAGE'
|
||||
Usage:
|
||||
scripts/run_object_batch_bench_abc.sh \
|
||||
--tool <warp|s3bench> --endpoint <url> --access-key <ak> --secret-key <sk> [options]
|
||||
|
||||
Required:
|
||||
--tool warp | s3bench
|
||||
--endpoint S3 endpoint
|
||||
--access-key S3 access key
|
||||
--secret-key S3 secret key
|
||||
|
||||
Core options:
|
||||
--group all|A|B|C (default: all)
|
||||
--bucket Bucket name (default: rustfs-bench)
|
||||
--region Region (default: us-east-1)
|
||||
--concurrency Default 128
|
||||
--rounds Default 3
|
||||
--retry-per-round Default 2
|
||||
--retry-sleep-secs Default 2
|
||||
--out-root Default target/bench/object-batch-abc-<timestamp>
|
||||
--baseline-root If set, use <baseline-root>/<group>/median_summary.csv
|
||||
--insecure Allow insecure TLS
|
||||
--dry-run Print commands without execution
|
||||
|
||||
Warp options:
|
||||
--warp-bin Default: warp
|
||||
--warp-mode get|put|mixed (default: mixed)
|
||||
--duration Default: 60s
|
||||
|
||||
s3bench options:
|
||||
--s3bench-bin Default: s3bench
|
||||
--samples Default: 20000
|
||||
|
||||
Hooks:
|
||||
--apply-cmd Optional command to apply/restart RustFS after profile env switch.
|
||||
Executed directly (no shell eval), e.g. "bash scripts/restart.sh"
|
||||
--apply-wait-secs Wait time after apply cmd (default: 20)
|
||||
|
||||
Extra:
|
||||
--extra-args Extra args passed to enhanced script, quoted as one string
|
||||
-h, --help Show this help
|
||||
|
||||
Examples:
|
||||
scripts/run_object_batch_bench_abc.sh \
|
||||
--tool warp --endpoint http://127.0.0.1:9000 \
|
||||
--access-key minioadmin --secret-key minioadmin \
|
||||
--bucket bench-obj --group all --duration 90s
|
||||
|
||||
scripts/run_object_batch_bench_abc.sh \
|
||||
--tool s3bench --endpoint http://127.0.0.1:9000 \
|
||||
--access-key minioadmin --secret-key minioadmin \
|
||||
--group B --samples 50000 --apply-cmd "bash scripts/run.capacity-object.lab.sh"
|
||||
USAGE
|
||||
}
|
||||
|
||||
require_cmd() {
|
||||
if ! command -v "$1" >/dev/null 2>&1; then
|
||||
echo "ERROR: command not found: $1" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
validate_positive_int() {
|
||||
local v="$1"
|
||||
local n="$2"
|
||||
if ! [[ "$v" =~ ^[0-9]+$ ]] || [[ "$v" -le 0 ]]; then
|
||||
echo "ERROR: $n must be a positive integer, got: $v" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
parse_apply_cmd() {
|
||||
local raw="$1"
|
||||
|
||||
if [[ "$raw" == *';'* || "$raw" == *'&&'* || "$raw" == *'||'* || "$raw" == *'|'* || "$raw" == *'<'* || "$raw" == *'>'* || "$raw" == *'`'* || "$raw" == *'$'* ]]; then
|
||||
echo "ERROR: --apply-cmd does not allow shell operators or expansions; pass a plain command and args only" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
IFS=$' \t\n' read -r -a APPLY_CMD_ARR <<< "$raw"
|
||||
if [[ "${#APPLY_CMD_ARR[@]}" -eq 0 ]]; then
|
||||
echo "ERROR: --apply-cmd must not be empty" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
require_cmd "${APPLY_CMD_ARR[0]}"
|
||||
}
|
||||
|
||||
parse_args() {
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--tool) TOOL="$2"; shift 2 ;;
|
||||
--endpoint) ENDPOINT="$2"; shift 2 ;;
|
||||
--access-key) ACCESS_KEY="$2"; shift 2 ;;
|
||||
--secret-key) SECRET_KEY="$2"; shift 2 ;;
|
||||
--group) GROUP="$2"; shift 2 ;;
|
||||
--bucket) BUCKET="$2"; shift 2 ;;
|
||||
--region) REGION="$2"; shift 2 ;;
|
||||
--concurrency) CONCURRENCY="$2"; shift 2 ;;
|
||||
--rounds) ROUNDS="$2"; shift 2 ;;
|
||||
--retry-per-round) RETRY_PER_ROUND="$2"; shift 2 ;;
|
||||
--retry-sleep-secs) RETRY_SLEEP_SECS="$2"; shift 2 ;;
|
||||
--out-root) OUT_ROOT="$2"; shift 2 ;;
|
||||
--baseline-root) BASELINE_ROOT="$2"; shift 2 ;;
|
||||
--insecure) INSECURE=true; shift ;;
|
||||
--dry-run) DRY_RUN=true; shift ;;
|
||||
--warp-bin) WARP_BIN="$2"; shift 2 ;;
|
||||
--warp-mode) WARP_MODE="$2"; shift 2 ;;
|
||||
--duration) DURATION="$2"; shift 2 ;;
|
||||
--s3bench-bin) S3BENCH_BIN="$2"; shift 2 ;;
|
||||
--samples) SAMPLES="$2"; shift 2 ;;
|
||||
--apply-cmd) APPLY_CMD="$2"; shift 2 ;;
|
||||
--apply-wait-secs) APPLY_WAIT_SECS="$2"; shift 2 ;;
|
||||
--extra-args)
|
||||
# shellcheck disable=SC2206
|
||||
EXTRA_ARGS=($2)
|
||||
shift 2
|
||||
;;
|
||||
-h|--help) usage; exit 0 ;;
|
||||
*)
|
||||
echo "ERROR: unknown arg: $1" >&2
|
||||
usage
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
}
|
||||
|
||||
validate_args() {
|
||||
if [[ "$TOOL" != "warp" && "$TOOL" != "s3bench" ]]; then
|
||||
echo "ERROR: --tool must be warp or s3bench" >&2
|
||||
exit 1
|
||||
fi
|
||||
case "$GROUP" in
|
||||
all|A|B|C) ;;
|
||||
*) echo "ERROR: --group must be all|A|B|C" >&2; exit 1 ;;
|
||||
esac
|
||||
if [[ -z "$ENDPOINT" || -z "$ACCESS_KEY" || -z "$SECRET_KEY" ]]; then
|
||||
echo "ERROR: --endpoint/--access-key/--secret-key are required" >&2
|
||||
exit 1
|
||||
fi
|
||||
validate_positive_int "$CONCURRENCY" "--concurrency"
|
||||
validate_positive_int "$ROUNDS" "--rounds"
|
||||
validate_positive_int "$RETRY_PER_ROUND" "--retry-per-round"
|
||||
validate_positive_int "$RETRY_SLEEP_SECS" "--retry-sleep-secs"
|
||||
validate_positive_int "$APPLY_WAIT_SECS" "--apply-wait-secs"
|
||||
if [[ "$TOOL" == "s3bench" ]]; then
|
||||
validate_positive_int "$SAMPLES" "--samples"
|
||||
fi
|
||||
if [[ -n "$APPLY_CMD" ]]; then
|
||||
parse_apply_cmd "$APPLY_CMD"
|
||||
fi
|
||||
}
|
||||
|
||||
setup_out_root() {
|
||||
if [[ -z "$OUT_ROOT" ]]; then
|
||||
OUT_ROOT="target/bench/object-batch-abc-$(date +%Y%m%d-%H%M%S)"
|
||||
fi
|
||||
mkdir -p "$OUT_ROOT"
|
||||
}
|
||||
|
||||
apply_capacity_common() {
|
||||
export RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
|
||||
export RUSTFS_CAPACITY_WRITE_TRIGGER_DELAY=8
|
||||
export RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=14
|
||||
export RUSTFS_CAPACITY_FAST_UPDATE_THRESHOLD=45
|
||||
export RUSTFS_CAPACITY_MAX_FILES_THRESHOLD=1000000
|
||||
export RUSTFS_CAPACITY_STAT_TIMEOUT=5
|
||||
export RUSTFS_CAPACITY_SAMPLE_RATE=100
|
||||
export RUSTFS_CAPACITY_METRICS_INTERVAL=120
|
||||
}
|
||||
|
||||
apply_object_profile_A() {
|
||||
export RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=128
|
||||
export RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=2097152
|
||||
export RUSTFS_OBJECT_GET_TIMEOUT=18
|
||||
export RUSTFS_OBJECT_DISK_READ_TIMEOUT=6
|
||||
export RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=4
|
||||
export RUSTFS_OBJECT_PRIORITY_SCHEDULING_ENABLE=true
|
||||
export RUSTFS_OBJECT_LOCK_OPTIMIZATION_ENABLE=true
|
||||
export RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=12
|
||||
export RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=6
|
||||
}
|
||||
|
||||
apply_object_profile_B() {
|
||||
export RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=112
|
||||
export RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=4194304
|
||||
export RUSTFS_OBJECT_GET_TIMEOUT=30
|
||||
export RUSTFS_OBJECT_DISK_READ_TIMEOUT=10
|
||||
export RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=5
|
||||
export RUSTFS_OBJECT_PRIORITY_SCHEDULING_ENABLE=true
|
||||
export RUSTFS_OBJECT_LOCK_OPTIMIZATION_ENABLE=true
|
||||
export RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=12
|
||||
export RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=6
|
||||
}
|
||||
|
||||
apply_object_profile_C() {
|
||||
export RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=72
|
||||
export RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
|
||||
export RUSTFS_OBJECT_GET_TIMEOUT=50
|
||||
export RUSTFS_OBJECT_DISK_READ_TIMEOUT=14
|
||||
export RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=6
|
||||
export RUSTFS_OBJECT_PRIORITY_SCHEDULING_ENABLE=true
|
||||
export RUSTFS_OBJECT_LOCK_OPTIMIZATION_ENABLE=true
|
||||
export RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=12
|
||||
export RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=6
|
||||
}
|
||||
|
||||
sizes_for_group() {
|
||||
case "$1" in
|
||||
A) echo "1KiB,4KiB,8KiB,16KiB,32KiB,100KiB" ;;
|
||||
B) echo "100KiB,512KiB,1MiB,2MiB" ;;
|
||||
C) echo "2MiB,5MiB,10MiB" ;;
|
||||
*) echo "" ;;
|
||||
esac
|
||||
}
|
||||
|
||||
run_apply_hook_if_needed() {
|
||||
local group="$1"
|
||||
if [[ "${#APPLY_CMD_ARR[@]}" -eq 0 ]]; then
|
||||
return
|
||||
fi
|
||||
echo "[${group}] running apply command..."
|
||||
if [[ "$DRY_RUN" == "true" ]]; then
|
||||
printf '[DRY-RUN] '
|
||||
printf '%q ' "${APPLY_CMD_ARR[@]}"
|
||||
printf '\n'
|
||||
echo "[DRY-RUN] sleep $APPLY_WAIT_SECS"
|
||||
else
|
||||
"${APPLY_CMD_ARR[@]}"
|
||||
echo "[${group}] waiting ${APPLY_WAIT_SECS}s for service readiness..."
|
||||
sleep "$APPLY_WAIT_SECS"
|
||||
fi
|
||||
}
|
||||
|
||||
write_env_snapshot() {
|
||||
local out_file="$1"
|
||||
cat > "$out_file" <<EOF
|
||||
RUSTFS_CAPACITY_SCHEDULED_INTERVAL=${RUSTFS_CAPACITY_SCHEDULED_INTERVAL}
|
||||
RUSTFS_CAPACITY_WRITE_TRIGGER_DELAY=${RUSTFS_CAPACITY_WRITE_TRIGGER_DELAY}
|
||||
RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=${RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD}
|
||||
RUSTFS_CAPACITY_FAST_UPDATE_THRESHOLD=${RUSTFS_CAPACITY_FAST_UPDATE_THRESHOLD}
|
||||
RUSTFS_CAPACITY_MAX_FILES_THRESHOLD=${RUSTFS_CAPACITY_MAX_FILES_THRESHOLD}
|
||||
RUSTFS_CAPACITY_STAT_TIMEOUT=${RUSTFS_CAPACITY_STAT_TIMEOUT}
|
||||
RUSTFS_CAPACITY_SAMPLE_RATE=${RUSTFS_CAPACITY_SAMPLE_RATE}
|
||||
RUSTFS_CAPACITY_METRICS_INTERVAL=${RUSTFS_CAPACITY_METRICS_INTERVAL}
|
||||
RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=${RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS}
|
||||
RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=${RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE}
|
||||
RUSTFS_OBJECT_GET_TIMEOUT=${RUSTFS_OBJECT_GET_TIMEOUT}
|
||||
RUSTFS_OBJECT_DISK_READ_TIMEOUT=${RUSTFS_OBJECT_DISK_READ_TIMEOUT}
|
||||
RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT}
|
||||
RUSTFS_OBJECT_PRIORITY_SCHEDULING_ENABLE=${RUSTFS_OBJECT_PRIORITY_SCHEDULING_ENABLE}
|
||||
RUSTFS_OBJECT_LOCK_OPTIMIZATION_ENABLE=${RUSTFS_OBJECT_LOCK_OPTIMIZATION_ENABLE}
|
||||
RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD}
|
||||
RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD}
|
||||
EOF
|
||||
}
|
||||
|
||||
run_group() {
|
||||
local g="$1"
|
||||
local sizes out_dir baseline_csv
|
||||
|
||||
apply_capacity_common
|
||||
case "$g" in
|
||||
A) apply_object_profile_A ;;
|
||||
B) apply_object_profile_B ;;
|
||||
C) apply_object_profile_C ;;
|
||||
*) echo "ERROR: unsupported group $g" >&2; exit 1 ;;
|
||||
esac
|
||||
|
||||
sizes="$(sizes_for_group "$g")"
|
||||
out_dir="$OUT_ROOT/$g"
|
||||
mkdir -p "$out_dir"
|
||||
write_env_snapshot "$out_dir/env_snapshot.env"
|
||||
|
||||
run_apply_hook_if_needed "$g"
|
||||
|
||||
baseline_csv=""
|
||||
if [[ -n "$BASELINE_ROOT" && -f "$BASELINE_ROOT/$g/median_summary.csv" ]]; then
|
||||
baseline_csv="$BASELINE_ROOT/$g/median_summary.csv"
|
||||
fi
|
||||
|
||||
local cmd=(
|
||||
"$ENHANCED_SCRIPT"
|
||||
"--tool" "$TOOL"
|
||||
"--endpoint" "$ENDPOINT"
|
||||
"--access-key" "$ACCESS_KEY"
|
||||
"--secret-key" "$SECRET_KEY"
|
||||
"--bucket" "$BUCKET"
|
||||
"--region" "$REGION"
|
||||
"--concurrency" "$CONCURRENCY"
|
||||
"--sizes" "$sizes"
|
||||
"--rounds" "$ROUNDS"
|
||||
"--retry-per-round" "$RETRY_PER_ROUND"
|
||||
"--retry-sleep-secs" "$RETRY_SLEEP_SECS"
|
||||
"--out-dir" "$out_dir"
|
||||
)
|
||||
|
||||
if [[ -n "$baseline_csv" ]]; then
|
||||
cmd+=("--baseline-csv" "$baseline_csv")
|
||||
fi
|
||||
if [[ "$INSECURE" == "true" ]]; then
|
||||
cmd+=("--insecure")
|
||||
fi
|
||||
if [[ "$DRY_RUN" == "true" ]]; then
|
||||
cmd+=("--dry-run")
|
||||
fi
|
||||
|
||||
if [[ "$TOOL" == "warp" ]]; then
|
||||
cmd+=("--warp-bin" "$WARP_BIN" "--warp-mode" "$WARP_MODE" "--duration" "$DURATION")
|
||||
else
|
||||
cmd+=("--s3bench-bin" "$S3BENCH_BIN" "--samples" "$SAMPLES")
|
||||
fi
|
||||
if [[ "${#EXTRA_ARGS[@]}" -gt 0 ]]; then
|
||||
local joined
|
||||
joined="$(printf '%s ' "${EXTRA_ARGS[@]}" | sed 's/[[:space:]]*$//')"
|
||||
cmd+=("--extra-args" "$joined")
|
||||
fi
|
||||
|
||||
echo
|
||||
echo "===== Running group ${g} ====="
|
||||
echo "Sizes: $sizes"
|
||||
echo "Output: $out_dir"
|
||||
if [[ "$DRY_RUN" == "true" ]]; then
|
||||
printf '[DRY-RUN] %q ' "${cmd[@]}"
|
||||
printf '\n'
|
||||
else
|
||||
"${cmd[@]}"
|
||||
fi
|
||||
}
|
||||
|
||||
main() {
|
||||
parse_args "$@"
|
||||
validate_args
|
||||
require_cmd awk
|
||||
require_cmd sed
|
||||
if [[ ! -x "$ENHANCED_SCRIPT" ]]; then
|
||||
echo "ERROR: enhanced script missing or not executable: $ENHANCED_SCRIPT" >&2
|
||||
exit 1
|
||||
fi
|
||||
setup_out_root
|
||||
|
||||
echo "Controller output root: $OUT_ROOT"
|
||||
echo "Tool=$TOOL Group=$GROUP Concurrency=$CONCURRENCY Rounds=$ROUNDS"
|
||||
|
||||
case "$GROUP" in
|
||||
all)
|
||||
run_group A
|
||||
run_group B
|
||||
run_group C
|
||||
;;
|
||||
A|B|C)
|
||||
run_group "$GROUP"
|
||||
;;
|
||||
esac
|
||||
|
||||
echo
|
||||
echo "Done. Group outputs are under: $OUT_ROOT"
|
||||
}
|
||||
|
||||
main "$@"
|
||||
+1458
File diff suppressed because it is too large
Load Diff
+601
@@ -0,0 +1,601 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# Large-object PUT stage-breakdown runner for rustfs/backlog#706.
|
||||
# This wrapper standardizes:
|
||||
# - object sizes and concurrency matrix for 16MiB / 32MiB PUT runs
|
||||
# - output directory layout under target/bench/
|
||||
# - artifact naming and run manifest capture
|
||||
# - optional baseline comparison reuse across per-concurrency runs
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
|
||||
ENHANCED_BENCH_SCRIPT="${PROJECT_ROOT}/scripts/run_object_batch_bench_enhanced.sh"
|
||||
SERVICE_METRICS_DELTA_SCRIPT="${PROJECT_ROOT}/scripts/analyze_put_service_metrics_deltas.py"
|
||||
|
||||
TOOL="warp"
|
||||
ENDPOINT=""
|
||||
ACCESS_KEY=""
|
||||
SECRET_KEY=""
|
||||
REGION="us-east-1"
|
||||
BUCKET_PREFIX="rustfs-put-large"
|
||||
CONCURRENCIES="16,32,64,96,128"
|
||||
SIZES="16MiB,32MiB"
|
||||
DURATION="120s"
|
||||
ROUNDS=3
|
||||
RETRY_PER_ROUND=1
|
||||
RETRY_SLEEP_SECS=2
|
||||
COOLDOWN_SECS=20
|
||||
WARP_BIN="${WARP_BIN:-warp}"
|
||||
OUT_DIR=""
|
||||
BASELINE_ROOT=""
|
||||
EXTRA_ARGS=""
|
||||
INSECURE=false
|
||||
DRY_RUN=false
|
||||
PYTHON_BIN="python3"
|
||||
SERVICE_METRICS_URL=""
|
||||
SERVICE_PROMETHEUS_QUERY_URL=""
|
||||
SERVICE_PROMETHEUS_QUERY=""
|
||||
SERVICE_METRICS_SERVICE_NAME=""
|
||||
SERVICE_METRICS_FILTER_REGEX=""
|
||||
SERVICE_METRICS_CAPTURE_ATTEMPTS=3
|
||||
SERVICE_METRICS_CAPTURE_RETRY_SECS=1
|
||||
SERVICE_METRICS_CONNECT_TIMEOUT_SECS=2
|
||||
SERVICE_METRICS_MAX_TIME_SECS=15
|
||||
SERVICE_METRICS_SETTLE_SECS=0
|
||||
|
||||
TOPOLOGY_NODES=""
|
||||
TOPOLOGY_DISKS_PER_NODE=""
|
||||
TOPOLOGY_TOTAL_DISKS=""
|
||||
TOPOLOGY_CPU_PER_NODE=""
|
||||
TOPOLOGY_MEM_PER_NODE=""
|
||||
TOPOLOGY_NETWORK=""
|
||||
TOPOLOGY_ENDPOINT_MODE=""
|
||||
TOPOLOGY_ERASURE_SET_DRIVE_COUNT=""
|
||||
CLIENT_HOST=""
|
||||
WORKLOAD_LABEL="backlog-706-large-put-stage-breakdown"
|
||||
|
||||
usage() {
|
||||
cat <<'USAGE'
|
||||
Usage:
|
||||
scripts/run_put_large_stage_breakdown.sh --endpoint <url> \
|
||||
--access-key <ak> --secret-key <sk> [options]
|
||||
|
||||
Required:
|
||||
--endpoint <url> S3 endpoint
|
||||
--access-key <ak> Access key
|
||||
--secret-key <sk> Secret key
|
||||
|
||||
Core options:
|
||||
--bucket-prefix <prefix> Bucket prefix (default: rustfs-put-large)
|
||||
--region <name> Region (default: us-east-1)
|
||||
--sizes <csv> Object sizes (default: 16MiB,32MiB)
|
||||
--concurrencies <csv> Concurrency matrix (default: 16,32,64,96,128)
|
||||
--duration <dur> Per-run duration (default: 120s)
|
||||
--rounds <n> Rounds per size (default: 3)
|
||||
--retry-per-round <n> Retries per failed round (default: 1)
|
||||
--retry-sleep-secs <n> Sleep between retries (default: 2)
|
||||
--cooldown-secs <n> Sleep between rounds/sizes/concurrency runs (default: 20)
|
||||
--round-cooldown-secs <n> Compatibility alias for --cooldown-secs
|
||||
--out-dir <dir> Output root (default: target/bench/put-large-stage-breakdown-<timestamp>)
|
||||
--baseline-root <dir> Existing root from a previous run of this script
|
||||
--extra-args "<args>" Extra args passed to run_object_batch_bench_enhanced.sh
|
||||
--warp-bin <path> warp binary (default: warp)
|
||||
--insecure Pass --insecure to warp
|
||||
--dry-run Print commands only
|
||||
--python-bin <path> Python binary for Prometheus query capture (default: python3)
|
||||
--service-metrics-url <url> Plain Prometheus text scrape URL
|
||||
--service-prometheus-query-url <url>
|
||||
Prometheus HTTP API /api/v1/query URL for OTLP-exported metrics
|
||||
--service-prometheus-query <promql> PromQL for --service-prometheus-query-url
|
||||
--service-metrics-service-name <name>
|
||||
Optional service.name/service_name filter for query results
|
||||
--service-metrics-filter-regex <regex>
|
||||
Regex for retained plain text metrics lines
|
||||
--service-metrics-attempts <n> Direct scrape attempts per snapshot (default: 3)
|
||||
--service-metrics-retry-secs <n> Sleep between direct scrape attempts (default: 1)
|
||||
--service-metrics-connect-timeout-secs <n>
|
||||
Curl connect timeout for direct scrape (default: 2)
|
||||
--service-metrics-max-time-secs <n> Curl max time for direct scrape (default: 15)
|
||||
--service-metrics-settle-secs <n> Sleep before after-snapshot capture (default: 0)
|
||||
|
||||
Topology metadata (optional but recommended):
|
||||
--nodes <n>
|
||||
--disks-per-node <n>
|
||||
--total-disks <n>
|
||||
--cpu-per-node <text>
|
||||
--mem-per-node <text>
|
||||
--network <text>
|
||||
--endpoint-mode <direct|lb>
|
||||
--erasure-set-drive-count <n>
|
||||
--client-host <text>
|
||||
--workload-label <text>
|
||||
|
||||
Output layout:
|
||||
<out-dir>/
|
||||
run_manifest.txt
|
||||
run_matrix.csv
|
||||
aggregate_median_summary.csv
|
||||
aggregate_baseline_compare.csv
|
||||
artifact_layout.txt
|
||||
runs/
|
||||
c16/
|
||||
c32/
|
||||
...
|
||||
|
||||
Each runs/cXX directory is a direct output directory of:
|
||||
scripts/run_object_batch_bench_enhanced.sh
|
||||
USAGE
|
||||
}
|
||||
|
||||
require_cmd() {
|
||||
if ! command -v "$1" >/dev/null 2>&1; then
|
||||
echo "ERROR: command not found: $1" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
arg_value() {
|
||||
local flag="$1"
|
||||
local value="${2:-}"
|
||||
if [[ -z "$value" || "$value" == --* ]]; then
|
||||
echo "ERROR: missing value for $flag" >&2
|
||||
exit 1
|
||||
fi
|
||||
printf '%s\n' "$value"
|
||||
}
|
||||
|
||||
arg_value_allow_option() {
|
||||
local flag="$1"
|
||||
local value="${2:-}"
|
||||
if [[ -z "$value" ]]; then
|
||||
echo "ERROR: missing value for $flag" >&2
|
||||
exit 1
|
||||
fi
|
||||
printf '%s\n' "$value"
|
||||
}
|
||||
|
||||
parse_args() {
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--endpoint) ENDPOINT="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--access-key) ACCESS_KEY="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--secret-key) SECRET_KEY="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--bucket-prefix) BUCKET_PREFIX="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--region) REGION="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--sizes) SIZES="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--concurrencies) CONCURRENCIES="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--duration) DURATION="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--rounds) ROUNDS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--retry-per-round) RETRY_PER_ROUND="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--retry-sleep-secs) RETRY_SLEEP_SECS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--cooldown-secs|--round-cooldown-secs) COOLDOWN_SECS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--out-dir) OUT_DIR="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--baseline-root) BASELINE_ROOT="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--extra-args) EXTRA_ARGS="$(arg_value_allow_option "$1" "${2:-}")"; shift 2 ;;
|
||||
--warp-bin) WARP_BIN="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--python-bin) PYTHON_BIN="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--service-metrics-url) SERVICE_METRICS_URL="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--service-prometheus-query-url) SERVICE_PROMETHEUS_QUERY_URL="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--service-prometheus-query) SERVICE_PROMETHEUS_QUERY="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--service-metrics-service-name) SERVICE_METRICS_SERVICE_NAME="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--service-metrics-filter-regex) SERVICE_METRICS_FILTER_REGEX="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--service-metrics-attempts) SERVICE_METRICS_CAPTURE_ATTEMPTS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--service-metrics-retry-secs) SERVICE_METRICS_CAPTURE_RETRY_SECS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--service-metrics-connect-timeout-secs) SERVICE_METRICS_CONNECT_TIMEOUT_SECS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--service-metrics-max-time-secs) SERVICE_METRICS_MAX_TIME_SECS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--service-metrics-settle-secs) SERVICE_METRICS_SETTLE_SECS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--nodes) TOPOLOGY_NODES="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--disks-per-node) TOPOLOGY_DISKS_PER_NODE="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--total-disks) TOPOLOGY_TOTAL_DISKS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--cpu-per-node) TOPOLOGY_CPU_PER_NODE="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--mem-per-node) TOPOLOGY_MEM_PER_NODE="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--network) TOPOLOGY_NETWORK="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--endpoint-mode) TOPOLOGY_ENDPOINT_MODE="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--erasure-set-drive-count) TOPOLOGY_ERASURE_SET_DRIVE_COUNT="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--client-host) CLIENT_HOST="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--workload-label) WORKLOAD_LABEL="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--insecure) INSECURE=true; shift ;;
|
||||
--dry-run) DRY_RUN=true; shift ;;
|
||||
-h|--help) usage; exit 0 ;;
|
||||
*)
|
||||
echo "ERROR: unknown arg: $1" >&2
|
||||
usage
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
}
|
||||
|
||||
is_positive_int() {
|
||||
[[ "$1" =~ ^[1-9][0-9]*$ ]]
|
||||
}
|
||||
|
||||
is_nonnegative_int() {
|
||||
[[ "$1" =~ ^[0-9]+$ ]]
|
||||
}
|
||||
|
||||
cooldown_sleep() {
|
||||
local secs="$1"
|
||||
if (( secs <= 0 )); then
|
||||
return 0
|
||||
fi
|
||||
if [[ "$DRY_RUN" == "true" ]]; then
|
||||
echo "[DRY-RUN] sleep ${secs}"
|
||||
else
|
||||
sleep "$secs"
|
||||
fi
|
||||
}
|
||||
|
||||
validate_args() {
|
||||
if [[ -z "$ENDPOINT" || -z "$ACCESS_KEY" || -z "$SECRET_KEY" ]]; then
|
||||
echo "ERROR: --endpoint, --access-key, and --secret-key are required" >&2
|
||||
exit 1
|
||||
fi
|
||||
if [[ "$TOOL" != "warp" ]]; then
|
||||
echo "ERROR: only warp is supported by this wrapper" >&2
|
||||
exit 1
|
||||
fi
|
||||
if ! is_positive_int "$ROUNDS" || ! is_positive_int "$RETRY_PER_ROUND" || ! is_positive_int "$RETRY_SLEEP_SECS" || ! is_nonnegative_int "$COOLDOWN_SECS"; then
|
||||
echo "ERROR: --rounds, --retry-per-round, and --retry-sleep-secs must be positive integers; --cooldown-secs must be a nonnegative integer" >&2
|
||||
exit 1
|
||||
fi
|
||||
if [[ -n "$BASELINE_ROOT" && ! -d "$BASELINE_ROOT" ]]; then
|
||||
echo "ERROR: --baseline-root does not exist: $BASELINE_ROOT" >&2
|
||||
exit 1
|
||||
fi
|
||||
if [[ -n "$SERVICE_METRICS_URL" && -n "$SERVICE_PROMETHEUS_QUERY_URL" ]]; then
|
||||
echo "ERROR: --service-metrics-url and --service-prometheus-query-url are mutually exclusive" >&2
|
||||
exit 1
|
||||
fi
|
||||
if ! is_positive_int "$SERVICE_METRICS_CAPTURE_ATTEMPTS" || ! is_nonnegative_int "$SERVICE_METRICS_CAPTURE_RETRY_SECS" || ! is_positive_int "$SERVICE_METRICS_CONNECT_TIMEOUT_SECS" || ! is_positive_int "$SERVICE_METRICS_MAX_TIME_SECS" || ! is_nonnegative_int "$SERVICE_METRICS_SETTLE_SECS"; then
|
||||
echo "ERROR: service metrics attempt/timeout/settle options must be valid integers" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
setup_output() {
|
||||
if [[ -z "$OUT_DIR" ]]; then
|
||||
OUT_DIR="target/bench/put-large-stage-breakdown-$(date -u +%Y%m%dT%H%M%SZ)"
|
||||
fi
|
||||
RUNS_DIR="${OUT_DIR}/runs"
|
||||
AGG_MEDIAN_CSV="${OUT_DIR}/aggregate_median_summary.csv"
|
||||
AGG_COMPARE_CSV="${OUT_DIR}/aggregate_baseline_compare.csv"
|
||||
AGG_SERVICE_METRICS_CSV="${OUT_DIR}/aggregate_service_metrics_captures.csv"
|
||||
AGG_SERVICE_METRICS_DELTAS_CSV="${OUT_DIR}/aggregate_service_metrics_deltas.csv"
|
||||
AGG_SERVICE_METRICS_PATH_SUMMARY_CSV="${OUT_DIR}/aggregate_service_metrics_path_summary.csv"
|
||||
AGG_SERVICE_METRICS_STAGE_SUMMARY_CSV="${OUT_DIR}/aggregate_service_metrics_stage_summary.csv"
|
||||
RUN_MATRIX_CSV="${OUT_DIR}/run_matrix.csv"
|
||||
mkdir -p "$RUNS_DIR"
|
||||
|
||||
echo "concurrency,size,tool,successful_rounds,failed_rounds,median_throughput_bps,median_reqps,median_latency_ms,bucket,run_dir" > "$AGG_MEDIAN_CSV"
|
||||
echo "concurrency,size,tool,new_median_reqps,baseline_median_reqps,delta_reqps_pct,new_median_latency_ms,baseline_median_latency_ms,delta_latency_pct,new_median_throughput_bps,baseline_median_throughput_bps,delta_throughput_pct,run_dir" > "$AGG_COMPARE_CSV"
|
||||
echo "concurrency,run_dir,size,tool,round,attempt,phase,source,status,capture_attempt,raw_bytes,snapshot_bytes,status_file,snapshot_file,filter_regex,prometheus_query" > "$AGG_SERVICE_METRICS_CSV"
|
||||
echo "concurrency,bucket,run_dir,baseline_csv,status" > "$RUN_MATRIX_CSV"
|
||||
}
|
||||
|
||||
join_bool() {
|
||||
if [[ "$1" == "true" ]]; then
|
||||
echo "true"
|
||||
else
|
||||
echo "false"
|
||||
fi
|
||||
}
|
||||
|
||||
write_manifest() {
|
||||
local git_commit git_branch git_dirty rustc_version
|
||||
git_commit="$(git -C "$PROJECT_ROOT" rev-parse HEAD 2>/dev/null || echo "unknown")"
|
||||
git_branch="$(git -C "$PROJECT_ROOT" branch --show-current 2>/dev/null || echo "unknown")"
|
||||
if [[ -n "$(git -C "$PROJECT_ROOT" status --porcelain 2>/dev/null || true)" ]]; then
|
||||
git_dirty="true"
|
||||
else
|
||||
git_dirty="false"
|
||||
fi
|
||||
rustc_version="$(rustc --version 2>/dev/null || echo "unknown")"
|
||||
|
||||
{
|
||||
echo "created_utc=$(date -u +%Y-%m-%dT%H:%M:%SZ)"
|
||||
echo "git_commit=${git_commit}"
|
||||
echo "git_branch=${git_branch}"
|
||||
echo "git_dirty=${git_dirty}"
|
||||
echo "rustc_version=${rustc_version}"
|
||||
echo "kernel=$(uname -srvmo 2>/dev/null || echo "unknown")"
|
||||
echo "tool=${TOOL}"
|
||||
echo "endpoint=${ENDPOINT}"
|
||||
echo "region=${REGION}"
|
||||
echo "bucket_prefix=${BUCKET_PREFIX}"
|
||||
echo "sizes=${SIZES}"
|
||||
echo "concurrencies=${CONCURRENCIES}"
|
||||
echo "duration=${DURATION}"
|
||||
echo "rounds=${ROUNDS}"
|
||||
echo "retry_per_round=${RETRY_PER_ROUND}"
|
||||
echo "retry_sleep_secs=${RETRY_SLEEP_SECS}"
|
||||
echo "cooldown_secs=${COOLDOWN_SECS}"
|
||||
echo "insecure=$(join_bool "$INSECURE")"
|
||||
echo "dry_run=$(join_bool "$DRY_RUN")"
|
||||
echo "baseline_root=${BASELINE_ROOT:-N/A}"
|
||||
echo "extra_args_present=$([[ -n "$EXTRA_ARGS" ]] && echo true || echo false)"
|
||||
echo "python_bin=${PYTHON_BIN}"
|
||||
echo "service_metrics_url=${SERVICE_METRICS_URL:-N/A}"
|
||||
echo "service_prometheus_query_url=${SERVICE_PROMETHEUS_QUERY_URL:-N/A}"
|
||||
echo "service_prometheus_query=${SERVICE_PROMETHEUS_QUERY:-N/A}"
|
||||
echo "service_metrics_service_name=${SERVICE_METRICS_SERVICE_NAME:-N/A}"
|
||||
echo "service_metrics_filter_regex=${SERVICE_METRICS_FILTER_REGEX:-N/A}"
|
||||
echo "service_metrics_capture_attempts=${SERVICE_METRICS_CAPTURE_ATTEMPTS}"
|
||||
echo "service_metrics_retry_secs=${SERVICE_METRICS_CAPTURE_RETRY_SECS}"
|
||||
echo "service_metrics_connect_timeout_secs=${SERVICE_METRICS_CONNECT_TIMEOUT_SECS}"
|
||||
echo "service_metrics_max_time_secs=${SERVICE_METRICS_MAX_TIME_SECS}"
|
||||
echo "service_metrics_settle_secs=${SERVICE_METRICS_SETTLE_SECS}"
|
||||
echo "workload_label=${WORKLOAD_LABEL}"
|
||||
echo "nodes=${TOPOLOGY_NODES:-N/A}"
|
||||
echo "disks_per_node=${TOPOLOGY_DISKS_PER_NODE:-N/A}"
|
||||
echo "total_disks=${TOPOLOGY_TOTAL_DISKS:-N/A}"
|
||||
echo "cpu_per_node=${TOPOLOGY_CPU_PER_NODE:-N/A}"
|
||||
echo "mem_per_node=${TOPOLOGY_MEM_PER_NODE:-N/A}"
|
||||
echo "network=${TOPOLOGY_NETWORK:-N/A}"
|
||||
echo "endpoint_mode=${TOPOLOGY_ENDPOINT_MODE:-N/A}"
|
||||
echo "erasure_set_drive_count=${TOPOLOGY_ERASURE_SET_DRIVE_COUNT:-N/A}"
|
||||
echo "client_host=${CLIENT_HOST:-N/A}"
|
||||
echo "access_key=REDACTED"
|
||||
echo "secret_key=REDACTED"
|
||||
} > "${OUT_DIR}/run_manifest.txt"
|
||||
}
|
||||
|
||||
write_artifact_layout() {
|
||||
cat > "${OUT_DIR}/artifact_layout.txt" <<'EOF'
|
||||
Top-level artifacts:
|
||||
- run_manifest.txt: run metadata, git revision, topology notes, and redacted execution context
|
||||
- run_matrix.csv: one row per concurrency run, including bucket and baseline linkage
|
||||
- aggregate_median_summary.csv: merged median_summary rows from every concurrency run
|
||||
- aggregate_baseline_compare.csv: merged baseline_compare rows when a matching baseline exists
|
||||
- aggregate_service_metrics_captures.csv: merged service metrics capture status from every concurrency run
|
||||
- aggregate_service_metrics_deltas.csv: per-round before/after service metric deltas
|
||||
- aggregate_service_metrics_path_summary.csv: per-round PUT path deltas
|
||||
- aggregate_service_metrics_stage_summary.csv: per-round PUT stage duration averages from deltas
|
||||
- runs/cXX/: direct output directory of scripts/run_object_batch_bench_enhanced.sh
|
||||
|
||||
Per-concurrency directory:
|
||||
- round_results.csv
|
||||
- median_summary.csv
|
||||
- service_metrics_captures.csv (when service metrics capture is enabled)
|
||||
- baseline_compare.csv (only when a baseline CSV is supplied)
|
||||
- logs/
|
||||
EOF
|
||||
}
|
||||
|
||||
trim() {
|
||||
echo "$1" | awk '{$1=$1;print}'
|
||||
}
|
||||
|
||||
csv_to_lines() {
|
||||
local csv="$1"
|
||||
IFS=',' read -r -a arr <<< "$csv"
|
||||
for raw in "${arr[@]}"; do
|
||||
local item
|
||||
item="$(trim "$raw")"
|
||||
[[ -z "$item" ]] && continue
|
||||
echo "$item"
|
||||
done
|
||||
}
|
||||
|
||||
sanitize_bucket_component() {
|
||||
echo "$1" | tr '[:upper:]' '[:lower:]' | sed -E 's/[^a-z0-9-]+/-/g; s/^-+//; s/-+$//; s/-{2,}/-/g'
|
||||
}
|
||||
|
||||
build_bucket_name() {
|
||||
local concurrency="$1"
|
||||
local run_id prefix raw
|
||||
run_id="$(date -u +%Y%m%d%H%M%S)"
|
||||
prefix="$(sanitize_bucket_component "$BUCKET_PREFIX")"
|
||||
raw="${prefix}-${run_id}-c${concurrency}"
|
||||
raw="$(sanitize_bucket_component "$raw")"
|
||||
echo "${raw:0:63}"
|
||||
}
|
||||
|
||||
resolve_baseline_csv() {
|
||||
local concurrency="$1"
|
||||
local candidate=""
|
||||
if [[ -z "$BASELINE_ROOT" ]]; then
|
||||
return 0
|
||||
fi
|
||||
|
||||
if [[ -f "${BASELINE_ROOT}/runs/c${concurrency}/median_summary.csv" ]]; then
|
||||
candidate="${BASELINE_ROOT}/runs/c${concurrency}/median_summary.csv"
|
||||
elif [[ -f "${BASELINE_ROOT}/c${concurrency}/median_summary.csv" ]]; then
|
||||
candidate="${BASELINE_ROOT}/c${concurrency}/median_summary.csv"
|
||||
fi
|
||||
|
||||
if [[ -n "$candidate" ]]; then
|
||||
printf '%s\n' "$candidate"
|
||||
fi
|
||||
}
|
||||
|
||||
append_aggregate_rows() {
|
||||
local concurrency="$1"
|
||||
local bucket="$2"
|
||||
local run_dir="$3"
|
||||
local median_csv="$4"
|
||||
local compare_csv="$5"
|
||||
local service_metrics_csv="${run_dir}/service_metrics_captures.csv"
|
||||
|
||||
awk -F',' -v c="$concurrency" -v b="$bucket" -v rd="$run_dir" 'NR>1 {print c "," $1 "," $2 "," $4 "," $5 "," $6 "," $7 "," $8 "," b "," rd}' "$median_csv" >> "$AGG_MEDIAN_CSV"
|
||||
|
||||
if [[ -f "$compare_csv" ]]; then
|
||||
awk -F',' -v c="$concurrency" -v rd="$run_dir" 'NR>1 {print c "," $1 "," $2 "," $4 "," $5 "," $6 "," $7 "," $8 "," $9 "," $10 "," $11 "," $12 "," rd}' "$compare_csv" >> "$AGG_COMPARE_CSV"
|
||||
fi
|
||||
if [[ -f "$service_metrics_csv" ]]; then
|
||||
awk -F',' -v c="$concurrency" -v rd="$run_dir" 'NR>1 {print c "," rd "," $0}' "$service_metrics_csv" >> "$AGG_SERVICE_METRICS_CSV"
|
||||
fi
|
||||
}
|
||||
|
||||
analyze_service_metric_deltas() {
|
||||
if [[ ! -s "$AGG_SERVICE_METRICS_CSV" ]]; then
|
||||
return 0
|
||||
fi
|
||||
if [[ "$DRY_RUN" == "true" ]]; then
|
||||
return 0
|
||||
fi
|
||||
if [[ ! -f "$SERVICE_METRICS_DELTA_SCRIPT" ]]; then
|
||||
echo "ERROR: missing service metrics delta script: $SERVICE_METRICS_DELTA_SCRIPT" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
"$PYTHON_BIN" "$SERVICE_METRICS_DELTA_SCRIPT" \
|
||||
--capture-csv "$AGG_SERVICE_METRICS_CSV" \
|
||||
--delta-csv "$AGG_SERVICE_METRICS_DELTAS_CSV" \
|
||||
--path-summary-csv "$AGG_SERVICE_METRICS_PATH_SUMMARY_CSV" \
|
||||
--stage-summary-csv "$AGG_SERVICE_METRICS_STAGE_SUMMARY_CSV"
|
||||
}
|
||||
|
||||
run_concurrency() {
|
||||
local concurrency="$1"
|
||||
local run_dir bucket baseline_csv status
|
||||
run_dir="${RUNS_DIR}/c${concurrency}"
|
||||
bucket="$(build_bucket_name "$concurrency")"
|
||||
baseline_csv="$(resolve_baseline_csv "$concurrency" || true)"
|
||||
status="pending"
|
||||
|
||||
local -a cmd=(
|
||||
bash "$ENHANCED_BENCH_SCRIPT"
|
||||
--tool "$TOOL"
|
||||
--endpoint "$ENDPOINT"
|
||||
--access-key "$ACCESS_KEY"
|
||||
--secret-key "$SECRET_KEY"
|
||||
--bucket "$bucket"
|
||||
--region "$REGION"
|
||||
--warp-bin "$WARP_BIN"
|
||||
--warp-mode put
|
||||
--sizes "$SIZES"
|
||||
--concurrency "$concurrency"
|
||||
--duration "$DURATION"
|
||||
--rounds "$ROUNDS"
|
||||
--retry-per-round "$RETRY_PER_ROUND"
|
||||
--retry-sleep-secs "$RETRY_SLEEP_SECS"
|
||||
--cooldown-secs "$COOLDOWN_SECS"
|
||||
--out-dir "$run_dir"
|
||||
)
|
||||
|
||||
if [[ "$INSECURE" == "true" ]]; then
|
||||
cmd+=(--insecure)
|
||||
fi
|
||||
if [[ -n "$baseline_csv" ]]; then
|
||||
cmd+=(--baseline-csv "$baseline_csv")
|
||||
fi
|
||||
if [[ -n "$EXTRA_ARGS" ]]; then
|
||||
cmd+=(--extra-args "$EXTRA_ARGS")
|
||||
fi
|
||||
if [[ -n "$SERVICE_METRICS_URL" || -n "$SERVICE_PROMETHEUS_QUERY_URL" ]]; then
|
||||
cmd+=(--service-metrics-dir "${run_dir}/service-metrics")
|
||||
fi
|
||||
if [[ -n "$SERVICE_METRICS_URL" ]]; then
|
||||
cmd+=(--service-metrics-url "$SERVICE_METRICS_URL")
|
||||
fi
|
||||
if [[ -n "$SERVICE_PROMETHEUS_QUERY_URL" ]]; then
|
||||
cmd+=(--service-prometheus-query-url "$SERVICE_PROMETHEUS_QUERY_URL")
|
||||
fi
|
||||
if [[ -n "$SERVICE_PROMETHEUS_QUERY" ]]; then
|
||||
cmd+=(--service-prometheus-query "$SERVICE_PROMETHEUS_QUERY")
|
||||
fi
|
||||
if [[ -n "$SERVICE_METRICS_SERVICE_NAME" ]]; then
|
||||
cmd+=(--service-metrics-service-name "$SERVICE_METRICS_SERVICE_NAME")
|
||||
fi
|
||||
if [[ -n "$SERVICE_METRICS_FILTER_REGEX" ]]; then
|
||||
cmd+=(--service-metrics-filter-regex "$SERVICE_METRICS_FILTER_REGEX")
|
||||
fi
|
||||
cmd+=(
|
||||
--python-bin "$PYTHON_BIN"
|
||||
--service-metrics-attempts "$SERVICE_METRICS_CAPTURE_ATTEMPTS"
|
||||
--service-metrics-retry-secs "$SERVICE_METRICS_CAPTURE_RETRY_SECS"
|
||||
--service-metrics-connect-timeout-secs "$SERVICE_METRICS_CONNECT_TIMEOUT_SECS"
|
||||
--service-metrics-max-time-secs "$SERVICE_METRICS_MAX_TIME_SECS"
|
||||
--service-metrics-settle-secs "$SERVICE_METRICS_SETTLE_SECS"
|
||||
)
|
||||
if [[ "$DRY_RUN" == "true" ]]; then
|
||||
cmd+=(--dry-run)
|
||||
fi
|
||||
|
||||
echo "==== concurrency=${concurrency} bucket=${bucket} ===="
|
||||
printf 'Command:'
|
||||
printf ' %q' "${cmd[@]}"
|
||||
printf '\n'
|
||||
|
||||
if "${cmd[@]}"; then
|
||||
status="ok"
|
||||
else
|
||||
status="failed"
|
||||
fi
|
||||
|
||||
echo "${concurrency},${bucket},${run_dir},${baseline_csv:-N/A},${status}" >> "$RUN_MATRIX_CSV"
|
||||
|
||||
if [[ "$status" != "ok" ]]; then
|
||||
echo "ERROR: concurrency=${concurrency} failed; aborting remaining matrix runs" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
append_aggregate_rows "$concurrency" "$bucket" "$run_dir" "${run_dir}/median_summary.csv" "${run_dir}/baseline_compare.csv"
|
||||
}
|
||||
|
||||
main() {
|
||||
parse_args "$@"
|
||||
validate_args
|
||||
require_cmd bash
|
||||
require_cmd git
|
||||
require_cmd awk
|
||||
require_cmd sed
|
||||
require_cmd sort
|
||||
require_cmd "$WARP_BIN"
|
||||
|
||||
if [[ ! -x "$ENHANCED_BENCH_SCRIPT" && ! -f "$ENHANCED_BENCH_SCRIPT" ]]; then
|
||||
echo "ERROR: missing dependency script: $ENHANCED_BENCH_SCRIPT" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
setup_output
|
||||
write_manifest
|
||||
write_artifact_layout
|
||||
|
||||
echo "Output dir: $OUT_DIR"
|
||||
echo "Concurrencies: $CONCURRENCIES"
|
||||
echo "Sizes: $SIZES"
|
||||
echo "Duration: $DURATION"
|
||||
echo "Rounds: $ROUNDS"
|
||||
echo "Cooldown secs: $COOLDOWN_SECS"
|
||||
if [[ -n "$SERVICE_METRICS_URL" || -n "$SERVICE_PROMETHEUS_QUERY_URL" ]]; then
|
||||
echo "Service metrics capture: enabled"
|
||||
echo "Service metrics source: $([[ -n "$SERVICE_PROMETHEUS_QUERY_URL" ]] && echo prometheus_query || echo prometheus_text)"
|
||||
fi
|
||||
|
||||
local conc_count conc_index
|
||||
conc_count="$(csv_to_lines "$CONCURRENCIES" | awk 'END{print NR+0}')"
|
||||
conc_index=0
|
||||
while IFS= read -r concurrency; do
|
||||
conc_index=$(( conc_index + 1 ))
|
||||
run_concurrency "$concurrency"
|
||||
if (( COOLDOWN_SECS > 0 && conc_index < conc_count )); then
|
||||
echo "Cooldown after concurrency=${concurrency}: ${COOLDOWN_SECS}s"
|
||||
cooldown_sleep "$COOLDOWN_SECS"
|
||||
fi
|
||||
done < <(csv_to_lines "$CONCURRENCIES")
|
||||
|
||||
analyze_service_metric_deltas
|
||||
|
||||
echo
|
||||
echo "Stage-breakdown run finished."
|
||||
echo "Artifacts written to: $OUT_DIR"
|
||||
echo "Top-level summaries:"
|
||||
echo " - $RUN_MATRIX_CSV"
|
||||
echo " - $AGG_MEDIAN_CSV"
|
||||
if [[ -s "$AGG_SERVICE_METRICS_CSV" ]]; then
|
||||
echo " - $AGG_SERVICE_METRICS_CSV"
|
||||
fi
|
||||
if [[ -s "$AGG_SERVICE_METRICS_DELTAS_CSV" ]]; then
|
||||
echo " - $AGG_SERVICE_METRICS_DELTAS_CSV"
|
||||
echo " - $AGG_SERVICE_METRICS_PATH_SUMMARY_CSV"
|
||||
echo " - $AGG_SERVICE_METRICS_STAGE_SUMMARY_CSV"
|
||||
fi
|
||||
if [[ -s "$AGG_COMPARE_CSV" ]]; then
|
||||
echo " - $AGG_COMPARE_CSV"
|
||||
fi
|
||||
}
|
||||
|
||||
main "$@"
|
||||
@@ -0,0 +1,406 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# One-shot wrapper for rustfs/backlog#706:
|
||||
# - runs the large PUT stage-breakdown benchmark matrix
|
||||
# - starts a parallel artifact collector that covers the benchmark window
|
||||
# - stores both outputs under the same run root
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
|
||||
BENCH_SCRIPT="${PROJECT_ROOT}/scripts/run_put_large_stage_breakdown.sh"
|
||||
CAPTURE_SCRIPT="${PROJECT_ROOT}/scripts/collect_put_large_stage_breakdown_artifacts.sh"
|
||||
|
||||
ENDPOINT=""
|
||||
ACCESS_KEY=""
|
||||
SECRET_KEY=""
|
||||
REGION="us-east-1"
|
||||
BUCKET_PREFIX="rustfs-put-large"
|
||||
CONCURRENCIES="16,32,64,96,128"
|
||||
SIZES="16MiB,32MiB"
|
||||
DURATION="120s"
|
||||
ROUNDS=3
|
||||
RETRY_PER_ROUND=1
|
||||
RETRY_SLEEP_SECS=2
|
||||
COOLDOWN_SECS=0
|
||||
OUT_DIR=""
|
||||
BASELINE_ROOT=""
|
||||
EXTRA_ARGS=""
|
||||
INSECURE=false
|
||||
DRY_RUN=false
|
||||
|
||||
WARP_BIN="${WARP_BIN:-warp}"
|
||||
|
||||
TOPOLOGY_NODES=""
|
||||
TOPOLOGY_DISKS_PER_NODE=""
|
||||
TOPOLOGY_TOTAL_DISKS=""
|
||||
TOPOLOGY_CPU_PER_NODE=""
|
||||
TOPOLOGY_MEM_PER_NODE=""
|
||||
TOPOLOGY_NETWORK=""
|
||||
TOPOLOGY_ENDPOINT_MODE=""
|
||||
TOPOLOGY_ERASURE_SET_DRIVE_COUNT=""
|
||||
CLIENT_HOST=""
|
||||
WORKLOAD_LABEL="backlog-706-large-put-stage-breakdown"
|
||||
|
||||
CAPTURE_LABEL="benchmark-window"
|
||||
CAPTURE_METRICS_ENDPOINTS=""
|
||||
CAPTURE_PROM_METRICS_URLS=""
|
||||
CAPTURE_DURATION_SECS=""
|
||||
CAPTURE_INTERVAL_SECS=15
|
||||
CAPTURE_RUSTFS_PID=""
|
||||
CAPTURE_SKIP_HOST_TELEMETRY=false
|
||||
SKIP_CAPTURE=false
|
||||
|
||||
usage() {
|
||||
cat <<'USAGE'
|
||||
Usage:
|
||||
scripts/run_put_large_stage_breakdown_with_capture.sh --endpoint <url> \
|
||||
--access-key <ak> --secret-key <sk> [options]
|
||||
|
||||
Benchmark options:
|
||||
--endpoint <url>
|
||||
--access-key <ak>
|
||||
--secret-key <sk>
|
||||
--bucket-prefix <prefix> Default: rustfs-put-large
|
||||
--region <name> Default: us-east-1
|
||||
--sizes <csv> Default: 16MiB,32MiB
|
||||
--concurrencies <csv> Default: 16,32,64,96,128
|
||||
--duration <dur> Default: 120s
|
||||
--rounds <n> Default: 3
|
||||
--retry-per-round <n> Default: 1
|
||||
--retry-sleep-secs <n> Default: 2
|
||||
--cooldown-secs <n> Sleep between rounds/sizes (default: 0)
|
||||
--out-dir <dir> Default: target/bench/put-large-stage-breakdown-<timestamp>
|
||||
--baseline-root <dir>
|
||||
--extra-args "<args>"
|
||||
--warp-bin <path> Default: warp
|
||||
--insecure
|
||||
--dry-run
|
||||
|
||||
Topology metadata:
|
||||
--nodes <n>
|
||||
--disks-per-node <n>
|
||||
--total-disks <n>
|
||||
--cpu-per-node <text>
|
||||
--mem-per-node <text>
|
||||
--network <text>
|
||||
--endpoint-mode <direct|lb>
|
||||
--erasure-set-drive-count <n>
|
||||
--client-host <text>
|
||||
--workload-label <text>
|
||||
|
||||
Capture options:
|
||||
--capture-label <name> Default: benchmark-window
|
||||
--capture-metrics-endpoints <csv> Signed admin metrics endpoints
|
||||
--capture-prom-metrics-urls <csv> Optional plain Prometheus text endpoints
|
||||
--capture-duration-secs <n> Override auto-derived capture window
|
||||
--capture-interval-secs <n> Default: 15
|
||||
--capture-rustfs-pid <pid>
|
||||
--capture-skip-host-telemetry
|
||||
--skip-capture Run benchmark only and do not start capture
|
||||
|
||||
Behavior:
|
||||
- If --capture-duration-secs is omitted, the wrapper derives a nominal
|
||||
capture window from the benchmark matrix and adds a safety slack.
|
||||
- The collector runs in the background while the benchmark matrix is in the foreground.
|
||||
USAGE
|
||||
}
|
||||
|
||||
require_cmd() {
|
||||
if ! command -v "$1" >/dev/null 2>&1; then
|
||||
echo "ERROR: command not found: $1" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
arg_value() {
|
||||
local flag="$1"
|
||||
local value="${2:-}"
|
||||
if [[ -z "$value" || "$value" == --* ]]; then
|
||||
echo "ERROR: missing value for $flag" >&2
|
||||
exit 1
|
||||
fi
|
||||
printf '%s\n' "$value"
|
||||
}
|
||||
|
||||
parse_args() {
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--endpoint) ENDPOINT="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--access-key) ACCESS_KEY="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--secret-key) SECRET_KEY="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--bucket-prefix) BUCKET_PREFIX="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--region) REGION="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--sizes) SIZES="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--concurrencies) CONCURRENCIES="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--duration) DURATION="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--rounds) ROUNDS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--retry-per-round) RETRY_PER_ROUND="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--retry-sleep-secs) RETRY_SLEEP_SECS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--cooldown-secs) COOLDOWN_SECS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--out-dir) OUT_DIR="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--baseline-root) BASELINE_ROOT="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--extra-args) EXTRA_ARGS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--warp-bin) WARP_BIN="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--nodes) TOPOLOGY_NODES="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--disks-per-node) TOPOLOGY_DISKS_PER_NODE="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--total-disks) TOPOLOGY_TOTAL_DISKS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--cpu-per-node) TOPOLOGY_CPU_PER_NODE="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--mem-per-node) TOPOLOGY_MEM_PER_NODE="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--network) TOPOLOGY_NETWORK="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--endpoint-mode) TOPOLOGY_ENDPOINT_MODE="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--erasure-set-drive-count) TOPOLOGY_ERASURE_SET_DRIVE_COUNT="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--client-host) CLIENT_HOST="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--workload-label) WORKLOAD_LABEL="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--capture-label) CAPTURE_LABEL="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--capture-metrics-endpoints) CAPTURE_METRICS_ENDPOINTS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--capture-prom-metrics-urls) CAPTURE_PROM_METRICS_URLS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--capture-duration-secs) CAPTURE_DURATION_SECS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--capture-interval-secs) CAPTURE_INTERVAL_SECS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--capture-rustfs-pid) CAPTURE_RUSTFS_PID="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--capture-skip-host-telemetry) CAPTURE_SKIP_HOST_TELEMETRY=true; shift ;;
|
||||
--skip-capture) SKIP_CAPTURE=true; shift ;;
|
||||
--insecure) INSECURE=true; shift ;;
|
||||
--dry-run) DRY_RUN=true; shift ;;
|
||||
-h|--help) usage; exit 0 ;;
|
||||
*)
|
||||
echo "ERROR: unknown arg: $1" >&2
|
||||
usage
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
}
|
||||
|
||||
is_positive_int() {
|
||||
[[ "$1" =~ ^[1-9][0-9]*$ ]]
|
||||
}
|
||||
|
||||
is_nonnegative_int() {
|
||||
[[ "$1" =~ ^[0-9]+$ ]]
|
||||
}
|
||||
|
||||
count_csv_items() {
|
||||
local csv="$1"
|
||||
local count=0 raw item
|
||||
IFS=',' read -r -a arr <<< "$csv"
|
||||
for raw in "${arr[@]}"; do
|
||||
item="$(echo "$raw" | awk '{$1=$1;print}')"
|
||||
[[ -z "$item" ]] && continue
|
||||
count=$((count + 1))
|
||||
done
|
||||
echo "$count"
|
||||
}
|
||||
|
||||
duration_to_seconds() {
|
||||
local value="$1"
|
||||
if [[ "$value" =~ ^([0-9]+)s$ ]]; then
|
||||
echo "${BASH_REMATCH[1]}"
|
||||
elif [[ "$value" =~ ^([0-9]+)m$ ]]; then
|
||||
echo $(( BASH_REMATCH[1] * 60 ))
|
||||
elif [[ "$value" =~ ^([0-9]+)h$ ]]; then
|
||||
echo $(( BASH_REMATCH[1] * 3600 ))
|
||||
elif [[ "$value" =~ ^[0-9]+$ ]]; then
|
||||
echo "$value"
|
||||
else
|
||||
echo "ERROR"
|
||||
fi
|
||||
}
|
||||
|
||||
validate_args() {
|
||||
if [[ -z "$ENDPOINT" || -z "$ACCESS_KEY" || -z "$SECRET_KEY" ]]; then
|
||||
echo "ERROR: --endpoint, --access-key, and --secret-key are required" >&2
|
||||
exit 1
|
||||
fi
|
||||
if ! is_positive_int "$ROUNDS" || ! is_positive_int "$RETRY_PER_ROUND" || ! is_positive_int "$RETRY_SLEEP_SECS" || ! is_nonnegative_int "$COOLDOWN_SECS"; then
|
||||
echo "ERROR: --rounds, --retry-per-round, and --retry-sleep-secs must be positive integers; --cooldown-secs must be a nonnegative integer" >&2
|
||||
exit 1
|
||||
fi
|
||||
if ! is_positive_int "$CAPTURE_INTERVAL_SECS"; then
|
||||
echo "ERROR: --capture-interval-secs must be a positive integer" >&2
|
||||
exit 1
|
||||
fi
|
||||
if [[ -n "$CAPTURE_DURATION_SECS" && ! "$CAPTURE_DURATION_SECS" =~ ^[0-9]+$ ]]; then
|
||||
echo "ERROR: --capture-duration-secs must be a nonnegative integer" >&2
|
||||
exit 1
|
||||
fi
|
||||
if [[ -n "$BASELINE_ROOT" && ! -d "$BASELINE_ROOT" ]]; then
|
||||
echo "ERROR: --baseline-root does not exist: $BASELINE_ROOT" >&2
|
||||
exit 1
|
||||
fi
|
||||
local duration_secs
|
||||
duration_secs="$(duration_to_seconds "$DURATION")"
|
||||
if [[ "$duration_secs" == "ERROR" ]]; then
|
||||
echo "ERROR: unsupported --duration format: $DURATION (expected e.g. 120s, 2m, 1h)" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
setup_output() {
|
||||
if [[ -z "$OUT_DIR" ]]; then
|
||||
OUT_DIR="target/bench/put-large-stage-breakdown-$(date -u +%Y%m%dT%H%M%SZ)"
|
||||
fi
|
||||
}
|
||||
|
||||
derive_capture_duration() {
|
||||
if [[ -n "$CAPTURE_DURATION_SECS" ]]; then
|
||||
echo "$CAPTURE_DURATION_SECS"
|
||||
return
|
||||
fi
|
||||
|
||||
local sizes_count conc_count duration_secs nominal bench_secs retry_slack cooldown_slack fixed_slack
|
||||
sizes_count="$(count_csv_items "$SIZES")"
|
||||
conc_count="$(count_csv_items "$CONCURRENCIES")"
|
||||
duration_secs="$(duration_to_seconds "$DURATION")"
|
||||
bench_secs=$(( sizes_count * conc_count * ROUNDS * duration_secs ))
|
||||
retry_slack=$(( conc_count * RETRY_PER_ROUND * RETRY_SLEEP_SECS + conc_count * 30 ))
|
||||
cooldown_slack=$(( (conc_count * (sizes_count * (ROUNDS - 1) + (sizes_count - 1)) + (conc_count - 1)) * COOLDOWN_SECS ))
|
||||
fixed_slack=120
|
||||
nominal=$(( bench_secs + retry_slack + cooldown_slack + fixed_slack ))
|
||||
echo "$nominal"
|
||||
}
|
||||
|
||||
run_capture() {
|
||||
local capture_duration="$1"
|
||||
local -a cmd=(
|
||||
bash "$CAPTURE_SCRIPT"
|
||||
--run-root "$OUT_DIR"
|
||||
--endpoint "$ENDPOINT"
|
||||
--label "$CAPTURE_LABEL"
|
||||
--access-key "$ACCESS_KEY"
|
||||
--secret-key-env RUSTFS_CAPTURE_SECRET_KEY
|
||||
--region "$REGION"
|
||||
--duration-secs "$capture_duration"
|
||||
--interval-secs "$CAPTURE_INTERVAL_SECS"
|
||||
)
|
||||
|
||||
if [[ -n "$CAPTURE_METRICS_ENDPOINTS" ]]; then
|
||||
cmd+=(--metrics-endpoints "$CAPTURE_METRICS_ENDPOINTS")
|
||||
fi
|
||||
if [[ -n "$CAPTURE_PROM_METRICS_URLS" ]]; then
|
||||
cmd+=(--prom-metrics-urls "$CAPTURE_PROM_METRICS_URLS")
|
||||
fi
|
||||
if [[ -n "$CAPTURE_RUSTFS_PID" ]]; then
|
||||
cmd+=(--rustfs-pid "$CAPTURE_RUSTFS_PID")
|
||||
fi
|
||||
if [[ "$CAPTURE_SKIP_HOST_TELEMETRY" == "true" ]]; then
|
||||
cmd+=(--skip-host-telemetry)
|
||||
fi
|
||||
if [[ "$DRY_RUN" == "true" ]]; then
|
||||
cmd+=(--dry-run)
|
||||
fi
|
||||
|
||||
printf 'Capture command:'
|
||||
printf ' %q' "${cmd[@]}"
|
||||
printf '\n'
|
||||
|
||||
RUSTFS_CAPTURE_SECRET_KEY="$SECRET_KEY" "${cmd[@]}" &
|
||||
CAPTURE_PID=$!
|
||||
}
|
||||
|
||||
run_benchmark() {
|
||||
local -a cmd=(
|
||||
bash "$BENCH_SCRIPT"
|
||||
--endpoint "$ENDPOINT"
|
||||
--access-key "$ACCESS_KEY"
|
||||
--secret-key "$SECRET_KEY"
|
||||
--bucket-prefix "$BUCKET_PREFIX"
|
||||
--region "$REGION"
|
||||
--sizes "$SIZES"
|
||||
--concurrencies "$CONCURRENCIES"
|
||||
--duration "$DURATION"
|
||||
--rounds "$ROUNDS"
|
||||
--retry-per-round "$RETRY_PER_ROUND"
|
||||
--retry-sleep-secs "$RETRY_SLEEP_SECS"
|
||||
--cooldown-secs "$COOLDOWN_SECS"
|
||||
--out-dir "$OUT_DIR"
|
||||
--warp-bin "$WARP_BIN"
|
||||
--workload-label "$WORKLOAD_LABEL"
|
||||
)
|
||||
|
||||
if [[ -n "$BASELINE_ROOT" ]]; then
|
||||
cmd+=(--baseline-root "$BASELINE_ROOT")
|
||||
fi
|
||||
if [[ -n "$EXTRA_ARGS" ]]; then
|
||||
cmd+=(--extra-args "$EXTRA_ARGS")
|
||||
fi
|
||||
if [[ -n "$TOPOLOGY_NODES" ]]; then
|
||||
cmd+=(--nodes "$TOPOLOGY_NODES")
|
||||
fi
|
||||
if [[ -n "$TOPOLOGY_DISKS_PER_NODE" ]]; then
|
||||
cmd+=(--disks-per-node "$TOPOLOGY_DISKS_PER_NODE")
|
||||
fi
|
||||
if [[ -n "$TOPOLOGY_TOTAL_DISKS" ]]; then
|
||||
cmd+=(--total-disks "$TOPOLOGY_TOTAL_DISKS")
|
||||
fi
|
||||
if [[ -n "$TOPOLOGY_CPU_PER_NODE" ]]; then
|
||||
cmd+=(--cpu-per-node "$TOPOLOGY_CPU_PER_NODE")
|
||||
fi
|
||||
if [[ -n "$TOPOLOGY_MEM_PER_NODE" ]]; then
|
||||
cmd+=(--mem-per-node "$TOPOLOGY_MEM_PER_NODE")
|
||||
fi
|
||||
if [[ -n "$TOPOLOGY_NETWORK" ]]; then
|
||||
cmd+=(--network "$TOPOLOGY_NETWORK")
|
||||
fi
|
||||
if [[ -n "$TOPOLOGY_ENDPOINT_MODE" ]]; then
|
||||
cmd+=(--endpoint-mode "$TOPOLOGY_ENDPOINT_MODE")
|
||||
fi
|
||||
if [[ -n "$TOPOLOGY_ERASURE_SET_DRIVE_COUNT" ]]; then
|
||||
cmd+=(--erasure-set-drive-count "$TOPOLOGY_ERASURE_SET_DRIVE_COUNT")
|
||||
fi
|
||||
if [[ -n "$CLIENT_HOST" ]]; then
|
||||
cmd+=(--client-host "$CLIENT_HOST")
|
||||
fi
|
||||
if [[ "$INSECURE" == "true" ]]; then
|
||||
cmd+=(--insecure)
|
||||
fi
|
||||
if [[ "$DRY_RUN" == "true" ]]; then
|
||||
cmd+=(--dry-run)
|
||||
fi
|
||||
|
||||
printf 'Benchmark command:'
|
||||
printf ' %q' "${cmd[@]}"
|
||||
printf '\n'
|
||||
|
||||
"${cmd[@]}"
|
||||
}
|
||||
|
||||
cleanup_capture() {
|
||||
if [[ -n "${CAPTURE_PID:-}" ]] && kill -0 "$CAPTURE_PID" >/dev/null 2>&1; then
|
||||
kill "$CAPTURE_PID" >/dev/null 2>&1 || true
|
||||
wait "$CAPTURE_PID" >/dev/null 2>&1 || true
|
||||
fi
|
||||
}
|
||||
|
||||
main() {
|
||||
parse_args "$@"
|
||||
validate_args
|
||||
require_cmd bash
|
||||
require_cmd awk
|
||||
require_cmd "$WARP_BIN"
|
||||
|
||||
setup_output
|
||||
echo "Output dir: $OUT_DIR"
|
||||
if [[ "$SKIP_CAPTURE" != "true" ]]; then
|
||||
local capture_duration
|
||||
capture_duration="$(derive_capture_duration)"
|
||||
echo "Derived capture duration seconds: $capture_duration"
|
||||
echo "Capture label: $CAPTURE_LABEL"
|
||||
trap cleanup_capture EXIT
|
||||
run_capture "$capture_duration"
|
||||
else
|
||||
echo "Capture: skipped"
|
||||
fi
|
||||
|
||||
run_benchmark
|
||||
if [[ "$SKIP_CAPTURE" != "true" ]]; then
|
||||
wait "$CAPTURE_PID"
|
||||
CAPTURE_PID=""
|
||||
fi
|
||||
|
||||
echo
|
||||
echo "One-shot benchmark + capture run finished."
|
||||
echo "Run root: $OUT_DIR"
|
||||
}
|
||||
|
||||
main "$@"
|
||||
+430
@@ -0,0 +1,430 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# rustfs/backlog#708 tuning matrix runner for large-object PUT.
|
||||
# This controller:
|
||||
# - defines high-priority tuning profiles for 16MiB / 32MiB PUT
|
||||
# - writes an env snapshot per profile
|
||||
# - optionally calls an apply hook after switching profile env
|
||||
# - reuses scripts/run_put_large_stage_breakdown_with_capture.sh for each profile
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
RUNNER_SCRIPT="${SCRIPT_DIR}/run_put_large_stage_breakdown_with_capture.sh"
|
||||
|
||||
GROUP="all" # all|baseline|inflight|io-buffer|runtime|duplex
|
||||
ENDPOINT=""
|
||||
ACCESS_KEY=""
|
||||
SECRET_KEY=""
|
||||
REGION="us-east-1"
|
||||
BUCKET_PREFIX="rustfs-put-large"
|
||||
CONCURRENCIES="16,32,64,96,128"
|
||||
SIZES="16MiB,32MiB"
|
||||
DURATION="120s"
|
||||
ROUNDS=3
|
||||
OUT_ROOT=""
|
||||
WARP_BIN="warp"
|
||||
EXTRA_ARGS=""
|
||||
INSECURE=false
|
||||
DRY_RUN=false
|
||||
COOLDOWN_SECS=30
|
||||
|
||||
TOPOLOGY_NODES=""
|
||||
TOPOLOGY_DISKS_PER_NODE=""
|
||||
TOPOLOGY_TOTAL_DISKS=""
|
||||
TOPOLOGY_CPU_PER_NODE=""
|
||||
TOPOLOGY_MEM_PER_NODE=""
|
||||
TOPOLOGY_NETWORK=""
|
||||
TOPOLOGY_ENDPOINT_MODE=""
|
||||
TOPOLOGY_ERASURE_SET_DRIVE_COUNT=""
|
||||
CLIENT_HOST=""
|
||||
|
||||
CAPTURE_METRICS_ENDPOINTS=""
|
||||
CAPTURE_PROM_METRICS_URLS=""
|
||||
CAPTURE_INTERVAL_SECS=15
|
||||
CAPTURE_RUSTFS_PID=""
|
||||
CAPTURE_SKIP_HOST_TELEMETRY=false
|
||||
SKIP_CAPTURE=false
|
||||
|
||||
APPLY_CMD=""
|
||||
APPLY_CMD_ARR=()
|
||||
APPLY_WAIT_SECS=20
|
||||
|
||||
BASE_OBJECT_IO_BUFFER_SIZE=262144
|
||||
BASE_OBJECT_DUPLEX_BUFFER_SIZE=8388608
|
||||
BASE_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
|
||||
BASE_RUNTIME_WORKER_THREADS=12
|
||||
BASE_RUNTIME_MAX_BLOCKING_THREADS=512
|
||||
|
||||
usage() {
|
||||
cat <<'USAGE'
|
||||
Usage:
|
||||
scripts/run_put_large_tuning_matrix.sh --endpoint <url> \
|
||||
--access-key <ak> --secret-key <sk> [options]
|
||||
|
||||
Required:
|
||||
--endpoint <url>
|
||||
--access-key <ak>
|
||||
--secret-key <sk>
|
||||
|
||||
Core options:
|
||||
--group <name> all|baseline|inflight|io-buffer|runtime|duplex
|
||||
--bucket-prefix <prefix> Default: rustfs-put-large
|
||||
--region <name> Default: us-east-1
|
||||
--concurrencies <csv> Default: 16,32,64,96,128
|
||||
--sizes <csv> Default: 16MiB,32MiB
|
||||
--duration <dur> Default: 120s
|
||||
--rounds <n> Default: 3
|
||||
--out-root <dir> Default: target/bench/put-large-tuning-<timestamp>
|
||||
--warp-bin <path> Default: warp
|
||||
--extra-args "<args>"
|
||||
--cooldown-secs <n> Sleep between profiles (default: 30)
|
||||
--insecure
|
||||
--dry-run
|
||||
|
||||
Topology metadata:
|
||||
--nodes <n>
|
||||
--disks-per-node <n>
|
||||
--total-disks <n>
|
||||
--cpu-per-node <text>
|
||||
--mem-per-node <text>
|
||||
--network <text>
|
||||
--endpoint-mode <direct|lb>
|
||||
--erasure-set-drive-count <n>
|
||||
--client-host <text>
|
||||
|
||||
Capture options:
|
||||
--capture-metrics-endpoints <csv>
|
||||
--capture-prom-metrics-urls <csv>
|
||||
--capture-interval-secs <n> Default: 15
|
||||
--capture-rustfs-pid <pid>
|
||||
--capture-skip-host-telemetry
|
||||
--skip-capture Reuse benchmark runner without starting capture
|
||||
|
||||
Optional apply hook:
|
||||
--apply-cmd "<cmd>" Optional plain command + args, no shell operators.
|
||||
Useful for restarting/reloading RustFS after env changes.
|
||||
--apply-wait-secs <n> Wait after apply command (default: 20)
|
||||
|
||||
Behavior:
|
||||
- Each profile writes: <out-root>/<profile>/env_snapshot.env
|
||||
- Each profile then calls scripts/run_put_large_stage_breakdown_with_capture.sh
|
||||
- Non-baseline profiles automatically compare against <out-root>/baseline
|
||||
USAGE
|
||||
}
|
||||
|
||||
require_cmd() {
|
||||
if ! command -v "$1" >/dev/null 2>&1; then
|
||||
echo "ERROR: command not found: $1" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
arg_value() {
|
||||
local flag="$1"
|
||||
local value="${2:-}"
|
||||
if [[ -z "$value" || "$value" == --* ]]; then
|
||||
echo "ERROR: missing value for $flag" >&2
|
||||
exit 1
|
||||
fi
|
||||
printf '%s\n' "$value"
|
||||
}
|
||||
|
||||
parse_apply_cmd() {
|
||||
local raw="$1"
|
||||
if [[ "$raw" == *';'* || "$raw" == *'&&'* || "$raw" == *'||'* || "$raw" == *'|'* || "$raw" == *'<'* || "$raw" == *'>'* || "$raw" == *'`'* || "$raw" == *'$'* ]]; then
|
||||
echo "ERROR: --apply-cmd does not allow shell operators or expansions; pass a plain command and args only" >&2
|
||||
exit 1
|
||||
fi
|
||||
IFS=$' \t\n' read -r -a APPLY_CMD_ARR <<< "$raw"
|
||||
if [[ "${#APPLY_CMD_ARR[@]}" -eq 0 ]]; then
|
||||
echo "ERROR: --apply-cmd must not be empty" >&2
|
||||
exit 1
|
||||
fi
|
||||
require_cmd "${APPLY_CMD_ARR[0]}"
|
||||
}
|
||||
|
||||
parse_args() {
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--endpoint) ENDPOINT="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--access-key) ACCESS_KEY="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--secret-key) SECRET_KEY="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--group) GROUP="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--bucket-prefix) BUCKET_PREFIX="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--region) REGION="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--concurrencies) CONCURRENCIES="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--sizes) SIZES="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--duration) DURATION="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--rounds) ROUNDS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--out-root) OUT_ROOT="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--warp-bin) WARP_BIN="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--extra-args) EXTRA_ARGS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--cooldown-secs) COOLDOWN_SECS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--nodes) TOPOLOGY_NODES="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--disks-per-node) TOPOLOGY_DISKS_PER_NODE="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--total-disks) TOPOLOGY_TOTAL_DISKS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--cpu-per-node) TOPOLOGY_CPU_PER_NODE="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--mem-per-node) TOPOLOGY_MEM_PER_NODE="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--network) TOPOLOGY_NETWORK="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--endpoint-mode) TOPOLOGY_ENDPOINT_MODE="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--erasure-set-drive-count) TOPOLOGY_ERASURE_SET_DRIVE_COUNT="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--client-host) CLIENT_HOST="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--capture-metrics-endpoints) CAPTURE_METRICS_ENDPOINTS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--capture-prom-metrics-urls) CAPTURE_PROM_METRICS_URLS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--capture-interval-secs) CAPTURE_INTERVAL_SECS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--capture-rustfs-pid) CAPTURE_RUSTFS_PID="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--capture-skip-host-telemetry) CAPTURE_SKIP_HOST_TELEMETRY=true; shift ;;
|
||||
--skip-capture) SKIP_CAPTURE=true; shift ;;
|
||||
--apply-cmd) APPLY_CMD="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--apply-wait-secs) APPLY_WAIT_SECS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
|
||||
--insecure) INSECURE=true; shift ;;
|
||||
--dry-run) DRY_RUN=true; shift ;;
|
||||
-h|--help) usage; exit 0 ;;
|
||||
*)
|
||||
echo "ERROR: unknown arg: $1" >&2
|
||||
usage
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
}
|
||||
|
||||
is_positive_int() {
|
||||
[[ "$1" =~ ^[1-9][0-9]*$ ]]
|
||||
}
|
||||
|
||||
validate_args() {
|
||||
if [[ -z "$ENDPOINT" || -z "$ACCESS_KEY" || -z "$SECRET_KEY" ]]; then
|
||||
echo "ERROR: --endpoint, --access-key, and --secret-key are required" >&2
|
||||
exit 1
|
||||
fi
|
||||
case "$GROUP" in
|
||||
all|baseline|inflight|io-buffer|runtime|duplex) ;;
|
||||
*) echo "ERROR: --group must be all|baseline|inflight|io-buffer|runtime|duplex" >&2; exit 1 ;;
|
||||
esac
|
||||
if ! is_positive_int "$ROUNDS" || ! is_positive_int "$CAPTURE_INTERVAL_SECS" || ! is_positive_int "$APPLY_WAIT_SECS" || ! [[ "$COOLDOWN_SECS" =~ ^[0-9]+$ ]]; then
|
||||
echo "ERROR: --rounds, --capture-interval-secs, and --apply-wait-secs must be positive integers; --cooldown-secs must be a nonnegative integer" >&2
|
||||
exit 1
|
||||
fi
|
||||
if [[ -n "$APPLY_CMD" ]]; then
|
||||
parse_apply_cmd "$APPLY_CMD"
|
||||
fi
|
||||
}
|
||||
|
||||
setup_out_root() {
|
||||
if [[ -z "$OUT_ROOT" ]]; then
|
||||
OUT_ROOT="target/bench/put-large-tuning-$(date -u +%Y%m%dT%H%M%SZ)"
|
||||
fi
|
||||
mkdir -p "$OUT_ROOT"
|
||||
}
|
||||
|
||||
apply_profile() {
|
||||
local profile="$1"
|
||||
export RUSTFS_OBJECT_IO_BUFFER_SIZE="$BASE_OBJECT_IO_BUFFER_SIZE"
|
||||
export RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE="$BASE_OBJECT_DUPLEX_BUFFER_SIZE"
|
||||
export RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES="$BASE_ERASURE_ENCODE_MAX_INFLIGHT_BYTES"
|
||||
export RUSTFS_RUNTIME_WORKER_THREADS="$BASE_RUNTIME_WORKER_THREADS"
|
||||
export RUSTFS_RUNTIME_MAX_BLOCKING_THREADS="$BASE_RUNTIME_MAX_BLOCKING_THREADS"
|
||||
|
||||
case "$profile" in
|
||||
baseline) ;;
|
||||
inflight-32m) export RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=33554432 ;;
|
||||
inflight-48m) export RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=50331648 ;;
|
||||
inflight-64m) export RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=67108864 ;;
|
||||
io-buffer-512k) export RUSTFS_OBJECT_IO_BUFFER_SIZE=524288 ;;
|
||||
io-buffer-1m) export RUSTFS_OBJECT_IO_BUFFER_SIZE=1048576 ;;
|
||||
runtime-w16-b768)
|
||||
export RUSTFS_RUNTIME_WORKER_THREADS=16
|
||||
export RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=768
|
||||
;;
|
||||
runtime-w20-b1024)
|
||||
export RUSTFS_RUNTIME_WORKER_THREADS=20
|
||||
export RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1024
|
||||
;;
|
||||
duplex-16m) export RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=16777216 ;;
|
||||
*)
|
||||
echo "ERROR: unsupported profile $profile" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
write_env_snapshot() {
|
||||
local out_file="$1"
|
||||
cat > "$out_file" <<EOF
|
||||
RUSTFS_OBJECT_IO_BUFFER_SIZE=${RUSTFS_OBJECT_IO_BUFFER_SIZE}
|
||||
RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=${RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE}
|
||||
RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=${RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES}
|
||||
RUSTFS_RUNTIME_WORKER_THREADS=${RUSTFS_RUNTIME_WORKER_THREADS}
|
||||
RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=${RUSTFS_RUNTIME_MAX_BLOCKING_THREADS}
|
||||
EOF
|
||||
}
|
||||
|
||||
run_apply_hook_if_needed() {
|
||||
local profile="$1"
|
||||
local env_file="$2"
|
||||
if [[ "${#APPLY_CMD_ARR[@]}" -eq 0 ]]; then
|
||||
return
|
||||
fi
|
||||
echo "[${profile}] running apply command..."
|
||||
if [[ "$DRY_RUN" == "true" ]]; then
|
||||
printf '[DRY-RUN] RUSTFS_TUNING_ENV_FILE=%q ' "$env_file"
|
||||
printf '%q ' "${APPLY_CMD_ARR[@]}"
|
||||
printf '\n'
|
||||
echo "[DRY-RUN] sleep $APPLY_WAIT_SECS"
|
||||
else
|
||||
RUSTFS_TUNING_ENV_FILE="$env_file" "${APPLY_CMD_ARR[@]}"
|
||||
echo "[${profile}] waiting ${APPLY_WAIT_SECS}s for service readiness..."
|
||||
sleep "$APPLY_WAIT_SECS"
|
||||
fi
|
||||
}
|
||||
|
||||
profiles_for_group() {
|
||||
case "$1" in
|
||||
baseline) echo "baseline" ;;
|
||||
inflight) echo "baseline inflight-32m inflight-48m inflight-64m" ;;
|
||||
io-buffer) echo "baseline io-buffer-512k io-buffer-1m" ;;
|
||||
runtime) echo "baseline runtime-w16-b768 runtime-w20-b1024" ;;
|
||||
duplex) echo "baseline duplex-16m" ;;
|
||||
all) echo "baseline inflight-32m inflight-48m inflight-64m io-buffer-512k io-buffer-1m runtime-w16-b768 runtime-w20-b1024 duplex-16m" ;;
|
||||
esac
|
||||
}
|
||||
|
||||
run_profile() {
|
||||
local profile="$1"
|
||||
local out_dir env_file baseline_root
|
||||
out_dir="${OUT_ROOT}/${profile}"
|
||||
mkdir -p "$out_dir"
|
||||
|
||||
apply_profile "$profile"
|
||||
env_file="${out_dir}/env_snapshot.env"
|
||||
write_env_snapshot "$env_file"
|
||||
run_apply_hook_if_needed "$profile" "$env_file"
|
||||
|
||||
baseline_root=""
|
||||
if [[ "$profile" != "baseline" ]]; then
|
||||
baseline_root="${OUT_ROOT}/baseline"
|
||||
fi
|
||||
|
||||
local -a cmd=(
|
||||
bash "$RUNNER_SCRIPT"
|
||||
--endpoint "$ENDPOINT"
|
||||
--access-key "$ACCESS_KEY"
|
||||
--secret-key "$SECRET_KEY"
|
||||
--bucket-prefix "${BUCKET_PREFIX}-${profile}"
|
||||
--region "$REGION"
|
||||
--concurrencies "$CONCURRENCIES"
|
||||
--sizes "$SIZES"
|
||||
--duration "$DURATION"
|
||||
--rounds "$ROUNDS"
|
||||
--out-dir "$out_dir"
|
||||
--warp-bin "$WARP_BIN"
|
||||
)
|
||||
|
||||
if [[ -n "$baseline_root" ]]; then
|
||||
cmd+=(--baseline-root "$baseline_root")
|
||||
fi
|
||||
if [[ -n "$EXTRA_ARGS" ]]; then
|
||||
cmd+=(--extra-args "$EXTRA_ARGS")
|
||||
fi
|
||||
if [[ -n "$TOPOLOGY_NODES" ]]; then
|
||||
cmd+=(--nodes "$TOPOLOGY_NODES")
|
||||
fi
|
||||
if [[ -n "$TOPOLOGY_DISKS_PER_NODE" ]]; then
|
||||
cmd+=(--disks-per-node "$TOPOLOGY_DISKS_PER_NODE")
|
||||
fi
|
||||
if [[ -n "$TOPOLOGY_TOTAL_DISKS" ]]; then
|
||||
cmd+=(--total-disks "$TOPOLOGY_TOTAL_DISKS")
|
||||
fi
|
||||
if [[ -n "$TOPOLOGY_CPU_PER_NODE" ]]; then
|
||||
cmd+=(--cpu-per-node "$TOPOLOGY_CPU_PER_NODE")
|
||||
fi
|
||||
if [[ -n "$TOPOLOGY_MEM_PER_NODE" ]]; then
|
||||
cmd+=(--mem-per-node "$TOPOLOGY_MEM_PER_NODE")
|
||||
fi
|
||||
if [[ -n "$TOPOLOGY_NETWORK" ]]; then
|
||||
cmd+=(--network "$TOPOLOGY_NETWORK")
|
||||
fi
|
||||
if [[ -n "$TOPOLOGY_ENDPOINT_MODE" ]]; then
|
||||
cmd+=(--endpoint-mode "$TOPOLOGY_ENDPOINT_MODE")
|
||||
fi
|
||||
if [[ -n "$TOPOLOGY_ERASURE_SET_DRIVE_COUNT" ]]; then
|
||||
cmd+=(--erasure-set-drive-count "$TOPOLOGY_ERASURE_SET_DRIVE_COUNT")
|
||||
fi
|
||||
if [[ -n "$CLIENT_HOST" ]]; then
|
||||
cmd+=(--client-host "$CLIENT_HOST")
|
||||
fi
|
||||
if [[ -n "$CAPTURE_METRICS_ENDPOINTS" ]]; then
|
||||
cmd+=(--capture-metrics-endpoints "$CAPTURE_METRICS_ENDPOINTS")
|
||||
fi
|
||||
if [[ -n "$CAPTURE_PROM_METRICS_URLS" ]]; then
|
||||
cmd+=(--capture-prom-metrics-urls "$CAPTURE_PROM_METRICS_URLS")
|
||||
fi
|
||||
if [[ -n "$CAPTURE_RUSTFS_PID" ]]; then
|
||||
cmd+=(--capture-rustfs-pid "$CAPTURE_RUSTFS_PID")
|
||||
fi
|
||||
if [[ "$CAPTURE_SKIP_HOST_TELEMETRY" == "true" ]]; then
|
||||
cmd+=(--capture-skip-host-telemetry)
|
||||
fi
|
||||
if [[ "$SKIP_CAPTURE" == "true" ]]; then
|
||||
cmd+=(--skip-capture)
|
||||
fi
|
||||
if [[ "$INSECURE" == "true" ]]; then
|
||||
cmd+=(--insecure)
|
||||
fi
|
||||
if [[ "$DRY_RUN" == "true" ]]; then
|
||||
cmd+=(--dry-run)
|
||||
fi
|
||||
|
||||
echo
|
||||
echo "===== Running profile ${profile} ====="
|
||||
echo "Output: $out_dir"
|
||||
echo "Env snapshot: $env_file"
|
||||
if [[ "$DRY_RUN" == "true" ]]; then
|
||||
printf '[DRY-RUN] '
|
||||
printf '%q ' "${cmd[@]}"
|
||||
printf '\n'
|
||||
else
|
||||
"${cmd[@]}"
|
||||
fi
|
||||
}
|
||||
|
||||
main() {
|
||||
parse_args "$@"
|
||||
validate_args
|
||||
require_cmd bash
|
||||
require_cmd awk
|
||||
if [[ ! -x "$RUNNER_SCRIPT" && ! -f "$RUNNER_SCRIPT" ]]; then
|
||||
echo "ERROR: missing dependency script: $RUNNER_SCRIPT" >&2
|
||||
exit 1
|
||||
fi
|
||||
setup_out_root
|
||||
|
||||
echo "Tuning output root: $OUT_ROOT"
|
||||
echo "Group: $GROUP"
|
||||
echo "Concurrencies: $CONCURRENCIES"
|
||||
echo "Sizes: $SIZES"
|
||||
echo "Cooldown between profiles: ${COOLDOWN_SECS}s"
|
||||
|
||||
local profiles profile index total_profiles
|
||||
profiles=($(profiles_for_group "$GROUP"))
|
||||
total_profiles="${#profiles[@]}"
|
||||
for ((index = 0; index < total_profiles; index++)); do
|
||||
profile="${profiles[$index]}"
|
||||
run_profile "$profile"
|
||||
if (( index + 1 < total_profiles )) && (( COOLDOWN_SECS > 0 )); then
|
||||
echo
|
||||
echo "[${profile}] cooldown ${COOLDOWN_SECS}s before next profile..."
|
||||
if [[ "$DRY_RUN" == "true" ]]; then
|
||||
echo "[DRY-RUN] sleep ${COOLDOWN_SECS}"
|
||||
else
|
||||
sleep "$COOLDOWN_SECS"
|
||||
fi
|
||||
fi
|
||||
done
|
||||
|
||||
echo
|
||||
echo "Done. Profile outputs are under: $OUT_ROOT"
|
||||
}
|
||||
|
||||
main "$@"
|
||||
Executable
+38
@@ -0,0 +1,38 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
# Setup test binaries for Docker build testing
|
||||
# This script creates temporary binary files for testing Docker build process
|
||||
|
||||
set -e
|
||||
|
||||
echo "Setting up test binaries for Docker build..."
|
||||
|
||||
# Create temporary rustfs binary
|
||||
./build-rustfs.sh -p x86_64-unknown-linux-gnu
|
||||
|
||||
# Create test directory structure
|
||||
mkdir -p test-releases/server/rustfs/release/linux-amd64/archive
|
||||
mkdir -p test-releases/server/rustfs/release/linux-arm64/archive
|
||||
|
||||
# Get version
|
||||
VERSION=$(git describe --abbrev=0 --tags 2>/dev/null || git rev-parse --short HEAD)
|
||||
|
||||
# Copy binaries
|
||||
cp target/x86_64-unknown-linux-gnu/release/rustfs test-releases/server/rustfs/release/linux-amd64/archive/rustfs.${VERSION}
|
||||
cp target/x86_64-unknown-linux-gnu/release/rustfs.sha256sum test-releases/server/rustfs/release/linux-amd64/archive/rustfs.${VERSION}.sha256sum
|
||||
|
||||
# Create dummy signatures
|
||||
echo "dummy signature" > test-releases/server/rustfs/release/linux-amd64/archive/rustfs.${VERSION}.minisig
|
||||
echo "dummy signature" > test-releases/server/rustfs/release/linux-arm64/archive/rustfs.${VERSION}.minisig
|
||||
|
||||
# Also copy for arm64 (using same binary for testing)
|
||||
cp target/aarch64-unknown-linux-gnu/release/rustfs test-releases/server/rustfs/release/linux-arm64/archive/rustfs.${VERSION}
|
||||
cp target/aarch64-unknown-linux-gnu/release/rustfs.sha256sum test-releases/server/rustfs/release/linux-arm64/archive/rustfs.${VERSION}.sha256sum
|
||||
|
||||
echo "Test binaries created for version: ${VERSION}"
|
||||
echo "You can now test Docker builds with these local binaries"
|
||||
echo ""
|
||||
echo "To start a local HTTP server for testing:"
|
||||
echo " cd test-releases && python3 -m http.server 8000"
|
||||
echo ""
|
||||
echo "Then modify Dockerfile to use http://host.docker.internal:8000 instead of https://dl.rustfs.com/artifacts/rustfs"
|
||||
+242
@@ -0,0 +1,242 @@
|
||||
#!/usr/bin/env bash
|
||||
# GET Optimization Stress Test Script
|
||||
# Usage: ./scripts/stress-test-get-optimization.sh [TARGET_HOST] [OUTPUT_DIR]
|
||||
#
|
||||
# Validates:
|
||||
# 1. Data correctness with early-stop enabled
|
||||
# 2. Concurrent GET performance
|
||||
# 3. Mixed read/write stability
|
||||
# 4. Early-stop behavior under slow disk conditions
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
TARGET_HOST="${1:-localhost:9000}"
|
||||
OUTPUT_DIR="${2:-./stress-test-results/$(date +%Y%m%d-%H%M%S)}"
|
||||
MC_ALIAS="${MC_ALIAS:-rustfs}"
|
||||
TEST_BUCKET="${TEST_BUCKET:-stress-test-$(date +%s)}"
|
||||
TEST_DURATION="${TEST_DURATION:-300s}"
|
||||
CONCURRENCY="${CONCURRENCY:-64}"
|
||||
|
||||
# S3 credentials
|
||||
export WARP_ACCESS_KEY="${WARP_ACCESS_KEY:-rustfsadmin}"
|
||||
export WARP_SECRET_KEY="${WARP_SECRET_KEY:-rustfsadmin}"
|
||||
|
||||
mkdir -p "$OUTPUT_DIR"
|
||||
|
||||
echo "=========================================="
|
||||
echo "GET Optimization Stress Test"
|
||||
echo "=========================================="
|
||||
echo "Target: $TARGET_HOST"
|
||||
echo "Output: $OUTPUT_DIR"
|
||||
echo "Bucket: $TEST_BUCKET"
|
||||
echo "Duration: $TEST_DURATION"
|
||||
echo "Concurrency: $CONCURRENCY"
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
|
||||
# Save test configuration
|
||||
cat > "$OUTPUT_DIR/test-config.txt" <<EOF
|
||||
Date: $(date -u +"%Y-%m-%dT%H:%M:%SZ")
|
||||
Target: $TARGET_HOST
|
||||
Bucket: $TEST_BUCKET
|
||||
Duration: $TEST_DURATION
|
||||
Concurrency: $CONCURRENCY
|
||||
|
||||
Environment Variables:
|
||||
RUSTFS_GET_METADATA_EARLY_STOP_ENABLE=${RUSTFS_GET_METADATA_EARLY_STOP_ENABLE:-true}
|
||||
RUSTFS_GET_CODEC_STREAMING_ENABLE=${RUSTFS_GET_CODEC_STREAMING_ENABLE:-false}
|
||||
RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE=${RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE:-true}
|
||||
EOF
|
||||
|
||||
# ============================================================================
|
||||
# Test 1: Data Correctness Validation
|
||||
# ============================================================================
|
||||
echo "[1/4] Data Correctness Validation"
|
||||
echo "=================================="
|
||||
|
||||
test_data_correctness() {
|
||||
local size=$1
|
||||
local count=$2
|
||||
local passed=0
|
||||
local failed=0
|
||||
|
||||
echo " Testing $count objects of size $size..."
|
||||
|
||||
for i in $(seq 1 $count); do
|
||||
# Generate test file
|
||||
local testfile="/tmp/stress-test-${size}-${i}.bin"
|
||||
if [ "$size" = "1KB" ]; then
|
||||
dd if=/dev/urandom of="$testfile" bs=1024 count=1 2>/dev/null
|
||||
elif [ "$size" = "1MB" ]; then
|
||||
dd if=/dev/urandom of="$testfile" bs=1048576 count=1 2>/dev/null
|
||||
elif [ "$size" = "10MB" ]; then
|
||||
dd if=/dev/urandom of="$testfile" bs=1048576 count=10 2>/dev/null
|
||||
fi
|
||||
|
||||
# Calculate original hash
|
||||
local original_hash
|
||||
original_hash=$(md5 -q "$testfile" 2>/dev/null || md5sum "$testfile" | awk '{print $1}')
|
||||
|
||||
# Upload
|
||||
mc cp "$testfile" "${MC_ALIAS}/${TEST_BUCKET}/correctness-${size}-${i}.bin" >/dev/null 2>&1
|
||||
|
||||
# Download and verify
|
||||
local downloadfile="/tmp/stress-test-${size}-${i}-download.bin"
|
||||
mc cp "${MC_ALIAS}/${TEST_BUCKET}/correctness-${size}-${i}.bin" "$downloadfile" >/dev/null 2>&1
|
||||
|
||||
local download_hash
|
||||
download_hash=$(md5 -q "$downloadfile" 2>/dev/null || md5sum "$downloadfile" | awk '{print $1}')
|
||||
|
||||
if [ "$original_hash" = "$download_hash" ]; then
|
||||
passed=$((passed + 1))
|
||||
else
|
||||
failed=$((failed + 1))
|
||||
echo " MISMATCH: correctness-${size}-${i}.bin (original=$original_hash, download=$download_hash)" >> "$OUTPUT_DIR/correctness-errors.log"
|
||||
fi
|
||||
|
||||
# Cleanup
|
||||
rm -f "$testfile" "$downloadfile"
|
||||
done
|
||||
|
||||
echo " Result: $passed passed, $failed failed"
|
||||
echo "correctness_${size}: passed=$passed failed=$failed" >> "$OUTPUT_DIR/correctness-results.txt"
|
||||
}
|
||||
|
||||
# Test different object sizes
|
||||
mc mb "${MC_ALIAS}/${TEST_BUCKET}" 2>/dev/null || true
|
||||
|
||||
test_data_correctness "1KB" 100
|
||||
test_data_correctness "1MB" 50
|
||||
test_data_correctness "10MB" 10
|
||||
|
||||
echo ""
|
||||
|
||||
# ============================================================================
|
||||
# Test 2: Concurrent GET Stress Test
|
||||
# ============================================================================
|
||||
echo "[2/4] Concurrent GET Stress Test"
|
||||
echo "================================="
|
||||
|
||||
# Prepare test objects
|
||||
echo " Preparing test objects..."
|
||||
for size in 1KiB 1MiB 4MiB 10MiB; do
|
||||
warp put --obj.size="$size" --num.objects=100 --host="$TARGET_HOST" --bucket="$TEST_BUCKET" --concurrent=16 >/dev/null 2>&1
|
||||
done
|
||||
|
||||
echo " Running concurrent GET tests..."
|
||||
|
||||
for size in 1KiB 1MiB 4MiB 10MiB; do
|
||||
for conc in 16 64 256; do
|
||||
echo " GET size=$size concurrency=$conc duration=$TEST_DURATION"
|
||||
output_file="$OUTPUT_DIR/get-${size}-c${conc}.json"
|
||||
|
||||
if warp get \
|
||||
--host="$TARGET_HOST" \
|
||||
--obj.size="$size" \
|
||||
--concurrent="$conc" \
|
||||
--duration="$TEST_DURATION" \
|
||||
--bucket="$TEST_BUCKET" \
|
||||
--json \
|
||||
> "$output_file" 2>/dev/null; then
|
||||
echo " OK"
|
||||
else
|
||||
echo " FAILED (see $output_file)"
|
||||
fi
|
||||
done
|
||||
done
|
||||
|
||||
echo ""
|
||||
|
||||
# ============================================================================
|
||||
# Test 3: Mixed Read/Write Stress Test
|
||||
# ============================================================================
|
||||
echo "[3/4] Mixed Read/Write Stress Test"
|
||||
echo "==================================="
|
||||
|
||||
echo " Running mixed workload for $TEST_DURATION..."
|
||||
|
||||
warp mixed \
|
||||
--host="$TARGET_HOST" \
|
||||
--obj.size=1MiB \
|
||||
--concurrent="$CONCURRENCY" \
|
||||
--duration="$TEST_DURATION" \
|
||||
--bucket="$TEST_BUCKET" \
|
||||
--json \
|
||||
> "$OUTPUT_DIR/mixed-1MiB-c${CONCURRENCY}.json" 2>/dev/null || echo " MIXED TEST FAILED"
|
||||
|
||||
echo " Mixed test complete"
|
||||
echo ""
|
||||
|
||||
# ============================================================================
|
||||
# Test 4: Early-Stop Behavior Validation
|
||||
# ============================================================================
|
||||
echo "[4/4] Early-Stop Behavior Validation"
|
||||
echo "====================================="
|
||||
|
||||
# This test verifies that early-stop works correctly by checking:
|
||||
# 1. All GET requests return correct data
|
||||
# 2. No timeouts or errors under normal conditions
|
||||
# 3. Performance is consistent
|
||||
|
||||
echo " Running early-stop validation with concurrent reads..."
|
||||
|
||||
# Create a test object
|
||||
dd if=/dev/urandom of=/tmp/early-stop-test.bin bs=1048576 count=10 2>/dev/null
|
||||
mc cp /tmp/early-stop-test.bin "${MC_ALIAS}/${TEST_BUCKET}/early-stop-test.bin" >/dev/null 2>&1
|
||||
|
||||
# Concurrent read test
|
||||
local_passed=0
|
||||
local_failed=0
|
||||
|
||||
for i in $(seq 1 100); do
|
||||
downloadfile="/tmp/early-stop-download-${i}.bin"
|
||||
if mc cp "${MC_ALIAS}/${TEST_BUCKET}/early-stop-test.bin" "$downloadfile" >/dev/null 2>&1; then
|
||||
# Verify file size
|
||||
local_size=$(stat -f%z "$downloadfile" 2>/dev/null || stat -c%s "$downloadfile" 2>/dev/null)
|
||||
if [ "$local_size" = "10485760" ]; then
|
||||
local_passed=$((local_passed + 1))
|
||||
else
|
||||
local_failed=$((local_failed + 1))
|
||||
echo " SIZE MISMATCH: expected 10485760, got $local_size" >> "$OUTPUT_DIR/early-stop-errors.log"
|
||||
fi
|
||||
else
|
||||
local_failed=$((local_failed + 1))
|
||||
echo " DOWNLOAD FAILED: iteration $i" >> "$OUTPUT_DIR/early-stop-errors.log"
|
||||
fi
|
||||
rm -f "$downloadfile"
|
||||
done
|
||||
|
||||
echo " Early-stop validation: $local_passed passed, $local_failed failed"
|
||||
echo "early_stop_validation: passed=$local_passed failed=$local_failed" >> "$OUTPUT_DIR/early-stop-results.txt"
|
||||
|
||||
# Cleanup
|
||||
rm -f /tmp/early-stop-test.bin
|
||||
mc rm "${MC_ALIAS}/${TEST_BUCKET}/early-stop-test.bin" >/dev/null 2>&1 || true
|
||||
|
||||
echo ""
|
||||
|
||||
# ============================================================================
|
||||
# Summary
|
||||
# ============================================================================
|
||||
echo "=========================================="
|
||||
echo "Stress Test Summary"
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
echo "Results saved to: $OUTPUT_DIR"
|
||||
echo ""
|
||||
echo "Files:"
|
||||
ls -la "$OUTPUT_DIR"
|
||||
echo ""
|
||||
echo "Review:"
|
||||
echo " - correctness-results.txt: Data correctness validation"
|
||||
echo " - early-stop-results.txt: Early-stop behavior validation"
|
||||
echo " - get-*.json: Concurrent GET performance results"
|
||||
echo " - mixed-*.json: Mixed workload results"
|
||||
echo ""
|
||||
|
||||
# Cleanup test bucket
|
||||
echo "Cleaning up test bucket..."
|
||||
mc rm --recursive --force "${MC_ALIAS}/${TEST_BUCKET}" >/dev/null 2>&1 || true
|
||||
mc rb "${MC_ALIAS}/${TEST_BUCKET}" >/dev/null 2>&1 || true
|
||||
|
||||
echo "Done!"
|
||||
Executable
+143
@@ -0,0 +1,143 @@
|
||||
#!/usr/bin/env bash
|
||||
# Copyright 2024 RustFS Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
|
||||
mkdir test
|
||||
|
||||
cd test
|
||||
|
||||
# make_bucket
|
||||
mc mb rustfs/mbmb
|
||||
mc ls rustfs
|
||||
mc rb rustfs/mbmb
|
||||
|
||||
mc mb rustfs/rb-force
|
||||
mc ls rustfs
|
||||
echo "123525" >> test.txt
|
||||
mc cp test.txt rustfs/rb-force
|
||||
|
||||
mc rb --force rustfs/rb-force
|
||||
rm test.txt
|
||||
|
||||
|
||||
mc mb rustfs/dada
|
||||
|
||||
echo "123525" >> test.txt
|
||||
mc cp test.txt rustfs/dada
|
||||
|
||||
mc get rustfs/dada/test.txt test2.txt
|
||||
|
||||
diff test.txt test2.txt
|
||||
|
||||
echo "33333" >> test2.txt
|
||||
|
||||
mc cp test2.txt rustfs/dada
|
||||
|
||||
mc get rustfs/dada/test2.txt test3.txt
|
||||
|
||||
diff test2.txt test3.txt
|
||||
|
||||
# list_buckets
|
||||
mc ls rustfs/dada
|
||||
|
||||
mc mb rustfs/dada2
|
||||
|
||||
mc ls rustfs
|
||||
|
||||
mc ls rustfs/dada
|
||||
|
||||
mc rm rustfs/dada/test2.txt
|
||||
|
||||
mc ls rustfs/dada
|
||||
|
||||
dd if=/dev/urandom of=50M.file bs=1m count=50
|
||||
|
||||
mc cp 50M.file rustfs/dada
|
||||
|
||||
mc ls rustfs/dada
|
||||
|
||||
|
||||
mc get rustfs/dada/50M.file 50m.file.d
|
||||
|
||||
diff 50M.file 50m.file.d
|
||||
|
||||
mc rm rustfs/dada/50M.file
|
||||
mc ls rustfs/dada
|
||||
|
||||
rm test.txt test2.txt test3.txt
|
||||
rm 50M.file 50m.file.d
|
||||
|
||||
|
||||
# object_tags
|
||||
echo "33333" >> tags.txt
|
||||
mc cp tags.txt rustfs/dada
|
||||
mc tag list rustfs/dada/tags.txt
|
||||
mc tag set rustfs/dada/tags.txt "key1=value1&key2=value2"
|
||||
mc tag list rustfs/dada/tags.txt
|
||||
mc tag remove rustfs/dada/tags.txt
|
||||
mc tag list rustfs/dada/tags.txt
|
||||
rm tags.txt
|
||||
|
||||
# bucket_tags
|
||||
mc tag list rustfs/dada
|
||||
mc tag set rustfs/dada "a=b&b=c&dada=yes&yy=11&77=99&99=23&11=11"
|
||||
mc tag list rustfs/dada
|
||||
mc tag remove rustfs/dada
|
||||
mc tag list rustfs/dada
|
||||
|
||||
|
||||
# bucket_versioning
|
||||
mc version info rustfs/dada
|
||||
mc version enable rustfs/dada
|
||||
mc version info rustfs/dada
|
||||
mc version suspend rustfs/dada
|
||||
mc version info rustfs/dada
|
||||
|
||||
# bucket_policy
|
||||
mc anonymous get rustfs/dada
|
||||
mc anonymous set public rustfs/dada
|
||||
mc anonymous set upload rustfs/dada
|
||||
mc anonymous set download rustfs/dada
|
||||
mc anonymous list rustfs/dada
|
||||
|
||||
# lifecycle
|
||||
mc ilm ls rustfs/dada
|
||||
mc ilm rule add --expire-days 90 --noncurrent-expire-days 30 rustfs/dada
|
||||
mc ilm ls rustfs/dada
|
||||
|
||||
# bucket_encryption
|
||||
mc encrypt info rustfs/dada
|
||||
mc encrypt set sse-kms rustfs-encryption-key rustfs/dada
|
||||
mc encrypt info rustfs/dada
|
||||
mc encrypt clear rustfs/dada
|
||||
|
||||
# object_lock_config
|
||||
mc mb --with-lock rustfs/lock
|
||||
mc retention info --default rustfs/lock
|
||||
mc retention set --default GOVERNANCE "30d" rustfs/lock
|
||||
mc retention info --default rustfs/lock
|
||||
mc retention clear --default rustfs/lock
|
||||
mc rb rustfs/lock
|
||||
|
||||
# bucket_notification
|
||||
mc event list rustfs/dada
|
||||
mc event add --event "put,delete" rustfs/dada arn:aws:sqs::primary:target
|
||||
mc event list rustfs/dada
|
||||
mc event rm --event "put,delete" rustfs/dada arn:aws:sqs::primary:target
|
||||
mc event list rustfs/dada
|
||||
|
||||
# bucket_quota ? admin/v3/get-bucket-quota
|
||||
# mc quota info rustfs/dada
|
||||
# bucket_target ?
|
||||
@@ -0,0 +1,59 @@
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": [
|
||||
"s3:GetObject"
|
||||
],
|
||||
"Resource": [
|
||||
"arn:aws:s3:::dada/*"
|
||||
],
|
||||
"Condition": {
|
||||
"StringEquals": {
|
||||
"s3:ExistingObjectTag/security": "public"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": [
|
||||
"s3:DeleteObjectTagging"
|
||||
],
|
||||
"Resource": [
|
||||
"arn:aws:s3:::dada/*"
|
||||
],
|
||||
"Condition": {
|
||||
"StringEquals": {
|
||||
"s3:ExistingObjectTag/security": "public"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": [
|
||||
"s3:DeleteObject"
|
||||
],
|
||||
"Resource": [
|
||||
"arn:aws:s3:::dada/*"
|
||||
]
|
||||
},
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": [
|
||||
"s3:PutObject"
|
||||
],
|
||||
"Resource": [
|
||||
"arn:aws:s3:::dada/*"
|
||||
],
|
||||
"Condition": {
|
||||
"ForAllValues:StringLike": {
|
||||
"s3:RequestObjectTagKeys": [
|
||||
"security",
|
||||
"virus"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
+177
@@ -0,0 +1,177 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
PROJECT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
COMPOSE_FILE="${COMPOSE_FILE:-docker-compose-simple.yml}"
|
||||
WAIT_TIMEOUT_SECS="${WAIT_TIMEOUT_SECS:-120}"
|
||||
KEEP_UP="${KEEP_UP:-false}"
|
||||
RUN_S3_TESTS="${RUN_S3_TESTS:-true}"
|
||||
BUILD_LOCAL_IMAGE="${BUILD_LOCAL_IMAGE:-true}"
|
||||
S3_HOST="${S3_HOST:-127.0.0.1}"
|
||||
S3_PORT="${S3_PORT:-9000}"
|
||||
|
||||
usage() {
|
||||
cat <<'USAGE'
|
||||
Usage:
|
||||
scripts/validate_issue_1365_docker.sh [options]
|
||||
|
||||
Options:
|
||||
--compose-file <path> docker compose file (default: docker-compose-simple.yml)
|
||||
--wait-timeout <secs> health wait timeout (default: 120)
|
||||
--keep-up keep compose services up after the script exits
|
||||
--skip-s3-tests skip scripts/s3-tests/run.sh
|
||||
--skip-build skip local Dockerfile.source image build
|
||||
-h, --help show help
|
||||
|
||||
Environment:
|
||||
COMPOSE_FILE
|
||||
WAIT_TIMEOUT_SECS
|
||||
KEEP_UP
|
||||
RUN_S3_TESTS
|
||||
BUILD_LOCAL_IMAGE
|
||||
S3_HOST
|
||||
S3_PORT
|
||||
USAGE
|
||||
}
|
||||
|
||||
log_info() {
|
||||
printf '[INFO] %s\n' "$*"
|
||||
}
|
||||
|
||||
log_error() {
|
||||
printf '[ERROR] %s\n' "$*" >&2
|
||||
}
|
||||
|
||||
require_cmd() {
|
||||
if ! command -v "$1" >/dev/null 2>&1; then
|
||||
log_error "command not found: $1"
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
compose() {
|
||||
local compose_path
|
||||
compose_path="$(resolve_compose_file)"
|
||||
docker compose -f "${compose_path}" "$@"
|
||||
}
|
||||
|
||||
resolve_compose_file() {
|
||||
if [[ "${COMPOSE_FILE}" = /* ]]; then
|
||||
printf '%s\n' "${COMPOSE_FILE}"
|
||||
else
|
||||
printf '%s\n' "${PROJECT_ROOT}/${COMPOSE_FILE}"
|
||||
fi
|
||||
}
|
||||
|
||||
cleanup() {
|
||||
if [[ "${KEEP_UP}" == "true" ]]; then
|
||||
log_info "KEEP_UP=true, leaving compose services running"
|
||||
return
|
||||
fi
|
||||
|
||||
log_info "Stopping docker compose services"
|
||||
compose down -v >/dev/null 2>&1 || true
|
||||
}
|
||||
|
||||
parse_args() {
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--compose-file)
|
||||
COMPOSE_FILE="$2"
|
||||
shift 2
|
||||
;;
|
||||
--wait-timeout)
|
||||
WAIT_TIMEOUT_SECS="$2"
|
||||
shift 2
|
||||
;;
|
||||
--keep-up)
|
||||
KEEP_UP=true
|
||||
shift
|
||||
;;
|
||||
--skip-s3-tests)
|
||||
RUN_S3_TESTS=false
|
||||
shift
|
||||
;;
|
||||
--skip-build)
|
||||
BUILD_LOCAL_IMAGE=false
|
||||
shift
|
||||
;;
|
||||
-h|--help)
|
||||
usage
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
log_error "unknown argument: $1"
|
||||
usage
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
}
|
||||
|
||||
wait_for_endpoint() {
|
||||
local url="$1"
|
||||
local start now
|
||||
start="$(date +%s)"
|
||||
|
||||
while true; do
|
||||
if curl -fsS --connect-timeout 2 --max-time 3 "${url}" >/dev/null 2>&1; then
|
||||
return 0
|
||||
fi
|
||||
|
||||
now="$(date +%s)"
|
||||
if (( now - start >= WAIT_TIMEOUT_SECS )); then
|
||||
log_error "timed out waiting for ${url}"
|
||||
compose ps || true
|
||||
compose logs rustfs --tail 200 || true
|
||||
return 1
|
||||
fi
|
||||
|
||||
sleep 2
|
||||
done
|
||||
}
|
||||
|
||||
main() {
|
||||
parse_args "$@"
|
||||
require_cmd docker
|
||||
require_cmd curl
|
||||
|
||||
trap cleanup EXIT INT TERM
|
||||
|
||||
if [[ "${BUILD_LOCAL_IMAGE}" == "true" ]]; then
|
||||
log_info "Building rustfs/rustfs:latest from Dockerfile.source"
|
||||
docker build -f "${PROJECT_ROOT}/Dockerfile.source" -t rustfs/rustfs:latest "${PROJECT_ROOT}"
|
||||
else
|
||||
log_info "Skipping local image build"
|
||||
fi
|
||||
|
||||
if [[ -z "${RUSTFS_UNSAFE_BYPASS_DISK_CHECK+x}" ]]; then
|
||||
export RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true
|
||||
log_info "RUSTFS_UNSAFE_BYPASS_DISK_CHECK not set; defaulting to true for local validation"
|
||||
fi
|
||||
|
||||
log_info "Starting docker compose from $(resolve_compose_file)"
|
||||
compose up -d
|
||||
|
||||
log_info "Waiting for RustFS health endpoint"
|
||||
wait_for_endpoint "http://${S3_HOST}:${S3_PORT}/health"
|
||||
|
||||
log_info "Waiting for RustFS readiness endpoint"
|
||||
wait_for_endpoint "http://${S3_HOST}:${S3_PORT}/health/ready"
|
||||
|
||||
log_info "Docker health checks passed"
|
||||
|
||||
if [[ "${RUN_S3_TESTS}" == "true" ]]; then
|
||||
log_info "Running S3 compatibility tests against the running dockerized service"
|
||||
(
|
||||
cd "${PROJECT_ROOT}"
|
||||
DEPLOY_MODE=existing S3_HOST="${S3_HOST}" S3_PORT="${S3_PORT}" ./scripts/s3-tests/run.sh
|
||||
)
|
||||
else
|
||||
log_info "Skipping S3 compatibility tests"
|
||||
fi
|
||||
|
||||
log_info "Issue 1365 docker validation completed successfully"
|
||||
}
|
||||
|
||||
main "$@"
|
||||
+298
@@ -0,0 +1,298 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# Issue 2723 verification runner
|
||||
# Validates site-replication behavior against Task 07 matrix/evidence.
|
||||
|
||||
SITE_A_ENDPOINT="${SITE_A_ENDPOINT:-}"
|
||||
SITE_B_ENDPOINT="${SITE_B_ENDPOINT:-}"
|
||||
ACCESS_KEY="${ACCESS_KEY:-rustfsadmin}"
|
||||
SECRET_KEY="${SECRET_KEY:-rustfsadmin}"
|
||||
REGION="${REGION:-us-east-1}"
|
||||
CA_CERT="${CA_CERT:-}"
|
||||
OUT_DIR="${OUT_DIR:-target/verify/issue-2723-$(date +%Y%m%d-%H%M%S)}"
|
||||
SITE_A_RESTART_CMD="${SITE_A_RESTART_CMD:-}"
|
||||
SITE_B_RESTART_CMD="${SITE_B_RESTART_CMD:-}"
|
||||
BUCKET="${BUCKET:-}"
|
||||
REPL_OBJECT_KEY="${REPL_OBJECT_KEY:-issue-2723-e2e-object.txt}"
|
||||
REPL_OBJECT_BODY="${REPL_OBJECT_BODY:-issue-2723-replication-check}"
|
||||
AWS_PROFILE="${AWS_PROFILE:-}"
|
||||
|
||||
AWSCURL_BIN="${AWSCURL_BIN:-awscurl}"
|
||||
AWS_BIN="${AWS_BIN:-aws}"
|
||||
HEALTHCHECK_FORCE_LOOPBACK_RESOLVE="${HEALTHCHECK_FORCE_LOOPBACK_RESOLVE:-false}"
|
||||
|
||||
usage() {
|
||||
cat <<'USAGE'
|
||||
Usage:
|
||||
scripts/validate_issue_2723_site_replication.sh [options]
|
||||
|
||||
Required:
|
||||
--site-a-endpoint <url> Site A admin endpoint, e.g. https://site-a.example.com:9000
|
||||
--site-b-endpoint <url> Site B admin endpoint, e.g. https://site-b.example.com:9000
|
||||
--access-key <ak>
|
||||
--secret-key <sk>
|
||||
|
||||
Optional:
|
||||
--region <name> AWS region for signing (default: us-east-1)
|
||||
--ca-cert <path> CA cert for strict HTTPS health checks
|
||||
--out-dir <dir> Artifact output directory
|
||||
--site-a-restart-cmd <cmd> Restart command for site A (optional)
|
||||
--site-b-restart-cmd <cmd> Restart command for site B (optional)
|
||||
--bucket <name> Replication validation bucket (optional)
|
||||
--repl-object-key <key> Replication validation object key
|
||||
--repl-object-body <text> Replication validation object body
|
||||
--awscurl-bin <path> awscurl binary (default: awscurl)
|
||||
--aws-bin <path> aws cli binary (default: aws)
|
||||
--aws-profile <profile> AWS CLI profile for object-flow checks
|
||||
--healthcheck-force-loopback-resolve
|
||||
Force HTTPS healthcheck `--resolve host:port:127.0.0.1`
|
||||
(default: false; intended for single-host local Docker)
|
||||
-h, --help Show help
|
||||
|
||||
Notes:
|
||||
1) If --bucket is provided and aws cli is available, script will run optional
|
||||
object-flow checks on both sites.
|
||||
2) Restart verification is skipped unless both restart commands are provided.
|
||||
USAGE
|
||||
}
|
||||
|
||||
log() {
|
||||
printf '[%s] %s\n' "$(date +%H:%M:%S)" "$*"
|
||||
}
|
||||
|
||||
fail() {
|
||||
log "ERROR: $*" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
require_cmd() {
|
||||
if ! command -v "$1" >/dev/null 2>&1; then
|
||||
fail "required command not found: $1"
|
||||
fi
|
||||
}
|
||||
|
||||
parse_args() {
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--site-a-endpoint) SITE_A_ENDPOINT="$2"; shift 2 ;;
|
||||
--site-b-endpoint) SITE_B_ENDPOINT="$2"; shift 2 ;;
|
||||
--access-key) ACCESS_KEY="$2"; shift 2 ;;
|
||||
--secret-key) SECRET_KEY="$2"; shift 2 ;;
|
||||
--region) REGION="$2"; shift 2 ;;
|
||||
--ca-cert) CA_CERT="$2"; shift 2 ;;
|
||||
--out-dir) OUT_DIR="$2"; shift 2 ;;
|
||||
--site-a-restart-cmd) SITE_A_RESTART_CMD="$2"; shift 2 ;;
|
||||
--site-b-restart-cmd) SITE_B_RESTART_CMD="$2"; shift 2 ;;
|
||||
--bucket) BUCKET="$2"; shift 2 ;;
|
||||
--repl-object-key) REPL_OBJECT_KEY="$2"; shift 2 ;;
|
||||
--repl-object-body) REPL_OBJECT_BODY="$2"; shift 2 ;;
|
||||
--awscurl-bin) AWSCURL_BIN="$2"; shift 2 ;;
|
||||
--aws-bin) AWS_BIN="$2"; shift 2 ;;
|
||||
--aws-profile) AWS_PROFILE="$2"; shift 2 ;;
|
||||
--healthcheck-force-loopback-resolve) HEALTHCHECK_FORCE_LOOPBACK_RESOLVE="true"; shift ;;
|
||||
-h|--help) usage; exit 0 ;;
|
||||
*)
|
||||
fail "unknown argument: $1"
|
||||
;;
|
||||
esac
|
||||
done
|
||||
}
|
||||
|
||||
endpoint_scheme() {
|
||||
local endpoint="$1"
|
||||
if [[ "$endpoint" == https://* ]]; then
|
||||
echo "https"
|
||||
elif [[ "$endpoint" == http://* ]]; then
|
||||
echo "http"
|
||||
else
|
||||
fail "endpoint must include scheme http:// or https:// : $endpoint"
|
||||
fi
|
||||
}
|
||||
|
||||
endpoint_hostport() {
|
||||
local endpoint="$1"
|
||||
local hostport
|
||||
hostport="${endpoint#http://}"
|
||||
hostport="${hostport#https://}"
|
||||
hostport="${hostport%%/*}"
|
||||
echo "$hostport"
|
||||
}
|
||||
|
||||
admin_get() {
|
||||
local endpoint="$1"
|
||||
local path="$2"
|
||||
local out_file="$3"
|
||||
local url="${endpoint%/}${path}"
|
||||
if [[ -n "$CA_CERT" ]]; then
|
||||
REQUESTS_CA_BUNDLE="$CA_CERT" SSL_CERT_FILE="$CA_CERT" \
|
||||
"$AWSCURL_BIN" --service s3 --region "$REGION" --access_key "$ACCESS_KEY" --secret_key "$SECRET_KEY" "$url" >"$out_file"
|
||||
else
|
||||
"$AWSCURL_BIN" --service s3 --region "$REGION" --access_key "$ACCESS_KEY" --secret_key "$SECRET_KEY" "$url" >"$out_file"
|
||||
fi
|
||||
}
|
||||
|
||||
strict_healthcheck() {
|
||||
local endpoint="$1"
|
||||
local label="$2"
|
||||
local out_file="$3"
|
||||
local scheme hostport host port
|
||||
scheme="$(endpoint_scheme "$endpoint")"
|
||||
hostport="$(endpoint_hostport "$endpoint")"
|
||||
host="${hostport%:*}"
|
||||
port="${hostport##*:}"
|
||||
if [[ "$port" == "$hostport" ]]; then
|
||||
port=$([[ "$scheme" == "https" ]] && echo "443" || echo "80")
|
||||
fi
|
||||
local url="${scheme}://${host}:${port}/health"
|
||||
|
||||
if [[ "$scheme" == "https" ]]; then
|
||||
if [[ -z "$CA_CERT" ]]; then
|
||||
fail "HTTPS endpoint requires --ca-cert for strict validation: $endpoint"
|
||||
fi
|
||||
if [[ "$HEALTHCHECK_FORCE_LOOPBACK_RESOLVE" == "true" ]]; then
|
||||
curl -fsS --cacert "$CA_CERT" --resolve "${host}:${port}:127.0.0.1" "$url" >"$out_file"
|
||||
else
|
||||
curl -fsS --cacert "$CA_CERT" "$url" >"$out_file"
|
||||
fi
|
||||
else
|
||||
curl -fsS "$url" >"$out_file"
|
||||
fi
|
||||
log "health check passed for ${label}: $url"
|
||||
}
|
||||
|
||||
analyze_duplicates() {
|
||||
local status_json="$1"
|
||||
local out_file="$2"
|
||||
jq -r '
|
||||
def identity_key($e):
|
||||
($e | sub("^https?://";"") | sub("/$";"") | ascii_downcase);
|
||||
(.sites // {})
|
||||
| to_entries
|
||||
| map(.value.endpoint // "")
|
||||
| map(select(. != ""))
|
||||
| map(identity_key(.))
|
||||
| group_by(.)
|
||||
| map({identity: .[0], count: length})
|
||||
| map(select(.count > 1))
|
||||
' "$status_json" >"$out_file"
|
||||
}
|
||||
|
||||
optional_object_flow_check() {
|
||||
local endpoint="$1"
|
||||
local label="$2"
|
||||
local put_out="$3"
|
||||
local get_out="$4"
|
||||
|
||||
if [[ -z "$BUCKET" ]]; then
|
||||
log "skip object-flow check for ${label}: --bucket not provided"
|
||||
return 0
|
||||
fi
|
||||
if ! command -v "$AWS_BIN" >/dev/null 2>&1; then
|
||||
log "skip object-flow check for ${label}: aws cli not found"
|
||||
return 0
|
||||
fi
|
||||
|
||||
local common=(
|
||||
--endpoint-url "$endpoint"
|
||||
--region "$REGION"
|
||||
--no-cli-pager
|
||||
)
|
||||
if [[ -n "$AWS_PROFILE" ]]; then
|
||||
common+=(--profile "$AWS_PROFILE")
|
||||
fi
|
||||
if [[ -n "$CA_CERT" ]]; then
|
||||
common+=(--ca-bundle "$CA_CERT")
|
||||
fi
|
||||
|
||||
AWS_ACCESS_KEY_ID="$ACCESS_KEY" AWS_SECRET_ACCESS_KEY="$SECRET_KEY" \
|
||||
"$AWS_BIN" "${common[@]}" s3api put-object \
|
||||
--bucket "$BUCKET" --key "$REPL_OBJECT_KEY" --body <(printf '%s' "$REPL_OBJECT_BODY") >"$put_out"
|
||||
|
||||
AWS_ACCESS_KEY_ID="$ACCESS_KEY" AWS_SECRET_ACCESS_KEY="$SECRET_KEY" \
|
||||
"$AWS_BIN" "${common[@]}" s3api head-object \
|
||||
--bucket "$BUCKET" --key "$REPL_OBJECT_KEY" >"$get_out"
|
||||
|
||||
log "object-flow check passed for ${label} on bucket=${BUCKET}, key=${REPL_OBJECT_KEY}"
|
||||
}
|
||||
|
||||
restart_if_configured() {
|
||||
if [[ -z "$SITE_A_RESTART_CMD" || -z "$SITE_B_RESTART_CMD" ]]; then
|
||||
log "skip restart verification: restart commands not fully provided"
|
||||
return 0
|
||||
fi
|
||||
log "running restart command for site A"
|
||||
bash -lc "$SITE_A_RESTART_CMD"
|
||||
log "running restart command for site B"
|
||||
bash -lc "$SITE_B_RESTART_CMD"
|
||||
}
|
||||
|
||||
main() {
|
||||
parse_args "$@"
|
||||
[[ -n "$SITE_A_ENDPOINT" ]] || fail "--site-a-endpoint is required"
|
||||
[[ -n "$SITE_B_ENDPOINT" ]] || fail "--site-b-endpoint is required"
|
||||
[[ -n "$ACCESS_KEY" ]] || fail "--access-key is required"
|
||||
[[ -n "$SECRET_KEY" ]] || fail "--secret-key is required"
|
||||
|
||||
require_cmd "$AWSCURL_BIN"
|
||||
require_cmd curl
|
||||
require_cmd jq
|
||||
|
||||
mkdir -p "$OUT_DIR"
|
||||
log "output directory: $OUT_DIR"
|
||||
|
||||
local a_status="$OUT_DIR/site-a.status.json"
|
||||
local a_info="$OUT_DIR/site-a.info.json"
|
||||
local b_status="$OUT_DIR/site-b.status.json"
|
||||
local b_info="$OUT_DIR/site-b.info.json"
|
||||
local a_health="$OUT_DIR/site-a.health.txt"
|
||||
local b_health="$OUT_DIR/site-b.health.txt"
|
||||
local a_dupes="$OUT_DIR/site-a.duplicates.json"
|
||||
local b_dupes="$OUT_DIR/site-b.duplicates.json"
|
||||
local summary="$OUT_DIR/summary.txt"
|
||||
|
||||
log "step 1/6: strict health checks"
|
||||
strict_healthcheck "$SITE_A_ENDPOINT" "site-a" "$a_health"
|
||||
strict_healthcheck "$SITE_B_ENDPOINT" "site-b" "$b_health"
|
||||
|
||||
log "step 2/6: collect site-replication status/info"
|
||||
admin_get "$SITE_A_ENDPOINT" "/rustfs/admin/v3/site-replication/status" "$a_status"
|
||||
admin_get "$SITE_A_ENDPOINT" "/rustfs/admin/v3/site-replication/info" "$a_info"
|
||||
admin_get "$SITE_B_ENDPOINT" "/rustfs/admin/v3/site-replication/status" "$b_status"
|
||||
admin_get "$SITE_B_ENDPOINT" "/rustfs/admin/v3/site-replication/info" "$b_info"
|
||||
|
||||
log "step 3/6: duplicate identity analysis"
|
||||
analyze_duplicates "$a_status" "$a_dupes"
|
||||
analyze_duplicates "$b_status" "$b_dupes"
|
||||
|
||||
log "step 4/6: optional object-flow checks"
|
||||
optional_object_flow_check "$SITE_A_ENDPOINT" "site-a" "$OUT_DIR/site-a.put.json" "$OUT_DIR/site-a.head.json"
|
||||
optional_object_flow_check "$SITE_B_ENDPOINT" "site-b" "$OUT_DIR/site-b.put.json" "$OUT_DIR/site-b.head.json"
|
||||
|
||||
log "step 5/6: optional restart verification"
|
||||
restart_if_configured
|
||||
if [[ -n "$SITE_A_RESTART_CMD" && -n "$SITE_B_RESTART_CMD" ]]; then
|
||||
admin_get "$SITE_A_ENDPOINT" "/rustfs/admin/v3/site-replication/status" "$OUT_DIR/site-a.status.after-restart.json"
|
||||
admin_get "$SITE_B_ENDPOINT" "/rustfs/admin/v3/site-replication/status" "$OUT_DIR/site-b.status.after-restart.json"
|
||||
fi
|
||||
|
||||
log "step 6/6: write summary"
|
||||
{
|
||||
echo "Issue 2723 verification summary"
|
||||
echo "site-a endpoint: $SITE_A_ENDPOINT"
|
||||
echo "site-b endpoint: $SITE_B_ENDPOINT"
|
||||
echo "region: $REGION"
|
||||
echo "ca-cert: ${CA_CERT:-<none>}"
|
||||
echo
|
||||
echo "Duplicate identities (site-a): $(jq 'length' "$a_dupes")"
|
||||
echo "Duplicate identities (site-b): $(jq 'length' "$b_dupes")"
|
||||
echo
|
||||
echo "Artifacts:"
|
||||
find "$OUT_DIR" -maxdepth 1 -type f | sort
|
||||
} >"$summary"
|
||||
|
||||
log "done. summary: $summary"
|
||||
cat "$summary"
|
||||
}
|
||||
|
||||
main "$@"
|
||||
@@ -0,0 +1,466 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
PROJECT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
CLUSTER_COMPOSE="${CLUSTER_COMPOSE:-${PROJECT_ROOT}/.docker/compose/docker-compose.cluster.local-build.yml}"
|
||||
PROJECT_NAME="${PROJECT_NAME:-rustfs-issue3031}"
|
||||
RUSTFS_IMAGE="${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}"
|
||||
BUILD_LOCAL_IMAGE="${BUILD_LOCAL_IMAGE:-true}"
|
||||
FORCE_BUILD="${FORCE_BUILD:-false}"
|
||||
KEEP_UP="${KEEP_UP:-false}"
|
||||
PRECHECK_AUTO_CLEANUP="${PRECHECK_AUTO_CLEANUP:-true}"
|
||||
WAIT_TIMEOUT_SECS="${WAIT_TIMEOUT_SECS:-180}"
|
||||
S3_READY_TIMEOUT_SECS="${S3_READY_TIMEOUT_SECS:-120}"
|
||||
|
||||
RUSTFS_ACCESS_KEY="${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}"
|
||||
RUSTFS_SECRET_KEY="${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}"
|
||||
RUSTFS_UNSAFE_BYPASS_DISK_CHECK="${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}"
|
||||
RUSTFS_ISSUE3031_DIAG_ENABLE="${RUSTFS_ISSUE3031_DIAG_ENABLE:-true}"
|
||||
RUSTFS_OBS_LOG_STDOUT_ENABLED="${RUSTFS_OBS_LOG_STDOUT_ENABLED:-true}"
|
||||
RUSTFS_OBS_USE_STDOUT="${RUSTFS_OBS_USE_STDOUT:-false}"
|
||||
RUSTFS_OBS_LOGGER_LEVEL="${RUSTFS_OBS_LOGGER_LEVEL:-info}"
|
||||
RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT="${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}"
|
||||
RUSTFS_LOCK_ACQUIRE_TIMEOUT="${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}"
|
||||
|
||||
ENDPOINT="${ENDPOINT:-http://127.0.0.1:9000}"
|
||||
WARP_HOST="${WARP_HOST:-node1:9000}"
|
||||
BUCKET="${BUCKET:-rustfs-multipart-repro}"
|
||||
SMOKE_OBJECTS="${SMOKE_OBJECTS:-16}"
|
||||
SMOKE_OBJECT_SIZE_MB="${SMOKE_OBJECT_SIZE_MB:-32}"
|
||||
SMOKE_PARALLELISM="${SMOKE_PARALLELISM:-8}"
|
||||
|
||||
WARP_DURATION="${WARP_DURATION:-5m}"
|
||||
WARP_CONCURRENT="${WARP_CONCURRENT:-16}"
|
||||
WARP_PARTS="${WARP_PARTS:-16}"
|
||||
WARP_PART_SIZE="${WARP_PART_SIZE:-16MiB}"
|
||||
WARP_PART_CONCURRENT="${WARP_PART_CONCURRENT:-4}"
|
||||
|
||||
MC_IMAGE="${MC_IMAGE:-minio/mc:latest}"
|
||||
WARP_IMAGE="${WARP_IMAGE:-minio/warp:latest}"
|
||||
OUT_DIR="${OUT_DIR:-${PROJECT_ROOT}/target/issue3031/$(date +%Y%m%d-%H%M%S)}"
|
||||
|
||||
usage() {
|
||||
cat <<'USAGE'
|
||||
Usage:
|
||||
scripts/validate_issue_3031_docker.sh [options]
|
||||
|
||||
Options:
|
||||
--skip-build skip local Docker image build
|
||||
--force-build rebuild even if the local image already exists
|
||||
--keep-up keep cluster running after the script exits
|
||||
--project-name <name> docker compose project name
|
||||
--out-dir <path> output directory
|
||||
--warp-duration <dur> warp duration (default: 5m)
|
||||
--warp-concurrent <n> warp --concurrent (default: 16)
|
||||
--warp-parts <n> warp --parts (default: 16)
|
||||
--warp-part-size <size> warp --part.size (default: 16MiB)
|
||||
--warp-part-concurrent <n> warp --part.concurrent (default: 4)
|
||||
--smoke-objects <n> plain write smoke object count (default: 16)
|
||||
--smoke-object-size-mb <n> plain write smoke object size MiB (default: 32)
|
||||
--smoke-parallelism <n> plain write smoke parallelism (default: 8)
|
||||
-h, --help show help
|
||||
|
||||
Environment:
|
||||
CLUSTER_COMPOSE PROJECT_NAME RUSTFS_IMAGE BUILD_LOCAL_IMAGE FORCE_BUILD KEEP_UP
|
||||
RUSTFS_ACCESS_KEY RUSTFS_SECRET_KEY RUSTFS_UNSAFE_BYPASS_DISK_CHECK
|
||||
RUSTFS_ISSUE3031_DIAG_ENABLE RUSTFS_OBS_LOG_STDOUT_ENABLED
|
||||
RUSTFS_OBS_USE_STDOUT RUSTFS_OBS_LOGGER_LEVEL
|
||||
RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT
|
||||
RUSTFS_LOCK_ACQUIRE_TIMEOUT
|
||||
S3_READY_TIMEOUT_SECS
|
||||
ENDPOINT BUCKET
|
||||
WARP_HOST
|
||||
SMOKE_OBJECTS SMOKE_OBJECT_SIZE_MB SMOKE_PARALLELISM
|
||||
WARP_DURATION WARP_CONCURRENT WARP_PARTS WARP_PART_SIZE WARP_PART_CONCURRENT
|
||||
MC_IMAGE WARP_IMAGE OUT_DIR
|
||||
USAGE
|
||||
}
|
||||
|
||||
log_info() {
|
||||
printf '[INFO] %s\n' "$*"
|
||||
}
|
||||
|
||||
log_warn() {
|
||||
printf '[WARN] %s\n' "$*"
|
||||
}
|
||||
|
||||
log_error() {
|
||||
printf '[ERROR] %s\n' "$*" >&2
|
||||
}
|
||||
|
||||
require_cmd() {
|
||||
if ! command -v "$1" >/dev/null 2>&1; then
|
||||
log_error "command not found: $1"
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
compose() {
|
||||
docker compose \
|
||||
--project-name "${PROJECT_NAME}" \
|
||||
-f "${CLUSTER_COMPOSE}" \
|
||||
"$@"
|
||||
}
|
||||
|
||||
cleanup() {
|
||||
if [[ "${KEEP_UP}" == "true" ]]; then
|
||||
log_info "KEEP_UP=true, leaving cluster running"
|
||||
return
|
||||
fi
|
||||
|
||||
log_info "Stopping docker compose services"
|
||||
compose down -v >/dev/null 2>&1 || true
|
||||
}
|
||||
|
||||
parse_args() {
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--skip-build)
|
||||
BUILD_LOCAL_IMAGE=false
|
||||
shift
|
||||
;;
|
||||
--keep-up)
|
||||
KEEP_UP=true
|
||||
shift
|
||||
;;
|
||||
--force-build)
|
||||
FORCE_BUILD=true
|
||||
shift
|
||||
;;
|
||||
--project-name)
|
||||
PROJECT_NAME="$2"
|
||||
shift 2
|
||||
;;
|
||||
--out-dir)
|
||||
OUT_DIR="$2"
|
||||
shift 2
|
||||
;;
|
||||
--warp-duration)
|
||||
WARP_DURATION="$2"
|
||||
shift 2
|
||||
;;
|
||||
--warp-concurrent)
|
||||
WARP_CONCURRENT="$2"
|
||||
shift 2
|
||||
;;
|
||||
--warp-parts)
|
||||
WARP_PARTS="$2"
|
||||
shift 2
|
||||
;;
|
||||
--warp-part-size)
|
||||
WARP_PART_SIZE="$2"
|
||||
shift 2
|
||||
;;
|
||||
--warp-part-concurrent)
|
||||
WARP_PART_CONCURRENT="$2"
|
||||
shift 2
|
||||
;;
|
||||
--smoke-objects)
|
||||
SMOKE_OBJECTS="$2"
|
||||
shift 2
|
||||
;;
|
||||
--smoke-object-size-mb)
|
||||
SMOKE_OBJECT_SIZE_MB="$2"
|
||||
shift 2
|
||||
;;
|
||||
--smoke-parallelism)
|
||||
SMOKE_PARALLELISM="$2"
|
||||
shift 2
|
||||
;;
|
||||
-h|--help)
|
||||
usage
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
log_error "unknown argument: $1"
|
||||
usage
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
}
|
||||
|
||||
wait_http_ok() {
|
||||
local url="$1"
|
||||
local start now
|
||||
start="$(date +%s)"
|
||||
|
||||
while true; do
|
||||
if curl -fsS --connect-timeout 2 --max-time 3 "${url}" >/dev/null 2>&1; then
|
||||
return 0
|
||||
fi
|
||||
|
||||
now="$(date +%s)"
|
||||
if (( now - start >= WAIT_TIMEOUT_SECS )); then
|
||||
log_error "timed out waiting for ${url}"
|
||||
return 1
|
||||
fi
|
||||
|
||||
sleep 2
|
||||
done
|
||||
}
|
||||
|
||||
cleanup_existing_project_containers() {
|
||||
local existing_ids
|
||||
existing_ids="$(docker ps -aq --filter "label=com.docker.compose.project=${PROJECT_NAME}")"
|
||||
|
||||
if [[ -z "${existing_ids}" ]]; then
|
||||
return 0
|
||||
fi
|
||||
|
||||
log_warn "Found existing containers for project ${PROJECT_NAME}."
|
||||
docker ps -a --filter "label=com.docker.compose.project=${PROJECT_NAME}" --format ' - {{.Names}} ({{.Status}})'
|
||||
|
||||
if [[ "${PRECHECK_AUTO_CLEANUP}" == "true" ]]; then
|
||||
log_info "PRECHECK_AUTO_CLEANUP=true, removing existing project containers."
|
||||
# shellcheck disable=SC2086
|
||||
docker rm -f ${existing_ids} >/dev/null
|
||||
else
|
||||
log_error "existing project containers detected and PRECHECK_AUTO_CLEANUP=false"
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
build_image_if_needed() {
|
||||
if [[ "${BUILD_LOCAL_IMAGE}" != "true" ]]; then
|
||||
log_info "Skipping local image build"
|
||||
return
|
||||
fi
|
||||
|
||||
if [[ "${FORCE_BUILD}" != "true" ]] && docker image inspect "${RUSTFS_IMAGE}" >/dev/null 2>&1; then
|
||||
log_info "Reusing existing local image ${RUSTFS_IMAGE}; pass --force-build to rebuild"
|
||||
return
|
||||
fi
|
||||
|
||||
log_info "Building ${RUSTFS_IMAGE} from Dockerfile.source"
|
||||
docker build -f "${PROJECT_ROOT}/Dockerfile.source" -t "${RUSTFS_IMAGE}" "${PROJECT_ROOT}"
|
||||
}
|
||||
|
||||
start_cluster() {
|
||||
mkdir -p "${OUT_DIR}"
|
||||
cleanup_existing_project_containers
|
||||
|
||||
log_info "Starting 4-node cluster from ${CLUSTER_COMPOSE}"
|
||||
compose up -d
|
||||
|
||||
log_info "Waiting for cluster health endpoint"
|
||||
wait_http_ok "${ENDPOINT}/health"
|
||||
|
||||
log_info "Waiting for cluster readiness endpoint"
|
||||
wait_http_ok "${ENDPOINT}/health/ready"
|
||||
|
||||
log_info "Waiting for S3 API readiness via mc alias/list"
|
||||
wait_s3_api_ready
|
||||
}
|
||||
|
||||
cluster_network() {
|
||||
echo "${PROJECT_NAME}_rustfs-cluster-net"
|
||||
}
|
||||
|
||||
run_mc_in_network() {
|
||||
docker run --rm --network "$(cluster_network)" "${MC_IMAGE}" "$@"
|
||||
}
|
||||
|
||||
run_mc_shell_in_network() {
|
||||
docker run --rm --network "$(cluster_network)" --entrypoint /bin/sh "${MC_IMAGE}" -lc "$1"
|
||||
}
|
||||
|
||||
run_warp_in_network() {
|
||||
docker run --rm --network "$(cluster_network)" -v "${OUT_DIR}:/out" "${WARP_IMAGE}" "$@"
|
||||
}
|
||||
|
||||
wait_s3_api_ready() {
|
||||
local start now
|
||||
start="$(date +%s)"
|
||||
|
||||
while true; do
|
||||
if run_mc_in_network alias set rustfs "http://node1:9000" "${RUSTFS_ACCESS_KEY}" "${RUSTFS_SECRET_KEY}" >/dev/null 2>&1; then
|
||||
if run_mc_in_network ls rustfs >/dev/null 2>&1; then
|
||||
return 0
|
||||
fi
|
||||
fi
|
||||
|
||||
now="$(date +%s)"
|
||||
if (( now - start >= S3_READY_TIMEOUT_SECS )); then
|
||||
log_error "timed out waiting for S3 API readiness via mc alias/list"
|
||||
return 1
|
||||
fi
|
||||
|
||||
sleep 2
|
||||
done
|
||||
}
|
||||
|
||||
collect_cluster_info() {
|
||||
local node
|
||||
|
||||
for node in node1 node2 node3 node4; do
|
||||
log_info "Collecting rustfs info from ${node}"
|
||||
compose exec -T "${node}" rustfs info --all --json > "${OUT_DIR}/${node}-info.json"
|
||||
done
|
||||
}
|
||||
|
||||
run_plain_smoke() {
|
||||
local prefix
|
||||
prefix="plain-smoke-$(date +%s)"
|
||||
|
||||
log_info "Preparing bucket ${BUCKET}"
|
||||
run_mc_in_network alias set rustfs "http://node1:9000" "${RUSTFS_ACCESS_KEY}" "${RUSTFS_SECRET_KEY}" >/dev/null
|
||||
run_mc_in_network ls "rustfs/${BUCKET}" >/dev/null 2>&1 || run_mc_in_network mb "rustfs/${BUCKET}" >/dev/null
|
||||
|
||||
log_info "Running plain smoke write/read/delete: objects=${SMOKE_OBJECTS} size_mb=${SMOKE_OBJECT_SIZE_MB} parallelism=${SMOKE_PARALLELISM}"
|
||||
run_mc_shell_in_network "
|
||||
set -euo pipefail
|
||||
i=1
|
||||
while [ \"\$i\" -le \"${SMOKE_OBJECTS}\" ]; do
|
||||
end=\$((i + ${SMOKE_PARALLELISM} - 1))
|
||||
[ \"\$end\" -le \"${SMOKE_OBJECTS}\" ] || end='${SMOKE_OBJECTS}'
|
||||
j=\"\$i\"
|
||||
while [ \"\$j\" -le \"\$end\" ]; do
|
||||
(
|
||||
dd if=/dev/urandom bs='${SMOKE_OBJECT_SIZE_MB}M' count=1 2>/dev/null | \
|
||||
mc pipe 'rustfs/${BUCKET}/${prefix}/obj-\${j}' >/dev/null
|
||||
) &
|
||||
j=\$((j + 1))
|
||||
done
|
||||
wait
|
||||
i=\$((end + 1))
|
||||
done
|
||||
mc rm --recursive --force 'rustfs/${BUCKET}/${prefix}' >/dev/null
|
||||
"
|
||||
}
|
||||
|
||||
run_warp_multipart() {
|
||||
log_info "Running warp multipart-put against ${WARP_HOST}"
|
||||
run_warp_in_network multipart-put \
|
||||
--host "${WARP_HOST}" \
|
||||
--access-key "${RUSTFS_ACCESS_KEY}" \
|
||||
--secret-key "${RUSTFS_SECRET_KEY}" \
|
||||
--bucket "${BUCKET}-warp" \
|
||||
--lookup path \
|
||||
--duration "${WARP_DURATION}" \
|
||||
--concurrent "${WARP_CONCURRENT}" \
|
||||
--parts "${WARP_PARTS}" \
|
||||
--part.size "${WARP_PART_SIZE}" \
|
||||
--part.concurrent "${WARP_PART_CONCURRENT}" \
|
||||
--benchdata /out/warp.csv.zst \
|
||||
--analyze.v \
|
||||
--no-color \
|
||||
| tee "${OUT_DIR}/warp.log"
|
||||
}
|
||||
|
||||
count_matches() {
|
||||
local pattern="$1"
|
||||
shift
|
||||
local total=0
|
||||
local file count
|
||||
|
||||
for file in "$@"; do
|
||||
count="$(grep -cE "${pattern}" "${file}" 2>/dev/null || true)"
|
||||
total=$((total + ${count:-0}))
|
||||
done
|
||||
|
||||
echo "${total}"
|
||||
}
|
||||
|
||||
collect_cluster_logs() {
|
||||
local node
|
||||
|
||||
for node in node1 node2 node3 node4; do
|
||||
compose logs --no-color "${node}" > "${OUT_DIR}/${node}.log" || true
|
||||
done
|
||||
}
|
||||
|
||||
summarize_results() {
|
||||
local warp_log summary
|
||||
local warp_errors quorum_not_reached connection_refused storage_insufficient
|
||||
local lock_timeout erasure_write_quorum readiness_probe_failed liveness_probe_failed
|
||||
local diag_read_parts diag_complete_part diag_rename_part
|
||||
local startup_range_oob startup_volume_not_found startup_remote_network startup_remote_faulty startup_remote_lock_rpc
|
||||
|
||||
warp_log="${OUT_DIR}/warp.log"
|
||||
summary="${OUT_DIR}/summary.txt"
|
||||
|
||||
warp_errors="$(count_matches 'warp: <ERROR>' "${warp_log}")"
|
||||
quorum_not_reached="$(count_matches 'Quorum not reached' "${warp_log}")"
|
||||
connection_refused="$(count_matches 'connection refused' "${warp_log}")"
|
||||
storage_insufficient="$(count_matches 'Storage resources are insufficient for the write operation' "${warp_log}")"
|
||||
lock_timeout="$(count_matches 'Lock acquisition timeout' "${warp_log}")"
|
||||
erasure_write_quorum="$(count_matches 'erasure write quorum' "${warp_log}")"
|
||||
readiness_probe_failed="$(count_matches 'readiness_probe_failed' "${warp_log}")"
|
||||
liveness_probe_failed="$(count_matches 'liveness_probe_failed' "${warp_log}")"
|
||||
|
||||
diag_read_parts="$(count_matches 'issue3031_read_parts_part_quorum' "${OUT_DIR}"/node*.log)"
|
||||
diag_complete_part="$(count_matches 'issue3031_complete_part_error' "${OUT_DIR}"/node*.log)"
|
||||
diag_rename_part="$(count_matches 'issue3031_rename_part_context' "${OUT_DIR}"/node*.log)"
|
||||
|
||||
startup_range_oob="$(count_matches 'Range \\[0, 4\\) is out of bounds' "${OUT_DIR}"/node*.log)"
|
||||
startup_volume_not_found="$(count_matches 'volume not found' "${OUT_DIR}"/node*.log)"
|
||||
startup_remote_network="$(count_matches 'Remote disk operation returned a network-like error' "${OUT_DIR}"/node*.log)"
|
||||
startup_remote_faulty="$(count_matches 'Remote disk marked faulty after timeout' "${OUT_DIR}"/node*.log)"
|
||||
startup_remote_lock_rpc="$(count_matches 'Evicting cached remote lock connection after RPC failure' "${OUT_DIR}"/node*.log)"
|
||||
|
||||
cat > "${summary}" <<EOF
|
||||
issue=3031
|
||||
endpoint=${ENDPOINT}
|
||||
bucket=${BUCKET}
|
||||
warp_duration=${WARP_DURATION}
|
||||
warp_concurrent=${WARP_CONCURRENT}
|
||||
warp_parts=${WARP_PARTS}
|
||||
warp_part_size=${WARP_PART_SIZE}
|
||||
warp_part_concurrent=${WARP_PART_CONCURRENT}
|
||||
|
||||
warp_errors=${warp_errors}
|
||||
quorum_not_reached=${quorum_not_reached}
|
||||
connection_refused=${connection_refused}
|
||||
storage_insufficient=${storage_insufficient}
|
||||
lock_timeout=${lock_timeout}
|
||||
erasure_write_quorum=${erasure_write_quorum}
|
||||
readiness_probe_failed=${readiness_probe_failed}
|
||||
liveness_probe_failed=${liveness_probe_failed}
|
||||
|
||||
issue3031_read_parts_part_quorum=${diag_read_parts}
|
||||
issue3031_complete_part_error=${diag_complete_part}
|
||||
issue3031_rename_part_context=${diag_rename_part}
|
||||
|
||||
startup_range_oob=${startup_range_oob}
|
||||
startup_volume_not_found=${startup_volume_not_found}
|
||||
startup_remote_network_error=${startup_remote_network}
|
||||
startup_remote_faulty=${startup_remote_faulty}
|
||||
startup_remote_lock_rpc=${startup_remote_lock_rpc}
|
||||
EOF
|
||||
|
||||
log_info "Summary written to ${summary}"
|
||||
cat "${summary}"
|
||||
}
|
||||
|
||||
main() {
|
||||
parse_args "$@"
|
||||
require_cmd docker
|
||||
require_cmd curl
|
||||
require_cmd grep
|
||||
require_cmd tee
|
||||
|
||||
mkdir -p "${OUT_DIR}"
|
||||
trap cleanup EXIT INT TERM
|
||||
|
||||
export RUSTFS_IMAGE
|
||||
export RUSTFS_ACCESS_KEY
|
||||
export RUSTFS_SECRET_KEY
|
||||
export RUSTFS_UNSAFE_BYPASS_DISK_CHECK
|
||||
export RUSTFS_ISSUE3031_DIAG_ENABLE
|
||||
export RUSTFS_OBS_LOG_STDOUT_ENABLED
|
||||
export RUSTFS_OBS_USE_STDOUT
|
||||
export RUSTFS_OBS_LOGGER_LEVEL
|
||||
export RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT
|
||||
export RUSTFS_LOCK_ACQUIRE_TIMEOUT
|
||||
|
||||
build_image_if_needed
|
||||
start_cluster
|
||||
collect_cluster_info
|
||||
run_plain_smoke
|
||||
run_warp_multipart
|
||||
collect_cluster_logs
|
||||
summarize_results
|
||||
}
|
||||
|
||||
main "$@"
|
||||
+152
@@ -0,0 +1,152 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# Acceptance runner for rustfs/backlog#785 and PR #4072
|
||||
PROJECT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
OUT_DIR="${OUT_DIR:-${PROJECT_ROOT}/target/issue-785-acceptance-$(date +%Y%m%d-%H%M%S)}"
|
||||
SKIP_LIVE="${SKIP_LIVE:-true}"
|
||||
ENDPOINT="${ENDPOINT:-http://127.0.0.1:9000}"
|
||||
|
||||
log_info() { printf '[INFO] %s\n' "$*"; }
|
||||
log_warn() { printf '[WARN] %s\n' "$*"; }
|
||||
log_error() { printf '[ERROR] %s\n' "$*" >&2; }
|
||||
|
||||
usage() {
|
||||
cat <<'USAGE'
|
||||
Usage:
|
||||
scripts/validate_issue_785_list_objects.sh [--full] [--no-skip-live]
|
||||
|
||||
Environment:
|
||||
OUT_DIR output directory (default: target/issue-785-acceptance-<ts>)
|
||||
SKIP_LIVE skip live S3 endpoint checks when true (default: true)
|
||||
ENDPOINT rustfs endpoint used by optional live checks (default: http://127.0.0.1:9000)
|
||||
USAGE
|
||||
}
|
||||
|
||||
require_cmd() {
|
||||
if ! command -v "$1" >/dev/null 2>&1; then
|
||||
log_error "command not found: $1"
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
run_unit_checks() {
|
||||
log_info "Running focused rustfs-ecstore tests for list_objects hot path"
|
||||
mkdir -p "$OUT_DIR"
|
||||
|
||||
local test_log="$OUT_DIR/issue-785-ecstore-tests.log"
|
||||
: > "$test_log"
|
||||
|
||||
local tests=(
|
||||
list_path_gather_results_returns_after_limit_without_waiting_for_input_close
|
||||
list_path_gather_results_keeps_marker_entry_for_version_marker_listing
|
||||
list_path_gather_results_skips_marker_entry_by_default
|
||||
list_path_forward_past_is_idempotent_for_same_marker
|
||||
list_path_parse_marker_replay_still_stable
|
||||
normalize_list_quorum_falls_back_to_strict
|
||||
list_objects_quorum_from_env_defaults_to_optimal
|
||||
)
|
||||
|
||||
local test
|
||||
for test in "${tests[@]}"; do
|
||||
echo "[TEST] $test" | tee -a "$test_log"
|
||||
(cd "$PROJECT_ROOT" && cargo test -p rustfs-ecstore "$test" -- --nocapture | tee -a "$test_log")
|
||||
done
|
||||
|
||||
log_info "Unit test log: $test_log"
|
||||
}
|
||||
|
||||
run_static_checks() {
|
||||
log_info "Running acceptance-focused static checks"
|
||||
local static_log="$OUT_DIR/static-checks.log"
|
||||
|
||||
: > "$static_log"
|
||||
|
||||
if rg -n "store_list_objects_list_path|store_list_objects_list_merged|sets_list_objects_list_path|sets_list_objects_list_merged|set_disks_list_objects_list_path|store_list_objects_gather" \
|
||||
"$PROJECT_ROOT/crates/ecstore/src/store/list_objects.rs" | tee -a "$static_log"; then
|
||||
log_info "Metric and stage names exist in list_objects.rs"
|
||||
else
|
||||
log_error "Required metric symbols not found in list_objects.rs"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
log_info "Static check log: $static_log"
|
||||
}
|
||||
|
||||
run_live_smoke() {
|
||||
if [[ "$SKIP_LIVE" == "true" ]]; then
|
||||
log_warn "SKIP_LIVE=true, skipping live endpoint checks"
|
||||
return 0
|
||||
fi
|
||||
|
||||
log_info "Running optional live readiness check: $ENDPOINT"
|
||||
if ! curl -fsS "${ENDPOINT}/health/ready" >/dev/null 2>&1; then
|
||||
log_error "Health ready endpoint check failed: ${ENDPOINT}/health/ready"
|
||||
return 1
|
||||
fi
|
||||
log_info "Live endpoint reachable"
|
||||
|
||||
log_warn "Optional live S3 pagination/candidates check is intentionally not enforced in this runner."
|
||||
log_warn "Please run your preferred client workload (mc/warp/rclone) and verify page continuity + duplicates manually."
|
||||
}
|
||||
|
||||
run_metrics_context() {
|
||||
log_info "Collecting local metric symbol index for review"
|
||||
local metric_log="$OUT_DIR/issue-785-metrics-context.log"
|
||||
if rg -n "record_stage_duration\(" "$PROJECT_ROOT/crates/ecstore/src/store/list_objects.rs" > "$metric_log"; then
|
||||
log_info "Metric context saved: $metric_log"
|
||||
else
|
||||
log_error "No metric stage context found"
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
parse_args() {
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--full)
|
||||
SKIP_LIVE="false"
|
||||
shift
|
||||
;;
|
||||
--no-skip-live)
|
||||
SKIP_LIVE="false"
|
||||
shift
|
||||
;;
|
||||
-h|--help)
|
||||
usage
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
log_error "unknown arg: $1"
|
||||
usage
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
:
|
||||
}
|
||||
|
||||
main() {
|
||||
parse_args "$@"
|
||||
require_cmd cargo
|
||||
require_cmd rg
|
||||
require_cmd tee
|
||||
require_cmd curl
|
||||
|
||||
mkdir -p "$OUT_DIR"
|
||||
|
||||
run_unit_checks
|
||||
run_static_checks
|
||||
run_metrics_context
|
||||
|
||||
if [[ "$SKIP_LIVE" != "true" ]]; then
|
||||
run_live_smoke
|
||||
fi
|
||||
|
||||
log_info "Validation summary:"
|
||||
log_info " - logs: $OUT_DIR"
|
||||
log_info " - status: pass"
|
||||
}
|
||||
|
||||
main "$@"
|
||||
+232
@@ -0,0 +1,232 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# Acceptance runner for rustfs/backlog#786 and PR #4072 follow-up
|
||||
PROJECT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
OUT_DIR="${OUT_DIR:-${PROJECT_ROOT}/target/issue-786-acceptance-$(date +%Y%m%d-%H%M%S)}"
|
||||
SKIP_LIVE="${SKIP_LIVE:-true}"
|
||||
ENDPOINT="${ENDPOINT:-http://127.0.0.1:9000}"
|
||||
LIVE_LIST_BUCKET="${LIVE_LIST_BUCKET:-}"
|
||||
LIVE_LIST_MAX_KEYS="${LIVE_LIST_MAX_KEYS:-2}"
|
||||
LIVE_LIST_REGION="${LIVE_LIST_REGION:-us-east-1}"
|
||||
|
||||
log_info() { printf '[INFO] %s\n' "$*"; }
|
||||
log_warn() { printf '[WARN] %s\n' "$*"; }
|
||||
log_error() { printf '[ERROR] %s\n' "$*" >&2; }
|
||||
|
||||
usage() {
|
||||
cat <<'USAGE'
|
||||
Usage:
|
||||
scripts/validate_issue_786_list_objects.sh [--full] [--no-skip-live]
|
||||
|
||||
Environment:
|
||||
OUT_DIR output directory (default: target/issue-786-acceptance-<ts>)
|
||||
SKIP_LIVE skip live S3 endpoint checks when true (default: true)
|
||||
ENDPOINT rustfs endpoint used by optional live checks (default: http://127.0.0.1:9000)
|
||||
LIVE_LIST_BUCKET S3 bucket used for pagination smoke (required)
|
||||
LIVE_LIST_MAX_KEYS maximum keys used per page for smoke (default: 2)
|
||||
LIVE_LIST_REGION signing region for curl --aws-sigv4 (default: us-east-1)
|
||||
USAGE
|
||||
}
|
||||
|
||||
require_cmd() {
|
||||
if ! command -v "$1" >/dev/null 2>&1; then
|
||||
log_error "command not found: $1"
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
run_unit_checks() {
|
||||
log_info "Running focused rustfs-ecstore tests for issue-786 cursor fallback"
|
||||
mkdir -p "$OUT_DIR"
|
||||
|
||||
local test_log="$OUT_DIR/issue-786-ecstore-tests.log"
|
||||
: > "$test_log"
|
||||
|
||||
local tests=(
|
||||
list_path_marker_round_trip_preserves_set_index
|
||||
list_path_marker_parser_uses_trailing_cache_tag
|
||||
list_path_marker_parser_accepts_legacy_v1_tag
|
||||
list_path_marker_parser_return_tag_forces_cache_refresh
|
||||
list_path_marker_parser_recovers_from_corrupt_set_index
|
||||
)
|
||||
|
||||
local test
|
||||
for test in "${tests[@]}"; do
|
||||
echo "[TEST] $test" | tee -a "$test_log"
|
||||
(cd "$PROJECT_ROOT" && cargo test -p rustfs-ecstore "$test" -- --nocapture | tee -a "$test_log")
|
||||
done
|
||||
|
||||
log_info "Unit test log: $test_log"
|
||||
}
|
||||
|
||||
run_static_checks() {
|
||||
log_info "Running issue-786 static marker checks"
|
||||
local static_log="$OUT_DIR/static-checks.log"
|
||||
|
||||
: > "$static_log"
|
||||
|
||||
if rg -n "struct ListContinuationV2|return:\\]|rustfs_cache:v2|MARKER_TAG_VERSION" "$PROJECT_ROOT/crates/ecstore/src/store/list_objects.rs" | tee -a "$static_log"; then
|
||||
log_info "ListContinuationV2 symbols found in list_objects.rs"
|
||||
else
|
||||
log_error "Required cursor symbols not found in list_objects.rs"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
log_info "Static check log: $static_log"
|
||||
}
|
||||
|
||||
extract_xml_keys() {
|
||||
local xml_text="$1"
|
||||
printf '%s\n' "$xml_text" | sed -n 's:.*<Key>\(.*\)</Key>.*:\1:p'
|
||||
}
|
||||
|
||||
run_live_smoke() {
|
||||
if [[ "$SKIP_LIVE" == "true" ]]; then
|
||||
log_warn "SKIP_LIVE=true, skipping live endpoint checks"
|
||||
return 0
|
||||
fi
|
||||
|
||||
local live_log="$OUT_DIR/live-pagination-smoke.log"
|
||||
: > "$live_log"
|
||||
|
||||
if ! curl -fsS "${ENDPOINT}/health/ready" >/dev/null 2>&1; then
|
||||
log_warn "Live endpoint not reachable: ${ENDPOINT}/health/ready"
|
||||
echo "live pagination smoke: skipped (endpoint not reachable)" | tee -a "$live_log"
|
||||
return 0
|
||||
fi
|
||||
log_info "Live endpoint reachable"
|
||||
|
||||
local access_key="${RUSTFS_ACCESS_KEY:-${WARP_ACCESS_KEY:-}}"
|
||||
local secret_key="${RUSTFS_SECRET_KEY:-${WARP_SECRET_KEY:-}}"
|
||||
if [[ -z "$access_key" || -z "$secret_key" ]]; then
|
||||
log_warn "Live pagination smoke skipped: missing RUSTFS_ACCESS_KEY / RUSTFS_SECRET_KEY or WARP_* alternatives"
|
||||
echo "live pagination smoke: skipped (missing credentials)" | tee -a "$live_log"
|
||||
return 0
|
||||
fi
|
||||
|
||||
if [[ -z "$LIVE_LIST_BUCKET" ]]; then
|
||||
log_warn "Live pagination smoke skipped: LIVE_LIST_BUCKET is empty"
|
||||
echo "live pagination smoke: skipped (bucket not configured)" | tee -a "$live_log"
|
||||
return 0
|
||||
fi
|
||||
|
||||
if ! command -v python3 >/dev/null 2>&1; then
|
||||
log_warn "Live pagination smoke skipped: python3 not available"
|
||||
echo "live pagination smoke: skipped (missing python3 parser)" | tee -a "$live_log"
|
||||
return 0
|
||||
fi
|
||||
|
||||
local sigv4_target="aws:amz:${LIVE_LIST_REGION}:s3"
|
||||
local base_url="${ENDPOINT}/${LIVE_LIST_BUCKET}?list-type=2&max-keys=${LIVE_LIST_MAX_KEYS}"
|
||||
|
||||
local page1
|
||||
page1=$(curl -fsS \
|
||||
--user "${access_key}:${secret_key}" \
|
||||
--aws-sigv4 "$sigv4_target" \
|
||||
-H "x-amz-content-sha256: UNSIGNED-PAYLOAD" \
|
||||
"$base_url")
|
||||
|
||||
local token1
|
||||
token1=$(printf '%s' "$page1" | sed -n 's:.*<NextContinuationToken>\(.*\)</NextContinuationToken>.*:\1:p' | head -n 1)
|
||||
|
||||
local page1_key_count
|
||||
page1_key_count=$(printf '%s' "$page1" | sed -n 's:.*<Key>\(.*\)</Key>.*:\1:p' | grep -cv '^$')
|
||||
|
||||
local -a keys1 keys2
|
||||
while IFS= read -r key; do
|
||||
keys1+=("$key")
|
||||
done < <(extract_xml_keys "$page1")
|
||||
|
||||
if [[ -z "$token1" ]]; then
|
||||
log_info "Live pagination smoke: single page only (key count=${page1_key_count})"
|
||||
echo "live pagination smoke: pass (single page, key_count=${page1_key_count})" | tee -a "$live_log"
|
||||
return 0
|
||||
fi
|
||||
|
||||
local token1_encoded
|
||||
token1_encoded=$(printf '%s' "$token1" | python3 -c 'import urllib.parse,sys; print(urllib.parse.quote(sys.stdin.read().strip(), safe=""))')
|
||||
|
||||
local page2
|
||||
page2=$(curl -fsS \
|
||||
--user "${access_key}:${secret_key}" \
|
||||
--aws-sigv4 "$sigv4_target" \
|
||||
-H "x-amz-content-sha256: UNSIGNED-PAYLOAD" \
|
||||
"${base_url}&continuation-token=${token1_encoded}")
|
||||
|
||||
local page2_key_count
|
||||
page2_key_count=$(printf '%s' "$page2" | sed -n 's:.*<Key>\(.*\)</Key>.*:\1:p' | grep -cv '^$')
|
||||
|
||||
if (( page2_key_count == 0 )); then
|
||||
log_error "Live pagination smoke: second page returned zero keys for token ${token1}"
|
||||
return 1
|
||||
fi
|
||||
|
||||
local duplicate
|
||||
local dkeys=()
|
||||
duplicate=0
|
||||
local key
|
||||
|
||||
while IFS= read -r key; do
|
||||
keys2+=("$key")
|
||||
if printf '%s\n' "${keys1[@]}" | grep -Fx -- "$key" >/dev/null 2>&1; then
|
||||
duplicate=1
|
||||
dkeys+=("$key")
|
||||
fi
|
||||
done < <(extract_xml_keys "$page2")
|
||||
|
||||
if (( duplicate )); then
|
||||
log_error "Live pagination smoke detected duplicated keys across page transitions: ${dkeys[*]}"
|
||||
printf '%s\n' "${dkeys[@]}" | tee -a "$live_log"
|
||||
return 1
|
||||
fi
|
||||
|
||||
log_info "Live pagination smoke: first page=${page1_key_count}, second page=${page2_key_count}, token=${token1}"
|
||||
echo "live pagination smoke: pass (continuous two-page check)" | tee -a "$live_log"
|
||||
echo "page1=${page1_key_count}" | tee -a "$live_log"
|
||||
echo "page2=${page2_key_count}" | tee -a "$live_log"
|
||||
log_info "Live pagination smoke log: $live_log"
|
||||
}
|
||||
|
||||
parse_args() {
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--full|--no-skip-live)
|
||||
SKIP_LIVE="false"
|
||||
shift
|
||||
;;
|
||||
-h|--help)
|
||||
usage
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
log_error "unknown arg: $1"
|
||||
usage
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
}
|
||||
|
||||
main() {
|
||||
parse_args "$@"
|
||||
require_cmd cargo
|
||||
require_cmd rg
|
||||
require_cmd tee
|
||||
require_cmd curl
|
||||
|
||||
mkdir -p "$OUT_DIR"
|
||||
|
||||
run_unit_checks
|
||||
run_static_checks
|
||||
|
||||
if [[ "$SKIP_LIVE" != "true" ]]; then
|
||||
run_live_smoke
|
||||
fi
|
||||
|
||||
log_info "Validation summary:"
|
||||
log_info " - logs: $OUT_DIR"
|
||||
log_info " - status: pass"
|
||||
}
|
||||
|
||||
main "$@"
|
||||
+348
@@ -0,0 +1,348 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# Validation runner for rustfs/backlog#787: list quorum tuning and index-readiness baseline
|
||||
PROJECT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
OUT_DIR="${OUT_DIR:-${PROJECT_ROOT}/target/issue-787-list-quorum-$(date +%Y%m%d-%H%M%S)}"
|
||||
SKIP_LIVE="${SKIP_LIVE:-true}"
|
||||
ENDPOINT="${ENDPOINT:-http://127.0.0.1:9000}"
|
||||
LIVE_LIST_BUCKET="${LIVE_LIST_BUCKET:-}"
|
||||
LIVE_LIST_PREFIX="${LIVE_LIST_PREFIX:-}"
|
||||
LIVE_LIST_MAX_KEYS="${LIVE_LIST_MAX_KEYS:-1000}"
|
||||
LIVE_LIST_REGION="${LIVE_LIST_REGION:-us-east-1}"
|
||||
MODE_LIST="${MODE_LIST:-strict,optimal,reduced}"
|
||||
SERVER_RESTART_CMD="${SERVER_RESTART_CMD:-}"
|
||||
SERVER_WAIT_SECONDS="${SERVER_WAIT_SECONDS:-20}"
|
||||
|
||||
log_info() { printf '[INFO] %s\n' "$*"; }
|
||||
log_warn() { printf '[WARN] %s\n' "$*"; }
|
||||
log_error() { printf '[ERROR] %s\n' "$*" >&2; }
|
||||
|
||||
usage() {
|
||||
cat <<'USAGE'
|
||||
Usage:
|
||||
scripts/validate_issue_787_list_quorum.sh [--full] [--no-skip-live]
|
||||
|
||||
Environment:
|
||||
OUT_DIR output directory (default: target/issue-787-list-quorum-<ts>)
|
||||
SKIP_LIVE skip live S3 endpoint checks when true (default: true)
|
||||
ENDPOINT rustfs endpoint for live checks (default: http://127.0.0.1:9000)
|
||||
LIVE_LIST_BUCKET bucket used for live pagination smoke (required when running live)
|
||||
LIVE_LIST_PREFIX optional prefix for list smoke (default: empty)
|
||||
LIVE_LIST_MAX_KEYS max-keys for live smoke page samples (default: 1000)
|
||||
LIVE_LIST_REGION signing region for curl (default: us-east-1)
|
||||
MODE_LIST comma-separated list of list quorum modes (default: strict,optimal,reduced)
|
||||
SERVER_RESTART_CMD optional command to restart server with RUSTFS_TUNING_ENV_FILE
|
||||
SERVER_WAIT_SECONDS seconds to wait after restart command before next checks (default: 20)
|
||||
|
||||
Run modes:
|
||||
--full or --no-skip-live run live checks
|
||||
--help this help
|
||||
USAGE
|
||||
}
|
||||
|
||||
require_cmd() {
|
||||
if ! command -v "$1" >/dev/null 2>&1; then
|
||||
log_error "command not found: $1"
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
timestamp_ms() {
|
||||
python3 - <<'PY'
|
||||
import time
|
||||
print(int(time.time() * 1000))
|
||||
PY
|
||||
}
|
||||
|
||||
extract_xml_keys() {
|
||||
local xml_text="$1"
|
||||
printf '%s\n' "$xml_text" | sed -n 's:.*<Key>\(.*\)</Key>.*:\1:p'
|
||||
}
|
||||
|
||||
wait_for_health() {
|
||||
local waited=0
|
||||
while (( waited < SERVER_WAIT_SECONDS )); do
|
||||
if curl -fsS "${ENDPOINT}/health/ready" >/dev/null 2>&1; then
|
||||
return 0
|
||||
fi
|
||||
sleep 1
|
||||
waited=$(( waited + 1 ))
|
||||
done
|
||||
|
||||
return 1
|
||||
}
|
||||
|
||||
run_unit_checks() {
|
||||
log_info "Running focused rustfs-ecstore tests for #787 list quorum tuning"
|
||||
mkdir -p "$OUT_DIR"
|
||||
|
||||
local test_log="$OUT_DIR/issue-787-ecstore-tests.log"
|
||||
: > "$test_log"
|
||||
|
||||
local tests=(
|
||||
list_path_gather_results_returns_after_limit_without_waiting_for_input_close
|
||||
list_path_marker_parser_uses_trailing_cache_tag
|
||||
normalize_list_quorum_accepts_supported_values
|
||||
normalize_list_quorum_falls_back_to_strict
|
||||
list_quorum_from_env_defaults_to_strict
|
||||
list_quorum_from_env_honors_supported_value
|
||||
list_objects_quorum_from_env_defaults_to_optimal
|
||||
list_objects_quorum_from_env_honors_supported_value
|
||||
)
|
||||
|
||||
local test
|
||||
for test in "${tests[@]}"; do
|
||||
echo "[TEST] $test" | tee -a "$test_log"
|
||||
(cd "$PROJECT_ROOT" && cargo test -p rustfs-ecstore "$test" -- --nocapture | tee -a "$test_log")
|
||||
done
|
||||
|
||||
log_info "Unit test log: $test_log"
|
||||
}
|
||||
|
||||
run_static_checks() {
|
||||
log_info "Running #787 static checks for list-quorum and list path hot path"
|
||||
local static_log="$OUT_DIR/static-checks.log"
|
||||
: > "$static_log"
|
||||
|
||||
if rg -n "ENV_API_LIST_OBJECTS_QUORUM|list_objects_quorum_from_env\(|ask_disks: list_objects_quorum_from_env\(\)" "$PROJECT_ROOT/crates/ecstore/src/store/list_objects.rs" | tee -a "$static_log"; then
|
||||
log_info "Required list quorum symbols found in list_objects.rs"
|
||||
else
|
||||
log_error "list quorum symbols missing in list_objects.rs"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if rg -n "store_list_objects_list_merged|sets_list_objects_list_merged|set_disks_list_objects_list_path|store list_merged finished|sets list_merged finished|set_disks list_path selected listing quorum" "$PROJECT_ROOT/crates/ecstore/src/store/list_objects.rs" | tee -a "$static_log"; then
|
||||
log_info "List path merged/path logging symbols found"
|
||||
else
|
||||
log_error "List path merged symbols missing, static guard failed"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if rg -n "index-backed|list index|list index path|list_index" "$PROJECT_ROOT/crates/ecstore/src/store/list_objects.rs" >/dev/null 2>&1; then
|
||||
log_info "Index-backed scaffold symbols detected in list_objects.rs"
|
||||
else
|
||||
log_warn "Index-backed symbols not yet present in list_objects.rs (expected for current phase)"
|
||||
fi
|
||||
|
||||
log_info "Static check log: $static_log"
|
||||
}
|
||||
|
||||
run_metrics_context() {
|
||||
log_info "Collecting local metric symbol index for list-quorum review"
|
||||
local metric_log="$OUT_DIR/list-quorum-metrics-context.log"
|
||||
|
||||
if rg -n "record_stage_duration\(\"(store|sets|set_disks)_list_objects|store_list_objects_gather|set_disks_list_objects_list_path\"" "$PROJECT_ROOT/crates/ecstore/src/store/list_objects.rs" > "$metric_log"; then
|
||||
log_info "Metric context saved: $metric_log"
|
||||
else
|
||||
log_error "No metric symbols found in list_objects.rs"
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
run_live_listing_two_page_smoke() {
|
||||
local mode="$1"
|
||||
local log_file="$OUT_DIR/live-${mode}.log"
|
||||
: > "$log_file"
|
||||
|
||||
if ! command -v python3 >/dev/null 2>&1; then
|
||||
log_warn "python3 missing, skip live smoke"
|
||||
echo "live pagination smoke: skipped (missing python3)" | tee -a "$log_file"
|
||||
return 0
|
||||
fi
|
||||
|
||||
local access_key="${RUSTFS_ACCESS_KEY:-${WARP_ACCESS_KEY:-}}"
|
||||
local secret_key="${RUSTFS_SECRET_KEY:-${WARP_SECRET_KEY:-}}"
|
||||
if [[ -z "$access_key" || -z "$secret_key" ]]; then
|
||||
log_warn "Missing credentials, skip live smoke"
|
||||
echo "live pagination smoke: skipped (missing credentials)" | tee -a "$log_file"
|
||||
return 0
|
||||
fi
|
||||
|
||||
if [[ -z "$LIVE_LIST_BUCKET" ]]; then
|
||||
log_warn "LIVE_LIST_BUCKET is empty, skip live smoke"
|
||||
echo "live pagination smoke: skipped (bucket not configured)" | tee -a "$log_file"
|
||||
return 0
|
||||
fi
|
||||
|
||||
if ! curl -fsS "${ENDPOINT}/health/ready" >/dev/null 2>&1; then
|
||||
log_error "health check failed: ${ENDPOINT}/health/ready"
|
||||
echo "live pagination smoke: failed (health check)" | tee -a "$log_file"
|
||||
return 1
|
||||
fi
|
||||
|
||||
local base_path="$LIVE_LIST_BUCKET"
|
||||
if [[ -n "$LIVE_LIST_PREFIX" ]]; then
|
||||
base_path="${LIVE_LIST_BUCKET}/${LIVE_LIST_PREFIX}"
|
||||
fi
|
||||
|
||||
local base_url="${ENDPOINT}/${base_path}?list-type=2&max-keys=${LIVE_LIST_MAX_KEYS}"
|
||||
local sig_target="aws:amz:${LIVE_LIST_REGION}:s3"
|
||||
|
||||
local start_ms page1 page2 token1 page1_key_count page2_key_count
|
||||
start_ms="$(timestamp_ms)"
|
||||
page1=$(curl -fsS \
|
||||
--user "${access_key}:${secret_key}" \
|
||||
--aws-sigv4 "$sig_target" \
|
||||
-H "x-amz-content-sha256: UNSIGNED-PAYLOAD" \
|
||||
"$base_url")
|
||||
page1_duration_ms="$(( $(timestamp_ms) - start_ms ))"
|
||||
|
||||
page1_key_count=$(extract_xml_keys "$page1" | grep -cv '^$' || true)
|
||||
token1=$(printf '%s' "$page1" | sed -n 's:.*<NextContinuationToken>\(.*\)</NextContinuationToken>.*:\1:p' | head -n 1)
|
||||
local -a page1_keys=()
|
||||
while IFS= read -r key; do
|
||||
[[ -n "$key" ]] || continue
|
||||
page1_keys+=("$key")
|
||||
done < <(extract_xml_keys "$page1")
|
||||
|
||||
log_info "Mode=${mode} first page keys=${page1_key_count} duration=${page1_duration_ms}ms" | tee -a "$log_file"
|
||||
|
||||
if [[ -z "$token1" ]]; then
|
||||
log_info "Mode=${mode} live pagination smoke: single page only" | tee -a "$log_file"
|
||||
echo "live pagination smoke: pass (single page, key_count=${page1_key_count}, duration_ms=${page1_duration_ms})" | tee -a "$log_file"
|
||||
return 0
|
||||
fi
|
||||
|
||||
local encoded_token
|
||||
encoded_token=$(printf '%s' "$token1" | python3 - <<'PY'
|
||||
import urllib.parse,sys
|
||||
print(urllib.parse.quote(sys.stdin.read().strip(), safe=""))
|
||||
PY)
|
||||
|
||||
start_ms="$(timestamp_ms)"
|
||||
page2=$(curl -fsS \
|
||||
--user "${access_key}:${secret_key}" \
|
||||
--aws-sigv4 "$sig_target" \
|
||||
-H "x-amz-content-sha256: UNSIGNED-PAYLOAD" \
|
||||
"${base_url}&continuation-token=${encoded_token}")
|
||||
page2_duration_ms="$(( $(timestamp_ms) - start_ms ))"
|
||||
|
||||
page2_key_count=$(extract_xml_keys "$page2" | grep -cv '^$' || true)
|
||||
|
||||
if (( page2_key_count == 0 )); then
|
||||
log_error "Mode=${mode} live smoke: page2 empty"
|
||||
return 1
|
||||
fi
|
||||
|
||||
local duplicate=0
|
||||
local -a dkeys=()
|
||||
local -a page2_keys=()
|
||||
while IFS= read -r key; do
|
||||
[[ -n "$key" ]] || continue
|
||||
page2_keys+=("$key")
|
||||
done < <(extract_xml_keys "$page2")
|
||||
|
||||
for key in "${page2_keys[@]}"; do
|
||||
if printf '%s\n' "${page1_keys[@]}" | grep -Fx -- "$key" >/dev/null 2>&1; then
|
||||
duplicate=1
|
||||
dkeys+=("$key")
|
||||
fi
|
||||
done
|
||||
|
||||
if (( duplicate )); then
|
||||
log_error "Mode=${mode} live pagination duplicates detected: ${dkeys[*]}"
|
||||
echo "live pagination smoke: fail (duplicate keys)" | tee -a "$log_file"
|
||||
return 1
|
||||
fi
|
||||
|
||||
echo "Mode=${mode}" | tee -a "$log_file"
|
||||
echo "first-page-count=${page1_key_count}" | tee -a "$log_file"
|
||||
echo "second-page-count=${page2_key_count}" | tee -a "$log_file"
|
||||
echo "page1-duration-ms=${page1_duration_ms}" | tee -a "$log_file"
|
||||
echo "page2-duration-ms=${page2_duration_ms}" | tee -a "$log_file"
|
||||
echo "live pagination smoke: pass (two-page continuity)" | tee -a "$log_file"
|
||||
|
||||
return 0
|
||||
}
|
||||
|
||||
run_mode_series() {
|
||||
if [[ -z "${MODE_LIST}" ]]; then
|
||||
log_warn "MODE_LIST is empty; skip mode series"
|
||||
return 0
|
||||
fi
|
||||
|
||||
local mode
|
||||
local mode_count=0
|
||||
IFS=',' read -r -a modes <<< "$MODE_LIST"
|
||||
|
||||
for mode in "${modes[@]}"; do
|
||||
[[ -z "$mode" ]] && continue
|
||||
log_info "Live check for list quorum mode=${mode}"
|
||||
mode=$(printf '%s' "$mode" | tr -d '[:space:]')
|
||||
|
||||
if [[ -n "$SERVER_RESTART_CMD" ]]; then
|
||||
local mode_cmd=""
|
||||
local mode_env_file=""
|
||||
|
||||
if [[ "$mode" == "default" ]]; then
|
||||
mode_cmd="$SERVER_RESTART_CMD"
|
||||
else
|
||||
mode_env_file="$OUT_DIR/.issue787-${mode}.env"
|
||||
printf 'RUSTFS_LIST_OBJECTS_QUORUM=%s\n' "$mode" > "$mode_env_file"
|
||||
mode_cmd="RUSTFS_TUNING_ENV_FILE=${mode_env_file} $SERVER_RESTART_CMD"
|
||||
fi
|
||||
|
||||
log_info "Restarting server for mode=${mode}"
|
||||
(cd "$PROJECT_ROOT" && eval "$mode_cmd")
|
||||
|
||||
if ! wait_for_health; then
|
||||
log_error "health check timeout during mode=${mode} restart"
|
||||
return 1
|
||||
fi
|
||||
elif (( mode_count > 0 )); then
|
||||
log_warn "SERVER_RESTART_CMD unset, mode=${mode} does not auto-switch. Skip remaining modes."
|
||||
break
|
||||
fi
|
||||
|
||||
mode_count=$(( mode_count + 1 ))
|
||||
|
||||
run_live_listing_two_page_smoke "$mode"
|
||||
done
|
||||
}
|
||||
|
||||
parse_args() {
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--full|--no-skip-live)
|
||||
SKIP_LIVE="false"
|
||||
shift
|
||||
;;
|
||||
-h|--help)
|
||||
usage
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
log_error "unknown arg: $1"
|
||||
usage
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
}
|
||||
|
||||
main() {
|
||||
parse_args "$@"
|
||||
|
||||
require_cmd cargo
|
||||
require_cmd rg
|
||||
require_cmd tee
|
||||
require_cmd curl
|
||||
|
||||
mkdir -p "$OUT_DIR"
|
||||
|
||||
run_unit_checks
|
||||
run_static_checks
|
||||
run_metrics_context
|
||||
|
||||
if [[ "$SKIP_LIVE" != "true" ]]; then
|
||||
require_cmd python3
|
||||
run_mode_series
|
||||
fi
|
||||
|
||||
log_info "Validation summary:"
|
||||
log_info " - logs: $OUT_DIR"
|
||||
log_info " - status: pass"
|
||||
}
|
||||
|
||||
main "$@"
|
||||
@@ -0,0 +1,102 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# Acceptance runner for rustfs/backlog#841.
|
||||
PROJECT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
OUT_DIR="${OUT_DIR:-${PROJECT_ROOT}/target/issue-841-acceptance-$(date +%Y%m%d-%H%M%S)}"
|
||||
|
||||
log_info() { printf '[INFO] %s\n' "$*"; }
|
||||
log_error() { printf '[ERROR] %s\n' "$*" >&2; }
|
||||
|
||||
usage() {
|
||||
cat <<'USAGE'
|
||||
Usage:
|
||||
scripts/validate_issue_841_list_objects_observability.sh
|
||||
|
||||
Environment:
|
||||
OUT_DIR output directory (default: target/issue-841-acceptance-<ts>)
|
||||
USAGE
|
||||
}
|
||||
|
||||
require_cmd() {
|
||||
if ! command -v "$1" >/dev/null 2>&1; then
|
||||
log_error "command not found: $1"
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
run_unit_checks() {
|
||||
log_info "Running ListObjects observability unit checks"
|
||||
local test_log="$OUT_DIR/issue-841-tests.log"
|
||||
: > "$test_log"
|
||||
|
||||
(
|
||||
cd "$PROJECT_ROOT"
|
||||
cargo test -p rustfs-io-metrics list_objects_metrics -- --nocapture
|
||||
cargo test -p rustfs-ecstore list_objects -- --nocapture
|
||||
) 2>&1 | tee "$test_log"
|
||||
|
||||
log_info "Unit test log: $test_log"
|
||||
}
|
||||
|
||||
run_static_checks() {
|
||||
log_info "Running ListObjects observability static checks"
|
||||
local static_log="$OUT_DIR/static-checks.log"
|
||||
: > "$static_log"
|
||||
|
||||
local required_patterns=(
|
||||
"rustfs_s3_list_objects_gather_total"
|
||||
"rustfs_s3_list_objects_gather_scan_amplification"
|
||||
"rustfs_s3_list_objects_merge_fan_in"
|
||||
"record_list_objects_gather"
|
||||
"record_list_objects_merge"
|
||||
"init_list_objects_metrics"
|
||||
"listobjects-v2-baseline-fixtures"
|
||||
)
|
||||
|
||||
local pattern
|
||||
for pattern in "${required_patterns[@]}"; do
|
||||
if rg --no-ignore -n "$pattern" "$PROJECT_ROOT/crates" "$PROJECT_ROOT/rustfs" "$PROJECT_ROOT/docs" "$PROJECT_ROOT/scripts" >> "$static_log"; then
|
||||
log_info "Found required symbol: $pattern"
|
||||
else
|
||||
log_error "Missing required symbol: $pattern"
|
||||
log_error "Static check log: $static_log"
|
||||
return 1
|
||||
fi
|
||||
done
|
||||
|
||||
log_info "Static check log: $static_log"
|
||||
}
|
||||
|
||||
parse_args() {
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
-h|--help)
|
||||
usage
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
log_error "unknown arg: $1"
|
||||
usage
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
}
|
||||
|
||||
main() {
|
||||
parse_args "$@"
|
||||
require_cmd cargo
|
||||
require_cmd rg
|
||||
require_cmd tee
|
||||
|
||||
mkdir -p "$OUT_DIR"
|
||||
run_unit_checks
|
||||
run_static_checks
|
||||
|
||||
log_info "Validation summary:"
|
||||
log_info " - logs: $OUT_DIR"
|
||||
log_info " - status: pass"
|
||||
}
|
||||
|
||||
main "$@"
|
||||
Reference in New Issue
Block a user