chore(scripts): add scripts index and archive one-shot scripts (#4822)

chore(scripts): index scripts/ and archive 29 one-shot scripts

backlog#1153 infra-13. scripts/ had 80+ unlabelled top-level entries
mixing CI gates with finished one-shot issue-validation scripts.

- scripts/README.md — one index row per entry with status
  (ci-gate / dev-tool / archived), purpose, and wiring; subdirectories
  get one row each. run_scanner_benchmarks.sh is annotated
  "disposition owned by backlog perf-10" and deliberately untouched.
- git mv 29 confirmed-stale one-shot entries to scripts/archive/:
  11 issue-scoped validation/perf-capture scripts, the 5-script
  backlog#706 large-PUT breakdown family, the 4-file GET-optimization
  stress suite, 2 gt1g one-shots, and 7 other orphaned one-shots.
  Evidence: a whole-tree boundary-aware reference census showed zero
  references from CI/Makefiles/docs/code for every moved entry (or
  references only from other scripts inside the same archived set);
  re-run after the move shows zero dangling references.
- docs/testing/README.md links the index.

Moves only — no script content changed.
This commit is contained in:
Zhengchao An
2026-07-15 11:43:26 +08:00
committed by GitHub
parent 5294f36669
commit eb392f24d6
31 changed files with 143 additions and 0 deletions
+119
View File
@@ -0,0 +1,119 @@
# GET Optimization Stress Test Scripts
## Quick Start
### Prerequisites
- `warp` installed (https://github.com/minio/warp)
- `mc` configured with access to the RustFS server
- RustFS server running with GET optimizations enabled
### Quick Validation (5 minutes)
```bash
./scripts/quick-validate-get-optimization.sh localhost:9000
```
This runs basic functional tests:
- Data integrity verification
- Concurrent GET stability
- Early-stop behavior validation
### Full Stress Test (30+ minutes)
```bash
./scripts/stress-test-get-optimization.sh localhost:9000 ./stress-results
```
This runs comprehensive tests:
- Data correctness validation (1KB, 1MB, 10MB objects)
- Concurrent GET stress test (16, 64, 256 concurrency)
- Mixed read/write workload
- Early-stop behavior under load
## Environment Variables
| Variable | Default | Description |
|----------|---------|-------------|
| `WARP_ACCESS_KEY` | rustfsadmin | S3 access key |
| `WARP_SECRET_KEY` | rustfsadmin | S3 secret key |
| `MC_ALIAS` | rustfs | mc alias for the server |
| `TEST_BUCKET` | auto-generated | Test bucket name |
| `TEST_DURATION` | 300s | Duration for stress tests |
| `CONCURRENCY` | 64 | Default concurrency level |
## Test Scenarios
### 1. Data Correctness Validation
Verifies that GET returns correct data with early-stop enabled:
- Uploads random data
- Downloads and compares MD5 hash
- Tests different object sizes (1KB, 1MB, 10MB)
### 2. Concurrent GET Stress Test
Tests performance under high concurrency:
- Object sizes: 1KiB, 1MiB, 4MiB, 10MiB
- Concurrency levels: 16, 64, 256
- Duration: configurable (default 300s)
### 3. Mixed Read/Write Stress Test
Tests stability under concurrent read/write:
- 50% reads, 50% writes
- 1MB objects
- 64 concurrent operations
### 4. Early-Stop Behavior Validation
Verifies early-stop works correctly:
- 100 sequential reads of the same object
- Verifies data size matches expected
- Checks for any download failures
## Expected Results
### Success Criteria
- **Data Correctness**: 100% pass rate (no data corruption)
- **Concurrent GET**: No errors, consistent latency
- **Mixed Workload**: No deadlocks or data corruption
- **Early-Stop**: All reads return correct data size
### Performance Baselines
| Object Size | Expected Throughput | Expected p95 Latency |
|-------------|--------------------|--------------------|
| 1KiB | > 5 MiB/s | < 10ms |
| 1MiB | > 500 MiB/s | < 20ms |
| 4MiB | > 1000 MiB/s | < 30ms |
| 10MiB | > 2000 MiB/s | < 50ms |
## Troubleshooting
### Common Issues
1. **warp not found**: Install with `go install github.com/minio/warp@latest`
2. **mc not configured**: Run `mc alias set rustfs http://localhost:9000 admin password`
3. **Connection refused**: Verify RustFS is running and accessible
4. **Permission denied**: Check S3 credentials
### Debug Mode
Enable debug logging:
```bash
RUST_LOG=rustfs_ecstore::bucket::lifecycle=debug ./scripts/stress-test-get-optimization.sh
```
## Output Files
| File | Description |
|------|-------------|
| `test-config.txt` | Test configuration |
| `correctness-results.txt` | Data correctness results |
| `early-stop-results.txt` | Early-stop validation results |
| `get-*.json` | Concurrent GET performance (warp JSON) |
| `mixed-*.json` | Mixed workload performance (warp JSON) |
| `correctness-errors.log` | Data correctness errors (if any) |
| `early-stop-errors.log` | Early-stop errors (if any) |
@@ -0,0 +1,322 @@
#!/usr/bin/env python3
"""Summarize per-round PUT service metric deltas from before/after snapshots."""
from __future__ import annotations
import argparse
import csv
import re
from collections import defaultdict
from pathlib import Path
PROM_LINE_RE = re.compile(
r"^([a-zA-Z_:][a-zA-Z0-9_:]*)(?:\{(.*)\})?\s+([-+]?(?:\d+(?:\.\d*)?|\.\d+)(?:[eE][-+]?\d+)?)"
)
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--capture-csv", required=True, help="aggregate_service_metrics_captures.csv path")
parser.add_argument("--delta-csv", required=True, help="Output generic metric delta CSV")
parser.add_argument("--path-summary-csv", required=True, help="Output PUT path delta summary CSV")
parser.add_argument("--stage-summary-csv", required=True, help="Output PUT stage duration summary CSV")
return parser.parse_args()
def split_label_items(labels: str) -> list[str]:
items: list[str] = []
start = 0
in_quotes = False
escaped = False
for index, char in enumerate(labels):
if escaped:
escaped = False
continue
if char == "\\":
escaped = True
continue
if char == '"':
in_quotes = not in_quotes
continue
if char == "," and not in_quotes:
items.append(labels[start:index])
start = index + 1
items.append(labels[start:])
return [item.strip() for item in items if item.strip()]
def parse_labels(labels: str) -> dict[str, str]:
parsed: dict[str, str] = {}
for item in split_label_items(labels):
key, sep, value = item.partition("=")
if not sep:
continue
value = value.strip()
if value.startswith('"') and value.endswith('"') and len(value) >= 2:
value = value[1:-1]
value = value.replace(r"\"", '"').replace(r"\\", "\\")
parsed[key.strip()] = value
return parsed
def canonical_labels(labels: dict[str, str]) -> str:
return ",".join(f'{key}="{escape_label_value(labels[key])}"' for key in sorted(labels))
def escape_label_value(value: str) -> str:
return value.replace("\\", r"\\").replace('"', r"\"").replace("\n", r"\n")
def parse_prom(path: Path) -> dict[tuple[str, str], float]:
metrics: dict[tuple[str, str], float] = {}
with path.open("r", encoding="utf-8") as file:
for line in file:
line = line.strip()
if not line or line.startswith("#"):
continue
match = PROM_LINE_RE.match(line)
if not match:
continue
metric, raw_labels, raw_value = match.groups()
labels = canonical_labels(parse_labels(raw_labels or ""))
try:
value = float(raw_value)
except ValueError:
continue
metrics[(metric, labels)] = value
return metrics
def classify_metric(metric: str) -> str:
if metric == "rustfs_s3_put_object_path_total":
return "path_total"
if metric == "rustfs_s3_put_object_diagnostics_total":
return "diagnostics_total"
if metric.endswith("_bucket"):
return "histogram_bucket"
if metric.endswith("_count"):
return "histogram_count"
if metric.endswith("_sum"):
return "histogram_sum"
if metric.endswith("_total"):
return "counter_total"
return "gauge_delta"
def is_monotonic_metric(metric: str) -> bool:
return (
metric.endswith("_total")
or metric.endswith("_bucket")
or metric.endswith("_count")
or metric.endswith("_sum")
)
def group_capture_rows(capture_csv: Path) -> dict[tuple[str, str, str, str, str, str], dict[str, dict[str, str]]]:
groups: dict[tuple[str, str, str, str, str, str], dict[str, dict[str, str]]] = defaultdict(dict)
with capture_csv.open("r", encoding="utf-8", newline="") as file:
for row in csv.DictReader(file):
phase = row.get("phase", "")
if phase not in {"before", "after"}:
continue
key = (
row.get("concurrency", ""),
row.get("run_dir", ""),
row.get("size", ""),
row.get("tool", ""),
row.get("round", ""),
row.get("attempt", ""),
)
groups[key][phase] = row
return groups
def label_value(labels: str, name: str) -> str:
return parse_labels(labels).get(name, "")
def write_outputs(args: argparse.Namespace) -> None:
capture_csv = Path(args.capture_csv)
delta_csv = Path(args.delta_csv)
path_summary_csv = Path(args.path_summary_csv)
stage_summary_csv = Path(args.stage_summary_csv)
for output in (delta_csv, path_summary_csv, stage_summary_csv):
output.parent.mkdir(parents=True, exist_ok=True)
groups = group_capture_rows(capture_csv)
delta_fields = [
"concurrency",
"run_dir",
"size",
"tool",
"round",
"attempt",
"source",
"before_status",
"after_status",
"metric",
"labels",
"before",
"after",
"delta",
"classification",
]
path_fields = [
"concurrency",
"run_dir",
"size",
"tool",
"round",
"attempt",
"path",
"delta",
"before_status",
"after_status",
]
stage_fields = [
"concurrency",
"run_dir",
"size",
"tool",
"round",
"attempt",
"stage",
"count_delta",
"sum_delta",
"avg_ms",
"before_status",
"after_status",
]
with (
delta_csv.open("w", encoding="utf-8", newline="") as delta_file,
path_summary_csv.open("w", encoding="utf-8", newline="") as path_file,
stage_summary_csv.open("w", encoding="utf-8", newline="") as stage_file,
):
delta_writer = csv.DictWriter(delta_file, fieldnames=delta_fields)
path_writer = csv.DictWriter(path_file, fieldnames=path_fields)
stage_writer = csv.DictWriter(stage_file, fieldnames=stage_fields)
delta_writer.writeheader()
path_writer.writeheader()
stage_writer.writeheader()
for key in sorted(groups):
phases = groups[key]
before_row = phases.get("before")
after_row = phases.get("after")
if not before_row or not after_row:
continue
before_path = Path(before_row.get("snapshot_file", ""))
after_path = Path(after_row.get("snapshot_file", ""))
if before_row.get("status") != "ok" or after_row.get("status") != "ok":
continue
if not before_path.is_file() or not after_path.is_file():
continue
before = parse_prom(before_path)
after = parse_prom(after_path)
concurrency, run_dir, size, tool, round_no, attempt = key
source = after_row.get("source", before_row.get("source", ""))
before_status = before_row.get("status", "")
after_status = after_row.get("status", "")
stage_counts: dict[str, float] = defaultdict(float)
stage_sums: dict[str, float] = defaultdict(float)
for metric_key in sorted(after):
metric, labels = metric_key
before_value = before.get(metric_key, 0.0)
after_value = after.get(metric_key, 0.0)
delta = after_value - before_value
if is_monotonic_metric(metric) and delta < 0:
continue
if delta == 0:
continue
classification = classify_metric(metric)
delta_writer.writerow(
{
"concurrency": concurrency,
"run_dir": run_dir,
"size": size,
"tool": tool,
"round": round_no,
"attempt": attempt,
"source": source,
"before_status": before_status,
"after_status": after_status,
"metric": metric,
"labels": labels,
"before": f"{before_value:.12g}",
"after": f"{after_value:.12g}",
"delta": f"{delta:.12g}",
"classification": classification,
}
)
if metric == "rustfs_s3_put_object_path_total":
path_writer.writerow(
{
"concurrency": concurrency,
"run_dir": run_dir,
"size": size,
"tool": tool,
"round": round_no,
"attempt": attempt,
"path": label_value(labels, "path"),
"delta": f"{delta:.12g}",
"before_status": before_status,
"after_status": after_status,
}
)
if metric.endswith("_stage_duration_ms_count"):
stage = label_value(labels, "stage")
if stage:
stage_counts[stage] += delta
elif metric.endswith("_stage_duration_ms_sum"):
stage = label_value(labels, "stage")
if stage:
stage_sums[stage] += delta
elif metric.endswith("_stage_duration_seconds_count"):
stage = label_value(labels, "stage")
if stage:
stage_counts[stage] += delta
elif metric.endswith("_stage_duration_seconds_sum"):
stage = label_value(labels, "stage")
if stage:
stage_sums[stage] += delta * 1000.0
for stage in sorted(set(stage_counts) | set(stage_sums)):
count_delta = stage_counts.get(stage, 0.0)
if count_delta <= 0:
continue
sum_delta = stage_sums.get(stage, 0.0)
avg_ms = sum_delta / count_delta if count_delta else 0.0
stage_writer.writerow(
{
"concurrency": concurrency,
"run_dir": run_dir,
"size": size,
"tool": tool,
"round": round_no,
"attempt": attempt,
"stage": stage,
"count_delta": f"{count_delta:.12g}",
"sum_delta": f"{sum_delta:.12g}",
"avg_ms": f"{avg_ms:.6f}",
"before_status": before_status,
"after_status": after_status,
}
)
def main() -> None:
write_outputs(parse_args())
if __name__ == "__main__":
main()
+166
View File
@@ -0,0 +1,166 @@
#!/usr/bin/env bash
# Benchmark script for GET small-file optimization (SF01-SF07)
# Matches baseline parameters from issue714-local-single-machine-multidisk-get-2026-06-26.md
set -euo pipefail
WARP_HOST="${WARP_HOST:-127.0.0.1:19031}"
export WARP_ACCESS_KEY="${WARP_ACCESS_KEY:-rustfsadmin}"
export WARP_SECRET_KEY="${WARP_SECRET_KEY:-rustfsadmin}"
SIZES="1KiB 4KiB 10KiB 100KiB 1MiB"
CONCURRENCY=32
DURATION=10s
ROUNDS=3
COOLDOWN=10
OBJECTS=8
OUT_DIR="target/bench/sf-optimization-$(date +%Y%m%d-%H%M%S)"
mkdir -p "$OUT_DIR"
echo "=========================================="
echo "GET Small-File Optimization Benchmark"
echo "=========================================="
echo "Host: $WARP_HOST"
echo "Output: $OUT_DIR"
echo ""
# Save environment info
cat > "$OUT_DIR/meta.env" <<EOF
DATE=$(date -u +%Y-%m-%dT%H:%M:%SZ)
HOST=$WARP_HOST
CONCURRENCY=$CONCURRENCY
DURATION=$DURATION
ROUNDS=$ROUNDS
OBJECTS=$OBJECTS
BRANCH=$(git rev-parse --abbrev-ref HEAD)
COMMIT=$(git rev-parse --short HEAD)
RUST_VERSION=$(rustc --version)
EOF
# CSV header for summary
echo "size,round,throughput_mib_s,requests_per_s,p50_ms,total_bytes,errors" > "$OUT_DIR/summary.csv"
for size in $SIZES; do
echo ""
echo "=========================================="
echo "Testing: $size"
echo "=========================================="
for round in $(seq 1 $ROUNDS); do
echo " Round $round/$ROUNDS..."
json_file="$OUT_DIR/get-${size}-round${round}.json"
warp get \
--host="$WARP_HOST" \
--obj.size="$size" \
--concurrent=$CONCURRENCY \
--duration=$DURATION \
--objects=$OBJECTS \
--noclear \
--lookup=path \
--analyze.out="$json_file" \
2>/dev/null
# Extract key metrics from JSON
if [ -f "$json_file" ]; then
throughput=$(python3 -c "
import json, sys
with open('$json_file') as f:
data = json.load(f)
for op in data.get('operations', []):
if op.get('operation') == 'GET':
mb_s = op.get('mb_per_sec', 0)
rps = op.get('requests_per_sec', 0)
p50 = 0
for t in op.get('throughput', []):
pass
# Get p50 from time_series_aggregated
tsa = op.get('time_series_aggregated', {})
if tsa:
p50 = tsa.get('median_ms', 0)
total = op.get('total_bytes', 0)
errors = op.get('requests_errors', 0) or 0
print(f'{mb_s:.2f},{rps:.2f},{p50:.1f},{total},{errors}')
sys.exit(0)
print('0,0,0,0,0')
" 2>/dev/null || echo "0,0,0,0,0")
echo "$size,$round,$throughput" >> "$OUT_DIR/summary.csv"
echo " -> $throughput"
else
echo "$size,$round,0,0,0,0,0" >> "$OUT_DIR/summary.csv"
echo " -> FAILED (no output)"
fi
echo " Cooling down ${COOLDOWN}s..."
sleep $COOLDOWN
done
# Extra cooldown between sizes
echo " Extra cooldown ${COOLDOWN}s between sizes..."
sleep $COOLDOWN
done
echo ""
echo "=========================================="
echo "Benchmark Complete"
echo "=========================================="
echo "Results: $OUT_DIR/summary.csv"
echo ""
# Print summary table
echo "Summary (MiB/s):"
echo "---------------------------------------------------"
printf "%-10s" "Size"
for r in $(seq 1 $ROUNDS); do
printf "%-12s" "Round $r"
done
printf "%-12s\n" "Median"
echo "---------------------------------------------------"
for size in $SIZES; do
printf "%-10s" "$size"
values=()
for r in $(seq 1 $ROUNDS); do
val=$(grep "^$size,$r," "$OUT_DIR/summary.csv" | cut -d',' -f3)
values+=("$val")
printf "%-12s" "$val"
done
# Calculate median
median=$(printf '%s\n' "${values[@]}" | sort -n | sed -n "$(((${#values[@]}+1)/2))p")
printf "%-12s\n" "$median"
done
echo ""
echo "Comparison with baseline (MiB/s):"
echo "---------------------------------------------------"
printf "%-10s %-12s %-12s %-12s %-12s\n" "Size" "MinIO" "RustFS main" "This branch" "vs main"
echo "---------------------------------------------------"
# Baseline data from issue714
declare -A MINIO_DATA=(
["1KiB"]="21.15" ["4KiB"]="51.19" ["10KiB"]="201.23" ["100KiB"]="1142.89" ["1MiB"]="7264.10"
)
declare -A MAIN_DATA=(
["1KiB"]="2.88" ["4KiB"]="11.30" ["10KiB"]="28.56" ["100KiB"]="277.49" ["1MiB"]="2270.27"
)
for size in $SIZES; do
values=()
for r in $(seq 1 $ROUNDS); do
val=$(grep "^$size,$r," "$OUT_DIR/summary.csv" | cut -d',' -f3)
values+=("$val")
done
median=$(printf '%s\n' "${values[@]}" | sort -n | sed -n "$(((${#values[@]}+1)/2))p")
main_val=${MAIN_DATA[$size]}
minio_val=${MINIO_DATA[$size]}
if [ "$main_val" != "0" ] && [ "$main_val" != "" ]; then
pct_change=$(python3 -c "print(f'{(($median - $main_val) / $main_val * 100):+.1f}%')" 2>/dev/null || echo "N/A")
else
pct_change="N/A"
fi
printf "%-10s %-12s %-12s %-12s %-12s\n" "$size" "$minio_val" "$main_val" "$median" "$pct_change"
done
@@ -0,0 +1,491 @@
#!/usr/bin/env bash
set -euo pipefail
# Helper artifact collector for rustfs/backlog#706 large-object PUT stage-breakdown runs.
# Designed to pair with scripts/run_put_large_stage_breakdown.sh and store
# supporting evidence under:
# <run-root>/captures/<label>/
#
# Captures:
# - health and readiness snapshots
# - signed admin metrics snapshots per endpoint
# - optional plain Prometheus text snapshots
# - optional host telemetry (pidstat/iostat/mpstat)
# - per-sample process snapshots when a rustfs pid is available
RUN_ROOT=""
LABEL=""
ENDPOINT=""
ACCESS_KEY="${RUSTFS_ACCESS_KEY:-}"
SECRET_KEY=""
SECRET_KEY_ENV="RUSTFS_SECRET_KEY"
REGION="us-east-1"
METRICS_ENDPOINTS=""
PROM_METRICS_URLS=""
DURATION_SECS=180
INTERVAL_SECS=15
RUSTFS_PID=""
SKIP_HOST_TELEMETRY=false
AWSCURL_BIN="awscurl"
CURL_BIN="curl"
JQ_BIN="jq"
DRY_RUN=false
AWSCURL_AVAILABLE=true
usage() {
cat <<'USAGE'
Usage:
scripts/collect_put_large_stage_breakdown_artifacts.sh --run-root <dir> \
--endpoint <url> [options]
Required:
--run-root <dir> Benchmark run root, usually from
scripts/run_put_large_stage_breakdown.sh
--endpoint <url> RustFS endpoint, e.g. http://127.0.0.1:9000
Auth / signed admin metrics:
--access-key <ak> Override RUSTFS_ACCESS_KEY
--secret-key-env <var> Secret-key environment variable
(default: RUSTFS_SECRET_KEY)
--region <name> SigV4 region (default: us-east-1)
Capture options:
--label <name> Capture label (default: capture-<timestamp>)
--metrics-endpoints <csv> Signed admin metrics endpoints. If omitted,
defaults to --endpoint.
--prom-metrics-urls <csv> Optional plain Prometheus text endpoints
captured with curl.
--duration-secs <n> Total capture duration (default: 180)
--interval-secs <n> Sample interval (default: 15)
--rustfs-pid <pid> RustFS pid. Auto-detect with pidof rustfs if omitted.
--skip-host-telemetry Skip pidstat/iostat/mpstat collection.
--dry-run Print intended actions without live capture.
Output:
<run-root>/captures/<label>/
capture-meta.txt
capture-layout.txt
capture-report.md
health/
metrics/
prom/
host/
USAGE
}
require_cmd() {
if ! command -v "$1" >/dev/null 2>&1; then
echo "ERROR: command not found: $1" >&2
exit 1
fi
}
arg_value() {
local flag="$1"
local value="${2:-}"
if [[ -z "$value" || "$value" == --* ]]; then
echo "ERROR: missing value for $flag" >&2
exit 1
fi
printf '%s\n' "$value"
}
parse_args() {
while [[ $# -gt 0 ]]; do
case "$1" in
--run-root) RUN_ROOT="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--label) LABEL="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--endpoint) ENDPOINT="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--access-key) ACCESS_KEY="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--secret-key-env) SECRET_KEY_ENV="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--region) REGION="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--metrics-endpoints) METRICS_ENDPOINTS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--prom-metrics-urls) PROM_METRICS_URLS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--duration-secs) DURATION_SECS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--interval-secs) INTERVAL_SECS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--rustfs-pid) RUSTFS_PID="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--skip-host-telemetry) SKIP_HOST_TELEMETRY=true; shift ;;
--dry-run) DRY_RUN=true; shift ;;
-h|--help) usage; exit 0 ;;
*)
echo "ERROR: unknown arg: $1" >&2
usage
exit 1
;;
esac
done
}
is_nonnegative_integer() {
[[ "$1" =~ ^[0-9]+$ ]]
}
is_positive_integer() {
[[ "$1" =~ ^[1-9][0-9]*$ ]]
}
validate_args() {
if [[ -z "$RUN_ROOT" || -z "$ENDPOINT" ]]; then
echo "ERROR: --run-root and --endpoint are required" >&2
exit 1
fi
if ! [[ "$SECRET_KEY_ENV" =~ ^[A-Za-z_][A-Za-z0-9_]*$ ]]; then
echo "ERROR: --secret-key-env must be a valid environment variable name" >&2
exit 1
fi
if ! is_nonnegative_integer "$DURATION_SECS" || ! is_positive_integer "$INTERVAL_SECS"; then
echo "ERROR: --duration-secs must be >= 0 and --interval-secs must be > 0" >&2
exit 1
fi
SECRET_KEY="${!SECRET_KEY_ENV:-}"
if [[ -z "$LABEL" ]]; then
LABEL="capture-$(date -u +%Y%m%dT%H%M%SZ)"
fi
}
setup_output() {
CAPTURE_ROOT="${RUN_ROOT}/captures/${LABEL}"
HEALTH_DIR="${CAPTURE_ROOT}/health"
METRICS_DIR="${CAPTURE_ROOT}/metrics"
PROM_DIR="${CAPTURE_ROOT}/prom"
HOST_DIR="${CAPTURE_ROOT}/host"
mkdir -p "$HEALTH_DIR" "$METRICS_DIR" "$PROM_DIR" "$HOST_DIR"
}
endpoint_label() {
local endpoint="$1"
local label
label="${endpoint#*://}"
label="${label%%/*}"
printf '%s' "$label" | tr -c 'A-Za-z0-9._-' '_'
}
csv_to_lines() {
local csv="$1"
local raw item
IFS=',' read -r -a arr <<< "$csv"
for raw in "${arr[@]}"; do
item="$(echo "$raw" | awk '{$1=$1;print}')"
[[ -z "$item" ]] && continue
echo "$item"
done
}
resolve_metrics_endpoints() {
if [[ -n "$METRICS_ENDPOINTS" ]]; then
csv_to_lines "$METRICS_ENDPOINTS"
else
printf '%s\n' "$ENDPOINT"
fi
}
resolve_pid() {
if [[ -n "$RUSTFS_PID" ]]; then
echo "$RUSTFS_PID"
return
fi
if command -v pidof >/dev/null 2>&1; then
pidof rustfs 2>/dev/null | awk '{print $1}' && return 0
fi
if command -v pgrep >/dev/null 2>&1; then
pgrep -f 'rustfs server' 2>/dev/null | head -n 1 && return 0
fi
if command -v ps >/dev/null 2>&1; then
ps -ef 2>/dev/null | awk '/[r]ustfs server/ {print $2; exit}' && return 0
fi
true
}
health_url() {
printf '%s/health\n' "${1%/}"
}
ready_url() {
printf '%s/health/ready\n' "${1%/}"
}
admin_metrics_url() {
printf '%s/rustfs/admin/v3/metrics?types=1&by-host=true&n=1\n' "${1%/}"
}
write_layout_file() {
cat > "${CAPTURE_ROOT}/capture-layout.txt" <<'EOF'
capture-meta.txt
capture-layout.txt
capture-report.md
health/
- health.<endpoint>.<index>.<timestamp>.txt
- ready.<endpoint>.<index>.<timestamp>.txt
metrics/
- admin-metrics.<endpoint>.<index>.<timestamp>.ndjson
prom/
- prometheus.<endpoint>.<index>.<timestamp>.prom
host/
- pidstat.txt
- iostat.txt
- mpstat.txt
- proc-status.<index>.<timestamp>.txt
- proc-io.<index>.<timestamp>.txt
- ps.<index>.<timestamp>.txt
EOF
}
write_meta_file() {
local git_commit git_branch
git_commit="$(git rev-parse HEAD 2>/dev/null || echo "unknown")"
git_branch="$(git branch --show-current 2>/dev/null || echo "unknown")"
cat > "${CAPTURE_ROOT}/capture-meta.txt" <<EOF
created_utc=$(date -u +%Y-%m-%dT%H:%M:%SZ)
run_root=${RUN_ROOT}
label=${LABEL}
endpoint=${ENDPOINT}
metrics_endpoints=${METRICS_ENDPOINTS:-$ENDPOINT}
prom_metrics_urls=${PROM_METRICS_URLS:-N/A}
region=${REGION}
duration_secs=${DURATION_SECS}
interval_secs=${INTERVAL_SECS}
rustfs_pid=${RUSTFS_PID:-auto}
resolved_pid=${RESOLVED_PID:-N/A}
skip_host_telemetry=${SKIP_HOST_TELEMETRY}
dry_run=${DRY_RUN}
git_branch=${git_branch}
git_commit=${git_commit}
access_key_present=$([[ -n "$ACCESS_KEY" ]] && echo true || echo false)
secret_key_env=${SECRET_KEY_ENV}
EOF
}
capture_health_sample() {
local index="$1"
local ts="$2"
local endpoint="$3"
local label file
label="$(endpoint_label "$endpoint")"
file="${HEALTH_DIR}/health.${label}.${index}.${ts}.txt"
if [[ "$DRY_RUN" == "true" ]]; then
echo "dry run" > "$file"
else
"$CURL_BIN" -fsS "$(health_url "$endpoint")" > "$file" 2>&1 || true
fi
file="${HEALTH_DIR}/ready.${label}.${index}.${ts}.txt"
if [[ "$DRY_RUN" == "true" ]]; then
echo "dry run" > "$file"
else
"$CURL_BIN" -fsS "$(ready_url "$endpoint")" > "$file" 2>&1 || true
fi
}
capture_admin_metrics_sample() {
local index="$1"
local ts="$2"
local endpoint="$3"
local label file
label="$(endpoint_label "$endpoint")"
file="${METRICS_DIR}/admin-metrics.${label}.${index}.${ts}.ndjson"
if [[ "$DRY_RUN" == "true" ]]; then
echo "dry run" > "$file"
return
fi
if [[ "$AWSCURL_AVAILABLE" != "true" ]]; then
echo "awscurl unavailable; signed admin metrics skipped" > "$file"
return
fi
if [[ -z "$ACCESS_KEY" || -z "$SECRET_KEY" ]]; then
echo "missing access/secret for signed metrics capture" > "$file"
return
fi
AWS_ACCESS_KEY_ID="$ACCESS_KEY" \
AWS_SECRET_ACCESS_KEY="$SECRET_KEY" \
AWS_DEFAULT_REGION="$REGION" \
"$AWSCURL_BIN" \
--service s3 \
--region "$REGION" \
--request GET \
"$(admin_metrics_url "$endpoint")" \
> "$file" 2>&1 || true
}
capture_prometheus_sample() {
local index="$1"
local ts="$2"
local endpoint="$3"
local label file
label="$(endpoint_label "$endpoint")"
file="${PROM_DIR}/prometheus.${label}.${index}.${ts}.prom"
if [[ "$DRY_RUN" == "true" ]]; then
echo "dry run" > "$file"
else
"$CURL_BIN" -fsS "$endpoint" > "$file" 2>&1 || true
fi
}
capture_proc_snapshot() {
local index="$1"
local ts="$2"
local pid="$3"
[[ -n "$pid" ]] || return 0
if [[ -r "/proc/${pid}/status" ]]; then
cp "/proc/${pid}/status" "${HOST_DIR}/proc-status.${index}.${ts}.txt" || true
fi
if [[ -r "/proc/${pid}/io" ]]; then
cp "/proc/${pid}/io" "${HOST_DIR}/proc-io.${index}.${ts}.txt" || true
fi
if command -v ps >/dev/null 2>&1; then
ps -p "$pid" -o pid,ppid,stat,pcpu,pmem,rss,vsz,etime,args > "${HOST_DIR}/ps.${index}.${ts}.txt" 2>&1 || true
fi
}
start_host_telemetry() {
HOST_TELEMETRY_PIDS=()
if [[ "$SKIP_HOST_TELEMETRY" == "true" ]]; then
return
fi
local count
count=$(( DURATION_SECS / INTERVAL_SECS + 1 ))
(( count < 1 )) && count=1
if [[ -n "$RESOLVED_PID" ]] && command -v pidstat >/dev/null 2>&1; then
pidstat -durwh -p "$RESOLVED_PID" "$INTERVAL_SECS" "$count" > "${HOST_DIR}/pidstat.txt" 2>&1 &
HOST_TELEMETRY_PIDS+=("$!")
fi
if command -v iostat >/dev/null 2>&1; then
iostat -xz "$INTERVAL_SECS" "$count" > "${HOST_DIR}/iostat.txt" 2>&1 &
HOST_TELEMETRY_PIDS+=("$!")
fi
if command -v mpstat >/dev/null 2>&1; then
mpstat "$INTERVAL_SECS" "$count" > "${HOST_DIR}/mpstat.txt" 2>&1 &
HOST_TELEMETRY_PIDS+=("$!")
fi
}
wait_host_telemetry() {
local pid
for pid in "${HOST_TELEMETRY_PIDS[@]:-}"; do
wait "$pid" || true
done
}
capture_one_sample() {
local index="$1"
local ts endpoint prom_url
ts="$(date -u +%Y%m%dT%H%M%SZ)"
while IFS= read -r endpoint; do
capture_health_sample "$index" "$ts" "$endpoint"
capture_admin_metrics_sample "$index" "$ts" "$endpoint"
done < <(resolve_metrics_endpoints)
if [[ -n "$PROM_METRICS_URLS" ]]; then
while IFS= read -r prom_url; do
capture_prometheus_sample "$index" "$ts" "$prom_url"
done < <(csv_to_lines "$PROM_METRICS_URLS")
fi
if [[ "$DRY_RUN" != "true" ]]; then
capture_proc_snapshot "$index" "$ts" "$RESOLVED_PID"
fi
}
capture_series() {
local total_samples index
total_samples=$(( DURATION_SECS / INTERVAL_SECS + 1 ))
(( total_samples < 1 )) && total_samples=1
for ((index = 1; index <= total_samples; index++)); do
capture_one_sample "$index"
if (( index < total_samples )); then
sleep "$INTERVAL_SECS"
fi
done
}
count_artifacts() {
local dir="$1"
local pattern="$2"
find "$dir" -type f -name "$pattern" 2>/dev/null | wc -l | tr -d ' '
}
write_report() {
local health_count metrics_count prom_count proc_count
health_count="$(count_artifacts "$HEALTH_DIR" 'health.*.txt')"
metrics_count="$(count_artifacts "$METRICS_DIR" 'admin-metrics.*.ndjson')"
prom_count="$(count_artifacts "$PROM_DIR" 'prometheus.*.prom')"
proc_count="$(count_artifacts "$HOST_DIR" 'proc-status.*.txt')"
cat > "${CAPTURE_ROOT}/capture-report.md" <<EOF
## Large PUT Stage-Breakdown Capture Report
Run root: ${RUN_ROOT}
Label: ${LABEL}
Endpoint: ${ENDPOINT}
Duration seconds: ${DURATION_SECS}
Interval seconds: ${INTERVAL_SECS}
Resolved pid: ${RESOLVED_PID:-N/A}
## Artifact Summary
- Health snapshots: ${health_count}
- Signed admin metrics snapshots: ${metrics_count}
- Plain Prometheus snapshots: ${prom_count}
- Process snapshots: ${proc_count}
- Host telemetry present: $([[ -s "${HOST_DIR}/pidstat.txt" || -s "${HOST_DIR}/iostat.txt" || -s "${HOST_DIR}/mpstat.txt" ]] && echo yes || echo no)
## Review Checklist
- Align capture timestamps with benchmark windows in \`runs/cXX/\`.
- Check whether health or readiness degraded during the benchmark window.
- Compare admin metrics snapshots with the Prometheus / Grafana queries used for stage interpretation.
- Attach pidstat/iostat/mpstat when evaluating CPU, RSS, and disk pressure.
- Use the capture directory together with \`aggregate_median_summary.csv\` when filling the stage-breakdown report template.
EOF
}
main() {
parse_args "$@"
validate_args
require_cmd bash
require_cmd "$CURL_BIN"
require_cmd awk
if [[ "$DRY_RUN" != "true" ]]; then
require_cmd date
fi
if [[ "$DRY_RUN" != "true" && -n "$ACCESS_KEY" && -n "$SECRET_KEY" ]]; then
if command -v "$AWSCURL_BIN" >/dev/null 2>&1; then
AWSCURL_AVAILABLE=true
else
AWSCURL_AVAILABLE=false
echo "WARN: ${AWSCURL_BIN} not found; signed admin metrics snapshots will be skipped" >&2
fi
fi
setup_output
RESOLVED_PID="$(resolve_pid)"
write_layout_file
write_meta_file
start_host_telemetry
capture_series
wait_host_telemetry
write_report
echo "Capture artifacts written to ${CAPTURE_ROOT}"
}
main "$@"
+249
View File
@@ -0,0 +1,249 @@
#!/usr/bin/env bash
set -euo pipefail
ENDPOINT=""
ACCESS_KEY=""
SECRET_KEY=""
BUCKET=""
PREFIX="bench/issue713"
OBJECTS="1GiB=plain-1g.bin,2GiB=plain-2g.bin"
REGION="us-east-1"
MC_BIN="${MC_BIN:-mc}"
FORCE=false
INSECURE=false
DRY_RUN=false
usage() {
cat <<'USAGE'
Usage:
scripts/prepare_gt1g_get_test_objects.sh \
--endpoint <url> --access-key <ak> --secret-key <sk> --bucket <bucket> [options]
Required:
--endpoint <url>
--access-key <ak>
--secret-key <sk>
--bucket <bucket>
Options:
--prefix <path> Default: bench/issue713
--objects <csv> Default: 1GiB=plain-1g.bin,2GiB=plain-2g.bin
Format: size=object-name,size=object-name
--region <name> Default: us-east-1
--mc-bin <path> Default: mc
--force Re-upload even if object already exists
--insecure Allow insecure TLS
--dry-run
-h, --help
Examples:
scripts/prepare_gt1g_get_test_objects.sh \
--endpoint http://127.0.0.1:9000 \
--access-key rustfsadmin \
--secret-key rustfsadmin \
--bucket rustfs-bench \
--prefix bench/issue713/plain
USAGE
}
arg_value() {
local flag="$1"
local value="${2:-}"
if [[ -z "$value" || "$value" == --* ]]; then
echo "ERROR: missing value for $flag" >&2
exit 1
fi
printf '%s\n' "$value"
}
require_cmd() {
if ! command -v "$1" >/dev/null 2>&1; then
echo "ERROR: command not found: $1" >&2
exit 1
fi
}
run_rust_helper_fallback() {
local objects_arg=""
local helper_out_dir="$OUT_DIR"
local raw_spec
IFS=',' read -r -a object_specs <<< "$OBJECTS"
for raw_spec in "${object_specs[@]}"; do
local spec size object_name
spec="$(echo "$raw_spec" | awk '{$1=$1;print}')"
[[ -z "$spec" ]] && continue
size="${spec%%=*}"
object_name="${spec#*=}"
objects_arg+="${size}=${PREFIX%/}/${object_name},"
done
objects_arg="${objects_arg%,}"
if [[ -n "$helper_out_dir" && "$helper_out_dir" != /* ]]; then
helper_out_dir="$PWD/$helper_out_dir"
fi
echo "mc not found; falling back to rustfs/tests/gt1g_get_benchmark_tool.rs"
GT1G_GET_ACTION=prepare \
GT1G_GET_ENDPOINT="$ENDPOINT" \
GT1G_GET_ACCESS_KEY="$ACCESS_KEY" \
GT1G_GET_SECRET_KEY="$SECRET_KEY" \
GT1G_GET_BUCKET="$BUCKET" \
GT1G_GET_REGION="$REGION" \
GT1G_GET_OBJECTS="$objects_arg" \
GT1G_GET_OUT_DIR="${helper_out_dir:-$PWD/target/bench/issue713-prepare-helper}" \
GT1G_GET_FORCE="$([[ "$FORCE" == "true" ]] && echo true || echo false)" \
cargo test -p rustfs --test gt1g_get_benchmark_tool gt1g_get_benchmark_tool -- --ignored --nocapture
}
parse_args() {
while [[ $# -gt 0 ]]; do
case "$1" in
--endpoint) ENDPOINT="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--access-key) ACCESS_KEY="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--secret-key) SECRET_KEY="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--bucket) BUCKET="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--prefix) PREFIX="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--objects) OBJECTS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--region) REGION="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--mc-bin) MC_BIN="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--force) FORCE=true; shift ;;
--insecure) INSECURE=true; shift ;;
--dry-run) DRY_RUN=true; shift ;;
-h|--help) usage; exit 0 ;;
*)
echo "ERROR: unknown arg: $1" >&2
usage
exit 1
;;
esac
done
}
validate_args() {
if [[ -z "$ENDPOINT" || -z "$ACCESS_KEY" || -z "$SECRET_KEY" || -z "$BUCKET" ]]; then
echo "ERROR: --endpoint, --access-key, --secret-key, and --bucket are required" >&2
exit 1
fi
}
size_to_bytes() {
local size="$1"
case "$size" in
*GiB)
local n="${size%GiB}"
echo $((n * 1024 * 1024 * 1024))
;;
*MiB)
local n="${size%MiB}"
echo $((n * 1024 * 1024))
;;
*KiB)
local n="${size%KiB}"
echo $((n * 1024))
;;
*B)
echo "${size%B}"
;;
*)
echo "ERROR"
;;
esac
}
create_sparse_file() {
local file_path="$1"
local bytes="$2"
if command -v mkfile >/dev/null 2>&1; then
mkfile -n "$bytes" "$file_path"
else
truncate -s "$bytes" "$file_path"
fi
}
object_exists() {
local alias_path="$1"
local -a cmd=("$MC_BIN" stat "$alias_path")
if [[ "$INSECURE" == "true" ]]; then
cmd=("$MC_BIN" --insecure stat "$alias_path")
fi
"${cmd[@]}" >/dev/null 2>&1
}
main() {
parse_args "$@"
validate_args
if ! command -v "$MC_BIN" >/dev/null 2>&1; then
run_rust_helper_fallback
exit 0
fi
local tmp_root
tmp_root="$(mktemp -d "${TMPDIR:-/tmp}/issue713-gt1g-get.XXXXXX")"
trap 'rm -rf "$tmp_root"' EXIT
local mc_config_dir="$tmp_root/mc"
mkdir -p "$mc_config_dir"
local -a mc_alias_cmd=("$MC_BIN" --config-dir "$mc_config_dir")
if [[ "$INSECURE" == "true" ]]; then
mc_alias_cmd+=("--insecure")
fi
if [[ "$DRY_RUN" == "true" ]]; then
echo "[DRY-RUN] ${mc_alias_cmd[*]} alias set issue713 ${ENDPOINT} REDACTED REDACTED"
else
"${mc_alias_cmd[@]}" alias set issue713 "$ENDPOINT" "$ACCESS_KEY" "$SECRET_KEY" >/dev/null
fi
IFS=',' read -r -a object_specs <<< "$OBJECTS"
for raw_spec in "${object_specs[@]}"; do
local spec size object_name bytes local_file object_key alias_path
spec="$(echo "$raw_spec" | awk '{$1=$1;print}')"
[[ -z "$spec" ]] && continue
size="${spec%%=*}"
object_name="${spec#*=}"
if [[ -z "$size" || -z "$object_name" || "$size" == "$object_name" ]]; then
echo "ERROR: invalid object spec: $spec" >&2
exit 1
fi
bytes="$(size_to_bytes "$size")"
if [[ "$bytes" == "ERROR" ]]; then
echo "ERROR: unsupported size label: $size" >&2
exit 1
fi
object_key="${PREFIX%/}/${object_name}"
alias_path="issue713/${BUCKET}/${object_key}"
if [[ "$FORCE" != "true" && "$DRY_RUN" != "true" ]] && object_exists "$alias_path"; then
echo "skip existing: s3://${BUCKET}/${object_key}"
continue
fi
local_file="${tmp_root}/${object_name}"
create_sparse_file "$local_file" "$bytes"
if [[ "$DRY_RUN" == "true" ]]; then
echo "[DRY-RUN] create sparse file ${local_file} (${bytes} bytes)"
echo "[DRY-RUN] ${mc_alias_cmd[*]} cp ${local_file} ${alias_path}"
else
echo "uploading: s3://${BUCKET}/${object_key} (${size})"
"${mc_alias_cmd[@]}" cp "$local_file" "$alias_path" >/dev/null
fi
done
echo "prepared objects:"
for raw_spec in "${object_specs[@]}"; do
local spec object_name
spec="$(echo "$raw_spec" | awk '{$1=$1;print}')"
[[ -z "$spec" ]] && continue
object_name="${spec#*=}"
echo " s3://${BUCKET}/${PREFIX%/}/${object_name}"
done
}
main "$@"
+122
View File
@@ -0,0 +1,122 @@
#!/usr/bin/env bash
# Quick Validation Script for GET Optimization
# Usage: ./scripts/quick-validate-get-optimization.sh [TARGET_HOST]
#
# Runs quick functional tests to validate GET optimization changes:
# 1. Basic GET correctness
# 2. Concurrent GET stability
# 3. Early-stop behavior
set -euo pipefail
TARGET_HOST="${1:-localhost:9000}"
MC_ALIAS="${MC_ALIAS:-rustfs}"
TEST_BUCKET="quick-validate-$(date +%s)"
echo "=========================================="
echo "Quick GET Optimization Validation"
echo "=========================================="
echo "Target: $TARGET_HOST"
echo ""
# Create test bucket
mc mb "${MC_ALIAS}/${TEST_BUCKET}" 2>/dev/null || true
# ============================================================================
# Test 1: Basic GET Correctness
# ============================================================================
echo "[1/3] Basic GET Correctness"
# Upload test file
dd if=/dev/urandom of=/tmp/quick-test.bin bs=1048576 count=1 2>/dev/null
original_hash=$(md5 -q /tmp/quick-test.bin 2>/dev/null || md5sum /tmp/quick-test.bin | awk '{print $1}')
mc cp /tmp/quick-test.bin "${MC_ALIAS}/${TEST_BUCKET}/test.bin" >/dev/null 2>&1
# Download and verify
mc cp "${MC_ALIAS}/${TEST_BUCKET}/test.bin" /tmp/quick-test-download.bin >/dev/null 2>&1
download_hash=$(md5 -q /tmp/quick-test-download.bin 2>/dev/null || md5sum /tmp/quick-test-download.bin | awk '{print $1}')
if [ "$original_hash" = "$download_hash" ]; then
echo " PASS: Data integrity verified"
else
echo " FAIL: Data mismatch (original=$original_hash, download=$download_hash)"
fi
rm -f /tmp/quick-test.bin /tmp/quick-test-download.bin
# ============================================================================
# Test 2: Concurrent GET Stability
# ============================================================================
echo "[2/3] Concurrent GET Stability"
# Upload multiple test files
for i in $(seq 1 10); do
dd if=/dev/urandom of="/tmp/quick-test-${i}.bin" bs=1048576 count=1 2>/dev/null
mc cp "/tmp/quick-test-${i}.bin" "${MC_ALIAS}/${TEST_BUCKET}/test-${i}.bin" >/dev/null 2>&1
rm -f "/tmp/quick-test-${i}.bin"
done
# Concurrent download
passed=0
failed=0
for i in $(seq 1 10); do
if mc cp "${MC_ALIAS}/${TEST_BUCKET}/test-${i}.bin" "/tmp/quick-download-${i}.bin" >/dev/null 2>&1; then
size=$(stat -f%z "/tmp/quick-download-${i}.bin" 2>/dev/null || stat -c%s "/tmp/quick-download-${i}.bin" 2>/dev/null)
if [ "$size" = "1048576" ]; then
passed=$((passed + 1))
else
failed=$((failed + 1))
fi
else
failed=$((failed + 1))
fi
rm -f "/tmp/quick-download-${i}.bin"
done
echo " Result: $passed passed, $failed failed"
# ============================================================================
# Test 3: Early-Stop Behavior
# ============================================================================
echo "[3/3] Early-Stop Behavior"
# Upload a larger test file
dd if=/dev/urandom of=/tmp/quick-test-large.bin bs=1048576 count=10 2>/dev/null
mc cp /tmp/quick-test-large.bin "${MC_ALIAS}/${TEST_BUCKET}/test-large.bin" >/dev/null 2>&1
# Multiple reads to trigger early-stop
passed=0
failed=0
for i in $(seq 1 20); do
if mc cp "${MC_ALIAS}/${TEST_BUCKET}/test-large.bin" "/tmp/quick-large-${i}.bin" >/dev/null 2>&1; then
size=$(stat -f%z "/tmp/quick-large-${i}.bin" 2>/dev/null || stat -c%s "/tmp/quick-large-${i}.bin" 2>/dev/null)
if [ "$size" = "10485760" ]; then
passed=$((passed + 1))
else
failed=$((failed + 1))
fi
else
failed=$((failed + 1))
fi
rm -f "/tmp/quick-large-${i}.bin"
done
echo " Result: $passed passed, $failed failed"
rm -f /tmp/quick-test-large.bin
# ============================================================================
# Cleanup
# ============================================================================
echo ""
echo "Cleaning up..."
mc rm --recursive --force "${MC_ALIAS}/${TEST_BUCKET}" >/dev/null 2>&1 || true
mc rb "${MC_ALIAS}/${TEST_BUCKET}" >/dev/null 2>&1 || true
echo ""
echo "=========================================="
echo "Quick Validation Complete"
echo "=========================================="
+514
View File
@@ -0,0 +1,514 @@
#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
ENHANCED_BENCH="${PROJECT_ROOT}/scripts/run_object_batch_bench_enhanced.sh"
ADDRESS="127.0.0.1:19031"
ACCESS_KEY="rustfsadmin"
SECRET_KEY="rustfsadmin"
REGION="us-east-1"
SIZE="10MiB"
CONCURRENCY=32
DURATION="10s"
ROUNDS=3
ROUND_COOLDOWN_SECS=10
RETRY_PER_ROUND=1
SEED_DURATION="5s"
SEED_CONCURRENCY=8
HEALTH_TIMEOUT_SECS=60
OUT_DIR=""
DATA_ROOT=""
BUCKET=""
RUSTFS_BIN="${PROJECT_ROOT}/target/release/rustfs"
WARP_BIN="warp"
BASELINE_CSV=""
DRY_RUN=false
SKIP_BUILD=false
SERVER_PID=""
usage() {
cat <<'USAGE'
Usage:
scripts/run_get_metrics_gate_smoke.sh [options]
Purpose:
Start one local single-node multi-disk RustFS server with observability
export disabled, seed 10MiB objects, and run a focused warp GET benchmark.
Options:
--address <host:port> RustFS listen address (default: 127.0.0.1:19031)
--access-key <value> Access key (default: rustfsadmin)
--secret-key <value> Secret key (default: rustfsadmin)
--region <value> Region (default: us-east-1)
--bucket <name> Benchmark bucket (default: auto-generated)
--size <label> Object size label (default: 10MiB)
--concurrency <n> warp GET concurrency (default: 32)
--duration <duration> warp GET duration per round (default: 10s)
--rounds <n> Benchmark rounds (default: 3)
--round-cooldown-secs <n> Cooldown after each round (default: 10)
--retry-per-round <n> Failed-round retries (default: 1)
--seed-duration <duration> warp PUT duration for object seeding (default: 5s)
--seed-concurrency <n> warp PUT concurrency for object seeding (default: 8)
--health-timeout-secs <n> Health wait timeout (default: 60)
--out-dir <path> Output directory (default: target/bench/get-metrics-gate-<timestamp>)
--data-root <path> Data root for d1..d4 (default: /private/tmp/get-metrics-gate-<timestamp>)
--rustfs-bin <path> RustFS binary (default: target/release/rustfs)
--warp-bin <path> warp binary (default: warp)
--baseline-csv <path> Optional baseline median_summary.csv for delta output
--skip-build Skip cargo build --release -p rustfs --bin rustfs
--dry-run Print commands only
-h, --help Show help
USAGE
}
die() {
echo "ERROR: $*" >&2
exit 1
}
require_cmd() {
if ! command -v "$1" >/dev/null 2>&1; then
die "command not found: $1"
fi
}
validate_positive_int() {
local value="$1"
local name="$2"
if ! [[ "$value" =~ ^[0-9]+$ ]] || [[ "$value" -le 0 ]]; then
die "$name must be a positive integer, got: $value"
fi
}
validate_non_negative_int() {
local value="$1"
local name="$2"
if ! [[ "$value" =~ ^[0-9]+$ ]]; then
die "$name must be a non-negative integer, got: $value"
fi
}
endpoint_url() {
echo "http://${ADDRESS}"
}
normalize_warp_host() {
local raw="$1"
raw="${raw#http://}"
raw="${raw#https://}"
raw="${raw%%/*}"
raw="${raw%%\?*}"
raw="${raw%%\#*}"
echo "$raw"
}
parse_args() {
while [[ $# -gt 0 ]]; do
case "$1" in
--address) ADDRESS="$2"; shift 2 ;;
--access-key) ACCESS_KEY="$2"; shift 2 ;;
--secret-key) SECRET_KEY="$2"; shift 2 ;;
--region) REGION="$2"; shift 2 ;;
--bucket) BUCKET="$2"; shift 2 ;;
--size) SIZE="$2"; shift 2 ;;
--concurrency) CONCURRENCY="$2"; shift 2 ;;
--duration) DURATION="$2"; shift 2 ;;
--rounds) ROUNDS="$2"; shift 2 ;;
--round-cooldown-secs) ROUND_COOLDOWN_SECS="$2"; shift 2 ;;
--retry-per-round) RETRY_PER_ROUND="$2"; shift 2 ;;
--seed-duration) SEED_DURATION="$2"; shift 2 ;;
--seed-concurrency) SEED_CONCURRENCY="$2"; shift 2 ;;
--health-timeout-secs) HEALTH_TIMEOUT_SECS="$2"; shift 2 ;;
--out-dir) OUT_DIR="$2"; shift 2 ;;
--data-root) DATA_ROOT="$2"; shift 2 ;;
--rustfs-bin) RUSTFS_BIN="$2"; shift 2 ;;
--warp-bin) WARP_BIN="$2"; shift 2 ;;
--baseline-csv) BASELINE_CSV="$2"; shift 2 ;;
--skip-build) SKIP_BUILD=true; shift ;;
--dry-run) DRY_RUN=true; shift ;;
-h|--help) usage; exit 0 ;;
*)
usage >&2
die "unknown arg: $1"
;;
esac
done
}
validate_args() {
[[ -n "$ADDRESS" ]] || die "--address must not be empty"
[[ -n "$ACCESS_KEY" ]] || die "--access-key must not be empty"
[[ -n "$SECRET_KEY" ]] || die "--secret-key must not be empty"
[[ -n "$REGION" ]] || die "--region must not be empty"
[[ -n "$SIZE" ]] || die "--size must not be empty"
validate_positive_int "$CONCURRENCY" "--concurrency"
validate_positive_int "$ROUNDS" "--rounds"
validate_positive_int "$RETRY_PER_ROUND" "--retry-per-round"
validate_positive_int "$SEED_CONCURRENCY" "--seed-concurrency"
validate_positive_int "$HEALTH_TIMEOUT_SECS" "--health-timeout-secs"
validate_non_negative_int "$ROUND_COOLDOWN_SECS" "--round-cooldown-secs"
[[ -x "$ENHANCED_BENCH" ]] || die "benchmark script is not executable: $ENHANCED_BENCH"
require_cmd curl
require_cmd git
require_cmd "$WARP_BIN"
if [[ "$DRY_RUN" != "true" && "$SKIP_BUILD" != "true" ]]; then
require_cmd cargo
fi
if [[ -n "$BASELINE_CSV" && ! -f "$BASELINE_CSV" ]]; then
die "--baseline-csv does not exist: $BASELINE_CSV"
fi
}
setup_paths() {
local timestamp
timestamp="$(date +%Y%m%d-%H%M%S)"
if [[ -z "$OUT_DIR" ]]; then
OUT_DIR="${PROJECT_ROOT}/target/bench/get-metrics-gate-${timestamp}"
fi
if [[ -z "$DATA_ROOT" ]]; then
DATA_ROOT="/private/tmp/get-metrics-gate-${timestamp}"
fi
if [[ -z "$BUCKET" ]]; then
BUCKET="rustfs-get-metrics-${timestamp}"
fi
mkdir -p "$OUT_DIR" "$DATA_ROOT"/d1 "$DATA_ROOT"/d2 "$DATA_ROOT"/d3 "$DATA_ROOT"/d4
}
build_rustfs_if_needed() {
if [[ "$DRY_RUN" == "true" || "$SKIP_BUILD" == "true" ]]; then
return
fi
cargo build --release -p rustfs --bin rustfs
}
write_manifest() {
local git_head
git_head="$(git -C "$PROJECT_ROOT" rev-parse HEAD)"
cat >"${OUT_DIR}/manifest.env" <<EOF
git_head=${git_head}
endpoint=$(endpoint_url)
address=${ADDRESS}
bucket=${BUCKET}
region=${REGION}
size=${SIZE}
concurrency=${CONCURRENCY}
duration=${DURATION}
rounds=${ROUNDS}
round_cooldown_secs=${ROUND_COOLDOWN_SECS}
retry_per_round=${RETRY_PER_ROUND}
seed_duration=${SEED_DURATION}
seed_concurrency=${SEED_CONCURRENCY}
rustfs_bin=${RUSTFS_BIN}
warp_bin=${WARP_BIN}
data_root=${DATA_ROOT}
RUST_LOG=off
RUSTFS_OBS_LOGGER_LEVEL=off
RUSTFS_OBS_TRACES_EXPORT_ENABLED=false
RUSTFS_OBS_METRICS_EXPORT_ENABLED=false
RUSTFS_OBS_LOGS_EXPORT_ENABLED=false
RUSTFS_OBS_PROFILING_EXPORT_ENABLED=false
RUSTFS_OBS_USE_STDOUT=false
RUSTFS_OBS_LOG_STDOUT_ENABLED=false
RUSTFS_SCANNER_ENABLED=false
RUSTFS_CONSOLE_ENABLE=false
RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true
EOF
}
stop_server() {
if [[ -n "$SERVER_PID" ]]; then
if kill -0 "$SERVER_PID" >/dev/null 2>&1; then
kill "$SERVER_PID" >/dev/null 2>&1 || true
wait "$SERVER_PID" >/dev/null 2>&1 || true
fi
SERVER_PID=""
fi
}
wait_for_health() {
local health_url
health_url="$(endpoint_url)/health"
for ((attempt = 1; attempt <= HEALTH_TIMEOUT_SECS; attempt++)); do
if curl -fsS --noproxy '*' --connect-timeout 2 --max-time 3 "$health_url" >/dev/null 2>&1; then
return
fi
if [[ -n "$SERVER_PID" ]] && ! kill -0 "$SERVER_PID" >/dev/null 2>&1; then
tail -n 80 "${OUT_DIR}/rustfs.log" >&2 || true
die "RustFS exited before health check passed"
fi
sleep 1
done
tail -n 80 "${OUT_DIR}/rustfs.log" >&2 || true
die "RustFS health check timed out"
}
start_server() {
if [[ "$DRY_RUN" == "true" ]]; then
echo "[DRY-RUN] start RustFS at $(endpoint_url)"
return
fi
[[ -x "$RUSTFS_BIN" ]] || die "RustFS binary is not executable: $RUSTFS_BIN"
(
export RUST_LOG=off
export RUSTFS_OBS_LOGGER_LEVEL=off
export RUSTFS_OBS_TRACES_EXPORT_ENABLED=false
export RUSTFS_OBS_METRICS_EXPORT_ENABLED=false
export RUSTFS_OBS_LOGS_EXPORT_ENABLED=false
export RUSTFS_OBS_PROFILING_EXPORT_ENABLED=false
export RUSTFS_OBS_USE_STDOUT=false
export RUSTFS_OBS_LOG_STDOUT_ENABLED=false
export RUSTFS_SCANNER_ENABLED=false
export RUSTFS_CONSOLE_ENABLE=false
export RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true
export RUSTFS_ADDRESS="$ADDRESS"
export RUSTFS_ACCESS_KEY="$ACCESS_KEY"
export RUSTFS_SECRET_KEY="$SECRET_KEY"
export RUSTFS_RPC_SECRET="rustfs-get-metrics-gate-rpc-secret"
export RUSTFS_REGION="$REGION"
exec "$RUSTFS_BIN" server \
"${DATA_ROOT}/d1" \
"${DATA_ROOT}/d2" \
"${DATA_ROOT}/d3" \
"${DATA_ROOT}/d4"
) >"${OUT_DIR}/rustfs.log" 2>&1 &
SERVER_PID="$!"
wait_for_health
}
run_bench() {
local cmd=(
"$ENHANCED_BENCH"
--tool warp
--endpoint "$(endpoint_url)"
--access-key "$ACCESS_KEY"
--secret-key "$SECRET_KEY"
--bucket "$BUCKET"
--region "$REGION"
--warp-bin "$WARP_BIN"
--warp-mode get
--sizes "$SIZE"
--concurrency "$CONCURRENCY"
--duration "$DURATION"
--rounds "$ROUNDS"
--retry-per-round "$RETRY_PER_ROUND"
--round-cooldown-secs "$ROUND_COOLDOWN_SECS"
--out-dir "${OUT_DIR}/warp"
)
if [[ -n "$BASELINE_CSV" ]]; then
cmd+=(--baseline-csv "$BASELINE_CSV")
fi
if [[ "$DRY_RUN" == "true" ]]; then
cmd+=(--dry-run)
fi
"${cmd[@]}"
}
to_bps() {
local human="$1"
local number unit factor
if [[ "$human" == "N/A" || -z "$human" ]]; then
echo "N/A"
return
fi
number="$(echo "$human" | awk '{print $1}')"
unit="$(echo "$human" | awk '{print $2}')"
case "$unit" in
GiB/s) factor=1073741824 ;;
MiB/s) factor=1048576 ;;
KiB/s) factor=1024 ;;
GB/s) factor=1000000000 ;;
MB/s) factor=1000000 ;;
KB/s) factor=1000 ;;
B/s) factor=1 ;;
*)
echo "N/A"
return
;;
esac
awk -v n="$number" -v f="$factor" 'BEGIN { printf "%.6f\n", n * f }'
}
to_ms() {
local human="$1"
local number unit factor
if [[ "$human" == "N/A" || -z "$human" ]]; then
echo "N/A"
return
fi
number="$(echo "$human" | awk '{print $1}')"
unit="$(echo "$human" | awk '{print $2}')"
case "$unit" in
s) factor=1000 ;;
ms) factor=1 ;;
us|µs) factor=0.001 ;;
*)
echo "N/A"
return
;;
esac
awk -v n="$number" -v f="$factor" 'BEGIN { printf "%.6f\n", n * f }'
}
extract_final_get_metrics() {
local log_file="$1"
local avg_line reqs_line throughput_human reqps latency_human
avg_line="$(awk '/^[[:space:]]*\* Average:/ { line=$0 } END { print line }' "$log_file")"
reqs_line="$(awk '/^[[:space:]]*\* Reqs: Avg:/ { line=$0 } END { print line }' "$log_file")"
throughput_human="$(
echo "$avg_line" \
| sed -nE 's/^[[:space:]]*\* Average: ([0-9]+(\.[0-9]+)? [A-Za-z\/]+), ([0-9]+(\.[0-9]+)?) obj\/s$/\1/p'
)"
reqps="$(
echo "$avg_line" \
| sed -nE 's/^[[:space:]]*\* Average: ([0-9]+(\.[0-9]+)? [A-Za-z\/]+), ([0-9]+(\.[0-9]+)?) obj\/s$/\3/p'
)"
latency_human="$(
echo "$reqs_line" \
| sed -nE 's/^[[:space:]]*\* Reqs: Avg: ([0-9]+(\.[0-9]+)?)(ms|us|µs|s),.*$/\1 \3/p'
)"
echo "${throughput_human:-N/A},${reqps:-N/A},${latency_human:-N/A}"
}
median_from_numbers() {
local values="$1"
local count
count="$(printf '%s\n' "$values" | awk 'NF{c++} END{print c+0}')"
if [[ "$count" -eq 0 ]]; then
echo "N/A"
return
fi
printf '%s\n' "$values" | awk 'NF' | sort -n | awk '
{a[NR]=$1}
END{
n=NR
if (n==0) { print "N/A"; exit }
if (n%2==1) {
printf "%.6f\n", a[(n+1)/2]
} else {
printf "%.6f\n", (a[n/2]+a[n/2+1])/2
}
}'
}
rebuild_round_results() {
local round_csv="${OUT_DIR}/warp/round_results.csv"
local tmp_csv
tmp_csv="$(mktemp)"
while IFS=',' read -r size tool round attempt concurrency status throughput_human throughput_bps reqps latency_human latency_ms log_file; do
if [[ "$size" == "size" ]]; then
echo "$size,$tool,$round,$attempt,$concurrency,$status,$throughput_human,$throughput_bps,$reqps,$latency_human,$latency_ms,$log_file" >> "$tmp_csv"
continue
fi
if [[ "$status" == "ok" ]]; then
local metrics
metrics="$(extract_final_get_metrics "$log_file")"
throughput_human="$(echo "$metrics" | cut -d',' -f1)"
reqps="$(echo "$metrics" | cut -d',' -f2)"
latency_human="$(echo "$metrics" | cut -d',' -f3)"
throughput_bps="$(to_bps "$throughput_human")"
latency_ms="$(to_ms "$latency_human")"
fi
echo "$size,$tool,$round,$attempt,$concurrency,$status,$throughput_human,$throughput_bps,$reqps,$latency_human,$latency_ms,$log_file" >> "$tmp_csv"
done < "$round_csv"
mv "$tmp_csv" "$round_csv"
}
rebuild_median_summary() {
local round_csv="${OUT_DIR}/warp/round_results.csv"
local median_csv="${OUT_DIR}/warp/median_summary.csv"
local ok_rounds fail_rounds t_vals r_vals l_vals m_t m_r m_l
ok_rounds="$(awk -F',' 'NR>1 && $6=="ok" {c++} END{print c+0}' "$round_csv")"
fail_rounds="$(awk -F',' 'NR>1 && $6!="ok" {c++} END{print c+0}' "$round_csv")"
t_vals="$(awk -F',' 'NR>1 && $6=="ok" && $8!="N/A" {print $8}' "$round_csv")"
r_vals="$(awk -F',' 'NR>1 && $6=="ok" && $9!="N/A" {print $9}' "$round_csv")"
l_vals="$(awk -F',' 'NR>1 && $6=="ok" && $11!="N/A" {print $11}' "$round_csv")"
m_t="$(median_from_numbers "$t_vals")"
m_r="$(median_from_numbers "$r_vals")"
m_l="$(median_from_numbers "$l_vals")"
{
echo "size,tool,concurrency,successful_rounds,failed_rounds,median_throughput_bps,median_reqps,median_latency_ms"
echo "$SIZE,warp,$CONCURRENCY,$ok_rounds,$fail_rounds,$m_t,$m_r,$m_l"
} > "$median_csv"
}
rebuild_baseline_compare() {
local compare_csv="${OUT_DIR}/warp/baseline_compare.csv"
if [[ -z "$BASELINE_CSV" ]]; then
return
fi
echo "size,tool,concurrency,new_median_reqps,baseline_median_reqps,delta_reqps_pct,new_median_latency_ms,baseline_median_latency_ms,delta_latency_pct,new_median_throughput_bps,baseline_median_throughput_bps,delta_throughput_pct" > "$compare_csv"
awk -F',' '
NR==FNR {
if (FNR==1) next
b_req=$7
b_lat=$8
b_thr=$6
next
}
FNR==2 {
n_thr=$6
n_req=$7
n_lat=$8
dr="N/A"; dl="N/A"; dt="N/A"
if (b_req!="N/A" && n_req!="N/A" && b_req+0!=0) dr=sprintf("%.2f", ((n_req-b_req)/b_req)*100)
if (b_lat!="N/A" && n_lat!="N/A" && b_lat+0!=0) dl=sprintf("%.2f", ((n_lat-b_lat)/b_lat)*100)
if (b_thr!="N/A" && n_thr!="N/A" && b_thr+0!=0) dt=sprintf("%.2f", ((n_thr-b_thr)/b_thr)*100)
print $1 "," $2 "," $3 "," n_req "," b_req "," dr "," n_lat "," b_lat "," dl "," n_thr "," b_thr "," dt
}
' "$BASELINE_CSV" "${OUT_DIR}/warp/median_summary.csv" >> "$compare_csv"
}
postprocess_results() {
rebuild_round_results
rebuild_median_summary
rebuild_baseline_compare
}
main() {
parse_args "$@"
validate_args
setup_paths
write_manifest
build_rustfs_if_needed
trap stop_server EXIT INT TERM
echo "Output dir: $OUT_DIR"
start_server
run_bench
postprocess_results
stop_server
}
main "$@"
@@ -0,0 +1,222 @@
#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
HOST="${HOST:-127.0.0.1:9000}"
ACCESS_KEY="${ACCESS_KEY:-}"
SECRET_KEY="${SECRET_KEY:-}"
BUCKET_PREFIX="${BUCKET_PREFIX:-issue712-gt1g-multipart-focus}"
LOOKUP="${LOOKUP:-path}"
DURATION="${DURATION:-10m}"
WARP_BIN="${WARP_BIN:-warp}"
OUT_DIR="${OUT_DIR:-${PROJECT_ROOT}/target/bench/gt1g-multipart-server-path-$(date +%Y%m%d-%H%M%S)}"
DRY_RUN=false
DEFAULT_PROFILES="1g-64m-pc4,2g-128m-pc4"
PROFILES="${PROFILES:-$DEFAULT_PROFILES}"
usage() {
cat <<'USAGE'
Usage:
scripts/run_gt1g_multipart_put_server_path_focus.sh --access-key <ak> --secret-key <sk> [options]
Required:
--access-key <ak>
--secret-key <sk>
Options:
--host <host:port> Default: 127.0.0.1:9000
--bucket-prefix <prefix> Default: issue712-gt1g-multipart-focus
--lookup <path|dns> Default: path
--duration <dur> Default: 10m
--profiles <csv> Default: 1g-64m-pc4,2g-128m-pc4
--warp-bin <path> Default: warp
--out-dir <dir> Default: target/bench/gt1g-multipart-server-path-<timestamp>
--dry-run
-h, --help
Profiles:
1g-64m-pc4 concurrent=4 parts=16 part.size=64MiB part.concurrent=4
2g-128m-pc4 concurrent=4 parts=16 part.size=128MiB part.concurrent=4
2g-256m-pc4 concurrent=4 parts=8 part.size=256MiB part.concurrent=4
USAGE
}
require_cmd() {
if ! command -v "$1" >/dev/null 2>&1; then
echo "ERROR: command not found: $1" >&2
exit 1
fi
}
trim() {
echo "$1" | awk '{$1=$1;print}'
}
parse_args() {
while [[ $# -gt 0 ]]; do
case "$1" in
--host) HOST="$2"; shift 2 ;;
--access-key) ACCESS_KEY="$2"; shift 2 ;;
--secret-key) SECRET_KEY="$2"; shift 2 ;;
--bucket-prefix) BUCKET_PREFIX="$2"; shift 2 ;;
--lookup) LOOKUP="$2"; shift 2 ;;
--duration) DURATION="$2"; shift 2 ;;
--profiles) PROFILES="$2"; shift 2 ;;
--warp-bin) WARP_BIN="$2"; shift 2 ;;
--out-dir) OUT_DIR="$2"; shift 2 ;;
--dry-run) DRY_RUN=true; shift ;;
-h|--help) usage; exit 0 ;;
*)
echo "ERROR: unknown arg: $1" >&2
usage
exit 1
;;
esac
done
}
validate_args() {
if [[ -z "$ACCESS_KEY" || -z "$SECRET_KEY" ]]; then
echo "ERROR: --access-key and --secret-key are required" >&2
exit 1
fi
}
setup_output() {
mkdir -p "$OUT_DIR/logs" "$OUT_DIR/benchdata"
SUMMARY_CSV="$OUT_DIR/summary.csv"
COMMANDS_TXT="$OUT_DIR/commands.txt"
MANIFEST_TXT="$OUT_DIR/run_manifest.txt"
echo "profile,host,bucket,duration,concurrent,parts,part_size,part_concurrent,throughput_human,reqps,latency_human,log_file,benchdata_file,status" > "$SUMMARY_CSV"
: > "$COMMANDS_TXT"
}
write_manifest() {
{
echo "created_at=$(date +%Y-%m-%dT%H:%M:%S%z)"
echo "host=${HOST}"
echo "bucket_prefix=${BUCKET_PREFIX}"
echo "lookup=${LOOKUP}"
echo "duration=${DURATION}"
echo "profiles=${PROFILES}"
echo "warp_bin=${WARP_BIN}"
echo "dry_run=${DRY_RUN}"
echo "access_key=REDACTED"
echo "secret_key=REDACTED"
} > "$MANIFEST_TXT"
}
profile_spec() {
case "$1" in
1g-64m-pc4) echo "4|16|64MiB|4" ;;
2g-128m-pc4) echo "4|16|128MiB|4" ;;
2g-256m-pc4) echo "4|8|256MiB|4" ;;
*)
echo "ERROR: unknown profile: $1" >&2
exit 1
;;
esac
}
extract_first() {
local regex="$1"
local file="$2"
rg -o "$regex" "$file" | head -n1 || true
}
extract_metrics() {
local log_file="$1"
local throughput reqps latency
throughput="$(extract_first '[0-9]+(\.[0-9]+)?[[:space:]]*(GiB/s|MiB/s|KiB/s|GB/s|MB/s|KB/s|B/s)' "$log_file")"
reqps="$(extract_first '[0-9]+(\.[0-9]+)?[[:space:]]*(obj/s|req/s|ops/s|requests/s)' "$log_file")"
latency="$(rg -o 'Reqs:[[:space:]]+Avg:[[:space:]]+[0-9]+(\.[0-9]+)?(ms|us|µs|s)' "$log_file" | head -n1 | sed -E 's/^Reqs:[[:space:]]+Avg:[[:space:]]+//')"
throughput="$(trim "${throughput:-N/A}")"
reqps="$(trim "${reqps:-N/A}")"
latency="$(trim "${latency:-N/A}")"
reqps="$(echo "$reqps" | awk '{print $1}')"
echo "${throughput},${reqps:-N/A},${latency}"
}
run_profile() {
local profile="$1"
local spec concurrent parts part_size part_concurrent
local bucket log_file benchdata_file status metrics throughput reqps latency
spec="$(profile_spec "$profile")"
IFS='|' read -r concurrent parts part_size part_concurrent <<< "$spec"
bucket="$(echo "${BUCKET_PREFIX}-${profile}-$(date +%Y%m%d%H%M%S)" | tr '[:upper:]' '[:lower:]' | sed -E 's/[^a-z0-9-]+/-/g; s/-{2,}/-/g; s/^-+//; s/-+$//')"
bucket="${bucket:0:63}"
log_file="$OUT_DIR/logs/${profile}.log"
benchdata_file="$OUT_DIR/benchdata/${profile}.csv.zst"
local -a cmd=(
"$WARP_BIN" multipart-put
--host "$HOST"
--access-key "$ACCESS_KEY"
--secret-key "$SECRET_KEY"
--bucket "$bucket"
--lookup "$LOOKUP"
--duration "$DURATION"
--concurrent "$concurrent"
--parts "$parts"
--part.size "$part_size"
--part.concurrent "$part_concurrent"
--benchdata "$benchdata_file"
--analyze.v
--no-color
)
printf '%q ' "${cmd[@]}" >> "$COMMANDS_TXT"
printf '\n' >> "$COMMANDS_TXT"
status="ok"
if [[ "$DRY_RUN" == "true" ]]; then
printf '[DRY-RUN] %q ' "${cmd[@]}"
printf '\n'
: > "$log_file"
else
if ! "${cmd[@]}" >"$log_file" 2>&1; then
status="failed"
fi
fi
metrics="$(extract_metrics "$log_file")"
throughput="$(echo "$metrics" | cut -d',' -f1)"
reqps="$(echo "$metrics" | cut -d',' -f2)"
latency="$(echo "$metrics" | cut -d',' -f3)"
echo "${profile},${HOST},${bucket},${DURATION},${concurrent},${parts},${part_size},${part_concurrent},${throughput},${reqps},${latency},${log_file},${benchdata_file},${status}" >> "$SUMMARY_CSV"
}
main() {
parse_args "$@"
validate_args
require_cmd "$WARP_BIN"
require_cmd rg
setup_output
write_manifest
echo "Output dir: $OUT_DIR"
echo "Profiles: $PROFILES"
IFS=',' read -r -a profiles <<< "$PROFILES"
for raw in "${profiles[@]}"; do
profile="$(trim "$raw")"
[[ -z "$profile" ]] && continue
run_profile "$profile"
done
echo
echo "Summary:"
cat "$SUMMARY_CSV"
}
main "$@"
@@ -0,0 +1,141 @@
#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
RUNNER_SCRIPT="${PROJECT_ROOT}/scripts/run_put_large_stage_breakdown_with_capture.sh"
ENDPOINT="${ENDPOINT:-http://127.0.0.1:9000}"
ACCESS_KEY="${ACCESS_KEY:-}"
SECRET_KEY="${SECRET_KEY:-}"
REGION="${REGION:-us-east-1}"
SIZES="${SIZES:-64MiB,128MiB,256MiB}"
CONCURRENCIES="${CONCURRENCIES:-16}"
DURATION="${DURATION:-60s}"
ROUNDS="${ROUNDS:-1}"
COOLDOWN_SECS="${COOLDOWN_SECS:-15}"
OUT_DIR="${OUT_DIR:-target/bench/issue712-deeper-zero-copy-capture-$(date -u +%Y%m%dT%H%M%SZ)}"
CAPTURE_INTERVAL_SECS="${CAPTURE_INTERVAL_SECS:-15}"
CAPTURE_PROM_METRICS_URLS="${CAPTURE_PROM_METRICS_URLS:-http://127.0.0.1:8889/metrics}"
CAPTURE_RUSTFS_PID="${CAPTURE_RUSTFS_PID:-}"
WORKLOAD_LABEL="${WORKLOAD_LABEL:-issue-712-deeper-zero-copy}"
DRY_RUN=false
usage() {
cat <<'USAGE'
Usage:
scripts/run_issue712_deeper_zero_copy_put_with_capture.sh \
--access-key <ak> --secret-key <sk> [options]
Required:
--access-key <ak>
--secret-key <sk>
Options:
--endpoint <url> Default: http://127.0.0.1:9000
--region <name> Default: us-east-1
--sizes <csv> Default: 64MiB,128MiB,256MiB
--concurrencies <csv> Default: 16
--duration <dur> Default: 60s
--rounds <n> Default: 1
--cooldown-secs <n> Default: 15
--out-dir <dir> Default: target/bench/issue712-deeper-zero-copy-capture-<timestamp>
--capture-interval-secs <n> Default: 15
--capture-prom-metrics-urls <csv>
Default: http://127.0.0.1:8889/metrics
--capture-rustfs-pid <pid> Optional explicit rustfs pid
--dry-run
-h, --help
Notes:
- This wrapper assumes the local RustFS server is already running with:
RUSTFS_ERASURE_ENCODE_BYTESMUT_INGEST=true
- It reuses scripts/run_put_large_stage_breakdown_with_capture.sh
and only narrows the matrix to the deeper-zero-copy focus area.
USAGE
}
arg_value() {
local flag="$1"
local value="${2:-}"
if [[ -z "$value" || "$value" == --* ]]; then
echo "ERROR: missing value for $flag" >&2
exit 1
fi
printf '%s\n' "$value"
}
parse_args() {
while [[ $# -gt 0 ]]; do
case "$1" in
--endpoint) ENDPOINT="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--access-key) ACCESS_KEY="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--secret-key) SECRET_KEY="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--region) REGION="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--sizes) SIZES="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--concurrencies) CONCURRENCIES="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--duration) DURATION="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--rounds) ROUNDS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--cooldown-secs) COOLDOWN_SECS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--out-dir) OUT_DIR="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--capture-interval-secs) CAPTURE_INTERVAL_SECS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--capture-prom-metrics-urls) CAPTURE_PROM_METRICS_URLS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--capture-rustfs-pid) CAPTURE_RUSTFS_PID="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--dry-run) DRY_RUN=true; shift ;;
-h|--help) usage; exit 0 ;;
*)
echo "ERROR: unknown arg: $1" >&2
usage
exit 1
;;
esac
done
}
validate_args() {
if [[ -z "$ACCESS_KEY" || -z "$SECRET_KEY" ]]; then
echo "ERROR: --access-key and --secret-key are required" >&2
exit 1
fi
}
main() {
parse_args "$@"
validate_args
local -a cmd=(
bash "$RUNNER_SCRIPT"
--endpoint "$ENDPOINT"
--access-key "$ACCESS_KEY"
--secret-key "$SECRET_KEY"
--region "$REGION"
--sizes "$SIZES"
--concurrencies "$CONCURRENCIES"
--duration "$DURATION"
--rounds "$ROUNDS"
--retry-per-round 1
--retry-sleep-secs 2
--cooldown-secs "$COOLDOWN_SECS"
--out-dir "$OUT_DIR"
--workload-label "$WORKLOAD_LABEL"
--capture-label deeper-zero-copy-window
--capture-interval-secs "$CAPTURE_INTERVAL_SECS"
--capture-prom-metrics-urls "$CAPTURE_PROM_METRICS_URLS"
)
if [[ -n "$CAPTURE_RUSTFS_PID" ]]; then
cmd+=(--capture-rustfs-pid "$CAPTURE_RUSTFS_PID")
fi
if [[ "$DRY_RUN" == "true" ]]; then
cmd+=(--dry-run)
fi
printf 'Command:'
printf ' %q' "${cmd[@]}"
printf '\n'
"${cmd[@]}"
}
main "$@"
+582
View File
@@ -0,0 +1,582 @@
#!/usr/bin/env bash
set -euo pipefail
# Local 4-node / 16-disk RustFS runner for rustfs/backlog#797.
# It starts four local RustFS processes, runs warp workloads, and captures
# health, logs, benchmark summaries, and optional signed admin metrics.
PROJECT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
RUSTFS_BIN="${RUSTFS_BIN:-${PROJECT_ROOT}/target/debug/rustfs}"
BUILD_BIN="${BUILD_BIN:-true}"
BASE_PORT="${BASE_PORT:-19100}"
NODE_COUNT="${NODE_COUNT:-4}"
DISKS_PER_NODE="${DISKS_PER_NODE:-4}"
ACCESS_KEY="${RUSTFS_ACCESS_KEY:-rustfsadmin}"
SECRET_KEY="${RUSTFS_SECRET_KEY:-rustfsadmin}"
REGION="${REGION:-us-east-1}"
SIZES="${SIZES:-4KiB,1MiB}"
CONCURRENCY="${CONCURRENCY:-8}"
DURATION="${DURATION:-60s}"
WARP_BIN="${WARP_BIN:-warp}"
WARP_MODE="${WARP_MODE:-mixed}"
WARP_EXTRA_ARGS="${WARP_EXTRA_ARGS:---noclear}"
PROFILES="${PROFILES:-baseline,metrics_logging}"
OUT_DIR="${OUT_DIR:-${PROJECT_ROOT}/target/bench/issue-797-local-4node-16disk-ab-$(date -u +%Y%m%dT%H%M%SZ)}"
DATA_ROOT="${DATA_ROOT:-/tmp/issue797-local-4node-16disk-ab-$(date -u +%Y%m%dT%H%M%SZ)}"
KEEP_DATA="${KEEP_DATA:-false}"
WAIT_TIMEOUT_SECS="${WAIT_TIMEOUT_SECS:-180}"
HEALTH_POLL_SECS="${HEALTH_POLL_SECS:-2}"
CAPTURE_ADMIN_METRICS="${CAPTURE_ADMIN_METRICS:-true}"
AWSCURL_BIN="${AWSCURL_BIN:-awscurl}"
CURL_BIN="${CURL_BIN:-curl}"
RG_BIN="${RG_BIN:-rg}"
DRY_RUN=false
PIDS=()
usage() {
cat <<'USAGE'
Usage:
scripts/run_issue797_local_4node_16disk_ab.sh [options]
Options:
--rustfs-bin <path> RustFS binary (default: target/debug/rustfs)
--skip-build Do not build rustfs before running
--base-port <port> First node port; uses port..port+3
--sizes <csv> Object sizes for warp (default: 4KiB,1MiB)
--concurrency <n> Warp concurrency (default: 8)
--duration <dur> Warp duration per size/profile (default: 60s)
--profiles <csv> baseline,metrics_logging,locality_on
--out-dir <dir> Output directory
--data-root <dir> Temporary disk root
--keep-data Keep data root after exit
--warp-extra-args <args> Extra args appended to warp (default: --noclear)
--skip-admin-metrics Do not attempt signed admin metrics capture
--dry-run Print planned layout without running
-h, --help Show help
Profiles:
baseline Metrics exports, file logging, shard locality, and batch
processor observation disabled.
metrics_logging Enables metrics export gate, bounded warn-level file
logging, shard locality observe mode, and batch processor
observe mode.
locality_on Same as metrics_logging, but uses shard locality on mode.
Notes:
Admin metrics are captured from /rustfs/admin/v3/metrics when awscurl or
curl --aws-sigv4 is available. If neither is available, benchmark still runs
and records a skipped metrics status file.
USAGE
}
log() {
printf '[INFO] %s\n' "$*"
}
warn() {
printf '[WARN] %s\n' "$*" >&2
}
die() {
printf '[ERROR] %s\n' "$*" >&2
exit 1
}
arg_value() {
local flag="$1"
local value="${2:-}"
if [[ -z "$value" || "$value" == --* ]]; then
die "missing value for ${flag}"
fi
printf '%s\n' "$value"
}
# Like arg_value, but accepts values that themselves start with `--`
# (e.g. `--warp-extra-args --noclear`).
arg_value_allow_dashes() {
local flag="$1"
local value="${2:-}"
if [[ -z "$value" ]]; then
die "missing value for ${flag}"
fi
printf '%s\n' "$value"
}
parse_args() {
while [[ $# -gt 0 ]]; do
case "$1" in
--rustfs-bin) RUSTFS_BIN="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--skip-build) BUILD_BIN=false; shift ;;
--base-port) BASE_PORT="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--sizes) SIZES="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--concurrency) CONCURRENCY="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--duration) DURATION="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--profiles) PROFILES="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--out-dir) OUT_DIR="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--data-root) DATA_ROOT="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--keep-data) KEEP_DATA=true; shift ;;
--warp-extra-args) WARP_EXTRA_ARGS="$(arg_value_allow_dashes "$1" "${2:-}")"; shift 2 ;;
--skip-admin-metrics) CAPTURE_ADMIN_METRICS=false; shift ;;
--dry-run) DRY_RUN=true; shift ;;
-h|--help) usage; exit 0 ;;
*) die "unknown arg: $1" ;;
esac
done
}
is_positive_integer() {
[[ "$1" =~ ^[1-9][0-9]*$ ]]
}
validate_args() {
is_positive_integer "$BASE_PORT" || die "--base-port must be a positive integer"
is_positive_integer "$CONCURRENCY" || die "--concurrency must be a positive integer"
[[ "$NODE_COUNT" == "4" ]] || die "NODE_COUNT is fixed to 4 for this runner"
[[ "$DISKS_PER_NODE" == "4" ]] || die "DISKS_PER_NODE is fixed to 4 for this runner"
}
require_cmd() {
if ! command -v "$1" >/dev/null 2>&1; then
die "command not found: $1"
fi
}
print_redacted_command() {
local redact_next=false
local arg
printf 'Command:'
for arg in "$@"; do
if [[ "$redact_next" == "true" ]]; then
printf ' %q' "REDACTED"
redact_next=false
continue
fi
case "$arg" in
--access-key|--secret-key)
printf ' %q' "$arg"
redact_next=true
;;
-accessKey=*|-secretKey=*)
printf ' %q' "${arg%%=*}=REDACTED"
;;
*)
printf ' %q' "$arg"
;;
esac
done
printf '\n'
}
endpoint_for_node() {
local endpoint_node_id="$1"
printf 'http://127.0.0.1:%s' "$((BASE_PORT + endpoint_node_id - 1))"
}
endpoint_label() {
local endpoint="$1"
local label
label="${endpoint#*://}"
label="${label%%/*}"
printf '%s' "$label" | tr -c 'A-Za-z0-9._-' '_'
}
all_endpoints_csv() {
local endpoint_loop_id
local endpoints=()
for ((endpoint_loop_id = 1; endpoint_loop_id <= NODE_COUNT; endpoint_loop_id++)); do
endpoints+=("$(endpoint_for_node "$endpoint_loop_id")")
done
local IFS=','
printf '%s' "${endpoints[*]}"
}
metrics_url() {
# SCANNER(1) + NET(32) + RPC(256): enough for readiness context plus
# internode aggregate traffic without collecting the heavier all-metrics set.
printf '%s/rustfs/admin/v3/metrics?types=289&by-host=true&n=1\n' "${1%/}"
}
curl_supports_aws_sigv4() {
"$CURL_BIN" --help all 2>/dev/null | grep -q -- '--aws-sigv4'
}
profile_is_supported() {
case "$1" in
baseline|metrics_logging|locality_on) return 0 ;;
*) return 1 ;;
esac
}
profile_env_file() {
local profile="$1"
local env_file="$2"
case "$profile" in
baseline)
cat >"$env_file" <<'EOF'
RUSTFS_OBS_LOGGER_LEVEL=off
RUSTFS_OBS_TRACES_EXPORT_ENABLED=false
RUSTFS_OBS_METRICS_EXPORT_ENABLED=false
RUSTFS_OBS_LOGS_EXPORT_ENABLED=false
RUSTFS_OBS_PROFILING_EXPORT_ENABLED=false
RUSTFS_OBS_USE_STDOUT=false
RUSTFS_OBS_LOG_STDOUT_ENABLED=false
RUSTFS_BATCH_PROCESSOR_ADAPTIVE=off
RUSTFS_SHARD_LOCALITY_SCHEDULING=off
EOF
;;
metrics_logging)
cat >"$env_file" <<'EOF'
RUSTFS_OBS_LOGGER_LEVEL=warn
RUSTFS_OBS_ENDPOINT=http://127.0.0.1:4318
RUSTFS_OBS_ENDPOINT_TIMEOUT_MILLIS=500
RUSTFS_OBS_TRACES_EXPORT_ENABLED=false
RUSTFS_OBS_METRICS_EXPORT_ENABLED=true
RUSTFS_OBS_LOGS_EXPORT_ENABLED=false
RUSTFS_OBS_PROFILING_EXPORT_ENABLED=false
RUSTFS_OBS_USE_STDOUT=false
RUSTFS_OBS_LOG_STDOUT_ENABLED=false
RUSTFS_OBS_METER_INTERVAL=5
RUSTFS_OBS_LOG_KEEP_FILES=2
RUSTFS_OBS_LOG_MAX_TOTAL_SIZE_BYTES=268435456
RUSTFS_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES=134217728
RUSTFS_OBS_LOG_MIN_FILE_AGE_SECONDS=0
RUSTFS_OBS_LOG_CLEANUP_INTERVAL_SECONDS=30
RUSTFS_BATCH_PROCESSOR_ADAPTIVE=observe
RUSTFS_SHARD_LOCALITY_SCHEDULING=observe
EOF
;;
locality_on)
cat >"$env_file" <<'EOF'
RUSTFS_OBS_LOGGER_LEVEL=warn
RUSTFS_OBS_ENDPOINT=http://127.0.0.1:4318
RUSTFS_OBS_ENDPOINT_TIMEOUT_MILLIS=500
RUSTFS_OBS_TRACES_EXPORT_ENABLED=false
RUSTFS_OBS_METRICS_EXPORT_ENABLED=true
RUSTFS_OBS_LOGS_EXPORT_ENABLED=false
RUSTFS_OBS_PROFILING_EXPORT_ENABLED=false
RUSTFS_OBS_USE_STDOUT=false
RUSTFS_OBS_LOG_STDOUT_ENABLED=false
RUSTFS_OBS_METER_INTERVAL=5
RUSTFS_OBS_LOG_KEEP_FILES=2
RUSTFS_OBS_LOG_MAX_TOTAL_SIZE_BYTES=268435456
RUSTFS_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES=134217728
RUSTFS_OBS_LOG_MIN_FILE_AGE_SECONDS=0
RUSTFS_OBS_LOG_CLEANUP_INTERVAL_SECONDS=30
RUSTFS_BATCH_PROCESSOR_ADAPTIVE=observe
RUSTFS_SHARD_LOCALITY_SCHEDULING=on
EOF
;;
esac
}
load_profile_env() {
local env_file="$1"
set -a
# shellcheck disable=SC1090
source "$env_file"
set +a
}
prepare_profile_layout() {
local profile="$1"
local profile_dir="$OUT_DIR/$profile"
mkdir -p "$profile_dir"/{bench,health,logs,metrics,pids}
profile_env_file "$profile" "$profile_dir/profile.env"
}
build_volumes() {
local profile="$1"
local node_index disk_index
local volumes=()
for ((node_index = 1; node_index <= NODE_COUNT; node_index++)); do
for ((disk_index = 1; disk_index <= DISKS_PER_NODE; disk_index++)); do
volumes+=("http://127.0.0.1:$((BASE_PORT + node_index - 1))${DATA_ROOT}/${profile}/node${node_index}/disk${disk_index}")
done
done
printf '%s ' "${volumes[@]}"
}
prepare_data_dirs() {
local profile="$1"
local node_index disk_index
for ((node_index = 1; node_index <= NODE_COUNT; node_index++)); do
for ((disk_index = 1; disk_index <= DISKS_PER_NODE; disk_index++)); do
mkdir -p "${DATA_ROOT}/${profile}/node${node_index}/disk${disk_index}"
done
done
}
stop_nodes() {
local pid
if [[ ${#PIDS[@]} -eq 0 ]]; then
return
fi
for pid in "${PIDS[@]}"; do
if kill -0 "$pid" >/dev/null 2>&1; then
kill "$pid" >/dev/null 2>&1 || true
fi
done
for ((_ = 1; _ <= 5; _++)); do
local remaining=0
for pid in "${PIDS[@]}"; do
if kill -0 "$pid" >/dev/null 2>&1; then
remaining=$((remaining + 1))
fi
done
[[ "$remaining" -eq 0 ]] && break
sleep 1
done
for pid in "${PIDS[@]}"; do
if kill -0 "$pid" >/dev/null 2>&1; then
kill -9 "$pid" >/dev/null 2>&1 || true
fi
wait "$pid" >/dev/null 2>&1 || true
done
PIDS=()
}
cleanup() {
stop_nodes
if [[ "$KEEP_DATA" != "true" && "$DRY_RUN" != "true" && -n "$DATA_ROOT" && -d "$DATA_ROOT" ]]; then
rm -rf "$DATA_ROOT"
fi
}
trap cleanup EXIT
start_nodes() {
local profile="$1"
local profile_dir="$OUT_DIR/$profile"
local volumes
volumes="$(build_volumes "$profile")"
prepare_data_dirs "$profile"
load_profile_env "$profile_dir/profile.env"
local node_index endpoint log_file pid_file
for ((node_index = 1; node_index <= NODE_COUNT; node_index++)); do
endpoint="$(endpoint_for_node "$node_index")"
log_file="$profile_dir/logs/node${node_index}.log"
pid_file="$profile_dir/pids/node${node_index}.pid"
(
export RUSTFS_ACCESS_KEY="$ACCESS_KEY"
export RUSTFS_SECRET_KEY="$SECRET_KEY"
export RUSTFS_ADDRESS="127.0.0.1:$((BASE_PORT + node_index - 1))"
export RUSTFS_CONSOLE_ENABLE=false
export RUSTFS_SCANNER_ENABLED=false
export RUSTFS_SCANNER_START_DELAY_SECS=3600
export RUSTFS_SCANNER_CYCLE=3600
export RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true
export RUSTFS_VOLUMES="$volumes"
export RUSTFS_OBS_SERVICE_NAME="rustfs-issue797-${profile}-node${node_index}"
export RUSTFS_OBS_LOG_DIRECTORY="$profile_dir/logs/node${node_index}"
mkdir -p "$RUSTFS_OBS_LOG_DIRECTORY"
# Keep both env and positional volumes: current CLI still requires the
# positional server volumes, while env makes the layout visible in logs.
# shellcheck disable=SC2086
exec "$RUSTFS_BIN" server $volumes
) >"$log_file" 2>&1 &
PIDS+=("$!")
printf '%s\n' "$!" >"$pid_file"
log "started ${profile} node${node_index} ${endpoint} pid=$!"
done
}
wait_for_health() {
local profile="$1"
local profile_dir="$OUT_DIR/$profile"
local deadline node_index endpoint status_file
deadline=$((SECONDS + WAIT_TIMEOUT_SECS))
while (( SECONDS < deadline )); do
local ready=0
for ((node_index = 1; node_index <= NODE_COUNT; node_index++)); do
endpoint="$(endpoint_for_node "$node_index")"
status_file="$profile_dir/health/node${node_index}.live"
if "$CURL_BIN" -fsS --max-time 2 "${endpoint}/health/live" >"$status_file" 2>"$status_file.err"; then
ready=$((ready + 1))
fi
done
if [[ "$ready" -eq "$NODE_COUNT" ]]; then
log "${profile}: all nodes are live"
return 0
fi
sleep "$HEALTH_POLL_SECS"
done
for ((node_index = 1; node_index <= NODE_COUNT; node_index++)); do
warn "${profile} node${node_index} log tail:"
tail -n 40 "$profile_dir/logs/node${node_index}.log" >&2 || true
done
die "${profile}: timed out waiting for health"
}
capture_admin_metrics() {
local profile="$1"
local phase="$2"
local profile_dir="$OUT_DIR/$profile"
local status_file="$profile_dir/metrics/${phase}.status"
local endpoint label metrics_file endpoints
if [[ "$CAPTURE_ADMIN_METRICS" != "true" ]]; then
echo "skipped: disabled" >"$status_file"
return 0
fi
local capture_method=""
if command -v "$AWSCURL_BIN" >/dev/null 2>&1; then
capture_method="awscurl"
elif curl_supports_aws_sigv4; then
capture_method="curl-sigv4"
else
echo "skipped: awscurl not found and curl lacks --aws-sigv4" >"$status_file"
return 0
fi
echo "capturing: ${capture_method}" >"$status_file"
IFS=',' read -r -a endpoints <<<"$(all_endpoints_csv)"
for endpoint in "${endpoints[@]}"; do
label="$(endpoint_label "$endpoint")"
metrics_file="$profile_dir/metrics/${phase}.${label}.ndjson"
if [[ "$capture_method" == "awscurl" ]]; then
AWS_ACCESS_KEY_ID="$ACCESS_KEY" \
AWS_SECRET_ACCESS_KEY="$SECRET_KEY" \
AWS_DEFAULT_REGION="$REGION" \
"$AWSCURL_BIN" \
--service s3 \
--region "$REGION" \
--request GET \
"$(metrics_url "$endpoint")" \
>"$metrics_file" 2>"$metrics_file.err" || true
else
local curl_config="$profile_dir/metrics/.curl-sigv4-${phase}.${label}.conf"
{
printf 'aws-sigv4 = "aws:amz:%s:s3"\n' "$REGION"
printf 'user = "%s:%s"\n' "$ACCESS_KEY" "$SECRET_KEY"
printf 'request = "GET"\n'
printf 'max-time = 10\n'
printf 'fail\n'
printf 'silent\n'
printf 'show-error\n'
} >"$curl_config"
chmod 600 "$curl_config"
"$CURL_BIN" --config "$curl_config" "$(metrics_url "$endpoint")" >"$metrics_file" 2>"$metrics_file.err" || true
rm -f "$curl_config"
fi
done
echo "done: ${capture_method}" >"$status_file"
}
run_bench() {
local profile="$1"
local profile_dir="$OUT_DIR/$profile"
local bucket_profile="${profile//_/-}"
local -a cmd=(
"${PROJECT_ROOT}/scripts/run_object_batch_bench.sh"
--tool warp
--endpoint "$(all_endpoints_csv)"
--access-key "$ACCESS_KEY"
--secret-key "$SECRET_KEY"
--region "$REGION"
--auto-new-bucket
--bucket-prefix "issue797-${bucket_profile}"
--sizes "$SIZES"
--concurrency "$CONCURRENCY"
--duration "$DURATION"
--warp-bin "$WARP_BIN"
--warp-mode "$WARP_MODE"
--out-dir "$profile_dir/bench"
)
if [[ -n "$WARP_EXTRA_ARGS" ]]; then
cmd+=(--extra-args "$WARP_EXTRA_ARGS")
fi
print_redacted_command "${cmd[@]}" >"$profile_dir/bench/command.txt"
log "${profile}: running warp benchmark"
"${cmd[@]}" 2>&1 | tee "$profile_dir/bench/run.log"
}
extract_tail_summary() {
local profile="$1"
local profile_dir="$OUT_DIR/$profile"
local output="$profile_dir/bench/tail_latency_summary.txt"
: >"$output"
if ! command -v "$RG_BIN" >/dev/null 2>&1; then
echo "rg not found; skipped" >"$output"
return 0
fi
"$RG_BIN" -n 'Average:|Median:|90th:|99th:|Fastest:|Slowest:|StdDev:|Total:|Throughput by host|warp: <ERROR>' \
"$profile_dir/bench"/*.log >"$output" || true
}
write_run_meta() {
local meta="$OUT_DIR/run-meta.txt"
mkdir -p "$OUT_DIR"
{
echo "created_utc=$(date -u +%Y-%m-%dT%H:%M:%SZ)"
echo "rustfs_bin=$RUSTFS_BIN"
echo "base_port=$BASE_PORT"
echo "node_count=$NODE_COUNT"
echo "disks_per_node=$DISKS_PER_NODE"
echo "sizes=$SIZES"
echo "concurrency=$CONCURRENCY"
echo "duration=$DURATION"
echo "profiles=$PROFILES"
echo "warp_extra_args=$WARP_EXTRA_ARGS"
echo "endpoints=$(all_endpoints_csv)"
echo "data_root=$DATA_ROOT"
echo "keep_data=$KEEP_DATA"
echo "capture_admin_metrics=$CAPTURE_ADMIN_METRICS"
} >"$meta"
}
run_profile() {
local profile="$1"
profile_is_supported "$profile" || die "unsupported profile: $profile"
prepare_profile_layout "$profile"
start_nodes "$profile"
wait_for_health "$profile"
capture_admin_metrics "$profile" before
run_bench "$profile"
capture_admin_metrics "$profile" after
extract_tail_summary "$profile"
stop_nodes
}
main() {
parse_args "$@"
validate_args
write_run_meta
if [[ "$DRY_RUN" == "true" ]]; then
log "dry run only"
cat "$OUT_DIR/run-meta.txt"
return 0
fi
require_cmd "$CURL_BIN"
require_cmd "$WARP_BIN"
if [[ "$BUILD_BIN" == "true" ]]; then
log "building rustfs binary"
cargo build -p rustfs --bin rustfs
fi
[[ -x "$RUSTFS_BIN" ]] || die "rustfs binary is not executable: $RUSTFS_BIN"
local profile
IFS=',' read -r -a profile_arr <<<"$PROFILES"
for profile in "${profile_arr[@]}"; do
profile="${profile//[[:space:]]/}"
[[ -z "$profile" ]] && continue
run_profile "$profile"
done
log "done. Output dir: $OUT_DIR"
}
main "$@"
+257
View File
@@ -0,0 +1,257 @@
#!/usr/bin/env bash
set -euo pipefail
# Issue 2573 acceptance runner
# Runs the key workload profiles discussed in docs/tasks/issue-2573/05-benchmark-and-acceptance.md
# and samples the RustFS process RSS during load and cooldown.
WARP_BIN="${WARP_BIN:-warp}"
HOST="${HOST:-http://127.0.0.1:9000}"
ACCESS_KEY="${ACCESS_KEY:-rustfsadmin}"
SECRET_KEY="${SECRET_KEY:-rustfsadmin}"
BUCKET="${BUCKET:-rustfs-issue-2573}"
REGION="${REGION:-us-east-1}"
CONCURRENCY="${CONCURRENCY:-30}"
DURATION="${DURATION:-60s}"
COOLDOWN_SECS="${COOLDOWN_SECS:-180}"
SAMPLE_SECS="${SAMPLE_SECS:-1}"
RUSTFS_PID="${RUSTFS_PID:-}"
OUT_DIR="${OUT_DIR:-target/bench/issue-2573-acceptance-$(date +%Y%m%d-%H%M%S)}"
INSECURE="${INSECURE:-false}"
usage() {
cat <<'USAGE'
Usage:
scripts/run_issue_2573_acceptance.sh [options]
Options:
--warp-bin <path> warp binary (default: warp)
--host <url> S3 endpoint; accepts either URL or host:port (default: http://127.0.0.1:9000)
--access-key <ak> access key (default: rustfsadmin)
--secret-key <sk> secret key (default: rustfsadmin)
--bucket <name> bucket name (default: rustfs-issue-2573)
--region <name> region (default: us-east-1)
--concurrency <n> warp concurrency (default: 30)
--duration <dur> warp duration per profile (default: 60s)
--cooldown-secs <n> cooldown sampling after each profile (default: 180)
--sample-secs <n> RSS sample interval seconds (default: 1)
--pid <pid> rustfs process pid (optional; auto-detect if omitted)
--out-dir <dir> output directory
--insecure pass --insecure to warp
-h, --help show help
Profiles executed:
1. 4KiB mixed
2. 11MiB mixed
3. 11MiB delete
USAGE
}
require_cmd() {
if ! command -v "$1" >/dev/null 2>&1; then
echo "ERROR: command not found: $1" >&2
exit 1
fi
}
parse_args() {
while [[ $# -gt 0 ]]; do
case "$1" in
--warp-bin) WARP_BIN="$2"; shift 2 ;;
--host) HOST="$2"; shift 2 ;;
--access-key) ACCESS_KEY="$2"; shift 2 ;;
--secret-key) SECRET_KEY="$2"; shift 2 ;;
--bucket) BUCKET="$2"; shift 2 ;;
--region) REGION="$2"; shift 2 ;;
--concurrency) CONCURRENCY="$2"; shift 2 ;;
--duration) DURATION="$2"; shift 2 ;;
--cooldown-secs) COOLDOWN_SECS="$2"; shift 2 ;;
--sample-secs) SAMPLE_SECS="$2"; shift 2 ;;
--pid) RUSTFS_PID="$2"; shift 2 ;;
--out-dir) OUT_DIR="$2"; shift 2 ;;
--insecure) INSECURE=true; shift ;;
-h|--help) usage; exit 0 ;;
*)
echo "ERROR: unknown arg: $1" >&2
usage
exit 1
;;
esac
done
}
resolve_pid() {
if [[ -n "$RUSTFS_PID" ]]; then
echo "$RUSTFS_PID"
return
fi
local pid
pid="$(pgrep -n rustfs || true)"
if [[ -z "$pid" ]]; then
echo ""
return
fi
echo "$pid"
}
normalize_warp_host() {
local raw="$1"
# Strip scheme when a URL is provided.
raw="${raw#http://}"
raw="${raw#https://}"
# Remove any path/query/fragment to satisfy warp's --host requirements.
raw="${raw%%/*}"
raw="${raw%%\?*}"
raw="${raw%%\#*}"
echo "$raw"
}
sample_rss_loop() {
local pid="$1"
local out_file="$2"
if [[ -z "$pid" ]]; then
return 0
fi
local started_at
started_at="$(date +%s)"
echo "timestamp,elapsed_seconds,rss_kib,vsz_kib" > "$out_file"
while kill -0 "$pid" >/dev/null 2>&1; do
local now elapsed sample
now="$(date +%s)"
elapsed="$((now - started_at))"
sample="$(ps -o rss=,vsz= -p "$pid" | awk 'NF>=2 {print $1","$2}')"
if [[ -n "$sample" ]]; then
echo "$(date +%Y-%m-%dT%H:%M:%S),${elapsed},${sample}" >> "$out_file"
fi
sleep "$SAMPLE_SECS"
done
}
sample_rss_window() {
local pid="$1"
local seconds="$2"
local out_file="$3"
if [[ -z "$pid" ]]; then
return 0
fi
local started_at deadline
started_at="$(date +%s)"
deadline="$((started_at + seconds))"
echo "timestamp,elapsed_seconds,rss_kib,vsz_kib" > "$out_file"
while true; do
local now elapsed sample
now="$(date +%s)"
if (( now > deadline )); then
break
fi
elapsed="$((now - started_at))"
if ! kill -0 "$pid" >/dev/null 2>&1; then
break
fi
sample="$(ps -o rss=,vsz= -p "$pid" | awk 'NF>=2 {print $1","$2}')"
if [[ -n "$sample" ]]; then
echo "$(date +%Y-%m-%dT%H:%M:%S),${elapsed},${sample}" >> "$out_file"
fi
sleep "$SAMPLE_SECS"
done
}
run_profile() {
local profile_name="$1"
local mode="$2"
local obj_size="$3"
local pid="$4"
local warp_host
warp_host="$(normalize_warp_host "$HOST")"
local benchdata="$OUT_DIR/${profile_name// /-}"
local warp_log="$OUT_DIR/${profile_name// /-}.warp.log"
local rss_during="$OUT_DIR/${profile_name// /-}.rss_during.csv"
local rss_cooldown="$OUT_DIR/${profile_name// /-}.rss_cooldown.csv"
local -a cmd=(
"$WARP_BIN" "$mode"
"--host" "$warp_host"
"--access-key" "$ACCESS_KEY"
"--secret-key" "$SECRET_KEY"
"--bucket" "$BUCKET"
"--region" "$REGION"
"--obj.size" "$obj_size"
"--concurrent" "$CONCURRENCY"
"--duration" "$DURATION"
"--benchdata" "$benchdata"
)
if [[ "$INSECURE" == "true" ]]; then
cmd+=("--insecure")
fi
echo "==== Running profile: $profile_name ===="
printf 'Command:'
printf ' %q' "${cmd[@]}"
printf '\n'
local sampler_pid=""
if [[ -n "$pid" ]]; then
sample_rss_loop "$pid" "$rss_during" &
sampler_pid=$!
else
echo "WARN: rustfs pid unavailable; skipping RSS sampling for $profile_name" >&2
fi
if ! "${cmd[@]}" 2>&1 | tee "$warp_log"; then
echo "ERROR: profile failed: $profile_name" >&2
if [[ -n "$sampler_pid" ]]; then
kill "$sampler_pid" >/dev/null 2>&1 || true
wait "$sampler_pid" >/dev/null 2>&1 || true
fi
exit 1
fi
if [[ -n "$sampler_pid" ]]; then
kill "$sampler_pid" >/dev/null 2>&1 || true
wait "$sampler_pid" >/dev/null 2>&1 || true
fi
echo "==== Cooldown sampling: $profile_name ($COOLDOWN_SECS s) ===="
sample_rss_window "$pid" "$COOLDOWN_SECS" "$rss_cooldown"
}
main() {
parse_args "$@"
require_cmd "$WARP_BIN"
require_cmd awk
require_cmd ps
require_cmd pgrep
require_cmd tee
mkdir -p "$OUT_DIR"
local pid
pid="$(resolve_pid)"
local warp_host
warp_host="$(normalize_warp_host "$HOST")"
echo "Output dir: $OUT_DIR"
if [[ -n "$pid" ]]; then
echo "RustFS pid: $pid"
else
echo "RustFS pid: auto-detect failed (continuing without RSS sampling)"
fi
echo "Host: $HOST"
echo "Warp host: $warp_host"
echo "Bucket: $BUCKET"
echo "Profiles:"
echo " - 4KiB mixed"
echo " - 11MiB mixed"
echo " - 11MiB delete"
run_profile "4KiB mixed" "mixed" "4KiB" "$pid"
run_profile "11MiB mixed" "mixed" "11MiB" "$pid"
run_profile "11MiB delete" "delete" "11MiB" "$pid"
echo
echo "Acceptance run finished."
echo "Artifacts:"
find "$OUT_DIR" -maxdepth 1 -type f | sort
}
main "$@"
+290
View File
@@ -0,0 +1,290 @@
#!/usr/bin/env bash
set -euo pipefail
PROJECT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
LABEL="${LABEL:-issue-2941}"
DURATION_SECS="${DURATION_SECS:-60}"
PERF_FREQ="${PERF_FREQ:-99}"
OUT_DIR="${OUT_DIR:-}"
RUSTFS_PID="${RUSTFS_PID:-}"
CONTAINER_NAME="${CONTAINER_NAME:-}"
ENDPOINT="${ENDPOINT:-http://127.0.0.1:9000}"
PERF_MODE="${PERF_MODE:-auto}" # auto|on|off
SUDO_CMD="${SUDO_CMD:-}" # example: sudo
usage() {
cat <<'USAGE'
Usage:
scripts/run_issue_2941_perf_capture.sh [options]
Options:
--label <name> artifact label prefix
--duration <secs> sample duration in seconds (default: 60)
--out-dir <dir> artifact output directory
--pid <pid> rustfs pid; auto-detect if omitted
--container <name> docker container name/id for extra stats
--endpoint <url> rustfs endpoint for health probes (default: http://127.0.0.1:9000)
--perf <auto|on|off> whether to run perf record (default: auto)
--perf-freq <hz> perf sample frequency (default: 99)
--sudo-cmd <cmd> optional prefix for privileged perf, e.g. "sudo"
-h, --help show help
Environment:
LABEL
DURATION_SECS
PERF_FREQ
OUT_DIR
RUSTFS_PID
CONTAINER_NAME
ENDPOINT
PERF_MODE
SUDO_CMD
Examples:
scripts/run_issue_2941_perf_capture.sh --label musl-baseline --container rustfs
scripts/run_issue_2941_perf_capture.sh --label glibc-test --pid 12345 --perf on --sudo-cmd sudo
USAGE
}
log() {
printf '[INFO] %s\n' "$*"
}
warn() {
printf '[WARN] %s\n' "$*" >&2
}
require_arg() {
local option="$1"
local value="${2-}"
if [[ $# -lt 2 || -z "${value}" || "${value}" == --* ]]; then
warn "missing value for ${option}"
usage
exit 1
fi
}
parse_args() {
while [[ $# -gt 0 ]]; do
case "$1" in
--label) require_arg "$1" "${2-}"; LABEL="$2"; shift 2 ;;
--duration) require_arg "$1" "${2-}"; DURATION_SECS="$2"; shift 2 ;;
--out-dir) require_arg "$1" "${2-}"; OUT_DIR="$2"; shift 2 ;;
--pid) require_arg "$1" "${2-}"; RUSTFS_PID="$2"; shift 2 ;;
--container) require_arg "$1" "${2-}"; CONTAINER_NAME="$2"; shift 2 ;;
--endpoint) require_arg "$1" "${2-}"; ENDPOINT="$2"; shift 2 ;;
--perf) require_arg "$1" "${2-}"; PERF_MODE="$2"; shift 2 ;;
--perf-freq) require_arg "$1" "${2-}"; PERF_FREQ="$2"; shift 2 ;;
--sudo-cmd) require_arg "$1" "${2-}"; SUDO_CMD="$2"; shift 2 ;;
-h|--help) usage; exit 0 ;;
*)
warn "unknown argument: $1"
usage
exit 1
;;
esac
done
}
finalize_defaults() {
if [[ -z "${OUT_DIR}" ]]; then
OUT_DIR="${PROJECT_ROOT}/target/perf/${LABEL}-$(date +%Y%m%d-%H%M%S)}"
fi
}
command_exists() {
command -v "$1" >/dev/null 2>&1
}
write_cmd_output() {
local out_file="$1"
shift
if "$@" >"$out_file" 2>&1; then
return 0
fi
warn "command failed, see ${out_file}"
return 1
}
resolve_pid() {
if [[ -n "${RUSTFS_PID}" ]]; then
printf '%s\n' "${RUSTFS_PID}"
return
fi
if [[ -n "${CONTAINER_NAME}" ]] && command_exists docker; then
local pid
pid="$(docker inspect --format '{{.State.Pid}}' "${CONTAINER_NAME}" 2>/dev/null || true)"
if [[ -n "${pid}" && "${pid}" != "0" ]]; then
printf '%s\n' "${pid}"
return
fi
fi
pgrep -n rustfs || true
}
snapshot_proc() {
local pid="$1"
local prefix="$2"
[[ -n "${pid}" ]] || return 0
[[ -r "/proc/${pid}/status" ]] && cp "/proc/${pid}/status" "${OUT_DIR}/${prefix}.proc-status.txt" || true
[[ -r "/proc/${pid}/io" ]] && cp "/proc/${pid}/io" "${OUT_DIR}/${prefix}.proc-io.txt" || true
[[ -r "/proc/${pid}/sched" ]] && cp "/proc/${pid}/sched" "${OUT_DIR}/${prefix}.proc-sched.txt" || true
[[ -r "/proc/${pid}/smaps_rollup" ]] && cp "/proc/${pid}/smaps_rollup" "${OUT_DIR}/${prefix}.proc-smaps-rollup.txt" || true
[[ -r "/proc/${pid}/limits" ]] && cp "/proc/${pid}/limits" "${OUT_DIR}/${prefix}.proc-limits.txt" || true
if command_exists ps; then
ps -p "${pid}" -o pid,ppid,stat,pcpu,pmem,rss,vsz,etime,args >"${OUT_DIR}/${prefix}.ps.txt" 2>&1 || true
ps -L -p "${pid}" -o pid,tid,psr,pcpu,stat,wchan:32,comm >"${OUT_DIR}/${prefix}.threads.txt" 2>&1 || true
fi
if command_exists top; then
if [[ "$(uname -s)" == "Linux" ]]; then
top -H -b -n 1 -p "${pid}" >"${OUT_DIR}/${prefix}.top.txt" 2>&1 || true
else
top -l 1 -pid "${pid}" >"${OUT_DIR}/${prefix}.top.txt" 2>&1 || true
fi
fi
}
capture_host_info() {
write_cmd_output "${OUT_DIR}/uname.txt" uname -a || true
command_exists lscpu && write_cmd_output "${OUT_DIR}/lscpu.txt" lscpu || true
command_exists free && write_cmd_output "${OUT_DIR}/free.txt" free -h || true
command_exists df && write_cmd_output "${OUT_DIR}/df.txt" df -h || true
command_exists mount && write_cmd_output "${OUT_DIR}/mount.txt" mount || true
}
capture_endpoint_info() {
if command_exists curl; then
curl -fsS "${ENDPOINT}/health" >"${OUT_DIR}/health.txt" 2>&1 || true
curl -fsS "${ENDPOINT}/health/ready" >"${OUT_DIR}/health-ready.txt" 2>&1 || true
fi
}
capture_container_info() {
[[ -n "${CONTAINER_NAME}" ]] || return 0
command_exists docker || return 0
docker inspect "${CONTAINER_NAME}" >"${OUT_DIR}/docker-inspect.json" 2>&1 || true
docker logs --tail 500 "${CONTAINER_NAME}" >"${OUT_DIR}/docker-logs-tail.txt" 2>&1 || true
docker stats --no-stream --format '{{json .}}' "${CONTAINER_NAME}" >"${OUT_DIR}/docker-stats-once.jsonl" 2>&1 || true
}
sample_container_stats_loop() {
[[ -n "${CONTAINER_NAME}" ]] || return 0
command_exists docker || return 0
local out_file="${OUT_DIR}/docker-stats-loop.jsonl"
: >"${out_file}"
local end_ts=$((SECONDS + DURATION_SECS))
while (( SECONDS < end_ts )); do
docker stats --no-stream --format '{{json .}}' "${CONTAINER_NAME}" >>"${out_file}" 2>/dev/null || true
sleep 1
done
}
sample_pidstat() {
local pid="$1"
[[ -n "${pid}" ]] || return 0
command_exists pidstat || {
echo "pidstat unavailable" >"${OUT_DIR}/pidstat.txt"
return 0
}
pidstat -durwh -p "${pid}" 1 "${DURATION_SECS}" >"${OUT_DIR}/pidstat.txt" 2>&1 || true
}
sample_perf() {
local pid="$1"
[[ -n "${pid}" ]] || return 0
[[ "${PERF_MODE}" == "off" ]] && return 0
command_exists perf || {
echo "perf unavailable" >"${OUT_DIR}/perf-record.log"
[[ "${PERF_MODE}" == "on" ]] && warn "perf requested but not installed"
return 0
}
local perf_data="${OUT_DIR}/perf.data"
local perf_log="${OUT_DIR}/perf-record.log"
local perf_report="${OUT_DIR}/perf-report.txt"
local -a prefix=()
if [[ -n "${SUDO_CMD}" ]]; then
read -r -a prefix <<<"${SUDO_CMD}"
fi
if "${prefix[@]}" perf record -F "${PERF_FREQ}" -g -p "${pid}" -o "${perf_data}" -- sleep "${DURATION_SECS}" \
>"${perf_log}" 2>&1; then
"${prefix[@]}" perf report --stdio -i "${perf_data}" >"${perf_report}" 2>&1 || true
else
if [[ "${PERF_MODE}" == "on" ]]; then
warn "perf record failed; see ${perf_log}"
fi
fi
}
capture_version_info() {
local pid="$1"
if [[ -n "${pid}" && -x "/proc/${pid}/exe" ]]; then
readlink "/proc/${pid}/exe" >"${OUT_DIR}/binary-path.txt" 2>&1 || true
"/proc/${pid}/exe" --help >"${OUT_DIR}/binary-help.txt" 2>&1 || true
fi
}
main() {
parse_args "$@"
finalize_defaults
mkdir -p "${OUT_DIR}"
local pid
pid="$(resolve_pid)"
if [[ -z "${pid}" ]]; then
warn "failed to detect rustfs pid automatically"
else
log "using rustfs pid=${pid}"
fi
cat >"${OUT_DIR}/capture-meta.txt" <<EOF
label=${LABEL}
duration_secs=${DURATION_SECS}
perf_freq=${PERF_FREQ}
endpoint=${ENDPOINT}
container_name=${CONTAINER_NAME}
rustfs_pid=${pid}
perf_mode=${PERF_MODE}
sudo_cmd=${SUDO_CMD}
started_at=$(date -u +%Y-%m-%dT%H:%M:%SZ)
project_root=${PROJECT_ROOT}
git_branch=$(git -C "${PROJECT_ROOT}" branch --show-current 2>/dev/null || true)
git_head=$(git -C "${PROJECT_ROOT}" rev-parse HEAD 2>/dev/null || true)
EOF
capture_host_info
capture_endpoint_info
capture_container_info
capture_version_info "${pid}"
snapshot_proc "${pid}" "start"
local bg_pids=()
sample_pidstat "${pid}" &
bg_pids+=($!)
sample_container_stats_loop &
bg_pids+=($!)
sample_perf "${pid}" &
bg_pids+=($!)
for bg_pid in "${bg_pids[@]}"; do
wait "${bg_pid}" || true
done
snapshot_proc "${pid}" "end"
capture_endpoint_info
log "issue-2941 perf capture artifacts written to ${OUT_DIR}"
find "${OUT_DIR}" -maxdepth 1 -type f | sort
}
main "$@"
File diff suppressed because it is too large Load Diff
+403
View File
@@ -0,0 +1,403 @@
#!/usr/bin/env bash
set -euo pipefail
# One-click controller:
# - Switches RUSTFS_CAPACITY_* and RUSTFS_OBJECT_* by profile A/B/C
# - Calls scripts/run_object_batch_bench_enhanced.sh for each profile
# - Supports optional "apply command" hook to reload/restart RustFS per profile
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ENHANCED_SCRIPT="$SCRIPT_DIR/run_object_batch_bench_enhanced.sh"
GROUP="all" # all|A|B|C
ENDPOINT=""
ACCESS_KEY=""
SECRET_KEY=""
BUCKET="rustfs-bench"
REGION="us-east-1"
TOOL="warp"
CONCURRENCY=128
ROUNDS=3
RETRY_PER_ROUND=2
RETRY_SLEEP_SECS=2
INSECURE=false
DRY_RUN=false
OUT_ROOT=""
BASELINE_ROOT=""
# tool-specific
WARP_BIN="warp"
WARP_MODE="mixed"
DURATION="60s"
S3BENCH_BIN="s3bench"
SAMPLES=20000
# optional hooks
APPLY_CMD=""
APPLY_CMD_ARR=()
APPLY_WAIT_SECS=20
EXTRA_ARGS=()
usage() {
cat <<'USAGE'
Usage:
scripts/run_object_batch_bench_abc.sh \
--tool <warp|s3bench> --endpoint <url> --access-key <ak> --secret-key <sk> [options]
Required:
--tool warp | s3bench
--endpoint S3 endpoint
--access-key S3 access key
--secret-key S3 secret key
Core options:
--group all|A|B|C (default: all)
--bucket Bucket name (default: rustfs-bench)
--region Region (default: us-east-1)
--concurrency Default 128
--rounds Default 3
--retry-per-round Default 2
--retry-sleep-secs Default 2
--out-root Default target/bench/object-batch-abc-<timestamp>
--baseline-root If set, use <baseline-root>/<group>/median_summary.csv
--insecure Allow insecure TLS
--dry-run Print commands without execution
Warp options:
--warp-bin Default: warp
--warp-mode get|put|mixed (default: mixed)
--duration Default: 60s
s3bench options:
--s3bench-bin Default: s3bench
--samples Default: 20000
Hooks:
--apply-cmd Optional command to apply/restart RustFS after profile env switch.
Executed directly (no shell eval), e.g. "bash scripts/restart.sh"
--apply-wait-secs Wait time after apply cmd (default: 20)
Extra:
--extra-args Extra args passed to enhanced script, quoted as one string
-h, --help Show this help
Examples:
scripts/run_object_batch_bench_abc.sh \
--tool warp --endpoint http://127.0.0.1:9000 \
--access-key minioadmin --secret-key minioadmin \
--bucket bench-obj --group all --duration 90s
scripts/run_object_batch_bench_abc.sh \
--tool s3bench --endpoint http://127.0.0.1:9000 \
--access-key minioadmin --secret-key minioadmin \
--group B --samples 50000 --apply-cmd "bash scripts/run.capacity-object.lab.sh"
USAGE
}
require_cmd() {
if ! command -v "$1" >/dev/null 2>&1; then
echo "ERROR: command not found: $1" >&2
exit 1
fi
}
validate_positive_int() {
local v="$1"
local n="$2"
if ! [[ "$v" =~ ^[0-9]+$ ]] || [[ "$v" -le 0 ]]; then
echo "ERROR: $n must be a positive integer, got: $v" >&2
exit 1
fi
}
parse_apply_cmd() {
local raw="$1"
if [[ "$raw" == *';'* || "$raw" == *'&&'* || "$raw" == *'||'* || "$raw" == *'|'* || "$raw" == *'<'* || "$raw" == *'>'* || "$raw" == *'`'* || "$raw" == *'$'* ]]; then
echo "ERROR: --apply-cmd does not allow shell operators or expansions; pass a plain command and args only" >&2
exit 1
fi
IFS=$' \t\n' read -r -a APPLY_CMD_ARR <<< "$raw"
if [[ "${#APPLY_CMD_ARR[@]}" -eq 0 ]]; then
echo "ERROR: --apply-cmd must not be empty" >&2
exit 1
fi
require_cmd "${APPLY_CMD_ARR[0]}"
}
parse_args() {
while [[ $# -gt 0 ]]; do
case "$1" in
--tool) TOOL="$2"; shift 2 ;;
--endpoint) ENDPOINT="$2"; shift 2 ;;
--access-key) ACCESS_KEY="$2"; shift 2 ;;
--secret-key) SECRET_KEY="$2"; shift 2 ;;
--group) GROUP="$2"; shift 2 ;;
--bucket) BUCKET="$2"; shift 2 ;;
--region) REGION="$2"; shift 2 ;;
--concurrency) CONCURRENCY="$2"; shift 2 ;;
--rounds) ROUNDS="$2"; shift 2 ;;
--retry-per-round) RETRY_PER_ROUND="$2"; shift 2 ;;
--retry-sleep-secs) RETRY_SLEEP_SECS="$2"; shift 2 ;;
--out-root) OUT_ROOT="$2"; shift 2 ;;
--baseline-root) BASELINE_ROOT="$2"; shift 2 ;;
--insecure) INSECURE=true; shift ;;
--dry-run) DRY_RUN=true; shift ;;
--warp-bin) WARP_BIN="$2"; shift 2 ;;
--warp-mode) WARP_MODE="$2"; shift 2 ;;
--duration) DURATION="$2"; shift 2 ;;
--s3bench-bin) S3BENCH_BIN="$2"; shift 2 ;;
--samples) SAMPLES="$2"; shift 2 ;;
--apply-cmd) APPLY_CMD="$2"; shift 2 ;;
--apply-wait-secs) APPLY_WAIT_SECS="$2"; shift 2 ;;
--extra-args)
# shellcheck disable=SC2206
EXTRA_ARGS=($2)
shift 2
;;
-h|--help) usage; exit 0 ;;
*)
echo "ERROR: unknown arg: $1" >&2
usage
exit 1
;;
esac
done
}
validate_args() {
if [[ "$TOOL" != "warp" && "$TOOL" != "s3bench" ]]; then
echo "ERROR: --tool must be warp or s3bench" >&2
exit 1
fi
case "$GROUP" in
all|A|B|C) ;;
*) echo "ERROR: --group must be all|A|B|C" >&2; exit 1 ;;
esac
if [[ -z "$ENDPOINT" || -z "$ACCESS_KEY" || -z "$SECRET_KEY" ]]; then
echo "ERROR: --endpoint/--access-key/--secret-key are required" >&2
exit 1
fi
validate_positive_int "$CONCURRENCY" "--concurrency"
validate_positive_int "$ROUNDS" "--rounds"
validate_positive_int "$RETRY_PER_ROUND" "--retry-per-round"
validate_positive_int "$RETRY_SLEEP_SECS" "--retry-sleep-secs"
validate_positive_int "$APPLY_WAIT_SECS" "--apply-wait-secs"
if [[ "$TOOL" == "s3bench" ]]; then
validate_positive_int "$SAMPLES" "--samples"
fi
if [[ -n "$APPLY_CMD" ]]; then
parse_apply_cmd "$APPLY_CMD"
fi
}
setup_out_root() {
if [[ -z "$OUT_ROOT" ]]; then
OUT_ROOT="target/bench/object-batch-abc-$(date +%Y%m%d-%H%M%S)"
fi
mkdir -p "$OUT_ROOT"
}
apply_capacity_common() {
export RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
export RUSTFS_CAPACITY_WRITE_TRIGGER_DELAY=8
export RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=14
export RUSTFS_CAPACITY_FAST_UPDATE_THRESHOLD=45
export RUSTFS_CAPACITY_MAX_FILES_THRESHOLD=1000000
export RUSTFS_CAPACITY_STAT_TIMEOUT=5
export RUSTFS_CAPACITY_SAMPLE_RATE=100
export RUSTFS_CAPACITY_METRICS_INTERVAL=120
}
apply_object_profile_A() {
export RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=128
export RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=2097152
export RUSTFS_OBJECT_GET_TIMEOUT=18
export RUSTFS_OBJECT_DISK_READ_TIMEOUT=6
export RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=4
export RUSTFS_OBJECT_PRIORITY_SCHEDULING_ENABLE=true
export RUSTFS_OBJECT_LOCK_OPTIMIZATION_ENABLE=true
export RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=12
export RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=6
}
apply_object_profile_B() {
export RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=112
export RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=4194304
export RUSTFS_OBJECT_GET_TIMEOUT=30
export RUSTFS_OBJECT_DISK_READ_TIMEOUT=10
export RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=5
export RUSTFS_OBJECT_PRIORITY_SCHEDULING_ENABLE=true
export RUSTFS_OBJECT_LOCK_OPTIMIZATION_ENABLE=true
export RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=12
export RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=6
}
apply_object_profile_C() {
export RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=72
export RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
export RUSTFS_OBJECT_GET_TIMEOUT=50
export RUSTFS_OBJECT_DISK_READ_TIMEOUT=14
export RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=6
export RUSTFS_OBJECT_PRIORITY_SCHEDULING_ENABLE=true
export RUSTFS_OBJECT_LOCK_OPTIMIZATION_ENABLE=true
export RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=12
export RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=6
}
sizes_for_group() {
case "$1" in
A) echo "1KiB,4KiB,8KiB,16KiB,32KiB,100KiB" ;;
B) echo "100KiB,512KiB,1MiB,2MiB" ;;
C) echo "2MiB,5MiB,10MiB" ;;
*) echo "" ;;
esac
}
run_apply_hook_if_needed() {
local group="$1"
if [[ "${#APPLY_CMD_ARR[@]}" -eq 0 ]]; then
return
fi
echo "[${group}] running apply command..."
if [[ "$DRY_RUN" == "true" ]]; then
printf '[DRY-RUN] '
printf '%q ' "${APPLY_CMD_ARR[@]}"
printf '\n'
echo "[DRY-RUN] sleep $APPLY_WAIT_SECS"
else
"${APPLY_CMD_ARR[@]}"
echo "[${group}] waiting ${APPLY_WAIT_SECS}s for service readiness..."
sleep "$APPLY_WAIT_SECS"
fi
}
write_env_snapshot() {
local out_file="$1"
cat > "$out_file" <<EOF
RUSTFS_CAPACITY_SCHEDULED_INTERVAL=${RUSTFS_CAPACITY_SCHEDULED_INTERVAL}
RUSTFS_CAPACITY_WRITE_TRIGGER_DELAY=${RUSTFS_CAPACITY_WRITE_TRIGGER_DELAY}
RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=${RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD}
RUSTFS_CAPACITY_FAST_UPDATE_THRESHOLD=${RUSTFS_CAPACITY_FAST_UPDATE_THRESHOLD}
RUSTFS_CAPACITY_MAX_FILES_THRESHOLD=${RUSTFS_CAPACITY_MAX_FILES_THRESHOLD}
RUSTFS_CAPACITY_STAT_TIMEOUT=${RUSTFS_CAPACITY_STAT_TIMEOUT}
RUSTFS_CAPACITY_SAMPLE_RATE=${RUSTFS_CAPACITY_SAMPLE_RATE}
RUSTFS_CAPACITY_METRICS_INTERVAL=${RUSTFS_CAPACITY_METRICS_INTERVAL}
RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=${RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS}
RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=${RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE}
RUSTFS_OBJECT_GET_TIMEOUT=${RUSTFS_OBJECT_GET_TIMEOUT}
RUSTFS_OBJECT_DISK_READ_TIMEOUT=${RUSTFS_OBJECT_DISK_READ_TIMEOUT}
RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT}
RUSTFS_OBJECT_PRIORITY_SCHEDULING_ENABLE=${RUSTFS_OBJECT_PRIORITY_SCHEDULING_ENABLE}
RUSTFS_OBJECT_LOCK_OPTIMIZATION_ENABLE=${RUSTFS_OBJECT_LOCK_OPTIMIZATION_ENABLE}
RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD}
RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD}
EOF
}
run_group() {
local g="$1"
local sizes out_dir baseline_csv
apply_capacity_common
case "$g" in
A) apply_object_profile_A ;;
B) apply_object_profile_B ;;
C) apply_object_profile_C ;;
*) echo "ERROR: unsupported group $g" >&2; exit 1 ;;
esac
sizes="$(sizes_for_group "$g")"
out_dir="$OUT_ROOT/$g"
mkdir -p "$out_dir"
write_env_snapshot "$out_dir/env_snapshot.env"
run_apply_hook_if_needed "$g"
baseline_csv=""
if [[ -n "$BASELINE_ROOT" && -f "$BASELINE_ROOT/$g/median_summary.csv" ]]; then
baseline_csv="$BASELINE_ROOT/$g/median_summary.csv"
fi
local cmd=(
"$ENHANCED_SCRIPT"
"--tool" "$TOOL"
"--endpoint" "$ENDPOINT"
"--access-key" "$ACCESS_KEY"
"--secret-key" "$SECRET_KEY"
"--bucket" "$BUCKET"
"--region" "$REGION"
"--concurrency" "$CONCURRENCY"
"--sizes" "$sizes"
"--rounds" "$ROUNDS"
"--retry-per-round" "$RETRY_PER_ROUND"
"--retry-sleep-secs" "$RETRY_SLEEP_SECS"
"--out-dir" "$out_dir"
)
if [[ -n "$baseline_csv" ]]; then
cmd+=("--baseline-csv" "$baseline_csv")
fi
if [[ "$INSECURE" == "true" ]]; then
cmd+=("--insecure")
fi
if [[ "$DRY_RUN" == "true" ]]; then
cmd+=("--dry-run")
fi
if [[ "$TOOL" == "warp" ]]; then
cmd+=("--warp-bin" "$WARP_BIN" "--warp-mode" "$WARP_MODE" "--duration" "$DURATION")
else
cmd+=("--s3bench-bin" "$S3BENCH_BIN" "--samples" "$SAMPLES")
fi
if [[ "${#EXTRA_ARGS[@]}" -gt 0 ]]; then
local joined
joined="$(printf '%s ' "${EXTRA_ARGS[@]}" | sed 's/[[:space:]]*$//')"
cmd+=("--extra-args" "$joined")
fi
echo
echo "===== Running group ${g} ====="
echo "Sizes: $sizes"
echo "Output: $out_dir"
if [[ "$DRY_RUN" == "true" ]]; then
printf '[DRY-RUN] %q ' "${cmd[@]}"
printf '\n'
else
"${cmd[@]}"
fi
}
main() {
parse_args "$@"
validate_args
require_cmd awk
require_cmd sed
if [[ ! -x "$ENHANCED_SCRIPT" ]]; then
echo "ERROR: enhanced script missing or not executable: $ENHANCED_SCRIPT" >&2
exit 1
fi
setup_out_root
echo "Controller output root: $OUT_ROOT"
echo "Tool=$TOOL Group=$GROUP Concurrency=$CONCURRENCY Rounds=$ROUNDS"
case "$GROUP" in
all)
run_group A
run_group B
run_group C
;;
A|B|C)
run_group "$GROUP"
;;
esac
echo
echo "Done. Group outputs are under: $OUT_ROOT"
}
main "$@"
File diff suppressed because it is too large Load Diff
+601
View File
@@ -0,0 +1,601 @@
#!/usr/bin/env bash
set -euo pipefail
# Large-object PUT stage-breakdown runner for rustfs/backlog#706.
# This wrapper standardizes:
# - object sizes and concurrency matrix for 16MiB / 32MiB PUT runs
# - output directory layout under target/bench/
# - artifact naming and run manifest capture
# - optional baseline comparison reuse across per-concurrency runs
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
ENHANCED_BENCH_SCRIPT="${PROJECT_ROOT}/scripts/run_object_batch_bench_enhanced.sh"
SERVICE_METRICS_DELTA_SCRIPT="${PROJECT_ROOT}/scripts/analyze_put_service_metrics_deltas.py"
TOOL="warp"
ENDPOINT=""
ACCESS_KEY=""
SECRET_KEY=""
REGION="us-east-1"
BUCKET_PREFIX="rustfs-put-large"
CONCURRENCIES="16,32,64,96,128"
SIZES="16MiB,32MiB"
DURATION="120s"
ROUNDS=3
RETRY_PER_ROUND=1
RETRY_SLEEP_SECS=2
COOLDOWN_SECS=20
WARP_BIN="${WARP_BIN:-warp}"
OUT_DIR=""
BASELINE_ROOT=""
EXTRA_ARGS=""
INSECURE=false
DRY_RUN=false
PYTHON_BIN="python3"
SERVICE_METRICS_URL=""
SERVICE_PROMETHEUS_QUERY_URL=""
SERVICE_PROMETHEUS_QUERY=""
SERVICE_METRICS_SERVICE_NAME=""
SERVICE_METRICS_FILTER_REGEX=""
SERVICE_METRICS_CAPTURE_ATTEMPTS=3
SERVICE_METRICS_CAPTURE_RETRY_SECS=1
SERVICE_METRICS_CONNECT_TIMEOUT_SECS=2
SERVICE_METRICS_MAX_TIME_SECS=15
SERVICE_METRICS_SETTLE_SECS=0
TOPOLOGY_NODES=""
TOPOLOGY_DISKS_PER_NODE=""
TOPOLOGY_TOTAL_DISKS=""
TOPOLOGY_CPU_PER_NODE=""
TOPOLOGY_MEM_PER_NODE=""
TOPOLOGY_NETWORK=""
TOPOLOGY_ENDPOINT_MODE=""
TOPOLOGY_ERASURE_SET_DRIVE_COUNT=""
CLIENT_HOST=""
WORKLOAD_LABEL="backlog-706-large-put-stage-breakdown"
usage() {
cat <<'USAGE'
Usage:
scripts/run_put_large_stage_breakdown.sh --endpoint <url> \
--access-key <ak> --secret-key <sk> [options]
Required:
--endpoint <url> S3 endpoint
--access-key <ak> Access key
--secret-key <sk> Secret key
Core options:
--bucket-prefix <prefix> Bucket prefix (default: rustfs-put-large)
--region <name> Region (default: us-east-1)
--sizes <csv> Object sizes (default: 16MiB,32MiB)
--concurrencies <csv> Concurrency matrix (default: 16,32,64,96,128)
--duration <dur> Per-run duration (default: 120s)
--rounds <n> Rounds per size (default: 3)
--retry-per-round <n> Retries per failed round (default: 1)
--retry-sleep-secs <n> Sleep between retries (default: 2)
--cooldown-secs <n> Sleep between rounds/sizes/concurrency runs (default: 20)
--round-cooldown-secs <n> Compatibility alias for --cooldown-secs
--out-dir <dir> Output root (default: target/bench/put-large-stage-breakdown-<timestamp>)
--baseline-root <dir> Existing root from a previous run of this script
--extra-args "<args>" Extra args passed to run_object_batch_bench_enhanced.sh
--warp-bin <path> warp binary (default: warp)
--insecure Pass --insecure to warp
--dry-run Print commands only
--python-bin <path> Python binary for Prometheus query capture (default: python3)
--service-metrics-url <url> Plain Prometheus text scrape URL
--service-prometheus-query-url <url>
Prometheus HTTP API /api/v1/query URL for OTLP-exported metrics
--service-prometheus-query <promql> PromQL for --service-prometheus-query-url
--service-metrics-service-name <name>
Optional service.name/service_name filter for query results
--service-metrics-filter-regex <regex>
Regex for retained plain text metrics lines
--service-metrics-attempts <n> Direct scrape attempts per snapshot (default: 3)
--service-metrics-retry-secs <n> Sleep between direct scrape attempts (default: 1)
--service-metrics-connect-timeout-secs <n>
Curl connect timeout for direct scrape (default: 2)
--service-metrics-max-time-secs <n> Curl max time for direct scrape (default: 15)
--service-metrics-settle-secs <n> Sleep before after-snapshot capture (default: 0)
Topology metadata (optional but recommended):
--nodes <n>
--disks-per-node <n>
--total-disks <n>
--cpu-per-node <text>
--mem-per-node <text>
--network <text>
--endpoint-mode <direct|lb>
--erasure-set-drive-count <n>
--client-host <text>
--workload-label <text>
Output layout:
<out-dir>/
run_manifest.txt
run_matrix.csv
aggregate_median_summary.csv
aggregate_baseline_compare.csv
artifact_layout.txt
runs/
c16/
c32/
...
Each runs/cXX directory is a direct output directory of:
scripts/run_object_batch_bench_enhanced.sh
USAGE
}
require_cmd() {
if ! command -v "$1" >/dev/null 2>&1; then
echo "ERROR: command not found: $1" >&2
exit 1
fi
}
arg_value() {
local flag="$1"
local value="${2:-}"
if [[ -z "$value" || "$value" == --* ]]; then
echo "ERROR: missing value for $flag" >&2
exit 1
fi
printf '%s\n' "$value"
}
arg_value_allow_option() {
local flag="$1"
local value="${2:-}"
if [[ -z "$value" ]]; then
echo "ERROR: missing value for $flag" >&2
exit 1
fi
printf '%s\n' "$value"
}
parse_args() {
while [[ $# -gt 0 ]]; do
case "$1" in
--endpoint) ENDPOINT="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--access-key) ACCESS_KEY="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--secret-key) SECRET_KEY="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--bucket-prefix) BUCKET_PREFIX="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--region) REGION="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--sizes) SIZES="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--concurrencies) CONCURRENCIES="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--duration) DURATION="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--rounds) ROUNDS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--retry-per-round) RETRY_PER_ROUND="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--retry-sleep-secs) RETRY_SLEEP_SECS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--cooldown-secs|--round-cooldown-secs) COOLDOWN_SECS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--out-dir) OUT_DIR="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--baseline-root) BASELINE_ROOT="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--extra-args) EXTRA_ARGS="$(arg_value_allow_option "$1" "${2:-}")"; shift 2 ;;
--warp-bin) WARP_BIN="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--python-bin) PYTHON_BIN="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--service-metrics-url) SERVICE_METRICS_URL="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--service-prometheus-query-url) SERVICE_PROMETHEUS_QUERY_URL="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--service-prometheus-query) SERVICE_PROMETHEUS_QUERY="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--service-metrics-service-name) SERVICE_METRICS_SERVICE_NAME="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--service-metrics-filter-regex) SERVICE_METRICS_FILTER_REGEX="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--service-metrics-attempts) SERVICE_METRICS_CAPTURE_ATTEMPTS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--service-metrics-retry-secs) SERVICE_METRICS_CAPTURE_RETRY_SECS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--service-metrics-connect-timeout-secs) SERVICE_METRICS_CONNECT_TIMEOUT_SECS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--service-metrics-max-time-secs) SERVICE_METRICS_MAX_TIME_SECS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--service-metrics-settle-secs) SERVICE_METRICS_SETTLE_SECS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--nodes) TOPOLOGY_NODES="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--disks-per-node) TOPOLOGY_DISKS_PER_NODE="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--total-disks) TOPOLOGY_TOTAL_DISKS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--cpu-per-node) TOPOLOGY_CPU_PER_NODE="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--mem-per-node) TOPOLOGY_MEM_PER_NODE="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--network) TOPOLOGY_NETWORK="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--endpoint-mode) TOPOLOGY_ENDPOINT_MODE="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--erasure-set-drive-count) TOPOLOGY_ERASURE_SET_DRIVE_COUNT="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--client-host) CLIENT_HOST="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--workload-label) WORKLOAD_LABEL="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--insecure) INSECURE=true; shift ;;
--dry-run) DRY_RUN=true; shift ;;
-h|--help) usage; exit 0 ;;
*)
echo "ERROR: unknown arg: $1" >&2
usage
exit 1
;;
esac
done
}
is_positive_int() {
[[ "$1" =~ ^[1-9][0-9]*$ ]]
}
is_nonnegative_int() {
[[ "$1" =~ ^[0-9]+$ ]]
}
cooldown_sleep() {
local secs="$1"
if (( secs <= 0 )); then
return 0
fi
if [[ "$DRY_RUN" == "true" ]]; then
echo "[DRY-RUN] sleep ${secs}"
else
sleep "$secs"
fi
}
validate_args() {
if [[ -z "$ENDPOINT" || -z "$ACCESS_KEY" || -z "$SECRET_KEY" ]]; then
echo "ERROR: --endpoint, --access-key, and --secret-key are required" >&2
exit 1
fi
if [[ "$TOOL" != "warp" ]]; then
echo "ERROR: only warp is supported by this wrapper" >&2
exit 1
fi
if ! is_positive_int "$ROUNDS" || ! is_positive_int "$RETRY_PER_ROUND" || ! is_positive_int "$RETRY_SLEEP_SECS" || ! is_nonnegative_int "$COOLDOWN_SECS"; then
echo "ERROR: --rounds, --retry-per-round, and --retry-sleep-secs must be positive integers; --cooldown-secs must be a nonnegative integer" >&2
exit 1
fi
if [[ -n "$BASELINE_ROOT" && ! -d "$BASELINE_ROOT" ]]; then
echo "ERROR: --baseline-root does not exist: $BASELINE_ROOT" >&2
exit 1
fi
if [[ -n "$SERVICE_METRICS_URL" && -n "$SERVICE_PROMETHEUS_QUERY_URL" ]]; then
echo "ERROR: --service-metrics-url and --service-prometheus-query-url are mutually exclusive" >&2
exit 1
fi
if ! is_positive_int "$SERVICE_METRICS_CAPTURE_ATTEMPTS" || ! is_nonnegative_int "$SERVICE_METRICS_CAPTURE_RETRY_SECS" || ! is_positive_int "$SERVICE_METRICS_CONNECT_TIMEOUT_SECS" || ! is_positive_int "$SERVICE_METRICS_MAX_TIME_SECS" || ! is_nonnegative_int "$SERVICE_METRICS_SETTLE_SECS"; then
echo "ERROR: service metrics attempt/timeout/settle options must be valid integers" >&2
exit 1
fi
}
setup_output() {
if [[ -z "$OUT_DIR" ]]; then
OUT_DIR="target/bench/put-large-stage-breakdown-$(date -u +%Y%m%dT%H%M%SZ)"
fi
RUNS_DIR="${OUT_DIR}/runs"
AGG_MEDIAN_CSV="${OUT_DIR}/aggregate_median_summary.csv"
AGG_COMPARE_CSV="${OUT_DIR}/aggregate_baseline_compare.csv"
AGG_SERVICE_METRICS_CSV="${OUT_DIR}/aggregate_service_metrics_captures.csv"
AGG_SERVICE_METRICS_DELTAS_CSV="${OUT_DIR}/aggregate_service_metrics_deltas.csv"
AGG_SERVICE_METRICS_PATH_SUMMARY_CSV="${OUT_DIR}/aggregate_service_metrics_path_summary.csv"
AGG_SERVICE_METRICS_STAGE_SUMMARY_CSV="${OUT_DIR}/aggregate_service_metrics_stage_summary.csv"
RUN_MATRIX_CSV="${OUT_DIR}/run_matrix.csv"
mkdir -p "$RUNS_DIR"
echo "concurrency,size,tool,successful_rounds,failed_rounds,median_throughput_bps,median_reqps,median_latency_ms,bucket,run_dir" > "$AGG_MEDIAN_CSV"
echo "concurrency,size,tool,new_median_reqps,baseline_median_reqps,delta_reqps_pct,new_median_latency_ms,baseline_median_latency_ms,delta_latency_pct,new_median_throughput_bps,baseline_median_throughput_bps,delta_throughput_pct,run_dir" > "$AGG_COMPARE_CSV"
echo "concurrency,run_dir,size,tool,round,attempt,phase,source,status,capture_attempt,raw_bytes,snapshot_bytes,status_file,snapshot_file,filter_regex,prometheus_query" > "$AGG_SERVICE_METRICS_CSV"
echo "concurrency,bucket,run_dir,baseline_csv,status" > "$RUN_MATRIX_CSV"
}
join_bool() {
if [[ "$1" == "true" ]]; then
echo "true"
else
echo "false"
fi
}
write_manifest() {
local git_commit git_branch git_dirty rustc_version
git_commit="$(git -C "$PROJECT_ROOT" rev-parse HEAD 2>/dev/null || echo "unknown")"
git_branch="$(git -C "$PROJECT_ROOT" branch --show-current 2>/dev/null || echo "unknown")"
if [[ -n "$(git -C "$PROJECT_ROOT" status --porcelain 2>/dev/null || true)" ]]; then
git_dirty="true"
else
git_dirty="false"
fi
rustc_version="$(rustc --version 2>/dev/null || echo "unknown")"
{
echo "created_utc=$(date -u +%Y-%m-%dT%H:%M:%SZ)"
echo "git_commit=${git_commit}"
echo "git_branch=${git_branch}"
echo "git_dirty=${git_dirty}"
echo "rustc_version=${rustc_version}"
echo "kernel=$(uname -srvmo 2>/dev/null || echo "unknown")"
echo "tool=${TOOL}"
echo "endpoint=${ENDPOINT}"
echo "region=${REGION}"
echo "bucket_prefix=${BUCKET_PREFIX}"
echo "sizes=${SIZES}"
echo "concurrencies=${CONCURRENCIES}"
echo "duration=${DURATION}"
echo "rounds=${ROUNDS}"
echo "retry_per_round=${RETRY_PER_ROUND}"
echo "retry_sleep_secs=${RETRY_SLEEP_SECS}"
echo "cooldown_secs=${COOLDOWN_SECS}"
echo "insecure=$(join_bool "$INSECURE")"
echo "dry_run=$(join_bool "$DRY_RUN")"
echo "baseline_root=${BASELINE_ROOT:-N/A}"
echo "extra_args_present=$([[ -n "$EXTRA_ARGS" ]] && echo true || echo false)"
echo "python_bin=${PYTHON_BIN}"
echo "service_metrics_url=${SERVICE_METRICS_URL:-N/A}"
echo "service_prometheus_query_url=${SERVICE_PROMETHEUS_QUERY_URL:-N/A}"
echo "service_prometheus_query=${SERVICE_PROMETHEUS_QUERY:-N/A}"
echo "service_metrics_service_name=${SERVICE_METRICS_SERVICE_NAME:-N/A}"
echo "service_metrics_filter_regex=${SERVICE_METRICS_FILTER_REGEX:-N/A}"
echo "service_metrics_capture_attempts=${SERVICE_METRICS_CAPTURE_ATTEMPTS}"
echo "service_metrics_retry_secs=${SERVICE_METRICS_CAPTURE_RETRY_SECS}"
echo "service_metrics_connect_timeout_secs=${SERVICE_METRICS_CONNECT_TIMEOUT_SECS}"
echo "service_metrics_max_time_secs=${SERVICE_METRICS_MAX_TIME_SECS}"
echo "service_metrics_settle_secs=${SERVICE_METRICS_SETTLE_SECS}"
echo "workload_label=${WORKLOAD_LABEL}"
echo "nodes=${TOPOLOGY_NODES:-N/A}"
echo "disks_per_node=${TOPOLOGY_DISKS_PER_NODE:-N/A}"
echo "total_disks=${TOPOLOGY_TOTAL_DISKS:-N/A}"
echo "cpu_per_node=${TOPOLOGY_CPU_PER_NODE:-N/A}"
echo "mem_per_node=${TOPOLOGY_MEM_PER_NODE:-N/A}"
echo "network=${TOPOLOGY_NETWORK:-N/A}"
echo "endpoint_mode=${TOPOLOGY_ENDPOINT_MODE:-N/A}"
echo "erasure_set_drive_count=${TOPOLOGY_ERASURE_SET_DRIVE_COUNT:-N/A}"
echo "client_host=${CLIENT_HOST:-N/A}"
echo "access_key=REDACTED"
echo "secret_key=REDACTED"
} > "${OUT_DIR}/run_manifest.txt"
}
write_artifact_layout() {
cat > "${OUT_DIR}/artifact_layout.txt" <<'EOF'
Top-level artifacts:
- run_manifest.txt: run metadata, git revision, topology notes, and redacted execution context
- run_matrix.csv: one row per concurrency run, including bucket and baseline linkage
- aggregate_median_summary.csv: merged median_summary rows from every concurrency run
- aggregate_baseline_compare.csv: merged baseline_compare rows when a matching baseline exists
- aggregate_service_metrics_captures.csv: merged service metrics capture status from every concurrency run
- aggregate_service_metrics_deltas.csv: per-round before/after service metric deltas
- aggregate_service_metrics_path_summary.csv: per-round PUT path deltas
- aggregate_service_metrics_stage_summary.csv: per-round PUT stage duration averages from deltas
- runs/cXX/: direct output directory of scripts/run_object_batch_bench_enhanced.sh
Per-concurrency directory:
- round_results.csv
- median_summary.csv
- service_metrics_captures.csv (when service metrics capture is enabled)
- baseline_compare.csv (only when a baseline CSV is supplied)
- logs/
EOF
}
trim() {
echo "$1" | awk '{$1=$1;print}'
}
csv_to_lines() {
local csv="$1"
IFS=',' read -r -a arr <<< "$csv"
for raw in "${arr[@]}"; do
local item
item="$(trim "$raw")"
[[ -z "$item" ]] && continue
echo "$item"
done
}
sanitize_bucket_component() {
echo "$1" | tr '[:upper:]' '[:lower:]' | sed -E 's/[^a-z0-9-]+/-/g; s/^-+//; s/-+$//; s/-{2,}/-/g'
}
build_bucket_name() {
local concurrency="$1"
local run_id prefix raw
run_id="$(date -u +%Y%m%d%H%M%S)"
prefix="$(sanitize_bucket_component "$BUCKET_PREFIX")"
raw="${prefix}-${run_id}-c${concurrency}"
raw="$(sanitize_bucket_component "$raw")"
echo "${raw:0:63}"
}
resolve_baseline_csv() {
local concurrency="$1"
local candidate=""
if [[ -z "$BASELINE_ROOT" ]]; then
return 0
fi
if [[ -f "${BASELINE_ROOT}/runs/c${concurrency}/median_summary.csv" ]]; then
candidate="${BASELINE_ROOT}/runs/c${concurrency}/median_summary.csv"
elif [[ -f "${BASELINE_ROOT}/c${concurrency}/median_summary.csv" ]]; then
candidate="${BASELINE_ROOT}/c${concurrency}/median_summary.csv"
fi
if [[ -n "$candidate" ]]; then
printf '%s\n' "$candidate"
fi
}
append_aggregate_rows() {
local concurrency="$1"
local bucket="$2"
local run_dir="$3"
local median_csv="$4"
local compare_csv="$5"
local service_metrics_csv="${run_dir}/service_metrics_captures.csv"
awk -F',' -v c="$concurrency" -v b="$bucket" -v rd="$run_dir" 'NR>1 {print c "," $1 "," $2 "," $4 "," $5 "," $6 "," $7 "," $8 "," b "," rd}' "$median_csv" >> "$AGG_MEDIAN_CSV"
if [[ -f "$compare_csv" ]]; then
awk -F',' -v c="$concurrency" -v rd="$run_dir" 'NR>1 {print c "," $1 "," $2 "," $4 "," $5 "," $6 "," $7 "," $8 "," $9 "," $10 "," $11 "," $12 "," rd}' "$compare_csv" >> "$AGG_COMPARE_CSV"
fi
if [[ -f "$service_metrics_csv" ]]; then
awk -F',' -v c="$concurrency" -v rd="$run_dir" 'NR>1 {print c "," rd "," $0}' "$service_metrics_csv" >> "$AGG_SERVICE_METRICS_CSV"
fi
}
analyze_service_metric_deltas() {
if [[ ! -s "$AGG_SERVICE_METRICS_CSV" ]]; then
return 0
fi
if [[ "$DRY_RUN" == "true" ]]; then
return 0
fi
if [[ ! -f "$SERVICE_METRICS_DELTA_SCRIPT" ]]; then
echo "ERROR: missing service metrics delta script: $SERVICE_METRICS_DELTA_SCRIPT" >&2
exit 1
fi
"$PYTHON_BIN" "$SERVICE_METRICS_DELTA_SCRIPT" \
--capture-csv "$AGG_SERVICE_METRICS_CSV" \
--delta-csv "$AGG_SERVICE_METRICS_DELTAS_CSV" \
--path-summary-csv "$AGG_SERVICE_METRICS_PATH_SUMMARY_CSV" \
--stage-summary-csv "$AGG_SERVICE_METRICS_STAGE_SUMMARY_CSV"
}
run_concurrency() {
local concurrency="$1"
local run_dir bucket baseline_csv status
run_dir="${RUNS_DIR}/c${concurrency}"
bucket="$(build_bucket_name "$concurrency")"
baseline_csv="$(resolve_baseline_csv "$concurrency" || true)"
status="pending"
local -a cmd=(
bash "$ENHANCED_BENCH_SCRIPT"
--tool "$TOOL"
--endpoint "$ENDPOINT"
--access-key "$ACCESS_KEY"
--secret-key "$SECRET_KEY"
--bucket "$bucket"
--region "$REGION"
--warp-bin "$WARP_BIN"
--warp-mode put
--sizes "$SIZES"
--concurrency "$concurrency"
--duration "$DURATION"
--rounds "$ROUNDS"
--retry-per-round "$RETRY_PER_ROUND"
--retry-sleep-secs "$RETRY_SLEEP_SECS"
--cooldown-secs "$COOLDOWN_SECS"
--out-dir "$run_dir"
)
if [[ "$INSECURE" == "true" ]]; then
cmd+=(--insecure)
fi
if [[ -n "$baseline_csv" ]]; then
cmd+=(--baseline-csv "$baseline_csv")
fi
if [[ -n "$EXTRA_ARGS" ]]; then
cmd+=(--extra-args "$EXTRA_ARGS")
fi
if [[ -n "$SERVICE_METRICS_URL" || -n "$SERVICE_PROMETHEUS_QUERY_URL" ]]; then
cmd+=(--service-metrics-dir "${run_dir}/service-metrics")
fi
if [[ -n "$SERVICE_METRICS_URL" ]]; then
cmd+=(--service-metrics-url "$SERVICE_METRICS_URL")
fi
if [[ -n "$SERVICE_PROMETHEUS_QUERY_URL" ]]; then
cmd+=(--service-prometheus-query-url "$SERVICE_PROMETHEUS_QUERY_URL")
fi
if [[ -n "$SERVICE_PROMETHEUS_QUERY" ]]; then
cmd+=(--service-prometheus-query "$SERVICE_PROMETHEUS_QUERY")
fi
if [[ -n "$SERVICE_METRICS_SERVICE_NAME" ]]; then
cmd+=(--service-metrics-service-name "$SERVICE_METRICS_SERVICE_NAME")
fi
if [[ -n "$SERVICE_METRICS_FILTER_REGEX" ]]; then
cmd+=(--service-metrics-filter-regex "$SERVICE_METRICS_FILTER_REGEX")
fi
cmd+=(
--python-bin "$PYTHON_BIN"
--service-metrics-attempts "$SERVICE_METRICS_CAPTURE_ATTEMPTS"
--service-metrics-retry-secs "$SERVICE_METRICS_CAPTURE_RETRY_SECS"
--service-metrics-connect-timeout-secs "$SERVICE_METRICS_CONNECT_TIMEOUT_SECS"
--service-metrics-max-time-secs "$SERVICE_METRICS_MAX_TIME_SECS"
--service-metrics-settle-secs "$SERVICE_METRICS_SETTLE_SECS"
)
if [[ "$DRY_RUN" == "true" ]]; then
cmd+=(--dry-run)
fi
echo "==== concurrency=${concurrency} bucket=${bucket} ===="
printf 'Command:'
printf ' %q' "${cmd[@]}"
printf '\n'
if "${cmd[@]}"; then
status="ok"
else
status="failed"
fi
echo "${concurrency},${bucket},${run_dir},${baseline_csv:-N/A},${status}" >> "$RUN_MATRIX_CSV"
if [[ "$status" != "ok" ]]; then
echo "ERROR: concurrency=${concurrency} failed; aborting remaining matrix runs" >&2
exit 1
fi
append_aggregate_rows "$concurrency" "$bucket" "$run_dir" "${run_dir}/median_summary.csv" "${run_dir}/baseline_compare.csv"
}
main() {
parse_args "$@"
validate_args
require_cmd bash
require_cmd git
require_cmd awk
require_cmd sed
require_cmd sort
require_cmd "$WARP_BIN"
if [[ ! -x "$ENHANCED_BENCH_SCRIPT" && ! -f "$ENHANCED_BENCH_SCRIPT" ]]; then
echo "ERROR: missing dependency script: $ENHANCED_BENCH_SCRIPT" >&2
exit 1
fi
setup_output
write_manifest
write_artifact_layout
echo "Output dir: $OUT_DIR"
echo "Concurrencies: $CONCURRENCIES"
echo "Sizes: $SIZES"
echo "Duration: $DURATION"
echo "Rounds: $ROUNDS"
echo "Cooldown secs: $COOLDOWN_SECS"
if [[ -n "$SERVICE_METRICS_URL" || -n "$SERVICE_PROMETHEUS_QUERY_URL" ]]; then
echo "Service metrics capture: enabled"
echo "Service metrics source: $([[ -n "$SERVICE_PROMETHEUS_QUERY_URL" ]] && echo prometheus_query || echo prometheus_text)"
fi
local conc_count conc_index
conc_count="$(csv_to_lines "$CONCURRENCIES" | awk 'END{print NR+0}')"
conc_index=0
while IFS= read -r concurrency; do
conc_index=$(( conc_index + 1 ))
run_concurrency "$concurrency"
if (( COOLDOWN_SECS > 0 && conc_index < conc_count )); then
echo "Cooldown after concurrency=${concurrency}: ${COOLDOWN_SECS}s"
cooldown_sleep "$COOLDOWN_SECS"
fi
done < <(csv_to_lines "$CONCURRENCIES")
analyze_service_metric_deltas
echo
echo "Stage-breakdown run finished."
echo "Artifacts written to: $OUT_DIR"
echo "Top-level summaries:"
echo " - $RUN_MATRIX_CSV"
echo " - $AGG_MEDIAN_CSV"
if [[ -s "$AGG_SERVICE_METRICS_CSV" ]]; then
echo " - $AGG_SERVICE_METRICS_CSV"
fi
if [[ -s "$AGG_SERVICE_METRICS_DELTAS_CSV" ]]; then
echo " - $AGG_SERVICE_METRICS_DELTAS_CSV"
echo " - $AGG_SERVICE_METRICS_PATH_SUMMARY_CSV"
echo " - $AGG_SERVICE_METRICS_STAGE_SUMMARY_CSV"
fi
if [[ -s "$AGG_COMPARE_CSV" ]]; then
echo " - $AGG_COMPARE_CSV"
fi
}
main "$@"
@@ -0,0 +1,406 @@
#!/usr/bin/env bash
set -euo pipefail
# One-shot wrapper for rustfs/backlog#706:
# - runs the large PUT stage-breakdown benchmark matrix
# - starts a parallel artifact collector that covers the benchmark window
# - stores both outputs under the same run root
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
BENCH_SCRIPT="${PROJECT_ROOT}/scripts/run_put_large_stage_breakdown.sh"
CAPTURE_SCRIPT="${PROJECT_ROOT}/scripts/collect_put_large_stage_breakdown_artifacts.sh"
ENDPOINT=""
ACCESS_KEY=""
SECRET_KEY=""
REGION="us-east-1"
BUCKET_PREFIX="rustfs-put-large"
CONCURRENCIES="16,32,64,96,128"
SIZES="16MiB,32MiB"
DURATION="120s"
ROUNDS=3
RETRY_PER_ROUND=1
RETRY_SLEEP_SECS=2
COOLDOWN_SECS=0
OUT_DIR=""
BASELINE_ROOT=""
EXTRA_ARGS=""
INSECURE=false
DRY_RUN=false
WARP_BIN="${WARP_BIN:-warp}"
TOPOLOGY_NODES=""
TOPOLOGY_DISKS_PER_NODE=""
TOPOLOGY_TOTAL_DISKS=""
TOPOLOGY_CPU_PER_NODE=""
TOPOLOGY_MEM_PER_NODE=""
TOPOLOGY_NETWORK=""
TOPOLOGY_ENDPOINT_MODE=""
TOPOLOGY_ERASURE_SET_DRIVE_COUNT=""
CLIENT_HOST=""
WORKLOAD_LABEL="backlog-706-large-put-stage-breakdown"
CAPTURE_LABEL="benchmark-window"
CAPTURE_METRICS_ENDPOINTS=""
CAPTURE_PROM_METRICS_URLS=""
CAPTURE_DURATION_SECS=""
CAPTURE_INTERVAL_SECS=15
CAPTURE_RUSTFS_PID=""
CAPTURE_SKIP_HOST_TELEMETRY=false
SKIP_CAPTURE=false
usage() {
cat <<'USAGE'
Usage:
scripts/run_put_large_stage_breakdown_with_capture.sh --endpoint <url> \
--access-key <ak> --secret-key <sk> [options]
Benchmark options:
--endpoint <url>
--access-key <ak>
--secret-key <sk>
--bucket-prefix <prefix> Default: rustfs-put-large
--region <name> Default: us-east-1
--sizes <csv> Default: 16MiB,32MiB
--concurrencies <csv> Default: 16,32,64,96,128
--duration <dur> Default: 120s
--rounds <n> Default: 3
--retry-per-round <n> Default: 1
--retry-sleep-secs <n> Default: 2
--cooldown-secs <n> Sleep between rounds/sizes (default: 0)
--out-dir <dir> Default: target/bench/put-large-stage-breakdown-<timestamp>
--baseline-root <dir>
--extra-args "<args>"
--warp-bin <path> Default: warp
--insecure
--dry-run
Topology metadata:
--nodes <n>
--disks-per-node <n>
--total-disks <n>
--cpu-per-node <text>
--mem-per-node <text>
--network <text>
--endpoint-mode <direct|lb>
--erasure-set-drive-count <n>
--client-host <text>
--workload-label <text>
Capture options:
--capture-label <name> Default: benchmark-window
--capture-metrics-endpoints <csv> Signed admin metrics endpoints
--capture-prom-metrics-urls <csv> Optional plain Prometheus text endpoints
--capture-duration-secs <n> Override auto-derived capture window
--capture-interval-secs <n> Default: 15
--capture-rustfs-pid <pid>
--capture-skip-host-telemetry
--skip-capture Run benchmark only and do not start capture
Behavior:
- If --capture-duration-secs is omitted, the wrapper derives a nominal
capture window from the benchmark matrix and adds a safety slack.
- The collector runs in the background while the benchmark matrix is in the foreground.
USAGE
}
require_cmd() {
if ! command -v "$1" >/dev/null 2>&1; then
echo "ERROR: command not found: $1" >&2
exit 1
fi
}
arg_value() {
local flag="$1"
local value="${2:-}"
if [[ -z "$value" || "$value" == --* ]]; then
echo "ERROR: missing value for $flag" >&2
exit 1
fi
printf '%s\n' "$value"
}
parse_args() {
while [[ $# -gt 0 ]]; do
case "$1" in
--endpoint) ENDPOINT="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--access-key) ACCESS_KEY="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--secret-key) SECRET_KEY="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--bucket-prefix) BUCKET_PREFIX="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--region) REGION="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--sizes) SIZES="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--concurrencies) CONCURRENCIES="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--duration) DURATION="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--rounds) ROUNDS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--retry-per-round) RETRY_PER_ROUND="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--retry-sleep-secs) RETRY_SLEEP_SECS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--cooldown-secs) COOLDOWN_SECS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--out-dir) OUT_DIR="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--baseline-root) BASELINE_ROOT="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--extra-args) EXTRA_ARGS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--warp-bin) WARP_BIN="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--nodes) TOPOLOGY_NODES="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--disks-per-node) TOPOLOGY_DISKS_PER_NODE="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--total-disks) TOPOLOGY_TOTAL_DISKS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--cpu-per-node) TOPOLOGY_CPU_PER_NODE="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--mem-per-node) TOPOLOGY_MEM_PER_NODE="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--network) TOPOLOGY_NETWORK="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--endpoint-mode) TOPOLOGY_ENDPOINT_MODE="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--erasure-set-drive-count) TOPOLOGY_ERASURE_SET_DRIVE_COUNT="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--client-host) CLIENT_HOST="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--workload-label) WORKLOAD_LABEL="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--capture-label) CAPTURE_LABEL="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--capture-metrics-endpoints) CAPTURE_METRICS_ENDPOINTS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--capture-prom-metrics-urls) CAPTURE_PROM_METRICS_URLS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--capture-duration-secs) CAPTURE_DURATION_SECS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--capture-interval-secs) CAPTURE_INTERVAL_SECS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--capture-rustfs-pid) CAPTURE_RUSTFS_PID="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--capture-skip-host-telemetry) CAPTURE_SKIP_HOST_TELEMETRY=true; shift ;;
--skip-capture) SKIP_CAPTURE=true; shift ;;
--insecure) INSECURE=true; shift ;;
--dry-run) DRY_RUN=true; shift ;;
-h|--help) usage; exit 0 ;;
*)
echo "ERROR: unknown arg: $1" >&2
usage
exit 1
;;
esac
done
}
is_positive_int() {
[[ "$1" =~ ^[1-9][0-9]*$ ]]
}
is_nonnegative_int() {
[[ "$1" =~ ^[0-9]+$ ]]
}
count_csv_items() {
local csv="$1"
local count=0 raw item
IFS=',' read -r -a arr <<< "$csv"
for raw in "${arr[@]}"; do
item="$(echo "$raw" | awk '{$1=$1;print}')"
[[ -z "$item" ]] && continue
count=$((count + 1))
done
echo "$count"
}
duration_to_seconds() {
local value="$1"
if [[ "$value" =~ ^([0-9]+)s$ ]]; then
echo "${BASH_REMATCH[1]}"
elif [[ "$value" =~ ^([0-9]+)m$ ]]; then
echo $(( BASH_REMATCH[1] * 60 ))
elif [[ "$value" =~ ^([0-9]+)h$ ]]; then
echo $(( BASH_REMATCH[1] * 3600 ))
elif [[ "$value" =~ ^[0-9]+$ ]]; then
echo "$value"
else
echo "ERROR"
fi
}
validate_args() {
if [[ -z "$ENDPOINT" || -z "$ACCESS_KEY" || -z "$SECRET_KEY" ]]; then
echo "ERROR: --endpoint, --access-key, and --secret-key are required" >&2
exit 1
fi
if ! is_positive_int "$ROUNDS" || ! is_positive_int "$RETRY_PER_ROUND" || ! is_positive_int "$RETRY_SLEEP_SECS" || ! is_nonnegative_int "$COOLDOWN_SECS"; then
echo "ERROR: --rounds, --retry-per-round, and --retry-sleep-secs must be positive integers; --cooldown-secs must be a nonnegative integer" >&2
exit 1
fi
if ! is_positive_int "$CAPTURE_INTERVAL_SECS"; then
echo "ERROR: --capture-interval-secs must be a positive integer" >&2
exit 1
fi
if [[ -n "$CAPTURE_DURATION_SECS" && ! "$CAPTURE_DURATION_SECS" =~ ^[0-9]+$ ]]; then
echo "ERROR: --capture-duration-secs must be a nonnegative integer" >&2
exit 1
fi
if [[ -n "$BASELINE_ROOT" && ! -d "$BASELINE_ROOT" ]]; then
echo "ERROR: --baseline-root does not exist: $BASELINE_ROOT" >&2
exit 1
fi
local duration_secs
duration_secs="$(duration_to_seconds "$DURATION")"
if [[ "$duration_secs" == "ERROR" ]]; then
echo "ERROR: unsupported --duration format: $DURATION (expected e.g. 120s, 2m, 1h)" >&2
exit 1
fi
}
setup_output() {
if [[ -z "$OUT_DIR" ]]; then
OUT_DIR="target/bench/put-large-stage-breakdown-$(date -u +%Y%m%dT%H%M%SZ)"
fi
}
derive_capture_duration() {
if [[ -n "$CAPTURE_DURATION_SECS" ]]; then
echo "$CAPTURE_DURATION_SECS"
return
fi
local sizes_count conc_count duration_secs nominal bench_secs retry_slack cooldown_slack fixed_slack
sizes_count="$(count_csv_items "$SIZES")"
conc_count="$(count_csv_items "$CONCURRENCIES")"
duration_secs="$(duration_to_seconds "$DURATION")"
bench_secs=$(( sizes_count * conc_count * ROUNDS * duration_secs ))
retry_slack=$(( conc_count * RETRY_PER_ROUND * RETRY_SLEEP_SECS + conc_count * 30 ))
cooldown_slack=$(( (conc_count * (sizes_count * (ROUNDS - 1) + (sizes_count - 1)) + (conc_count - 1)) * COOLDOWN_SECS ))
fixed_slack=120
nominal=$(( bench_secs + retry_slack + cooldown_slack + fixed_slack ))
echo "$nominal"
}
run_capture() {
local capture_duration="$1"
local -a cmd=(
bash "$CAPTURE_SCRIPT"
--run-root "$OUT_DIR"
--endpoint "$ENDPOINT"
--label "$CAPTURE_LABEL"
--access-key "$ACCESS_KEY"
--secret-key-env RUSTFS_CAPTURE_SECRET_KEY
--region "$REGION"
--duration-secs "$capture_duration"
--interval-secs "$CAPTURE_INTERVAL_SECS"
)
if [[ -n "$CAPTURE_METRICS_ENDPOINTS" ]]; then
cmd+=(--metrics-endpoints "$CAPTURE_METRICS_ENDPOINTS")
fi
if [[ -n "$CAPTURE_PROM_METRICS_URLS" ]]; then
cmd+=(--prom-metrics-urls "$CAPTURE_PROM_METRICS_URLS")
fi
if [[ -n "$CAPTURE_RUSTFS_PID" ]]; then
cmd+=(--rustfs-pid "$CAPTURE_RUSTFS_PID")
fi
if [[ "$CAPTURE_SKIP_HOST_TELEMETRY" == "true" ]]; then
cmd+=(--skip-host-telemetry)
fi
if [[ "$DRY_RUN" == "true" ]]; then
cmd+=(--dry-run)
fi
printf 'Capture command:'
printf ' %q' "${cmd[@]}"
printf '\n'
RUSTFS_CAPTURE_SECRET_KEY="$SECRET_KEY" "${cmd[@]}" &
CAPTURE_PID=$!
}
run_benchmark() {
local -a cmd=(
bash "$BENCH_SCRIPT"
--endpoint "$ENDPOINT"
--access-key "$ACCESS_KEY"
--secret-key "$SECRET_KEY"
--bucket-prefix "$BUCKET_PREFIX"
--region "$REGION"
--sizes "$SIZES"
--concurrencies "$CONCURRENCIES"
--duration "$DURATION"
--rounds "$ROUNDS"
--retry-per-round "$RETRY_PER_ROUND"
--retry-sleep-secs "$RETRY_SLEEP_SECS"
--cooldown-secs "$COOLDOWN_SECS"
--out-dir "$OUT_DIR"
--warp-bin "$WARP_BIN"
--workload-label "$WORKLOAD_LABEL"
)
if [[ -n "$BASELINE_ROOT" ]]; then
cmd+=(--baseline-root "$BASELINE_ROOT")
fi
if [[ -n "$EXTRA_ARGS" ]]; then
cmd+=(--extra-args "$EXTRA_ARGS")
fi
if [[ -n "$TOPOLOGY_NODES" ]]; then
cmd+=(--nodes "$TOPOLOGY_NODES")
fi
if [[ -n "$TOPOLOGY_DISKS_PER_NODE" ]]; then
cmd+=(--disks-per-node "$TOPOLOGY_DISKS_PER_NODE")
fi
if [[ -n "$TOPOLOGY_TOTAL_DISKS" ]]; then
cmd+=(--total-disks "$TOPOLOGY_TOTAL_DISKS")
fi
if [[ -n "$TOPOLOGY_CPU_PER_NODE" ]]; then
cmd+=(--cpu-per-node "$TOPOLOGY_CPU_PER_NODE")
fi
if [[ -n "$TOPOLOGY_MEM_PER_NODE" ]]; then
cmd+=(--mem-per-node "$TOPOLOGY_MEM_PER_NODE")
fi
if [[ -n "$TOPOLOGY_NETWORK" ]]; then
cmd+=(--network "$TOPOLOGY_NETWORK")
fi
if [[ -n "$TOPOLOGY_ENDPOINT_MODE" ]]; then
cmd+=(--endpoint-mode "$TOPOLOGY_ENDPOINT_MODE")
fi
if [[ -n "$TOPOLOGY_ERASURE_SET_DRIVE_COUNT" ]]; then
cmd+=(--erasure-set-drive-count "$TOPOLOGY_ERASURE_SET_DRIVE_COUNT")
fi
if [[ -n "$CLIENT_HOST" ]]; then
cmd+=(--client-host "$CLIENT_HOST")
fi
if [[ "$INSECURE" == "true" ]]; then
cmd+=(--insecure)
fi
if [[ "$DRY_RUN" == "true" ]]; then
cmd+=(--dry-run)
fi
printf 'Benchmark command:'
printf ' %q' "${cmd[@]}"
printf '\n'
"${cmd[@]}"
}
cleanup_capture() {
if [[ -n "${CAPTURE_PID:-}" ]] && kill -0 "$CAPTURE_PID" >/dev/null 2>&1; then
kill "$CAPTURE_PID" >/dev/null 2>&1 || true
wait "$CAPTURE_PID" >/dev/null 2>&1 || true
fi
}
main() {
parse_args "$@"
validate_args
require_cmd bash
require_cmd awk
require_cmd "$WARP_BIN"
setup_output
echo "Output dir: $OUT_DIR"
if [[ "$SKIP_CAPTURE" != "true" ]]; then
local capture_duration
capture_duration="$(derive_capture_duration)"
echo "Derived capture duration seconds: $capture_duration"
echo "Capture label: $CAPTURE_LABEL"
trap cleanup_capture EXIT
run_capture "$capture_duration"
else
echo "Capture: skipped"
fi
run_benchmark
if [[ "$SKIP_CAPTURE" != "true" ]]; then
wait "$CAPTURE_PID"
CAPTURE_PID=""
fi
echo
echo "One-shot benchmark + capture run finished."
echo "Run root: $OUT_DIR"
}
main "$@"
+430
View File
@@ -0,0 +1,430 @@
#!/usr/bin/env bash
set -euo pipefail
# rustfs/backlog#708 tuning matrix runner for large-object PUT.
# This controller:
# - defines high-priority tuning profiles for 16MiB / 32MiB PUT
# - writes an env snapshot per profile
# - optionally calls an apply hook after switching profile env
# - reuses scripts/run_put_large_stage_breakdown_with_capture.sh for each profile
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
RUNNER_SCRIPT="${SCRIPT_DIR}/run_put_large_stage_breakdown_with_capture.sh"
GROUP="all" # all|baseline|inflight|io-buffer|runtime|duplex
ENDPOINT=""
ACCESS_KEY=""
SECRET_KEY=""
REGION="us-east-1"
BUCKET_PREFIX="rustfs-put-large"
CONCURRENCIES="16,32,64,96,128"
SIZES="16MiB,32MiB"
DURATION="120s"
ROUNDS=3
OUT_ROOT=""
WARP_BIN="warp"
EXTRA_ARGS=""
INSECURE=false
DRY_RUN=false
COOLDOWN_SECS=30
TOPOLOGY_NODES=""
TOPOLOGY_DISKS_PER_NODE=""
TOPOLOGY_TOTAL_DISKS=""
TOPOLOGY_CPU_PER_NODE=""
TOPOLOGY_MEM_PER_NODE=""
TOPOLOGY_NETWORK=""
TOPOLOGY_ENDPOINT_MODE=""
TOPOLOGY_ERASURE_SET_DRIVE_COUNT=""
CLIENT_HOST=""
CAPTURE_METRICS_ENDPOINTS=""
CAPTURE_PROM_METRICS_URLS=""
CAPTURE_INTERVAL_SECS=15
CAPTURE_RUSTFS_PID=""
CAPTURE_SKIP_HOST_TELEMETRY=false
SKIP_CAPTURE=false
APPLY_CMD=""
APPLY_CMD_ARR=()
APPLY_WAIT_SECS=20
BASE_OBJECT_IO_BUFFER_SIZE=262144
BASE_OBJECT_DUPLEX_BUFFER_SIZE=8388608
BASE_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
BASE_RUNTIME_WORKER_THREADS=12
BASE_RUNTIME_MAX_BLOCKING_THREADS=512
usage() {
cat <<'USAGE'
Usage:
scripts/run_put_large_tuning_matrix.sh --endpoint <url> \
--access-key <ak> --secret-key <sk> [options]
Required:
--endpoint <url>
--access-key <ak>
--secret-key <sk>
Core options:
--group <name> all|baseline|inflight|io-buffer|runtime|duplex
--bucket-prefix <prefix> Default: rustfs-put-large
--region <name> Default: us-east-1
--concurrencies <csv> Default: 16,32,64,96,128
--sizes <csv> Default: 16MiB,32MiB
--duration <dur> Default: 120s
--rounds <n> Default: 3
--out-root <dir> Default: target/bench/put-large-tuning-<timestamp>
--warp-bin <path> Default: warp
--extra-args "<args>"
--cooldown-secs <n> Sleep between profiles (default: 30)
--insecure
--dry-run
Topology metadata:
--nodes <n>
--disks-per-node <n>
--total-disks <n>
--cpu-per-node <text>
--mem-per-node <text>
--network <text>
--endpoint-mode <direct|lb>
--erasure-set-drive-count <n>
--client-host <text>
Capture options:
--capture-metrics-endpoints <csv>
--capture-prom-metrics-urls <csv>
--capture-interval-secs <n> Default: 15
--capture-rustfs-pid <pid>
--capture-skip-host-telemetry
--skip-capture Reuse benchmark runner without starting capture
Optional apply hook:
--apply-cmd "<cmd>" Optional plain command + args, no shell operators.
Useful for restarting/reloading RustFS after env changes.
--apply-wait-secs <n> Wait after apply command (default: 20)
Behavior:
- Each profile writes: <out-root>/<profile>/env_snapshot.env
- Each profile then calls scripts/run_put_large_stage_breakdown_with_capture.sh
- Non-baseline profiles automatically compare against <out-root>/baseline
USAGE
}
require_cmd() {
if ! command -v "$1" >/dev/null 2>&1; then
echo "ERROR: command not found: $1" >&2
exit 1
fi
}
arg_value() {
local flag="$1"
local value="${2:-}"
if [[ -z "$value" || "$value" == --* ]]; then
echo "ERROR: missing value for $flag" >&2
exit 1
fi
printf '%s\n' "$value"
}
parse_apply_cmd() {
local raw="$1"
if [[ "$raw" == *';'* || "$raw" == *'&&'* || "$raw" == *'||'* || "$raw" == *'|'* || "$raw" == *'<'* || "$raw" == *'>'* || "$raw" == *'`'* || "$raw" == *'$'* ]]; then
echo "ERROR: --apply-cmd does not allow shell operators or expansions; pass a plain command and args only" >&2
exit 1
fi
IFS=$' \t\n' read -r -a APPLY_CMD_ARR <<< "$raw"
if [[ "${#APPLY_CMD_ARR[@]}" -eq 0 ]]; then
echo "ERROR: --apply-cmd must not be empty" >&2
exit 1
fi
require_cmd "${APPLY_CMD_ARR[0]}"
}
parse_args() {
while [[ $# -gt 0 ]]; do
case "$1" in
--endpoint) ENDPOINT="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--access-key) ACCESS_KEY="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--secret-key) SECRET_KEY="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--group) GROUP="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--bucket-prefix) BUCKET_PREFIX="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--region) REGION="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--concurrencies) CONCURRENCIES="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--sizes) SIZES="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--duration) DURATION="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--rounds) ROUNDS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--out-root) OUT_ROOT="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--warp-bin) WARP_BIN="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--extra-args) EXTRA_ARGS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--cooldown-secs) COOLDOWN_SECS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--nodes) TOPOLOGY_NODES="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--disks-per-node) TOPOLOGY_DISKS_PER_NODE="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--total-disks) TOPOLOGY_TOTAL_DISKS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--cpu-per-node) TOPOLOGY_CPU_PER_NODE="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--mem-per-node) TOPOLOGY_MEM_PER_NODE="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--network) TOPOLOGY_NETWORK="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--endpoint-mode) TOPOLOGY_ENDPOINT_MODE="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--erasure-set-drive-count) TOPOLOGY_ERASURE_SET_DRIVE_COUNT="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--client-host) CLIENT_HOST="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--capture-metrics-endpoints) CAPTURE_METRICS_ENDPOINTS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--capture-prom-metrics-urls) CAPTURE_PROM_METRICS_URLS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--capture-interval-secs) CAPTURE_INTERVAL_SECS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--capture-rustfs-pid) CAPTURE_RUSTFS_PID="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--capture-skip-host-telemetry) CAPTURE_SKIP_HOST_TELEMETRY=true; shift ;;
--skip-capture) SKIP_CAPTURE=true; shift ;;
--apply-cmd) APPLY_CMD="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--apply-wait-secs) APPLY_WAIT_SECS="$(arg_value "$1" "${2:-}")"; shift 2 ;;
--insecure) INSECURE=true; shift ;;
--dry-run) DRY_RUN=true; shift ;;
-h|--help) usage; exit 0 ;;
*)
echo "ERROR: unknown arg: $1" >&2
usage
exit 1
;;
esac
done
}
is_positive_int() {
[[ "$1" =~ ^[1-9][0-9]*$ ]]
}
validate_args() {
if [[ -z "$ENDPOINT" || -z "$ACCESS_KEY" || -z "$SECRET_KEY" ]]; then
echo "ERROR: --endpoint, --access-key, and --secret-key are required" >&2
exit 1
fi
case "$GROUP" in
all|baseline|inflight|io-buffer|runtime|duplex) ;;
*) echo "ERROR: --group must be all|baseline|inflight|io-buffer|runtime|duplex" >&2; exit 1 ;;
esac
if ! is_positive_int "$ROUNDS" || ! is_positive_int "$CAPTURE_INTERVAL_SECS" || ! is_positive_int "$APPLY_WAIT_SECS" || ! [[ "$COOLDOWN_SECS" =~ ^[0-9]+$ ]]; then
echo "ERROR: --rounds, --capture-interval-secs, and --apply-wait-secs must be positive integers; --cooldown-secs must be a nonnegative integer" >&2
exit 1
fi
if [[ -n "$APPLY_CMD" ]]; then
parse_apply_cmd "$APPLY_CMD"
fi
}
setup_out_root() {
if [[ -z "$OUT_ROOT" ]]; then
OUT_ROOT="target/bench/put-large-tuning-$(date -u +%Y%m%dT%H%M%SZ)"
fi
mkdir -p "$OUT_ROOT"
}
apply_profile() {
local profile="$1"
export RUSTFS_OBJECT_IO_BUFFER_SIZE="$BASE_OBJECT_IO_BUFFER_SIZE"
export RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE="$BASE_OBJECT_DUPLEX_BUFFER_SIZE"
export RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES="$BASE_ERASURE_ENCODE_MAX_INFLIGHT_BYTES"
export RUSTFS_RUNTIME_WORKER_THREADS="$BASE_RUNTIME_WORKER_THREADS"
export RUSTFS_RUNTIME_MAX_BLOCKING_THREADS="$BASE_RUNTIME_MAX_BLOCKING_THREADS"
case "$profile" in
baseline) ;;
inflight-32m) export RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=33554432 ;;
inflight-48m) export RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=50331648 ;;
inflight-64m) export RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=67108864 ;;
io-buffer-512k) export RUSTFS_OBJECT_IO_BUFFER_SIZE=524288 ;;
io-buffer-1m) export RUSTFS_OBJECT_IO_BUFFER_SIZE=1048576 ;;
runtime-w16-b768)
export RUSTFS_RUNTIME_WORKER_THREADS=16
export RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=768
;;
runtime-w20-b1024)
export RUSTFS_RUNTIME_WORKER_THREADS=20
export RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1024
;;
duplex-16m) export RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=16777216 ;;
*)
echo "ERROR: unsupported profile $profile" >&2
exit 1
;;
esac
}
write_env_snapshot() {
local out_file="$1"
cat > "$out_file" <<EOF
RUSTFS_OBJECT_IO_BUFFER_SIZE=${RUSTFS_OBJECT_IO_BUFFER_SIZE}
RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=${RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE}
RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=${RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES}
RUSTFS_RUNTIME_WORKER_THREADS=${RUSTFS_RUNTIME_WORKER_THREADS}
RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=${RUSTFS_RUNTIME_MAX_BLOCKING_THREADS}
EOF
}
run_apply_hook_if_needed() {
local profile="$1"
local env_file="$2"
if [[ "${#APPLY_CMD_ARR[@]}" -eq 0 ]]; then
return
fi
echo "[${profile}] running apply command..."
if [[ "$DRY_RUN" == "true" ]]; then
printf '[DRY-RUN] RUSTFS_TUNING_ENV_FILE=%q ' "$env_file"
printf '%q ' "${APPLY_CMD_ARR[@]}"
printf '\n'
echo "[DRY-RUN] sleep $APPLY_WAIT_SECS"
else
RUSTFS_TUNING_ENV_FILE="$env_file" "${APPLY_CMD_ARR[@]}"
echo "[${profile}] waiting ${APPLY_WAIT_SECS}s for service readiness..."
sleep "$APPLY_WAIT_SECS"
fi
}
profiles_for_group() {
case "$1" in
baseline) echo "baseline" ;;
inflight) echo "baseline inflight-32m inflight-48m inflight-64m" ;;
io-buffer) echo "baseline io-buffer-512k io-buffer-1m" ;;
runtime) echo "baseline runtime-w16-b768 runtime-w20-b1024" ;;
duplex) echo "baseline duplex-16m" ;;
all) echo "baseline inflight-32m inflight-48m inflight-64m io-buffer-512k io-buffer-1m runtime-w16-b768 runtime-w20-b1024 duplex-16m" ;;
esac
}
run_profile() {
local profile="$1"
local out_dir env_file baseline_root
out_dir="${OUT_ROOT}/${profile}"
mkdir -p "$out_dir"
apply_profile "$profile"
env_file="${out_dir}/env_snapshot.env"
write_env_snapshot "$env_file"
run_apply_hook_if_needed "$profile" "$env_file"
baseline_root=""
if [[ "$profile" != "baseline" ]]; then
baseline_root="${OUT_ROOT}/baseline"
fi
local -a cmd=(
bash "$RUNNER_SCRIPT"
--endpoint "$ENDPOINT"
--access-key "$ACCESS_KEY"
--secret-key "$SECRET_KEY"
--bucket-prefix "${BUCKET_PREFIX}-${profile}"
--region "$REGION"
--concurrencies "$CONCURRENCIES"
--sizes "$SIZES"
--duration "$DURATION"
--rounds "$ROUNDS"
--out-dir "$out_dir"
--warp-bin "$WARP_BIN"
)
if [[ -n "$baseline_root" ]]; then
cmd+=(--baseline-root "$baseline_root")
fi
if [[ -n "$EXTRA_ARGS" ]]; then
cmd+=(--extra-args "$EXTRA_ARGS")
fi
if [[ -n "$TOPOLOGY_NODES" ]]; then
cmd+=(--nodes "$TOPOLOGY_NODES")
fi
if [[ -n "$TOPOLOGY_DISKS_PER_NODE" ]]; then
cmd+=(--disks-per-node "$TOPOLOGY_DISKS_PER_NODE")
fi
if [[ -n "$TOPOLOGY_TOTAL_DISKS" ]]; then
cmd+=(--total-disks "$TOPOLOGY_TOTAL_DISKS")
fi
if [[ -n "$TOPOLOGY_CPU_PER_NODE" ]]; then
cmd+=(--cpu-per-node "$TOPOLOGY_CPU_PER_NODE")
fi
if [[ -n "$TOPOLOGY_MEM_PER_NODE" ]]; then
cmd+=(--mem-per-node "$TOPOLOGY_MEM_PER_NODE")
fi
if [[ -n "$TOPOLOGY_NETWORK" ]]; then
cmd+=(--network "$TOPOLOGY_NETWORK")
fi
if [[ -n "$TOPOLOGY_ENDPOINT_MODE" ]]; then
cmd+=(--endpoint-mode "$TOPOLOGY_ENDPOINT_MODE")
fi
if [[ -n "$TOPOLOGY_ERASURE_SET_DRIVE_COUNT" ]]; then
cmd+=(--erasure-set-drive-count "$TOPOLOGY_ERASURE_SET_DRIVE_COUNT")
fi
if [[ -n "$CLIENT_HOST" ]]; then
cmd+=(--client-host "$CLIENT_HOST")
fi
if [[ -n "$CAPTURE_METRICS_ENDPOINTS" ]]; then
cmd+=(--capture-metrics-endpoints "$CAPTURE_METRICS_ENDPOINTS")
fi
if [[ -n "$CAPTURE_PROM_METRICS_URLS" ]]; then
cmd+=(--capture-prom-metrics-urls "$CAPTURE_PROM_METRICS_URLS")
fi
if [[ -n "$CAPTURE_RUSTFS_PID" ]]; then
cmd+=(--capture-rustfs-pid "$CAPTURE_RUSTFS_PID")
fi
if [[ "$CAPTURE_SKIP_HOST_TELEMETRY" == "true" ]]; then
cmd+=(--capture-skip-host-telemetry)
fi
if [[ "$SKIP_CAPTURE" == "true" ]]; then
cmd+=(--skip-capture)
fi
if [[ "$INSECURE" == "true" ]]; then
cmd+=(--insecure)
fi
if [[ "$DRY_RUN" == "true" ]]; then
cmd+=(--dry-run)
fi
echo
echo "===== Running profile ${profile} ====="
echo "Output: $out_dir"
echo "Env snapshot: $env_file"
if [[ "$DRY_RUN" == "true" ]]; then
printf '[DRY-RUN] '
printf '%q ' "${cmd[@]}"
printf '\n'
else
"${cmd[@]}"
fi
}
main() {
parse_args "$@"
validate_args
require_cmd bash
require_cmd awk
if [[ ! -x "$RUNNER_SCRIPT" && ! -f "$RUNNER_SCRIPT" ]]; then
echo "ERROR: missing dependency script: $RUNNER_SCRIPT" >&2
exit 1
fi
setup_out_root
echo "Tuning output root: $OUT_ROOT"
echo "Group: $GROUP"
echo "Concurrencies: $CONCURRENCIES"
echo "Sizes: $SIZES"
echo "Cooldown between profiles: ${COOLDOWN_SECS}s"
local profiles profile index total_profiles
profiles=($(profiles_for_group "$GROUP"))
total_profiles="${#profiles[@]}"
for ((index = 0; index < total_profiles; index++)); do
profile="${profiles[$index]}"
run_profile "$profile"
if (( index + 1 < total_profiles )) && (( COOLDOWN_SECS > 0 )); then
echo
echo "[${profile}] cooldown ${COOLDOWN_SECS}s before next profile..."
if [[ "$DRY_RUN" == "true" ]]; then
echo "[DRY-RUN] sleep ${COOLDOWN_SECS}"
else
sleep "$COOLDOWN_SECS"
fi
fi
done
echo
echo "Done. Profile outputs are under: $OUT_ROOT"
}
main "$@"
+38
View File
@@ -0,0 +1,38 @@
#!/usr/bin/env bash
# Setup test binaries for Docker build testing
# This script creates temporary binary files for testing Docker build process
set -e
echo "Setting up test binaries for Docker build..."
# Create temporary rustfs binary
./build-rustfs.sh -p x86_64-unknown-linux-gnu
# Create test directory structure
mkdir -p test-releases/server/rustfs/release/linux-amd64/archive
mkdir -p test-releases/server/rustfs/release/linux-arm64/archive
# Get version
VERSION=$(git describe --abbrev=0 --tags 2>/dev/null || git rev-parse --short HEAD)
# Copy binaries
cp target/x86_64-unknown-linux-gnu/release/rustfs test-releases/server/rustfs/release/linux-amd64/archive/rustfs.${VERSION}
cp target/x86_64-unknown-linux-gnu/release/rustfs.sha256sum test-releases/server/rustfs/release/linux-amd64/archive/rustfs.${VERSION}.sha256sum
# Create dummy signatures
echo "dummy signature" > test-releases/server/rustfs/release/linux-amd64/archive/rustfs.${VERSION}.minisig
echo "dummy signature" > test-releases/server/rustfs/release/linux-arm64/archive/rustfs.${VERSION}.minisig
# Also copy for arm64 (using same binary for testing)
cp target/aarch64-unknown-linux-gnu/release/rustfs test-releases/server/rustfs/release/linux-arm64/archive/rustfs.${VERSION}
cp target/aarch64-unknown-linux-gnu/release/rustfs.sha256sum test-releases/server/rustfs/release/linux-arm64/archive/rustfs.${VERSION}.sha256sum
echo "Test binaries created for version: ${VERSION}"
echo "You can now test Docker builds with these local binaries"
echo ""
echo "To start a local HTTP server for testing:"
echo " cd test-releases && python3 -m http.server 8000"
echo ""
echo "Then modify Dockerfile to use http://host.docker.internal:8000 instead of https://dl.rustfs.com/artifacts/rustfs"
+242
View File
@@ -0,0 +1,242 @@
#!/usr/bin/env bash
# GET Optimization Stress Test Script
# Usage: ./scripts/stress-test-get-optimization.sh [TARGET_HOST] [OUTPUT_DIR]
#
# Validates:
# 1. Data correctness with early-stop enabled
# 2. Concurrent GET performance
# 3. Mixed read/write stability
# 4. Early-stop behavior under slow disk conditions
set -euo pipefail
TARGET_HOST="${1:-localhost:9000}"
OUTPUT_DIR="${2:-./stress-test-results/$(date +%Y%m%d-%H%M%S)}"
MC_ALIAS="${MC_ALIAS:-rustfs}"
TEST_BUCKET="${TEST_BUCKET:-stress-test-$(date +%s)}"
TEST_DURATION="${TEST_DURATION:-300s}"
CONCURRENCY="${CONCURRENCY:-64}"
# S3 credentials
export WARP_ACCESS_KEY="${WARP_ACCESS_KEY:-rustfsadmin}"
export WARP_SECRET_KEY="${WARP_SECRET_KEY:-rustfsadmin}"
mkdir -p "$OUTPUT_DIR"
echo "=========================================="
echo "GET Optimization Stress Test"
echo "=========================================="
echo "Target: $TARGET_HOST"
echo "Output: $OUTPUT_DIR"
echo "Bucket: $TEST_BUCKET"
echo "Duration: $TEST_DURATION"
echo "Concurrency: $CONCURRENCY"
echo "=========================================="
echo ""
# Save test configuration
cat > "$OUTPUT_DIR/test-config.txt" <<EOF
Date: $(date -u +"%Y-%m-%dT%H:%M:%SZ")
Target: $TARGET_HOST
Bucket: $TEST_BUCKET
Duration: $TEST_DURATION
Concurrency: $CONCURRENCY
Environment Variables:
RUSTFS_GET_METADATA_EARLY_STOP_ENABLE=${RUSTFS_GET_METADATA_EARLY_STOP_ENABLE:-true}
RUSTFS_GET_CODEC_STREAMING_ENABLE=${RUSTFS_GET_CODEC_STREAMING_ENABLE:-false}
RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE=${RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE:-true}
EOF
# ============================================================================
# Test 1: Data Correctness Validation
# ============================================================================
echo "[1/4] Data Correctness Validation"
echo "=================================="
test_data_correctness() {
local size=$1
local count=$2
local passed=0
local failed=0
echo " Testing $count objects of size $size..."
for i in $(seq 1 $count); do
# Generate test file
local testfile="/tmp/stress-test-${size}-${i}.bin"
if [ "$size" = "1KB" ]; then
dd if=/dev/urandom of="$testfile" bs=1024 count=1 2>/dev/null
elif [ "$size" = "1MB" ]; then
dd if=/dev/urandom of="$testfile" bs=1048576 count=1 2>/dev/null
elif [ "$size" = "10MB" ]; then
dd if=/dev/urandom of="$testfile" bs=1048576 count=10 2>/dev/null
fi
# Calculate original hash
local original_hash
original_hash=$(md5 -q "$testfile" 2>/dev/null || md5sum "$testfile" | awk '{print $1}')
# Upload
mc cp "$testfile" "${MC_ALIAS}/${TEST_BUCKET}/correctness-${size}-${i}.bin" >/dev/null 2>&1
# Download and verify
local downloadfile="/tmp/stress-test-${size}-${i}-download.bin"
mc cp "${MC_ALIAS}/${TEST_BUCKET}/correctness-${size}-${i}.bin" "$downloadfile" >/dev/null 2>&1
local download_hash
download_hash=$(md5 -q "$downloadfile" 2>/dev/null || md5sum "$downloadfile" | awk '{print $1}')
if [ "$original_hash" = "$download_hash" ]; then
passed=$((passed + 1))
else
failed=$((failed + 1))
echo " MISMATCH: correctness-${size}-${i}.bin (original=$original_hash, download=$download_hash)" >> "$OUTPUT_DIR/correctness-errors.log"
fi
# Cleanup
rm -f "$testfile" "$downloadfile"
done
echo " Result: $passed passed, $failed failed"
echo "correctness_${size}: passed=$passed failed=$failed" >> "$OUTPUT_DIR/correctness-results.txt"
}
# Test different object sizes
mc mb "${MC_ALIAS}/${TEST_BUCKET}" 2>/dev/null || true
test_data_correctness "1KB" 100
test_data_correctness "1MB" 50
test_data_correctness "10MB" 10
echo ""
# ============================================================================
# Test 2: Concurrent GET Stress Test
# ============================================================================
echo "[2/4] Concurrent GET Stress Test"
echo "================================="
# Prepare test objects
echo " Preparing test objects..."
for size in 1KiB 1MiB 4MiB 10MiB; do
warp put --obj.size="$size" --num.objects=100 --host="$TARGET_HOST" --bucket="$TEST_BUCKET" --concurrent=16 >/dev/null 2>&1
done
echo " Running concurrent GET tests..."
for size in 1KiB 1MiB 4MiB 10MiB; do
for conc in 16 64 256; do
echo " GET size=$size concurrency=$conc duration=$TEST_DURATION"
output_file="$OUTPUT_DIR/get-${size}-c${conc}.json"
if warp get \
--host="$TARGET_HOST" \
--obj.size="$size" \
--concurrent="$conc" \
--duration="$TEST_DURATION" \
--bucket="$TEST_BUCKET" \
--json \
> "$output_file" 2>/dev/null; then
echo " OK"
else
echo " FAILED (see $output_file)"
fi
done
done
echo ""
# ============================================================================
# Test 3: Mixed Read/Write Stress Test
# ============================================================================
echo "[3/4] Mixed Read/Write Stress Test"
echo "==================================="
echo " Running mixed workload for $TEST_DURATION..."
warp mixed \
--host="$TARGET_HOST" \
--obj.size=1MiB \
--concurrent="$CONCURRENCY" \
--duration="$TEST_DURATION" \
--bucket="$TEST_BUCKET" \
--json \
> "$OUTPUT_DIR/mixed-1MiB-c${CONCURRENCY}.json" 2>/dev/null || echo " MIXED TEST FAILED"
echo " Mixed test complete"
echo ""
# ============================================================================
# Test 4: Early-Stop Behavior Validation
# ============================================================================
echo "[4/4] Early-Stop Behavior Validation"
echo "====================================="
# This test verifies that early-stop works correctly by checking:
# 1. All GET requests return correct data
# 2. No timeouts or errors under normal conditions
# 3. Performance is consistent
echo " Running early-stop validation with concurrent reads..."
# Create a test object
dd if=/dev/urandom of=/tmp/early-stop-test.bin bs=1048576 count=10 2>/dev/null
mc cp /tmp/early-stop-test.bin "${MC_ALIAS}/${TEST_BUCKET}/early-stop-test.bin" >/dev/null 2>&1
# Concurrent read test
local_passed=0
local_failed=0
for i in $(seq 1 100); do
downloadfile="/tmp/early-stop-download-${i}.bin"
if mc cp "${MC_ALIAS}/${TEST_BUCKET}/early-stop-test.bin" "$downloadfile" >/dev/null 2>&1; then
# Verify file size
local_size=$(stat -f%z "$downloadfile" 2>/dev/null || stat -c%s "$downloadfile" 2>/dev/null)
if [ "$local_size" = "10485760" ]; then
local_passed=$((local_passed + 1))
else
local_failed=$((local_failed + 1))
echo " SIZE MISMATCH: expected 10485760, got $local_size" >> "$OUTPUT_DIR/early-stop-errors.log"
fi
else
local_failed=$((local_failed + 1))
echo " DOWNLOAD FAILED: iteration $i" >> "$OUTPUT_DIR/early-stop-errors.log"
fi
rm -f "$downloadfile"
done
echo " Early-stop validation: $local_passed passed, $local_failed failed"
echo "early_stop_validation: passed=$local_passed failed=$local_failed" >> "$OUTPUT_DIR/early-stop-results.txt"
# Cleanup
rm -f /tmp/early-stop-test.bin
mc rm "${MC_ALIAS}/${TEST_BUCKET}/early-stop-test.bin" >/dev/null 2>&1 || true
echo ""
# ============================================================================
# Summary
# ============================================================================
echo "=========================================="
echo "Stress Test Summary"
echo "=========================================="
echo ""
echo "Results saved to: $OUTPUT_DIR"
echo ""
echo "Files:"
ls -la "$OUTPUT_DIR"
echo ""
echo "Review:"
echo " - correctness-results.txt: Data correctness validation"
echo " - early-stop-results.txt: Early-stop behavior validation"
echo " - get-*.json: Concurrent GET performance results"
echo " - mixed-*.json: Mixed workload results"
echo ""
# Cleanup test bucket
echo "Cleaning up test bucket..."
mc rm --recursive --force "${MC_ALIAS}/${TEST_BUCKET}" >/dev/null 2>&1 || true
mc rb "${MC_ALIAS}/${TEST_BUCKET}" >/dev/null 2>&1 || true
echo "Done!"
+143
View File
@@ -0,0 +1,143 @@
#!/usr/bin/env bash
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
mkdir test
cd test
# make_bucket
mc mb rustfs/mbmb
mc ls rustfs
mc rb rustfs/mbmb
mc mb rustfs/rb-force
mc ls rustfs
echo "123525" >> test.txt
mc cp test.txt rustfs/rb-force
mc rb --force rustfs/rb-force
rm test.txt
mc mb rustfs/dada
echo "123525" >> test.txt
mc cp test.txt rustfs/dada
mc get rustfs/dada/test.txt test2.txt
diff test.txt test2.txt
echo "33333" >> test2.txt
mc cp test2.txt rustfs/dada
mc get rustfs/dada/test2.txt test3.txt
diff test2.txt test3.txt
# list_buckets
mc ls rustfs/dada
mc mb rustfs/dada2
mc ls rustfs
mc ls rustfs/dada
mc rm rustfs/dada/test2.txt
mc ls rustfs/dada
dd if=/dev/urandom of=50M.file bs=1m count=50
mc cp 50M.file rustfs/dada
mc ls rustfs/dada
mc get rustfs/dada/50M.file 50m.file.d
diff 50M.file 50m.file.d
mc rm rustfs/dada/50M.file
mc ls rustfs/dada
rm test.txt test2.txt test3.txt
rm 50M.file 50m.file.d
# object_tags
echo "33333" >> tags.txt
mc cp tags.txt rustfs/dada
mc tag list rustfs/dada/tags.txt
mc tag set rustfs/dada/tags.txt "key1=value1&key2=value2"
mc tag list rustfs/dada/tags.txt
mc tag remove rustfs/dada/tags.txt
mc tag list rustfs/dada/tags.txt
rm tags.txt
# bucket_tags
mc tag list rustfs/dada
mc tag set rustfs/dada "a=b&b=c&dada=yes&yy=11&77=99&99=23&11=11"
mc tag list rustfs/dada
mc tag remove rustfs/dada
mc tag list rustfs/dada
# bucket_versioning
mc version info rustfs/dada
mc version enable rustfs/dada
mc version info rustfs/dada
mc version suspend rustfs/dada
mc version info rustfs/dada
# bucket_policy
mc anonymous get rustfs/dada
mc anonymous set public rustfs/dada
mc anonymous set upload rustfs/dada
mc anonymous set download rustfs/dada
mc anonymous list rustfs/dada
# lifecycle
mc ilm ls rustfs/dada
mc ilm rule add --expire-days 90 --noncurrent-expire-days 30 rustfs/dada
mc ilm ls rustfs/dada
# bucket_encryption
mc encrypt info rustfs/dada
mc encrypt set sse-kms rustfs-encryption-key rustfs/dada
mc encrypt info rustfs/dada
mc encrypt clear rustfs/dada
# object_lock_config
mc mb --with-lock rustfs/lock
mc retention info --default rustfs/lock
mc retention set --default GOVERNANCE "30d" rustfs/lock
mc retention info --default rustfs/lock
mc retention clear --default rustfs/lock
mc rb rustfs/lock
# bucket_notification
mc event list rustfs/dada
mc event add --event "put,delete" rustfs/dada arn:aws:sqs::primary:target
mc event list rustfs/dada
mc event rm --event "put,delete" rustfs/dada arn:aws:sqs::primary:target
mc event list rustfs/dada
# bucket_quota ? admin/v3/get-bucket-quota
# mc quota info rustfs/dada
# bucket_target ?
+59
View File
@@ -0,0 +1,59 @@
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::dada/*"
],
"Condition": {
"StringEquals": {
"s3:ExistingObjectTag/security": "public"
}
}
},
{
"Effect": "Allow",
"Action": [
"s3:DeleteObjectTagging"
],
"Resource": [
"arn:aws:s3:::dada/*"
],
"Condition": {
"StringEquals": {
"s3:ExistingObjectTag/security": "public"
}
}
},
{
"Effect": "Allow",
"Action": [
"s3:DeleteObject"
],
"Resource": [
"arn:aws:s3:::dada/*"
]
},
{
"Effect": "Allow",
"Action": [
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::dada/*"
],
"Condition": {
"ForAllValues:StringLike": {
"s3:RequestObjectTagKeys": [
"security",
"virus"
]
}
}
}
]
}
+177
View File
@@ -0,0 +1,177 @@
#!/usr/bin/env bash
set -euo pipefail
PROJECT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
COMPOSE_FILE="${COMPOSE_FILE:-docker-compose-simple.yml}"
WAIT_TIMEOUT_SECS="${WAIT_TIMEOUT_SECS:-120}"
KEEP_UP="${KEEP_UP:-false}"
RUN_S3_TESTS="${RUN_S3_TESTS:-true}"
BUILD_LOCAL_IMAGE="${BUILD_LOCAL_IMAGE:-true}"
S3_HOST="${S3_HOST:-127.0.0.1}"
S3_PORT="${S3_PORT:-9000}"
usage() {
cat <<'USAGE'
Usage:
scripts/validate_issue_1365_docker.sh [options]
Options:
--compose-file <path> docker compose file (default: docker-compose-simple.yml)
--wait-timeout <secs> health wait timeout (default: 120)
--keep-up keep compose services up after the script exits
--skip-s3-tests skip scripts/s3-tests/run.sh
--skip-build skip local Dockerfile.source image build
-h, --help show help
Environment:
COMPOSE_FILE
WAIT_TIMEOUT_SECS
KEEP_UP
RUN_S3_TESTS
BUILD_LOCAL_IMAGE
S3_HOST
S3_PORT
USAGE
}
log_info() {
printf '[INFO] %s\n' "$*"
}
log_error() {
printf '[ERROR] %s\n' "$*" >&2
}
require_cmd() {
if ! command -v "$1" >/dev/null 2>&1; then
log_error "command not found: $1"
exit 1
fi
}
compose() {
local compose_path
compose_path="$(resolve_compose_file)"
docker compose -f "${compose_path}" "$@"
}
resolve_compose_file() {
if [[ "${COMPOSE_FILE}" = /* ]]; then
printf '%s\n' "${COMPOSE_FILE}"
else
printf '%s\n' "${PROJECT_ROOT}/${COMPOSE_FILE}"
fi
}
cleanup() {
if [[ "${KEEP_UP}" == "true" ]]; then
log_info "KEEP_UP=true, leaving compose services running"
return
fi
log_info "Stopping docker compose services"
compose down -v >/dev/null 2>&1 || true
}
parse_args() {
while [[ $# -gt 0 ]]; do
case "$1" in
--compose-file)
COMPOSE_FILE="$2"
shift 2
;;
--wait-timeout)
WAIT_TIMEOUT_SECS="$2"
shift 2
;;
--keep-up)
KEEP_UP=true
shift
;;
--skip-s3-tests)
RUN_S3_TESTS=false
shift
;;
--skip-build)
BUILD_LOCAL_IMAGE=false
shift
;;
-h|--help)
usage
exit 0
;;
*)
log_error "unknown argument: $1"
usage
exit 1
;;
esac
done
}
wait_for_endpoint() {
local url="$1"
local start now
start="$(date +%s)"
while true; do
if curl -fsS --connect-timeout 2 --max-time 3 "${url}" >/dev/null 2>&1; then
return 0
fi
now="$(date +%s)"
if (( now - start >= WAIT_TIMEOUT_SECS )); then
log_error "timed out waiting for ${url}"
compose ps || true
compose logs rustfs --tail 200 || true
return 1
fi
sleep 2
done
}
main() {
parse_args "$@"
require_cmd docker
require_cmd curl
trap cleanup EXIT INT TERM
if [[ "${BUILD_LOCAL_IMAGE}" == "true" ]]; then
log_info "Building rustfs/rustfs:latest from Dockerfile.source"
docker build -f "${PROJECT_ROOT}/Dockerfile.source" -t rustfs/rustfs:latest "${PROJECT_ROOT}"
else
log_info "Skipping local image build"
fi
if [[ -z "${RUSTFS_UNSAFE_BYPASS_DISK_CHECK+x}" ]]; then
export RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true
log_info "RUSTFS_UNSAFE_BYPASS_DISK_CHECK not set; defaulting to true for local validation"
fi
log_info "Starting docker compose from $(resolve_compose_file)"
compose up -d
log_info "Waiting for RustFS health endpoint"
wait_for_endpoint "http://${S3_HOST}:${S3_PORT}/health"
log_info "Waiting for RustFS readiness endpoint"
wait_for_endpoint "http://${S3_HOST}:${S3_PORT}/health/ready"
log_info "Docker health checks passed"
if [[ "${RUN_S3_TESTS}" == "true" ]]; then
log_info "Running S3 compatibility tests against the running dockerized service"
(
cd "${PROJECT_ROOT}"
DEPLOY_MODE=existing S3_HOST="${S3_HOST}" S3_PORT="${S3_PORT}" ./scripts/s3-tests/run.sh
)
else
log_info "Skipping S3 compatibility tests"
fi
log_info "Issue 1365 docker validation completed successfully"
}
main "$@"
+298
View File
@@ -0,0 +1,298 @@
#!/usr/bin/env bash
set -euo pipefail
# Issue 2723 verification runner
# Validates site-replication behavior against Task 07 matrix/evidence.
SITE_A_ENDPOINT="${SITE_A_ENDPOINT:-}"
SITE_B_ENDPOINT="${SITE_B_ENDPOINT:-}"
ACCESS_KEY="${ACCESS_KEY:-rustfsadmin}"
SECRET_KEY="${SECRET_KEY:-rustfsadmin}"
REGION="${REGION:-us-east-1}"
CA_CERT="${CA_CERT:-}"
OUT_DIR="${OUT_DIR:-target/verify/issue-2723-$(date +%Y%m%d-%H%M%S)}"
SITE_A_RESTART_CMD="${SITE_A_RESTART_CMD:-}"
SITE_B_RESTART_CMD="${SITE_B_RESTART_CMD:-}"
BUCKET="${BUCKET:-}"
REPL_OBJECT_KEY="${REPL_OBJECT_KEY:-issue-2723-e2e-object.txt}"
REPL_OBJECT_BODY="${REPL_OBJECT_BODY:-issue-2723-replication-check}"
AWS_PROFILE="${AWS_PROFILE:-}"
AWSCURL_BIN="${AWSCURL_BIN:-awscurl}"
AWS_BIN="${AWS_BIN:-aws}"
HEALTHCHECK_FORCE_LOOPBACK_RESOLVE="${HEALTHCHECK_FORCE_LOOPBACK_RESOLVE:-false}"
usage() {
cat <<'USAGE'
Usage:
scripts/validate_issue_2723_site_replication.sh [options]
Required:
--site-a-endpoint <url> Site A admin endpoint, e.g. https://site-a.example.com:9000
--site-b-endpoint <url> Site B admin endpoint, e.g. https://site-b.example.com:9000
--access-key <ak>
--secret-key <sk>
Optional:
--region <name> AWS region for signing (default: us-east-1)
--ca-cert <path> CA cert for strict HTTPS health checks
--out-dir <dir> Artifact output directory
--site-a-restart-cmd <cmd> Restart command for site A (optional)
--site-b-restart-cmd <cmd> Restart command for site B (optional)
--bucket <name> Replication validation bucket (optional)
--repl-object-key <key> Replication validation object key
--repl-object-body <text> Replication validation object body
--awscurl-bin <path> awscurl binary (default: awscurl)
--aws-bin <path> aws cli binary (default: aws)
--aws-profile <profile> AWS CLI profile for object-flow checks
--healthcheck-force-loopback-resolve
Force HTTPS healthcheck `--resolve host:port:127.0.0.1`
(default: false; intended for single-host local Docker)
-h, --help Show help
Notes:
1) If --bucket is provided and aws cli is available, script will run optional
object-flow checks on both sites.
2) Restart verification is skipped unless both restart commands are provided.
USAGE
}
log() {
printf '[%s] %s\n' "$(date +%H:%M:%S)" "$*"
}
fail() {
log "ERROR: $*" >&2
exit 1
}
require_cmd() {
if ! command -v "$1" >/dev/null 2>&1; then
fail "required command not found: $1"
fi
}
parse_args() {
while [[ $# -gt 0 ]]; do
case "$1" in
--site-a-endpoint) SITE_A_ENDPOINT="$2"; shift 2 ;;
--site-b-endpoint) SITE_B_ENDPOINT="$2"; shift 2 ;;
--access-key) ACCESS_KEY="$2"; shift 2 ;;
--secret-key) SECRET_KEY="$2"; shift 2 ;;
--region) REGION="$2"; shift 2 ;;
--ca-cert) CA_CERT="$2"; shift 2 ;;
--out-dir) OUT_DIR="$2"; shift 2 ;;
--site-a-restart-cmd) SITE_A_RESTART_CMD="$2"; shift 2 ;;
--site-b-restart-cmd) SITE_B_RESTART_CMD="$2"; shift 2 ;;
--bucket) BUCKET="$2"; shift 2 ;;
--repl-object-key) REPL_OBJECT_KEY="$2"; shift 2 ;;
--repl-object-body) REPL_OBJECT_BODY="$2"; shift 2 ;;
--awscurl-bin) AWSCURL_BIN="$2"; shift 2 ;;
--aws-bin) AWS_BIN="$2"; shift 2 ;;
--aws-profile) AWS_PROFILE="$2"; shift 2 ;;
--healthcheck-force-loopback-resolve) HEALTHCHECK_FORCE_LOOPBACK_RESOLVE="true"; shift ;;
-h|--help) usage; exit 0 ;;
*)
fail "unknown argument: $1"
;;
esac
done
}
endpoint_scheme() {
local endpoint="$1"
if [[ "$endpoint" == https://* ]]; then
echo "https"
elif [[ "$endpoint" == http://* ]]; then
echo "http"
else
fail "endpoint must include scheme http:// or https:// : $endpoint"
fi
}
endpoint_hostport() {
local endpoint="$1"
local hostport
hostport="${endpoint#http://}"
hostport="${hostport#https://}"
hostport="${hostport%%/*}"
echo "$hostport"
}
admin_get() {
local endpoint="$1"
local path="$2"
local out_file="$3"
local url="${endpoint%/}${path}"
if [[ -n "$CA_CERT" ]]; then
REQUESTS_CA_BUNDLE="$CA_CERT" SSL_CERT_FILE="$CA_CERT" \
"$AWSCURL_BIN" --service s3 --region "$REGION" --access_key "$ACCESS_KEY" --secret_key "$SECRET_KEY" "$url" >"$out_file"
else
"$AWSCURL_BIN" --service s3 --region "$REGION" --access_key "$ACCESS_KEY" --secret_key "$SECRET_KEY" "$url" >"$out_file"
fi
}
strict_healthcheck() {
local endpoint="$1"
local label="$2"
local out_file="$3"
local scheme hostport host port
scheme="$(endpoint_scheme "$endpoint")"
hostport="$(endpoint_hostport "$endpoint")"
host="${hostport%:*}"
port="${hostport##*:}"
if [[ "$port" == "$hostport" ]]; then
port=$([[ "$scheme" == "https" ]] && echo "443" || echo "80")
fi
local url="${scheme}://${host}:${port}/health"
if [[ "$scheme" == "https" ]]; then
if [[ -z "$CA_CERT" ]]; then
fail "HTTPS endpoint requires --ca-cert for strict validation: $endpoint"
fi
if [[ "$HEALTHCHECK_FORCE_LOOPBACK_RESOLVE" == "true" ]]; then
curl -fsS --cacert "$CA_CERT" --resolve "${host}:${port}:127.0.0.1" "$url" >"$out_file"
else
curl -fsS --cacert "$CA_CERT" "$url" >"$out_file"
fi
else
curl -fsS "$url" >"$out_file"
fi
log "health check passed for ${label}: $url"
}
analyze_duplicates() {
local status_json="$1"
local out_file="$2"
jq -r '
def identity_key($e):
($e | sub("^https?://";"") | sub("/$";"") | ascii_downcase);
(.sites // {})
| to_entries
| map(.value.endpoint // "")
| map(select(. != ""))
| map(identity_key(.))
| group_by(.)
| map({identity: .[0], count: length})
| map(select(.count > 1))
' "$status_json" >"$out_file"
}
optional_object_flow_check() {
local endpoint="$1"
local label="$2"
local put_out="$3"
local get_out="$4"
if [[ -z "$BUCKET" ]]; then
log "skip object-flow check for ${label}: --bucket not provided"
return 0
fi
if ! command -v "$AWS_BIN" >/dev/null 2>&1; then
log "skip object-flow check for ${label}: aws cli not found"
return 0
fi
local common=(
--endpoint-url "$endpoint"
--region "$REGION"
--no-cli-pager
)
if [[ -n "$AWS_PROFILE" ]]; then
common+=(--profile "$AWS_PROFILE")
fi
if [[ -n "$CA_CERT" ]]; then
common+=(--ca-bundle "$CA_CERT")
fi
AWS_ACCESS_KEY_ID="$ACCESS_KEY" AWS_SECRET_ACCESS_KEY="$SECRET_KEY" \
"$AWS_BIN" "${common[@]}" s3api put-object \
--bucket "$BUCKET" --key "$REPL_OBJECT_KEY" --body <(printf '%s' "$REPL_OBJECT_BODY") >"$put_out"
AWS_ACCESS_KEY_ID="$ACCESS_KEY" AWS_SECRET_ACCESS_KEY="$SECRET_KEY" \
"$AWS_BIN" "${common[@]}" s3api head-object \
--bucket "$BUCKET" --key "$REPL_OBJECT_KEY" >"$get_out"
log "object-flow check passed for ${label} on bucket=${BUCKET}, key=${REPL_OBJECT_KEY}"
}
restart_if_configured() {
if [[ -z "$SITE_A_RESTART_CMD" || -z "$SITE_B_RESTART_CMD" ]]; then
log "skip restart verification: restart commands not fully provided"
return 0
fi
log "running restart command for site A"
bash -lc "$SITE_A_RESTART_CMD"
log "running restart command for site B"
bash -lc "$SITE_B_RESTART_CMD"
}
main() {
parse_args "$@"
[[ -n "$SITE_A_ENDPOINT" ]] || fail "--site-a-endpoint is required"
[[ -n "$SITE_B_ENDPOINT" ]] || fail "--site-b-endpoint is required"
[[ -n "$ACCESS_KEY" ]] || fail "--access-key is required"
[[ -n "$SECRET_KEY" ]] || fail "--secret-key is required"
require_cmd "$AWSCURL_BIN"
require_cmd curl
require_cmd jq
mkdir -p "$OUT_DIR"
log "output directory: $OUT_DIR"
local a_status="$OUT_DIR/site-a.status.json"
local a_info="$OUT_DIR/site-a.info.json"
local b_status="$OUT_DIR/site-b.status.json"
local b_info="$OUT_DIR/site-b.info.json"
local a_health="$OUT_DIR/site-a.health.txt"
local b_health="$OUT_DIR/site-b.health.txt"
local a_dupes="$OUT_DIR/site-a.duplicates.json"
local b_dupes="$OUT_DIR/site-b.duplicates.json"
local summary="$OUT_DIR/summary.txt"
log "step 1/6: strict health checks"
strict_healthcheck "$SITE_A_ENDPOINT" "site-a" "$a_health"
strict_healthcheck "$SITE_B_ENDPOINT" "site-b" "$b_health"
log "step 2/6: collect site-replication status/info"
admin_get "$SITE_A_ENDPOINT" "/rustfs/admin/v3/site-replication/status" "$a_status"
admin_get "$SITE_A_ENDPOINT" "/rustfs/admin/v3/site-replication/info" "$a_info"
admin_get "$SITE_B_ENDPOINT" "/rustfs/admin/v3/site-replication/status" "$b_status"
admin_get "$SITE_B_ENDPOINT" "/rustfs/admin/v3/site-replication/info" "$b_info"
log "step 3/6: duplicate identity analysis"
analyze_duplicates "$a_status" "$a_dupes"
analyze_duplicates "$b_status" "$b_dupes"
log "step 4/6: optional object-flow checks"
optional_object_flow_check "$SITE_A_ENDPOINT" "site-a" "$OUT_DIR/site-a.put.json" "$OUT_DIR/site-a.head.json"
optional_object_flow_check "$SITE_B_ENDPOINT" "site-b" "$OUT_DIR/site-b.put.json" "$OUT_DIR/site-b.head.json"
log "step 5/6: optional restart verification"
restart_if_configured
if [[ -n "$SITE_A_RESTART_CMD" && -n "$SITE_B_RESTART_CMD" ]]; then
admin_get "$SITE_A_ENDPOINT" "/rustfs/admin/v3/site-replication/status" "$OUT_DIR/site-a.status.after-restart.json"
admin_get "$SITE_B_ENDPOINT" "/rustfs/admin/v3/site-replication/status" "$OUT_DIR/site-b.status.after-restart.json"
fi
log "step 6/6: write summary"
{
echo "Issue 2723 verification summary"
echo "site-a endpoint: $SITE_A_ENDPOINT"
echo "site-b endpoint: $SITE_B_ENDPOINT"
echo "region: $REGION"
echo "ca-cert: ${CA_CERT:-<none>}"
echo
echo "Duplicate identities (site-a): $(jq 'length' "$a_dupes")"
echo "Duplicate identities (site-b): $(jq 'length' "$b_dupes")"
echo
echo "Artifacts:"
find "$OUT_DIR" -maxdepth 1 -type f | sort
} >"$summary"
log "done. summary: $summary"
cat "$summary"
}
main "$@"
@@ -0,0 +1,466 @@
#!/usr/bin/env bash
set -euo pipefail
PROJECT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
CLUSTER_COMPOSE="${CLUSTER_COMPOSE:-${PROJECT_ROOT}/.docker/compose/docker-compose.cluster.local-build.yml}"
PROJECT_NAME="${PROJECT_NAME:-rustfs-issue3031}"
RUSTFS_IMAGE="${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}"
BUILD_LOCAL_IMAGE="${BUILD_LOCAL_IMAGE:-true}"
FORCE_BUILD="${FORCE_BUILD:-false}"
KEEP_UP="${KEEP_UP:-false}"
PRECHECK_AUTO_CLEANUP="${PRECHECK_AUTO_CLEANUP:-true}"
WAIT_TIMEOUT_SECS="${WAIT_TIMEOUT_SECS:-180}"
S3_READY_TIMEOUT_SECS="${S3_READY_TIMEOUT_SECS:-120}"
RUSTFS_ACCESS_KEY="${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}"
RUSTFS_SECRET_KEY="${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}"
RUSTFS_UNSAFE_BYPASS_DISK_CHECK="${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}"
RUSTFS_ISSUE3031_DIAG_ENABLE="${RUSTFS_ISSUE3031_DIAG_ENABLE:-true}"
RUSTFS_OBS_LOG_STDOUT_ENABLED="${RUSTFS_OBS_LOG_STDOUT_ENABLED:-true}"
RUSTFS_OBS_USE_STDOUT="${RUSTFS_OBS_USE_STDOUT:-false}"
RUSTFS_OBS_LOGGER_LEVEL="${RUSTFS_OBS_LOGGER_LEVEL:-info}"
RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT="${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}"
RUSTFS_LOCK_ACQUIRE_TIMEOUT="${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}"
ENDPOINT="${ENDPOINT:-http://127.0.0.1:9000}"
WARP_HOST="${WARP_HOST:-node1:9000}"
BUCKET="${BUCKET:-rustfs-multipart-repro}"
SMOKE_OBJECTS="${SMOKE_OBJECTS:-16}"
SMOKE_OBJECT_SIZE_MB="${SMOKE_OBJECT_SIZE_MB:-32}"
SMOKE_PARALLELISM="${SMOKE_PARALLELISM:-8}"
WARP_DURATION="${WARP_DURATION:-5m}"
WARP_CONCURRENT="${WARP_CONCURRENT:-16}"
WARP_PARTS="${WARP_PARTS:-16}"
WARP_PART_SIZE="${WARP_PART_SIZE:-16MiB}"
WARP_PART_CONCURRENT="${WARP_PART_CONCURRENT:-4}"
MC_IMAGE="${MC_IMAGE:-minio/mc:latest}"
WARP_IMAGE="${WARP_IMAGE:-minio/warp:latest}"
OUT_DIR="${OUT_DIR:-${PROJECT_ROOT}/target/issue3031/$(date +%Y%m%d-%H%M%S)}"
usage() {
cat <<'USAGE'
Usage:
scripts/validate_issue_3031_docker.sh [options]
Options:
--skip-build skip local Docker image build
--force-build rebuild even if the local image already exists
--keep-up keep cluster running after the script exits
--project-name <name> docker compose project name
--out-dir <path> output directory
--warp-duration <dur> warp duration (default: 5m)
--warp-concurrent <n> warp --concurrent (default: 16)
--warp-parts <n> warp --parts (default: 16)
--warp-part-size <size> warp --part.size (default: 16MiB)
--warp-part-concurrent <n> warp --part.concurrent (default: 4)
--smoke-objects <n> plain write smoke object count (default: 16)
--smoke-object-size-mb <n> plain write smoke object size MiB (default: 32)
--smoke-parallelism <n> plain write smoke parallelism (default: 8)
-h, --help show help
Environment:
CLUSTER_COMPOSE PROJECT_NAME RUSTFS_IMAGE BUILD_LOCAL_IMAGE FORCE_BUILD KEEP_UP
RUSTFS_ACCESS_KEY RUSTFS_SECRET_KEY RUSTFS_UNSAFE_BYPASS_DISK_CHECK
RUSTFS_ISSUE3031_DIAG_ENABLE RUSTFS_OBS_LOG_STDOUT_ENABLED
RUSTFS_OBS_USE_STDOUT RUSTFS_OBS_LOGGER_LEVEL
RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT
RUSTFS_LOCK_ACQUIRE_TIMEOUT
S3_READY_TIMEOUT_SECS
ENDPOINT BUCKET
WARP_HOST
SMOKE_OBJECTS SMOKE_OBJECT_SIZE_MB SMOKE_PARALLELISM
WARP_DURATION WARP_CONCURRENT WARP_PARTS WARP_PART_SIZE WARP_PART_CONCURRENT
MC_IMAGE WARP_IMAGE OUT_DIR
USAGE
}
log_info() {
printf '[INFO] %s\n' "$*"
}
log_warn() {
printf '[WARN] %s\n' "$*"
}
log_error() {
printf '[ERROR] %s\n' "$*" >&2
}
require_cmd() {
if ! command -v "$1" >/dev/null 2>&1; then
log_error "command not found: $1"
exit 1
fi
}
compose() {
docker compose \
--project-name "${PROJECT_NAME}" \
-f "${CLUSTER_COMPOSE}" \
"$@"
}
cleanup() {
if [[ "${KEEP_UP}" == "true" ]]; then
log_info "KEEP_UP=true, leaving cluster running"
return
fi
log_info "Stopping docker compose services"
compose down -v >/dev/null 2>&1 || true
}
parse_args() {
while [[ $# -gt 0 ]]; do
case "$1" in
--skip-build)
BUILD_LOCAL_IMAGE=false
shift
;;
--keep-up)
KEEP_UP=true
shift
;;
--force-build)
FORCE_BUILD=true
shift
;;
--project-name)
PROJECT_NAME="$2"
shift 2
;;
--out-dir)
OUT_DIR="$2"
shift 2
;;
--warp-duration)
WARP_DURATION="$2"
shift 2
;;
--warp-concurrent)
WARP_CONCURRENT="$2"
shift 2
;;
--warp-parts)
WARP_PARTS="$2"
shift 2
;;
--warp-part-size)
WARP_PART_SIZE="$2"
shift 2
;;
--warp-part-concurrent)
WARP_PART_CONCURRENT="$2"
shift 2
;;
--smoke-objects)
SMOKE_OBJECTS="$2"
shift 2
;;
--smoke-object-size-mb)
SMOKE_OBJECT_SIZE_MB="$2"
shift 2
;;
--smoke-parallelism)
SMOKE_PARALLELISM="$2"
shift 2
;;
-h|--help)
usage
exit 0
;;
*)
log_error "unknown argument: $1"
usage
exit 1
;;
esac
done
}
wait_http_ok() {
local url="$1"
local start now
start="$(date +%s)"
while true; do
if curl -fsS --connect-timeout 2 --max-time 3 "${url}" >/dev/null 2>&1; then
return 0
fi
now="$(date +%s)"
if (( now - start >= WAIT_TIMEOUT_SECS )); then
log_error "timed out waiting for ${url}"
return 1
fi
sleep 2
done
}
cleanup_existing_project_containers() {
local existing_ids
existing_ids="$(docker ps -aq --filter "label=com.docker.compose.project=${PROJECT_NAME}")"
if [[ -z "${existing_ids}" ]]; then
return 0
fi
log_warn "Found existing containers for project ${PROJECT_NAME}."
docker ps -a --filter "label=com.docker.compose.project=${PROJECT_NAME}" --format ' - {{.Names}} ({{.Status}})'
if [[ "${PRECHECK_AUTO_CLEANUP}" == "true" ]]; then
log_info "PRECHECK_AUTO_CLEANUP=true, removing existing project containers."
# shellcheck disable=SC2086
docker rm -f ${existing_ids} >/dev/null
else
log_error "existing project containers detected and PRECHECK_AUTO_CLEANUP=false"
exit 1
fi
}
build_image_if_needed() {
if [[ "${BUILD_LOCAL_IMAGE}" != "true" ]]; then
log_info "Skipping local image build"
return
fi
if [[ "${FORCE_BUILD}" != "true" ]] && docker image inspect "${RUSTFS_IMAGE}" >/dev/null 2>&1; then
log_info "Reusing existing local image ${RUSTFS_IMAGE}; pass --force-build to rebuild"
return
fi
log_info "Building ${RUSTFS_IMAGE} from Dockerfile.source"
docker build -f "${PROJECT_ROOT}/Dockerfile.source" -t "${RUSTFS_IMAGE}" "${PROJECT_ROOT}"
}
start_cluster() {
mkdir -p "${OUT_DIR}"
cleanup_existing_project_containers
log_info "Starting 4-node cluster from ${CLUSTER_COMPOSE}"
compose up -d
log_info "Waiting for cluster health endpoint"
wait_http_ok "${ENDPOINT}/health"
log_info "Waiting for cluster readiness endpoint"
wait_http_ok "${ENDPOINT}/health/ready"
log_info "Waiting for S3 API readiness via mc alias/list"
wait_s3_api_ready
}
cluster_network() {
echo "${PROJECT_NAME}_rustfs-cluster-net"
}
run_mc_in_network() {
docker run --rm --network "$(cluster_network)" "${MC_IMAGE}" "$@"
}
run_mc_shell_in_network() {
docker run --rm --network "$(cluster_network)" --entrypoint /bin/sh "${MC_IMAGE}" -lc "$1"
}
run_warp_in_network() {
docker run --rm --network "$(cluster_network)" -v "${OUT_DIR}:/out" "${WARP_IMAGE}" "$@"
}
wait_s3_api_ready() {
local start now
start="$(date +%s)"
while true; do
if run_mc_in_network alias set rustfs "http://node1:9000" "${RUSTFS_ACCESS_KEY}" "${RUSTFS_SECRET_KEY}" >/dev/null 2>&1; then
if run_mc_in_network ls rustfs >/dev/null 2>&1; then
return 0
fi
fi
now="$(date +%s)"
if (( now - start >= S3_READY_TIMEOUT_SECS )); then
log_error "timed out waiting for S3 API readiness via mc alias/list"
return 1
fi
sleep 2
done
}
collect_cluster_info() {
local node
for node in node1 node2 node3 node4; do
log_info "Collecting rustfs info from ${node}"
compose exec -T "${node}" rustfs info --all --json > "${OUT_DIR}/${node}-info.json"
done
}
run_plain_smoke() {
local prefix
prefix="plain-smoke-$(date +%s)"
log_info "Preparing bucket ${BUCKET}"
run_mc_in_network alias set rustfs "http://node1:9000" "${RUSTFS_ACCESS_KEY}" "${RUSTFS_SECRET_KEY}" >/dev/null
run_mc_in_network ls "rustfs/${BUCKET}" >/dev/null 2>&1 || run_mc_in_network mb "rustfs/${BUCKET}" >/dev/null
log_info "Running plain smoke write/read/delete: objects=${SMOKE_OBJECTS} size_mb=${SMOKE_OBJECT_SIZE_MB} parallelism=${SMOKE_PARALLELISM}"
run_mc_shell_in_network "
set -euo pipefail
i=1
while [ \"\$i\" -le \"${SMOKE_OBJECTS}\" ]; do
end=\$((i + ${SMOKE_PARALLELISM} - 1))
[ \"\$end\" -le \"${SMOKE_OBJECTS}\" ] || end='${SMOKE_OBJECTS}'
j=\"\$i\"
while [ \"\$j\" -le \"\$end\" ]; do
(
dd if=/dev/urandom bs='${SMOKE_OBJECT_SIZE_MB}M' count=1 2>/dev/null | \
mc pipe 'rustfs/${BUCKET}/${prefix}/obj-\${j}' >/dev/null
) &
j=\$((j + 1))
done
wait
i=\$((end + 1))
done
mc rm --recursive --force 'rustfs/${BUCKET}/${prefix}' >/dev/null
"
}
run_warp_multipart() {
log_info "Running warp multipart-put against ${WARP_HOST}"
run_warp_in_network multipart-put \
--host "${WARP_HOST}" \
--access-key "${RUSTFS_ACCESS_KEY}" \
--secret-key "${RUSTFS_SECRET_KEY}" \
--bucket "${BUCKET}-warp" \
--lookup path \
--duration "${WARP_DURATION}" \
--concurrent "${WARP_CONCURRENT}" \
--parts "${WARP_PARTS}" \
--part.size "${WARP_PART_SIZE}" \
--part.concurrent "${WARP_PART_CONCURRENT}" \
--benchdata /out/warp.csv.zst \
--analyze.v \
--no-color \
| tee "${OUT_DIR}/warp.log"
}
count_matches() {
local pattern="$1"
shift
local total=0
local file count
for file in "$@"; do
count="$(grep -cE "${pattern}" "${file}" 2>/dev/null || true)"
total=$((total + ${count:-0}))
done
echo "${total}"
}
collect_cluster_logs() {
local node
for node in node1 node2 node3 node4; do
compose logs --no-color "${node}" > "${OUT_DIR}/${node}.log" || true
done
}
summarize_results() {
local warp_log summary
local warp_errors quorum_not_reached connection_refused storage_insufficient
local lock_timeout erasure_write_quorum readiness_probe_failed liveness_probe_failed
local diag_read_parts diag_complete_part diag_rename_part
local startup_range_oob startup_volume_not_found startup_remote_network startup_remote_faulty startup_remote_lock_rpc
warp_log="${OUT_DIR}/warp.log"
summary="${OUT_DIR}/summary.txt"
warp_errors="$(count_matches 'warp: <ERROR>' "${warp_log}")"
quorum_not_reached="$(count_matches 'Quorum not reached' "${warp_log}")"
connection_refused="$(count_matches 'connection refused' "${warp_log}")"
storage_insufficient="$(count_matches 'Storage resources are insufficient for the write operation' "${warp_log}")"
lock_timeout="$(count_matches 'Lock acquisition timeout' "${warp_log}")"
erasure_write_quorum="$(count_matches 'erasure write quorum' "${warp_log}")"
readiness_probe_failed="$(count_matches 'readiness_probe_failed' "${warp_log}")"
liveness_probe_failed="$(count_matches 'liveness_probe_failed' "${warp_log}")"
diag_read_parts="$(count_matches 'issue3031_read_parts_part_quorum' "${OUT_DIR}"/node*.log)"
diag_complete_part="$(count_matches 'issue3031_complete_part_error' "${OUT_DIR}"/node*.log)"
diag_rename_part="$(count_matches 'issue3031_rename_part_context' "${OUT_DIR}"/node*.log)"
startup_range_oob="$(count_matches 'Range \\[0, 4\\) is out of bounds' "${OUT_DIR}"/node*.log)"
startup_volume_not_found="$(count_matches 'volume not found' "${OUT_DIR}"/node*.log)"
startup_remote_network="$(count_matches 'Remote disk operation returned a network-like error' "${OUT_DIR}"/node*.log)"
startup_remote_faulty="$(count_matches 'Remote disk marked faulty after timeout' "${OUT_DIR}"/node*.log)"
startup_remote_lock_rpc="$(count_matches 'Evicting cached remote lock connection after RPC failure' "${OUT_DIR}"/node*.log)"
cat > "${summary}" <<EOF
issue=3031
endpoint=${ENDPOINT}
bucket=${BUCKET}
warp_duration=${WARP_DURATION}
warp_concurrent=${WARP_CONCURRENT}
warp_parts=${WARP_PARTS}
warp_part_size=${WARP_PART_SIZE}
warp_part_concurrent=${WARP_PART_CONCURRENT}
warp_errors=${warp_errors}
quorum_not_reached=${quorum_not_reached}
connection_refused=${connection_refused}
storage_insufficient=${storage_insufficient}
lock_timeout=${lock_timeout}
erasure_write_quorum=${erasure_write_quorum}
readiness_probe_failed=${readiness_probe_failed}
liveness_probe_failed=${liveness_probe_failed}
issue3031_read_parts_part_quorum=${diag_read_parts}
issue3031_complete_part_error=${diag_complete_part}
issue3031_rename_part_context=${diag_rename_part}
startup_range_oob=${startup_range_oob}
startup_volume_not_found=${startup_volume_not_found}
startup_remote_network_error=${startup_remote_network}
startup_remote_faulty=${startup_remote_faulty}
startup_remote_lock_rpc=${startup_remote_lock_rpc}
EOF
log_info "Summary written to ${summary}"
cat "${summary}"
}
main() {
parse_args "$@"
require_cmd docker
require_cmd curl
require_cmd grep
require_cmd tee
mkdir -p "${OUT_DIR}"
trap cleanup EXIT INT TERM
export RUSTFS_IMAGE
export RUSTFS_ACCESS_KEY
export RUSTFS_SECRET_KEY
export RUSTFS_UNSAFE_BYPASS_DISK_CHECK
export RUSTFS_ISSUE3031_DIAG_ENABLE
export RUSTFS_OBS_LOG_STDOUT_ENABLED
export RUSTFS_OBS_USE_STDOUT
export RUSTFS_OBS_LOGGER_LEVEL
export RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT
export RUSTFS_LOCK_ACQUIRE_TIMEOUT
build_image_if_needed
start_cluster
collect_cluster_info
run_plain_smoke
run_warp_multipart
collect_cluster_logs
summarize_results
}
main "$@"
+152
View File
@@ -0,0 +1,152 @@
#!/usr/bin/env bash
set -euo pipefail
# Acceptance runner for rustfs/backlog#785 and PR #4072
PROJECT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
OUT_DIR="${OUT_DIR:-${PROJECT_ROOT}/target/issue-785-acceptance-$(date +%Y%m%d-%H%M%S)}"
SKIP_LIVE="${SKIP_LIVE:-true}"
ENDPOINT="${ENDPOINT:-http://127.0.0.1:9000}"
log_info() { printf '[INFO] %s\n' "$*"; }
log_warn() { printf '[WARN] %s\n' "$*"; }
log_error() { printf '[ERROR] %s\n' "$*" >&2; }
usage() {
cat <<'USAGE'
Usage:
scripts/validate_issue_785_list_objects.sh [--full] [--no-skip-live]
Environment:
OUT_DIR output directory (default: target/issue-785-acceptance-<ts>)
SKIP_LIVE skip live S3 endpoint checks when true (default: true)
ENDPOINT rustfs endpoint used by optional live checks (default: http://127.0.0.1:9000)
USAGE
}
require_cmd() {
if ! command -v "$1" >/dev/null 2>&1; then
log_error "command not found: $1"
exit 1
fi
}
run_unit_checks() {
log_info "Running focused rustfs-ecstore tests for list_objects hot path"
mkdir -p "$OUT_DIR"
local test_log="$OUT_DIR/issue-785-ecstore-tests.log"
: > "$test_log"
local tests=(
list_path_gather_results_returns_after_limit_without_waiting_for_input_close
list_path_gather_results_keeps_marker_entry_for_version_marker_listing
list_path_gather_results_skips_marker_entry_by_default
list_path_forward_past_is_idempotent_for_same_marker
list_path_parse_marker_replay_still_stable
normalize_list_quorum_falls_back_to_strict
list_objects_quorum_from_env_defaults_to_optimal
)
local test
for test in "${tests[@]}"; do
echo "[TEST] $test" | tee -a "$test_log"
(cd "$PROJECT_ROOT" && cargo test -p rustfs-ecstore "$test" -- --nocapture | tee -a "$test_log")
done
log_info "Unit test log: $test_log"
}
run_static_checks() {
log_info "Running acceptance-focused static checks"
local static_log="$OUT_DIR/static-checks.log"
: > "$static_log"
if rg -n "store_list_objects_list_path|store_list_objects_list_merged|sets_list_objects_list_path|sets_list_objects_list_merged|set_disks_list_objects_list_path|store_list_objects_gather" \
"$PROJECT_ROOT/crates/ecstore/src/store/list_objects.rs" | tee -a "$static_log"; then
log_info "Metric and stage names exist in list_objects.rs"
else
log_error "Required metric symbols not found in list_objects.rs"
exit 1
fi
log_info "Static check log: $static_log"
}
run_live_smoke() {
if [[ "$SKIP_LIVE" == "true" ]]; then
log_warn "SKIP_LIVE=true, skipping live endpoint checks"
return 0
fi
log_info "Running optional live readiness check: $ENDPOINT"
if ! curl -fsS "${ENDPOINT}/health/ready" >/dev/null 2>&1; then
log_error "Health ready endpoint check failed: ${ENDPOINT}/health/ready"
return 1
fi
log_info "Live endpoint reachable"
log_warn "Optional live S3 pagination/candidates check is intentionally not enforced in this runner."
log_warn "Please run your preferred client workload (mc/warp/rclone) and verify page continuity + duplicates manually."
}
run_metrics_context() {
log_info "Collecting local metric symbol index for review"
local metric_log="$OUT_DIR/issue-785-metrics-context.log"
if rg -n "record_stage_duration\(" "$PROJECT_ROOT/crates/ecstore/src/store/list_objects.rs" > "$metric_log"; then
log_info "Metric context saved: $metric_log"
else
log_error "No metric stage context found"
return 1
fi
}
parse_args() {
while [[ $# -gt 0 ]]; do
case "$1" in
--full)
SKIP_LIVE="false"
shift
;;
--no-skip-live)
SKIP_LIVE="false"
shift
;;
-h|--help)
usage
exit 0
;;
*)
log_error "unknown arg: $1"
usage
exit 1
;;
esac
done
:
}
main() {
parse_args "$@"
require_cmd cargo
require_cmd rg
require_cmd tee
require_cmd curl
mkdir -p "$OUT_DIR"
run_unit_checks
run_static_checks
run_metrics_context
if [[ "$SKIP_LIVE" != "true" ]]; then
run_live_smoke
fi
log_info "Validation summary:"
log_info " - logs: $OUT_DIR"
log_info " - status: pass"
}
main "$@"
+232
View File
@@ -0,0 +1,232 @@
#!/usr/bin/env bash
set -euo pipefail
# Acceptance runner for rustfs/backlog#786 and PR #4072 follow-up
PROJECT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
OUT_DIR="${OUT_DIR:-${PROJECT_ROOT}/target/issue-786-acceptance-$(date +%Y%m%d-%H%M%S)}"
SKIP_LIVE="${SKIP_LIVE:-true}"
ENDPOINT="${ENDPOINT:-http://127.0.0.1:9000}"
LIVE_LIST_BUCKET="${LIVE_LIST_BUCKET:-}"
LIVE_LIST_MAX_KEYS="${LIVE_LIST_MAX_KEYS:-2}"
LIVE_LIST_REGION="${LIVE_LIST_REGION:-us-east-1}"
log_info() { printf '[INFO] %s\n' "$*"; }
log_warn() { printf '[WARN] %s\n' "$*"; }
log_error() { printf '[ERROR] %s\n' "$*" >&2; }
usage() {
cat <<'USAGE'
Usage:
scripts/validate_issue_786_list_objects.sh [--full] [--no-skip-live]
Environment:
OUT_DIR output directory (default: target/issue-786-acceptance-<ts>)
SKIP_LIVE skip live S3 endpoint checks when true (default: true)
ENDPOINT rustfs endpoint used by optional live checks (default: http://127.0.0.1:9000)
LIVE_LIST_BUCKET S3 bucket used for pagination smoke (required)
LIVE_LIST_MAX_KEYS maximum keys used per page for smoke (default: 2)
LIVE_LIST_REGION signing region for curl --aws-sigv4 (default: us-east-1)
USAGE
}
require_cmd() {
if ! command -v "$1" >/dev/null 2>&1; then
log_error "command not found: $1"
exit 1
fi
}
run_unit_checks() {
log_info "Running focused rustfs-ecstore tests for issue-786 cursor fallback"
mkdir -p "$OUT_DIR"
local test_log="$OUT_DIR/issue-786-ecstore-tests.log"
: > "$test_log"
local tests=(
list_path_marker_round_trip_preserves_set_index
list_path_marker_parser_uses_trailing_cache_tag
list_path_marker_parser_accepts_legacy_v1_tag
list_path_marker_parser_return_tag_forces_cache_refresh
list_path_marker_parser_recovers_from_corrupt_set_index
)
local test
for test in "${tests[@]}"; do
echo "[TEST] $test" | tee -a "$test_log"
(cd "$PROJECT_ROOT" && cargo test -p rustfs-ecstore "$test" -- --nocapture | tee -a "$test_log")
done
log_info "Unit test log: $test_log"
}
run_static_checks() {
log_info "Running issue-786 static marker checks"
local static_log="$OUT_DIR/static-checks.log"
: > "$static_log"
if rg -n "struct ListContinuationV2|return:\\]|rustfs_cache:v2|MARKER_TAG_VERSION" "$PROJECT_ROOT/crates/ecstore/src/store/list_objects.rs" | tee -a "$static_log"; then
log_info "ListContinuationV2 symbols found in list_objects.rs"
else
log_error "Required cursor symbols not found in list_objects.rs"
exit 1
fi
log_info "Static check log: $static_log"
}
extract_xml_keys() {
local xml_text="$1"
printf '%s\n' "$xml_text" | sed -n 's:.*<Key>\(.*\)</Key>.*:\1:p'
}
run_live_smoke() {
if [[ "$SKIP_LIVE" == "true" ]]; then
log_warn "SKIP_LIVE=true, skipping live endpoint checks"
return 0
fi
local live_log="$OUT_DIR/live-pagination-smoke.log"
: > "$live_log"
if ! curl -fsS "${ENDPOINT}/health/ready" >/dev/null 2>&1; then
log_warn "Live endpoint not reachable: ${ENDPOINT}/health/ready"
echo "live pagination smoke: skipped (endpoint not reachable)" | tee -a "$live_log"
return 0
fi
log_info "Live endpoint reachable"
local access_key="${RUSTFS_ACCESS_KEY:-${WARP_ACCESS_KEY:-}}"
local secret_key="${RUSTFS_SECRET_KEY:-${WARP_SECRET_KEY:-}}"
if [[ -z "$access_key" || -z "$secret_key" ]]; then
log_warn "Live pagination smoke skipped: missing RUSTFS_ACCESS_KEY / RUSTFS_SECRET_KEY or WARP_* alternatives"
echo "live pagination smoke: skipped (missing credentials)" | tee -a "$live_log"
return 0
fi
if [[ -z "$LIVE_LIST_BUCKET" ]]; then
log_warn "Live pagination smoke skipped: LIVE_LIST_BUCKET is empty"
echo "live pagination smoke: skipped (bucket not configured)" | tee -a "$live_log"
return 0
fi
if ! command -v python3 >/dev/null 2>&1; then
log_warn "Live pagination smoke skipped: python3 not available"
echo "live pagination smoke: skipped (missing python3 parser)" | tee -a "$live_log"
return 0
fi
local sigv4_target="aws:amz:${LIVE_LIST_REGION}:s3"
local base_url="${ENDPOINT}/${LIVE_LIST_BUCKET}?list-type=2&max-keys=${LIVE_LIST_MAX_KEYS}"
local page1
page1=$(curl -fsS \
--user "${access_key}:${secret_key}" \
--aws-sigv4 "$sigv4_target" \
-H "x-amz-content-sha256: UNSIGNED-PAYLOAD" \
"$base_url")
local token1
token1=$(printf '%s' "$page1" | sed -n 's:.*<NextContinuationToken>\(.*\)</NextContinuationToken>.*:\1:p' | head -n 1)
local page1_key_count
page1_key_count=$(printf '%s' "$page1" | sed -n 's:.*<Key>\(.*\)</Key>.*:\1:p' | grep -cv '^$')
local -a keys1 keys2
while IFS= read -r key; do
keys1+=("$key")
done < <(extract_xml_keys "$page1")
if [[ -z "$token1" ]]; then
log_info "Live pagination smoke: single page only (key count=${page1_key_count})"
echo "live pagination smoke: pass (single page, key_count=${page1_key_count})" | tee -a "$live_log"
return 0
fi
local token1_encoded
token1_encoded=$(printf '%s' "$token1" | python3 -c 'import urllib.parse,sys; print(urllib.parse.quote(sys.stdin.read().strip(), safe=""))')
local page2
page2=$(curl -fsS \
--user "${access_key}:${secret_key}" \
--aws-sigv4 "$sigv4_target" \
-H "x-amz-content-sha256: UNSIGNED-PAYLOAD" \
"${base_url}&continuation-token=${token1_encoded}")
local page2_key_count
page2_key_count=$(printf '%s' "$page2" | sed -n 's:.*<Key>\(.*\)</Key>.*:\1:p' | grep -cv '^$')
if (( page2_key_count == 0 )); then
log_error "Live pagination smoke: second page returned zero keys for token ${token1}"
return 1
fi
local duplicate
local dkeys=()
duplicate=0
local key
while IFS= read -r key; do
keys2+=("$key")
if printf '%s\n' "${keys1[@]}" | grep -Fx -- "$key" >/dev/null 2>&1; then
duplicate=1
dkeys+=("$key")
fi
done < <(extract_xml_keys "$page2")
if (( duplicate )); then
log_error "Live pagination smoke detected duplicated keys across page transitions: ${dkeys[*]}"
printf '%s\n' "${dkeys[@]}" | tee -a "$live_log"
return 1
fi
log_info "Live pagination smoke: first page=${page1_key_count}, second page=${page2_key_count}, token=${token1}"
echo "live pagination smoke: pass (continuous two-page check)" | tee -a "$live_log"
echo "page1=${page1_key_count}" | tee -a "$live_log"
echo "page2=${page2_key_count}" | tee -a "$live_log"
log_info "Live pagination smoke log: $live_log"
}
parse_args() {
while [[ $# -gt 0 ]]; do
case "$1" in
--full|--no-skip-live)
SKIP_LIVE="false"
shift
;;
-h|--help)
usage
exit 0
;;
*)
log_error "unknown arg: $1"
usage
exit 1
;;
esac
done
}
main() {
parse_args "$@"
require_cmd cargo
require_cmd rg
require_cmd tee
require_cmd curl
mkdir -p "$OUT_DIR"
run_unit_checks
run_static_checks
if [[ "$SKIP_LIVE" != "true" ]]; then
run_live_smoke
fi
log_info "Validation summary:"
log_info " - logs: $OUT_DIR"
log_info " - status: pass"
}
main "$@"
+348
View File
@@ -0,0 +1,348 @@
#!/usr/bin/env bash
set -euo pipefail
# Validation runner for rustfs/backlog#787: list quorum tuning and index-readiness baseline
PROJECT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
OUT_DIR="${OUT_DIR:-${PROJECT_ROOT}/target/issue-787-list-quorum-$(date +%Y%m%d-%H%M%S)}"
SKIP_LIVE="${SKIP_LIVE:-true}"
ENDPOINT="${ENDPOINT:-http://127.0.0.1:9000}"
LIVE_LIST_BUCKET="${LIVE_LIST_BUCKET:-}"
LIVE_LIST_PREFIX="${LIVE_LIST_PREFIX:-}"
LIVE_LIST_MAX_KEYS="${LIVE_LIST_MAX_KEYS:-1000}"
LIVE_LIST_REGION="${LIVE_LIST_REGION:-us-east-1}"
MODE_LIST="${MODE_LIST:-strict,optimal,reduced}"
SERVER_RESTART_CMD="${SERVER_RESTART_CMD:-}"
SERVER_WAIT_SECONDS="${SERVER_WAIT_SECONDS:-20}"
log_info() { printf '[INFO] %s\n' "$*"; }
log_warn() { printf '[WARN] %s\n' "$*"; }
log_error() { printf '[ERROR] %s\n' "$*" >&2; }
usage() {
cat <<'USAGE'
Usage:
scripts/validate_issue_787_list_quorum.sh [--full] [--no-skip-live]
Environment:
OUT_DIR output directory (default: target/issue-787-list-quorum-<ts>)
SKIP_LIVE skip live S3 endpoint checks when true (default: true)
ENDPOINT rustfs endpoint for live checks (default: http://127.0.0.1:9000)
LIVE_LIST_BUCKET bucket used for live pagination smoke (required when running live)
LIVE_LIST_PREFIX optional prefix for list smoke (default: empty)
LIVE_LIST_MAX_KEYS max-keys for live smoke page samples (default: 1000)
LIVE_LIST_REGION signing region for curl (default: us-east-1)
MODE_LIST comma-separated list of list quorum modes (default: strict,optimal,reduced)
SERVER_RESTART_CMD optional command to restart server with RUSTFS_TUNING_ENV_FILE
SERVER_WAIT_SECONDS seconds to wait after restart command before next checks (default: 20)
Run modes:
--full or --no-skip-live run live checks
--help this help
USAGE
}
require_cmd() {
if ! command -v "$1" >/dev/null 2>&1; then
log_error "command not found: $1"
exit 1
fi
}
timestamp_ms() {
python3 - <<'PY'
import time
print(int(time.time() * 1000))
PY
}
extract_xml_keys() {
local xml_text="$1"
printf '%s\n' "$xml_text" | sed -n 's:.*<Key>\(.*\)</Key>.*:\1:p'
}
wait_for_health() {
local waited=0
while (( waited < SERVER_WAIT_SECONDS )); do
if curl -fsS "${ENDPOINT}/health/ready" >/dev/null 2>&1; then
return 0
fi
sleep 1
waited=$(( waited + 1 ))
done
return 1
}
run_unit_checks() {
log_info "Running focused rustfs-ecstore tests for #787 list quorum tuning"
mkdir -p "$OUT_DIR"
local test_log="$OUT_DIR/issue-787-ecstore-tests.log"
: > "$test_log"
local tests=(
list_path_gather_results_returns_after_limit_without_waiting_for_input_close
list_path_marker_parser_uses_trailing_cache_tag
normalize_list_quorum_accepts_supported_values
normalize_list_quorum_falls_back_to_strict
list_quorum_from_env_defaults_to_strict
list_quorum_from_env_honors_supported_value
list_objects_quorum_from_env_defaults_to_optimal
list_objects_quorum_from_env_honors_supported_value
)
local test
for test in "${tests[@]}"; do
echo "[TEST] $test" | tee -a "$test_log"
(cd "$PROJECT_ROOT" && cargo test -p rustfs-ecstore "$test" -- --nocapture | tee -a "$test_log")
done
log_info "Unit test log: $test_log"
}
run_static_checks() {
log_info "Running #787 static checks for list-quorum and list path hot path"
local static_log="$OUT_DIR/static-checks.log"
: > "$static_log"
if rg -n "ENV_API_LIST_OBJECTS_QUORUM|list_objects_quorum_from_env\(|ask_disks: list_objects_quorum_from_env\(\)" "$PROJECT_ROOT/crates/ecstore/src/store/list_objects.rs" | tee -a "$static_log"; then
log_info "Required list quorum symbols found in list_objects.rs"
else
log_error "list quorum symbols missing in list_objects.rs"
exit 1
fi
if rg -n "store_list_objects_list_merged|sets_list_objects_list_merged|set_disks_list_objects_list_path|store list_merged finished|sets list_merged finished|set_disks list_path selected listing quorum" "$PROJECT_ROOT/crates/ecstore/src/store/list_objects.rs" | tee -a "$static_log"; then
log_info "List path merged/path logging symbols found"
else
log_error "List path merged symbols missing, static guard failed"
exit 1
fi
if rg -n "index-backed|list index|list index path|list_index" "$PROJECT_ROOT/crates/ecstore/src/store/list_objects.rs" >/dev/null 2>&1; then
log_info "Index-backed scaffold symbols detected in list_objects.rs"
else
log_warn "Index-backed symbols not yet present in list_objects.rs (expected for current phase)"
fi
log_info "Static check log: $static_log"
}
run_metrics_context() {
log_info "Collecting local metric symbol index for list-quorum review"
local metric_log="$OUT_DIR/list-quorum-metrics-context.log"
if rg -n "record_stage_duration\(\"(store|sets|set_disks)_list_objects|store_list_objects_gather|set_disks_list_objects_list_path\"" "$PROJECT_ROOT/crates/ecstore/src/store/list_objects.rs" > "$metric_log"; then
log_info "Metric context saved: $metric_log"
else
log_error "No metric symbols found in list_objects.rs"
return 1
fi
}
run_live_listing_two_page_smoke() {
local mode="$1"
local log_file="$OUT_DIR/live-${mode}.log"
: > "$log_file"
if ! command -v python3 >/dev/null 2>&1; then
log_warn "python3 missing, skip live smoke"
echo "live pagination smoke: skipped (missing python3)" | tee -a "$log_file"
return 0
fi
local access_key="${RUSTFS_ACCESS_KEY:-${WARP_ACCESS_KEY:-}}"
local secret_key="${RUSTFS_SECRET_KEY:-${WARP_SECRET_KEY:-}}"
if [[ -z "$access_key" || -z "$secret_key" ]]; then
log_warn "Missing credentials, skip live smoke"
echo "live pagination smoke: skipped (missing credentials)" | tee -a "$log_file"
return 0
fi
if [[ -z "$LIVE_LIST_BUCKET" ]]; then
log_warn "LIVE_LIST_BUCKET is empty, skip live smoke"
echo "live pagination smoke: skipped (bucket not configured)" | tee -a "$log_file"
return 0
fi
if ! curl -fsS "${ENDPOINT}/health/ready" >/dev/null 2>&1; then
log_error "health check failed: ${ENDPOINT}/health/ready"
echo "live pagination smoke: failed (health check)" | tee -a "$log_file"
return 1
fi
local base_path="$LIVE_LIST_BUCKET"
if [[ -n "$LIVE_LIST_PREFIX" ]]; then
base_path="${LIVE_LIST_BUCKET}/${LIVE_LIST_PREFIX}"
fi
local base_url="${ENDPOINT}/${base_path}?list-type=2&max-keys=${LIVE_LIST_MAX_KEYS}"
local sig_target="aws:amz:${LIVE_LIST_REGION}:s3"
local start_ms page1 page2 token1 page1_key_count page2_key_count
start_ms="$(timestamp_ms)"
page1=$(curl -fsS \
--user "${access_key}:${secret_key}" \
--aws-sigv4 "$sig_target" \
-H "x-amz-content-sha256: UNSIGNED-PAYLOAD" \
"$base_url")
page1_duration_ms="$(( $(timestamp_ms) - start_ms ))"
page1_key_count=$(extract_xml_keys "$page1" | grep -cv '^$' || true)
token1=$(printf '%s' "$page1" | sed -n 's:.*<NextContinuationToken>\(.*\)</NextContinuationToken>.*:\1:p' | head -n 1)
local -a page1_keys=()
while IFS= read -r key; do
[[ -n "$key" ]] || continue
page1_keys+=("$key")
done < <(extract_xml_keys "$page1")
log_info "Mode=${mode} first page keys=${page1_key_count} duration=${page1_duration_ms}ms" | tee -a "$log_file"
if [[ -z "$token1" ]]; then
log_info "Mode=${mode} live pagination smoke: single page only" | tee -a "$log_file"
echo "live pagination smoke: pass (single page, key_count=${page1_key_count}, duration_ms=${page1_duration_ms})" | tee -a "$log_file"
return 0
fi
local encoded_token
encoded_token=$(printf '%s' "$token1" | python3 - <<'PY'
import urllib.parse,sys
print(urllib.parse.quote(sys.stdin.read().strip(), safe=""))
PY)
start_ms="$(timestamp_ms)"
page2=$(curl -fsS \
--user "${access_key}:${secret_key}" \
--aws-sigv4 "$sig_target" \
-H "x-amz-content-sha256: UNSIGNED-PAYLOAD" \
"${base_url}&continuation-token=${encoded_token}")
page2_duration_ms="$(( $(timestamp_ms) - start_ms ))"
page2_key_count=$(extract_xml_keys "$page2" | grep -cv '^$' || true)
if (( page2_key_count == 0 )); then
log_error "Mode=${mode} live smoke: page2 empty"
return 1
fi
local duplicate=0
local -a dkeys=()
local -a page2_keys=()
while IFS= read -r key; do
[[ -n "$key" ]] || continue
page2_keys+=("$key")
done < <(extract_xml_keys "$page2")
for key in "${page2_keys[@]}"; do
if printf '%s\n' "${page1_keys[@]}" | grep -Fx -- "$key" >/dev/null 2>&1; then
duplicate=1
dkeys+=("$key")
fi
done
if (( duplicate )); then
log_error "Mode=${mode} live pagination duplicates detected: ${dkeys[*]}"
echo "live pagination smoke: fail (duplicate keys)" | tee -a "$log_file"
return 1
fi
echo "Mode=${mode}" | tee -a "$log_file"
echo "first-page-count=${page1_key_count}" | tee -a "$log_file"
echo "second-page-count=${page2_key_count}" | tee -a "$log_file"
echo "page1-duration-ms=${page1_duration_ms}" | tee -a "$log_file"
echo "page2-duration-ms=${page2_duration_ms}" | tee -a "$log_file"
echo "live pagination smoke: pass (two-page continuity)" | tee -a "$log_file"
return 0
}
run_mode_series() {
if [[ -z "${MODE_LIST}" ]]; then
log_warn "MODE_LIST is empty; skip mode series"
return 0
fi
local mode
local mode_count=0
IFS=',' read -r -a modes <<< "$MODE_LIST"
for mode in "${modes[@]}"; do
[[ -z "$mode" ]] && continue
log_info "Live check for list quorum mode=${mode}"
mode=$(printf '%s' "$mode" | tr -d '[:space:]')
if [[ -n "$SERVER_RESTART_CMD" ]]; then
local mode_cmd=""
local mode_env_file=""
if [[ "$mode" == "default" ]]; then
mode_cmd="$SERVER_RESTART_CMD"
else
mode_env_file="$OUT_DIR/.issue787-${mode}.env"
printf 'RUSTFS_LIST_OBJECTS_QUORUM=%s\n' "$mode" > "$mode_env_file"
mode_cmd="RUSTFS_TUNING_ENV_FILE=${mode_env_file} $SERVER_RESTART_CMD"
fi
log_info "Restarting server for mode=${mode}"
(cd "$PROJECT_ROOT" && eval "$mode_cmd")
if ! wait_for_health; then
log_error "health check timeout during mode=${mode} restart"
return 1
fi
elif (( mode_count > 0 )); then
log_warn "SERVER_RESTART_CMD unset, mode=${mode} does not auto-switch. Skip remaining modes."
break
fi
mode_count=$(( mode_count + 1 ))
run_live_listing_two_page_smoke "$mode"
done
}
parse_args() {
while [[ $# -gt 0 ]]; do
case "$1" in
--full|--no-skip-live)
SKIP_LIVE="false"
shift
;;
-h|--help)
usage
exit 0
;;
*)
log_error "unknown arg: $1"
usage
exit 1
;;
esac
done
}
main() {
parse_args "$@"
require_cmd cargo
require_cmd rg
require_cmd tee
require_cmd curl
mkdir -p "$OUT_DIR"
run_unit_checks
run_static_checks
run_metrics_context
if [[ "$SKIP_LIVE" != "true" ]]; then
require_cmd python3
run_mode_series
fi
log_info "Validation summary:"
log_info " - logs: $OUT_DIR"
log_info " - status: pass"
}
main "$@"
@@ -0,0 +1,102 @@
#!/usr/bin/env bash
set -euo pipefail
# Acceptance runner for rustfs/backlog#841.
PROJECT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
OUT_DIR="${OUT_DIR:-${PROJECT_ROOT}/target/issue-841-acceptance-$(date +%Y%m%d-%H%M%S)}"
log_info() { printf '[INFO] %s\n' "$*"; }
log_error() { printf '[ERROR] %s\n' "$*" >&2; }
usage() {
cat <<'USAGE'
Usage:
scripts/validate_issue_841_list_objects_observability.sh
Environment:
OUT_DIR output directory (default: target/issue-841-acceptance-<ts>)
USAGE
}
require_cmd() {
if ! command -v "$1" >/dev/null 2>&1; then
log_error "command not found: $1"
exit 1
fi
}
run_unit_checks() {
log_info "Running ListObjects observability unit checks"
local test_log="$OUT_DIR/issue-841-tests.log"
: > "$test_log"
(
cd "$PROJECT_ROOT"
cargo test -p rustfs-io-metrics list_objects_metrics -- --nocapture
cargo test -p rustfs-ecstore list_objects -- --nocapture
) 2>&1 | tee "$test_log"
log_info "Unit test log: $test_log"
}
run_static_checks() {
log_info "Running ListObjects observability static checks"
local static_log="$OUT_DIR/static-checks.log"
: > "$static_log"
local required_patterns=(
"rustfs_s3_list_objects_gather_total"
"rustfs_s3_list_objects_gather_scan_amplification"
"rustfs_s3_list_objects_merge_fan_in"
"record_list_objects_gather"
"record_list_objects_merge"
"init_list_objects_metrics"
"listobjects-v2-baseline-fixtures"
)
local pattern
for pattern in "${required_patterns[@]}"; do
if rg --no-ignore -n "$pattern" "$PROJECT_ROOT/crates" "$PROJECT_ROOT/rustfs" "$PROJECT_ROOT/docs" "$PROJECT_ROOT/scripts" >> "$static_log"; then
log_info "Found required symbol: $pattern"
else
log_error "Missing required symbol: $pattern"
log_error "Static check log: $static_log"
return 1
fi
done
log_info "Static check log: $static_log"
}
parse_args() {
while [[ $# -gt 0 ]]; do
case "$1" in
-h|--help)
usage
exit 0
;;
*)
log_error "unknown arg: $1"
usage
exit 1
;;
esac
done
}
main() {
parse_args "$@"
require_cmd cargo
require_cmd rg
require_cmd tee
mkdir -p "$OUT_DIR"
run_unit_checks
run_static_checks
log_info "Validation summary:"
log_info " - logs: $OUT_DIR"
log_info " - status: pass"
}
main "$@"