mirror of
https://github.com/rustfs/rustfs.git
synced 2026-07-26 16:28:15 +00:00
perf(server): lighten internode data-plane stack (#3735)
* refactor(server): split internode dispatch scaffold
* test(server): cover internode dispatch prefix split
* refactor(server): name internode stack boundaries
* perf(server): skip internode request logging layer
* perf(server): skip internode trace layer
* perf(server): use lite internode request context
* feat(metrics): track internode rpc duration
* feat(ecstore): add put object stage summary logs
* test(metrics): update internode descriptor expectations
* fix(server): tighten internode path matching
* fix(pr): address review follow-up comments
* style(ecstore): simplify commit tail duration field
* refactor(ecstore): group put stage summary fields
* refactor(ecstore): inline put stage summary log
* fix(s3): return storage class for object attributes
* merge: sync latest main and resolve object attributes conflict
* fmt
* fix(server): remove duplicate rpc imports
* build(deps): bump memmap2 for RUSTSEC-2026-0186
* fix(s3select): align object_store with datafusion
* chore(deps): prune workspace dependencies
* perf(fuzz): optimize CI runtime with build/run split and matrix parallelization
Separate fuzz harness compilation from execution to eliminate redundant
builds across targets. Introduce matrix-based parallel execution for
PR smoke and nightly fuzz jobs.
Changes:
- Split CI workflow into `fuzz-build` (compile once) and matrix run jobs
(`pr-fuzz-smoke`, `nightly-fuzz-corpus`) that execute targets in parallel
- Add `BUILD_ONLY` mode to run_ci_targets.sh / run_nightly_targets.sh
- Add run_single_target.sh for matrix jobs (no build phase)
- Optimize `local_metadata` fuzz target: reduce prefix iterations from
8-10 (4 functions each) to 5 critical prefixes (parser-only), cutting
per-iteration cost by ~3-5x
- Move archive path validation (`validate_extract_relative_path`,
`normalize_extract_entry_key`) from `rustfs` to `rustfs-utils::path`,
eliminating `rustfs` binary crate dependency from fuzz harness
- Remove `rustfs` from fuzz/Cargo.toml (drops significant transitive deps)
- Add unit tests for archive path validation in rustfs-utils
- Update fuzz/README.md with new workflow and script documentation
Expected CI improvement: PR smoke wall-clock from ~120min (frequent
timeout) to ~40min; nightly from ~180min to ~60min.
* refactor(fuzz): consolidate scripts and fix prefix test alignment
Replace three duplicated shell scripts (run_ci_targets.sh,
run_nightly_targets.sh, run_single_target.sh) with a single
parameterized run.sh that supports BUILD_ONLY, SKIP_BUILD, and
MAX_TOTAL_TIME environment variables.
Fix local_metadata fuzz target prefix testing: replace always-true
'len > 0' guard with lengths aligned to xl.meta binary layout
(4/5/8/12 bytes for magic+version+header fields). Remove redundant
empty-slice test.
Hoist RUSTFLAGS to workflow top-level env to eliminate per-job
duplication. Update README with unified script documentation.
Net: -118 lines, zero functionality loss.
* perf(fuzz): optimize CI runtime with build/run split and matrix parallelization
Restructure fuzz CI workflow to eliminate redundant compilation and
run targets in parallel via matrix strategy.
Workflow changes:
- Split into fuzz-build (compile once) and matrix run jobs
- PR smoke: 3 targets parallel, 60s each, timeout 30min (was 120min)
- Nightly: 3 targets parallel, 300s each, timeout 60min (was 180min)
- Pass compiled harness via actions/artifact between jobs
- Hoist RUSTFLAGS to workflow top-level env
Script consolidation:
- Replace 3 duplicated scripts with single parameterized run.sh
- Supports BUILD_ONLY, SKIP_BUILD, MAX_TOTAL_TIME env vars
Target optimizations:
- Remove rustfs binary crate from fuzz dependencies (was pulling
979 transitive deps); move archive path validation to rustfs-utils
- Optimize local_metadata: reduce prefix iterations from 8-10x4
calls to 5 prefixes with parser-only (no decompress), aligned
with xl.meta binary layout (4/5/8/12 bytes)
- Add unit tests for archive path validation in rustfs-utils
- Update fuzz/README.md with unified script documentation
Expected: PR smoke wall-clock from ~120min (frequent timeout)
to ~40min; nightly from ~180min to ~60min.
* fix(rpc): resolve internode metrics via app context
* fmt
* ci: speed up fuzz smoke artifact restore
---------
Signed-off-by: houseme <housemecn@gmail.com>
This commit is contained in:
houseme
Executable
+82
@@ -0,0 +1,82 @@
|
||||
#!/bin/sh
|
||||
|
||||
# Unified fuzz runner script.
|
||||
#
|
||||
# Modes:
|
||||
# ./scripts/fuzz/run.sh # build + run all smoke targets (60s each)
|
||||
# BUILD_ONLY=1 ./scripts/fuzz/run.sh # build only, no fuzz run
|
||||
# FUZZ_TARGET=path_containment ./scripts/fuzz/run.sh # build + run single target
|
||||
# MAX_TOTAL_TIME=300 ./scripts/fuzz/run.sh # nightly-style 300s per target
|
||||
#
|
||||
# Environment variables:
|
||||
# FUZZ_TARGET — run only this target (default: all smoke targets)
|
||||
# MAX_TOTAL_TIME — seconds to fuzz per target (default: 60)
|
||||
# ARTIFACT_ROOT — artifact output directory (default: artifacts)
|
||||
# BUILD_ONLY — set to 1 to skip fuzz runs (default: 0)
|
||||
# SKIP_BUILD — set to 1 to skip build phase (default: 0)
|
||||
# USE_PREBUILT_BINARY — set to 1 to run a prebuilt fuzz binary directly
|
||||
# PREBUILT_BINARY_DIR — directory containing prebuilt fuzz binaries
|
||||
|
||||
set -eu
|
||||
|
||||
SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
|
||||
REPO_ROOT=$(CDPATH= cd -- "$SCRIPT_DIR/../.." && pwd)
|
||||
FUZZ_DIR="$REPO_ROOT/fuzz"
|
||||
MAX_TOTAL_TIME=${MAX_TOTAL_TIME:-60}
|
||||
ARTIFACT_ROOT=${ARTIFACT_ROOT:-artifacts}
|
||||
FUZZ_TARGET=${FUZZ_TARGET:-}
|
||||
BUILD_ONLY=${BUILD_ONLY:-0}
|
||||
SKIP_BUILD=${SKIP_BUILD:-0}
|
||||
USE_PREBUILT_BINARY=${USE_PREBUILT_BINARY:-0}
|
||||
PREBUILT_BINARY_DIR=${PREBUILT_BINARY_DIR:-}
|
||||
|
||||
cd "$FUZZ_DIR"
|
||||
mkdir -p "$ARTIFACT_ROOT"
|
||||
|
||||
targets="path_containment bucket_validation local_metadata"
|
||||
if [ -n "$FUZZ_TARGET" ]; then
|
||||
targets="$FUZZ_TARGET"
|
||||
fi
|
||||
|
||||
# Phase 1: build (unless skipped)
|
||||
if [ "$SKIP_BUILD" != "1" ]; then
|
||||
for target in $targets; do
|
||||
echo "==> cargo +nightly fuzz build $target"
|
||||
cargo +nightly fuzz build "$target"
|
||||
done
|
||||
fi
|
||||
|
||||
if [ "$BUILD_ONLY" = "1" ]; then
|
||||
echo "==> Build-only mode; skipping fuzz runs."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Phase 2: run each target (incremental — no recompilation if already built)
|
||||
for target in $targets; do
|
||||
artifact_dir="$ARTIFACT_ROOT/$target"
|
||||
corpus_dir="$FUZZ_DIR/corpus/$target"
|
||||
mkdir -p "$artifact_dir"
|
||||
mkdir -p "$corpus_dir"
|
||||
|
||||
if [ "$USE_PREBUILT_BINARY" = "1" ]; then
|
||||
binary_dir="$PREBUILT_BINARY_DIR"
|
||||
if [ -z "$binary_dir" ]; then
|
||||
if [ -n "${CARGO_BUILD_TARGET:-}" ]; then
|
||||
binary_dir="$FUZZ_DIR/target/${CARGO_BUILD_TARGET}/release"
|
||||
else
|
||||
binary_dir="$FUZZ_DIR/target/release"
|
||||
fi
|
||||
fi
|
||||
binary_path="$binary_dir/$target"
|
||||
if [ ! -x "$binary_path" ]; then
|
||||
echo "Missing executable prebuilt fuzz binary: $binary_path" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "==> $binary_path (-max_total_time=$MAX_TOTAL_TIME, -artifact_prefix=$artifact_dir/, corpus=$corpus_dir)"
|
||||
"$binary_path" -max_total_time="$MAX_TOTAL_TIME" -artifact_prefix="$artifact_dir/" "$corpus_dir"
|
||||
continue
|
||||
fi
|
||||
|
||||
echo "==> cargo +nightly fuzz run $target (-max_total_time=$MAX_TOTAL_TIME, -artifact_prefix=$artifact_dir/)"
|
||||
cargo +nightly fuzz run "$target" -- -max_total_time="$MAX_TOTAL_TIME" -artifact_prefix="$artifact_dir/"
|
||||
done
|
||||
@@ -1,25 +0,0 @@
|
||||
#!/bin/sh
|
||||
|
||||
set -eu
|
||||
|
||||
SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
|
||||
REPO_ROOT=$(CDPATH= cd -- "$SCRIPT_DIR/../.." && pwd)
|
||||
FUZZ_DIR="$REPO_ROOT/fuzz"
|
||||
MAX_TOTAL_TIME=${MAX_TOTAL_TIME:-60}
|
||||
ARTIFACT_ROOT=${ARTIFACT_ROOT:-artifacts}
|
||||
FUZZ_TARGET=${FUZZ_TARGET:-}
|
||||
|
||||
cd "$FUZZ_DIR"
|
||||
mkdir -p "$ARTIFACT_ROOT"
|
||||
|
||||
targets="path_containment bucket_validation local_metadata"
|
||||
if [ -n "$FUZZ_TARGET" ]; then
|
||||
targets="$FUZZ_TARGET"
|
||||
fi
|
||||
|
||||
for target in $targets; do
|
||||
artifact_dir="$ARTIFACT_ROOT/$target"
|
||||
mkdir -p "$artifact_dir"
|
||||
echo "==> cargo +nightly fuzz run $target (-max_total_time=$MAX_TOTAL_TIME, -artifact_prefix=$artifact_dir/)"
|
||||
cargo +nightly fuzz run "$target" -- -max_total_time="$MAX_TOTAL_TIME" -artifact_prefix="$artifact_dir/"
|
||||
done
|
||||
@@ -1,25 +0,0 @@
|
||||
#!/bin/sh
|
||||
|
||||
set -eu
|
||||
|
||||
SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
|
||||
REPO_ROOT=$(CDPATH= cd -- "$SCRIPT_DIR/../.." && pwd)
|
||||
FUZZ_DIR="$REPO_ROOT/fuzz"
|
||||
MAX_TOTAL_TIME=${MAX_TOTAL_TIME:-300}
|
||||
ARTIFACT_ROOT=${ARTIFACT_ROOT:-artifacts}
|
||||
FUZZ_TARGET=${FUZZ_TARGET:-}
|
||||
|
||||
cd "$FUZZ_DIR"
|
||||
mkdir -p "$ARTIFACT_ROOT"
|
||||
|
||||
targets="path_containment bucket_validation local_metadata"
|
||||
if [ -n "$FUZZ_TARGET" ]; then
|
||||
targets="$FUZZ_TARGET"
|
||||
fi
|
||||
|
||||
for target in $targets; do
|
||||
artifact_dir="$ARTIFACT_ROOT/$target"
|
||||
mkdir -p "$artifact_dir"
|
||||
echo "==> cargo +nightly fuzz run $target (-max_total_time=$MAX_TOTAL_TIME, -artifact_prefix=$artifact_dir/)"
|
||||
cargo +nightly fuzz run "$target" -- -max_total_time="$MAX_TOTAL_TIME" -artifact_prefix="$artifact_dir/"
|
||||
done
|
||||
Reference in New Issue
Block a user