mirror of
https://github.com/rustfs/rustfs.git
synced 2026-07-26 16:28:15 +00:00
houseme
0a00d8d500
* refactor(server): split internode dispatch scaffold
* test(server): cover internode dispatch prefix split
* refactor(server): name internode stack boundaries
* perf(server): skip internode request logging layer
* perf(server): skip internode trace layer
* perf(server): use lite internode request context
* feat(metrics): track internode rpc duration
* feat(ecstore): add put object stage summary logs
* test(metrics): update internode descriptor expectations
* fix(server): tighten internode path matching
* fix(pr): address review follow-up comments
* style(ecstore): simplify commit tail duration field
* refactor(ecstore): group put stage summary fields
* refactor(ecstore): inline put stage summary log
* fix(s3): return storage class for object attributes
* merge: sync latest main and resolve object attributes conflict
* fmt
* fix(server): remove duplicate rpc imports
* build(deps): bump memmap2 for RUSTSEC-2026-0186
* fix(s3select): align object_store with datafusion
* chore(deps): prune workspace dependencies
* perf(fuzz): optimize CI runtime with build/run split and matrix parallelization
Separate fuzz harness compilation from execution to eliminate redundant
builds across targets. Introduce matrix-based parallel execution for
PR smoke and nightly fuzz jobs.
Changes:
- Split CI workflow into `fuzz-build` (compile once) and matrix run jobs
(`pr-fuzz-smoke`, `nightly-fuzz-corpus`) that execute targets in parallel
- Add `BUILD_ONLY` mode to run_ci_targets.sh / run_nightly_targets.sh
- Add run_single_target.sh for matrix jobs (no build phase)
- Optimize `local_metadata` fuzz target: reduce prefix iterations from
8-10 (4 functions each) to 5 critical prefixes (parser-only), cutting
per-iteration cost by ~3-5x
- Move archive path validation (`validate_extract_relative_path`,
`normalize_extract_entry_key`) from `rustfs` to `rustfs-utils::path`,
eliminating `rustfs` binary crate dependency from fuzz harness
- Remove `rustfs` from fuzz/Cargo.toml (drops significant transitive deps)
- Add unit tests for archive path validation in rustfs-utils
- Update fuzz/README.md with new workflow and script documentation
Expected CI improvement: PR smoke wall-clock from ~120min (frequent
timeout) to ~40min; nightly from ~180min to ~60min.
* refactor(fuzz): consolidate scripts and fix prefix test alignment
Replace three duplicated shell scripts (run_ci_targets.sh,
run_nightly_targets.sh, run_single_target.sh) with a single
parameterized run.sh that supports BUILD_ONLY, SKIP_BUILD, and
MAX_TOTAL_TIME environment variables.
Fix local_metadata fuzz target prefix testing: replace always-true
'len > 0' guard with lengths aligned to xl.meta binary layout
(4/5/8/12 bytes for magic+version+header fields). Remove redundant
empty-slice test.
Hoist RUSTFLAGS to workflow top-level env to eliminate per-job
duplication. Update README with unified script documentation.
Net: -118 lines, zero functionality loss.
* perf(fuzz): optimize CI runtime with build/run split and matrix parallelization
Restructure fuzz CI workflow to eliminate redundant compilation and
run targets in parallel via matrix strategy.
Workflow changes:
- Split into fuzz-build (compile once) and matrix run jobs
- PR smoke: 3 targets parallel, 60s each, timeout 30min (was 120min)
- Nightly: 3 targets parallel, 300s each, timeout 60min (was 180min)
- Pass compiled harness via actions/artifact between jobs
- Hoist RUSTFLAGS to workflow top-level env
Script consolidation:
- Replace 3 duplicated scripts with single parameterized run.sh
- Supports BUILD_ONLY, SKIP_BUILD, MAX_TOTAL_TIME env vars
Target optimizations:
- Remove rustfs binary crate from fuzz dependencies (was pulling
979 transitive deps); move archive path validation to rustfs-utils
- Optimize local_metadata: reduce prefix iterations from 8-10x4
calls to 5 prefixes with parser-only (no decompress), aligned
with xl.meta binary layout (4/5/8/12 bytes)
- Add unit tests for archive path validation in rustfs-utils
- Update fuzz/README.md with unified script documentation
Expected: PR smoke wall-clock from ~120min (frequent timeout)
to ~40min; nightly from ~180min to ~60min.
* fix(rpc): resolve internode metrics via app context
* fmt
* ci: speed up fuzz smoke artifact restore
---------
Signed-off-by: houseme <housemecn@gmail.com>
83 lines
3.0 KiB
Bash
Executable File
83 lines
3.0 KiB
Bash
Executable File
#!/bin/sh
|
|
|
|
# Unified fuzz runner script.
|
|
#
|
|
# Modes:
|
|
# ./scripts/fuzz/run.sh # build + run all smoke targets (60s each)
|
|
# BUILD_ONLY=1 ./scripts/fuzz/run.sh # build only, no fuzz run
|
|
# FUZZ_TARGET=path_containment ./scripts/fuzz/run.sh # build + run single target
|
|
# MAX_TOTAL_TIME=300 ./scripts/fuzz/run.sh # nightly-style 300s per target
|
|
#
|
|
# Environment variables:
|
|
# FUZZ_TARGET — run only this target (default: all smoke targets)
|
|
# MAX_TOTAL_TIME — seconds to fuzz per target (default: 60)
|
|
# ARTIFACT_ROOT — artifact output directory (default: artifacts)
|
|
# BUILD_ONLY — set to 1 to skip fuzz runs (default: 0)
|
|
# SKIP_BUILD — set to 1 to skip build phase (default: 0)
|
|
# USE_PREBUILT_BINARY — set to 1 to run a prebuilt fuzz binary directly
|
|
# PREBUILT_BINARY_DIR — directory containing prebuilt fuzz binaries
|
|
|
|
set -eu
|
|
|
|
SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
|
|
REPO_ROOT=$(CDPATH= cd -- "$SCRIPT_DIR/../.." && pwd)
|
|
FUZZ_DIR="$REPO_ROOT/fuzz"
|
|
MAX_TOTAL_TIME=${MAX_TOTAL_TIME:-60}
|
|
ARTIFACT_ROOT=${ARTIFACT_ROOT:-artifacts}
|
|
FUZZ_TARGET=${FUZZ_TARGET:-}
|
|
BUILD_ONLY=${BUILD_ONLY:-0}
|
|
SKIP_BUILD=${SKIP_BUILD:-0}
|
|
USE_PREBUILT_BINARY=${USE_PREBUILT_BINARY:-0}
|
|
PREBUILT_BINARY_DIR=${PREBUILT_BINARY_DIR:-}
|
|
|
|
cd "$FUZZ_DIR"
|
|
mkdir -p "$ARTIFACT_ROOT"
|
|
|
|
targets="path_containment bucket_validation local_metadata"
|
|
if [ -n "$FUZZ_TARGET" ]; then
|
|
targets="$FUZZ_TARGET"
|
|
fi
|
|
|
|
# Phase 1: build (unless skipped)
|
|
if [ "$SKIP_BUILD" != "1" ]; then
|
|
for target in $targets; do
|
|
echo "==> cargo +nightly fuzz build $target"
|
|
cargo +nightly fuzz build "$target"
|
|
done
|
|
fi
|
|
|
|
if [ "$BUILD_ONLY" = "1" ]; then
|
|
echo "==> Build-only mode; skipping fuzz runs."
|
|
exit 0
|
|
fi
|
|
|
|
# Phase 2: run each target (incremental — no recompilation if already built)
|
|
for target in $targets; do
|
|
artifact_dir="$ARTIFACT_ROOT/$target"
|
|
corpus_dir="$FUZZ_DIR/corpus/$target"
|
|
mkdir -p "$artifact_dir"
|
|
mkdir -p "$corpus_dir"
|
|
|
|
if [ "$USE_PREBUILT_BINARY" = "1" ]; then
|
|
binary_dir="$PREBUILT_BINARY_DIR"
|
|
if [ -z "$binary_dir" ]; then
|
|
if [ -n "${CARGO_BUILD_TARGET:-}" ]; then
|
|
binary_dir="$FUZZ_DIR/target/${CARGO_BUILD_TARGET}/release"
|
|
else
|
|
binary_dir="$FUZZ_DIR/target/release"
|
|
fi
|
|
fi
|
|
binary_path="$binary_dir/$target"
|
|
if [ ! -x "$binary_path" ]; then
|
|
echo "Missing executable prebuilt fuzz binary: $binary_path" >&2
|
|
exit 1
|
|
fi
|
|
echo "==> $binary_path (-max_total_time=$MAX_TOTAL_TIME, -artifact_prefix=$artifact_dir/, corpus=$corpus_dir)"
|
|
"$binary_path" -max_total_time="$MAX_TOTAL_TIME" -artifact_prefix="$artifact_dir/" "$corpus_dir"
|
|
continue
|
|
fi
|
|
|
|
echo "==> cargo +nightly fuzz run $target (-max_total_time=$MAX_TOTAL_TIME, -artifact_prefix=$artifact_dir/)"
|
|
cargo +nightly fuzz run "$target" -- -max_total_time="$MAX_TOTAL_TIME" -artifact_prefix="$artifact_dir/"
|
|
done
|