mirror of
https://github.com/rustfs/rustfs.git
synced 2026-07-26 08:18:18 +00:00
houseme
0a00d8d500
* refactor(server): split internode dispatch scaffold
* test(server): cover internode dispatch prefix split
* refactor(server): name internode stack boundaries
* perf(server): skip internode request logging layer
* perf(server): skip internode trace layer
* perf(server): use lite internode request context
* feat(metrics): track internode rpc duration
* feat(ecstore): add put object stage summary logs
* test(metrics): update internode descriptor expectations
* fix(server): tighten internode path matching
* fix(pr): address review follow-up comments
* style(ecstore): simplify commit tail duration field
* refactor(ecstore): group put stage summary fields
* refactor(ecstore): inline put stage summary log
* fix(s3): return storage class for object attributes
* merge: sync latest main and resolve object attributes conflict
* fmt
* fix(server): remove duplicate rpc imports
* build(deps): bump memmap2 for RUSTSEC-2026-0186
* fix(s3select): align object_store with datafusion
* chore(deps): prune workspace dependencies
* perf(fuzz): optimize CI runtime with build/run split and matrix parallelization
Separate fuzz harness compilation from execution to eliminate redundant
builds across targets. Introduce matrix-based parallel execution for
PR smoke and nightly fuzz jobs.
Changes:
- Split CI workflow into `fuzz-build` (compile once) and matrix run jobs
(`pr-fuzz-smoke`, `nightly-fuzz-corpus`) that execute targets in parallel
- Add `BUILD_ONLY` mode to run_ci_targets.sh / run_nightly_targets.sh
- Add run_single_target.sh for matrix jobs (no build phase)
- Optimize `local_metadata` fuzz target: reduce prefix iterations from
8-10 (4 functions each) to 5 critical prefixes (parser-only), cutting
per-iteration cost by ~3-5x
- Move archive path validation (`validate_extract_relative_path`,
`normalize_extract_entry_key`) from `rustfs` to `rustfs-utils::path`,
eliminating `rustfs` binary crate dependency from fuzz harness
- Remove `rustfs` from fuzz/Cargo.toml (drops significant transitive deps)
- Add unit tests for archive path validation in rustfs-utils
- Update fuzz/README.md with new workflow and script documentation
Expected CI improvement: PR smoke wall-clock from ~120min (frequent
timeout) to ~40min; nightly from ~180min to ~60min.
* refactor(fuzz): consolidate scripts and fix prefix test alignment
Replace three duplicated shell scripts (run_ci_targets.sh,
run_nightly_targets.sh, run_single_target.sh) with a single
parameterized run.sh that supports BUILD_ONLY, SKIP_BUILD, and
MAX_TOTAL_TIME environment variables.
Fix local_metadata fuzz target prefix testing: replace always-true
'len > 0' guard with lengths aligned to xl.meta binary layout
(4/5/8/12 bytes for magic+version+header fields). Remove redundant
empty-slice test.
Hoist RUSTFLAGS to workflow top-level env to eliminate per-job
duplication. Update README with unified script documentation.
Net: -118 lines, zero functionality loss.
* perf(fuzz): optimize CI runtime with build/run split and matrix parallelization
Restructure fuzz CI workflow to eliminate redundant compilation and
run targets in parallel via matrix strategy.
Workflow changes:
- Split into fuzz-build (compile once) and matrix run jobs
- PR smoke: 3 targets parallel, 60s each, timeout 30min (was 120min)
- Nightly: 3 targets parallel, 300s each, timeout 60min (was 180min)
- Pass compiled harness via actions/artifact between jobs
- Hoist RUSTFLAGS to workflow top-level env
Script consolidation:
- Replace 3 duplicated scripts with single parameterized run.sh
- Supports BUILD_ONLY, SKIP_BUILD, MAX_TOTAL_TIME env vars
Target optimizations:
- Remove rustfs binary crate from fuzz dependencies (was pulling
979 transitive deps); move archive path validation to rustfs-utils
- Optimize local_metadata: reduce prefix iterations from 8-10x4
calls to 5 prefixes with parser-only (no decompress), aligned
with xl.meta binary layout (4/5/8/12 bytes)
- Add unit tests for archive path validation in rustfs-utils
- Update fuzz/README.md with unified script documentation
Expected: PR smoke wall-clock from ~120min (frequent timeout)
to ~40min; nightly from ~180min to ~60min.
* fix(rpc): resolve internode metrics via app context
* fmt
* ci: speed up fuzz smoke artifact restore
---------
Signed-off-by: houseme <housemecn@gmail.com>
94 lines
2.8 KiB
Markdown
94 lines
2.8 KiB
Markdown
# RustFS Fuzz Harness
|
|
|
|
This directory contains the standalone `cargo-fuzz` harness for RustFS.
|
|
|
|
It is intentionally isolated from the main workspace so that:
|
|
|
|
- root `cargo fmt --all`
|
|
- root `cargo nextest`
|
|
- root `make pre-commit`
|
|
|
|
continue to behave exactly as they do today.
|
|
|
|
## Prerequisites
|
|
|
|
Install `cargo-fuzz` and a nightly toolchain locally:
|
|
|
|
```bash
|
|
cargo install cargo-fuzz
|
|
rustup toolchain install nightly
|
|
```
|
|
|
|
## Layout
|
|
|
|
```text
|
|
fuzz/
|
|
Cargo.toml
|
|
fuzz_targets/
|
|
corpus/
|
|
artifacts/
|
|
```
|
|
|
|
Crash reproducers are written under `fuzz/artifacts/<target>/`.
|
|
|
|
## Targets
|
|
|
|
- `bucket_validation`
|
|
- Exercises RustFS bucket-name and bucket/object argument validation.
|
|
- `archive_extract`
|
|
- Exercises archive entry path normalization, prefix application, and bucket-namespace containment checks.
|
|
- `path_containment`
|
|
- Exercises object-path acceptance and containment invariants using public path helpers.
|
|
- `local_metadata`
|
|
- Exercises `rustfs-filemeta` metadata decoding and `rustfs-utils` block decompression.
|
|
- `policy_ingress`
|
|
- Exercises RustFS-owned bucket policy and policy-doc JSON ingress behavior, including strict-vs-tolerant unknown-field expectations.
|
|
|
|
## Usage
|
|
|
|
Run a single target:
|
|
|
|
```bash
|
|
cd fuzz
|
|
cargo +nightly fuzz run path_containment
|
|
```
|
|
|
|
Use the unified runner script from the repository root:
|
|
|
|
```bash
|
|
# Build + run all smoke targets (60s each)
|
|
./scripts/fuzz/run.sh
|
|
|
|
# Build only (no fuzz run)
|
|
BUILD_ONLY=1 ./scripts/fuzz/run.sh
|
|
|
|
# Build + run a single target
|
|
FUZZ_TARGET=path_containment ./scripts/fuzz/run.sh
|
|
|
|
# Nightly-style: 300s per target
|
|
MAX_TOTAL_TIME=300 ./scripts/fuzz/run.sh
|
|
|
|
# Skip build (use pre-built harness)
|
|
SKIP_BUILD=1 FUZZ_TARGET=local_metadata ./scripts/fuzz/run.sh
|
|
```
|
|
|
|
## CI Workflow
|
|
|
|
The GitHub Actions workflow (`.github/workflows/fuzz.yml`) uses a **build/run separation** pattern:
|
|
|
|
1. **`fuzz-build`** — compiles all fuzz targets once, then uploads only the prebuilt smoke harness binaries needed by later jobs.
|
|
2. **`pr-fuzz-smoke`** — matrix job that runs each target in parallel (60s each). Downloads the prebuilt binaries and executes them directly, so the job does not need to restore the full `fuzz/target/` tree or reinstall `cargo-fuzz`.
|
|
3. **`nightly-fuzz-corpus`** — matrix job that reuses the same prebuilt binaries and runs each target in parallel (300s each) on a daily schedule.
|
|
|
|
This design avoids redundant compilation across targets and keeps wall-clock time low.
|
|
|
|
## Seed Corpus
|
|
|
|
Initial seed directories live under `fuzz/corpus/`.
|
|
|
|
- Keep small, representative, and target-specific inputs here.
|
|
- Prefer checked-in fixtures for stable formats like `xl.meta`.
|
|
- Use `cargo fuzz cmin` / `cargo fuzz tmin` to shrink corpora and crashes before committing them.
|
|
|
|
The helper scripts also invoke `cargo +nightly fuzz ...` explicitly so local execution does not depend on the default toolchain.
|