mirror of
https://github.com/rustfs/rustfs.git
synced 2026-07-26 16:28:15 +00:00
0a00d8d500
* refactor(server): split internode dispatch scaffold
* test(server): cover internode dispatch prefix split
* refactor(server): name internode stack boundaries
* perf(server): skip internode request logging layer
* perf(server): skip internode trace layer
* perf(server): use lite internode request context
* feat(metrics): track internode rpc duration
* feat(ecstore): add put object stage summary logs
* test(metrics): update internode descriptor expectations
* fix(server): tighten internode path matching
* fix(pr): address review follow-up comments
* style(ecstore): simplify commit tail duration field
* refactor(ecstore): group put stage summary fields
* refactor(ecstore): inline put stage summary log
* fix(s3): return storage class for object attributes
* merge: sync latest main and resolve object attributes conflict
* fmt
* fix(server): remove duplicate rpc imports
* build(deps): bump memmap2 for RUSTSEC-2026-0186
* fix(s3select): align object_store with datafusion
* chore(deps): prune workspace dependencies
* perf(fuzz): optimize CI runtime with build/run split and matrix parallelization
Separate fuzz harness compilation from execution to eliminate redundant
builds across targets. Introduce matrix-based parallel execution for
PR smoke and nightly fuzz jobs.
Changes:
- Split CI workflow into `fuzz-build` (compile once) and matrix run jobs
(`pr-fuzz-smoke`, `nightly-fuzz-corpus`) that execute targets in parallel
- Add `BUILD_ONLY` mode to run_ci_targets.sh / run_nightly_targets.sh
- Add run_single_target.sh for matrix jobs (no build phase)
- Optimize `local_metadata` fuzz target: reduce prefix iterations from
8-10 (4 functions each) to 5 critical prefixes (parser-only), cutting
per-iteration cost by ~3-5x
- Move archive path validation (`validate_extract_relative_path`,
`normalize_extract_entry_key`) from `rustfs` to `rustfs-utils::path`,
eliminating `rustfs` binary crate dependency from fuzz harness
- Remove `rustfs` from fuzz/Cargo.toml (drops significant transitive deps)
- Add unit tests for archive path validation in rustfs-utils
- Update fuzz/README.md with new workflow and script documentation
Expected CI improvement: PR smoke wall-clock from ~120min (frequent
timeout) to ~40min; nightly from ~180min to ~60min.
* refactor(fuzz): consolidate scripts and fix prefix test alignment
Replace three duplicated shell scripts (run_ci_targets.sh,
run_nightly_targets.sh, run_single_target.sh) with a single
parameterized run.sh that supports BUILD_ONLY, SKIP_BUILD, and
MAX_TOTAL_TIME environment variables.
Fix local_metadata fuzz target prefix testing: replace always-true
'len > 0' guard with lengths aligned to xl.meta binary layout
(4/5/8/12 bytes for magic+version+header fields). Remove redundant
empty-slice test.
Hoist RUSTFLAGS to workflow top-level env to eliminate per-job
duplication. Update README with unified script documentation.
Net: -118 lines, zero functionality loss.
* perf(fuzz): optimize CI runtime with build/run split and matrix parallelization
Restructure fuzz CI workflow to eliminate redundant compilation and
run targets in parallel via matrix strategy.
Workflow changes:
- Split into fuzz-build (compile once) and matrix run jobs
- PR smoke: 3 targets parallel, 60s each, timeout 30min (was 120min)
- Nightly: 3 targets parallel, 300s each, timeout 60min (was 180min)
- Pass compiled harness via actions/artifact between jobs
- Hoist RUSTFLAGS to workflow top-level env
Script consolidation:
- Replace 3 duplicated scripts with single parameterized run.sh
- Supports BUILD_ONLY, SKIP_BUILD, MAX_TOTAL_TIME env vars
Target optimizations:
- Remove rustfs binary crate from fuzz dependencies (was pulling
979 transitive deps); move archive path validation to rustfs-utils
- Optimize local_metadata: reduce prefix iterations from 8-10x4
calls to 5 prefixes with parser-only (no decompress), aligned
with xl.meta binary layout (4/5/8/12 bytes)
- Add unit tests for archive path validation in rustfs-utils
- Update fuzz/README.md with unified script documentation
Expected: PR smoke wall-clock from ~120min (frequent timeout)
to ~40min; nightly from ~180min to ~60min.
* fix(rpc): resolve internode metrics via app context
* fmt
* ci: speed up fuzz smoke artifact restore
---------
Signed-off-by: houseme <housemecn@gmail.com>
207 lines
7.6 KiB
YAML
207 lines
7.6 KiB
YAML
# Copyright 2024 RustFS Team
|
|
#
|
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
# you may not use this file except in compliance with the License.
|
|
# You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing, software
|
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
# See the License for the specific language governing permissions and
|
|
# limitations under the License.
|
|
|
|
name: Fuzz
|
|
|
|
on:
|
|
pull_request:
|
|
paths:
|
|
- "fuzz/**"
|
|
- "scripts/fuzz/**"
|
|
- ".github/workflows/fuzz.yml"
|
|
- "crates/ecstore/**"
|
|
- "crates/filemeta/**"
|
|
- "crates/utils/**"
|
|
schedule:
|
|
- cron: "0 2 * * *"
|
|
workflow_dispatch:
|
|
inputs:
|
|
profile:
|
|
description: "Which fuzz profile to run"
|
|
required: false
|
|
default: "both"
|
|
type: choice
|
|
options:
|
|
- smoke
|
|
- nightly
|
|
- both
|
|
|
|
permissions:
|
|
contents: read
|
|
|
|
concurrency:
|
|
group: ${{ github.workflow }}-${{ github.ref }}
|
|
cancel-in-progress: true
|
|
|
|
env:
|
|
CARGO_TERM_COLOR: always
|
|
CARGO_BUILD_TARGET: x86_64-unknown-linux-gnu
|
|
RUSTFLAGS: "--cfg tokio_unstable -C target-feature=-crt-static"
|
|
|
|
jobs:
|
|
# ──────────────────────────────────────────────────────────────
|
|
# Phase 1: Build all fuzz harness binaries once.
|
|
# ──────────────────────────────────────────────────────────────
|
|
fuzz-build:
|
|
name: Build Fuzz Harness
|
|
if: >
|
|
github.event_name == 'pull_request' ||
|
|
github.event_name == 'schedule' ||
|
|
github.event_name == 'workflow_dispatch'
|
|
runs-on: ubicloud-standard-4
|
|
timeout-minutes: 45
|
|
env:
|
|
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
|
|
steps:
|
|
- name: Checkout repository
|
|
uses: actions/checkout@v7
|
|
|
|
- name: Setup Rust environment
|
|
uses: ./.github/actions/setup
|
|
with:
|
|
rust-version: nightly
|
|
cache-shared-key: fuzz-${{ hashFiles('fuzz/Cargo.lock') }}
|
|
github-token: ${{ secrets.GITHUB_TOKEN }}
|
|
cache-save-if: ${{ github.ref == 'refs/heads/main' || github.event_name == 'schedule' }}
|
|
|
|
- name: Install cargo-fuzz
|
|
uses: taiki-e/install-action@v2
|
|
with:
|
|
tool: cargo-fuzz
|
|
|
|
- name: Build all fuzz targets
|
|
run: BUILD_ONLY=1 ./scripts/fuzz/run.sh
|
|
|
|
- name: Stage prebuilt fuzz binaries
|
|
run: |
|
|
binary_root="fuzz/prebuilt/${CARGO_BUILD_TARGET}/release"
|
|
mkdir -p "${binary_root}"
|
|
for target in path_containment bucket_validation local_metadata; do
|
|
install -Dm755 "fuzz/target/${CARGO_BUILD_TARGET}/release/${target}" "${binary_root}/${target}"
|
|
done
|
|
|
|
- name: Upload prebuilt fuzz binaries
|
|
uses: actions/upload-artifact@v6
|
|
with:
|
|
name: fuzz-prebuilt-binaries-${{ github.run_number }}
|
|
path: |
|
|
fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release/path_containment
|
|
fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release/bucket_validation
|
|
fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release/local_metadata
|
|
if-no-files-found: error
|
|
retention-days: 1
|
|
compression-level: 0
|
|
|
|
# ──────────────────────────────────────────────────────────────
|
|
# Phase 2 (PR): Run smoke targets in parallel via matrix.
|
|
# ──────────────────────────────────────────────────────────────
|
|
pr-fuzz-smoke:
|
|
name: "Smoke / ${{ matrix.target }}"
|
|
needs: fuzz-build
|
|
if: >
|
|
github.event_name == 'pull_request' ||
|
|
(github.event_name == 'workflow_dispatch' &&
|
|
(github.event.inputs.profile == 'smoke' || github.event.inputs.profile == 'both'))
|
|
runs-on: ubicloud-standard-4
|
|
timeout-minutes: 30
|
|
strategy:
|
|
fail-fast: false
|
|
matrix:
|
|
target: [path_containment, bucket_validation, local_metadata]
|
|
env:
|
|
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
|
|
steps:
|
|
- name: Checkout repository
|
|
uses: actions/checkout@v7
|
|
|
|
- name: Download prebuilt fuzz binaries
|
|
uses: actions/download-artifact@v7
|
|
with:
|
|
name: fuzz-prebuilt-binaries-${{ github.run_number }}
|
|
path: fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release
|
|
|
|
- name: Restore executable bit
|
|
run: chmod +x "fuzz/prebuilt/${CARGO_BUILD_TARGET}/release/${{ matrix.target }}"
|
|
|
|
- name: Run ${{ matrix.target }}
|
|
env:
|
|
FUZZ_TARGET: ${{ matrix.target }}
|
|
MAX_TOTAL_TIME: 60
|
|
SKIP_BUILD: 1
|
|
USE_PREBUILT_BINARY: 1
|
|
PREBUILT_BINARY_DIR: ${{ github.workspace }}/fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release
|
|
run: ./scripts/fuzz/run.sh
|
|
|
|
- name: Upload fuzz smoke artifacts
|
|
if: always()
|
|
uses: actions/upload-artifact@v6
|
|
with:
|
|
name: fuzz-smoke-${{ matrix.target }}-${{ github.run_number }}
|
|
path: |
|
|
fuzz/artifacts/**
|
|
fuzz/corpus/${{ matrix.target }}/**
|
|
if-no-files-found: ignore
|
|
retention-days: 7
|
|
|
|
# ──────────────────────────────────────────────────────────────
|
|
# Phase 2 (Nightly): Run all targets in parallel via matrix.
|
|
# ──────────────────────────────────────────────────────────────
|
|
nightly-fuzz-corpus:
|
|
name: "Nightly / ${{ matrix.target }}"
|
|
needs: fuzz-build
|
|
if: >
|
|
github.event_name == 'schedule' ||
|
|
(github.event_name == 'workflow_dispatch' &&
|
|
(github.event.inputs.profile == 'nightly' || github.event.inputs.profile == 'both'))
|
|
runs-on: ubicloud-standard-4
|
|
timeout-minutes: 60
|
|
strategy:
|
|
fail-fast: false
|
|
matrix:
|
|
target: [path_containment, bucket_validation, local_metadata]
|
|
env:
|
|
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
|
|
steps:
|
|
- name: Checkout repository
|
|
uses: actions/checkout@v7
|
|
|
|
- name: Download prebuilt fuzz binaries
|
|
uses: actions/download-artifact@v7
|
|
with:
|
|
name: fuzz-prebuilt-binaries-${{ github.run_number }}
|
|
path: fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release
|
|
|
|
- name: Restore executable bit
|
|
run: chmod +x "fuzz/prebuilt/${CARGO_BUILD_TARGET}/release/${{ matrix.target }}"
|
|
|
|
- name: Run ${{ matrix.target }}
|
|
env:
|
|
FUZZ_TARGET: ${{ matrix.target }}
|
|
MAX_TOTAL_TIME: 300
|
|
SKIP_BUILD: 1
|
|
USE_PREBUILT_BINARY: 1
|
|
PREBUILT_BINARY_DIR: ${{ github.workspace }}/fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release
|
|
run: ./scripts/fuzz/run.sh
|
|
|
|
- name: Upload nightly fuzz artifacts
|
|
if: always()
|
|
uses: actions/upload-artifact@v6
|
|
with:
|
|
name: fuzz-nightly-${{ matrix.target }}-${{ github.run_number }}
|
|
path: |
|
|
fuzz/artifacts/**
|
|
fuzz/corpus/${{ matrix.target }}/**
|
|
if-no-files-found: ignore
|
|
retention-days: 30
|