gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-07-27 00:38:16 +00:00
Code Issues Packages Projects Releases Wiki Activity

perf(server): lighten internode data-plane stack (#3735)

Browse Source 
* refactor(server): split internode dispatch scaffold

* test(server): cover internode dispatch prefix split

* refactor(server): name internode stack boundaries

* perf(server): skip internode request logging layer

* perf(server): skip internode trace layer

* perf(server): use lite internode request context

* feat(metrics): track internode rpc duration

* feat(ecstore): add put object stage summary logs

* test(metrics): update internode descriptor expectations

* fix(server): tighten internode path matching

* fix(pr): address review follow-up comments

* style(ecstore): simplify commit tail duration field

* refactor(ecstore): group put stage summary fields

* refactor(ecstore): inline put stage summary log

* fix(s3): return storage class for object attributes

* merge: sync latest main and resolve object attributes conflict

* fmt

* fix(server): remove duplicate rpc imports

* build(deps): bump memmap2 for RUSTSEC-2026-0186

* fix(s3select): align object_store with datafusion

* chore(deps): prune workspace dependencies

* perf(fuzz): optimize CI runtime with build/run split and matrix parallelization

  Separate fuzz harness compilation from execution to eliminate redundant
  builds across targets. Introduce matrix-based parallel execution for
  PR smoke and nightly fuzz jobs.

  Changes:

  - Split CI workflow into `fuzz-build` (compile once) and matrix run jobs
    (`pr-fuzz-smoke`, `nightly-fuzz-corpus`) that execute targets in parallel
  - Add `BUILD_ONLY` mode to run_ci_targets.sh / run_nightly_targets.sh
  - Add run_single_target.sh for matrix jobs (no build phase)
  - Optimize `local_metadata` fuzz target: reduce prefix iterations from
    8-10 (4 functions each) to 5 critical prefixes (parser-only), cutting
    per-iteration cost by ~3-5x
  - Move archive path validation (`validate_extract_relative_path`,
    `normalize_extract_entry_key`) from `rustfs` to `rustfs-utils::path`,
    eliminating `rustfs` binary crate dependency from fuzz harness
  - Remove `rustfs` from fuzz/Cargo.toml (drops significant transitive deps)
  - Add unit tests for archive path validation in rustfs-utils
  - Update fuzz/README.md with new workflow and script documentation

  Expected CI improvement: PR smoke wall-clock from ~120min (frequent
  timeout) to ~40min; nightly from ~180min to ~60min.

* refactor(fuzz): consolidate scripts and fix prefix test alignment

Replace three duplicated shell scripts (run_ci_targets.sh,
run_nightly_targets.sh, run_single_target.sh) with a single
parameterized run.sh that supports BUILD_ONLY, SKIP_BUILD, and
MAX_TOTAL_TIME environment variables.

Fix local_metadata fuzz target prefix testing: replace always-true
'len > 0' guard with lengths aligned to xl.meta binary layout
(4/5/8/12 bytes for magic+version+header fields). Remove redundant
empty-slice test.

Hoist RUSTFLAGS to workflow top-level env to eliminate per-job
duplication. Update README with unified script documentation.

Net: -118 lines, zero functionality loss.

* perf(fuzz): optimize CI runtime with build/run split and matrix parallelization

Restructure fuzz CI workflow to eliminate redundant compilation and
run targets in parallel via matrix strategy.

Workflow changes:
- Split into fuzz-build (compile once) and matrix run jobs
- PR smoke: 3 targets parallel, 60s each, timeout 30min (was 120min)
- Nightly: 3 targets parallel, 300s each, timeout 60min (was 180min)
- Pass compiled harness via actions/artifact between jobs
- Hoist RUSTFLAGS to workflow top-level env

Script consolidation:
- Replace 3 duplicated scripts with single parameterized run.sh
- Supports BUILD_ONLY, SKIP_BUILD, MAX_TOTAL_TIME env vars

Target optimizations:
- Remove rustfs binary crate from fuzz dependencies (was pulling
  979 transitive deps); move archive path validation to rustfs-utils
- Optimize local_metadata: reduce prefix iterations from 8-10x4
  calls to 5 prefixes with parser-only (no decompress), aligned
  with xl.meta binary layout (4/5/8/12 bytes)
- Add unit tests for archive path validation in rustfs-utils
- Update fuzz/README.md with unified script documentation

Expected: PR smoke wall-clock from ~120min (frequent timeout)
to ~40min; nightly from ~180min to ~60min.

* fix(rpc): resolve internode metrics via app context

* fmt

* ci: speed up fuzz smoke artifact restore

---------

Signed-off-by: houseme <housemecn@gmail.com>
This commit is contained in:
houseme
2026-06-23 21:36:39 +08:00
committed by GitHub
parent 7bdb25ae9d
commit 0a00d8d500
23 changed files with 1103 additions and 5816 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/fuzz.yml
+105 -53
View File
@@ -47,108 +47,160 @@ concurrency:
env:
CARGO_TERM_COLOR: always
CARGO_BUILD_TARGET: x86_64-unknown-linux-gnu
RUSTFLAGS: "--cfg tokio_unstable -C target-feature=-crt-static"
jobs:
# ──────────────────────────────────────────────────────────────
# Phase 1: Build all fuzz harness binaries once.
# ──────────────────────────────────────────────────────────────
fuzz-build:
name: Build Fuzz Harness
if: >
github.event_name == 'pull_request' ||
github.event_name == 'schedule' ||
github.event_name == 'workflow_dispatch'
runs-on: ubicloud-standard-4
timeout-minutes: 45
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@v7
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: nightly
cache-shared-key: fuzz-${{ hashFiles('fuzz/Cargo.lock') }}
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' || github.event_name == 'schedule' }}
- name: Install cargo-fuzz
uses: taiki-e/install-action@v2
with:
tool: cargo-fuzz
- name: Build all fuzz targets
run: BUILD_ONLY=1 ./scripts/fuzz/run.sh
- name: Stage prebuilt fuzz binaries
run: |
binary_root="fuzz/prebuilt/${CARGO_BUILD_TARGET}/release"
mkdir -p "${binary_root}"
for target in path_containment bucket_validation local_metadata; do
install -Dm755 "fuzz/target/${CARGO_BUILD_TARGET}/release/${target}" "${binary_root}/${target}"
done
- name: Upload prebuilt fuzz binaries
uses: actions/upload-artifact@v6
with:
name: fuzz-prebuilt-binaries-${{ github.run_number }}
path: |
fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release/path_containment
fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release/bucket_validation
fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release/local_metadata
if-no-files-found: error
retention-days: 1
compression-level: 0
# ──────────────────────────────────────────────────────────────
# Phase 2 (PR): Run smoke targets in parallel via matrix.
# ──────────────────────────────────────────────────────────────
pr-fuzz-smoke:
name: PR Fuzz Smoke
name: "Smoke / ${{ matrix.target }}"
needs: fuzz-build
if: >
github.event_name == 'pull_request' ||
(github.event_name == 'workflow_dispatch' &&
(github.event.inputs.profile == 'smoke' || github.event.inputs.profile == 'both'))
runs-on: ubicloud-standard-4
timeout-minutes: 120
timeout-minutes: 30
strategy:
fail-fast: false
matrix:
target: [path_containment, bucket_validation, local_metadata]
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
MAX_TOTAL_TIME: 60
ARTIFACT_ROOT: artifacts
RUSTFLAGS: "--cfg tokio_unstable -C target-feature=-crt-static"
steps:
- name: Checkout repository
uses: actions/checkout@v7
- name: Setup Rust environment
uses: ./.github/actions/setup
- name: Download prebuilt fuzz binaries
uses: actions/download-artifact@v7
with:
rust-version: nightly
cache-shared-key: fuzz-smoke-${{ hashFiles('fuzz/Cargo.lock') }}
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
name: fuzz-prebuilt-binaries-${{ github.run_number }}
path: fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release
- name: Install cargo-fuzz
uses: taiki-e/install-action@v2
with:
tool: cargo-fuzz
- name: Restore executable bit
run: chmod +x "fuzz/prebuilt/${CARGO_BUILD_TARGET}/release/${{ matrix.target }}"
- name: Run path_containment smoke target
- name: Run ${{ matrix.target }}
env:
FUZZ_TARGET: path_containment
run: ./scripts/fuzz/run_ci_targets.sh
- name: Run bucket_validation smoke target
env:
FUZZ_TARGET: bucket_validation
run: ./scripts/fuzz/run_ci_targets.sh
- name: Run local_metadata smoke target
env:
FUZZ_TARGET: local_metadata
run: ./scripts/fuzz/run_ci_targets.sh
- name: Run bounded fuzz smoke targets
if: ${{ false }}
run: ./scripts/fuzz/run_ci_targets.sh
FUZZ_TARGET: ${{ matrix.target }}
MAX_TOTAL_TIME: 60
SKIP_BUILD: 1
USE_PREBUILT_BINARY: 1
PREBUILT_BINARY_DIR: ${{ github.workspace }}/fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release
run: ./scripts/fuzz/run.sh
- name: Upload fuzz smoke artifacts
if: always()
uses: actions/upload-artifact@v6
with:
name: fuzz-smoke-${{ github.run_number }}
name: fuzz-smoke-${{ matrix.target }}-${{ github.run_number }}
path: |
fuzz/artifacts/**
fuzz/corpus/**
fuzz/corpus/${{ matrix.target }}/**
if-no-files-found: ignore
retention-days: 7
# ──────────────────────────────────────────────────────────────
# Phase 2 (Nightly): Run all targets in parallel via matrix.
# ──────────────────────────────────────────────────────────────
nightly-fuzz-corpus:
name: Nightly Fuzz Corpus
name: "Nightly / ${{ matrix.target }}"
needs: fuzz-build
if: >
github.event_name == 'schedule' ||
(github.event_name == 'workflow_dispatch' &&
(github.event.inputs.profile == 'nightly' || github.event.inputs.profile == 'both'))
runs-on: ubicloud-standard-4
timeout-minutes: 180
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
target: [path_containment, bucket_validation, local_metadata]
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
MAX_TOTAL_TIME: 300
ARTIFACT_ROOT: artifacts
RUSTFLAGS: "--cfg tokio_unstable -C target-feature=-crt-static"
steps:
- name: Checkout repository
uses: actions/checkout@v7
- name: Setup Rust environment
uses: ./.github/actions/setup
- name: Download prebuilt fuzz binaries
uses: actions/download-artifact@v7
with:
rust-version: nightly
cache-shared-key: fuzz-nightly-${{ hashFiles('fuzz/Cargo.lock') }}
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
name: fuzz-prebuilt-binaries-${{ github.run_number }}
path: fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release
- name: Install cargo-fuzz
uses: taiki-e/install-action@v2
with:
tool: cargo-fuzz
- name: Restore executable bit
run: chmod +x "fuzz/prebuilt/${CARGO_BUILD_TARGET}/release/${{ matrix.target }}"
- name: Run nightly fuzz targets
run: ./scripts/fuzz/run_nightly_targets.sh
- name: Run ${{ matrix.target }}
env:
FUZZ_TARGET: ${{ matrix.target }}
MAX_TOTAL_TIME: 300
SKIP_BUILD: 1
USE_PREBUILT_BINARY: 1
PREBUILT_BINARY_DIR: ${{ github.workspace }}/fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release
run: ./scripts/fuzz/run.sh
- name: Upload nightly fuzz artifacts
if: always()
uses: actions/upload-artifact@v6
with:
name: fuzz-nightly-${{ github.run_number }}
name: fuzz-nightly-${{ matrix.target }}-${{ github.run_number }}
path: |
fuzz/artifacts/**
fuzz/corpus/**
fuzz/corpus/${{ matrix.target }}/**
if-no-files-found: ignore
retention-days: 30
Cargo.lock
Generated
+16 -21
View File
@@ -7830,9 +7830,9 @@ dependencies = [
[[package]]
name = "primefield"
version = "0.14.0-rc.12"
version = "0.14.0-rc.13"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8675564771a62f69a0af716b03e89b917b963c7b173b5855575e84fd4f605ca0"
checksum = "2db02b39ea98560a1fec81df6266f3c1ef7fdde06ac5ef17f69aee6101602630"
dependencies = [
"crypto-bigint 0.7.5",
"crypto-common 0.2.2",
@@ -8250,9 +8250,9 @@ dependencies = [
[[package]]
name = "quinn"
version = "0.11.9"
version = "0.11.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b9e20a958963c291dc322d98411f541009df2ced7b5a4f2bd52337638cfccf20"
checksum = "0c1a41e437b6bbd489372cd4971de128e85c855f56c57f283d20ff016cf7c0a8"
dependencies = [
"bytes",
"cfg_aliases",
@@ -8270,9 +8270,9 @@ dependencies = [
[[package]]
name = "quinn-proto"
version = "0.11.14"
version = "0.11.15"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "434b42fec591c96ef50e21e886936e66d3cc3f737104fdb9b737c40ffb94c098"
checksum = "4fcb935c5bec503c2f0e306bdd3e58bb9029dcb14fa8d9ac76e3a5256ac0763e"
dependencies = [
"aws-lc-rs",
"bytes",
@@ -8306,9 +8306,9 @@ dependencies = [
[[package]]
name = "quote"
version = "1.0.45"
version = "1.0.46"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "41f2619966050689382d2b44f664f4bc593e129785a36d6ee376ddf37259b924"
checksum = "dfbc457d0c7a0759a614551b11a6409e5951f6c7537be1f1b7682b9ae9230368"
dependencies = [
"proc-macro2",
]
@@ -9347,7 +9347,6 @@ dependencies = [
"rmp-serde",
"rustfs-checksums",
"rustfs-common",
"rustfs-concurrency",
"rustfs-config",
"rustfs-credentials",
"rustfs-data-usage",
@@ -9628,7 +9627,6 @@ dependencies = [
"rustfs-utils",
"serde",
"serde_json",
"sha2 0.11.0",
"temp-env",
"tempfile",
"thiserror 2.0.18",
@@ -9694,14 +9692,12 @@ dependencies = [
"rustfs-ecstore",
"rustfs-s3-ops",
"rustfs-s3-types",
"rustfs-storage-api",
"rustfs-targets",
"rustfs-utils",
"serde",
"serde_json",
"starshard",
"thiserror 2.0.18",
"time",
"tokio",
"tracing",
"tracing-subscriber",
@@ -9966,7 +9962,6 @@ dependencies = [
"futures-core",
"http 1.4.2",
"metrics",
"object_store",
"parking_lot",
"pin-project-lite",
"rustfs-common",
@@ -10282,9 +10277,9 @@ dependencies = [
[[package]]
name = "rustls"
version = "0.23.40"
version = "0.23.41"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ef86cd5876211988985292b91c96a8f2d298df24e75989a43a3c73f2d4d8168b"
checksum = "6b92b125634d9b795e7beca796cc790df15a7fb38323bf3196fda83292d06b1f"
dependencies = [
"aws-lc-rs",
"log",
@@ -11454,9 +11449,9 @@ dependencies = [
[[package]]
name = "sysinfo"
version = "0.39.4"
version = "0.39.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a71c821502e2786cc74c965f86561090a2b08fea91507cb599b5da291486a2aa"
checksum = "2c8bd2130a9b60bee2581bf82cfe89ee836424d1f37dcfa4ce21509611684673"
dependencies = [
"libc",
"memchr",
@@ -11666,9 +11661,9 @@ dependencies = [
[[package]]
name = "time"
version = "0.3.49"
version = "0.3.51"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "711a53c2d47bbd818258c498c8dbfe186a2526c631495cfe7e078567f86b8469"
checksum = "85c17d80feb7334b40c484e45ed1a5273dfd8bfda537c3be2e74a06a6686f327"
dependencies = [
"deranged",
"js-sys",
@@ -11689,9 +11684,9 @@ checksum = "9e1c906769ad99c88eaa54e728060edef082f8e358ff32030cb7c7d315e81109"
[[package]]
name = "time-macros"
version = "0.2.29"
version = "0.2.30"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "71c652a3727a9cbb9a02f707f530b618ce00d0ccd762009c8c23bd191df3c17d"
checksum = "dcef1a61bdb119096e153208ec5cbec23944ce8bca13be5c7f60c634f7403935"
dependencies = [
"num-conv",
"time-core",
Cargo.toml
+3 -4
View File
@@ -185,7 +185,7 @@ jsonwebtoken = { version = "10.4.0", features = ["aws_lc_rs"] }
openidconnect = { version = "4.0", default-features = false }
pbkdf2 = "0.13.0"
rsa = { version = "0.10.0-rc.18" }
rustls = { version = "0.23.40", default-features = false, features = ["aws-lc-rs", "logging", "tls12", "prefer-post-quantum", "std"] }
rustls = { version = "0.23.41", default-features = false, features = ["aws-lc-rs", "logging", "tls12", "prefer-post-quantum", "std"] }
rustls-native-certs = "0.8"
rustls-pki-types = "1.14.1"
sha1 = "0.11.0"
@@ -197,7 +197,7 @@ zeroize = { version = "1.9.0", features = ["derive"] }
chrono = { version = "0.4.45", features = ["serde"] }
humantime = "2.3.0"
jiff = { version = "0.2.29", features = ["serde"] }
time = { version = "0.3.49", features = ["std", "parsing", "formatting", "macros", "serde"] }
time = { version = "0.3.51", features = ["std", "parsing", "formatting", "macros", "serde"] }
# Database
deadpool-postgres = { version = "0.14", features = ["rt_tokio_1"] }
@@ -253,7 +253,6 @@ moka = { version = "0.12.15", features = ["future"] }
netif = "0.1.6"
num_cpus = { version = "1.17.0" }
nvml-wrapper = "0.12.1"
object_store = "0.13.2"
parking_lot = "0.12.5"
path-absolutize = "3.1.1"
path-clean = "1.0.1"
@@ -282,7 +281,7 @@ snafu = "0.9.1"
snap = "1.1.1"
starshard = { version = "2.2.1", features = ["rayon", "async", "serde"] }
strum = { version = "0.28.0", features = ["derive"] }
sysinfo = "0.39.4"
sysinfo = "0.39.5"
temp-env = "0.3.6"
tempfile = "3.27.0"
test-case = "3.3.1"
crates/ecstore/Cargo.toml
-1
View File
@@ -110,7 +110,6 @@ memmap2 = { workspace = true }
libc.workspace = true
rustix = { workspace = true }
rustfs-madmin.workspace = true
rustfs-concurrency.workspace = true
reqwest = { workspace = true }
aes-gcm.workspace = true
chacha20poly1305.workspace = true
crates/ecstore/src/set_disk.rs
+56 -22
View File
@@ -148,6 +148,7 @@ const EVENT_SET_DISK_MULTIPART: &str = "set_disk_multipart";
const EVENT_SET_DISK_WRITE: &str = "set_disk_write";
const EVENT_SET_DISK_HEAL: &str = "set_disk_heal";
const EVENT_SET_DISK_COMMIT_TAIL_SLOW: &str = "set_disk_commit_tail_slow";
const EVENT_SET_DISK_PUT_OBJECT_STAGE_SUMMARY: &str = "set_disk_put_object_stage_summary";
const SET_DISK_COMMIT_TAIL_WARN_THRESHOLD_MS: u128 = 5_000;
use crate::rio::{EtagResolvable, HashReader, HashReaderMut, TryGetIndex as _};
@@ -1138,10 +1139,8 @@ impl rustfs_storage_api::ObjectIO for SetDisks {
writers.push(w);
errors.push(e);
}
rustfs_io_metrics::record_put_object_stage_duration(
"set_disk_writer_setup",
writer_setup_stage_start.elapsed().as_secs_f64() * 1000.0,
);
let writer_setup_ms = writer_setup_stage_start.elapsed().as_millis() as u64;
rustfs_io_metrics::record_put_object_stage_duration("set_disk_writer_setup", writer_setup_ms as f64);
let nil_count = errors.iter().filter(|&e| e.is_none()).count();
if nil_count < write_quorum {
@@ -1202,10 +1201,8 @@ impl rustfs_storage_api::ObjectIO for SetDisks {
}
},
};
rustfs_io_metrics::record_put_object_stage_duration(
"set_disk_encode",
encode_stage_start.elapsed().as_secs_f64() * 1000.0,
);
let encode_ms = encode_stage_start.elapsed().as_millis() as u64;
rustfs_io_metrics::record_put_object_stage_duration("set_disk_encode", encode_ms as f64);
let _ = mem::replace(&mut data.stream, reader);
// if let Err(err) = close_bitrot_writers(&mut writers).await {
@@ -1314,12 +1311,9 @@ impl rustfs_storage_api::ObjectIO for SetDisks {
write_quorum,
)
.await?;
rustfs_io_metrics::record_put_object_stage_duration(
"set_disk_rename",
rename_stage_start.elapsed().as_secs_f64() * 1000.0,
);
let rename_stage_ms = rename_stage_start.elapsed().as_millis();
if rename_stage_ms >= SET_DISK_COMMIT_TAIL_WARN_THRESHOLD_MS {
let rename_stage_ms = rename_stage_start.elapsed().as_millis() as u64;
rustfs_io_metrics::record_put_object_stage_duration("set_disk_rename", rename_stage_ms as f64);
if (rename_stage_ms as u128) >= SET_DISK_COMMIT_TAIL_WARN_THRESHOLD_MS {
warn!(
event = EVENT_SET_DISK_COMMIT_TAIL_SLOW,
component = LOG_COMPONENT_ECSTORE,
@@ -1328,23 +1322,22 @@ impl rustfs_storage_api::ObjectIO for SetDisks {
bucket = %bucket,
object = %object,
tmp_dir = %tmp_dir,
duration_ms = rename_stage_ms as u64,
duration_ms = { rename_stage_ms },
write_quorum,
state = "slow",
"SetDisk commit tail stage is slow"
);
}
let mut cleanup_stage_ms: Option<u64> = None;
if let Some(old_dir) = op_old_dir {
let cleanup_stage_start = Instant::now();
self.commit_rename_data_dir(&cleanup_disks, bucket, object, &old_dir.to_string(), write_quorum)
.await?;
rustfs_io_metrics::record_put_object_stage_duration(
"set_disk_old_data_cleanup",
cleanup_stage_start.elapsed().as_secs_f64() * 1000.0,
);
let cleanup_stage_ms = cleanup_stage_start.elapsed().as_millis();
if cleanup_stage_ms >= SET_DISK_COMMIT_TAIL_WARN_THRESHOLD_MS {
let cleanup_ms = cleanup_stage_start.elapsed().as_millis() as u64;
cleanup_stage_ms = Some(cleanup_ms);
rustfs_io_metrics::record_put_object_stage_duration("set_disk_old_data_cleanup", cleanup_ms as f64);
if (cleanup_ms as u128) >= SET_DISK_COMMIT_TAIL_WARN_THRESHOLD_MS {
warn!(
event = EVENT_SET_DISK_COMMIT_TAIL_SLOW,
component = LOG_COMPONENT_ECSTORE,
@@ -1354,7 +1347,7 @@ impl rustfs_storage_api::ObjectIO for SetDisks {
object = %object,
tmp_dir = %tmp_dir,
old_dir = %old_dir,
duration_ms = cleanup_stage_ms as u64,
duration_ms = cleanup_ms,
write_quorum,
state = "slow",
"SetDisk commit tail stage is slow"
@@ -1411,10 +1404,51 @@ impl rustfs_storage_api::ObjectIO for SetDisks {
);
}
if issue3031_diag_enabled() {
warn!(
event = EVENT_SET_DISK_PUT_OBJECT_STAGE_SUMMARY,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_SET_DISK,
bucket = %bucket,
object = %object,
write_quorum,
write_path = write_path.metric_label(),
writer_setup_ms,
encode_ms,
rename_ms = rename_stage_ms,
cleanup_ms = cleanup_stage_ms.unwrap_or_default(),
cleanup_present = cleanup_stage_ms.is_some(),
commit_tail_ms = total_commit_tail_ms as u64,
result = "success",
"SetDisk put_object stage summary"
);
}
Ok(ObjectInfo::from_file_info(&fi, bucket, object, opts.versioned || opts.version_suspended))
}
.await;
if issue3031_diag_enabled()
&& let Err(err) = &result
{
let stage_hint = if err.to_string().contains("not enough disks to write") {
"writer_setup_or_quorum"
} else {
"unknown"
};
warn!(
event = EVENT_SET_DISK_PUT_OBJECT_STAGE_SUMMARY,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_SET_DISK,
bucket = %bucket,
object = %object,
result = "error",
stage_hint,
error = %err,
"SetDisk put_object stage summary"
);
}
if issue3031_diag_enabled() {
warn!(
target: "rustfs_ecstore::set_disk",
crates/io-metrics/src/internode_metrics.rs
+22 -7
View File
@@ -38,6 +38,7 @@ const INTERNODE_OPERATION_RECV_BYTES_TOTAL: &str = "rustfs_system_network_intern
const INTERNODE_OPERATION_REQUESTS_OUTGOING_TOTAL: &str = "rustfs_system_network_internode_operation_requests_outgoing_total";
const INTERNODE_OPERATION_REQUESTS_INCOMING_TOTAL: &str = "rustfs_system_network_internode_operation_requests_incoming_total";
const INTERNODE_OPERATION_ERRORS_TOTAL: &str = "rustfs_system_network_internode_operation_errors_total";
const INTERNODE_OPERATION_DURATION_MS: &str = "rustfs_system_network_internode_operation_duration_ms";
const INTERNODE_OPERATION_CLASSIFIED_ERRORS_TOTAL: &str = "rustfs_system_network_internode_operation_classified_errors_total";
const INTERNODE_OPERATION_RETRIES_TOTAL: &str = "rustfs_system_network_internode_operation_retries_total";
const INTERNODE_OPERATION_RETRY_SUCCESSES_TOTAL: &str = "rustfs_system_network_internode_operation_retry_successes_total";
@@ -74,6 +75,10 @@ pub const INTERNODE_OPERATION_METRICS: &[InternodeOperationMetricDescriptor] = &
name: INTERNODE_OPERATION_ERRORS_TOTAL,
labels: OPERATION_BACKEND_LABELS,
},
InternodeOperationMetricDescriptor {
name: INTERNODE_OPERATION_DURATION_MS,
labels: OPERATION_BACKEND_LABELS,
},
InternodeOperationMetricDescriptor {
name: INTERNODE_OPERATION_CLASSIFIED_ERRORS_TOTAL,
labels: OPERATION_BACKEND_CLASSIFICATION_LABELS,
@@ -208,6 +213,12 @@ impl InternodeMetrics {
counter!(INTERNODE_OPERATION_ERRORS_TOTAL, OPERATION_LABEL => operation, BACKEND_LABEL => backend).increment(1);
}
pub fn record_duration_for_operation_and_backend(&self, operation: &'static str, backend: &'static str, duration: Duration) {
let duration_ms = duration.as_secs_f64() * 1000.0;
metrics::histogram!(INTERNODE_OPERATION_DURATION_MS, OPERATION_LABEL => operation, BACKEND_LABEL => backend)
.record(duration_ms);
}
pub fn record_classified_error_for_operation_and_backend(
&self,
operation: &'static str,
@@ -382,14 +393,14 @@ mod tests {
#[test]
fn operation_metric_descriptors_include_backend_and_operation_labels() {
assert_eq!(INTERNODE_OPERATION_METRICS.len(), 9);
for metric in &INTERNODE_OPERATION_METRICS[..5] {
assert_eq!(INTERNODE_OPERATION_METRICS.len(), 10);
for metric in &INTERNODE_OPERATION_METRICS[..6] {
assert_eq!(metric.labels, &[OPERATION_LABEL, BACKEND_LABEL]);
}
for metric in &INTERNODE_OPERATION_METRICS[5..8] {
for metric in &INTERNODE_OPERATION_METRICS[6..9] {
assert_eq!(metric.labels, &[OPERATION_LABEL, BACKEND_LABEL, CLASSIFICATION_LABEL]);
}
assert_eq!(INTERNODE_OPERATION_METRICS[8].labels, &[STAGE_LABEL, DOMINANT_ERROR_LABEL]);
assert_eq!(INTERNODE_OPERATION_METRICS[9].labels, &[STAGE_LABEL, DOMINANT_ERROR_LABEL]);
}
#[test]
@@ -406,18 +417,22 @@ mod tests {
assert_eq!(
INTERNODE_OPERATION_METRICS[5].name,
"rustfs_system_network_internode_operation_classified_errors_total"
"rustfs_system_network_internode_operation_duration_ms"
);
assert_eq!(
INTERNODE_OPERATION_METRICS[6].name,
"rustfs_system_network_internode_operation_retries_total"
"rustfs_system_network_internode_operation_classified_errors_total"
);
assert_eq!(
INTERNODE_OPERATION_METRICS[7].name,
"rustfs_system_network_internode_operation_retry_successes_total"
"rustfs_system_network_internode_operation_retries_total"
);
assert_eq!(
INTERNODE_OPERATION_METRICS[8].name,
"rustfs_system_network_internode_operation_retry_successes_total"
);
assert_eq!(
INTERNODE_OPERATION_METRICS[9].name,
"rustfs_system_storage_erasure_write_quorum_failures_total"
);
}
crates/kms/Cargo.toml
-1
View File
@@ -43,7 +43,6 @@ aes-gcm = { workspace = true }
argon2 = { workspace = true }
chacha20poly1305 = { workspace = true }
rand = { workspace = true }
sha2 = { workspace = true }
base64 = { workspace = true }
zeroize = { workspace = true, features = ["derive"] }
crates/notify/Cargo.toml
-2
View File
@@ -60,10 +60,8 @@ quick-xml = { workspace = true, features = ["serialize", "serde-types", "encodin
tokio = { workspace = true, features = ["test-util"] }
tracing-subscriber = { workspace = true, features = ["env-filter"] }
axum = { workspace = true }
rustfs-storage-api = { workspace = true }
rustfs-utils = { workspace = true, features = ["path"] }
serde_json = { workspace = true }
time = { workspace = true }
criterion = { workspace = true }
[lints]
crates/s3select-api/Cargo.toml
-1
View File
@@ -37,7 +37,6 @@ rustfs-storage-api.workspace = true
futures = { workspace = true }
futures-core = { workspace = true }
http.workspace = true
object_store = { workspace = true }
pin-project-lite.workspace = true
s3s.workspace = true
serde_json = { workspace = true }
crates/s3select-api/src/object_store.rs
+5 -5
View File
@@ -20,14 +20,14 @@ use crate::{
use async_trait::async_trait;
use bytes::Bytes;
use chrono::Utc;
use datafusion::object_store::{
Attributes, CopyOptions, Error as o_Error, GetOptions, GetRange, GetResult, GetResultPayload, ListResult, MultipartUpload,
ObjectMeta, ObjectStore, PutMultipartOptions, PutOptions, PutPayload, PutResult, Result, path::Path,
};
use futures::pin_mut;
use futures::{Stream, StreamExt, future::ready, stream};
use futures_core::stream::BoxStream;
use http::{HeaderMap, HeaderValue, header::HeaderName};
use object_store::{
Attributes, CopyOptions, Error as o_Error, GetOptions, GetRange, GetResult, GetResultPayload, ListResult, MultipartUpload,
ObjectMeta, ObjectStore, PutMultipartOptions, PutOptions, PutPayload, PutResult, Result, path::Path,
};
use pin_project_lite::pin_project;
use rustfs_common::DEFAULT_DELIMITER;
use rustfs_storage_api::{HTTPRangeSpec, ObjectIO as _, ObjectOperations as _};
@@ -1080,8 +1080,8 @@ mod test {
scan_range_read_start, scan_range_stream, select_read_headers,
};
use bytes::Bytes;
use datafusion::object_store::{self, GetRange};
use futures::{StreamExt, TryStreamExt, stream};
use object_store::GetRange;
use s3s::dto::{
CSVInput, CSVOutput, ExpressionType, InputSerialization, OutputSerialization, SelectObjectContentInput,
SelectObjectContentRequest,
crates/s3select-api/src/query/session.rs
+2 -2
View File
@@ -21,10 +21,10 @@ use datafusion::{
record_batch::RecordBatch,
},
execution::{SessionStateBuilder, context::SessionState, runtime_env::RuntimeEnvBuilder},
object_store::{ObjectStore, ObjectStoreExt, memory::InMemory, path::Path},
parquet::arrow::ArrowWriter,
prelude::SessionContext,
};
use object_store::{ObjectStore, ObjectStoreExt, memory::InMemory, path::Path};
use std::sync::Arc;
use tracing::error;
@@ -110,7 +110,7 @@ impl SessionCtxFactory {
QueryError::StoreError { e: e.to_string() }
})?;
df_session_state.with_object_store(&store_url, Arc::new(store)).build()
df_session_state.with_object_store(&store_url, store).build()
} else {
let store: EcObjectStore =
EcObjectStore::new(context.input.clone()).map_err(|_| QueryError::NotImplemented { err: String::new() })?;
crates/utils/src/path.rs
+79
View File
@@ -12,6 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use std::path::Component;
use std::path::Path;
use std::path::PathBuf;
@@ -550,6 +551,52 @@ pub fn trim_etag(etag: &str) -> String {
etag.trim_matches('"').to_string()
}
/// Returns `true` if `path` contains a `..` component (parent directory traversal).
fn contains_parent_dir_component(path: &str) -> bool {
path.split(['/', '\\']).any(|component| component == "..")
}
/// Validates that an archive entry path does not escape the target bucket.
///
/// Rejects paths containing `..` components, root prefixes, or device/prefix
/// components that would allow path traversal outside the extraction root.
///
/// Returns `Ok(())` if the path is safe, or `Err(message)` describing the violation.
pub fn validate_extract_relative_path(path: &str) -> Result<(), String> {
let p = Path::new(path);
if p.components()
.any(|c| matches!(c, Component::Prefix(_) | Component::RootDir | Component::ParentDir))
|| contains_parent_dir_component(p.to_string_lossy().as_ref())
{
return Err("archive entry path must stay within the target bucket".to_string());
}
Ok(())
}
/// Normalizes an archive entry key by applying a prefix, trimming slashes,
/// and ensuring directory entries end with `/`.
///
/// Validates both the raw path and the final key against path traversal.
///
/// Returns `Ok(normalized_key)` or `Err(message)` if the path is unsafe.
pub fn normalize_extract_entry_key(path: &str, prefix: Option<&str>, is_dir: bool) -> Result<String, String> {
validate_extract_relative_path(path)?;
let path = path.trim_matches('/');
let mut key = match prefix {
Some(prefix) if !path.is_empty() => format!("{prefix}/{path}"),
Some(prefix) => prefix.to_string(),
None => path.to_string(),
};
if is_dir && !key.ends_with('/') {
key.push('/');
}
validate_extract_relative_path(&key)?;
Ok(key)
}
#[cfg(test)]
mod tests {
use super::*;
@@ -918,4 +965,36 @@ mod tests {
}
}
}
#[test]
fn test_validate_extract_relative_path_accepts_safe_paths() {
assert!(validate_extract_relative_path("file.txt").is_ok());
assert!(validate_extract_relative_path("dir/file.txt").is_ok());
assert!(validate_extract_relative_path("a/b/c").is_ok());
assert!(validate_extract_relative_path("").is_ok());
}
#[test]
fn test_validate_extract_relative_path_rejects_traversal() {
assert!(validate_extract_relative_path("../escape.txt").is_err());
assert!(validate_extract_relative_path("dir/../../../escape.txt").is_err());
assert!(validate_extract_relative_path("/absolute/path").is_err());
assert!(validate_extract_relative_path("dir/..").is_err());
assert!(validate_extract_relative_path("..").is_err());
}
#[test]
fn test_normalize_extract_entry_key_basic() {
assert_eq!(normalize_extract_entry_key("file.txt", None, false).unwrap(), "file.txt");
assert_eq!(normalize_extract_entry_key("file.txt", Some("prefix"), false).unwrap(), "prefix/file.txt");
assert_eq!(normalize_extract_entry_key("dir", None, true).unwrap(), "dir/");
assert_eq!(normalize_extract_entry_key("", Some("prefix"), false).unwrap(), "prefix");
}
#[test]
fn test_normalize_extract_entry_key_rejects_traversal() {
assert!(normalize_extract_entry_key("../escape.txt", None, false).is_err());
assert!(normalize_extract_entry_key("file.txt", Some("../bad"), false).is_err());
assert!(normalize_extract_entry_key("/absolute", None, false).is_err());
}
}
fuzz/Cargo.lock
Generated
+145 -5349
View File
File diff suppressed because it is too large Load Diff
fuzz/Cargo.toml
-1
View File
@@ -12,7 +12,6 @@ cargo-fuzz = true
[dependencies]
arbitrary = "1"
libfuzzer-sys = "0.4"
rustfs = { path = "../rustfs", default-features = false }
rustfs-ecstore = { path = "../crates/ecstore" }
rustfs-filemeta = { path = "../crates/filemeta" }
rustfs-policy = { path = "../crates/policy" }
fuzz/README.md
+24 -7
View File
@@ -34,7 +34,7 @@ Crash reproducers are written under `fuzz/artifacts/<target>/`.
## Targets
- `bucket_validation`
- Exercises RustFS bucket-name and bucket/object argument validation without pulling in the full `rustfs` binary crate graph.
- Exercises RustFS bucket-name and bucket/object argument validation.
- `archive_extract`
- Exercises archive entry path normalization, prefix application, and bucket-namespace containment checks.
- `path_containment`
@@ -53,17 +53,34 @@ cd fuzz
cargo +nightly fuzz run path_containment
```
Run bounded smoke targets from the repository root:
Use the unified runner script from the repository root:
```bash
./scripts/fuzz/run_ci_targets.sh
# Build + run all smoke targets (60s each)
./scripts/fuzz/run.sh
# Build only (no fuzz run)
BUILD_ONLY=1 ./scripts/fuzz/run.sh
# Build + run a single target
FUZZ_TARGET=path_containment ./scripts/fuzz/run.sh
# Nightly-style: 300s per target
MAX_TOTAL_TIME=300 ./scripts/fuzz/run.sh
# Skip build (use pre-built harness)
SKIP_BUILD=1 FUZZ_TARGET=local_metadata ./scripts/fuzz/run.sh
```
Run a longer local/nightly-style pass:
## CI Workflow
```bash
./scripts/fuzz/run_nightly_targets.sh
```
The GitHub Actions workflow (`.github/workflows/fuzz.yml`) uses a **build/run separation** pattern:
1. **`fuzz-build`** — compiles all fuzz targets once, then uploads only the prebuilt smoke harness binaries needed by later jobs.
2. **`pr-fuzz-smoke`** — matrix job that runs each target in parallel (60s each). Downloads the prebuilt binaries and executes them directly, so the job does not need to restore the full `fuzz/target/` tree or reinstall `cargo-fuzz`.
3. **`nightly-fuzz-corpus`** — matrix job that reuses the same prebuilt binaries and runs each target in parallel (300s each) on a daily schedule.
This design avoids redundant compilation across targets and keeps wall-clock time low.
## Seed Corpus
fuzz/fuzz_targets/archive_extract.rs
+1 -1
View File
@@ -1,7 +1,7 @@
#![no_main]
use libfuzzer_sys::fuzz_target;
use rustfs::app::object_usecase::{normalize_extract_entry_key, validate_extract_relative_path};
use rustfs_utils::path::{normalize_extract_entry_key, validate_extract_relative_path};
fn parse_case(data: &[u8]) -> (String, Option<String>, bool, Vec<String>) {
let text = String::from_utf8_lossy(data);
fuzz/fuzz_targets/local_metadata.rs
+25 -10
View File
@@ -3,7 +3,6 @@
use libfuzzer_sys::fuzz_target;
use rustfs_filemeta::FileMeta;
use rustfs_utils::compress::{CompressionAlgorithm, decompress_block};
use std::collections::BTreeSet;
fn pick_algorithm(tag: u8) -> CompressionAlgorithm {
match tag % 6 {
@@ -16,26 +15,42 @@ fn pick_algorithm(tag: u8) -> CompressionAlgorithm {
}
}
fn exercise_payload(payload: &[u8], algorithm: CompressionAlgorithm) {
/// Full exercise: all parsers + decompression on the complete payload.
fn exercise_full(payload: &[u8], algorithm: CompressionAlgorithm) {
let _ = FileMeta::load(payload);
let _ = FileMeta::load_or_convert(payload);
let _ = FileMeta::read_format_versions(payload);
let _ = decompress_block(payload, algorithm);
}
fn interesting_prefix_lengths(len: usize) -> BTreeSet<usize> {
let mut lengths = BTreeSet::from([0usize, 1, 2, 4, 5, 8, 16, 32]);
lengths.insert(len);
lengths.insert(len.saturating_sub(1));
lengths.into_iter().filter(|candidate| *candidate <= len).collect()
/// Lightweight exercise: only metadata parsers on a prefix (no decompression).
/// Prefix tests catch partial-input / truncated-header panics cheaply.
fn exercise_prefix(payload: &[u8]) {
let _ = FileMeta::load(payload);
let _ = FileMeta::read_format_versions(payload);
}
fuzz_target!(|data: &[u8]| {
let algorithm = pick_algorithm(data.first().copied().unwrap_or_default());
exercise_payload(data, algorithm);
// Full exercise on the complete input.
exercise_full(data, algorithm);
for prefix_len in interesting_prefix_lengths(data.len()) {
exercise_payload(&data[..prefix_len], algorithm);
// Prefix probing aligned with xl.meta binary layout:
// 4 = magic header ("XL2 ")
// 5 = magic + version byte
// 8 = magic + version + start of bin32 length field
// 12 = magic + version + bin32 length (4 bytes) + CRC start
// len-1 = truncated-at-end
//
// Only test prefixes that are strictly smaller than the full input
// (the full input is already tested above).
for prefix_len in [4usize, 5, 8, 12] {
if prefix_len < data.len() {
exercise_prefix(&data[..prefix_len]);
}
}
if data.len() > 1 {
exercise_prefix(&data[..data.len() - 1]);
}
});
rustfs/src/app/object_usecase.rs
+53 -77
View File
@@ -14,12 +14,43 @@
//! Object application use-case contracts.
use super::object_api_utils::to_s3s_etag;
use super::quota::checker::QuotaChecker;
use super::storageclass;
// Performance metrics recording (with zero-copy-metrics integration)
use super::ECStore;
use super::{AppReplicationConfigExt as _, AppVersioningConfigExt as _, predict_lifecycle_expiration, validate_restore_request};
use super::{DiskError, is_all_buckets_not_found};
use super::{DynReader, HashReader, WritePlan, wrap_reader};
use super::{Error as EcstoreError, StorageError, is_err_bucket_not_found, is_err_object_not_found, is_err_version_not_found};
use super::{MIN_DISK_COMPRESSIBLE_SIZE, is_disk_compressible};
use super::{get_lock_acquire_timeout, is_valid_storage_class};
use super::{
lifecycle::{
bucket_lifecycle_audit::LcEventSrc,
bucket_lifecycle_ops::{enqueue_transition_immediate, post_restore_opts},
lifecycle::{self, TransitionOptions},
},
metadata_sys,
object_lock::{
objectlock::{get_object_legalhold_meta, get_object_retention_meta},
objectlock_sys::{BucketObjectLockSys, check_object_lock_for_deletion, is_retention_active},
},
quota::QuotaOperation,
replication::{
DeletedObjectReplicationInfo, ObjectOpts as ReplicationObjectOpts, check_replicate_delete, get_must_replicate_options,
must_replicate, schedule_replication, schedule_replication_delete,
},
tagging::decode_tags,
versioning_sys::BucketVersioningSys,
};
use crate::app::context::{
AppContext, get_global_app_context, resolve_notify_interface_for_context, resolve_object_store_handle_for_context,
};
use crate::config::RustFSBufferConfig;
use crate::delete_tail_activity::{DeleteTailActivityGuard, DeleteTailStage};
use crate::error::ApiError;
use crate::server::convert_ecstore_object_info;
use crate::storage::access::{PostObjectRequestMarker, authorize_request, has_bypass_governance_header, req_info_mut};
use crate::storage::concurrency::{
ConcurrencyManager, GetObjectGuard, get_concurrency_aware_buffer_size, get_concurrency_manager,
@@ -46,38 +77,6 @@ use http::{HeaderMap, HeaderValue, StatusCode};
use md5::Context as Md5Context;
use metrics::{counter, histogram};
use pin_project_lite::pin_project;
use rustfs_object_capacity::capacity_manager::get_capacity_manager;
// Performance metrics recording (with zero-copy-metrics integration)
use super::ECStore;
use super::object_api_utils::to_s3s_etag;
use super::quota::checker::QuotaChecker;
use super::storageclass;
use super::{AppReplicationConfigExt as _, AppVersioningConfigExt as _, predict_lifecycle_expiration, validate_restore_request};
use super::{DiskError, is_all_buckets_not_found};
use super::{DynReader, HashReader, WritePlan, wrap_reader};
use super::{Error as EcstoreError, StorageError, is_err_bucket_not_found, is_err_object_not_found, is_err_version_not_found};
use super::{MIN_DISK_COMPRESSIBLE_SIZE, is_disk_compressible};
use super::{get_lock_acquire_timeout, is_valid_storage_class};
use super::{
lifecycle::{
bucket_lifecycle_audit::LcEventSrc,
bucket_lifecycle_ops::{enqueue_transition_immediate, post_restore_opts},
lifecycle::{self, TransitionOptions},
},
metadata_sys,
object_lock::{
objectlock::{get_object_legalhold_meta, get_object_retention_meta},
objectlock_sys::{BucketObjectLockSys, check_object_lock_for_deletion, is_retention_active},
},
quota::QuotaOperation,
replication::{
DeletedObjectReplicationInfo, ObjectOpts as ReplicationObjectOpts, check_replicate_delete, get_must_replicate_options,
must_replicate, schedule_replication, schedule_replication_delete,
},
tagging::decode_tags,
versioning_sys::BucketVersioningSys,
};
use crate::server::convert_ecstore_object_info;
use rustfs_concurrency::GetObjectQueueSnapshot;
use rustfs_filemeta::{
REPLICATE_INCOMING_DELETE, ReplicateDecision, ReplicateTargetDecision, ReplicationState, ReplicationStatusType,
@@ -88,6 +87,7 @@ use rustfs_io_core::{BytesPool, PooledBuffer};
use rustfs_io_metrics;
use rustfs_lock::NamespaceLockGuard;
use rustfs_notify::EventArgsBuilder;
use rustfs_object_capacity::capacity_manager::get_capacity_manager;
use rustfs_policy::policy::action::{Action, S3Action};
use rustfs_s3_ops::{S3Operation, delete_event_name_for_marker, put_event_name_for_post_object};
use rustfs_s3select_api::object_store::bytes_stream;
@@ -119,7 +119,7 @@ use s3s::header::{X_AMZ_RESTORE, X_AMZ_RESTORE_OUTPUT_PATH};
use s3s::{S3Error, S3ErrorCode, S3Request, S3Response, S3Result, s3_error};
use std::collections::HashMap;
use std::ops::Add;
use std::path::{Component, Path};
use std::path::Path;
use std::pin::Pin;
use std::task::{Context, Poll};
@@ -337,14 +337,14 @@ impl MemoryTrackedBytesStream {
impl futures::Stream for MemoryTrackedBytesStream {
type Item = std::io::Result<Bytes>;
fn poll_next(self: std::pin::Pin<&mut Self>, _cx: &mut std::task::Context<'_>) -> std::task::Poll<Option<Self::Item>> {
fn poll_next(self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<Option<Self::Item>> {
let this = self.project();
if *this.emitted {
return std::task::Poll::Ready(None);
return Poll::Ready(None);
}
*this.emitted = true;
std::task::Poll::Ready(Some(Ok(this.bytes.clone())))
Poll::Ready(Some(Ok(this.bytes.clone())))
}
}
@@ -360,16 +360,12 @@ impl<R> ExtractArchiveEtagReader<R> {
}
impl<R: AsyncRead> AsyncRead for ExtractArchiveEtagReader<R> {
fn poll_read(
self: std::pin::Pin<&mut Self>,
cx: &mut std::task::Context<'_>,
buf: &mut ReadBuf<'_>,
) -> std::task::Poll<std::io::Result<()>> {
fn poll_read(self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &mut ReadBuf<'_>) -> Poll<std::io::Result<()>> {
let this = self.project();
let before = buf.filled().len();
match this.inner.poll_read(cx, buf) {
std::task::Poll::Pending => std::task::Poll::Pending,
std::task::Poll::Ready(Ok(())) => {
Poll::Pending => Poll::Pending,
Poll::Ready(Ok(())) => {
let filled = &buf.filled()[before..];
if !filled.is_empty() {
this.md5.consume(filled);
@@ -379,9 +375,9 @@ impl<R: AsyncRead> AsyncRead for ExtractArchiveEtagReader<R> {
*etag = Some(format!("{:x}", this.md5.clone().finalize()));
}
}
std::task::Poll::Ready(Ok(()))
Poll::Ready(Ok(()))
}
std::task::Poll::Ready(Err(err)) => std::task::Poll::Ready(Err(err)),
Poll::Ready(Err(err)) => Poll::Ready(Err(err)),
}
}
}
@@ -988,21 +984,12 @@ fn snowball_meta_flag(headers: &HeaderMap, exact_keys: &[&str], suffix_lower: &s
snowball_meta_value(headers, exact_keys, suffix_lower).is_some_and(|value| value.eq_ignore_ascii_case("true"))
}
fn contains_parent_dir_component(path: &str) -> bool {
path.split(['/', '\\']).any(|component| component == "..")
}
/// Validates that an archive entry path does not escape the target bucket.
///
/// Delegates to [`rustfs_utils::path::validate_extract_relative_path`] and wraps
/// the result as an S3 error on failure.
pub fn validate_extract_relative_path(path: &str) -> S3Result<()> {
let path = Path::new(path);
if path
.components()
.any(|component| matches!(component, Component::Prefix(_) | Component::RootDir | Component::ParentDir))
|| contains_parent_dir_component(path.to_string_lossy().as_ref())
{
return Err(s3_error!(InvalidArgument, "archive entry path must stay within the target bucket"));
}
Ok(())
rustfs_utils::path::validate_extract_relative_path(path).map_err(|msg| s3_error!(InvalidArgument, "{msg}"))
}
fn normalize_snowball_prefix(prefix: &str) -> S3Result<Option<String>> {
@@ -1016,22 +1003,13 @@ fn normalize_snowball_prefix(prefix: &str) -> S3Result<Option<String>> {
Ok(Some(normalized.to_string()))
}
/// Normalizes an archive entry key by applying a prefix, trimming slashes,
/// and ensuring directory entries end with `/`.
///
/// Delegates to [`rustfs_utils::path::normalize_extract_entry_key`] and wraps
/// the result as an S3 error on failure.
pub fn normalize_extract_entry_key(path: &str, prefix: Option<&str>, is_dir: bool) -> S3Result<String> {
validate_extract_relative_path(path)?;
let path = path.trim_matches('/');
let mut key = match prefix {
Some(prefix) if !path.is_empty() => format!("{prefix}/{path}"),
Some(prefix) => prefix.to_string(),
None => path.to_string(),
};
if is_dir && !key.ends_with('/') {
key.push('/');
}
validate_extract_relative_path(&key)?;
Ok(key)
rustfs_utils::path::normalize_extract_entry_key(path, prefix, is_dir).map_err(|msg| s3_error!(InvalidArgument, "{msg}"))
}
fn map_extract_archive_error(err: impl std::fmt::Display) -> S3Error {
@@ -1614,7 +1592,7 @@ impl DefaultObjectUsecase {
#[allow(clippy::too_many_arguments)]
async fn prepare_get_object_read(
req: &S3Request<GetObjectInput>,
store: &super::ECStore,
store: &ECStore,
manager: &ConcurrencyManager,
bucket: &str,
key: &str,
@@ -2895,7 +2873,6 @@ impl DefaultObjectUsecase {
validate_ssec_for_read(&info.user_defined, sse_customer_key.as_ref(), sse_customer_key_md5.as_ref())?;
let metadata_map = info.user_defined.clone();
debug!(
"GetObjectAttributes raw object_attributes={:?}",
object_attributes.iter().map(|value| value.as_str()).collect::<Vec<_>>()
@@ -2952,7 +2929,6 @@ impl DefaultObjectUsecase {
} else {
None
};
let object_parts = if requested(ObjectAttributes::OBJECT_PARTS) && info.is_multipart() {
let params = parse_list_parts_params(part_number_marker, max_parts)?;
let mut parts = Vec::new();
rustfs/src/server/http.rs
+475 -202
View File
@@ -31,11 +31,13 @@ use crate::server::{
},
};
use crate::storage;
use crate::storage::request_context::{RequestContext, extract_request_id_from_headers};
use crate::storage::rpc::InternodeRpcService;
use crate::storage::tonic_service::make_server;
use crate::storage::{TONIC_RPC_PREFIX, verify_rpc_signature};
use bytes::Bytes;
use http::{HeaderMap, Method, Request as HttpRequest, Response};
use hyper::body::Incoming;
use hyper_util::{
rt::{TokioExecutor, TokioIo, TokioTimer},
server::conn::auto::Builder as ConnBuilder,
@@ -56,12 +58,14 @@ use s3s::{host::MultiDomain, service::S3Service, service::S3ServiceBuilder};
use socket2::{SockRef, TcpKeepalive};
use std::io::{Error, Result};
use std::net::SocketAddr;
use std::pin::Pin;
use std::sync::Arc;
use std::sync::atomic::{AtomicU64, Ordering};
use std::task::{Context, Poll};
use std::time::Duration;
use tokio::net::{TcpListener, TcpStream};
use tonic::{Request, Status};
use tower::ServiceBuilder;
use tower::{Service, ServiceBuilder};
use tower_http::add_extension::AddExtensionLayer;
use tower_http::catch_panic::CatchPanicLayer;
use tower_http::compression::CompressionLayer;
@@ -220,6 +224,34 @@ pub(crate) fn active_http_requests() -> u64 {
ACTIVE_HTTP_REQUESTS.load(Ordering::Relaxed)
}
fn trace_on_response<ResBody>(response: &Response<ResBody>, latency: Duration, span: &Span) {
span.record("status_code", tracing::field::display(response.status()));
let _enter = span.enter();
let status_class = status_class_label(response.status());
record_active_http_requests(-1);
histogram!(
METRIC_HTTP_SERVER_REQUEST_DURATION_SECONDS,
LABEL_HTTP_STATUS_CLASS => status_class
)
.record(latency.as_secs_f64());
if response.status().is_client_error() || response.status().is_server_error() {
counter!(
METRIC_HTTP_SERVER_FAILURES_TOTAL,
LABEL_HTTP_STATUS_CLASS => status_class
)
.increment(1);
}
if let Some(cl) = response.headers().get("content-length")
&& let Some(len) = cl.to_str().ok().and_then(|s| s.parse::<u64>().ok())
{
histogram!(
METRIC_HTTP_SERVER_RESPONSE_BODY_SIZE_BYTES,
LABEL_HTTP_STATUS_CLASS => status_class
)
.record(len as f64);
}
}
pub async fn start_http_server(config: &config::Config, readiness: Arc<GlobalReadiness>) -> Result<(ShutdownHandle, SocketAddr)> {
let server_addr = parse_and_resolve_address(config.address.as_str()).map_err(Error::other)?;
@@ -800,6 +832,90 @@ struct ConnectionContext {
trusted_proxy_layer: Option<rustfs_trusted_proxies::TrustedProxyLayer>,
}
#[derive(Clone)]
struct PathDispatchService<A, B> {
external: A,
internode: B,
}
#[derive(Clone, Default)]
struct InternodeRequestContextLiteLayer;
impl<S> tower::Layer<S> for InternodeRequestContextLiteLayer {
type Service = InternodeRequestContextLiteService<S>;
fn layer(&self, inner: S) -> Self::Service {
InternodeRequestContextLiteService { inner }
}
}
#[derive(Clone)]
struct InternodeRequestContextLiteService<S> {
inner: S,
}
impl<S, B> Service<HttpRequest<B>> for InternodeRequestContextLiteService<S>
where
S: Service<HttpRequest<B>> + Clone,
{
type Response = S::Response;
type Error = S::Error;
type Future = S::Future;
fn poll_ready(&mut self, cx: &mut Context<'_>) -> Poll<std::result::Result<(), Self::Error>> {
self.inner.poll_ready(cx)
}
fn call(&mut self, mut req: HttpRequest<B>) -> Self::Future {
let request_id = extract_request_id_from_headers(req.headers());
req.extensions_mut().insert(RequestContext {
x_amz_request_id: request_id.clone(),
request_id,
trace_id: None,
span_id: None,
start_time: std::time::Instant::now(),
});
self.inner.call(req)
}
}
impl<A, B> PathDispatchService<A, B> {
fn new(external: A, internode: B) -> Self {
Self { external, internode }
}
fn is_internode_path(path: &str) -> bool {
crate::server::has_path_prefix(path, crate::server::RPC_PREFIX)
}
}
impl<A, B> Service<HttpRequest<Incoming>> for PathDispatchService<A, B>
where
A: Service<HttpRequest<Incoming>> + Clone + Send + 'static,
A::Future: Send + 'static,
B: Service<HttpRequest<Incoming>, Response = A::Response, Error = A::Error> + Clone + Send + 'static,
B::Future: Send + 'static,
{
type Response = A::Response;
type Error = A::Error;
type Future = Pin<Box<dyn Future<Output = std::result::Result<Self::Response, Self::Error>> + Send>>;
fn poll_ready(&mut self, cx: &mut Context<'_>) -> Poll<std::result::Result<(), Self::Error>> {
match self.external.poll_ready(cx)? {
Poll::Ready(()) => self.internode.poll_ready(cx),
Poll::Pending => Poll::Pending,
}
}
fn call(&mut self, req: HttpRequest<Incoming>) -> Self::Future {
if Self::is_internode_path(req.uri().path()) {
Box::pin(self.internode.call(req))
} else {
Box::pin(self.external.call(req))
}
}
}
/// Adapter that implements the OpenTelemetry [`Extractor`] trait for Hyper's
/// [`HeaderMap`], enabling trace context propagation by extracting
/// OpenTelemetry headers from incoming HTTP requests.
@@ -912,7 +1028,8 @@ fn process_connection(
let http_service = s3_service;
let http_service = InternodeRpcService::new(http_service);
let service = hybrid(http_service, rpc_service);
let external_service = hybrid(http_service.clone(), rpc_service.clone());
let internode_service = hybrid(http_service, rpc_service);
let remote_addr = match socket.peer_addr() {
Ok(addr) => Some(RemoteAddr(addr)),
@@ -954,217 +1071,325 @@ fn process_connection(
// 21. PublicHealthEndpointLayer — handles public health before s3s host parsing
// 22. VirtualHostStyleHintLayer — actionable error for unroutable virtual-hosted-style (conditional)
// ─────────────────────────────────────────────────────────────
let hybrid_service = ServiceBuilder::new()
// NOTE: Both extension types are intentionally inserted to maintain compatibility:
// 1. `Option<RemoteAddr>` - Used by existing admin/storage handlers throughout the codebase
// 2. `std::net::SocketAddr` - Required by TrustedProxyMiddleware for proxy validation
// This dual insertion is necessary because the middleware expects the raw SocketAddr type
// while our application code uses the RemoteAddr wrapper. Consolidating these would
// require either modifying the third-party middleware or refactoring all existing handlers.
.layer(AddExtensionLayer::new(remote_addr))
.option_layer(remote_addr.map(|ra| AddExtensionLayer::new(ra.0)))
// Add TrustedProxyLayer to handle X-Forwarded-For and other proxy headers
// This should be placed before TraceLayer so that logs reflect the real client IP
// Pre-computed in ConnectionContext to avoid per-connection is_enabled() check.
.option_layer(trusted_proxy_layer)
.layer(SetRequestIdLayer::x_request_id(MakeRequestUuid))
.layer(RequestContextLayer)
.layer(EmptyBodyContentLengthCompatLayer)
.layer(CatchPanicLayer::new())
// CRITICAL: Insert ReadinessGateLayer before business logic
// This stops requests from hitting IAMAuth or Storage if they are not ready.
.layer(ReadinessGateLayer::new(readiness))
// Add Keystone authentication middleware
// This validates X-Auth-Token headers and stores credentials in task-local storage
// Must be placed AFTER ReadinessGateLayer but BEFORE business logic
// Pre-computed in ConnectionContext to avoid per-connection OnceLock read.
.layer(KeystoneAuthLayer::new(keystone_auth))
.layer(
TraceLayer::new_for_http()
.make_span_with(|request: &HttpRequest<_>| {
let request_context = request.extensions().get::<crate::storage::request_context::RequestContext>();
let request_id = request_context
.map(|ctx| ctx.request_id.as_str())
.unwrap_or("unknown");
let trace_id = request_context
.and_then(|ctx| ctx.trace_id.as_deref())
.unwrap_or("unknown");
let span_id = request_context
.and_then(|ctx| ctx.span_id.as_deref())
.unwrap_or("unknown");
// Batch 1 intentionally keeps the external and internode stacks behaviorally
// identical while giving each path family a named construction boundary.
// Later batches will trim internode-only middleware without risking drift in
// the public HTTP stack.
let build_external_stack = |service| {
ServiceBuilder::new()
// NOTE: Both extension types are intentionally inserted to maintain compatibility:
// 1. `Option<RemoteAddr>` - Used by existing admin/storage handlers throughout the codebase
// 2. `std::net::SocketAddr` - Required by TrustedProxyMiddleware for proxy validation
// This dual insertion is necessary because the middleware expects the raw SocketAddr type
// while our application code uses the RemoteAddr wrapper. Consolidating these would
// require either modifying the third-party middleware or refactoring all existing handlers.
.layer(AddExtensionLayer::new(remote_addr))
.option_layer(remote_addr.map(|ra| AddExtensionLayer::new(ra.0)))
// Add TrustedProxyLayer to handle X-Forwarded-For and other proxy headers
// This should be placed before TraceLayer so that logs reflect the real client IP
// Pre-computed in ConnectionContext to avoid per-connection is_enabled() check.
.option_layer(trusted_proxy_layer.clone())
.layer(SetRequestIdLayer::x_request_id(MakeRequestUuid))
.layer(InternodeRequestContextLiteLayer)
.layer(EmptyBodyContentLengthCompatLayer)
.layer(CatchPanicLayer::new())
// CRITICAL: Insert ReadinessGateLayer before business logic
// This stops requests from hitting IAMAuth or Storage if they are not ready.
.layer(ReadinessGateLayer::new(readiness.clone()))
// Add Keystone authentication middleware
// This validates X-Auth-Token headers and stores credentials in task-local storage
// Must be placed AFTER ReadinessGateLayer but BEFORE business logic
// Pre-computed in ConnectionContext to avoid per-connection OnceLock read.
.layer(KeystoneAuthLayer::new(keystone_auth.clone()))
.layer(
TraceLayer::new_for_http()
.make_span_with(|request: &HttpRequest<_>| {
let request_context = request.extensions().get::<crate::storage::request_context::RequestContext>();
let request_id = request_context
.map(|ctx| ctx.request_id.as_str())
.unwrap_or("unknown");
let trace_id = request_context
.and_then(|ctx| ctx.trace_id.as_deref())
.unwrap_or("unknown");
let span_id = request_context
.and_then(|ctx| ctx.span_id.as_deref())
.unwrap_or("unknown");
let parent_context = global::get_text_map_propagator(|propagator| {
propagator.extract(&HeaderMapCarrier::new(request.headers()))
});
let parent_context = global::get_text_map_propagator(|propagator| {
propagator.extract(&HeaderMapCarrier::new(request.headers()))
});
// Log trace context extraction for debugging distributed tracing
if parent_context.has_active_span() {
let span_ref = parent_context.span();
trace!(
otel_trace_id = %span_ref.span_context().trace_id(),
otel_parent_span_id = %span_ref.span_context().span_id(),
sampled = span_ref.span_context().is_sampled(),
"Extracted trace context from incoming request headers"
);
} else {
trace!("No trace context found in request headers, will create root span");
}
// Extract real client IP from trusted proxy middleware if available
let client_info = request.extensions().get::<ClientInfo>();
let peer_addr = client_info
.map(|info| info.real_ip.to_string())
.or_else(|| request.extensions().get::<RemoteAddr>().map(|addr| addr.0.to_string()))
.unwrap_or_else(|| "unknown".to_string());
let span = tracing::info_span!("http-request",
request_id = %request_id,
trace_id = %trace_id,
span_id = %span_id,
status_code = tracing::field::Empty,
method = %request.method(),
peer_addr = %peer_addr,
uri = %redact_sensitive_uri_query(request.uri()),
version = ?request.version(),
user_agent = tracing::field::Empty,
content_type = tracing::field::Empty,
content_length = tracing::field::Empty,
);
if span.is_disabled() {
return span;
}
if let Err(e) = span.set_parent(parent_context) {
debug!(component = LOG_COMPONENT_SERVER, subsystem = LOG_SUBSYSTEM_HTTP, error = ?e, "Failed to propagate tracing context");
}
for (header_name, header_value) in request.headers() {
let value = header_value.to_str().unwrap_or("invalid");
if header_name == "user-agent" {
span.record("user_agent", value);
} else if header_name == "content-type" {
span.record("content_type", value);
} else if header_name == "content-length" {
span.record("content_length", value);
if parent_context.has_active_span() {
let span_ref = parent_context.span();
trace!(
otel_trace_id = %span_ref.span_context().trace_id(),
otel_parent_span_id = %span_ref.span_context().span_id(),
sampled = span_ref.span_context().is_sampled(),
"Extracted trace context from incoming request headers"
);
} else {
trace!("No trace context found in request headers, will create root span");
}
}
let client_info = request.extensions().get::<ClientInfo>();
let peer_addr = client_info
.map(|info| info.real_ip.to_string())
.or_else(|| request.extensions().get::<RemoteAddr>().map(|addr| addr.0.to_string()))
.unwrap_or_else(|| "unknown".to_string());
span
})
.on_request(|request: &HttpRequest<_>, span: &Span| {
let _enter = span.enter();
trace!("HTTP request started");
let method = request_method_label(request.method());
record_active_http_requests(1);
counter!(
METRIC_HTTP_SERVER_REQUESTS_TOTAL,
LABEL_HTTP_METHOD => method
)
.increment(1);
let span = tracing::info_span!("http-request",
request_id = %request_id,
trace_id = %trace_id,
span_id = %span_id,
status_code = tracing::field::Empty,
method = %request.method(),
peer_addr = %peer_addr,
uri = %redact_sensitive_uri_query(request.uri()),
version = ?request.version(),
user_agent = tracing::field::Empty,
content_type = tracing::field::Empty,
content_length = tracing::field::Empty,
);
if span.is_disabled() {
return span;
}
if let Err(e) = span.set_parent(parent_context) {
debug!(component = LOG_COMPONENT_SERVER, subsystem = LOG_SUBSYSTEM_HTTP, error = ?e, "Failed to propagate tracing context");
}
for (header_name, header_value) in request.headers() {
let value = header_value.to_str().unwrap_or("invalid");
if header_name == "user-agent" {
span.record("user_agent", value);
} else if header_name == "content-type" {
span.record("content_type", value);
} else if header_name == "content-length" {
span.record("content_length", value);
}
}
if let Some(cl) = request.headers().get("content-length")
&& let Some(len) = cl.to_str().ok().and_then(|s| s.parse::<u64>().ok())
{
counter!(METRIC_HTTP_SERVER_REQUEST_BODY_BYTES_TOTAL).increment(len);
histogram!(
METRIC_HTTP_SERVER_REQUEST_BODY_SIZE_BYTES,
span
})
.on_request(|request: &HttpRequest<_>, span: &Span| {
let _enter = span.enter();
trace!("HTTP request started");
let method = request_method_label(request.method());
record_active_http_requests(1);
counter!(
METRIC_HTTP_SERVER_REQUESTS_TOTAL,
LABEL_HTTP_METHOD => method
)
.record(len as f64);
}
})
.on_response(|response: &Response<_>, latency: Duration, span: &Span| {
span.record("status_code", tracing::field::display(response.status()));
let _enter = span.enter();
let status_class = status_class_label(response.status());
record_active_http_requests(-1);
histogram!(
METRIC_HTTP_SERVER_REQUEST_DURATION_SECONDS,
LABEL_HTTP_STATUS_CLASS => status_class
)
.record(latency.as_secs_f64());
if response.status().is_client_error() || response.status().is_server_error() {
.increment(1);
if let Some(cl) = request.headers().get("content-length")
&& let Some(len) = cl.to_str().ok().and_then(|s| s.parse::<u64>().ok())
{
counter!(METRIC_HTTP_SERVER_REQUEST_BODY_BYTES_TOTAL).increment(len);
histogram!(
METRIC_HTTP_SERVER_REQUEST_BODY_SIZE_BYTES,
LABEL_HTTP_METHOD => method
)
.record(len as f64);
}
})
.on_response(trace_on_response)
.on_body_chunk(|chunk: &Bytes, latency: Duration, span: &Span| {
counter!(METRIC_HTTP_SERVER_RESPONSE_BODY_BYTES_TOTAL).increment(chunk.len() as u64);
#[cfg(feature = "tracing-chunk-debug")]
{
let _enter = span.enter();
debug!(chunk_bytes = chunk.len(), duration_ms = duration_ms(latency), "HTTP response body chunk sent");
}
#[cfg(not(feature = "tracing-chunk-debug"))]
{
let _ = (latency, span);
}
})
.on_eos(|_trailers: Option<&HeaderMap>, stream_duration: Duration, span: &Span| {
#[cfg(feature = "tracing-chunk-debug")]
{
let _enter = span.enter();
debug!(duration_ms = duration_ms(stream_duration), "HTTP response stream closed");
}
#[cfg(not(feature = "tracing-chunk-debug"))]
{
let _ = (_trailers, stream_duration, span);
}
})
.on_failure(|error, latency: Duration, span: &Span| {
let _enter = span.enter();
record_active_http_requests(-1);
counter!(
METRIC_HTTP_SERVER_FAILURES_TOTAL,
LABEL_HTTP_STATUS_CLASS => status_class
LABEL_HTTP_STATUS_CLASS => "transport"
)
.increment(1);
}
if let Some(cl) = response.headers().get("content-length")
&& let Some(len) = cl.to_str().ok().and_then(|s| s.parse::<u64>().ok())
{
histogram!(
METRIC_HTTP_SERVER_RESPONSE_BODY_SIZE_BYTES,
LABEL_HTTP_STATUS_CLASS => status_class
trace!(error = ?error, duration_ms = duration_ms(latency), "HTTP request failure captured by trace layer");
}),
)
.layer(RequestLoggingLayer)
.layer(PropagateRequestIdLayer::x_request_id())
.layer(CompressionLayer::new().compress_when(PathAwareHttpCompressionPredicate::new(compression_config.clone())))
.layer(PathCategoryInjectionLayer)
.layer(S3ErrorMessageCompatLayer)
.layer(ObjectAttributesEtagFixLayer)
.layer(ConditionalCorsLayer::new())
.option_layer(if is_console { Some(RedirectLayer) } else { None })
.layer(BodylessStatusFixLayer)
.layer(HeadRequestBodyFixLayer)
.layer(PublicHealthEndpointLayer)
.option_layer((!server_domains_configured && !is_console).then_some(VirtualHostStyleHintLayer))
.service(service)
};
let build_internode_stack = |service| {
ServiceBuilder::new()
.layer(AddExtensionLayer::new(remote_addr))
.option_layer(remote_addr.map(|ra| AddExtensionLayer::new(ra.0)))
.option_layer(trusted_proxy_layer.clone())
.layer(SetRequestIdLayer::x_request_id(MakeRequestUuid))
.layer(RequestContextLayer)
.layer(EmptyBodyContentLengthCompatLayer)
.layer(CatchPanicLayer::new())
.layer(ReadinessGateLayer::new(readiness.clone()))
.layer(KeystoneAuthLayer::new(keystone_auth.clone()))
.layer(
TraceLayer::new_for_http()
.make_span_with(|request: &HttpRequest<_>| {
let request_context = request.extensions().get::<crate::storage::request_context::RequestContext>();
let request_id = request_context
.map(|ctx| ctx.request_id.as_str())
.unwrap_or("unknown");
let trace_id = request_context
.and_then(|ctx| ctx.trace_id.as_deref())
.unwrap_or("unknown");
let span_id = request_context
.and_then(|ctx| ctx.span_id.as_deref())
.unwrap_or("unknown");
let parent_context = global::get_text_map_propagator(|propagator| {
propagator.extract(&HeaderMapCarrier::new(request.headers()))
});
if parent_context.has_active_span() {
let span_ref = parent_context.span();
trace!(
otel_trace_id = %span_ref.span_context().trace_id(),
otel_parent_span_id = %span_ref.span_context().span_id(),
sampled = span_ref.span_context().is_sampled(),
"Extracted trace context from incoming request headers"
);
} else {
trace!("No trace context found in request headers, will create root span");
}
let client_info = request.extensions().get::<ClientInfo>();
let peer_addr = client_info
.map(|info| info.real_ip.to_string())
.or_else(|| request.extensions().get::<RemoteAddr>().map(|addr| addr.0.to_string()))
.unwrap_or_else(|| "unknown".to_string());
let span = tracing::info_span!("http-request",
request_id = %request_id,
trace_id = %trace_id,
span_id = %span_id,
status_code = tracing::field::Empty,
method = %request.method(),
peer_addr = %peer_addr,
uri = %redact_sensitive_uri_query(request.uri()),
version = ?request.version(),
user_agent = tracing::field::Empty,
content_type = tracing::field::Empty,
content_length = tracing::field::Empty,
);
if span.is_disabled() {
return span;
}
if let Err(e) = span.set_parent(parent_context) {
debug!(component = LOG_COMPONENT_SERVER, subsystem = LOG_SUBSYSTEM_HTTP, error = ?e, "Failed to propagate tracing context");
}
for (header_name, header_value) in request.headers() {
let value = header_value.to_str().unwrap_or("invalid");
if header_name == "user-agent" {
span.record("user_agent", value);
} else if header_name == "content-type" {
span.record("content_type", value);
} else if header_name == "content-length" {
span.record("content_length", value);
}
}
span
})
.on_request(|request: &HttpRequest<_>, span: &Span| {
let _enter = span.enter();
trace!("HTTP request started");
let method = request_method_label(request.method());
record_active_http_requests(1);
counter!(
METRIC_HTTP_SERVER_REQUESTS_TOTAL,
LABEL_HTTP_METHOD => method
)
.record(len as f64);
}
})
.on_body_chunk(|chunk: &Bytes, latency: Duration, span: &Span| {
counter!(METRIC_HTTP_SERVER_RESPONSE_BODY_BYTES_TOTAL).increment(chunk.len() as u64);
#[cfg(feature = "tracing-chunk-debug")]
{
.increment(1);
if let Some(cl) = request.headers().get("content-length")
&& let Some(len) = cl.to_str().ok().and_then(|s| s.parse::<u64>().ok())
{
counter!(METRIC_HTTP_SERVER_REQUEST_BODY_BYTES_TOTAL).increment(len);
histogram!(
METRIC_HTTP_SERVER_REQUEST_BODY_SIZE_BYTES,
LABEL_HTTP_METHOD => method
)
.record(len as f64);
}
})
.on_response(trace_on_response)
.on_body_chunk(|chunk: &Bytes, latency: Duration, span: &Span| {
counter!(METRIC_HTTP_SERVER_RESPONSE_BODY_BYTES_TOTAL).increment(chunk.len() as u64);
#[cfg(feature = "tracing-chunk-debug")]
{
let _enter = span.enter();
debug!(chunk_bytes = chunk.len(), duration_ms = duration_ms(latency), "HTTP response body chunk sent");
}
#[cfg(not(feature = "tracing-chunk-debug"))]
{
let _ = (latency, span);
}
})
.on_eos(|_trailers: Option<&HeaderMap>, stream_duration: Duration, span: &Span| {
#[cfg(feature = "tracing-chunk-debug")]
{
let _enter = span.enter();
debug!(duration_ms = duration_ms(stream_duration), "HTTP response stream closed");
}
#[cfg(not(feature = "tracing-chunk-debug"))]
{
let _ = (_trailers, stream_duration, span);
}
})
.on_failure(|error, latency: Duration, span: &Span| {
let _enter = span.enter();
debug!(chunk_bytes = chunk.len(), duration_ms = duration_ms(latency), "HTTP response body chunk sent");
}
#[cfg(not(feature = "tracing-chunk-debug"))]
{
let _ = (latency, span);
}
})
.on_eos(|_trailers: Option<&HeaderMap>, stream_duration: Duration, span: &Span| {
#[cfg(feature = "tracing-chunk-debug")]
{
let _enter = span.enter();
debug!(duration_ms = duration_ms(stream_duration), "HTTP response stream closed");
}
#[cfg(not(feature = "tracing-chunk-debug"))]
{
let _ = (_trailers, stream_duration, span);
}
})
.on_failure(|_error, latency: Duration, span: &Span| {
let _enter = span.enter();
record_active_http_requests(-1);
counter!(
METRIC_HTTP_SERVER_FAILURES_TOTAL,
LABEL_HTTP_STATUS_CLASS => "transport"
)
.increment(1);
trace!(error = ?_error, duration_ms = duration_ms(latency), "HTTP request failure captured by trace layer");
}),
)
.layer(RequestLoggingLayer)
.layer(PropagateRequestIdLayer::x_request_id())
// Compress responses based on whitelist configuration
// Only compresses when enabled and matches configured extensions/MIME types
.layer(CompressionLayer::new().compress_when(PathAwareHttpCompressionPredicate::new(compression_config)))
.layer(PathCategoryInjectionLayer)
.layer(S3ErrorMessageCompatLayer)
.layer(ObjectAttributesEtagFixLayer)
// Conditional CORS layer: only applies to S3 API requests (not Admin, not Console)
// Admin has its own CORS handling in router.rs
// Console has its own CORS layer in setup_console_middleware_stack()
// S3 API uses this system default CORS (RUSTFS_CORS_ALLOWED_ORIGINS)
// Bucket-level CORS takes precedence when configured (handled in router.rs for OPTIONS, and in ecfs.rs for actual requests)
.layer(ConditionalCorsLayer::new())
.option_layer(if is_console { Some(RedirectLayer) } else { None })
// Must run before outer response-transforming layers: clear the body and remove
// Content-Length, Content-Type, and Transfer-Encoding for statuses
// that MUST NOT carry a body (1xx/204/304). Placed inside those
// layers so they see the already-bodyless
// response and so no layer (e.g. CORS) re-adds body headers afterward.
.layer(BodylessStatusFixLayer)
// HEAD responses must not send body bytes even when the inner S3 layer
// serializes an XML error payload.
.layer(HeadRequestBodyFixLayer)
// Health probes are public admin routes, but s3s parses virtual-host
// buckets before custom routes. Handle them here so SERVER_DOMAINS
// cannot turn /health into an S3 bucket request.
.layer(PublicHealthEndpointLayer)
// Virtual-hosted-style S3 requests (the AWS SDK / Terraform default) cannot be
// routed when no server domain is configured: s3s parses them path-style and
// returns an opaque 501. When RUSTFS_SERVER_DOMAINS is unset, return an actionable
// error pointing at the fix. Inert (not installed) once domains are configured.
.option_layer((!server_domains_configured && !is_console).then_some(VirtualHostStyleHintLayer))
.service(service);
record_active_http_requests(-1);
counter!(
METRIC_HTTP_SERVER_FAILURES_TOTAL,
LABEL_HTTP_STATUS_CLASS => "transport"
)
.increment(1);
trace!(error = ?error, duration_ms = duration_ms(latency), "HTTP request failure captured by trace layer");
}),
)
.layer(PropagateRequestIdLayer::x_request_id())
.layer(CompressionLayer::new().compress_when(PathAwareHttpCompressionPredicate::new(compression_config.clone())))
.layer(PathCategoryInjectionLayer)
.layer(S3ErrorMessageCompatLayer)
.layer(ObjectAttributesEtagFixLayer)
.layer(ConditionalCorsLayer::new())
.option_layer(if is_console { Some(RedirectLayer) } else { None })
.layer(BodylessStatusFixLayer)
.layer(HeadRequestBodyFixLayer)
.layer(PublicHealthEndpointLayer)
.option_layer((!server_domains_configured && !is_console).then_some(VirtualHostStyleHintLayer))
.service(service)
};
let external_stack_service = build_external_stack(external_service);
let internode_stack_service = build_internode_stack(internode_service);
let hybrid_service = PathDispatchService::new(external_stack_service, internode_stack_service);
let hybrid_service = TowerToHyperService::new(hybrid_service);
@@ -1397,12 +1622,13 @@ mod tests {
use super::*;
use crate::server::compress::RequestPathCategory;
use bytes::Bytes;
use http::HeaderMap;
use http::Request as HttpRequest;
use http_body_util::Empty;
use http::{HeaderMap, StatusCode};
use http_body_util::{Empty, Full};
use opentelemetry::propagation::Extractor;
use std::convert::Infallible;
use std::future::Ready;
use std::sync::{Arc, Mutex};
use std::task::{Context, Poll};
use tower::{Layer, Service, ServiceBuilder};
@@ -1620,6 +1846,36 @@ mod tests {
}
}
#[derive(Clone)]
struct MarkerService {
name: &'static str,
hits: Arc<Mutex<Vec<&'static str>>>,
}
impl MarkerService {
fn new(name: &'static str, hits: Arc<Mutex<Vec<&'static str>>>) -> Self {
Self { name, hits }
}
}
impl<ReqBody> Service<HttpRequest<ReqBody>> for MarkerService {
type Response = Response<Full<Bytes>>;
type Error = Infallible;
type Future = Ready<std::result::Result<Response<Full<Bytes>>, Infallible>>;
fn poll_ready(&mut self, _cx: &mut Context<'_>) -> Poll<std::result::Result<(), Self::Error>> {
Poll::Ready(Ok(()))
}
fn call(&mut self, _req: HttpRequest<ReqBody>) -> Self::Future {
self.hits.lock().expect("hits").push(self.name);
std::future::ready(Ok(Response::builder()
.status(StatusCode::OK)
.body(Full::from(Bytes::from_static(self.name.as_bytes())))
.expect("response")))
}
}
#[test]
fn test_service_builder_order_regression_for_response_extensions() {
let request = HttpRequest::builder().uri("/bucket/archive.zip").body(()).expect("request");
@@ -1648,4 +1904,21 @@ mod tests {
Some("true")
);
}
#[test]
fn path_dispatch_service_identifies_rpc_prefix() {
let hits = Arc::new(Mutex::new(Vec::new()));
let service =
PathDispatchService::new(MarkerService::new("external", Arc::clone(&hits)), MarkerService::new("internode", hits));
assert!(PathDispatchService::<MarkerService, MarkerService>::is_internode_path(&format!(
"{}/put_file_stream",
crate::server::RPC_PREFIX
)));
assert!(!PathDispatchService::<MarkerService, MarkerService>::is_internode_path(
"/bucket/object.txt"
));
assert!(!PathDispatchService::<MarkerService, MarkerService>::is_internode_path("/rustfs/rpcx"));
let _ = service;
}
}
rustfs/src/storage/rpc/http_service.rs
+10
View File
@@ -38,6 +38,7 @@ use serde_urlencoded::from_bytes;
use std::future::Future;
use std::pin::Pin;
use std::task::{Context, Poll};
use std::time::Instant;
use tokio::io::{self, AsyncWriteExt};
use tokio_util::io::ReaderStream;
use tower::Service;
@@ -280,6 +281,7 @@ fn is_internode_rpc_path(path: &str) -> bool {
async fn handle_internode_rpc(req: Request<Incoming>) -> Response<Body> {
let operation = internode_http_operation(req.uri().path());
let started_at = Instant::now();
if let Err(response) = verify_internode_rpc_signature(req.uri(), req.method(), req.headers()) {
record_internode_rpc_error(operation);
return *response;
@@ -299,6 +301,14 @@ async fn handle_internode_rpc(req: Request<Incoming>) -> Response<Body> {
record_internode_rpc_error(operation);
}
if let Some(operation) = operation {
resolve_internode_metrics().record_duration_for_operation_and_backend(
operation,
INTERNODE_TRANSPORT_BACKEND_TCP_HTTP,
started_at.elapsed(),
);
}
response
}
scripts/fuzz/run.sh
Executable
+82
View File
@@ -0,0 +1,82 @@
#!/bin/sh
# Unified fuzz runner script.
#
# Modes:
# ./scripts/fuzz/run.sh # build + run all smoke targets (60s each)
# BUILD_ONLY=1 ./scripts/fuzz/run.sh # build only, no fuzz run
# FUZZ_TARGET=path_containment ./scripts/fuzz/run.sh # build + run single target
# MAX_TOTAL_TIME=300 ./scripts/fuzz/run.sh # nightly-style 300s per target
#
# Environment variables:
# FUZZ_TARGET — run only this target (default: all smoke targets)
# MAX_TOTAL_TIME — seconds to fuzz per target (default: 60)
# ARTIFACT_ROOT — artifact output directory (default: artifacts)
# BUILD_ONLY — set to 1 to skip fuzz runs (default: 0)
# SKIP_BUILD — set to 1 to skip build phase (default: 0)
# USE_PREBUILT_BINARY — set to 1 to run a prebuilt fuzz binary directly
# PREBUILT_BINARY_DIR — directory containing prebuilt fuzz binaries
set -eu
SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
REPO_ROOT=$(CDPATH= cd -- "$SCRIPT_DIR/../.." && pwd)
FUZZ_DIR="$REPO_ROOT/fuzz"
MAX_TOTAL_TIME=${MAX_TOTAL_TIME:-60}
ARTIFACT_ROOT=${ARTIFACT_ROOT:-artifacts}
FUZZ_TARGET=${FUZZ_TARGET:-}
BUILD_ONLY=${BUILD_ONLY:-0}
SKIP_BUILD=${SKIP_BUILD:-0}
USE_PREBUILT_BINARY=${USE_PREBUILT_BINARY:-0}
PREBUILT_BINARY_DIR=${PREBUILT_BINARY_DIR:-}
cd "$FUZZ_DIR"
mkdir -p "$ARTIFACT_ROOT"
targets="path_containment bucket_validation local_metadata"
if [ -n "$FUZZ_TARGET" ]; then
targets="$FUZZ_TARGET"
fi
# Phase 1: build (unless skipped)
if [ "$SKIP_BUILD" != "1" ]; then
for target in $targets; do
echo "==> cargo +nightly fuzz build $target"
cargo +nightly fuzz build "$target"
done
fi
if [ "$BUILD_ONLY" = "1" ]; then
echo "==> Build-only mode; skipping fuzz runs."
exit 0
fi
# Phase 2: run each target (incremental — no recompilation if already built)
for target in $targets; do
artifact_dir="$ARTIFACT_ROOT/$target"
corpus_dir="$FUZZ_DIR/corpus/$target"
mkdir -p "$artifact_dir"
mkdir -p "$corpus_dir"
if [ "$USE_PREBUILT_BINARY" = "1" ]; then
binary_dir="$PREBUILT_BINARY_DIR"
if [ -z "$binary_dir" ]; then
if [ -n "${CARGO_BUILD_TARGET:-}" ]; then
binary_dir="$FUZZ_DIR/target/${CARGO_BUILD_TARGET}/release"
else
binary_dir="$FUZZ_DIR/target/release"
fi
fi
binary_path="$binary_dir/$target"
if [ ! -x "$binary_path" ]; then
echo "Missing executable prebuilt fuzz binary: $binary_path" >&2
exit 1
fi
echo "==> $binary_path (-max_total_time=$MAX_TOTAL_TIME, -artifact_prefix=$artifact_dir/, corpus=$corpus_dir)"
"$binary_path" -max_total_time="$MAX_TOTAL_TIME" -artifact_prefix="$artifact_dir/" "$corpus_dir"
continue
fi
echo "==> cargo +nightly fuzz run $target (-max_total_time=$MAX_TOTAL_TIME, -artifact_prefix=$artifact_dir/)"
cargo +nightly fuzz run "$target" -- -max_total_time="$MAX_TOTAL_TIME" -artifact_prefix="$artifact_dir/"
done
scripts/fuzz/run_ci_targets.sh
-25
View File
@@ -1,25 +0,0 @@
#!/bin/sh
set -eu
SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
REPO_ROOT=$(CDPATH= cd -- "$SCRIPT_DIR/../.." && pwd)
FUZZ_DIR="$REPO_ROOT/fuzz"
MAX_TOTAL_TIME=${MAX_TOTAL_TIME:-60}
ARTIFACT_ROOT=${ARTIFACT_ROOT:-artifacts}
FUZZ_TARGET=${FUZZ_TARGET:-}
cd "$FUZZ_DIR"
mkdir -p "$ARTIFACT_ROOT"
targets="path_containment bucket_validation local_metadata"
if [ -n "$FUZZ_TARGET" ]; then
targets="$FUZZ_TARGET"
fi
for target in $targets; do
artifact_dir="$ARTIFACT_ROOT/$target"
mkdir -p "$artifact_dir"
echo "==> cargo +nightly fuzz run $target (-max_total_time=$MAX_TOTAL_TIME, -artifact_prefix=$artifact_dir/)"
cargo +nightly fuzz run "$target" -- -max_total_time="$MAX_TOTAL_TIME" -artifact_prefix="$artifact_dir/"
done
scripts/fuzz/run_nightly_targets.sh
-25
View File
@@ -1,25 +0,0 @@
#!/bin/sh
set -eu
SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
REPO_ROOT=$(CDPATH= cd -- "$SCRIPT_DIR/../.." && pwd)
FUZZ_DIR="$REPO_ROOT/fuzz"
MAX_TOTAL_TIME=${MAX_TOTAL_TIME:-300}
ARTIFACT_ROOT=${ARTIFACT_ROOT:-artifacts}
FUZZ_TARGET=${FUZZ_TARGET:-}
cd "$FUZZ_DIR"
mkdir -p "$ARTIFACT_ROOT"
targets="path_containment bucket_validation local_metadata"
if [ -n "$FUZZ_TARGET" ]; then
targets="$FUZZ_TARGET"
fi
for target in $targets; do
artifact_dir="$ARTIFACT_ROOT/$target"
mkdir -p "$artifact_dir"
echo "==> cargo +nightly fuzz run $target (-max_total_time=$MAX_TOTAL_TIME, -artifact_prefix=$artifact_dir/)"
cargo +nightly fuzz run "$target" -- -max_total_time="$MAX_TOTAL_TIME" -artifact_prefix="$artifact_dir/"
done