gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-08-11 23:56:53 +00:00
Code Issues Packages Projects Releases Wiki Activity

perf(server): lighten internode data-plane stack (#3735)

Browse Source 
* refactor(server): split internode dispatch scaffold

* test(server): cover internode dispatch prefix split

* refactor(server): name internode stack boundaries

* perf(server): skip internode request logging layer

* perf(server): skip internode trace layer

* perf(server): use lite internode request context

* feat(metrics): track internode rpc duration

* feat(ecstore): add put object stage summary logs

* test(metrics): update internode descriptor expectations

* fix(server): tighten internode path matching

* fix(pr): address review follow-up comments

* style(ecstore): simplify commit tail duration field

* refactor(ecstore): group put stage summary fields

* refactor(ecstore): inline put stage summary log

* fix(s3): return storage class for object attributes

* merge: sync latest main and resolve object attributes conflict

* fmt

* fix(server): remove duplicate rpc imports

* build(deps): bump memmap2 for RUSTSEC-2026-0186

* fix(s3select): align object_store with datafusion

* chore(deps): prune workspace dependencies

* perf(fuzz): optimize CI runtime with build/run split and matrix parallelization

  Separate fuzz harness compilation from execution to eliminate redundant
  builds across targets. Introduce matrix-based parallel execution for
  PR smoke and nightly fuzz jobs.

  Changes:

  - Split CI workflow into `fuzz-build` (compile once) and matrix run jobs
    (`pr-fuzz-smoke`, `nightly-fuzz-corpus`) that execute targets in parallel
  - Add `BUILD_ONLY` mode to run_ci_targets.sh / run_nightly_targets.sh
  - Add run_single_target.sh for matrix jobs (no build phase)
  - Optimize `local_metadata` fuzz target: reduce prefix iterations from
    8-10 (4 functions each) to 5 critical prefixes (parser-only), cutting
    per-iteration cost by ~3-5x
  - Move archive path validation (`validate_extract_relative_path`,
    `normalize_extract_entry_key`) from `rustfs` to `rustfs-utils::path`,
    eliminating `rustfs` binary crate dependency from fuzz harness
  - Remove `rustfs` from fuzz/Cargo.toml (drops significant transitive deps)
  - Add unit tests for archive path validation in rustfs-utils
  - Update fuzz/README.md with new workflow and script documentation

  Expected CI improvement: PR smoke wall-clock from ~120min (frequent
  timeout) to ~40min; nightly from ~180min to ~60min.

* refactor(fuzz): consolidate scripts and fix prefix test alignment

Replace three duplicated shell scripts (run_ci_targets.sh,
run_nightly_targets.sh, run_single_target.sh) with a single
parameterized run.sh that supports BUILD_ONLY, SKIP_BUILD, and
MAX_TOTAL_TIME environment variables.

Fix local_metadata fuzz target prefix testing: replace always-true
'len > 0' guard with lengths aligned to xl.meta binary layout
(4/5/8/12 bytes for magic+version+header fields). Remove redundant
empty-slice test.

Hoist RUSTFLAGS to workflow top-level env to eliminate per-job
duplication. Update README with unified script documentation.

Net: -118 lines, zero functionality loss.

* perf(fuzz): optimize CI runtime with build/run split and matrix parallelization

Restructure fuzz CI workflow to eliminate redundant compilation and
run targets in parallel via matrix strategy.

Workflow changes:
- Split into fuzz-build (compile once) and matrix run jobs
- PR smoke: 3 targets parallel, 60s each, timeout 30min (was 120min)
- Nightly: 3 targets parallel, 300s each, timeout 60min (was 180min)
- Pass compiled harness via actions/artifact between jobs
- Hoist RUSTFLAGS to workflow top-level env

Script consolidation:
- Replace 3 duplicated scripts with single parameterized run.sh
- Supports BUILD_ONLY, SKIP_BUILD, MAX_TOTAL_TIME env vars

Target optimizations:
- Remove rustfs binary crate from fuzz dependencies (was pulling
  979 transitive deps); move archive path validation to rustfs-utils
- Optimize local_metadata: reduce prefix iterations from 8-10x4
  calls to 5 prefixes with parser-only (no decompress), aligned
  with xl.meta binary layout (4/5/8/12 bytes)
- Add unit tests for archive path validation in rustfs-utils
- Update fuzz/README.md with unified script documentation

Expected: PR smoke wall-clock from ~120min (frequent timeout)
to ~40min; nightly from ~180min to ~60min.

* fix(rpc): resolve internode metrics via app context

* fmt

* ci: speed up fuzz smoke artifact restore

---------

Signed-off-by: houseme <housemecn@gmail.com>
This commit is contained in:
houseme
2026-06-23 21:36:39 +08:00
committed by GitHub
parent 7bdb25ae9d
commit 0a00d8d500
23 changed files with 1103 additions and 5816 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/ecstore/src/set_disk.rs
+56 -22
View File
@@ -148,6 +148,7 @@ const EVENT_SET_DISK_MULTIPART: &str = "set_disk_multipart";
const EVENT_SET_DISK_WRITE: &str = "set_disk_write";
const EVENT_SET_DISK_HEAL: &str = "set_disk_heal";
const EVENT_SET_DISK_COMMIT_TAIL_SLOW: &str = "set_disk_commit_tail_slow";
const EVENT_SET_DISK_PUT_OBJECT_STAGE_SUMMARY: &str = "set_disk_put_object_stage_summary";
const SET_DISK_COMMIT_TAIL_WARN_THRESHOLD_MS: u128 = 5_000;
use crate::rio::{EtagResolvable, HashReader, HashReaderMut, TryGetIndex as _};
@@ -1138,10 +1139,8 @@ impl rustfs_storage_api::ObjectIO for SetDisks {
writers.push(w);
errors.push(e);
}
rustfs_io_metrics::record_put_object_stage_duration(
"set_disk_writer_setup",
writer_setup_stage_start.elapsed().as_secs_f64() * 1000.0,
);
let writer_setup_ms = writer_setup_stage_start.elapsed().as_millis() as u64;
rustfs_io_metrics::record_put_object_stage_duration("set_disk_writer_setup", writer_setup_ms as f64);
let nil_count = errors.iter().filter(|&e| e.is_none()).count();
if nil_count < write_quorum {
@@ -1202,10 +1201,8 @@ impl rustfs_storage_api::ObjectIO for SetDisks {
}
},
};
rustfs_io_metrics::record_put_object_stage_duration(
"set_disk_encode",
encode_stage_start.elapsed().as_secs_f64() * 1000.0,
);
let encode_ms = encode_stage_start.elapsed().as_millis() as u64;
rustfs_io_metrics::record_put_object_stage_duration("set_disk_encode", encode_ms as f64);
let _ = mem::replace(&mut data.stream, reader);
// if let Err(err) = close_bitrot_writers(&mut writers).await {
@@ -1314,12 +1311,9 @@ impl rustfs_storage_api::ObjectIO for SetDisks {
write_quorum,
)
.await?;
rustfs_io_metrics::record_put_object_stage_duration(
"set_disk_rename",
rename_stage_start.elapsed().as_secs_f64() * 1000.0,
);
let rename_stage_ms = rename_stage_start.elapsed().as_millis();
if rename_stage_ms >= SET_DISK_COMMIT_TAIL_WARN_THRESHOLD_MS {
let rename_stage_ms = rename_stage_start.elapsed().as_millis() as u64;
rustfs_io_metrics::record_put_object_stage_duration("set_disk_rename", rename_stage_ms as f64);
if (rename_stage_ms as u128) >= SET_DISK_COMMIT_TAIL_WARN_THRESHOLD_MS {
warn!(
event = EVENT_SET_DISK_COMMIT_TAIL_SLOW,
component = LOG_COMPONENT_ECSTORE,
@@ -1328,23 +1322,22 @@ impl rustfs_storage_api::ObjectIO for SetDisks {
bucket = %bucket,
object = %object,
tmp_dir = %tmp_dir,
duration_ms = rename_stage_ms as u64,
duration_ms = { rename_stage_ms },
write_quorum,
state = "slow",
"SetDisk commit tail stage is slow"
);
}
let mut cleanup_stage_ms: Option<u64> = None;
if let Some(old_dir) = op_old_dir {
let cleanup_stage_start = Instant::now();
self.commit_rename_data_dir(&cleanup_disks, bucket, object, &old_dir.to_string(), write_quorum)
.await?;
rustfs_io_metrics::record_put_object_stage_duration(
"set_disk_old_data_cleanup",
cleanup_stage_start.elapsed().as_secs_f64() * 1000.0,
);
let cleanup_stage_ms = cleanup_stage_start.elapsed().as_millis();
if cleanup_stage_ms >= SET_DISK_COMMIT_TAIL_WARN_THRESHOLD_MS {
let cleanup_ms = cleanup_stage_start.elapsed().as_millis() as u64;
cleanup_stage_ms = Some(cleanup_ms);
rustfs_io_metrics::record_put_object_stage_duration("set_disk_old_data_cleanup", cleanup_ms as f64);
if (cleanup_ms as u128) >= SET_DISK_COMMIT_TAIL_WARN_THRESHOLD_MS {
warn!(
event = EVENT_SET_DISK_COMMIT_TAIL_SLOW,
component = LOG_COMPONENT_ECSTORE,
@@ -1354,7 +1347,7 @@ impl rustfs_storage_api::ObjectIO for SetDisks {
object = %object,
tmp_dir = %tmp_dir,
old_dir = %old_dir,
duration_ms = cleanup_stage_ms as u64,
duration_ms = cleanup_ms,
write_quorum,
state = "slow",
"SetDisk commit tail stage is slow"
@@ -1411,10 +1404,51 @@ impl rustfs_storage_api::ObjectIO for SetDisks {
);
}
if issue3031_diag_enabled() {
warn!(
event = EVENT_SET_DISK_PUT_OBJECT_STAGE_SUMMARY,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_SET_DISK,
bucket = %bucket,
object = %object,
write_quorum,
write_path = write_path.metric_label(),
writer_setup_ms,
encode_ms,
rename_ms = rename_stage_ms,
cleanup_ms = cleanup_stage_ms.unwrap_or_default(),
cleanup_present = cleanup_stage_ms.is_some(),
commit_tail_ms = total_commit_tail_ms as u64,
result = "success",
"SetDisk put_object stage summary"
);
}
Ok(ObjectInfo::from_file_info(&fi, bucket, object, opts.versioned || opts.version_suspended))
}
.await;
if issue3031_diag_enabled()
&& let Err(err) = &result
{
let stage_hint = if err.to_string().contains("not enough disks to write") {
"writer_setup_or_quorum"
} else {
"unknown"
};
warn!(
event = EVENT_SET_DISK_PUT_OBJECT_STAGE_SUMMARY,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_SET_DISK,
bucket = %bucket,
object = %object,
result = "error",
stage_hint,
error = %err,
"SetDisk put_object stage summary"
);
}
if issue3031_diag_enabled() {
warn!(
target: "rustfs_ecstore::set_disk",