perf(server): lighten internode data-plane stack (#3735)

* refactor(server): split internode dispatch scaffold * test(server): cover internode dispatch prefix split * refactor(server): name internode stack boundaries * perf(server): skip internode request logging layer * perf(server): skip internode trace layer * perf(server): use lite internode request context * feat(metrics): track internode rpc duration * feat(ecstore): add put object stage summary logs * test(metrics): update internode descriptor expectations * fix(server): tighten internode path matching * fix(pr): address review follow-up comments * style(ecstore): simplify commit tail duration field * refactor(ecstore): group put stage summary fields * refactor(ecstore): inline put stage summary log * fix(s3): return storage class for object attributes * merge: sync latest main and resolve object attributes conflict * fmt * fix(server): remove duplicate rpc imports * build(deps): bump memmap2 for RUSTSEC-2026-0186 * fix(s3select): align object_store with datafusion * chore(deps): prune workspace dependencies * perf(fuzz): optimize CI runtime with build/run split and matrix parallelization Separate fuzz harness compilation from execution to eliminate redundant builds across targets. Introduce matrix-based parallel execution for PR smoke and nightly fuzz jobs. Changes: - Split CI workflow into `fuzz-build` (compile once) and matrix run jobs (`pr-fuzz-smoke`, `nightly-fuzz-corpus`) that execute targets in parallel - Add `BUILD_ONLY` mode to run_ci_targets.sh / run_nightly_targets.sh - Add run_single_target.sh for matrix jobs (no build phase) - Optimize `local_metadata` fuzz target: reduce prefix iterations from 8-10 (4 functions each) to 5 critical prefixes (parser-only), cutting per-iteration cost by ~3-5x - Move archive path validation (`validate_extract_relative_path`, `normalize_extract_entry_key`) from `rustfs` to `rustfs-utils::path`, eliminating `rustfs` binary crate dependency from fuzz harness - Remove `rustfs` from fuzz/Cargo.toml (drops significant transitive deps) - Add unit tests for archive path validation in rustfs-utils - Update fuzz/README.md with new workflow and script documentation Expected CI improvement: PR smoke wall-clock from ~120min (frequent timeout) to ~40min; nightly from ~180min to ~60min. * refactor(fuzz): consolidate scripts and fix prefix test alignment Replace three duplicated shell scripts (run_ci_targets.sh, run_nightly_targets.sh, run_single_target.sh) with a single parameterized run.sh that supports BUILD_ONLY, SKIP_BUILD, and MAX_TOTAL_TIME environment variables. Fix local_metadata fuzz target prefix testing: replace always-true 'len > 0' guard with lengths aligned to xl.meta binary layout (4/5/8/12 bytes for magic+version+header fields). Remove redundant empty-slice test. Hoist RUSTFLAGS to workflow top-level env to eliminate per-job duplication. Update README with unified script documentation. Net: -118 lines, zero functionality loss. * perf(fuzz): optimize CI runtime with build/run split and matrix parallelization Restructure fuzz CI workflow to eliminate redundant compilation and run targets in parallel via matrix strategy. Workflow changes: - Split into fuzz-build (compile once) and matrix run jobs - PR smoke: 3 targets parallel, 60s each, timeout 30min (was 120min) - Nightly: 3 targets parallel, 300s each, timeout 60min (was 180min) - Pass compiled harness via actions/artifact between jobs - Hoist RUSTFLAGS to workflow top-level env Script consolidation: - Replace 3 duplicated scripts with single parameterized run.sh - Supports BUILD_ONLY, SKIP_BUILD, MAX_TOTAL_TIME env vars Target optimizations: - Remove rustfs binary crate from fuzz dependencies (was pulling 979 transitive deps); move archive path validation to rustfs-utils - Optimize local_metadata: reduce prefix iterations from 8-10x4 calls to 5 prefixes with parser-only (no decompress), aligned with xl.meta binary layout (4/5/8/12 bytes) - Add unit tests for archive path validation in rustfs-utils - Update fuzz/README.md with unified script documentation Expected: PR smoke wall-clock from ~120min (frequent timeout) to ~40min; nightly from ~180min to ~60min. * fix(rpc): resolve internode metrics via app context * fmt * ci: speed up fuzz smoke artifact restore --------- Signed-off-by: houseme <housemecn@gmail.com>
2026-08-10 23:26:53 +00:00 · 2026-06-23 21:36:39 +08:00
parent 7bdb25ae9d
commit 0a00d8d500
23 changed files with 1103 additions and 5816 deletions
@@ -110,7 +110,6 @@ memmap2 = { workspace = true }
 libc.workspace = true
 rustix = { workspace = true }
 rustfs-madmin.workspace = true
-rustfs-concurrency.workspace = true
 reqwest = { workspace = true }
 aes-gcm.workspace = true
 chacha20poly1305.workspace = true
@@ -148,6 +148,7 @@ const EVENT_SET_DISK_MULTIPART: &str = "set_disk_multipart";
 const EVENT_SET_DISK_WRITE: &str = "set_disk_write";
 const EVENT_SET_DISK_HEAL: &str = "set_disk_heal";
 const EVENT_SET_DISK_COMMIT_TAIL_SLOW: &str = "set_disk_commit_tail_slow";
+const EVENT_SET_DISK_PUT_OBJECT_STAGE_SUMMARY: &str = "set_disk_put_object_stage_summary";
 const SET_DISK_COMMIT_TAIL_WARN_THRESHOLD_MS: u128 = 5_000;

 use crate::rio::{EtagResolvable, HashReader, HashReaderMut, TryGetIndex as _};
@@ -1138,10 +1139,8 @@ impl rustfs_storage_api::ObjectIO for SetDisks {
                writers.push(w);
                errors.push(e);
            }
-            rustfs_io_metrics::record_put_object_stage_duration(
-                "set_disk_writer_setup",
-                writer_setup_stage_start.elapsed().as_secs_f64() * 1000.0,
-            );
+            let writer_setup_ms = writer_setup_stage_start.elapsed().as_millis() as u64;
+            rustfs_io_metrics::record_put_object_stage_duration("set_disk_writer_setup", writer_setup_ms as f64);

            let nil_count = errors.iter().filter(|&e| e.is_none()).count();
            if nil_count < write_quorum {
@@ -1202,10 +1201,8 @@ impl rustfs_storage_api::ObjectIO for SetDisks {
                    }
                },
            };
-            rustfs_io_metrics::record_put_object_stage_duration(
-                "set_disk_encode",
-                encode_stage_start.elapsed().as_secs_f64() * 1000.0,
-            );
+            let encode_ms = encode_stage_start.elapsed().as_millis() as u64;
+            rustfs_io_metrics::record_put_object_stage_duration("set_disk_encode", encode_ms as f64);

            let _ = mem::replace(&mut data.stream, reader);
            // if let Err(err) = close_bitrot_writers(&mut writers).await {
@@ -1314,12 +1311,9 @@ impl rustfs_storage_api::ObjectIO for SetDisks {
                write_quorum,
            )
            .await?;
-            rustfs_io_metrics::record_put_object_stage_duration(
-                "set_disk_rename",
-                rename_stage_start.elapsed().as_secs_f64() * 1000.0,
-            );
-            let rename_stage_ms = rename_stage_start.elapsed().as_millis();
-            if rename_stage_ms >= SET_DISK_COMMIT_TAIL_WARN_THRESHOLD_MS {
+            let rename_stage_ms = rename_stage_start.elapsed().as_millis() as u64;
+            rustfs_io_metrics::record_put_object_stage_duration("set_disk_rename", rename_stage_ms as f64);
+            if (rename_stage_ms as u128) >= SET_DISK_COMMIT_TAIL_WARN_THRESHOLD_MS {
                warn!(
                    event = EVENT_SET_DISK_COMMIT_TAIL_SLOW,
                    component = LOG_COMPONENT_ECSTORE,
@@ -1328,23 +1322,22 @@ impl rustfs_storage_api::ObjectIO for SetDisks {
                    bucket = %bucket,
                    object = %object,
                    tmp_dir = %tmp_dir,
-                    duration_ms = rename_stage_ms as u64,
+                    duration_ms = { rename_stage_ms },
                    write_quorum,
                    state = "slow",
                    "SetDisk commit tail stage is slow"
                );
            }

+            let mut cleanup_stage_ms: Option<u64> = None;
            if let Some(old_dir) = op_old_dir {
                let cleanup_stage_start = Instant::now();
                self.commit_rename_data_dir(&cleanup_disks, bucket, object, &old_dir.to_string(), write_quorum)
                    .await?;
-                rustfs_io_metrics::record_put_object_stage_duration(
-                    "set_disk_old_data_cleanup",
-                    cleanup_stage_start.elapsed().as_secs_f64() * 1000.0,
-                );
-                let cleanup_stage_ms = cleanup_stage_start.elapsed().as_millis();
-                if cleanup_stage_ms >= SET_DISK_COMMIT_TAIL_WARN_THRESHOLD_MS {
+                let cleanup_ms = cleanup_stage_start.elapsed().as_millis() as u64;
+                cleanup_stage_ms = Some(cleanup_ms);
+                rustfs_io_metrics::record_put_object_stage_duration("set_disk_old_data_cleanup", cleanup_ms as f64);
+                if (cleanup_ms as u128) >= SET_DISK_COMMIT_TAIL_WARN_THRESHOLD_MS {
                    warn!(
                        event = EVENT_SET_DISK_COMMIT_TAIL_SLOW,
                        component = LOG_COMPONENT_ECSTORE,
@@ -1354,7 +1347,7 @@ impl rustfs_storage_api::ObjectIO for SetDisks {
                        object = %object,
                        tmp_dir = %tmp_dir,
                        old_dir = %old_dir,
-                        duration_ms = cleanup_stage_ms as u64,
+                        duration_ms = cleanup_ms,
                        write_quorum,
                        state = "slow",
                        "SetDisk commit tail stage is slow"
@@ -1411,10 +1404,51 @@ impl rustfs_storage_api::ObjectIO for SetDisks {
                );
            }

+            if issue3031_diag_enabled() {
+                warn!(
+                    event = EVENT_SET_DISK_PUT_OBJECT_STAGE_SUMMARY,
+                    component = LOG_COMPONENT_ECSTORE,
+                    subsystem = LOG_SUBSYSTEM_SET_DISK,
+                    bucket = %bucket,
+                    object = %object,
+                    write_quorum,
+                    write_path = write_path.metric_label(),
+                    writer_setup_ms,
+                    encode_ms,
+                    rename_ms = rename_stage_ms,
+                    cleanup_ms = cleanup_stage_ms.unwrap_or_default(),
+                    cleanup_present = cleanup_stage_ms.is_some(),
+                    commit_tail_ms = total_commit_tail_ms as u64,
+                    result = "success",
+                    "SetDisk put_object stage summary"
+                );
+            }
+
            Ok(ObjectInfo::from_file_info(&fi, bucket, object, opts.versioned || opts.version_suspended))
        }
        .await;

+        if issue3031_diag_enabled()
+            && let Err(err) = &result
+        {
+            let stage_hint = if err.to_string().contains("not enough disks to write") {
+                "writer_setup_or_quorum"
+            } else {
+                "unknown"
+            };
+            warn!(
+                event = EVENT_SET_DISK_PUT_OBJECT_STAGE_SUMMARY,
+                component = LOG_COMPONENT_ECSTORE,
+                subsystem = LOG_SUBSYSTEM_SET_DISK,
+                bucket = %bucket,
+                object = %object,
+                result = "error",
+                stage_hint,
+                error = %err,
+                "SetDisk put_object stage summary"
+            );
+        }
+
        if issue3031_diag_enabled() {
            warn!(
                target: "rustfs_ecstore::set_disk",
@@ -38,6 +38,7 @@ const INTERNODE_OPERATION_RECV_BYTES_TOTAL: &str = "rustfs_system_network_intern
 const INTERNODE_OPERATION_REQUESTS_OUTGOING_TOTAL: &str = "rustfs_system_network_internode_operation_requests_outgoing_total";
 const INTERNODE_OPERATION_REQUESTS_INCOMING_TOTAL: &str = "rustfs_system_network_internode_operation_requests_incoming_total";
 const INTERNODE_OPERATION_ERRORS_TOTAL: &str = "rustfs_system_network_internode_operation_errors_total";
+const INTERNODE_OPERATION_DURATION_MS: &str = "rustfs_system_network_internode_operation_duration_ms";
 const INTERNODE_OPERATION_CLASSIFIED_ERRORS_TOTAL: &str = "rustfs_system_network_internode_operation_classified_errors_total";
 const INTERNODE_OPERATION_RETRIES_TOTAL: &str = "rustfs_system_network_internode_operation_retries_total";
 const INTERNODE_OPERATION_RETRY_SUCCESSES_TOTAL: &str = "rustfs_system_network_internode_operation_retry_successes_total";
@@ -74,6 +75,10 @@ pub const INTERNODE_OPERATION_METRICS: &[InternodeOperationMetricDescriptor] = &
        name: INTERNODE_OPERATION_ERRORS_TOTAL,
        labels: OPERATION_BACKEND_LABELS,
    },
+    InternodeOperationMetricDescriptor {
+        name: INTERNODE_OPERATION_DURATION_MS,
+        labels: OPERATION_BACKEND_LABELS,
+    },
    InternodeOperationMetricDescriptor {
        name: INTERNODE_OPERATION_CLASSIFIED_ERRORS_TOTAL,
        labels: OPERATION_BACKEND_CLASSIFICATION_LABELS,
@@ -208,6 +213,12 @@ impl InternodeMetrics {
        counter!(INTERNODE_OPERATION_ERRORS_TOTAL, OPERATION_LABEL => operation, BACKEND_LABEL => backend).increment(1);
    }

+    pub fn record_duration_for_operation_and_backend(&self, operation: &'static str, backend: &'static str, duration: Duration) {
+        let duration_ms = duration.as_secs_f64() * 1000.0;
+        metrics::histogram!(INTERNODE_OPERATION_DURATION_MS, OPERATION_LABEL => operation, BACKEND_LABEL => backend)
+            .record(duration_ms);
+    }
+
    pub fn record_classified_error_for_operation_and_backend(
        &self,
        operation: &'static str,
@@ -382,14 +393,14 @@ mod tests {

    #[test]
    fn operation_metric_descriptors_include_backend_and_operation_labels() {
-        assert_eq!(INTERNODE_OPERATION_METRICS.len(), 9);
-        for metric in &INTERNODE_OPERATION_METRICS[..5] {
+        assert_eq!(INTERNODE_OPERATION_METRICS.len(), 10);
+        for metric in &INTERNODE_OPERATION_METRICS[..6] {
            assert_eq!(metric.labels, &[OPERATION_LABEL, BACKEND_LABEL]);
        }
-        for metric in &INTERNODE_OPERATION_METRICS[5..8] {
+        for metric in &INTERNODE_OPERATION_METRICS[6..9] {
            assert_eq!(metric.labels, &[OPERATION_LABEL, BACKEND_LABEL, CLASSIFICATION_LABEL]);
        }
-        assert_eq!(INTERNODE_OPERATION_METRICS[8].labels, &[STAGE_LABEL, DOMINANT_ERROR_LABEL]);
+        assert_eq!(INTERNODE_OPERATION_METRICS[9].labels, &[STAGE_LABEL, DOMINANT_ERROR_LABEL]);
    }

    #[test]
@@ -406,18 +417,22 @@ mod tests {

        assert_eq!(
            INTERNODE_OPERATION_METRICS[5].name,
-            "rustfs_system_network_internode_operation_classified_errors_total"
+            "rustfs_system_network_internode_operation_duration_ms"
        );
        assert_eq!(
            INTERNODE_OPERATION_METRICS[6].name,
-            "rustfs_system_network_internode_operation_retries_total"
+            "rustfs_system_network_internode_operation_classified_errors_total"
        );
        assert_eq!(
            INTERNODE_OPERATION_METRICS[7].name,
-            "rustfs_system_network_internode_operation_retry_successes_total"
+            "rustfs_system_network_internode_operation_retries_total"
        );
        assert_eq!(
            INTERNODE_OPERATION_METRICS[8].name,
+            "rustfs_system_network_internode_operation_retry_successes_total"
+        );
+        assert_eq!(
+            INTERNODE_OPERATION_METRICS[9].name,
            "rustfs_system_storage_erasure_write_quorum_failures_total"
        );
    }
@@ -43,7 +43,6 @@ aes-gcm = { workspace = true }
 argon2 = { workspace = true }
 chacha20poly1305 = { workspace = true }
 rand = { workspace = true }
-sha2 = { workspace = true }
 base64 = { workspace = true }
 zeroize = { workspace = true, features = ["derive"] }

@@ -60,10 +60,8 @@ quick-xml = { workspace = true, features = ["serialize", "serde-types", "encodin
 tokio = { workspace = true, features = ["test-util"] }
 tracing-subscriber = { workspace = true, features = ["env-filter"] }
 axum = { workspace = true }
-rustfs-storage-api = { workspace = true }
 rustfs-utils = { workspace = true, features = ["path"] }
 serde_json = { workspace = true }
-time = { workspace = true }
 criterion = { workspace = true }

 [lints]
@@ -37,7 +37,6 @@ rustfs-storage-api.workspace = true
 futures = { workspace = true }
 futures-core = { workspace = true }
 http.workspace = true
-object_store = { workspace = true }
 pin-project-lite.workspace = true
 s3s.workspace = true
 serde_json = { workspace = true }
@@ -20,14 +20,14 @@ use crate::{
 use async_trait::async_trait;
 use bytes::Bytes;
 use chrono::Utc;
+use datafusion::object_store::{
+    Attributes, CopyOptions, Error as o_Error, GetOptions, GetRange, GetResult, GetResultPayload, ListResult, MultipartUpload,
+    ObjectMeta, ObjectStore, PutMultipartOptions, PutOptions, PutPayload, PutResult, Result, path::Path,
+};
 use futures::pin_mut;
 use futures::{Stream, StreamExt, future::ready, stream};
 use futures_core::stream::BoxStream;
 use http::{HeaderMap, HeaderValue, header::HeaderName};
-use object_store::{
-    Attributes, CopyOptions, Error as o_Error, GetOptions, GetRange, GetResult, GetResultPayload, ListResult, MultipartUpload,
-    ObjectMeta, ObjectStore, PutMultipartOptions, PutOptions, PutPayload, PutResult, Result, path::Path,
-};
 use pin_project_lite::pin_project;
 use rustfs_common::DEFAULT_DELIMITER;
 use rustfs_storage_api::{HTTPRangeSpec, ObjectIO as _, ObjectOperations as _};
@@ -1080,8 +1080,8 @@ mod test {
        scan_range_read_start, scan_range_stream, select_read_headers,
    };
    use bytes::Bytes;
+    use datafusion::object_store::{self, GetRange};
    use futures::{StreamExt, TryStreamExt, stream};
-    use object_store::GetRange;
    use s3s::dto::{
        CSVInput, CSVOutput, ExpressionType, InputSerialization, OutputSerialization, SelectObjectContentInput,
        SelectObjectContentRequest,
@@ -21,10 +21,10 @@ use datafusion::{
        record_batch::RecordBatch,
    },
    execution::{SessionStateBuilder, context::SessionState, runtime_env::RuntimeEnvBuilder},
+    object_store::{ObjectStore, ObjectStoreExt, memory::InMemory, path::Path},
    parquet::arrow::ArrowWriter,
    prelude::SessionContext,
 };
-use object_store::{ObjectStore, ObjectStoreExt, memory::InMemory, path::Path};
 use std::sync::Arc;
 use tracing::error;

@@ -110,7 +110,7 @@ impl SessionCtxFactory {
                QueryError::StoreError { e: e.to_string() }
            })?;

-            df_session_state.with_object_store(&store_url, Arc::new(store)).build()
+            df_session_state.with_object_store(&store_url, store).build()
        } else {
            let store: EcObjectStore =
                EcObjectStore::new(context.input.clone()).map_err(|_| QueryError::NotImplemented { err: String::new() })?;
@@ -12,6 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.

+use std::path::Component;
 use std::path::Path;
 use std::path::PathBuf;

@@ -550,6 +551,52 @@ pub fn trim_etag(etag: &str) -> String {
    etag.trim_matches('"').to_string()
 }

+/// Returns `true` if `path` contains a `..` component (parent directory traversal).
+fn contains_parent_dir_component(path: &str) -> bool {
+    path.split(['/', '\\']).any(|component| component == "..")
+}
+
+/// Validates that an archive entry path does not escape the target bucket.
+///
+/// Rejects paths containing `..` components, root prefixes, or device/prefix
+/// components that would allow path traversal outside the extraction root.
+///
+/// Returns `Ok(())` if the path is safe, or `Err(message)` describing the violation.
+pub fn validate_extract_relative_path(path: &str) -> Result<(), String> {
+    let p = Path::new(path);
+    if p.components()
+        .any(|c| matches!(c, Component::Prefix(_) | Component::RootDir | Component::ParentDir))
+        || contains_parent_dir_component(p.to_string_lossy().as_ref())
+    {
+        return Err("archive entry path must stay within the target bucket".to_string());
+    }
+    Ok(())
+}
+
+/// Normalizes an archive entry key by applying a prefix, trimming slashes,
+/// and ensuring directory entries end with `/`.
+///
+/// Validates both the raw path and the final key against path traversal.
+///
+/// Returns `Ok(normalized_key)` or `Err(message)` if the path is unsafe.
+pub fn normalize_extract_entry_key(path: &str, prefix: Option<&str>, is_dir: bool) -> Result<String, String> {
+    validate_extract_relative_path(path)?;
+    let path = path.trim_matches('/');
+    let mut key = match prefix {
+        Some(prefix) if !path.is_empty() => format!("{prefix}/{path}"),
+        Some(prefix) => prefix.to_string(),
+        None => path.to_string(),
+    };
+
+    if is_dir && !key.ends_with('/') {
+        key.push('/');
+    }
+
+    validate_extract_relative_path(&key)?;
+
+    Ok(key)
+}
+
 #[cfg(test)]
 mod tests {
    use super::*;
@@ -918,4 +965,36 @@ mod tests {
            }
        }
    }
+
+    #[test]
+    fn test_validate_extract_relative_path_accepts_safe_paths() {
+        assert!(validate_extract_relative_path("file.txt").is_ok());
+        assert!(validate_extract_relative_path("dir/file.txt").is_ok());
+        assert!(validate_extract_relative_path("a/b/c").is_ok());
+        assert!(validate_extract_relative_path("").is_ok());
+    }
+
+    #[test]
+    fn test_validate_extract_relative_path_rejects_traversal() {
+        assert!(validate_extract_relative_path("../escape.txt").is_err());
+        assert!(validate_extract_relative_path("dir/../../../escape.txt").is_err());
+        assert!(validate_extract_relative_path("/absolute/path").is_err());
+        assert!(validate_extract_relative_path("dir/..").is_err());
+        assert!(validate_extract_relative_path("..").is_err());
+    }
+
+    #[test]
+    fn test_normalize_extract_entry_key_basic() {
+        assert_eq!(normalize_extract_entry_key("file.txt", None, false).unwrap(), "file.txt");
+        assert_eq!(normalize_extract_entry_key("file.txt", Some("prefix"), false).unwrap(), "prefix/file.txt");
+        assert_eq!(normalize_extract_entry_key("dir", None, true).unwrap(), "dir/");
+        assert_eq!(normalize_extract_entry_key("", Some("prefix"), false).unwrap(), "prefix");
+    }
+
+    #[test]
+    fn test_normalize_extract_entry_key_rejects_traversal() {
+        assert!(normalize_extract_entry_key("../escape.txt", None, false).is_err());
+        assert!(normalize_extract_entry_key("file.txt", Some("../bad"), false).is_err());
+        assert!(normalize_extract_entry_key("/absolute", None, false).is_err());
+    }
 }