Files
rustfs/Cargo.toml
T
houseme 0a00d8d500 perf(server): lighten internode data-plane stack (#3735)
* refactor(server): split internode dispatch scaffold

* test(server): cover internode dispatch prefix split

* refactor(server): name internode stack boundaries

* perf(server): skip internode request logging layer

* perf(server): skip internode trace layer

* perf(server): use lite internode request context

* feat(metrics): track internode rpc duration

* feat(ecstore): add put object stage summary logs

* test(metrics): update internode descriptor expectations

* fix(server): tighten internode path matching

* fix(pr): address review follow-up comments

* style(ecstore): simplify commit tail duration field

* refactor(ecstore): group put stage summary fields

* refactor(ecstore): inline put stage summary log

* fix(s3): return storage class for object attributes

* merge: sync latest main and resolve object attributes conflict

* fmt

* fix(server): remove duplicate rpc imports

* build(deps): bump memmap2 for RUSTSEC-2026-0186

* fix(s3select): align object_store with datafusion

* chore(deps): prune workspace dependencies

* perf(fuzz): optimize CI runtime with build/run split and matrix parallelization

  Separate fuzz harness compilation from execution to eliminate redundant
  builds across targets. Introduce matrix-based parallel execution for
  PR smoke and nightly fuzz jobs.

  Changes:

  - Split CI workflow into `fuzz-build` (compile once) and matrix run jobs
    (`pr-fuzz-smoke`, `nightly-fuzz-corpus`) that execute targets in parallel
  - Add `BUILD_ONLY` mode to run_ci_targets.sh / run_nightly_targets.sh
  - Add run_single_target.sh for matrix jobs (no build phase)
  - Optimize `local_metadata` fuzz target: reduce prefix iterations from
    8-10 (4 functions each) to 5 critical prefixes (parser-only), cutting
    per-iteration cost by ~3-5x
  - Move archive path validation (`validate_extract_relative_path`,
    `normalize_extract_entry_key`) from `rustfs` to `rustfs-utils::path`,
    eliminating `rustfs` binary crate dependency from fuzz harness
  - Remove `rustfs` from fuzz/Cargo.toml (drops significant transitive deps)
  - Add unit tests for archive path validation in rustfs-utils
  - Update fuzz/README.md with new workflow and script documentation

  Expected CI improvement: PR smoke wall-clock from ~120min (frequent
  timeout) to ~40min; nightly from ~180min to ~60min.

* refactor(fuzz): consolidate scripts and fix prefix test alignment

Replace three duplicated shell scripts (run_ci_targets.sh,
run_nightly_targets.sh, run_single_target.sh) with a single
parameterized run.sh that supports BUILD_ONLY, SKIP_BUILD, and
MAX_TOTAL_TIME environment variables.

Fix local_metadata fuzz target prefix testing: replace always-true
'len > 0' guard with lengths aligned to xl.meta binary layout
(4/5/8/12 bytes for magic+version+header fields). Remove redundant
empty-slice test.

Hoist RUSTFLAGS to workflow top-level env to eliminate per-job
duplication. Update README with unified script documentation.

Net: -118 lines, zero functionality loss.

* perf(fuzz): optimize CI runtime with build/run split and matrix parallelization

Restructure fuzz CI workflow to eliminate redundant compilation and
run targets in parallel via matrix strategy.

Workflow changes:
- Split into fuzz-build (compile once) and matrix run jobs
- PR smoke: 3 targets parallel, 60s each, timeout 30min (was 120min)
- Nightly: 3 targets parallel, 300s each, timeout 60min (was 180min)
- Pass compiled harness via actions/artifact between jobs
- Hoist RUSTFLAGS to workflow top-level env

Script consolidation:
- Replace 3 duplicated scripts with single parameterized run.sh
- Supports BUILD_ONLY, SKIP_BUILD, MAX_TOTAL_TIME env vars

Target optimizations:
- Remove rustfs binary crate from fuzz dependencies (was pulling
  979 transitive deps); move archive path validation to rustfs-utils
- Optimize local_metadata: reduce prefix iterations from 8-10x4
  calls to 5 prefixes with parser-only (no decompress), aligned
  with xl.meta binary layout (4/5/8/12 bytes)
- Add unit tests for archive path validation in rustfs-utils
- Update fuzz/README.md with unified script documentation

Expected: PR smoke wall-clock from ~120min (frequent timeout)
to ~40min; nightly from ~180min to ~60min.

* fix(rpc): resolve internode metrics via app context

* fmt

* ci: speed up fuzz smoke artifact restore

---------

Signed-off-by: houseme <housemecn@gmail.com>
2026-06-23 21:36:39 +08:00

357 lines
16 KiB
TOML

# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
[workspace]
members = [
"rustfs", # Core file system implementation
"crates/audit", # Audit target management system with multi-target fan-out
"crates/checksums", # client checksums
"crates/common", # Shared utilities and data structures
"crates/data-usage", # Shared data usage models and algorithms
"crates/config", # Configuration management
"crates/credentials", # Credential management system
"crates/crypto", # Cryptography and security features
"crates/ecstore", # Erasure coding storage implementation
"crates/e2e_test", # End-to-end test suite
"crates/filemeta", # File metadata management
"crates/heal", # Erasure set and object healing
"crates/iam", # Identity and Access Management
"crates/keystone", # OpenStack Keystone integration
"crates/kms", # Key Management Service
"crates/lock", # Distributed locking implementation
"crates/madmin", # Management dashboard and admin API interface
"crates/notify", # Notification system for events
"crates/obs", # Observability utilities
"crates/object-capacity", # Capacity scan and refresh core
"crates/policy", # Policy management
"crates/protocols", # Protocol implementations (FTPS, SFTP, etc.)
"crates/protos", # Protocol buffer definitions
"crates/rio", # Rust I/O utilities and abstractions
"crates/rio-v2", # Next-generation Rust I/O compatibility layer
"crates/concurrency", # Concurrency management for RustFS - timeout, locking, backpressure, and I/O scheduling
"crates/s3-types", # S3 event type definitions
"crates/s3-ops", # S3 operation definitions and mapping
"crates/s3select-api", # S3 Select API interface
"crates/s3select-query", # S3 Select query engine
"crates/scanner", # Scanner for data integrity checks and health monitoring
"crates/security-governance", # Security governance contracts
"crates/extension-schema", # Extension schema contracts
"crates/signer", # client signer
"crates/storage-api", # Storage API contracts
"crates/targets", # Target-specific configurations and utilities
"crates/trusted-proxies", # Trusted proxies management
"crates/tls-runtime", # Project-wide TLS runtime foundation
"crates/utils", # Utility functions and helpers
"crates/io-metrics", # Zero-copy metrics collection for performance analysis
"crates/io-core", # Zero-copy core reader and writer implementations
"crates/zip", # ZIP file handling and compression
]
resolver = "3"
[workspace.package]
edition = "2024"
license = "Apache-2.0"
repository = "https://github.com/rustfs/rustfs"
rust-version = "1.96.0"
version = "1.0.0-beta.8"
homepage = "https://rustfs.com"
description = "RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. "
keywords = ["RustFS", "Minio", "object-storage", "filesystem", "s3"]
categories = ["web-programming", "development-tools", "filesystem", "network-programming"]
authors = ["RustFS Team"]
[workspace.lints.rust]
unsafe_code = "deny"
[workspace.lints.clippy]
all = "warn"
needless_collect = "warn"
redundant_clone = "warn"
[workspace.dependencies]
# RustFS Internal Crates
rustfs = { path = "./rustfs", version = "1.0.0-beta.8" }
rustfs-heal = { path = "crates/heal", version = "1.0.0-beta.8" }
rustfs-audit = { path = "crates/audit", version = "1.0.0-beta.8" }
rustfs-checksums = { path = "crates/checksums", version = "1.0.0-beta.8" }
rustfs-common = { path = "crates/common", version = "1.0.0-beta.8" }
rustfs-data-usage = { path = "crates/data-usage", version = "1.0.0-beta.8" }
rustfs-config = { path = "./crates/config", version = "1.0.0-beta.8" }
rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.0-beta.8" }
rustfs-credentials = { path = "crates/credentials", version = "1.0.0-beta.8" }
rustfs-crypto = { path = "crates/crypto", version = "1.0.0-beta.8" }
rustfs-ecstore = { path = "crates/ecstore", version = "1.0.0-beta.8" }
rustfs-filemeta = { path = "crates/filemeta", version = "1.0.0-beta.8" }
rustfs-iam = { path = "crates/iam", version = "1.0.0-beta.8" }
rustfs-keystone = { path = "crates/keystone", version = "1.0.0-beta.8" }
rustfs-kms = { path = "crates/kms", version = "1.0.0-beta.8" }
rustfs-lock = { path = "crates/lock", version = "1.0.0-beta.8" }
rustfs-madmin = { path = "crates/madmin", version = "1.0.0-beta.8" }
rustfs-notify = { path = "crates/notify", version = "1.0.0-beta.8" }
rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.0-beta.8" }
rustfs-io-core = { path = "crates/io-core", version = "1.0.0-beta.8" }
rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.0-beta.8" }
rustfs-obs = { path = "crates/obs", version = "1.0.0-beta.8" }
rustfs-policy = { path = "crates/policy", version = "1.0.0-beta.8" }
rustfs-protos = { path = "crates/protos", version = "1.0.0-beta.8" }
rustfs-protocols = { path = "crates/protocols", version = "1.0.0-beta.8" }
rustfs-rio = { path = "crates/rio", version = "1.0.0-beta.8" }
rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.0-beta.8" }
rustfs-s3-types = { path = "crates/s3-types", version = "1.0.0-beta.8" }
rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.0-beta.8" }
rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.0-beta.8" }
rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.0-beta.8" }
rustfs-scanner = { path = "crates/scanner", version = "1.0.0-beta.8" }
rustfs-security-governance = { path = "crates/security-governance", version = "1.0.0-beta.8" }
rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.0-beta.8" }
rustfs-signer = { path = "crates/signer", version = "1.0.0-beta.8" }
rustfs-storage-api = { path = "crates/storage-api", version = "1.0.0-beta.8" }
rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.0-beta.8" }
rustfs-targets = { path = "crates/targets", version = "1.0.0-beta.8" }
rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.0-beta.8" }
rustfs-utils = { path = "crates/utils", version = "1.0.0-beta.8" }
rustfs-zip = { path = "./crates/zip", version = "1.0.0-beta.8" }
# Async Runtime and Networking
async-channel = "2.5.0"
async_zip = { version = "0.0.18", default-features = false, features = ["tokio", "deflate"] }
mysql_async = { version = "0.37", default-features = false, features = ["default-rustls", "tracing"] }
async-compression = { version = "0.4.42" }
async-recursion = "1.1.1"
async-trait = "0.1.89"
async-nats = "0.49.1"
axum = "0.8.9"
futures = "0.3.32"
futures-core = "0.3.32"
futures-lite = "2.6.1"
futures-util = "0.3.32"
pollster = "0.4.0"
pulsar = { version = "6.8.0", default-features = false, features = ["tokio-rustls-runtime", "telemetry"] }
lapin = { version = "4.10.0", default-features = false, features = ["tokio", "rustls", "rustls--aws_lc_rs"] }
hyper = { version = "1.10.1", features = ["http2", "http1", "server"] }
hyper-rustls = { version = "0.27.9", default-features = false, features = ["native-tokio", "http1", "tls12", "logging", "http2", "aws-lc-rs", "webpki-roots"] }
hyper-util = { version = "0.1.20", features = ["tokio", "server-auto", "server-graceful", "tracing"] }
http = "1.4.2"
http-body = "1.0.1"
http-body-util = "0.1.3"
reqwest = { version = "0.13.4", default-features = false, features = ["rustls", "charset", "http2", "system-proxy", "stream", "json", "blocking", "query", "form"] }
rustfs-kafka-async = { version = "1.2.0" }
socket2 = { version = "0.6.4", features = ["all"] }
tokio = { version = "1.52.3", features = ["fs", "rt-multi-thread"] }
tokio-rustls = { version = "0.26.4", default-features = false, features = ["logging", "tls12", "aws-lc-rs"] }
tokio-stream = { version = "0.1.18" }
tokio-test = "0.4.5"
tokio-util = { version = "0.7.18", features = ["io", "compat"] }
tonic = { version = "0.14.6", features = ["gzip", "deflate"] }
tonic-prost = { version = "0.14.6" }
tonic-prost-build = { version = "0.14.6" }
tower = { version = "0.5.3", features = ["timeout"] }
tower-http = { version = "0.7.0", features = ["cors"] }
# Serialization and Data Formats
apache-avro = "0.21.0"
bytes = { version = "1.12.0", features = ["serde"] }
bytesize = "2.4.0"
byteorder = "1.5.0"
flatbuffers = "25.12.19"
form_urlencoded = "1.2.2"
prost = "0.14.4"
quick-xml = "0.40.1"
rmp = { version = "0.8.15" }
rmp-serde = { version = "1.3.1" }
serde = { version = "1.0.228", features = ["derive"] }
serde_json = { version = "1.0.150", features = ["raw_value"] }
serde_urlencoded = "0.7.1"
# Cryptography and Security
aes-gcm = { version = "0.11.0-rc.4", features = ["rand_core"] }
argon2 = { version = "0.6.0-rc.8" }
blake2 = "0.11.0-rc.6"
chacha20poly1305 = { version = "0.11.0-rc.3" }
crc-fast = "1.10.0"
hmac = { version = "0.13.0" }
jsonwebtoken = { version = "10.4.0", features = ["aws_lc_rs"] }
openidconnect = { version = "4.0", default-features = false }
pbkdf2 = "0.13.0"
rsa = { version = "0.10.0-rc.18" }
rustls = { version = "0.23.41", default-features = false, features = ["aws-lc-rs", "logging", "tls12", "prefer-post-quantum", "std"] }
rustls-native-certs = "0.8"
rustls-pki-types = "1.14.1"
sha1 = "0.11.0"
sha2 = "0.11.0"
subtle = "2.6"
zeroize = { version = "1.9.0", features = ["derive"] }
# Time and Date
chrono = { version = "0.4.45", features = ["serde"] }
humantime = "2.3.0"
jiff = { version = "0.2.29", features = ["serde"] }
time = { version = "0.3.51", features = ["std", "parsing", "formatting", "macros", "serde"] }
# Database
deadpool-postgres = { version = "0.14", features = ["rt_tokio_1"] }
tokio-postgres = { version = "0.7.18", default-features = false, features = ["runtime", "with-serde_json-1"] }
tokio-postgres-rustls = "0.14.0"
# Utilities and Tools
anyhow = "1.0.102"
arc-swap = "1.9.1"
astral-tokio-tar = "0.6.2"
atoi = "3.0.0"
atomic_enum = "0.3.0"
aws-config = { version = "1.8.18" }
aws-credential-types = { version = "1.2.14" }
aws-sdk-s3 = { version = "1.137.0", default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
aws-smithy-http-client = { version = "1.1.13", default-features = false, features = ["default-client", "rustls-aws-lc"] }
aws-smithy-types = { version = "1.5.0" }
base64 = "0.22.1"
base64-simd = "0.8.0"
brotli = "8.0.4"
clap = { version = "4.6.1", features = ["derive", "env"] }
const-str = { version = "1.1.0", features = ["std", "proc"] }
convert_case = "0.11.0"
criterion = { version = "0.8", features = ["html_reports"] }
crossbeam-queue = "0.3.12"
crossbeam-channel = "0.5.15"
crossbeam-deque = "0.8.6"
crossbeam-utils = "0.8.21"
datafusion = "54.0.0"
derive_builder = "0.20.2"
enumset = "1.1.13"
faster-hex = "0.10.0"
flate2 = "1.1.9"
glob = "0.3.3"
google-cloud-storage = "1.15.0"
google-cloud-auth = "1.13.0"
hashbrown = { version = "0.17.1", features = ["serde", "rayon"] }
hex = "0.4.3"
hex-simd = "0.8.0"
highway = { version = "1.3.0" }
ipnetwork = { version = "0.21.1", features = ["serde"] }
lazy_static = "1.5.0"
libc = "0.2.186"
libsystemd = "0.7.2"
local-ip-address = "0.6.13"
memmap2 = "0.9.11"
lz4 = "1.28.1"
matchit = "0.9.2"
md-5 = "0.11.0"
md5 = "0.8.0"
mime_guess = "2.0.5"
moka = { version = "0.12.15", features = ["future"] }
netif = "0.1.6"
num_cpus = { version = "1.17.0" }
nvml-wrapper = "0.12.1"
parking_lot = "0.12.5"
path-absolutize = "3.1.1"
path-clean = "1.0.1"
percent-encoding = "2.3.2"
pin-project-lite = "0.2.17"
pretty_assertions = "1.4.1"
rand = { version = "0.10.1", features = ["serde"] }
ratelimit = "0.10.1"
rayon = "1.12.0"
reed-solomon-erasure = { package = "rustfs-erasure-codec", version = "7.0.0", features = ["simd-accel"] }
#reed-solomon-erasure = { version = "6.0", features = ["simd-accel"], git = "https://github.com/houseme/reed-solomon-erasure",rev = "main" }
reed-solomon-simd = "3.1.0"
regex = { version = "1.12.4" }
rumqttc = { package = "rumqttc-next", version = "0.33.2", features = ["websocket"] }
redis = { version = "1.2.4", features = ["connection-manager", "tokio-rustls-comp", "tls-rustls-insecure"] }
rustix = { version = "1.1.4", features = ["fs"] }
rust-embed = { version = "8.11.0" }
rustc-hash = { version = "2.1.2" }
s3s = { git = "https://github.com/rustfs/s3s", rev = "300c657fb895b87e01e30b06e41a7996e57191f9", features = ["minio"] }
serial_test = "3.5.0"
shadow-rs = { version = "2.0.0", default-features = false }
siphasher = "1.0.3"
smallvec = { version = "1.15.2", features = ["serde"] }
smartstring = "1.0.1"
snafu = "0.9.1"
snap = "1.1.1"
starshard = { version = "2.2.1", features = ["rayon", "async", "serde"] }
strum = { version = "0.28.0", features = ["derive"] }
sysinfo = "0.39.5"
temp-env = "0.3.6"
tempfile = "3.27.0"
test-case = "3.3.1"
thiserror = "2.0.18"
tracing = { version = "0.1.44" }
tracing-appender = "0.2.5"
tracing-error = "0.2.1"
tracing-opentelemetry = { version = "0.33" }
tracing-subscriber = { version = "0.3.23", features = ["env-filter", "time"] }
transform-stream = "0.3.1"
url = "2.5.8"
urlencoding = "2.1.3"
uuid = { version = "1.23.3", features = ["v4", "fast-rng", "macro-diagnostics"] }
vaultrs = { version = "0.8.0" }
walkdir = "2.5.0"
wildmatch = { version = "2.6.1", features = ["serde"] }
windows = { version = "0.62.2" }
xxhash-rust = { version = "0.8.15", features = ["xxh64", "xxh3"] }
zip = "8.6.0"
zstd = "0.13.3"
# Observability and Metrics
metrics = "0.24.6"
dial9-tokio-telemetry = "0.3"
opentelemetry = { version = "0.32.0" }
opentelemetry-appender-tracing = { version = "0.32.0", features = ["experimental_span_attributes", "experimental_metadata_attributes"] }
opentelemetry-otlp = { version = "0.32.0", features = ["gzip-http", "reqwest-rustls"] }
opentelemetry_sdk = { version = "0.32.1", features = ["rt-tokio"] }
opentelemetry-semantic-conventions = { version = "0.32.0", features = ["semconv_experimental"] }
opentelemetry-stdout = { version = "0.32.0" }
pyroscope = { version = "=2.0.5", features = ["backend-pprof-rs", "backend-jemalloc"] }
# FTP and SFTP
libunftp = { version = "0.23.0", features = ["experimental"] }
unftp-core = "0.1.0"
suppaftp = { version = "9.0.0", features = ["tokio", "tokio-rustls-aws-lc-rs"] }
rcgen = "0.14.8"
russh = { version = "0.61.2", features = ["serde"] }
russh-sftp = "2.3.0"
# WebDAV
dav-server = "0.11.0"
# Performance Analysis and Memory Profiling
mimalloc = "0.1"
# Use tikv-jemallocator as memory allocator and enable performance analysis
tikv-jemallocator = { version = "0.6", features = ["profiling", "stats", "unprefixed_malloc_on_supported_platforms", "background_threads"] }
# Used to control and obtain statistics for jemalloc at runtime
tikv-jemalloc-ctl = { version = "0.6", features = ["use_std", "stats", "profiling"] }
# Used to generate pprof-compatible memory profiling data and support symbolization and flame graphs
jemalloc_pprof = { version = "0.8.2", features = ["symbolize", "flamegraph"] }
# Used to generate CPU performance analysis data and flame diagrams
# pprof = { version = "0.15.0", features = ["flamegraph", "protobuf-codec"] }
# Use pprof-pyroscope-fork to match pyroscope 2.0.5's internal pprof dependency (v0.1500.4).
# Cargo unifies them into a single instance, avoiding duplicate perf_signal_handler symbols.
# NOTE: Do NOT upgrade pyroscope to >=2.0.6 — it vendors pprof-rs internally, causing symbol collision.
pprof = { package = "pprof-pyroscope-fork", version = "0.1500.4", features = ["flamegraph", "protobuf-codec"] }
[workspace.metadata.cargo-shear]
ignored = ["rustfs"]
[profile.release]
opt-level = 3
[profile.production]
inherits = "release"
lto = "fat"
codegen-units = 1
[profile.profiling]
inherits = "release"
debug = true