gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-08-08 06:13:14 +00:00
Code Issues Packages Projects Releases Wiki Activity

Implement adaptive buffer sizing with workload profiles - Production-Ready Implementation (#878)

Browse Source 
* Initial plan

* Add workload profiles module with adaptive buffer sizing

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* Add comprehensive adaptive buffer sizing documentation

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* Add implementation summary documentation

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* Implement Phase 2: Opt-In Usage for buffer profiles

- Add RUSTFS_BUFFER_PROFILE_ENABLE and RUSTFS_BUFFER_PROFILE config options
- Implement global buffer configuration with enable/disable flag
- Add WorkloadProfile::from_name() for parsing profile names
- Update put_object, put_object_extract, and upload_part to use opt-in function
- Add init_buffer_profile_system() in main.rs for initialization
- Add comprehensive tests for opt-in functionality
- Update documentation with Phase 2 usage examples

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* Implement Phase 3: Default Enablement for buffer profiles

- Change default to enabled with GeneralPurpose profile
- Replace --buffer-profile-enable with --buffer-profile-disable for opt-out
- Update init_buffer_profile_system() to reflect Phase 3 logic
- Add comprehensive Phase 3 documentation
- Create MIGRATION_PHASE3.md guide for smooth transition
- Add test for Phase 3 default behavior
- Update IMPLEMENTATION_SUMMARY.md to mark Phase 3 as complete

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* Implement Phase 4: Full Integration with deprecated legacy and metrics

- Deprecate get_adaptive_buffer_size() function (backward compatible)
- Implement profile-only buffer sizing in get_buffer_size_opt_in()
- Add performance metrics collection (optional, via metrics feature)
- Even disabled mode now uses GeneralPurpose profile (no hardcoded values)
- Add comprehensive Phase 4 documentation and guide
- Add test_phase4_full_integration() test
- Update IMPLEMENTATION_SUMMARY.md to mark Phase 4 as complete

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* Finalize adaptive buffer sizing: remove deprecated code and improve documentation

English:
- Remove deprecated get_adaptive_buffer_size() function
- Remove deprecated test cases (test_adaptive_buffer_size, Phase 3 legacy comparisons)
- Improve all code comments to be clear and professional English
- Add comprehensive bilingual implementation summary (COMPLETE_SUMMARY.md)
- Update buffer sizing comments in put_object, put_object_extract, upload_part
- Enhance init_buffer_profile_system documentation
- Clean up phase-specific references in comments

Chinese (中文):
- 移除已弃用的 get_adaptive_buffer_size() 函数
- 移除已弃用的测试用例（test_adaptive_buffer_size、Phase 3 旧版比较）
- 改进所有代码注释，使用清晰专业的英文
- 添加全面的双语实现摘要（COMPLETE_SUMMARY.md）
- 更新 put_object、put_object_extract、upload_part 中的缓冲区调整注释
- 增强 init_buffer_profile_system 文档
- 清理注释中的特定阶段引用

This commit completes the adaptive buffer sizing implementation by:
1. Removing all deprecated legacy code and tests
2. Improving code documentation quality
3. Providing comprehensive bilingual summary

本提交完成自适应缓冲区大小实现：
1. 移除所有已弃用的旧代码和测试
2. 提高代码文档质量
3. 提供全面的双语摘要

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* fmt

* fix

* fix

* fix

* fix

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
This commit is contained in:
Copilot
2025-11-18 13:32:02 +08:00
committed by GitHub
parent 85bc0ce2d5
commit 6da5766ea2
11 changed files with 3113 additions and 50 deletions
Download Patch File Download Diff File Expand all files Collapse all files
rustfs/Cargo.toml
+2 -1
View File
@@ -31,7 +31,8 @@ name = "rustfs"
path = "src/main.rs"
[features]
default = []
default = ["metrics"]
metrics = []
[lints]
workspace = true
rustfs/src/config/mod.rs
+12
View File
@@ -17,6 +17,8 @@ use const_str::concat;
use std::string::ToString;
shadow_rs::shadow!(build);
pub mod workload_profiles;
#[cfg(test)]
mod config_test;
@@ -112,6 +114,16 @@ pub struct Opt {
/// Default KMS key ID for encryption
#[arg(long, env = "RUSTFS_KMS_DEFAULT_KEY_ID")]
pub kms_default_key_id: Option<String>,
/// Disable adaptive buffer sizing with workload profiles
/// Set this flag to use legacy fixed-size buffer behavior from PR #869
#[arg(long, default_value_t = false, env = "RUSTFS_BUFFER_PROFILE_DISABLE")]
pub buffer_profile_disable: bool,
/// Workload profile for adaptive buffer sizing
/// Options: GeneralPurpose, AiTraining, DataAnalytics, WebWorkload, IndustrialIoT, SecureStorage
#[arg(long, default_value_t = String::from("GeneralPurpose"), env = "RUSTFS_BUFFER_PROFILE")]
pub buffer_profile: String,
}
// lazy_static::lazy_static! {
rustfs/src/config/workload_profiles.rs
+632
View File
@@ -0,0 +1,632 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#![allow(dead_code)]
//! Adaptive buffer sizing optimization for different workload types.
//!
//! This module provides intelligent buffer size selection based on file size and workload profile
//! to achieve optimal balance between performance, memory usage, and security.
use rustfs_config::{KI_B, MI_B};
use std::sync::OnceLock;
use std::sync::atomic::{AtomicBool, Ordering};
/// Global buffer configuration that can be set at application startup
static GLOBAL_BUFFER_CONFIG: OnceLock<RustFSBufferConfig> = OnceLock::new();
/// Global flag indicating whether buffer profiles are enabled
static BUFFER_PROFILE_ENABLED: AtomicBool = AtomicBool::new(false);
/// Enable or disable buffer profiling globally
///
/// This controls whether the opt-in buffer profiling feature is active.
///
/// # Arguments
/// * `enabled` - Whether to enable buffer profiling
pub fn set_buffer_profile_enabled(enabled: bool) {
BUFFER_PROFILE_ENABLED.store(enabled, Ordering::Relaxed);
}
/// Check if buffer profiling is enabled globally
pub fn is_buffer_profile_enabled() -> bool {
BUFFER_PROFILE_ENABLED.load(Ordering::Relaxed)
}
/// Initialize the global buffer configuration
///
/// This should be called once at application startup with the desired profile.
/// If not called, the default GeneralPurpose profile will be used.
///
/// # Arguments
/// * `config` - The buffer configuration to use globally
///
/// # Examples
/// ```ignore
/// use rustfs::config::workload_profiles::{RustFSBufferConfig, WorkloadProfile};
///
/// // Initialize with AiTraining profile
/// init_global_buffer_config(RustFSBufferConfig::new(WorkloadProfile::AiTraining));
/// ```
pub fn init_global_buffer_config(config: RustFSBufferConfig) {
let _ = GLOBAL_BUFFER_CONFIG.set(config);
}
/// Get the global buffer configuration
///
/// Returns the configured profile, or GeneralPurpose if not initialized.
pub fn get_global_buffer_config() -> &'static RustFSBufferConfig {
GLOBAL_BUFFER_CONFIG.get_or_init(RustFSBufferConfig::default)
}
/// Workload profile types that define buffer sizing strategies
#[derive(Debug, Clone, PartialEq)]
pub enum WorkloadProfile {
/// General purpose - default configuration with balanced performance and memory
GeneralPurpose,
/// AI/ML training: optimized for large sequential reads with maximum throughput
AiTraining,
/// Data analytics: mixed read-write patterns with moderate buffer sizes
DataAnalytics,
/// Web workloads: small file intensive with minimal memory overhead
WebWorkload,
/// Industrial IoT: real-time streaming with low latency priority
IndustrialIoT,
/// Secure storage: security first, memory constrained for compliance
SecureStorage,
/// Custom configuration for specialized requirements
Custom(BufferConfig),
}
/// Buffer size configuration for adaptive buffering
#[derive(Debug, Clone, PartialEq)]
pub struct BufferConfig {
/// Minimum buffer size in bytes (for very small files or memory-constrained environments)
pub min_size: usize,
/// Maximum buffer size in bytes (cap for large files to prevent excessive memory usage)
pub max_size: usize,
/// Default size for unknown file size scenarios (streaming/chunked uploads)
pub default_unknown: usize,
/// File size thresholds and corresponding buffer sizes: (file_size_threshold, buffer_size)
/// Thresholds should be in ascending order
pub thresholds: Vec<(i64, usize)>,
}
/// Complete buffer configuration for RustFS
#[derive(Debug, Clone)]
pub struct RustFSBufferConfig {
/// Selected workload profile
pub workload: WorkloadProfile,
/// Computed buffer configuration (either from profile or custom)
pub base_config: BufferConfig,
}
impl WorkloadProfile {
/// Parse a workload profile from a string name
///
/// # Arguments
/// * `name` - The name of the profile (case-insensitive)
///
/// # Returns
/// The corresponding WorkloadProfile, or GeneralPurpose if name is not recognized
///
/// # Examples
/// ```
/// use rustfs::config::workload_profiles::WorkloadProfile;
///
/// let profile = WorkloadProfile::from_name("AiTraining");
/// let profile2 = WorkloadProfile::from_name("aitraining"); // case-insensitive
/// let profile3 = WorkloadProfile::from_name("unknown"); // defaults to GeneralPurpose
/// ```
pub fn from_name(name: &str) -> Self {
match name.to_lowercase().as_str() {
"generalpurpose" | "general" => WorkloadProfile::GeneralPurpose,
"aitraining" | "ai" => WorkloadProfile::AiTraining,
"dataanalytics" | "analytics" => WorkloadProfile::DataAnalytics,
"webworkload" | "web" => WorkloadProfile::WebWorkload,
"industrialiot" | "iot" => WorkloadProfile::IndustrialIoT,
"securestorage" | "secure" => WorkloadProfile::SecureStorage,
_ => {
// Default to GeneralPurpose for unknown profiles
WorkloadProfile::GeneralPurpose
}
}
}
/// Get the buffer configuration for this workload profile
pub fn config(&self) -> BufferConfig {
match self {
WorkloadProfile::GeneralPurpose => Self::general_purpose_config(),
WorkloadProfile::AiTraining => Self::ai_training_config(),
WorkloadProfile::DataAnalytics => Self::data_analytics_config(),
WorkloadProfile::WebWorkload => Self::web_workload_config(),
WorkloadProfile::IndustrialIoT => Self::industrial_iot_config(),
WorkloadProfile::SecureStorage => Self::secure_storage_config(),
WorkloadProfile::Custom(config) => config.clone(),
}
}
/// General purpose configuration: balanced performance and memory usage
/// - Small files (< 1MB): 64KB buffer
/// - Medium files (1MB-100MB): 256KB buffer
/// - Large files (>= 100MB): 1MB buffer
fn general_purpose_config() -> BufferConfig {
BufferConfig {
min_size: 64 * KI_B,
max_size: MI_B,
default_unknown: MI_B,
thresholds: vec![
(MI_B as i64, 64 * KI_B), // < 1MB: 64KB
(100 * MI_B as i64, 256 * KI_B), // 1MB-100MB: 256KB
(i64::MAX, MI_B), // >= 100MB: 1MB
],
}
}
/// AI/ML training configuration: optimized for large sequential reads
/// - Small files (< 10MB): 512KB buffer
/// - Medium files (10MB-500MB): 2MB buffer
/// - Large files (>= 500MB): 4MB buffer for maximum throughput
fn ai_training_config() -> BufferConfig {
BufferConfig {
min_size: 512 * KI_B,
max_size: 4 * MI_B,
default_unknown: 2 * MI_B,
thresholds: vec![
(10 * MI_B as i64, 512 * KI_B), // < 10MB: 512KB
(500 * MI_B as i64, 2 * MI_B), // 10MB-500MB: 2MB
(i64::MAX, 4 * MI_B), // >= 500MB: 4MB
],
}
}
/// Data analytics configuration: mixed read-write patterns
/// - Small files (< 5MB): 128KB buffer
/// - Medium files (5MB-200MB): 512KB buffer
/// - Large files (>= 200MB): 2MB buffer
fn data_analytics_config() -> BufferConfig {
BufferConfig {
min_size: 128 * KI_B,
max_size: 2 * MI_B,
default_unknown: 512 * KI_B,
thresholds: vec![
(5 * MI_B as i64, 128 * KI_B), // < 5MB: 128KB
(200 * MI_B as i64, 512 * KI_B), // 5MB-200MB: 512KB
(i64::MAX, 2 * MI_B), // >= 200MB: 2MB
],
}
}
/// Web workload configuration: small file intensive
/// - Small files (< 512KB): 32KB buffer to minimize memory
/// - Medium files (512KB-10MB): 128KB buffer
/// - Large files (>= 10MB): 256KB buffer (rare for web assets)
fn web_workload_config() -> BufferConfig {
BufferConfig {
min_size: 32 * KI_B,
max_size: 256 * KI_B,
default_unknown: 128 * KI_B,
thresholds: vec![
(512 * KI_B as i64, 32 * KI_B), // < 512KB: 32KB
(10 * MI_B as i64, 128 * KI_B), // 512KB-10MB: 128KB
(i64::MAX, 256 * KI_B), // >= 10MB: 256KB
],
}
}
/// Industrial IoT configuration: real-time streaming with low latency
/// - Small files (< 1MB): 64KB buffer for quick processing
/// - Medium files (1MB-50MB): 256KB buffer
/// - Large files (>= 50MB): 512KB buffer (cap for memory constraints)
fn industrial_iot_config() -> BufferConfig {
BufferConfig {
min_size: 64 * KI_B,
max_size: 512 * KI_B,
default_unknown: 256 * KI_B,
thresholds: vec![
(MI_B as i64, 64 * KI_B), // < 1MB: 64KB
(50 * MI_B as i64, 256 * KI_B), // 1MB-50MB: 256KB
(i64::MAX, 512 * KI_B), // >= 50MB: 512KB
],
}
}
/// Secure storage configuration: security first, memory constrained
/// - Small files (< 1MB): 32KB buffer (minimal memory footprint)
/// - Medium files (1MB-50MB): 128KB buffer
/// - Large files (>= 50MB): 256KB buffer (strict memory limit for compliance)
fn secure_storage_config() -> BufferConfig {
BufferConfig {
min_size: 32 * KI_B,
max_size: 256 * KI_B,
default_unknown: 128 * KI_B,
thresholds: vec![
(MI_B as i64, 32 * KI_B), // < 1MB: 32KB
(50 * MI_B as i64, 128 * KI_B), // 1MB-50MB: 128KB
(i64::MAX, 256 * KI_B), // >= 50MB: 256KB
],
}
}
/// Detect special OS environment and return appropriate workload profile
/// Supports Chinese secure operating systems (Kylin, NeoKylin, Unity OS, etc.)
pub fn detect_os_environment() -> Option<WorkloadProfile> {
#[cfg(target_os = "linux")]
{
// Read /etc/os-release to detect Chinese secure OS distributions
if let Ok(content) = std::fs::read_to_string("/etc/os-release") {
let content_lower = content.to_lowercase();
// Check for Chinese secure OS distributions
if content_lower.contains("kylin")
|| content_lower.contains("neokylin")
|| content_lower.contains("uos")
|| content_lower.contains("unity")
|| content_lower.contains("openkylin")
{
// Use SecureStorage profile for Chinese secure OS environments
return Some(WorkloadProfile::SecureStorage);
}
}
}
None
}
}
impl BufferConfig {
/// Calculate the optimal buffer size for a given file size
///
/// # Arguments
/// * `file_size` - The size of the file in bytes, or -1 if unknown
///
/// # Returns
/// Optimal buffer size in bytes based on the configuration
pub fn calculate_buffer_size(&self, file_size: i64) -> usize {
// Handle unknown or negative file sizes
if file_size < 0 {
return self.default_unknown.clamp(self.min_size, self.max_size);
}
// Find the appropriate buffer size from thresholds
for (threshold, buffer_size) in &self.thresholds {
if file_size < *threshold {
return (*buffer_size).clamp(self.min_size, self.max_size);
}
}
// Fallback to max_size if no threshold matched (shouldn't happen with i64::MAX threshold)
self.max_size
}
/// Validate the buffer configuration
pub fn validate(&self) -> Result<(), String> {
if self.min_size == 0 {
return Err("min_size must be greater than 0".to_string());
}
if self.max_size < self.min_size {
return Err("max_size must be >= min_size".to_string());
}
if self.default_unknown < self.min_size || self.default_unknown > self.max_size {
return Err("default_unknown must be between min_size and max_size".to_string());
}
if self.thresholds.is_empty() {
return Err("thresholds cannot be empty".to_string());
}
// Validate thresholds are in ascending order
let mut prev_threshold = -1i64;
for (threshold, buffer_size) in &self.thresholds {
if *threshold <= prev_threshold {
return Err("thresholds must be in ascending order".to_string());
}
if *buffer_size < self.min_size || *buffer_size > self.max_size {
return Err(format!(
"buffer_size {} must be between min_size {} and max_size {}",
buffer_size, self.min_size, self.max_size
));
}
prev_threshold = *threshold;
}
Ok(())
}
}
impl RustFSBufferConfig {
/// Create a new buffer configuration with the given workload profile
pub fn new(workload: WorkloadProfile) -> Self {
let base_config = workload.config();
Self { workload, base_config }
}
/// Create a configuration with auto-detected OS environment
/// Falls back to GeneralPurpose if no special environment detected
pub fn with_auto_detect() -> Self {
let workload = WorkloadProfile::detect_os_environment().unwrap_or(WorkloadProfile::GeneralPurpose);
Self::new(workload)
}
/// Get the buffer size for a given file size
pub fn get_buffer_size(&self, file_size: i64) -> usize {
self.base_config.calculate_buffer_size(file_size)
}
}
impl Default for RustFSBufferConfig {
fn default() -> Self {
Self::new(WorkloadProfile::GeneralPurpose)
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_general_purpose_config() {
let config = WorkloadProfile::GeneralPurpose.config();
// Test small files (< 1MB) - should use 64KB
assert_eq!(config.calculate_buffer_size(0), 64 * KI_B);
assert_eq!(config.calculate_buffer_size(512 * KI_B as i64), 64 * KI_B);
assert_eq!(config.calculate_buffer_size((MI_B - 1) as i64), 64 * KI_B);
// Test medium files (1MB - 100MB) - should use 256KB
assert_eq!(config.calculate_buffer_size(MI_B as i64), 256 * KI_B);
assert_eq!(config.calculate_buffer_size((50 * MI_B) as i64), 256 * KI_B);
assert_eq!(config.calculate_buffer_size((100 * MI_B - 1) as i64), 256 * KI_B);
// Test large files (>= 100MB) - should use 1MB
assert_eq!(config.calculate_buffer_size((100 * MI_B) as i64), MI_B);
assert_eq!(config.calculate_buffer_size((500 * MI_B) as i64), MI_B);
assert_eq!(config.calculate_buffer_size((10 * 1024 * MI_B) as i64), MI_B);
// Test unknown size
assert_eq!(config.calculate_buffer_size(-1), MI_B);
}
#[test]
fn test_ai_training_config() {
let config = WorkloadProfile::AiTraining.config();
// Test small files
assert_eq!(config.calculate_buffer_size((5 * MI_B) as i64), 512 * KI_B);
assert_eq!(config.calculate_buffer_size((10 * MI_B - 1) as i64), 512 * KI_B);
// Test medium files
assert_eq!(config.calculate_buffer_size((10 * MI_B) as i64), 2 * MI_B);
assert_eq!(config.calculate_buffer_size((100 * MI_B) as i64), 2 * MI_B);
assert_eq!(config.calculate_buffer_size((500 * MI_B - 1) as i64), 2 * MI_B);
// Test large files
assert_eq!(config.calculate_buffer_size((500 * MI_B) as i64), 4 * MI_B);
assert_eq!(config.calculate_buffer_size((1024 * MI_B) as i64), 4 * MI_B);
// Test unknown size
assert_eq!(config.calculate_buffer_size(-1), 2 * MI_B);
}
#[test]
fn test_web_workload_config() {
let config = WorkloadProfile::WebWorkload.config();
// Test small files
assert_eq!(config.calculate_buffer_size((100 * KI_B) as i64), 32 * KI_B);
assert_eq!(config.calculate_buffer_size((512 * KI_B - 1) as i64), 32 * KI_B);
// Test medium files
assert_eq!(config.calculate_buffer_size((512 * KI_B) as i64), 128 * KI_B);
assert_eq!(config.calculate_buffer_size((5 * MI_B) as i64), 128 * KI_B);
assert_eq!(config.calculate_buffer_size((10 * MI_B - 1) as i64), 128 * KI_B);
// Test large files
assert_eq!(config.calculate_buffer_size((10 * MI_B) as i64), 256 * KI_B);
assert_eq!(config.calculate_buffer_size((50 * MI_B) as i64), 256 * KI_B);
// Test unknown size
assert_eq!(config.calculate_buffer_size(-1), 128 * KI_B);
}
#[test]
fn test_secure_storage_config() {
let config = WorkloadProfile::SecureStorage.config();
// Test small files
assert_eq!(config.calculate_buffer_size((500 * KI_B) as i64), 32 * KI_B);
assert_eq!(config.calculate_buffer_size((MI_B - 1) as i64), 32 * KI_B);
// Test medium files
assert_eq!(config.calculate_buffer_size(MI_B as i64), 128 * KI_B);
assert_eq!(config.calculate_buffer_size((25 * MI_B) as i64), 128 * KI_B);
assert_eq!(config.calculate_buffer_size((50 * MI_B - 1) as i64), 128 * KI_B);
// Test large files
assert_eq!(config.calculate_buffer_size((50 * MI_B) as i64), 256 * KI_B);
assert_eq!(config.calculate_buffer_size((100 * MI_B) as i64), 256 * KI_B);
// Test unknown size
assert_eq!(config.calculate_buffer_size(-1), 128 * KI_B);
}
#[test]
fn test_industrial_iot_config() {
let config = WorkloadProfile::IndustrialIoT.config();
// Test configuration
assert_eq!(config.calculate_buffer_size((500 * KI_B) as i64), 64 * KI_B);
assert_eq!(config.calculate_buffer_size((25 * MI_B) as i64), 256 * KI_B);
assert_eq!(config.calculate_buffer_size((100 * MI_B) as i64), 512 * KI_B);
assert_eq!(config.calculate_buffer_size(-1), 256 * KI_B);
}
#[test]
fn test_data_analytics_config() {
let config = WorkloadProfile::DataAnalytics.config();
// Test configuration
assert_eq!(config.calculate_buffer_size((2 * MI_B) as i64), 128 * KI_B);
assert_eq!(config.calculate_buffer_size((100 * MI_B) as i64), 512 * KI_B);
assert_eq!(config.calculate_buffer_size((500 * MI_B) as i64), 2 * MI_B);
assert_eq!(config.calculate_buffer_size(-1), 512 * KI_B);
}
#[test]
fn test_custom_config() {
let custom_config = BufferConfig {
min_size: 16 * KI_B,
max_size: 512 * KI_B,
default_unknown: 128 * KI_B,
thresholds: vec![(MI_B as i64, 64 * KI_B), (i64::MAX, 256 * KI_B)],
};
let profile = WorkloadProfile::Custom(custom_config.clone());
let config = profile.config();
assert_eq!(config.calculate_buffer_size(512 * KI_B as i64), 64 * KI_B);
assert_eq!(config.calculate_buffer_size(2 * MI_B as i64), 256 * KI_B);
assert_eq!(config.calculate_buffer_size(-1), 128 * KI_B);
}
#[test]
fn test_buffer_config_validation() {
// Valid configuration
let valid_config = BufferConfig {
min_size: 32 * KI_B,
max_size: MI_B,
default_unknown: 256 * KI_B,
thresholds: vec![(MI_B as i64, 128 * KI_B), (i64::MAX, 512 * KI_B)],
};
assert!(valid_config.validate().is_ok());
// Invalid: min_size is 0
let invalid_config = BufferConfig {
min_size: 0,
max_size: MI_B,
default_unknown: 256 * KI_B,
thresholds: vec![(MI_B as i64, 128 * KI_B)],
};
assert!(invalid_config.validate().is_err());
// Invalid: max_size < min_size
let invalid_config = BufferConfig {
min_size: MI_B,
max_size: 32 * KI_B,
default_unknown: 256 * KI_B,
thresholds: vec![(MI_B as i64, 128 * KI_B)],
};
assert!(invalid_config.validate().is_err());
// Invalid: default_unknown out of range
let invalid_config = BufferConfig {
min_size: 32 * KI_B,
max_size: 256 * KI_B,
default_unknown: MI_B,
thresholds: vec![(MI_B as i64, 128 * KI_B)],
};
assert!(invalid_config.validate().is_err());
// Invalid: empty thresholds
let invalid_config = BufferConfig {
min_size: 32 * KI_B,
max_size: MI_B,
default_unknown: 256 * KI_B,
thresholds: vec![],
};
assert!(invalid_config.validate().is_err());
// Invalid: thresholds not in ascending order
let invalid_config = BufferConfig {
min_size: 32 * KI_B,
max_size: MI_B,
default_unknown: 256 * KI_B,
thresholds: vec![(100 * MI_B as i64, 512 * KI_B), (MI_B as i64, 128 * KI_B)],
};
assert!(invalid_config.validate().is_err());
}
#[test]
fn test_rustfs_buffer_config() {
let config = RustFSBufferConfig::new(WorkloadProfile::GeneralPurpose);
assert_eq!(config.get_buffer_size(500 * KI_B as i64), 64 * KI_B);
assert_eq!(config.get_buffer_size(50 * MI_B as i64), 256 * KI_B);
assert_eq!(config.get_buffer_size(200 * MI_B as i64), MI_B);
let default_config = RustFSBufferConfig::default();
assert_eq!(default_config.get_buffer_size(500 * KI_B as i64), 64 * KI_B);
}
#[test]
fn test_workload_profile_equality() {
assert_eq!(WorkloadProfile::GeneralPurpose, WorkloadProfile::GeneralPurpose);
assert_ne!(WorkloadProfile::GeneralPurpose, WorkloadProfile::AiTraining);
let custom1 = BufferConfig {
min_size: 32 * KI_B,
max_size: MI_B,
default_unknown: 256 * KI_B,
thresholds: vec![(MI_B as i64, 128 * KI_B)],
};
let custom2 = custom1.clone();
assert_eq!(WorkloadProfile::Custom(custom1.clone()), WorkloadProfile::Custom(custom2));
}
#[test]
fn test_workload_profile_from_name() {
// Test exact matches (case-insensitive)
assert_eq!(WorkloadProfile::from_name("GeneralPurpose"), WorkloadProfile::GeneralPurpose);
assert_eq!(WorkloadProfile::from_name("generalpurpose"), WorkloadProfile::GeneralPurpose);
assert_eq!(WorkloadProfile::from_name("GENERALPURPOSE"), WorkloadProfile::GeneralPurpose);
assert_eq!(WorkloadProfile::from_name("general"), WorkloadProfile::GeneralPurpose);
assert_eq!(WorkloadProfile::from_name("AiTraining"), WorkloadProfile::AiTraining);
assert_eq!(WorkloadProfile::from_name("aitraining"), WorkloadProfile::AiTraining);
assert_eq!(WorkloadProfile::from_name("ai"), WorkloadProfile::AiTraining);
assert_eq!(WorkloadProfile::from_name("DataAnalytics"), WorkloadProfile::DataAnalytics);
assert_eq!(WorkloadProfile::from_name("dataanalytics"), WorkloadProfile::DataAnalytics);
assert_eq!(WorkloadProfile::from_name("analytics"), WorkloadProfile::DataAnalytics);
assert_eq!(WorkloadProfile::from_name("WebWorkload"), WorkloadProfile::WebWorkload);
assert_eq!(WorkloadProfile::from_name("webworkload"), WorkloadProfile::WebWorkload);
assert_eq!(WorkloadProfile::from_name("web"), WorkloadProfile::WebWorkload);
assert_eq!(WorkloadProfile::from_name("IndustrialIoT"), WorkloadProfile::IndustrialIoT);
assert_eq!(WorkloadProfile::from_name("industrialiot"), WorkloadProfile::IndustrialIoT);
assert_eq!(WorkloadProfile::from_name("iot"), WorkloadProfile::IndustrialIoT);
assert_eq!(WorkloadProfile::from_name("SecureStorage"), WorkloadProfile::SecureStorage);
assert_eq!(WorkloadProfile::from_name("securestorage"), WorkloadProfile::SecureStorage);
assert_eq!(WorkloadProfile::from_name("secure"), WorkloadProfile::SecureStorage);
// Test unknown name defaults to GeneralPurpose
assert_eq!(WorkloadProfile::from_name("unknown"), WorkloadProfile::GeneralPurpose);
assert_eq!(WorkloadProfile::from_name("invalid"), WorkloadProfile::GeneralPurpose);
assert_eq!(WorkloadProfile::from_name(""), WorkloadProfile::GeneralPurpose);
}
#[test]
fn test_global_buffer_config() {
use super::{is_buffer_profile_enabled, set_buffer_profile_enabled};
// Test enable/disable
set_buffer_profile_enabled(true);
assert!(is_buffer_profile_enabled());
set_buffer_profile_enabled(false);
assert!(!is_buffer_profile_enabled());
// Reset for other tests
set_buffer_profile_enabled(false);
}
}
rustfs/src/main.rs
+45
View File
@@ -256,6 +256,9 @@ async fn run(opt: config::Opt) -> Result<()> {
// Initialize KMS system if enabled
init_kms_system(&opt).await?;
// Initialize buffer profiling system
init_buffer_profile_system(&opt);
// Initialize event notifier
init_event_notifier().await;
// Start the audit system
@@ -651,3 +654,45 @@ async fn init_kms_system(opt: &config::Opt) -> Result<()> {
Ok(())
}
/// Initialize the adaptive buffer sizing system with workload profile configuration.
///
/// This system provides intelligent buffer size selection based on file size and workload type.
/// Workload-aware buffer sizing is enabled by default with the GeneralPurpose profile,
/// which provides the same buffer sizes as the original implementation for compatibility.
///
/// # Configuration
/// - Default: Enabled with GeneralPurpose profile
/// - Opt-out: Use `--buffer-profile-disable` flag
/// - Custom profile: Set via `--buffer-profile` or `RUSTFS_BUFFER_PROFILE` environment variable
///
/// # Arguments
/// * `opt` - The application configuration options
fn init_buffer_profile_system(opt: &config::Opt) {
use crate::config::workload_profiles::{
RustFSBufferConfig, WorkloadProfile, init_global_buffer_config, set_buffer_profile_enabled,
};
if opt.buffer_profile_disable {
// User explicitly disabled buffer profiling - use GeneralPurpose profile in disabled mode
info!("Buffer profiling disabled via --buffer-profile-disable, using GeneralPurpose profile");
set_buffer_profile_enabled(false);
} else {
// Enabled by default: use configured workload profile
info!("Buffer profiling enabled with profile: {}", opt.buffer_profile);
// Parse the workload profile from configuration string
let profile = WorkloadProfile::from_name(&opt.buffer_profile);
// Log the selected profile for operational visibility
info!("Active buffer profile: {:?}", profile);
// Initialize the global buffer configuration
init_global_buffer_config(RustFSBufferConfig::new(profile));
// Enable buffer profiling globally
set_buffer_profile_enabled(true);
info!("Buffer profiling system initialized successfully");
}
}
rustfs/src/storage/ecfs.rs
+292 -49
View File
@@ -13,6 +13,9 @@
// limitations under the License.
use crate::auth::get_condition_values;
use crate::config::workload_profiles::{
RustFSBufferConfig, WorkloadProfile, get_global_buffer_config, is_buffer_profile_enabled,
};
use crate::error::ApiError;
use crate::storage::entity;
use crate::storage::helper::OperationHelper;
@@ -33,7 +36,6 @@ use datafusion::arrow::{
use futures::StreamExt;
use http::{HeaderMap, StatusCode};
use metrics::counter;
use rustfs_config::{KI_B, MI_B};
use rustfs_ecstore::{
bucket::{
lifecycle::{
@@ -150,30 +152,101 @@ static RUSTFS_OWNER: LazyLock<Owner> = LazyLock::new(|| Owner {
id: Some("c19050dbcee97fda828689dda99097a6321af2248fa760517237346e5d9c8a66".to_owned()),
});
/// Calculate adaptive buffer size based on file size for optimal streaming performance.
/// Calculate adaptive buffer size with workload profile support.
///
/// This function implements adaptive buffering to balance memory usage and performance:
/// - Small files (< 1MB): 64KB buffer - minimize memory overhead
/// - Medium files (1MB-100MB): 256KB buffer - balanced approach
/// - Large files (>= 100MB): 1MB buffer - maximize throughput, minimize syscalls
/// This enhanced version supports different workload profiles for optimal performance
/// across various use cases (AI/ML, web workloads, secure storage, etc.).
///
/// # Arguments
/// * `file_size` - The size of the file in bytes, or -1 if unknown
/// * `profile` - Optional workload profile. If None, uses auto-detection or GeneralPurpose
///
/// # Returns
/// Optimal buffer size in bytes based on the workload profile and file size
///
/// # Examples
/// ```ignore
/// // Use general purpose profile (default)
/// let buffer_size = get_adaptive_buffer_size_with_profile(1024 * 1024, None);
///
/// // Use AI training profile for large model files
/// let buffer_size = get_adaptive_buffer_size_with_profile(
/// 500 * 1024 * 1024,
/// Some(WorkloadProfile::AiTraining)
/// );
///
/// // Use secure storage profile for compliance scenarios
/// let buffer_size = get_adaptive_buffer_size_with_profile(
/// 10 * 1024 * 1024,
/// Some(WorkloadProfile::SecureStorage)
/// );
/// ```
///
#[allow(dead_code)]
fn get_adaptive_buffer_size_with_profile(file_size: i64, profile: Option<WorkloadProfile>) -> usize {
let config = match profile {
Some(p) => RustFSBufferConfig::new(p),
None => {
// Auto-detect OS environment or use general purpose
RustFSBufferConfig::with_auto_detect()
}
};
config.get_buffer_size(file_size)
}
/// Get adaptive buffer size using global workload profile configuration.
///
/// This is the primary buffer sizing function that uses the workload profile
/// system configured at startup to provide optimal buffer sizes for different scenarios.
///
/// The function automatically selects buffer sizes based on:
/// - Configured workload profile (default: GeneralPurpose)
/// - File size characteristics
/// - Optional performance metrics collection
///
/// # Arguments
/// * `file_size` - The size of the file in bytes, or -1 if unknown
///
/// # Returns
/// Optimal buffer size in bytes
/// Optimal buffer size in bytes based on the configured workload profile
///
fn get_adaptive_buffer_size(file_size: i64) -> usize {
match file_size {
// Unknown size or negative (chunked/streaming): use default large buffer for safety
size if size < 0 => DEFAULT_READ_BUFFER_SIZE,
// Small files (< 1MB): use 64KB to minimize memory overhead
size if size < MI_B as i64 => 64 * KI_B,
// Medium files (1MB - 100MB): use 256KB for balanced performance
size if size < (100 * MI_B) as i64 => 256 * KI_B,
// Large files (>= 100MB): use 1MB buffer for maximum throughput
_ => DEFAULT_READ_BUFFER_SIZE,
/// # Performance Metrics
/// When compiled with the `metrics` feature flag, this function tracks:
/// - Buffer size distribution
/// - Selection frequency
/// - Buffer-to-file size ratios
///
/// # Examples
/// ```ignore
/// // Uses configured profile (default: GeneralPurpose)
/// let buffer_size = get_buffer_size_opt_in(file_size);
/// ```
fn get_buffer_size_opt_in(file_size: i64) -> usize {
let buffer_size = if is_buffer_profile_enabled() {
// Use globally configured workload profile (enabled by default in Phase 3)
let config = get_global_buffer_config();
config.get_buffer_size(file_size)
} else {
// Opt-out mode: Use GeneralPurpose profile for consistent behavior
let config = RustFSBufferConfig::new(WorkloadProfile::GeneralPurpose);
config.get_buffer_size(file_size)
};
// Optional performance metrics collection for monitoring and optimization
#[cfg(feature = "metrics")]
{
use metrics::histogram;
histogram!("rustfs_buffer_size_bytes").record(buffer_size as f64);
counter!("rustfs_buffer_size_selections").increment(1);
if file_size >= 0 {
let ratio = buffer_size as f64 / file_size as f64;
histogram!("rustfs_buffer_to_file_ratio").record(ratio);
}
}
buffer_size
}
#[derive(Debug, Clone)]
@@ -411,11 +484,10 @@ impl FS {
}
};
// Use adaptive buffer sizing based on file size for optimal performance:
// - Small files (< 1MB): 64KB buffer to minimize memory overhead
// - Medium files (1MB-100MB): 256KB buffer for balanced performance
// - Large files (>= 100MB): 1MB buffer to prevent chunked stream read timeouts
let buffer_size = get_adaptive_buffer_size(size);
// Apply adaptive buffer sizing based on file size for optimal streaming performance.
// Uses workload profile configuration (enabled by default) to select appropriate buffer size.
// Buffer sizes range from 32KB to 4MB depending on file size and configured workload profile.
let buffer_size = get_buffer_size_opt_in(size);
let body = tokio::io::BufReader::with_capacity(
buffer_size,
StreamReader::new(body.map(|f| f.map_err(|e| std::io::Error::other(e.to_string())))),
@@ -2361,11 +2433,10 @@ impl S3 for FS {
return Err(s3_error!(UnexpectedContent));
}
// Use adaptive buffer sizing based on file size for optimal performance:
// - Small files (< 1MB): 64KB buffer to minimize memory overhead
// - Medium files (1MB-100MB): 256KB buffer for balanced performance
// - Large files (>= 100MB): 1MB buffer to prevent chunked stream read timeouts
let buffer_size = get_adaptive_buffer_size(size);
// Apply adaptive buffer sizing based on file size for optimal streaming performance.
// Uses workload profile configuration (enabled by default) to select appropriate buffer size.
// Buffer sizes range from 32KB to 4MB depending on file size and configured workload profile.
let buffer_size = get_buffer_size_opt_in(size);
let body = tokio::io::BufReader::with_capacity(
buffer_size,
StreamReader::new(body.map(|f| f.map_err(|e| std::io::Error::other(e.to_string())))),
@@ -2889,11 +2960,10 @@ impl S3 for FS {
let mut size = size.ok_or_else(|| s3_error!(UnexpectedContent))?;
// Use adaptive buffer sizing based on part size for optimal performance:
// - Small parts (< 1MB): 64KB buffer to minimize memory overhead
// - Medium parts (1MB-100MB): 256KB buffer for balanced performance
// - Large parts (>= 100MB): 1MB buffer to prevent chunked stream read timeouts
let buffer_size = get_adaptive_buffer_size(size);
// Apply adaptive buffer sizing based on part size for optimal streaming performance.
// Uses workload profile configuration (enabled by default) to select appropriate buffer size.
// Buffer sizes range from 32KB to 4MB depending on part size and configured workload profile.
let buffer_size = get_buffer_size_opt_in(size);
let body = tokio::io::BufReader::with_capacity(
buffer_size,
StreamReader::new(body_stream.map(|f| f.map_err(|e| std::io::Error::other(e.to_string())))),
@@ -4923,6 +4993,7 @@ pub(crate) async fn has_replication_rules(bucket: &str, objects: &[ObjectToDelet
#[cfg(test)]
mod tests {
use super::*;
use rustfs_config::MI_B;
#[test]
fn test_fs_creation() {
@@ -4996,29 +5067,201 @@ mod tests {
}
#[test]
fn test_adaptive_buffer_size() {
fn test_adaptive_buffer_size_with_profile() {
const KB: i64 = 1024;
const MB: i64 = 1024 * 1024;
// Test unknown/negative size (chunked/streaming)
assert_eq!(get_adaptive_buffer_size(-1), DEFAULT_READ_BUFFER_SIZE);
assert_eq!(get_adaptive_buffer_size(-100), DEFAULT_READ_BUFFER_SIZE);
// Test GeneralPurpose profile (default behavior, should match get_adaptive_buffer_size)
assert_eq!(
get_adaptive_buffer_size_with_profile(500 * KB, Some(WorkloadProfile::GeneralPurpose)),
64 * KB as usize
);
assert_eq!(
get_adaptive_buffer_size_with_profile(50 * MB, Some(WorkloadProfile::GeneralPurpose)),
256 * KB as usize
);
assert_eq!(
get_adaptive_buffer_size_with_profile(200 * MB, Some(WorkloadProfile::GeneralPurpose)),
DEFAULT_READ_BUFFER_SIZE
);
// Test small files (< 1MB) - should use 64KB
assert_eq!(get_adaptive_buffer_size(0), 64 * KB as usize);
assert_eq!(get_adaptive_buffer_size(512 * KB), 64 * KB as usize);
assert_eq!(get_adaptive_buffer_size(MB - 1), 64 * KB as usize);
// Test AiTraining profile - larger buffers for large files
assert_eq!(
get_adaptive_buffer_size_with_profile(5 * MB, Some(WorkloadProfile::AiTraining)),
512 * KB as usize
);
assert_eq!(
get_adaptive_buffer_size_with_profile(100 * MB, Some(WorkloadProfile::AiTraining)),
2 * MB as usize
);
assert_eq!(
get_adaptive_buffer_size_with_profile(600 * MB, Some(WorkloadProfile::AiTraining)),
4 * MB as usize
);
// Test medium files (1MB - 100MB) - should use 256KB
assert_eq!(get_adaptive_buffer_size(MB), 256 * KB as usize);
assert_eq!(get_adaptive_buffer_size(50 * MB), 256 * KB as usize);
assert_eq!(get_adaptive_buffer_size(100 * MB - 1), 256 * KB as usize);
// Test WebWorkload profile - smaller buffers for web assets
assert_eq!(
get_adaptive_buffer_size_with_profile(100 * KB, Some(WorkloadProfile::WebWorkload)),
32 * KB as usize
);
assert_eq!(
get_adaptive_buffer_size_with_profile(5 * MB, Some(WorkloadProfile::WebWorkload)),
128 * KB as usize
);
assert_eq!(
get_adaptive_buffer_size_with_profile(50 * MB, Some(WorkloadProfile::WebWorkload)),
256 * KB as usize
);
// Test large files (>= 100MB) - should use 1MB (DEFAULT_READ_BUFFER_SIZE)
assert_eq!(get_adaptive_buffer_size(100 * MB), DEFAULT_READ_BUFFER_SIZE);
assert_eq!(get_adaptive_buffer_size(500 * MB), DEFAULT_READ_BUFFER_SIZE);
assert_eq!(get_adaptive_buffer_size(10 * 1024 * MB), DEFAULT_READ_BUFFER_SIZE); // 10GB
assert_eq!(get_adaptive_buffer_size(20 * 1024 * MB), DEFAULT_READ_BUFFER_SIZE); // 20GB
// Test SecureStorage profile - memory-constrained buffers
assert_eq!(
get_adaptive_buffer_size_with_profile(500 * KB, Some(WorkloadProfile::SecureStorage)),
32 * KB as usize
);
assert_eq!(
get_adaptive_buffer_size_with_profile(25 * MB, Some(WorkloadProfile::SecureStorage)),
128 * KB as usize
);
assert_eq!(
get_adaptive_buffer_size_with_profile(100 * MB, Some(WorkloadProfile::SecureStorage)),
256 * KB as usize
);
// Test IndustrialIoT profile - low latency, moderate buffers
assert_eq!(
get_adaptive_buffer_size_with_profile(512 * KB, Some(WorkloadProfile::IndustrialIoT)),
64 * KB as usize
);
assert_eq!(
get_adaptive_buffer_size_with_profile(25 * MB, Some(WorkloadProfile::IndustrialIoT)),
256 * KB as usize
);
assert_eq!(
get_adaptive_buffer_size_with_profile(100 * MB, Some(WorkloadProfile::IndustrialIoT)),
512 * KB as usize
);
// Test DataAnalytics profile
assert_eq!(
get_adaptive_buffer_size_with_profile(2 * MB, Some(WorkloadProfile::DataAnalytics)),
128 * KB as usize
);
assert_eq!(
get_adaptive_buffer_size_with_profile(100 * MB, Some(WorkloadProfile::DataAnalytics)),
512 * KB as usize
);
assert_eq!(
get_adaptive_buffer_size_with_profile(500 * MB, Some(WorkloadProfile::DataAnalytics)),
2 * MB as usize
);
// Test with None (should auto-detect or use GeneralPurpose)
let result = get_adaptive_buffer_size_with_profile(50 * MB, None);
// Should be either SecureStorage (if on special OS) or GeneralPurpose
assert!(result == 128 * KB as usize || result == 256 * KB as usize);
// Test unknown file size with different profiles
assert_eq!(
get_adaptive_buffer_size_with_profile(-1, Some(WorkloadProfile::AiTraining)),
2 * MB as usize
);
assert_eq!(
get_adaptive_buffer_size_with_profile(-1, Some(WorkloadProfile::WebWorkload)),
128 * KB as usize
);
assert_eq!(
get_adaptive_buffer_size_with_profile(-1, Some(WorkloadProfile::SecureStorage)),
128 * KB as usize
);
}
#[test]
fn test_phase3_default_behavior() {
use crate::config::workload_profiles::{
RustFSBufferConfig, WorkloadProfile, init_global_buffer_config, set_buffer_profile_enabled,
};
const KB: i64 = 1024;
const MB: i64 = 1024 * 1024;
// Test Phase 3: Enabled by default with GeneralPurpose profile
set_buffer_profile_enabled(true);
init_global_buffer_config(RustFSBufferConfig::new(WorkloadProfile::GeneralPurpose));
// Verify GeneralPurpose profile provides consistent buffer sizes
assert_eq!(get_buffer_size_opt_in(500 * KB), 64 * KB as usize);
assert_eq!(get_buffer_size_opt_in(50 * MB), 256 * KB as usize);
assert_eq!(get_buffer_size_opt_in(200 * MB), MI_B);
assert_eq!(get_buffer_size_opt_in(-1), MI_B); // Unknown size
// Reset for other tests
set_buffer_profile_enabled(false);
}
#[test]
fn test_buffer_size_opt_in() {
use crate::config::workload_profiles::{is_buffer_profile_enabled, set_buffer_profile_enabled};
const KB: i64 = 1024;
const MB: i64 = 1024 * 1024;
// \[1\] Default state: profile is not enabled, global configuration is not explicitly initialized
// get_buffer_size_opt_in should be equivalent to the GeneralPurpose configuration
set_buffer_profile_enabled(false);
assert!(!is_buffer_profile_enabled());
// GeneralPurpose rules:
// \< 1MB -> 64KB，1MB-100MB -> 256KB，\>=100MB -> 1MB
assert_eq!(get_buffer_size_opt_in(500 * KB), 64 * KB as usize);
assert_eq!(get_buffer_size_opt_in(50 * MB), 256 * KB as usize);
assert_eq!(get_buffer_size_opt_in(200 * MB), MI_B);
// \[2\] Enable the profile switch, but the global configuration is still the default GeneralPurpose
set_buffer_profile_enabled(true);
assert!(is_buffer_profile_enabled());
assert_eq!(get_buffer_size_opt_in(500 * KB), 64 * KB as usize);
assert_eq!(get_buffer_size_opt_in(50 * MB), 256 * KB as usize);
assert_eq!(get_buffer_size_opt_in(200 * MB), MI_B);
// \[3\] Close again to ensure unchanged behavior
set_buffer_profile_enabled(false);
assert!(!is_buffer_profile_enabled());
assert_eq!(get_buffer_size_opt_in(500 * KB), 64 * KB as usize);
}
#[test]
fn test_phase4_full_integration() {
use crate::config::workload_profiles::{
RustFSBufferConfig, WorkloadProfile, init_global_buffer_config, set_buffer_profile_enabled,
};
const KB: i64 = 1024;
const MB: i64 = 1024 * 1024;
// \[1\] During the entire test process, the global configuration is initialized only once.
// In order not to interfere with other tests, use GeneralPurpose (consistent with the default).
// If it has been initialized elsewhere, this call will be ignored by OnceLock and the behavior will still be GeneralPurpose.
init_global_buffer_config(RustFSBufferConfig::new(WorkloadProfile::GeneralPurpose));
// Make sure to turn off profile initially
set_buffer_profile_enabled(false);
// \[2\] Verify behavior of get_buffer_size_opt_in in disabled profile (GeneralPurpose)
assert_eq!(get_buffer_size_opt_in(500 * KB), 64 * KB as usize);
assert_eq!(get_buffer_size_opt_in(50 * MB), 256 * KB as usize);
assert_eq!(get_buffer_size_opt_in(200 * MB), MI_B);
// \[3\] When profile is enabled, the behavior remains consistent with the global GeneralPurpose configuration
set_buffer_profile_enabled(true);
assert_eq!(get_buffer_size_opt_in(500 * KB), 64 * KB as usize);
assert_eq!(get_buffer_size_opt_in(50 * MB), 256 * KB as usize);
assert_eq!(get_buffer_size_opt_in(200 * MB), MI_B);
// \[4\] Complex scenes, boundary values: such as unknown size
assert_eq!(get_buffer_size_opt_in(-1), MI_B);
set_buffer_profile_enabled(false);
}
// Note: S3Request structure is complex and requires many fields.