refactor: replace lazy_static with LazyLock and notify crate registry create_targets_from_config (#311)

* improve code for notify

* improve code for logger and fix typo (#272)

* Add GNU to  build.yml (#275)

* fix unzip error

* fix url change error

fix url change error

* Simplify user experience and integrate console and endpoint

Simplify user experience and integrate console and endpoint

* Add gnu to  build.yml

* upgrade version

* feat: add `cargo clippy --fix --allow-dirty` to pre-commit command (#282)

Resolves #277

- Add --fix flag to automatically fix clippy warnings
- Add --allow-dirty flag to run on dirty Git trees
- Improves code quality in pre-commit workflow

* fix: the issue where preview fails when the path length exceeds 255 characters (#280)

* fix

* fix: improve Windows build support and CI/CD workflow (#283)

- Fix Windows zip command issue by using PowerShell Compress-Archive
- Add Windows support for OSS upload with ossutil
- Replace Chinese comments with English in build.yml
- Fix bash syntax error in package_zip function
- Improve code formatting and consistency
- Update various configuration files for better cross-platform support

Resolves Windows build failures in GitHub Actions.

* fix: update link in README.md leading to a 404 error (#285)

* add rustfs.spec for rustfs (#103)

add support on loongarch64

* improve cargo.lock

* build(deps): bump the dependencies group with 5 updates (#289)

Bumps the dependencies group with 5 updates:

| Package | From | To |
| --- | --- | --- |
| [hyper-util](https://github.com/hyperium/hyper-util) | `0.1.15` | `0.1.16` |
| [rand](https://github.com/rust-random/rand) | `0.9.1` | `0.9.2` |
| [serde_json](https://github.com/serde-rs/json) | `1.0.140` | `1.0.141` |
| [strum](https://github.com/Peternator7/strum) | `0.27.1` | `0.27.2` |
| [sysinfo](https://github.com/GuillaumeGomez/sysinfo) | `0.36.0` | `0.36.1` |


Updates `hyper-util` from 0.1.15 to 0.1.16
- [Release notes](https://github.com/hyperium/hyper-util/releases)
- [Changelog](https://github.com/hyperium/hyper-util/blob/master/CHANGELOG.md)
- [Commits](https://github.com/hyperium/hyper-util/compare/v0.1.15...v0.1.16)

Updates `rand` from 0.9.1 to 0.9.2
- [Release notes](https://github.com/rust-random/rand/releases)
- [Changelog](https://github.com/rust-random/rand/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-random/rand/compare/rand_core-0.9.1...rand_core-0.9.2)

Updates `serde_json` from 1.0.140 to 1.0.141
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.140...v1.0.141)

Updates `strum` from 0.27.1 to 0.27.2
- [Release notes](https://github.com/Peternator7/strum/releases)
- [Changelog](https://github.com/Peternator7/strum/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Peternator7/strum/compare/v0.27.1...v0.27.2)

Updates `sysinfo` from 0.36.0 to 0.36.1
- [Changelog](https://github.com/GuillaumeGomez/sysinfo/blob/master/CHANGELOG.md)
- [Commits](https://github.com/GuillaumeGomez/sysinfo/compare/v0.36.0...v0.36.1)

---
updated-dependencies:
- dependency-name: hyper-util
  dependency-version: 0.1.16
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: rand
  dependency-version: 0.9.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: serde_json
  dependency-version: 1.0.141
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: strum
  dependency-version: 0.27.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: sysinfo
  dependency-version: 0.36.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* improve code for logger

* improve

* upgrade

* refactor: 优化构建工作流,统一 latest 文件处理和简化制品上传 (#293)

* Refactor: DatabaseManagerSystem as global

Signed-off-by: junxiang Mu <1948535941@qq.com>

* fix: fmt

Signed-off-by: junxiang Mu <1948535941@qq.com>

* Test: add e2e_test for s3select

Signed-off-by: junxiang Mu <1948535941@qq.com>

* Test: add test script for e2e

Signed-off-by: junxiang Mu <1948535941@qq.com>

* improve code for registry and intergation

* improve code for registry `create_targets_from_config`

* fix

* Feature up/ilm (#305)

* fix

* fix

* fix

* fix delete-marker expiration. add api_restore.

* fix

* time retry object upload

* lock file

* make fmt

* fix

* restore object

* fix

* fix

* serde-rs-xml -> quick-xml

* fix

* checksum

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* transfer lang to english

* upgrade clap version from 4.5.41 to 4.5.42

* refactor: replace `lazy_static` with `LazyLock`

* add router

* fix: modify comment

* improve code

* fix typos

* fix

* fix: modify name and fmt

* improve code for registry

* fix test

---------

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: junxiang Mu <1948535941@qq.com>
Co-authored-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: shiro.lee <69624924+shiroleeee@users.noreply.github.com>
Co-authored-by: Marco Orlandin <mipnamic@mipnamic.net>
Co-authored-by: zhangwenlong <zhangwenlong@loongson.cn>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: junxiang Mu <1948535941@qq.com>
Co-authored-by: likewu <likewu@126.com>
This commit is contained in:
houseme
2025-07-30 19:02:10 +08:00
committed by GitHub
parent 4f1770d3fe
commit c5df1f92c2
49 changed files with 1575 additions and 1435 deletions
+18 -8
View File
@@ -15,9 +15,9 @@
use const_str::concat;
/// Application name
/// Default value: RustFs
/// Default value: RustFS
/// Environment variable: RUSTFS_APP_NAME
pub const APP_NAME: &str = "RustFs";
pub const APP_NAME: &str = "RustFS";
/// Application version
/// Default value: 1.0.0
/// Environment variable: RUSTFS_VERSION
@@ -71,6 +71,16 @@ pub const DEFAULT_ACCESS_KEY: &str = "rustfsadmin";
/// Example: --secret-key rustfsadmin
pub const DEFAULT_SECRET_KEY: &str = "rustfsadmin";
/// Default console enable
/// This is the default value for the console server.
/// It is used to enable or disable the console server.
/// Default value: true
/// Environment variable: RUSTFS_CONSOLE_ENABLE
/// Command line argument: --console-enable
/// Example: RUSTFS_CONSOLE_ENABLE=true
/// Example: --console-enable true
pub const DEFAULT_CONSOLE_ENABLE: bool = true;
/// Default OBS configuration endpoint
/// Environment variable: DEFAULT_OBS_ENDPOINT
/// Command line argument: --obs-endpoint
@@ -126,28 +136,28 @@ pub const DEFAULT_SINK_FILE_LOG_FILE: &str = concat!(DEFAULT_LOG_FILENAME, "-sin
/// This is the default log directory for rustfs.
/// It is used to store the logs of the application.
/// Default value: logs
/// Environment variable: RUSTFS_OBSERVABILITY_LOG_DIRECTORY
pub const DEFAULT_LOG_DIR: &str = "/logs";
/// Environment variable: RUSTFS_LOG_DIRECTORY
pub const DEFAULT_LOG_DIR: &str = "logs";
/// Default log rotation size mb for rustfs
/// This is the default log rotation size for rustfs.
/// It is used to rotate the logs of the application.
/// Default value: 100 MB
/// Environment variable: RUSTFS_OBSERVABILITY_LOG_ROTATION_SIZE_MB
/// Environment variable: RUSTFS_OBS_LOG_ROTATION_SIZE_MB
pub const DEFAULT_LOG_ROTATION_SIZE_MB: u64 = 100;
/// Default log rotation time for rustfs
/// This is the default log rotation time for rustfs.
/// It is used to rotate the logs of the application.
/// Default value: hour, eg: day,hour,minute,second
/// Environment variable: RUSTFS_OBSERVABILITY_LOG_ROTATION_TIME
/// Environment variable: RUSTFS_OBS_LOG_ROTATION_TIME
pub const DEFAULT_LOG_ROTATION_TIME: &str = "day";
/// Default log keep files for rustfs
/// This is the default log keep files for rustfs.
/// It is used to keep the logs of the application.
/// Default value: 30
/// Environment variable: RUSTFS_OBSERVABILITY_LOG_KEEP_FILES
/// Environment variable: RUSTFS_OBS_LOG_KEEP_FILES
pub const DEFAULT_LOG_KEEP_FILES: u16 = 30;
#[cfg(test)]
@@ -157,7 +167,7 @@ mod tests {
#[test]
fn test_app_basic_constants() {
// Test application basic constants
assert_eq!(APP_NAME, "RustFs");
assert_eq!(APP_NAME, "RustFS");
assert!(!APP_NAME.contains(' '), "App name should not contain spaces");
assert_eq!(VERSION, "0.0.1");
+9 -1
View File
@@ -27,7 +27,15 @@ pub const DEFAULT_TARGET: &str = "1";
pub const NOTIFY_PREFIX: &str = "notify";
pub const NOTIFY_ROUTE_PREFIX: &str = "notify_";
pub const NOTIFY_ROUTE_PREFIX: &str = const_str::concat!(NOTIFY_PREFIX, "_");
/// Standard config keys and values.
pub const ENABLE_KEY: &str = "enable";
pub const COMMENT_KEY: &str = "comment";
/// Enable values
pub const ENABLE_ON: &str = "on";
pub const ENABLE_OFF: &str = "off";
#[allow(dead_code)]
pub const NOTIFY_SUB_SYSTEMS: &[&str] = &[NOTIFY_MQTT_SUB_SYS, NOTIFY_WEBHOOK_SUB_SYS];
+30
View File
@@ -12,6 +12,8 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::notify::{COMMENT_KEY, ENABLE_KEY};
// MQTT Keys
pub const MQTT_BROKER: &str = "broker";
pub const MQTT_TOPIC: &str = "topic";
@@ -23,6 +25,21 @@ pub const MQTT_KEEP_ALIVE_INTERVAL: &str = "keep_alive_interval";
pub const MQTT_QUEUE_DIR: &str = "queue_dir";
pub const MQTT_QUEUE_LIMIT: &str = "queue_limit";
/// A list of all valid configuration keys for an MQTT target.
pub const NOTIFY_MQTT_KEYS: &[&str] = &[
ENABLE_KEY, // "enable" is a common key
MQTT_BROKER,
MQTT_TOPIC,
MQTT_QOS,
MQTT_USERNAME,
MQTT_PASSWORD,
MQTT_RECONNECT_INTERVAL,
MQTT_KEEP_ALIVE_INTERVAL,
MQTT_QUEUE_DIR,
MQTT_QUEUE_LIMIT,
COMMENT_KEY,
];
// MQTT Environment Variables
pub const ENV_MQTT_ENABLE: &str = "RUSTFS_NOTIFY_MQTT_ENABLE";
pub const ENV_MQTT_BROKER: &str = "RUSTFS_NOTIFY_MQTT_BROKER";
@@ -34,3 +51,16 @@ pub const ENV_MQTT_RECONNECT_INTERVAL: &str = "RUSTFS_NOTIFY_MQTT_RECONNECT_INTE
pub const ENV_MQTT_KEEP_ALIVE_INTERVAL: &str = "RUSTFS_NOTIFY_MQTT_KEEP_ALIVE_INTERVAL";
pub const ENV_MQTT_QUEUE_DIR: &str = "RUSTFS_NOTIFY_MQTT_QUEUE_DIR";
pub const ENV_MQTT_QUEUE_LIMIT: &str = "RUSTFS_NOTIFY_MQTT_QUEUE_LIMIT";
pub const ENV_NOTIFY_MQTT_KEYS: &[&str; 10] = &[
ENV_MQTT_ENABLE,
ENV_MQTT_BROKER,
ENV_MQTT_TOPIC,
ENV_MQTT_QOS,
ENV_MQTT_USERNAME,
ENV_MQTT_PASSWORD,
ENV_MQTT_RECONNECT_INTERVAL,
ENV_MQTT_KEEP_ALIVE_INTERVAL,
ENV_MQTT_QUEUE_DIR,
ENV_MQTT_QUEUE_LIMIT,
];
+24
View File
@@ -12,6 +12,8 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::notify::{COMMENT_KEY, ENABLE_KEY};
// Webhook Keys
pub const WEBHOOK_ENDPOINT: &str = "endpoint";
pub const WEBHOOK_AUTH_TOKEN: &str = "auth_token";
@@ -20,6 +22,18 @@ pub const WEBHOOK_QUEUE_DIR: &str = "queue_dir";
pub const WEBHOOK_CLIENT_CERT: &str = "client_cert";
pub const WEBHOOK_CLIENT_KEY: &str = "client_key";
/// A list of all valid configuration keys for a webhook target.
pub const NOTIFY_WEBHOOK_KEYS: &[&str] = &[
ENABLE_KEY, // "enable" is a common key
WEBHOOK_ENDPOINT,
WEBHOOK_AUTH_TOKEN,
WEBHOOK_QUEUE_LIMIT,
WEBHOOK_QUEUE_DIR,
WEBHOOK_CLIENT_CERT,
WEBHOOK_CLIENT_KEY,
COMMENT_KEY,
];
// Webhook Environment Variables
pub const ENV_WEBHOOK_ENABLE: &str = "RUSTFS_NOTIFY_WEBHOOK_ENABLE";
pub const ENV_WEBHOOK_ENDPOINT: &str = "RUSTFS_NOTIFY_WEBHOOK_ENDPOINT";
@@ -28,3 +42,13 @@ pub const ENV_WEBHOOK_QUEUE_LIMIT: &str = "RUSTFS_NOTIFY_WEBHOOK_QUEUE_LIMIT";
pub const ENV_WEBHOOK_QUEUE_DIR: &str = "RUSTFS_NOTIFY_WEBHOOK_QUEUE_DIR";
pub const ENV_WEBHOOK_CLIENT_CERT: &str = "RUSTFS_NOTIFY_WEBHOOK_CLIENT_CERT";
pub const ENV_WEBHOOK_CLIENT_KEY: &str = "RUSTFS_NOTIFY_WEBHOOK_CLIENT_KEY";
pub const ENV_NOTIFY_WEBHOOK_KEYS: &[&str; 7] = &[
ENV_WEBHOOK_ENABLE,
ENV_WEBHOOK_ENDPOINT,
ENV_WEBHOOK_AUTH_TOKEN,
ENV_WEBHOOK_QUEUE_LIMIT,
ENV_WEBHOOK_QUEUE_DIR,
ENV_WEBHOOK_CLIENT_CERT,
ENV_WEBHOOK_CLIENT_KEY,
];
+18 -273
View File
@@ -12,279 +12,24 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::observability::logger::LoggerConfig;
use crate::observability::otel::OtelConfig;
use crate::observability::sink::SinkConfig;
use serde::{Deserialize, Serialize};
// Observability Keys
/// Observability configuration
#[derive(Debug, Deserialize, Serialize, Clone)]
pub struct ObservabilityConfig {
pub otel: OtelConfig,
pub sinks: Vec<SinkConfig>,
pub logger: Option<LoggerConfig>,
}
pub const ENV_OBS_ENDPOINT: &str = "RUSTFS_OBS_ENDPOINT";
pub const ENV_OBS_USE_STDOUT: &str = "RUSTFS_OBS_USE_STDOUT";
pub const ENV_OBS_SAMPLE_RATIO: &str = "RUSTFS_OBS_SAMPLE_RATIO";
pub const ENV_OBS_METER_INTERVAL: &str = "RUSTFS_OBS_METER_INTERVAL";
pub const ENV_OBS_SERVICE_NAME: &str = "RUSTFS_OBS_SERVICE_NAME";
pub const ENV_OBS_SERVICE_VERSION: &str = "RUSTFS_OBS_SERVICE_VERSION";
pub const ENV_OBS_ENVIRONMENT: &str = "RUSTFS_OBS_ENVIRONMENT";
pub const ENV_OBS_LOGGER_LEVEL: &str = "RUSTFS_OBS_LOGGER_LEVEL";
pub const ENV_OBS_LOCAL_LOGGING_ENABLED: &str = "RUSTFS_OBS_LOCAL_LOGGING_ENABLED";
pub const ENV_OBS_LOG_DIRECTORY: &str = "RUSTFS_OBS_LOG_DIRECTORY";
pub const ENV_OBS_LOG_FILENAME: &str = "RUSTFS_OBS_LOG_FILENAME";
pub const ENV_OBS_LOG_ROTATION_SIZE_MB: &str = "RUSTFS_OBS_LOG_ROTATION_SIZE_MB";
pub const ENV_OBS_LOG_ROTATION_TIME: &str = "RUSTFS_OBS_LOG_ROTATION_TIME";
pub const ENV_OBS_LOG_KEEP_FILES: &str = "RUSTFS_OBS_LOG_KEEP_FILES";
impl ObservabilityConfig {
pub fn new() -> Self {
Self {
otel: OtelConfig::new(),
sinks: vec![SinkConfig::new()],
logger: Some(LoggerConfig::new()),
}
}
}
pub const ENV_AUDIT_LOGGER_QUEUE_CAPACITY: &str = "RUSTFS_AUDIT_LOGGER_QUEUE_CAPACITY";
impl Default for ObservabilityConfig {
fn default() -> Self {
Self::new()
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_observability_config_new() {
let config = ObservabilityConfig::new();
// Verify OTEL config is initialized
assert!(config.otel.use_stdout.is_some(), "OTEL use_stdout should be configured");
assert!(config.otel.sample_ratio.is_some(), "OTEL sample_ratio should be configured");
assert!(config.otel.meter_interval.is_some(), "OTEL meter_interval should be configured");
assert!(config.otel.service_name.is_some(), "OTEL service_name should be configured");
assert!(config.otel.service_version.is_some(), "OTEL service_version should be configured");
assert!(config.otel.environment.is_some(), "OTEL environment should be configured");
assert!(config.otel.logger_level.is_some(), "OTEL logger_level should be configured");
// Verify sinks are initialized
assert!(!config.sinks.is_empty(), "Sinks should not be empty");
assert_eq!(config.sinks.len(), 1, "Should have exactly one default sink");
// Verify logger is initialized
assert!(config.logger.is_some(), "Logger should be configured");
}
#[test]
fn test_observability_config_default() {
let config = ObservabilityConfig::default();
let new_config = ObservabilityConfig::new();
// Default should be equivalent to new()
assert_eq!(config.sinks.len(), new_config.sinks.len());
assert_eq!(config.logger.is_some(), new_config.logger.is_some());
// OTEL configs should be equivalent
assert_eq!(config.otel.use_stdout, new_config.otel.use_stdout);
assert_eq!(config.otel.sample_ratio, new_config.otel.sample_ratio);
assert_eq!(config.otel.meter_interval, new_config.otel.meter_interval);
assert_eq!(config.otel.service_name, new_config.otel.service_name);
assert_eq!(config.otel.service_version, new_config.otel.service_version);
assert_eq!(config.otel.environment, new_config.otel.environment);
assert_eq!(config.otel.logger_level, new_config.otel.logger_level);
}
#[test]
fn test_observability_config_otel_defaults() {
let config = ObservabilityConfig::new();
// Test OTEL default values
if let Some(_use_stdout) = config.otel.use_stdout {
// Test boolean values - any boolean value is valid
}
if let Some(sample_ratio) = config.otel.sample_ratio {
assert!((0.0..=1.0).contains(&sample_ratio), "Sample ratio should be between 0.0 and 1.0");
}
if let Some(meter_interval) = config.otel.meter_interval {
assert!(meter_interval > 0, "Meter interval should be positive");
assert!(meter_interval <= 3600, "Meter interval should be reasonable (≤ 1 hour)");
}
if let Some(service_name) = &config.otel.service_name {
assert!(!service_name.is_empty(), "Service name should not be empty");
assert!(!service_name.contains(' '), "Service name should not contain spaces");
}
if let Some(service_version) = &config.otel.service_version {
assert!(!service_version.is_empty(), "Service version should not be empty");
}
if let Some(environment) = &config.otel.environment {
assert!(!environment.is_empty(), "Environment should not be empty");
assert!(
["development", "staging", "production", "test"].contains(&environment.as_str()),
"Environment should be a standard environment name"
);
}
if let Some(logger_level) = &config.otel.logger_level {
assert!(
["trace", "debug", "info", "warn", "error"].contains(&logger_level.as_str()),
"Logger level should be a valid tracing level"
);
}
}
#[test]
fn test_observability_config_sinks() {
let config = ObservabilityConfig::new();
// Test default sink configuration
assert_eq!(config.sinks.len(), 1, "Should have exactly one default sink");
let _default_sink = &config.sinks[0];
// Test that the sink has valid configuration
// Note: We can't test specific values without knowing SinkConfig implementation
// but we can test that it's properly initialized
// Test that we can add more sinks
let mut config_mut = config.clone();
config_mut.sinks.push(SinkConfig::new());
assert_eq!(config_mut.sinks.len(), 2, "Should be able to add more sinks");
}
#[test]
fn test_observability_config_logger() {
let config = ObservabilityConfig::new();
// Test logger configuration
assert!(config.logger.is_some(), "Logger should be configured by default");
if let Some(_logger) = &config.logger {
// Test that logger has valid configuration
// Note: We can't test specific values without knowing LoggerConfig implementation
// but we can test that it's properly initialized
}
// Test that logger can be disabled
let mut config_mut = config.clone();
config_mut.logger = None;
assert!(config_mut.logger.is_none(), "Logger should be able to be disabled");
}
#[test]
fn test_observability_config_serialization() {
let config = ObservabilityConfig::new();
// Test serialization to JSON
let json_result = serde_json::to_string(&config);
assert!(json_result.is_ok(), "Config should be serializable to JSON");
let json_str = json_result.unwrap();
assert!(!json_str.is_empty(), "Serialized JSON should not be empty");
assert!(json_str.contains("otel"), "JSON should contain otel configuration");
assert!(json_str.contains("sinks"), "JSON should contain sinks configuration");
assert!(json_str.contains("logger"), "JSON should contain logger configuration");
// Test deserialization from JSON
let deserialized_result: Result<ObservabilityConfig, _> = serde_json::from_str(&json_str);
assert!(deserialized_result.is_ok(), "Config should be deserializable from JSON");
let deserialized_config = deserialized_result.unwrap();
assert_eq!(deserialized_config.sinks.len(), config.sinks.len());
assert_eq!(deserialized_config.logger.is_some(), config.logger.is_some());
}
#[test]
fn test_observability_config_debug_format() {
let config = ObservabilityConfig::new();
let debug_str = format!("{config:?}");
assert!(!debug_str.is_empty(), "Debug output should not be empty");
assert!(debug_str.contains("ObservabilityConfig"), "Debug output should contain struct name");
assert!(debug_str.contains("otel"), "Debug output should contain otel field");
assert!(debug_str.contains("sinks"), "Debug output should contain sinks field");
assert!(debug_str.contains("logger"), "Debug output should contain logger field");
}
#[test]
fn test_observability_config_clone() {
let config = ObservabilityConfig::new();
let cloned_config = config.clone();
// Test that clone creates an independent copy
assert_eq!(cloned_config.sinks.len(), config.sinks.len());
assert_eq!(cloned_config.logger.is_some(), config.logger.is_some());
assert_eq!(cloned_config.otel.endpoint, config.otel.endpoint);
assert_eq!(cloned_config.otel.use_stdout, config.otel.use_stdout);
assert_eq!(cloned_config.otel.sample_ratio, config.otel.sample_ratio);
assert_eq!(cloned_config.otel.meter_interval, config.otel.meter_interval);
assert_eq!(cloned_config.otel.service_name, config.otel.service_name);
assert_eq!(cloned_config.otel.service_version, config.otel.service_version);
assert_eq!(cloned_config.otel.environment, config.otel.environment);
assert_eq!(cloned_config.otel.logger_level, config.otel.logger_level);
}
#[test]
fn test_observability_config_modification() {
let mut config = ObservabilityConfig::new();
// Test modifying OTEL endpoint
let original_endpoint = config.otel.endpoint.clone();
config.otel.endpoint = "http://localhost:4317".to_string();
assert_ne!(config.otel.endpoint, original_endpoint);
assert_eq!(config.otel.endpoint, "http://localhost:4317");
// Test modifying sinks
let original_sinks_len = config.sinks.len();
config.sinks.push(SinkConfig::new());
assert_eq!(config.sinks.len(), original_sinks_len + 1);
// Test disabling logger
config.logger = None;
assert!(config.logger.is_none());
}
#[test]
fn test_observability_config_edge_cases() {
// Test with empty sinks
let mut config = ObservabilityConfig::new();
config.sinks.clear();
assert!(config.sinks.is_empty(), "Sinks should be empty after clearing");
// Test serialization with empty sinks
let json_result = serde_json::to_string(&config);
assert!(json_result.is_ok(), "Config with empty sinks should be serializable");
// Test with no logger
config.logger = None;
let json_result = serde_json::to_string(&config);
assert!(json_result.is_ok(), "Config with no logger should be serializable");
}
#[test]
fn test_observability_config_memory_efficiency() {
let config = ObservabilityConfig::new();
// Test that config doesn't use excessive memory
let config_size = std::mem::size_of_val(&config);
assert!(config_size < 5000, "Config should not use excessive memory");
// Test that endpoint string is not excessively long
assert!(config.otel.endpoint.len() < 1000, "Endpoint should not be excessively long");
// Test that collections are reasonably sized
assert!(config.sinks.len() < 100, "Sinks collection should be reasonably sized");
}
#[test]
fn test_observability_config_consistency() {
// Create multiple configs and ensure they're consistent
let config1 = ObservabilityConfig::new();
let config2 = ObservabilityConfig::new();
// Both configs should have the same default structure
assert_eq!(config1.sinks.len(), config2.sinks.len());
assert_eq!(config1.logger.is_some(), config2.logger.is_some());
assert_eq!(config1.otel.use_stdout, config2.otel.use_stdout);
assert_eq!(config1.otel.sample_ratio, config2.otel.sample_ratio);
assert_eq!(config1.otel.meter_interval, config2.otel.meter_interval);
assert_eq!(config1.otel.service_name, config2.otel.service_name);
assert_eq!(config1.otel.service_version, config2.otel.service_version);
assert_eq!(config1.otel.environment, config2.otel.environment);
assert_eq!(config1.otel.logger_level, config2.otel.logger_level);
}
}
// Default values for observability configuration
pub const DEFAULT_AUDIT_LOGGER_QUEUE_CAPACITY: usize = 10000;
+11 -56
View File
@@ -12,62 +12,17 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use serde::{Deserialize, Serialize};
use std::env;
// RUSTFS_SINKS_FILE_PATH
pub const ENV_SINKS_FILE_PATH: &str = "RUSTFS_SINKS_FILE_PATH";
// RUSTFS_SINKS_FILE_BUFFER_SIZE
pub const ENV_SINKS_FILE_BUFFER_SIZE: &str = "RUSTFS_SINKS_FILE_BUFFER_SIZE";
// RUSTFS_SINKS_FILE_FLUSH_INTERVAL_MS
pub const ENV_SINKS_FILE_FLUSH_INTERVAL_MS: &str = "RUSTFS_SINKS_FILE_FLUSH_INTERVAL_MS";
// RUSTFS_SINKS_FILE_FLUSH_THRESHOLD
pub const ENV_SINKS_FILE_FLUSH_THRESHOLD: &str = "RUSTFS_SINKS_FILE_FLUSH_THRESHOLD";
/// File sink configuration
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct FileSink {
pub path: String,
#[serde(default = "default_buffer_size")]
pub buffer_size: Option<usize>,
#[serde(default = "default_flush_interval_ms")]
pub flush_interval_ms: Option<u64>,
#[serde(default = "default_flush_threshold")]
pub flush_threshold: Option<usize>,
}
pub const DEFAULT_SINKS_FILE_BUFFER_SIZE: usize = 8192;
impl FileSink {
pub fn new() -> Self {
Self {
path: env::var("RUSTFS_SINKS_FILE_PATH")
.ok()
.filter(|s| !s.trim().is_empty())
.unwrap_or_else(default_path),
buffer_size: default_buffer_size(),
flush_interval_ms: default_flush_interval_ms(),
flush_threshold: default_flush_threshold(),
}
}
}
pub const DEFAULT_SINKS_FILE_FLUSH_INTERVAL_MS: u64 = 1000;
impl Default for FileSink {
fn default() -> Self {
Self::new()
}
}
fn default_buffer_size() -> Option<usize> {
Some(8192)
}
fn default_flush_interval_ms() -> Option<u64> {
Some(1000)
}
fn default_flush_threshold() -> Option<usize> {
Some(100)
}
fn default_path() -> String {
let temp_dir = env::temp_dir().join("rustfs");
if let Err(e) = std::fs::create_dir_all(&temp_dir) {
eprintln!("Failed to create log directory: {e}");
return "rustfs/rustfs.log".to_string();
}
temp_dir
.join("rustfs.log")
.to_str()
.unwrap_or("rustfs/rustfs.log")
.to_string()
}
pub const DEFAULT_SINKS_FILE_FLUSH_THRESHOLD: usize = 100;
+12 -35
View File
@@ -12,39 +12,16 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use serde::{Deserialize, Serialize};
// RUSTFS_SINKS_KAFKA_BROKERS
pub const ENV_SINKS_KAFKA_BROKERS: &str = "RUSTFS_SINKS_KAFKA_BROKERS";
pub const ENV_SINKS_KAFKA_TOPIC: &str = "RUSTFS_SINKS_KAFKA_TOPIC";
// batch_size
pub const ENV_SINKS_KAFKA_BATCH_SIZE: &str = "RUSTFS_SINKS_KAFKA_BATCH_SIZE";
// batch_timeout_ms
pub const ENV_SINKS_KAFKA_BATCH_TIMEOUT_MS: &str = "RUSTFS_SINKS_KAFKA_BATCH_TIMEOUT_MS";
/// Kafka sink configuration
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct KafkaSink {
pub brokers: String,
pub topic: String,
#[serde(default = "default_batch_size")]
pub batch_size: Option<usize>,
#[serde(default = "default_batch_timeout_ms")]
pub batch_timeout_ms: Option<u64>,
}
impl KafkaSink {
pub fn new() -> Self {
Self {
brokers: "localhost:9092".to_string(),
topic: "rustfs".to_string(),
batch_size: default_batch_size(),
batch_timeout_ms: default_batch_timeout_ms(),
}
}
}
impl Default for KafkaSink {
fn default() -> Self {
Self::new()
}
}
fn default_batch_size() -> Option<usize> {
Some(100)
}
fn default_batch_timeout_ms() -> Option<u64> {
Some(1000)
}
// brokers
pub const DEFAULT_SINKS_KAFKA_BROKERS: &str = "localhost:9092";
pub const DEFAULT_SINKS_KAFKA_TOPIC: &str = "rustfs-sinks";
pub const DEFAULT_SINKS_KAFKA_BATCH_SIZE: usize = 100;
pub const DEFAULT_SINKS_KAFKA_BATCH_TIMEOUT_MS: u64 = 1000;
-35
View File
@@ -1,35 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use serde::{Deserialize, Serialize};
/// Logger configuration
#[derive(Debug, Deserialize, Serialize, Clone)]
pub struct LoggerConfig {
pub queue_capacity: Option<usize>,
}
impl LoggerConfig {
pub fn new() -> Self {
Self {
queue_capacity: Some(10000),
}
}
}
impl Default for LoggerConfig {
fn default() -> Self {
Self::new()
}
}
+9 -7
View File
@@ -12,10 +12,12 @@
// See the License for the specific language governing permissions and
// limitations under the License.
pub(crate) mod config;
pub(crate) mod file;
pub(crate) mod kafka;
pub(crate) mod logger;
pub(crate) mod otel;
pub(crate) mod sink;
pub(crate) mod webhook;
mod config;
mod file;
mod kafka;
mod webhook;
pub use config::*;
pub use file::*;
pub use kafka::*;
pub use webhook::*;
-83
View File
@@ -1,83 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::constants::app::{ENVIRONMENT, METER_INTERVAL, SAMPLE_RATIO, SERVICE_VERSION, USE_STDOUT};
use crate::{APP_NAME, DEFAULT_LOG_LEVEL};
use serde::{Deserialize, Serialize};
use std::env;
/// OpenTelemetry configuration
#[derive(Debug, Deserialize, Serialize, Clone)]
pub struct OtelConfig {
pub endpoint: String, // Endpoint for metric collection
pub use_stdout: Option<bool>, // Output to stdout
pub sample_ratio: Option<f64>, // Trace sampling ratio
pub meter_interval: Option<u64>, // Metric collection interval
pub service_name: Option<String>, // Service name
pub service_version: Option<String>, // Service version
pub environment: Option<String>, // Environment
pub logger_level: Option<String>, // Logger level
pub local_logging_enabled: Option<bool>, // Local logging enabled
}
impl OtelConfig {
pub fn new() -> Self {
extract_otel_config_from_env()
}
}
impl Default for OtelConfig {
fn default() -> Self {
Self::new()
}
}
// Helper function: Extract observable configuration from environment variables
fn extract_otel_config_from_env() -> OtelConfig {
OtelConfig {
endpoint: env::var("RUSTFS_OBSERVABILITY_ENDPOINT").unwrap_or_else(|_| "".to_string()),
use_stdout: env::var("RUSTFS_OBSERVABILITY_USE_STDOUT")
.ok()
.and_then(|v| v.parse().ok())
.or(Some(USE_STDOUT)),
sample_ratio: env::var("RUSTFS_OBSERVABILITY_SAMPLE_RATIO")
.ok()
.and_then(|v| v.parse().ok())
.or(Some(SAMPLE_RATIO)),
meter_interval: env::var("RUSTFS_OBSERVABILITY_METER_INTERVAL")
.ok()
.and_then(|v| v.parse().ok())
.or(Some(METER_INTERVAL)),
service_name: env::var("RUSTFS_OBSERVABILITY_SERVICE_NAME")
.ok()
.and_then(|v| v.parse().ok())
.or(Some(APP_NAME.to_string())),
service_version: env::var("RUSTFS_OBSERVABILITY_SERVICE_VERSION")
.ok()
.and_then(|v| v.parse().ok())
.or(Some(SERVICE_VERSION.to_string())),
environment: env::var("RUSTFS_OBSERVABILITY_ENVIRONMENT")
.ok()
.and_then(|v| v.parse().ok())
.or(Some(ENVIRONMENT.to_string())),
logger_level: env::var("RUSTFS_OBSERVABILITY_LOGGER_LEVEL")
.ok()
.and_then(|v| v.parse().ok())
.or(Some(DEFAULT_LOG_LEVEL.to_string())),
local_logging_enabled: env::var("RUSTFS_OBSERVABILITY_LOCAL_LOGGING_ENABLED")
.ok()
.and_then(|v| v.parse().ok())
.or(Some(false)),
}
}
-39
View File
@@ -1,39 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::observability::file::FileSink;
use crate::observability::kafka::KafkaSink;
use crate::observability::webhook::WebhookSink;
use serde::{Deserialize, Serialize};
/// Sink configuration
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(tag = "type")]
pub enum SinkConfig {
Kafka(KafkaSink),
Webhook(WebhookSink),
File(FileSink),
}
impl SinkConfig {
pub fn new() -> Self {
Self::File(FileSink::new())
}
}
impl Default for SinkConfig {
fn default() -> Self {
Self::new()
}
}
+13 -38
View File
@@ -12,42 +12,17 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use serde::{Deserialize, Serialize};
use std::collections::HashMap;
// RUSTFS_SINKS_WEBHOOK_ENDPOINT
pub const ENV_SINKS_WEBHOOK_ENDPOINT: &str = "RUSTFS_SINKS_WEBHOOK_ENDPOINT";
// RUSTFS_SINKS_WEBHOOK_AUTH_TOKEN
pub const ENV_SINKS_WEBHOOK_AUTH_TOKEN: &str = "RUSTFS_SINKS_WEBHOOK_AUTH_TOKEN";
// max_retries
pub const ENV_SINKS_WEBHOOK_MAX_RETRIES: &str = "RUSTFS_SINKS_WEBHOOK_MAX_RETRIES";
// retry_delay_ms
pub const ENV_SINKS_WEBHOOK_RETRY_DELAY_MS: &str = "RUSTFS_SINKS_WEBHOOK_RETRY_DELAY_MS";
/// Webhook sink configuration
#[derive(Debug, Deserialize, Serialize, Clone)]
pub struct WebhookSink {
pub endpoint: String,
pub auth_token: String,
pub headers: Option<HashMap<String, String>>,
#[serde(default = "default_max_retries")]
pub max_retries: Option<usize>,
#[serde(default = "default_retry_delay_ms")]
pub retry_delay_ms: Option<u64>,
}
impl WebhookSink {
pub fn new() -> Self {
Self {
endpoint: "".to_string(),
auth_token: "".to_string(),
headers: Some(HashMap::new()),
max_retries: default_max_retries(),
retry_delay_ms: default_retry_delay_ms(),
}
}
}
impl Default for WebhookSink {
fn default() -> Self {
Self::new()
}
}
fn default_max_retries() -> Option<usize> {
Some(3)
}
fn default_retry_delay_ms() -> Option<u64> {
Some(100)
}
// Default values for webhook sink configuration
pub const DEFAULT_SINKS_WEBHOOK_ENDPOINT: &str = "http://localhost:8080";
pub const DEFAULT_SINKS_WEBHOOK_AUTH_TOKEN: &str = "";
pub const DEFAULT_SINKS_WEBHOOK_MAX_RETRIES: usize = 3;
pub const DEFAULT_SINKS_WEBHOOK_RETRY_DELAY_MS: u64 = 100;