gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-08-27 15:37:02 +00:00
Code Issues Packages Projects Releases Wiki Activity

refactor(obs): enhance log rotation robustness and refine filter logic (#2155)

Browse Source 
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
This commit is contained in:
houseme
2026-03-14 09:20:35 +08:00
committed by GitHub
parent 593a58c161
commit 6e0f034ad1
12 changed files with 422 additions and 202 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/obs/README.md
+66 -65
View File
@@ -7,15 +7,15 @@ logging, distributed tracing, metrics via OpenTelemetry, and continuous profilin
## Features
| Feature | Description |
|---------|-------------|
| **Structured logging** | JSON-formatted logs via `tracing-subscriber` |
| **Rolling-file logging** | Daily / hourly rotation with automatic cleanup and high-precision timestamps |
| **Distributed tracing** | OTLP/HTTP export to Jaeger, Tempo, or any OTel collector |
| **Metrics** | OTLP/HTTP export, bridged from the `metrics` crate facade |
| **Continuous Profiling** | CPU/Memory profiling export to Pyroscope |
| **Log cleanup** | Background task: size limits, gzip compression, retention policies |
| **GPU metrics** *(optional)* | Enable with the `gpu` feature flag |
| Feature | Description |
|------------------------------|------------------------------------------------------------------------------|
| **Structured logging** | JSON-formatted logs via `tracing-subscriber` |
| **Rolling-file logging** | Daily / hourly rotation with automatic cleanup and high-precision timestamps |
| **Distributed tracing** | OTLP/HTTP export to Jaeger, Tempo, or any OTel collector |
| **Metrics** | OTLP/HTTP export, bridged from the `metrics` crate facade |
| **Continuous Profiling** | CPU/Memory profiling export to Pyroscope |
| **Log cleanup** | Background task: size limits, gzip compression, retention policies |
| **GPU metrics** *(optional)* | Enable with the `gpu` feature flag |
---
@@ -44,7 +44,7 @@ async fn main() {
}
```
> **Keep `_guard` alive** for the lifetime of your application. Dropping it
> **Keep `_guard` alive** for the lifetime of your application. Dropping it
> triggers an ordered shutdown of every OpenTelemetry provider.
---
@@ -57,8 +57,8 @@ async fn main() {
use rustfs_obs::init_obs;
let _guard = init_obs(Some("http://otel-collector:4318".to_string()))
.await
.expect("observability init failed");
.await
.expect("observability init failed");
```
### With a custom config struct
@@ -67,9 +67,9 @@ let _guard = init_obs(Some("http://otel-collector:4318".to_string()))
use rustfs_obs::{AppConfig, OtelConfig, init_obs_with_config};
let config = AppConfig::new_with_endpoint(Some("http://localhost:4318".to_string()));
let _guard = init_obs_with_config(&config.observability)
.await
.expect("observability init failed");
let _guard = init_obs_with_config( & config.observability)
.await
.expect("observability init failed");
```
---
@@ -92,10 +92,12 @@ The library selects a backend automatically based on configuration:
```
**Key Points:**
- When **no log directory** is configured, logs automatically go to **stdout only** (perfect for development)
- When a **log directory** is set, logs go to **rolling files** in that directory
- In **non-production environments**, stdout is automatically mirrored alongside file logging for visibility
- In **production** mode, you must explicitly set `RUSTFS_OBS_LOG_STDOUT_ENABLED=true` to see stdout in addition to files
- In **production** mode, you must explicitly set `RUSTFS_OBS_LOG_STDOUT_ENABLED=true` to see stdout in addition to
files
---
@@ -105,56 +107,55 @@ All configuration is read from environment variables at startup.
### OTLP / Export
| Variable | Default | Description |
|----------|---------|-------------|
| `RUSTFS_OBS_ENDPOINT` | _(empty)_ | Root OTLP/HTTP endpoint, e.g. `http://otel-collector:4318` |
| `RUSTFS_OBS_TRACE_ENDPOINT` | _(empty)_ | Dedicated trace endpoint (overrides root + `/v1/traces`) |
| `RUSTFS_OBS_METRIC_ENDPOINT` | _(empty)_ | Dedicated metrics endpoint |
| `RUSTFS_OBS_LOG_ENDPOINT` | _(empty)_ | Dedicated log endpoint |
| `RUSTFS_OBS_PROFILING_ENDPOINT` | _(empty)_ | Dedicated profiling endpoint (e.g. Pyroscope) |
| `RUSTFS_OBS_TRACES_EXPORT_ENABLED` | `true` | Toggle trace export |
| `RUSTFS_OBS_METRICS_EXPORT_ENABLED` | `true` | Toggle metrics export |
| `RUSTFS_OBS_LOGS_EXPORT_ENABLED` | `true` | Toggle OTLP log export |
| `RUSTFS_OBS_PROFILING_EXPORT_ENABLED` | `true` | Toggle profiling export |
| `RUSTFS_OBS_USE_STDOUT` | `false` | Mirror all signals to stdout alongside OTLP |
| `RUSTFS_OBS_SAMPLE_RATIO` | `0.1` | Trace sampling ratio `0.0`–`1.0` |
| `RUSTFS_OBS_METER_INTERVAL` | `15` | Metrics export interval (seconds) |
| Variable | Default | Description |
|---------------------------------------|-----------|------------------------------------------------------------|
| `RUSTFS_OBS_ENDPOINT` | _(empty)_ | Root OTLP/HTTP endpoint, e.g. `http://otel-collector:4318` |
| `RUSTFS_OBS_TRACE_ENDPOINT` | _(empty)_ | Dedicated trace endpoint (overrides root + `/v1/traces`) |
| `RUSTFS_OBS_METRIC_ENDPOINT` | _(empty)_ | Dedicated metrics endpoint |
| `RUSTFS_OBS_LOG_ENDPOINT` | _(empty)_ | Dedicated log endpoint |
| `RUSTFS_OBS_PROFILING_ENDPOINT` | _(empty)_ | Dedicated profiling endpoint (e.g. Pyroscope) |
| `RUSTFS_OBS_TRACES_EXPORT_ENABLED` | `true` | Toggle trace export |
| `RUSTFS_OBS_METRICS_EXPORT_ENABLED` | `true` | Toggle metrics export |
| `RUSTFS_OBS_LOGS_EXPORT_ENABLED` | `true` | Toggle OTLP log export |
| `RUSTFS_OBS_PROFILING_EXPORT_ENABLED` | `true` | Toggle profiling export |
| `RUSTFS_OBS_USE_STDOUT` | `false` | Mirror all signals to stdout alongside OTLP |
| `RUSTFS_OBS_SAMPLE_RATIO` | `0.1` | Trace sampling ratio `0.0`–`1.0` |
| `RUSTFS_OBS_METER_INTERVAL` | `15` | Metrics export interval (seconds) |
### Service identity
| Variable | Default | Description |
|----------|---------|-------------|
| `RUSTFS_OBS_SERVICE_NAME` | `rustfs` | OTel `service.name` |
| `RUSTFS_OBS_SERVICE_VERSION` | _(crate version)_ | OTel `service.version` |
| `RUSTFS_OBS_ENVIRONMENT` | `development` | Deployment environment (`production`, `development`, …) |
| Variable | Default | Description |
|------------------------------|-------------------|---------------------------------------------------------|
| `RUSTFS_OBS_SERVICE_NAME` | `rustfs` | OTel `service.name` |
| `RUSTFS_OBS_SERVICE_VERSION` | _(crate version)_ | OTel `service.version` |
| `RUSTFS_OBS_ENVIRONMENT` | `development` | Deployment environment (`production`, `development`, …) |
### Local logging
| Variable | Default | Description |
|----------|---------|-------------|
| `RUSTFS_OBS_LOGGER_LEVEL` | `info` | Log level; `RUST_LOG` syntax supported |
| `RUSTFS_OBS_LOG_STDOUT_ENABLED` | `false` | When file logging is active, also mirror to stdout |
| `RUSTFS_OBS_LOG_DIRECTORY` | _(empty)_ | **Directory for rolling log files. When empty, logs go to stdout only** |
| `RUSTFS_OBS_LOG_FILENAME` | `rustfs.log` | Base filename for rolling logs. Rotated archives include a high-precision timestamp and counter. With the default `RUSTFS_OBS_LOG_MATCH_MODE=suffix`, names look like `<timestamp>-<counter>.rustfs.log` (e.g., `20231027103001.123456-0.rustfs.log`); with `prefix`, they look like `rustfs.log.<timestamp>-<counter>` (e.g., `rustfs.log.20231027103001.123456-0`). |
| `RUSTFS_OBS_LOG_ROTATION_TIME` | `hourly` | Rotation granularity: `minutely`, `hourly`, or `daily` |
| `RUSTFS_OBS_LOG_KEEP_FILES` | `30` | Number of rolling files to keep (also used by cleaner) |
| `RUSTFS_OBS_LOG_MATCH_MODE` | `suffix` | File matching mode: `prefix` or `suffix` |
| Variable | Default | Description |
|---------------------------------|--------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `RUSTFS_OBS_LOGGER_LEVEL` | `info` | Log level; `RUST_LOG` syntax supported |
| `RUSTFS_OBS_LOG_STDOUT_ENABLED` | `false` | When file logging is active, also mirror to stdout |
| `RUSTFS_OBS_LOG_DIRECTORY` | _(empty)_ | **Directory for rolling log files. When empty, logs go to stdout only** |
| `RUSTFS_OBS_LOG_FILENAME` | `rustfs.log` | Base filename for rolling logs. Rotated archives include a high-precision timestamp and counter. With the default `RUSTFS_OBS_LOG_MATCH_MODE=suffix`, names look like `<timestamp>-<counter>.rustfs.log` (e.g., `20231027103001.123456-0.rustfs.log`); with `prefix`, they look like `rustfs.log.<timestamp>-<counter>` (e.g., `rustfs.log.20231027103001.123456-0`). |
| `RUSTFS_OBS_LOG_ROTATION_TIME` | `hourly` | Rotation granularity: `minutely`, `hourly`, or `daily` |
| `RUSTFS_OBS_LOG_KEEP_FILES` | `30` | Number of rolling files to keep (also used by cleaner) |
| `RUSTFS_OBS_LOG_MATCH_MODE` | `suffix` | File matching mode: `prefix` or `suffix` |
### Log cleanup
| Variable | Default | Description |
|----------|---------|-------------|
| `RUSTFS_OBS_LOG_MAX_TOTAL_SIZE_BYTES` | `2147483648` | Hard cap on total log directory size (2 GiB) |
| `RUSTFS_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES` | `0` | Per-file size cap; `0` = unlimited |
| `RUSTFS_OBS_LOG_COMPRESS_OLD_FILES` | `true` | Gzip-compress files before deleting |
| `RUSTFS_OBS_LOG_GZIP_COMPRESSION_LEVEL` | `6` | Gzip level `1` (fastest) – `9` (best) |
| `RUSTFS_OBS_LOG_COMPRESSED_FILE_RETENTION_DAYS` | `30` | Delete `.gz` archives older than N days; `0` = keep forever |
| `RUSTFS_OBS_LOG_EXCLUDE_PATTERNS` | _(empty)_ | Comma-separated glob patterns to never clean up |
| `RUSTFS_OBS_LOG_DELETE_EMPTY_FILES` | `true` | Remove zero-byte files |
| `RUSTFS_OBS_LOG_MIN_FILE_AGE_SECONDS` | `3600` | Minimum file age (seconds) before cleanup |
| `RUSTFS_OBS_LOG_CLEANUP_INTERVAL_SECONDS` | `1800` | How often the cleanup task runs (0.5 hours) |
| `RUSTFS_OBS_LOG_DRY_RUN` | `false` | Report deletions without actually removing files |
| Variable | Default | Description |
|-------------------------------------------------|--------------|-------------------------------------------------------------|
| `RUSTFS_OBS_LOG_MAX_TOTAL_SIZE_BYTES` | `2147483648` | Hard cap on total log directory size (2 GiB) |
| `RUSTFS_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES` | `0` | Per-file size cap; `0` = unlimited |
| `RUSTFS_OBS_LOG_COMPRESS_OLD_FILES` | `true` | Gzip-compress files before deleting |
| `RUSTFS_OBS_LOG_GZIP_COMPRESSION_LEVEL` | `6` | Gzip level `1` (fastest) – `9` (best) |
| `RUSTFS_OBS_LOG_COMPRESSED_FILE_RETENTION_DAYS` | `30` | Delete `.gz` archives older than N days; `0` = keep forever |
| `RUSTFS_OBS_LOG_EXCLUDE_PATTERNS` | _(empty)_ | Comma-separated glob patterns to never clean up |
| `RUSTFS_OBS_LOG_DELETE_EMPTY_FILES` | `true` | Remove zero-byte files |
| `RUSTFS_OBS_LOG_MIN_FILE_AGE_SECONDS` | `3600` | Minimum file age (seconds) before cleanup |
| `RUSTFS_OBS_LOG_CLEANUP_INTERVAL_SECONDS` | `1800` | How often the cleanup task runs (0.5 hours) |
| `RUSTFS_OBS_LOG_DRY_RUN` | `false` | Report deletions without actually removing files |
---
@@ -251,9 +252,9 @@ use rustfs_obs::LogCleaner;
use rustfs_obs::types::FileMatchMode;
let cleaner = LogCleaner::builder(
PathBuf::from("/var/log/rustfs"),
"rustfs.log.".to_string(), // file_pattern
"rustfs.log".to_string(), // active_filename
PathBuf::from("/var/log/rustfs"),
"rustfs.log.".to_string(), // file_pattern
"rustfs.log".to_string(), // active_filename
)
.match_mode(FileMatchMode::Prefix)
.keep_files(10)
@@ -261,7 +262,7 @@ let cleaner = LogCleaner::builder(
.max_single_file_size_bytes(0) // unlimited
.compress_old_files(true)
.gzip_compression_level(6)
.compressed_file_retention_days(30)
.compressed_file_retention_days(7)
.exclude_patterns(vec!["current.log".to_string()])
.delete_empty_files(true)
.min_file_age_seconds(3600) // 1 hour
@@ -276,11 +277,11 @@ println!("Deleted {deleted} files, freed {freed_bytes} bytes");
## Feature Flags
| Flag | Description |
|------|-------------|
| Flag | Description |
|-------------|------------------------------------|
| _(default)_ | Core logging, tracing, and metrics |
| `gpu` | GPU utilisation metrics via `nvml` |
| `full` | All features enabled |
| `gpu` | GPU utilisation metrics via `nvml` |
| `full` | All features enabled |
```toml
# Enable GPU monitoring