# rustfs-obs Observability library for [RustFS](https://github.com/rustfs/rustfs) providing structured JSON logging, distributed tracing, metrics via OpenTelemetry, and continuous profiling via Pyroscope. --- ## Features | Feature | Description | |------------------------------|------------------------------------------------------------------------------| | **Structured logging** | JSON-formatted logs via `tracing-subscriber` | | **Rolling-file logging** | Daily / hourly rotation with automatic cleanup and high-precision timestamps | | **Distributed tracing** | OTLP/HTTP export to Jaeger, Tempo, or any OTel collector | | **Metrics** | OTLP/HTTP export, bridged from the `metrics` crate facade | | **Continuous Profiling** | CPU/Memory profiling export to Pyroscope | | **Log cleanup** | Background task: size limits, zstd/gzip compression, retention policies | | **GPU metrics** *(optional)* | Enable with the `gpu` feature flag | --- ## Quick Start ```toml # Cargo.toml [dependencies] rustfs-obs = { version = "0.0.5" } # GPU metrics support rustfs-obs = { version = "0.0.5", features = ["gpu"] } ``` ```rust use rustfs_obs::init_obs; #[tokio::main] async fn main() { // Build config from environment variables, then initialise all backends. let _guard = init_obs(None).await.expect("failed to initialise observability"); tracing::info!("RustFS started"); // _guard is dropped here — all providers are flushed and shut down. } ``` > **Keep `_guard` alive** for the lifetime of your application. Dropping it > triggers an ordered shutdown of every OpenTelemetry provider. --- ## Initialisation ### With an explicit OTLP endpoint ```rust use rustfs_obs::init_obs; let _guard = init_obs(Some("http://otel-collector:4318".to_string())) .await .expect("observability init failed"); ``` ### With a custom config struct ```rust use rustfs_obs::{AppConfig, OtelConfig, init_obs_with_config}; let config = AppConfig::new_with_endpoint(Some("http://localhost:4318".to_string())); let _guard = init_obs_with_config( & config.observability) .await .expect("observability init failed"); ``` --- ## Routing Logic The library selects a backend automatically based on configuration: ``` 1. Any OTLP endpoint set? └─ YES → Full OTLP/HTTP pipeline (traces + metrics + logs) + Profiling (Pyroscope) only if: - RUSTFS_OBS_PROFILING_EXPORT_ENABLED=true (explicit opt-in, default: false) 2. RUSTFS_OBS_LOG_DIRECTORY set to a non-empty path? └─ YES → Rolling-file JSON logging + Stdout mirror enabled if: - RUSTFS_OBS_LOG_STDOUT_ENABLED=true (explicit), OR - RUSTFS_OBS_ENVIRONMENT != "production" (automatic) 3. Default → Stdout-only JSON logging (all signals) ``` **Key Points:** - When **no log directory** is configured, logs automatically go to **stdout only** (perfect for development) - When a **log directory** is set, logs go to **rolling files** in that directory - In **non-production environments**, stdout is automatically mirrored alongside file logging for visibility - In **production** mode, you must explicitly set `RUSTFS_OBS_LOG_STDOUT_ENABLED=true` to see stdout in addition to files --- ## Environment Variables All configuration is read from environment variables at startup. ### OTLP / Export | Variable | Default | Description | |---------------------------------------|-----------|------------------------------------------------------------| | `RUSTFS_OBS_ENDPOINT` | _(empty)_ | Root OTLP/HTTP endpoint, e.g. `http://otel-collector:4318` | | `RUSTFS_OBS_TRACE_ENDPOINT` | _(empty)_ | Dedicated trace endpoint (overrides root + `/v1/traces`) | | `RUSTFS_OBS_METRIC_ENDPOINT` | _(empty)_ | Dedicated metrics endpoint | | `RUSTFS_OBS_LOG_ENDPOINT` | _(empty)_ | Dedicated log endpoint | | `RUSTFS_OBS_PROFILING_ENDPOINT` | _(empty)_ | Dedicated profiling endpoint (e.g. Pyroscope) | | `RUSTFS_OBS_TRACES_EXPORT_ENABLED` | `true` | Toggle trace export | | `RUSTFS_OBS_METRICS_EXPORT_ENABLED` | `true` | Toggle metrics export | | `RUSTFS_OBS_LOGS_EXPORT_ENABLED` | `true` | Toggle OTLP log export | | `RUSTFS_OBS_PROFILING_EXPORT_ENABLED` | `false` | Toggle profiling export | | `RUSTFS_OBS_USE_STDOUT` | `false` | Mirror all signals to stdout alongside OTLP | | `RUSTFS_OBS_SAMPLE_RATIO` | `0.1` | Trace sampling ratio `0.0`–`1.0` | | `RUSTFS_OBS_METER_INTERVAL` | `15` | Metrics export interval (seconds) | ### Service identity | Variable | Default | Description | |------------------------------|-------------------|---------------------------------------------------------| | `RUSTFS_OBS_SERVICE_NAME` | `rustfs` | OTel `service.name` | | `RUSTFS_OBS_SERVICE_VERSION` | _(crate version)_ | OTel `service.version` | | `RUSTFS_OBS_ENVIRONMENT` | `development` | Deployment environment (`production`, `development`, …) | ### Local logging | Variable | Default | Description | |---------------------------------|--------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `RUSTFS_OBS_LOGGER_LEVEL` | `info` | Log level; `RUST_LOG` syntax supported | | `RUSTFS_OBS_LOG_STDOUT_ENABLED` | `false` | When file logging is active, also mirror to stdout | | `RUSTFS_OBS_LOG_DIRECTORY` | _(empty)_ | **Directory for rolling log files. When empty, logs go to stdout only** | | `RUSTFS_OBS_LOG_FILENAME` | `rustfs.log` | Base filename for rolling logs. Rotated archives include a high-precision timestamp and counter. With the default `RUSTFS_OBS_LOG_MATCH_MODE=suffix`, names look like `-.rustfs.log` (e.g., `20231027103001.123456-0.rustfs.log`); with `prefix`, they look like `rustfs.log.-` (e.g., `rustfs.log.20231027103001.123456-0`). | | `RUSTFS_OBS_LOG_ROTATION_TIME` | `hourly` | Rotation granularity: `minutely`, `hourly`, or `daily` | | `RUSTFS_OBS_LOG_KEEP_FILES` | `30` | Number of rolling files to keep (also used by cleaner) | | `RUSTFS_OBS_LOG_MATCH_MODE` | `suffix` | File matching mode: `prefix` or `suffix` | ### Log cleanup | Variable | Default | Description | |-------------------------------------------------|--------------|-------------------------------------------------------------| | `RUSTFS_OBS_LOG_MAX_TOTAL_SIZE_BYTES` | `2147483648` | Hard cap on total log directory size (2 GiB) | | `RUSTFS_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES` | `0` | Per-file size cap; `0` = unlimited | | `RUSTFS_OBS_LOG_COMPRESS_OLD_FILES` | `true` | Compress files before deleting | | `RUSTFS_OBS_LOG_GZIP_COMPRESSION_LEVEL` | `6` | Gzip level `1` (fastest) – `9` (best) | | `RUSTFS_OBS_LOG_COMPRESSION_ALGORITHM` | `zstd` | Compression codec: `zstd` or `gzip` | | `RUSTFS_OBS_LOG_PARALLEL_COMPRESS` | `true` | Enable work-stealing parallel compression | | `RUSTFS_OBS_LOG_PARALLEL_WORKERS` | `6` | Number of cleaner worker threads | | `RUSTFS_OBS_LOG_ZSTD_COMPRESSION_LEVEL` | `8` | Zstd level `1` (fastest) – `21` (best ratio) | | `RUSTFS_OBS_LOG_ZSTD_FALLBACK_TO_GZIP` | `true` | Fallback to gzip when zstd compression fails | | `RUSTFS_OBS_LOG_ZSTD_WORKERS` | `1` | zstdmt worker threads per compression task | | `RUSTFS_OBS_LOG_COMPRESSED_FILE_RETENTION_DAYS` | `30` | Delete `.gz` / `.zst` archives older than N days; `0` = keep forever | | `RUSTFS_OBS_LOG_EXCLUDE_PATTERNS` | _(empty)_ | Comma-separated glob patterns to never clean up | | `RUSTFS_OBS_LOG_DELETE_EMPTY_FILES` | `true` | Remove zero-byte files | | `RUSTFS_OBS_LOG_MIN_FILE_AGE_SECONDS` | `3600` | Minimum file age (seconds) before cleanup | | `RUSTFS_OBS_LOG_CLEANUP_INTERVAL_SECONDS` | `1800` | How often the cleanup task runs (0.5 hours) | | `RUSTFS_OBS_LOG_DRY_RUN` | `false` | Report deletions without actually removing files | --- ## Cleaner & Rotation Metrics The log rotation and cleanup pipeline emits these metrics (via the `metrics` facade): | Metric | Type | Description | |---|---|---| | `rustfs_log_cleaner_deleted_files_total` | counter | Number of files deleted per cleanup pass | | `rustfs_log_cleaner_freed_bytes_total` | counter | Bytes reclaimed by deletion | | `rustfs_log_cleaner_compress_duration_seconds` | histogram | Compression stage duration | | `rustfs_log_cleaner_steal_success_rate` | gauge | Work-stealing success ratio in parallel mode | | `rustfs_log_cleaner_runs_total` | counter | Successful cleanup loop runs | | `rustfs_log_cleaner_run_failures_total` | counter | Failed or panicked cleanup loop runs | | `rustfs_log_cleaner_rotation_total` | counter | Successful file rotations | | `rustfs_log_cleaner_rotation_failures_total` | counter | Failed file rotations | | `rustfs_log_cleaner_rotation_duration_seconds` | histogram | Rotation latency | | `rustfs_log_cleaner_active_file_size_bytes` | gauge | Current active log file size | These metrics cover compression, cleanup, and file rotation end-to-end. ### Metric Semantics - `deleted_files_total` and `freed_bytes_total` are emitted after each cleanup pass and include both normal log cleanup and expired compressed archive cleanup. - `compress_duration_seconds` measures compression stage wall-clock time for both serial and parallel modes. - `steal_success_rate` is updated by the parallel work-stealing path and remains at the last computed value. - `rotation_*` metrics are emitted by `RollingAppender` and include retries; a failed final rotation increments `rotation_failures_total`. - `active_file_size_bytes` is sampled on writes and after successful roll, so dashboards can track current active file growth. ### Grafana Dashboard JSON Draft (Ready to Import) > Save this as `rustfs-log-cleaner-dashboard.json`, then import from Grafana UI. > The canonical metric names use underscore notation, for example > `rustfs_log_cleaner_deleted_files_total`. > > The same panels are now checked in at: > `.docker/observability/grafana/dashboards/rustfs.json` > (row title: `Log Cleaner`). ```json { "uid": "rustfs-log-cleaner", "title": "RustFS Log Cleaner", "timezone": "browser", "schemaVersion": 39, "version": 1, "refresh": "10s", "tags": ["rustfs", "observability", "log-cleaner"], "time": { "from": "now-6h", "to": "now" }, "panels": [ { "id": 1, "title": "Cleanup Runs / Failures", "type": "timeseries", "targets": [ { "refId": "A", "expr": "sum(rate(rustfs_log_cleaner_runs_total[5m]))", "legendFormat": "runs/s" }, { "refId": "B", "expr": "sum(rate(rustfs_log_cleaner_run_failures_total[5m]))", "legendFormat": "failures/s" } ], "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 } }, { "id": 2, "title": "Freed Bytes / Deleted Files", "type": "timeseries", "targets": [ { "refId": "A", "expr": "sum(rate(rustfs_log_cleaner_freed_bytes_total[15m]))", "legendFormat": "bytes/s" }, { "refId": "B", "expr": "sum(rate(rustfs_log_cleaner_deleted_files_total[15m]))", "legendFormat": "files/s" } ], "gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 } }, { "id": 3, "title": "Compression P95 Latency", "type": "timeseries", "targets": [ { "refId": "A", "expr": "histogram_quantile(0.95, sum(rate(rustfs_log_cleaner_compress_duration_seconds_bucket[5m])) by (le))", "legendFormat": "p95" } ], "gridPos": { "h": 8, "w": 12, "x": 0, "y": 8 } }, { "id": 4, "title": "Rotation Success / Failure", "type": "timeseries", "targets": [ { "refId": "A", "expr": "sum(rate(rustfs_log_cleaner_rotation_total[5m]))", "legendFormat": "rotation/s" }, { "refId": "B", "expr": "sum(rate(rustfs_log_cleaner_rotation_failures_total[5m]))", "legendFormat": "rotation_failures/s" } ], "gridPos": { "h": 8, "w": 12, "x": 12, "y": 8 } }, { "id": 5, "title": "Steal Success Rate", "type": "timeseries", "targets": [ { "refId": "A", "expr": "max(rustfs_log_cleaner_steal_success_rate)", "legendFormat": "ratio" } ], "gridPos": { "h": 8, "w": 12, "x": 0, "y": 16 } }, { "id": 6, "title": "Active File Size", "type": "timeseries", "targets": [ { "refId": "A", "expr": "max(rustfs_log_cleaner_active_file_size_bytes)", "legendFormat": "bytes" } ], "gridPos": { "h": 8, "w": 12, "x": 12, "y": 16 } } ] } ``` ### PromQL Templates Use these templates directly in Grafana panels/alerts. - **Cleanup run rate** - `sum(rate(rustfs_log_cleaner_runs_total[$__rate_interval]))` - **Cleanup failure rate** - `sum(rate(rustfs_log_cleaner_run_failures_total[$__rate_interval]))` - **Cleanup failure ratio** - `sum(rate(rustfs_log_cleaner_run_failures_total[$__rate_interval])) / clamp_min(sum(rate(rustfs_log_cleaner_runs_total[$__rate_interval])), 1e-9)` - **Freed bytes throughput** - `sum(rate(rustfs_log_cleaner_freed_bytes_total[$__rate_interval]))` - **Deleted files throughput** - `sum(rate(rustfs_log_cleaner_deleted_files_total[$__rate_interval]))` - **Compression p95 latency** - `histogram_quantile(0.95, sum(rate(rustfs_log_cleaner_compress_duration_seconds_bucket[$__rate_interval])) by (le))` - **Rotation failure ratio** - `sum(rate(rustfs_log_cleaner_rotation_failures_total[$__rate_interval])) / clamp_min(sum(rate(rustfs_log_cleaner_rotation_total[$__rate_interval])), 1e-9)` - **Work-stealing efficiency (latest)** - `max(rustfs_log_cleaner_steal_success_rate)` - **Active file size (latest)** - `max(rustfs_log_cleaner_active_file_size_bytes)` ### Suggested Alerts - **CleanupFailureRatioHigh**: failure ratio > 0.05 for 10m. - **CompressionLatencyP95High**: p95 above your baseline SLO for 15m. - **RotationFailuresDetected**: rotation failure rate > 0 for 3 consecutive windows. - **NoCleanupActivity**: runs rate == 0 for expected active environments. ### Metrics Compatibility The project is currently in active development. Metric names and labels are updated directly when architecture evolves, and no backward-compatibility shim is maintained for old names. Use the metric names documented in this README as the current source of truth. --- ## Examples ### Stdout-only (development default) ```bash # No RUSTFS_OBS_LOG_DIRECTORY set → stdout JSON RUSTFS_OBS_LOGGER_LEVEL=debug ./rustfs ``` ### Rolling-file logging ```bash export RUSTFS_OBS_LOG_DIRECTORY=/var/log/rustfs export RUSTFS_OBS_LOGGER_LEVEL=info export RUSTFS_OBS_LOG_KEEP_FILES=30 export RUSTFS_OBS_LOG_MAX_TOTAL_SIZE_BYTES=5368709120 # 5 GiB ./rustfs ``` ### Full OTLP pipeline (production) ```bash export RUSTFS_OBS_ENDPOINT=http://otel-collector:4318 export RUSTFS_OBS_ENVIRONMENT=production export RUSTFS_OBS_SAMPLE_RATIO=0.05 # 5% trace sampling export RUSTFS_OBS_LOG_DIRECTORY=/var/log/rustfs export RUSTFS_OBS_LOG_STDOUT_ENABLED=false ./rustfs ``` ### Separate per-signal endpoints ```bash export RUSTFS_OBS_TRACE_ENDPOINT=http://tempo:4318/v1/traces export RUSTFS_OBS_METRIC_ENDPOINT=http://prometheus-otel:4318/v1/metrics export RUSTFS_OBS_LOG_ENDPOINT=http://loki-otel:4318/v1/logs ./rustfs ``` ### Dry-run cleanup audit ```bash export RUSTFS_OBS_LOG_DIRECTORY=/var/log/rustfs export RUSTFS_OBS_LOG_DRY_RUN=true ./rustfs # Observe log output — no files will actually be deleted. ``` ### Parallel zstd cleanup (recommended production profile) ```bash export RUSTFS_OBS_LOG_DIRECTORY=/var/log/rustfs export RUSTFS_OBS_LOG_COMPRESSION_ALGORITHM=zstd export RUSTFS_OBS_LOG_PARALLEL_COMPRESS=true export RUSTFS_OBS_LOG_PARALLEL_WORKERS=6 export RUSTFS_OBS_LOG_ZSTD_COMPRESSION_LEVEL=8 export RUSTFS_OBS_LOG_ZSTD_FALLBACK_TO_GZIP=true export RUSTFS_OBS_LOG_ZSTD_WORKERS=1 ./rustfs ``` --- ## Module Structure ``` rustfs-obs/src/ ├── lib.rs # Crate root; public re-exports ├── config.rs # OtelConfig + AppConfig; env-var loading ├── error.rs # TelemetryError type ├── global.rs # init_obs / init_obs_with_config entry points │ ├── telemetry/ # Backend initialisation │ ├── mod.rs # init_telemetry routing logic │ ├── guard.rs # OtelGuard RAII lifecycle manager │ ├── filter.rs # EnvFilter construction helpers │ ├── resource.rs # OTel Resource builder │ ├── local.rs # Stdout-only and rolling-file backends │ ├── otel.rs # Full OTLP/HTTP pipeline │ └── recorder.rs # metrics-crate → OTel bridge (Recorder) │ ├── cleaner/ # Background log-file cleanup subsystem │ ├── mod.rs # LogCleaner public API + tests │ ├── types.rs # Shared cleaner types (match mode, compression codec, FileInfo) │ ├── scanner.rs # Filesystem discovery │ ├── compress.rs # Gzip/Zstd compression helper │ └── core.rs # Selection, compression, deletion logic │ └── system/ # Host metrics (CPU, memory, disk, GPU) ├── mod.rs ├── attributes.rs ├── collector.rs ├── metrics.rs └── gpu.rs # GPU metrics (feature = "gpu") ``` --- ## Using `LogCleaner` Directly ```rust use std::path::PathBuf; use rustfs_obs::LogCleaner; use rustfs_obs::types::FileMatchMode; let cleaner = LogCleaner::builder( PathBuf::from("/var/log/rustfs"), "rustfs.log.".to_string(), // file_pattern "rustfs.log".to_string(), // active_filename ) .match_mode(FileMatchMode::Prefix) .keep_files(10) .max_total_size_bytes(2 * 1024 * 1024 * 1024) // 2 GiB .max_single_file_size_bytes(0) // unlimited .compress_old_files(true) .gzip_compression_level(6) .compressed_file_retention_days(7) .exclude_patterns(vec!["current.log".to_string()]) .delete_empty_files(true) .min_file_age_seconds(3600) // 1 hour .dry_run(false) .build(); let (deleted, freed_bytes) = cleaner.cleanup().expect("cleanup failed"); println!("Deleted {deleted} files, freed {freed_bytes} bytes"); ``` --- ## Feature Flags | Flag | Description | |-------------|------------------------------------| | _(default)_ | Core logging, tracing, and metrics | | `gpu` | GPU utilisation metrics via `nvml` | | `full` | All features enabled | ```toml # Enable GPU monitoring rustfs-obs = { version = "0.0.5", features = ["gpu"] } # Enable everything rustfs-obs = { version = "0.0.5", features = ["full"] } ``` --- ## License Apache 2.0 — see [LICENSE](../../LICENSE).