gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-08-31 09:18:28 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(obs): add init_obs_with_config API and signature guard test (#2175)

Browse Source 
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
This commit is contained in:
houseme
2026-03-16 18:17:55 +08:00
committed by GitHub
parent 06dff96c09
commit 94cdb89e29
18 changed files with 2597 additions and 325 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/obs/src/cleaner/README.md
+102 -43
View File
@@ -1,71 +1,130 @@
# Log Cleaner Subsystem
The `cleaner` module provides a robust, background log-file lifecycle manager for RustFS. It is designed to run periodically to enforce retention policies, compress old logs, and prevent disk exhaustion.
The `cleaner` module is a production-focused background lifecycle manager for RustFS log archives.
It periodically discovers rolled files, applies retention constraints, compresses candidates, and then deletes sources safely.
## Architecture
The subsystem is designed to be conservative by default:
The cleaner operates as a pipeline:
- it never touches the currently active log file;
- it refuses symlink deletion during the destructive phase;
- it keeps compression and source deletion as separate steps;
- it supports a full dry-run mode for policy verification.
1. **Discovery (`scanner.rs`)**: Scans the configured log directory for eligible files.
* **Non-recursive**: Only scans the top-level directory for safety.
* **Filtering**: Ignores the currently active log file, files matching exclude patterns, and files that do not match the configured prefix/suffix pattern.
* **Performance**: Uses `std::fs::read_dir` directly to minimize overhead and syscalls.
## Execution Pipeline
2. **Selection (`core.rs`)**: Applies retention policies to select files for deletion.
* **Keep Count**: Ensures at least `N` recent files are kept.
* **Total Size**: Deletes oldest files if the total size exceeds the limit.
* **Single File Size**: Deletes individual files that exceed a size limit (e.g., runaway logs).
1. **Discovery (`scanner.rs`)**
- Performs a shallow `read_dir` scan (no recursion) for predictable latency.
- Excludes the active log file, exclusion-pattern matches, and files younger than the age threshold.
- Classifies regular logs and compressed archives (`.gz` / `.zst`) in one pass.
3. **Action (`core.rs` / `compress.rs`)**:
* **Compression**: Optionally compresses selected files using Gzip (level 1-9) before deletion.
* **Deletion**: Removes the original file (and eventually the compressed archive based on retention days).
2. **Selection (`core.rs`)**
- Enforces keep-count first.
- Applies total-size and single-file-size constraints to oldest files.
- Produces an ordered list of files to process.
## Configuration
3. **Compression + Deletion (`core.rs` + `compress.rs`)**
- Supports `zstd` and `gzip` codecs.
- Uses parallel work stealing when enabled (`Injector + Worker::new_fifo + Stealer`).
- Always deletes source files in a serial pass after compression to minimize file-lock race issues.
The cleaner is configured via `LogCleanerBuilder`. When initialized via `rustfs-obs::init_obs`, it reads from environment variables.
4. **Archive Expiry (`core.rs`)**
- Applies a separate retention window to already-compressed files.
- Keeps archive expiration independent from plain-log keep-count logic.
| Parameter | Env Var | Description |
|-----------|---------|-------------|
| `log_dir` | `RUSTFS_OBS_LOG_DIRECTORY` | The directory to scan. |
| `file_pattern` | `RUSTFS_OBS_LOG_FILENAME` | The base filename pattern (e.g., `rustfs.log`). |
| `active_filename` | (Derived) | The exact name of the currently active log file, excluded from cleanup. |
| `match_mode` | `RUSTFS_OBS_LOG_MATCH_MODE` | `prefix` or `suffix`. Determines how `file_pattern` is matched against filenames. |
| `keep_files` | `RUSTFS_OBS_LOG_KEEP_FILES` | Minimum number of rolling log files to keep. |
| `max_total_size_bytes` | `RUSTFS_OBS_LOG_MAX_TOTAL_SIZE_BYTES` | Maximum aggregate size of all log files. Oldest files are deleted to satisfy this. |
| `compress_old_files` | `RUSTFS_OBS_LOG_COMPRESS_OLD_FILES` | If `true`, files selected for removal are first gzipped. |
| `compressed_file_retention_days` | `RUSTFS_OBS_LOG_COMPRESSED_FILE_RETENTION_DAYS` | Age in days after which `.gz` files are deleted. |
## Compression Modes
## Timestamp Format & Rotation
- **Primary codec**: `zstd` (default) for better ratio and faster decompression.
- **Fallback codec**: `gzip` when zstd fallback is enabled.
- **Dry-run**: reports planned compression/deletion operations without touching filesystem state.
The cleaner works in tandem with the `RollingAppender` in `telemetry/rolling.rs`.
## Safety Model
* **Rotation**: Logs are rotated based on time (Daily/Hourly/Minutely) or Size.
* **Naming**: Archived logs use a high-precision timestamp format: `YYYYMMDDHHMMSS.uuuuuu` (microseconds), plus a unique counter to prevent collisions.
* **Suffix Mode**: `<timestamp>-<counter>.<filename>` (e.g., `20231027103001.123456-0.rustfs.log`)
* **Prefix Mode**: `<filename>.<timestamp>-<counter>` (e.g., `rustfs.log.20231027103001.123456-0`)
- **No recursive traversal**: the scanner only inspects the immediate log directory.
- **No symlink following**: filesystem metadata is collected with `symlink_metadata`.
- **Idempotent archives**: an existing `*.gz` or `*.zst` target means the file is treated as already compressed.
- **Best-effort cleanup**: individual file failures are logged and do not abort the whole maintenance pass.
This high-precision naming ensures that files sort chronologically by name, and collisions are virtually impossible even under high load.
## Work-Stealing Strategy
## Usage Example
The parallel path in `core.rs` uses this fixed lookup sequence per worker:
1. `local_worker.pop()`
2. `injector.steal_batch_and_pop(&local_worker)`
3. randomized victim polling via `Steal::from_iter(...)`
This strategy keeps local cache affinity while still balancing stragglers.
## Metrics and Tracing
The cleaner emits tracing events and runtime metrics:
- `rustfs.log_cleaner.deleted_files_total` (counter)
- `rustfs.log_cleaner.freed_bytes_total` (counter)
- `rustfs.log_cleaner.compress_duration_seconds` (histogram)
- `rustfs.log_cleaner.steal_success_rate` (gauge)
- `rustfs.log_cleaner.rotation_total` (counter)
- `rustfs.log_cleaner.rotation_failures_total` (counter)
- `rustfs.log_cleaner.rotation_duration_seconds` (histogram)
- `rustfs.log_cleaner.active_file_size_bytes` (gauge)
These values can be wired into dashboards and alert rules for cleanup health.
## Retention Decision Order
For regular logs, the cleaner evaluates candidates in this order:
1. keep at least `keep_files` newest matching generations;
2. remove older files if total retained size still exceeds `max_total_size_bytes`;
3. remove any file whose individual size exceeds `max_single_file_size_bytes`;
4. if compression is enabled, archive before deletion;
5. delete the original file only after successful compression.
## Key Environment Variables
| Env Var | Meaning |
|---|---|
| `RUSTFS_OBS_LOG_COMPRESSION_ALGORITHM` | `zstd` or `gzip` |
| `RUSTFS_OBS_LOG_PARALLEL_COMPRESS` | Enable work-stealing compression |
| `RUSTFS_OBS_LOG_PARALLEL_WORKERS` | Worker count for parallel compressor |
| `RUSTFS_OBS_LOG_ZSTD_COMPRESSION_LEVEL` | Zstd level (1-21) |
| `RUSTFS_OBS_LOG_ZSTD_FALLBACK_TO_GZIP` | Fallback switch on zstd failure |
| `RUSTFS_OBS_LOG_ZSTD_WORKERS` | zstdmt worker threads per compression task |
| `RUSTFS_OBS_LOG_DRY_RUN` | Dry-run mode |
| `RUSTFS_OBS_LOG_COMPRESSED_FILE_RETENTION_DAYS` | Retention window for `*.gz` / `*.zst` archives |
| `RUSTFS_OBS_LOG_DELETE_EMPTY_FILES` | Remove zero-byte regular log files during scanning |
| `RUSTFS_OBS_LOG_MIN_FILE_AGE_SECONDS` | Minimum age for regular log eligibility |
## Builder Example
```rust
use rustfs_obs::LogCleaner;
use rustfs_obs::types::FileMatchMode;
use rustfs_obs::types::{CompressionAlgorithm, FileMatchMode};
use std::path::PathBuf;
let cleaner = LogCleaner::builder(
PathBuf::from("/var/log/rustfs"),
"rustfs.log.".to_string(),
"rustfs.log".to_string(),
"rustfs.log".to_string(),
)
.match_mode(FileMatchMode::Prefix)
.keep_files(10)
.max_total_size_bytes(1024 * 1024 * 100) // 100 MB
.match_mode(FileMatchMode::Suffix)
.keep_files(30)
.max_total_size_bytes(2 * 1024 * 1024 * 1024)
.compress_old_files(true)
.compression_algorithm(CompressionAlgorithm::Zstd)
.parallel_compress(true)
.parallel_workers(6)
.zstd_compression_level(8)
.zstd_fallback_to_gzip(true)
.zstd_workers(1)
.dry_run(false)
.build();
// Run cleanup (blocking operation, spawn in a background task)
if let Ok((deleted, freed)) = cleaner.cleanup() {
println!("Cleaned up {} files, freed {} bytes", deleted, freed);
}
let _ = cleaner.cleanup();
```
## Operational Notes
- Prefer `FileMatchMode::Suffix` when rotations prepend timestamps to the filename.
- Prefer `FileMatchMode::Prefix` when rotations append counters or timestamps after a stable base name.
- Keep `parallel_workers` modest when `zstd_workers` is greater than `1`, because each compression task may already use internal codec threads.