rustfs/crates/obs/src/cleaner

Log Cleaner Subsystem

The cleaner module is a production-focused background lifecycle manager for RustFS log archives. It periodically discovers rolled files, applies retention constraints, compresses candidates, and then deletes sources safely.

The subsystem is designed to be conservative by default:

• it never touches the currently active log file;
• it refuses symlink deletion during the destructive phase;
• it keeps compression and source deletion as separate steps;
• it supports a full dry-run mode for policy verification.

Execution Pipeline

1. Discovery (scanner.rs)

   • Performs a shallow read_dir scan (no recursion) for predictable latency.
   • Excludes the active log file, exclusion-pattern matches, and files younger than the age threshold.
   • Classifies regular logs and compressed archives (.gz / .zst) in one pass.

2. Selection (core.rs)

   • Enforces keep-count first.
   • Applies total-size and single-file-size constraints to oldest files.
   • Produces an ordered list of files to process.

3. Compression + Deletion (core.rs + compress.rs)

   • Supports zstd and gzip codecs.
   • Uses parallel work stealing when enabled (Injector + Worker::new_fifo + Stealer).
   • Always deletes source files in a serial pass after compression to minimize file-lock race issues.

4. Archive Expiry (core.rs)

   • Applies a separate retention window to already-compressed files.
   • Keeps archive expiration independent from plain-log keep-count logic.

Compression Modes

• Primary codec: zstd (default) for better ratio and faster decompression.
• Fallback codec: gzip when zstd fallback is enabled.
• Dry-run: reports planned compression/deletion operations without touching filesystem state.

Safety Model

• No recursive traversal: the scanner only inspects the immediate log directory.
• No symlink following: filesystem metadata is collected with symlink_metadata.
• Idempotent archives: an existing *.gz or *.zst target means the file is treated as already compressed.
• Best-effort cleanup: individual file failures are logged and do not abort the whole maintenance pass.

Work-Stealing Strategy

The parallel path in core.rs uses this fixed lookup sequence per worker:

1. local_worker.pop()
2. injector.steal_batch_and_pop(&local_worker)
3. randomized victim polling via Steal::from_iter(...)

This strategy keeps local cache affinity while still balancing stragglers.

Metrics and Tracing

The cleaner emits tracing events and runtime metrics:

• rustfs_log_cleaner_deleted_files_total (counter)
• rustfs_log_cleaner_freed_bytes_total (counter)
• rustfs_log_cleaner_compress_duration_seconds (histogram)
• rustfs_log_cleaner_steal_success_rate (gauge)
• rustfs_log_cleaner_rotation_total (counter)
• rustfs_log_cleaner_rotation_failures_total (counter)
• rustfs_log_cleaner_rotation_duration_seconds (histogram)
• rustfs_log_cleaner_active_file_size_bytes (gauge)

These values can be wired into dashboards and alert rules for cleanup health.

Retention Decision Order

For regular logs, the cleaner evaluates candidates in this order:

1. keep at most keep_files newest matching generations;
2. remove older files if total retained size still exceeds max_total_size_bytes;
3. remove any file whose individual size exceeds max_single_file_size_bytes;
4. if compression is enabled, archive before deletion;
5. delete the original file only after successful compression.

Key Environment Variables

Env Var	Meaning
RUSTFS_OBS_LOG_COMPRESSION_ALGORITHM	zstd or gzip
RUSTFS_OBS_LOG_PARALLEL_COMPRESS	Enable work-stealing compression
RUSTFS_OBS_LOG_PARALLEL_WORKERS	Worker count for parallel compressor
RUSTFS_OBS_LOG_ZSTD_COMPRESSION_LEVEL	Zstd level (1-21)
RUSTFS_OBS_LOG_ZSTD_FALLBACK_TO_GZIP	Fallback switch on zstd failure
RUSTFS_OBS_LOG_ZSTD_WORKERS	zstdmt worker threads per compression task
RUSTFS_OBS_LOG_DRY_RUN	Dry-run mode
RUSTFS_OBS_LOG_COMPRESSED_FILE_RETENTION_DAYS	Retention window for *.gz / *.zst archives
RUSTFS_OBS_LOG_DELETE_EMPTY_FILES	Remove zero-byte regular log files during scanning
RUSTFS_OBS_LOG_MIN_FILE_AGE_SECONDS	Minimum age for regular log eligibility

Builder Example

use rustfs_obs::LogCleaner;
use rustfs_obs::types::{CompressionAlgorithm, FileMatchMode};
use std::path::PathBuf;

let cleaner = LogCleaner::builder(
    PathBuf::from("/var/log/rustfs"),
    "rustfs.log".to_string(),
    "rustfs.log".to_string(),
)
.match_mode(FileMatchMode::Suffix)
.keep_files(30)
.max_total_size_bytes(2 * 1024 * 1024 * 1024)
.compress_old_files(true)
.compression_algorithm(CompressionAlgorithm::Zstd)
.parallel_compress(true)
.parallel_workers(6)
.zstd_compression_level(8)
.zstd_fallback_to_gzip(true)
.zstd_workers(1)
.dry_run(false)
.build();

let _ = cleaner.cleanup();

Operational Notes

• Prefer FileMatchMode::Suffix when rotations prepend timestamps to the filename.
• Prefer FileMatchMode::Prefix when rotations append counters or timestamps after a stable base name.
• Keep parallel_workers modest when zstd_workers is greater than 1, because each compression task may already use internal codec threads.