mirror of
https://github.com/rustfs/rustfs.git
synced 2026-07-26 08:18:18 +00:00
3b139e5267
* fix(obs/cleaner): fsync archive and log dir before deleting source logs OLC-01: the compression path flushed the BufWriter but never synced the archive data or the parent directory before renaming, and the source was then unlinked with no durability barrier. A crash after rename but before the page cache reached disk could leave a truncated/zero-length archive while the source was already gone — permanent log/audit data loss. Because the archive is always renamed to a brand-new name (guaranteed by the existing exists() guard), ext4 auto_da_alloc does not mask this. Hand the underlying File back from the writer closure, sync_all() it before rename, fsync the parent directory, and fsync the log directory after the unlinks so a delete cannot be reordered ahead of the archive it justified. Guard the temp file with an RAII cleanup so an early return or panic cannot leak a *.tmp orphan. Ref: rustfs/backlog#1194 (audit rustfs/backlog#1193) Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs/cleaner): create compression temp file with O_EXCL and O_NOFOLLOW OLC-03: the temp archive was opened with File::create at a predictable `<source>.gz.tmp` path with no O_EXCL/O_NOFOLLOW, so an actor with write access to the log directory could pre-plant that path as a symlink and have the compressor follow it — truncating and overwriting an arbitrary external file, then chmod-ing it to the source log's mode. This mirrors the symlink refusal already enforced on the deletion path (secure_delete). Route temp creation through create_tmp_archive(), which uses create_new (O_CREAT|O_EXCL) to refuse a pre-existing entry and, on Unix, O_NOFOLLOW to refuse a symlink at the final path component. Ref: rustfs/backlog#1196 (audit rustfs/backlog#1193) Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs/cleaner): create compression temp file with restrictive mode OLC-06: File::create left the temp archive world-readable (0644 & ~umask) for the entire duration of compressing a large log, exposing the full plaintext of a possibly-0600 audit log on shared hosts until the mode was copied only after the write completed. Pass the source mode into create_tmp_archive and open the temp file with it (default 0600) so it is restrictive from creation; the post-write chmod still tightens/matches the source mode exactly. Ref: rustfs/backlog#1199 (audit rustfs/backlog#1193) Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs/cleaner): validate existing archive before skipping recompression OLC-02: the idempotency guard used Path::exists() (which follows symlinks) and trusted whatever it found, then the caller deleted the source. A planted `<archive>.gz` symlink, or a zero-length/truncated archive left by a crashed run (OLC-01), would green-light deleting the source with no valid backup — data loss / log destruction. Replace exists() with symlink_metadata (no follow) and only treat the entry as a completed prior result when it is a regular, non-empty file whose header matches the codec magic (gzip 1f 8b / zstd 28 b5 2f fd). Anything else falls through to recompression, whose atomic create_new+rename replaces the bad entry (a symlink is replaced, never followed or deleted through). Ref: rustfs/backlog#1195 (audit rustfs/backlog#1193) Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs/cleaner): stop dry-run overstating reclaimed bytes for compression OLC-08: in dry-run, compress_with_writer returned output_bytes = 0, so projected_freed_bytes = input and delete_files reported the full input as freed. A real run keeps the archive on disk (freed = input - archive), so dry-run overstated reclaim by the whole archive footprint. Estimate the archive with a deliberately conservative ratio so the projection never exceeds what a real run reclaims. Ref: rustfs/backlog#1201 (audit rustfs/backlog#1193) Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs/cleaner): make freed-byte accounting resilient; document steal metric OLC-12: input/output byte sizes were read via metadata().unwrap_or(0), which silently reports 0 on failure and skews freed-byte metrics (input - 0 = full input, overstating reclaim). Use the copy() byte count as the authoritative input size and, when the archive metadata read fails, conservatively assume no savings instead of 0. Also document that the steal_success_rate counts only victim steals (batch = one success), so it reads as a relative rebalancing signal, not absolute task acquisition. Ref: rustfs/backlog#1205 (audit rustfs/backlog#1193) Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs/cleaner): preserve level-0 semantics, allow zstd 22, log effective levels OLC-09: build() and the codec calls clamped gzip/zstd levels to [1,9]/[1,21], silently rewriting gzip level 0 (store) and zstd level 0 (codec default) to 1 and blocking the legal zstd maximum of 22. Clamp to [0,9]/[0,22] so those meanings survive, and echo the effective (post-clamp) levels in the startup log via new effective_gzip_level()/effective_zstd_level() getters so the log matches what actually runs. Ref: rustfs/backlog#1202 (audit rustfs/backlog#1193) Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs/cleaner): bound compressed archives by byte cap; warn on retention=0 OLC-04: archive expiry was gated on compressed_file_retention_days > 0, so retention=0 disabled it entirely while compression kept producing archives, and max_total_size_bytes only ever bounded uncompressed logs — unbounded disk growth. Replace select_expired_compressed with select_archives_to_delete, which applies age expiry (when retention is on) and, regardless of retention, trims the oldest archives until the set fits under max_total_size_bytes. Also warn at startup when compression is on with retention=0 so the "keep forever" semantics are not mistaken for "delete immediately". Ref: rustfs/backlog#1197 (audit rustfs/backlog#1193) Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs/cleaner): warn on invalid exclude glob instead of dropping silently OLC-05: build() dropped unparseable exclude globs via filter_map(...ok()), so a typo (or a literal comma splitting a char-class in the config string) turned "protect this file" into "delete this file" with no signal. Log a warning per rejected pattern with the raw string and parse error so the misconfiguration is visible. Ref: rustfs/backlog#1198 (audit rustfs/backlog#1193) Co-Authored-By: heihutu <heihutu@gmail.com> * perf(obs/cleaner): backoff idle workers, cap worker count, lower small-host floor OLC-11: the work-stealing loop re-spun on Steal::Retry with no yield and used yield_now on the empty path, burning CPU during redistribution windows; worker_count had no upper bound so a mis-set parallel_workers over a directory of thousands of logs could spawn thousands of threads; and default_parallel_workers forced >=4 workers even on 1-2 vCPU hosts. Use crossbeam_utils::Backoff (spin->yield, reset on work) on the idle paths, clamp worker_count to MAX_PARALLEL_COMPRESS_WORKERS, and lower the default floor to 1. Ref: rustfs/backlog#1204 (audit rustfs/backlog#1193) Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs/cleaner): warn when active-file guard is disabled by empty filename OLC-13 (defense-in-depth): the scanner protects the live log purely by exact filename equality against active_filename. An empty active_filename silently disables that protection, so a non-empty file_pattern could make the live log a deletion candidate via the public builder. Warn in build() when that unsafe combination is configured. The audit's "never delete the newest match" structural guard is intentionally not implemented: it would conflict with the legitimate keep_files=0 semantics (purge all rotated logs). The naming contract is instead locked by regression tests (OLC-14). Ref: rustfs/backlog#1206 (audit rustfs/backlog#1193) Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs/cleaner): warn on unknown algorithm/match_mode, echo match_mode OLC-07: from_config_str silently fell back to defaults for unrecognized compression algorithm and match mode (any non-"prefix" value became Suffix), hiding operator typos like "prefixx" that could make the cleaner match no rotated logs. Warn on a non-empty unrecognized value in both parsers, and echo the resolved match_mode in the startup log alongside the algorithm. Ref: rustfs/backlog#1200 (audit rustfs/backlog#1193) Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs/cleaner): derive orphan .tmp suffixes and exempt them from min age OLC-10: orphan `*.gz.tmp`/`*.zst.tmp` cleanup was gated by min_file_age_seconds (default 3600), so crash-left orphans lingered up to an hour, and the tmp suffix list was hardcoded rather than derived from compressed_suffixes() — a new codec would leave `*.<ext>.tmp` orphans the scanner never recognizes. Derive the temp suffix from CompressionAlgorithm::compressed_suffixes(), and gate orphan removal on a small fixed grace window instead of min_file_age (orphans are never live-written after the rename that would promote them). Ref: rustfs/backlog#1203 (audit rustfs/backlog#1193) Co-Authored-By: heihutu <heihutu@gmail.com> * test(obs/cleaner): cover symlink, archive expiry, idempotency, and edge cases OLC-14: add regression tests for the previously-untested safety/correctness branches — symlink rejection (external target never deleted), archive age expiry vs fresh retention, archive byte-cap trim with retention disabled, gz/zst classification, max_single_file_size selection, min_age protecting a fresh non-empty log, active-file exclusion when the active name also matches the pattern, invalid exclude glob not aborting build, dry-run + compression creating no archive, gzip round-trip validity, and the idempotent-archive branch trusting a valid prior archive. Ref: rustfs/backlog#1207 (audit rustfs/backlog#1193) Co-Authored-By: heihutu <heihutu@gmail.com> * style(obs/cleaner): apply rustfmt and collapse nested if (clippy) Formatting-only cleanup over the audit fix series: rustfmt normalization of the multi-line expressions introduced in compress.rs/core.rs, plus collapsing the delete_files directory-fsync into a single let-chain to satisfy clippy::collapsible_if. No behavior change. Co-Authored-By: heihutu <heihutu@gmail.com> * refactor(obs/cleaner): collapse redundant source stats in compression path The per-issue fixes to compress_with_writer accumulated three metadata() syscalls on the source in the real compression path: an input_bytes read that was immediately shadowed by the copied byte count (a dead read), a source_mode read (OLC-06), and the pre-OLC-06 post-write chmod re-reading the same mode. Collapse to a single fd-based read — move the dry-run input_bytes read into the dry-run branch, read source_mode from the already-open fd (no path stat, no TOCTOU), and reuse it for the post-write chmod. Behavior is unchanged (same inode's mode, written for input size); 3 source stats -> 1. Co-Authored-By: heihutu <heihutu@gmail.com> * chore(obs/cleaner): fix typo flagged by CI (mis-set -> misconfigured) Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com>
121 lines
5.2 KiB
TOML
121 lines
5.2 KiB
TOML
# Copyright 2024 RustFS Team
|
|
#
|
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
# you may not use this file except in compliance with the License.
|
|
# You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing, software
|
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
# See the License for the specific language governing permissions and
|
|
# limitations under the License.
|
|
|
|
[package]
|
|
name = "rustfs-obs"
|
|
edition.workspace = true
|
|
license.workspace = true
|
|
repository.workspace = true
|
|
rust-version.workspace = true
|
|
version.workspace = true
|
|
homepage.workspace = true
|
|
description = "Observability and monitoring tools for RustFS, providing metrics, logging, and tracing capabilities."
|
|
keywords = ["observability", "metrics", "logging", "tracing", "RustFS"]
|
|
categories = ["web-programming", "development-tools::profiling", "asynchronous", "api-bindings", "development-tools::debugging"]
|
|
documentation = "https://docs.rs/rustfs-obs/latest/rustfs_obs/"
|
|
|
|
[features]
|
|
default = []
|
|
# Tokio runtime-level telemetry. Requires a `--cfg tokio_unstable` build; the
|
|
# build script fails the compile when that flag is missing. Off by default so
|
|
# ordinary builds neither pay for nor depend on Tokio's unstable API.
|
|
dial9 = ["dep:dial9-tokio-telemetry"]
|
|
#
|
|
# NOTE: there is deliberately no `dial9-taskdump` feature. dial9 only captures a
|
|
# task dump for futures it wrapped itself, i.e. those spawned via
|
|
# `dial9_tokio_telemetry::spawn`. RustFS spawns with `tokio::spawn` throughout,
|
|
# so enabling `tokio/taskdump` would cost a Linux-only build constraint and
|
|
# record nothing. Measured on an identical workload: 0 dumps via `tokio::spawn`,
|
|
# 14709 via `dial9::spawn`. Re-adding this feature only makes sense together
|
|
# with migrating the paths under investigation to dial9's spawner.
|
|
# See rustfs/backlog#1157 (D9-16) and dial9-rs/dial9#477.
|
|
#
|
|
# NOTE: there is deliberately no `dial9-s3` feature. dial9's `worker-s3` feature
|
|
# pulls aws-sdk-s3-transfer-manager 0.1.3 (its latest), which pins
|
|
# aws-smithy-http-client onto hyper-rustls 0.24 / rustls-webpki 0.101.7. That
|
|
# webpki carries RUSTSEC-2026-0098, -0099 and -0104, and `cargo deny` rejects it.
|
|
# Feature unification cannot drop a transitive dependency, so this can only be
|
|
# fixed upstream. Tracked in rustfs/backlog#1157 (D9-14).
|
|
gpu = ["dep:nvml-wrapper"]
|
|
pyroscope = ["dep:pyroscope"]
|
|
|
|
[[example]]
|
|
name = "dial9_smoke"
|
|
required-features = ["dial9"]
|
|
|
|
[lints]
|
|
workspace = true
|
|
|
|
[dependencies]
|
|
rustfs-audit = { workspace = true }
|
|
rustfs-common = { workspace = true }
|
|
rustfs-config = { workspace = true, features = ["constants", "observability"] }
|
|
# NOTE: This dependency on rustfs-ecstore is a known architectural limitation.
|
|
# The obs crate imports types from ecstore for metrics collection.
|
|
# Breaking this dependency would require defining traits in obs and
|
|
# implementing them in ecstore, which is a significant refactoring.
|
|
# See docs/architecture/obs-ecstore-dependency-inventory.md and
|
|
# https://github.com/rustfs/backlog/issues/735 for discussion.
|
|
rustfs-ecstore = { workspace = true }
|
|
rustfs-iam = { workspace = true }
|
|
rustfs-io-metrics = { workspace = true }
|
|
rustfs-notify = { workspace = true }
|
|
rustfs-security-governance = { workspace = true }
|
|
rustfs-storage-api = { workspace = true }
|
|
rustfs-utils = { workspace = true, features = ["ip"] }
|
|
rustix = { workspace = true, features = ["fs", "stdio"] }
|
|
chrono = { workspace = true }
|
|
flate2 = { workspace = true }
|
|
glob = { workspace = true }
|
|
jiff = { workspace = true }
|
|
metrics = { workspace = true }
|
|
crossbeam-channel = { workspace = true }
|
|
crossbeam-deque = { workspace = true }
|
|
crossbeam-utils = { workspace = true }
|
|
futures-util = { workspace = true }
|
|
num_cpus = { workspace = true }
|
|
opentelemetry = { workspace = true }
|
|
opentelemetry-appender-tracing = { workspace = true }
|
|
opentelemetry_sdk = { workspace = true }
|
|
opentelemetry-stdout = { workspace = true }
|
|
opentelemetry-otlp = { workspace = true }
|
|
opentelemetry-semantic-conventions = { workspace = true }
|
|
percent-encoding = { workspace = true }
|
|
serde = { workspace = true }
|
|
serde_json = { workspace = true }
|
|
tracing = { workspace = true, features = ["std", "attributes"] }
|
|
tracing-appender = { workspace = true }
|
|
tracing-error = { workspace = true }
|
|
tracing-opentelemetry = { workspace = true }
|
|
tracing-subscriber = { workspace = true, features = ["registry", "std", "fmt", "env-filter", "tracing-log", "time", "local-time", "json"] }
|
|
tokio = { workspace = true, features = ["sync", "fs", "rt-multi-thread", "rt", "time", "macros"] }
|
|
tokio-util = { workspace = true }
|
|
dial9-tokio-telemetry = { workspace = true, optional = true }
|
|
thiserror = { workspace = true }
|
|
zstd = { workspace = true, features = ["zstdmt"] }
|
|
sysinfo = { workspace = true }
|
|
nvml-wrapper = { workspace = true, optional = true }
|
|
|
|
[target.'cfg(any(target_os = "macos", all(target_os = "linux", target_env = "gnu", target_arch = "x86_64")))'.dependencies]
|
|
pyroscope = { workspace = true, features = ["backend-pprof-rs"], optional = true }
|
|
|
|
[target.'cfg(unix)'.dependencies]
|
|
libc = { workspace = true }
|
|
|
|
|
|
[dev-dependencies]
|
|
tokio = { workspace = true, features = ["full"] }
|
|
tempfile = { workspace = true }
|
|
temp-env = { workspace = true }
|