mirror of
https://github.com/rustfs/rustfs.git
synced 2026-07-26 08:18:18 +00:00
Zhengchao An
3ed682be42
* feat(log-analyzer): add crate skeleton and unified event model
Implements LA-1 (rustfs/backlog#1282) of the log fault-analysis system
(rustfs/backlog#1281): new synchronous rustfs-log-analyzer crate with the
LogEvent/LogLevel/SourceRef/EventKind/ParseStats model shared by all
later stages. No tokio, no rustfs-* internal deps by design.
Note: thiserror listed in the issue is deferred until a stage actually
defines error types (LA-3/LA-4) to avoid an unused dependency.
* feat(log-analyzer): add line parsing layer
Implements LA-2 (rustfs/backlog#1283): four parse channels tried in order
per line — native tracing JSON, container-prefix stripping (K8s CRI /
docker compose / journald) with JSON retry, multi-line Rust panic block
folding (both pre- and post-1.65 formats, stderr has no JSON logger), and
a plain-text fallback that never fails. Parse accounting feeds the report
parse-ratio disclosure.
* feat(log-analyzer): add ingest layer for directories and archives
Implements LA-3 (rustfs/backlog#1284): expands customer inputs (files,
directories, zip/tar/tar.gz/.zst/.gz, stdin-like readers) into parsed
events. Magic-byte detection with extension fallback, recursive archive
walking with depth/entry/byte/memory caps (every capped input disclosed
in IngestReport.skipped), first-level directory names become node labels,
and nothing is ever extracted to disk so hostile entry paths are inert.
Adds tar 0.4 to workspace deps (sync; the async astral-tokio-tar used by
rustfs-zip does not fit this crate's no-tokio contract).
* feat(log-analyzer): add rule model, matching engine, and finding aggregation
Implements LA-4 (rustfs/backlog#1285): owned serde-round-trippable Rule/
Matcher/Severity types (external JSON rules deserialize into the same
types later), fail-fast RuleSet validation that reports every problem at
once, a linear-scan engine with regexes compiled once, and an
order-independent FindingsCollector (commutative aggregates only; the
test asserts byte-identical output across shuffled input orders).
* feat(log-analyzer): add built-in seed rule library (68 rules, 12 categories)
Implements LA-5 (rustfs/backlog#1286): the 2026-07 repository-wide
failure-log survey distilled into rules across disk health, erasure/
bitrot, quorum, network/RPC, distributed locks, heal, scanner, IAM,
startup/config/TLS, capacity, decommission/rebalance, and process panics.
Every anchor was verified verbatim against the source tree (94/94 hits,
zero corrections needed). Quorum rules pre-fill implies_root_cause for
the Phase-2 folding (rustfs/backlog#1290); client-side rules carry burst
thresholds (min_count) so isolated client mistakes don't clutter reports.
Tests: one realistic positive sample per rule (table-driven), exact-set
smoke samples including the intentional internode/client signature
double-hit, and negative cases.
* feat(log-analyzer): add analysis orchestration, report rendering, and redaction
Implements LA-6 (rustfs/backlog#1287): a single-pass Analyzer that does
rule matching, minute-bucket timelines (gap-filled, merged to <=60
buckets), unmatched WARN/ERROR template clustering (placeholders for
numbers/uuids/paths/addresses/quotes, 5000-template cap disclosed as
<overflow>), mixed-UTC-offset detection, and below-min_count demotion to
a low-confidence section. Renderers: pipe-friendly terminal text, stable
JSON (schema_version=1), and ticket-pasteable Markdown. --redact hashes
customer identifiers (stable h:sha256[..8]) in samples/evidence/messages
while keeping rule ids, targets, and panic locations intact.
* feat(rustfs): add 'rustfs diagnose' subcommand for offline log fault analysis
Implements LA-7 (rustfs/backlog#1288): wires rustfs-log-analyzer into the
main binary as a diagnose subcommand that short-circuits before
observability/storage init (same pattern as 'info' / 'tls inspect') so
the report on stdout is never wrapped by the JSON logger.
rustfs diagnose <paths>... [--format text|json|md] [--since 24h]
[--until ...] [--min-level warn] [--redact] [--top N] [--samples N]
Accepts files, directories, archives (.zip/.tar/.tar.gz/.zst/.gz) and '-'
for stdin. Exit codes: 0 = diagnosis completed (findings never fail the
process), 2 = bad arguments / no readable input.
diagnose_e2e covers the six MVP acceptance scenarios from
rustfs/backlog#1281 (directory+zst archive, multi-node zip attribution,
CRI-prefixed kubectl logs, panic folding, stable JSON schema, CLI parsing
incl. the legacy 'rustfs <volume>' preprocessor regression); the
full-binary smoke test is #[ignore]d (run with -- --ignored).
Usage doc: docs/operations/log-diagnose.md.
* ci(log-analyzer): guard rule anchors against log-message drift
Implements LA-8 (rustfs/backlog#1289): every seed-rule anchor must exist
verbatim in the rustfs source tree, so changing a log message without
updating its rule fails the gate instead of silently killing the rule.
- la-dump-anchors bin emits 'rule_id<TAB>anchor' TSV;
- scripts/check_log_analyzer_rules.sh greps each anchor (fixed-string,
*.rs only, excluding crates/log-analyzer itself to avoid self-matches);
- RuleSet::new now rejects anchors that are blank, contain tab/newline,
or are shorter than 8 bytes (no discriminating power); the '[FATAL]'
anchor gained its trailing space to meet the floor while still matching
the emit_fatal_stderr format string;
- wired as log-analyzer-rules-check into the pre-pr gate (it compiles the
crate, so it stays out of the fast pre-commit set).
Negative self-test: breaking an anchor makes the script exit 1 naming the
rule ('MISSING anchor for rule inconsistent-drive: zzz-not-exist-anchor').
* refactor(log-analyzer): use root-relative provenance for directory inputs
Binary smoke run showed report samples citing full absolute paths, which
drowns the useful part. Directory inputs now label sources as
"<root-name>/<relative-path>" (e.g. "smoke-logs/node1/rustfs.log");
archives and single files keep their existing provenance.
* chore(log-analyzer): reword comment to satisfy the typos gate
* fix(log-analyzer): declare chrono serde feature locally after workspace feature localization
* fix(log-analyzer): bound line reads so a newline-less input cannot bypass the byte cap
read_until grew the line buffer with the entire remaining stream before the
max_total_bytes check ran, so a single multi-GB line (decompression bomb or
corrupt file) could allocate unboundedly. Replace it with a capped reader that
enforces the remaining global budget chunk-by-chunk and adds a per-line cap
(IngestOptions::max_line_bytes, default 1 MiB); over-cap tails are discarded
but still charged, and truncation is disclosed once per file as the new
line_too_long skip reason. Flagged by Codex review on #4876.
* fix(log-analyzer): redact field-shaped identifiers inside message text and widen the hash to 64 bits
--redact only hashed IPv4 literals in unstructured message text, so
bucket/object/access-key values embedded in messages (access_key=AK123,
'bucket: media, object: private/a.bin') leaked into reports documented as
safe to forward. Apply the SENSITIVE_FIELDS list to key=value / key: value
shapes in message text with the same hash as structured fields, and extend
the hash from 8 to 16 hex chars so cross-identifier collisions stay
negligible. Flagged by Codex and Copilot review on #4876.
* fix(log-analyzer): strip collector prefixes before panic-block absorption
An open panic block tested continuation lines against absorbs() before their
CRI/compose/journald prefix was stripped, so containerized panics stored the
prefix in the payload and split into a truncated panic plus text noise as soon
as the note/backtrace lines arrived. Stripping now happens once at the top of
feed() and every channel judges the payload. Flagged by Codex review on #4876.
* fix(log-analyzer): remove two input-order dependencies in representative selection
The unmatched-cluster target stayed pinned to the first-seen event while the
representative sample could be replaced, so sample and target could come from
different events and vary with input order; the pair now updates together by
lexicographic (sample, target) min. Sample selection tie-broke on line number
alone, which is only unique within one file; the key now includes the source
file. Flagged by Copilot review on #4876.
* fix(rustfs): reject negative relative times in diagnose --since/--until
parse_time_arg accepted "-24h" and produced a future timestamp, contradicting
the documented 'counted back from now' semantics. The amount now parses as
unsigned, so a leading '-' fails with the usual invalid-time error. Flagged by
Copilot review on #4876.
* feat(log-analyzer): Phase 2 — causal folding, timeline anomalies, external rules (LA-9) (#4942)
* feat(log-analyzer): collapse cascade symptoms under their root-cause finding
Phase-2 sub-item A (rustfs/backlog#1290): a finding whose rule declares
implies_root_cause edges folds under a qualifying root — root.first_seen <=
symptom.first_seen + 5min and root.last_seen >= symptom.first_seen - 30min,
existence-based when either side has no timestamps (pure stderr panics).
Findings gain collapsed_into/caused; text/markdown render the root block with
an indented cascade line and stop listing collapsed symptoms flat, while JSON
keeps every finding. Roots are promoted to the most severe position among
their block so the report top still answers 'the most likely cause'.
* feat(log-analyzer): detect timeline/clock anomalies (schema v2)
Phase-2 sub-item B (rustfs/backlog#1290): three deterministic hints rendered
between the summary and findings — mixed UTC offsets (with a clock-skew note
when signature-mismatch findings coexist), per-node time ranges that do not
overlap at all (both nodes >100 timestamped events), and log gaps of at least
max(15min, 3x bucket width) after >=3 consecutive active minutes, upgraded to
restart evidence when a startup-class finding begins within 5min after the
gap. JSON gains timeline_anomalies and schema_version bumps to 2.
* feat(diagnose): load external rules with --rules <file.json>
Phase-2 sub-item C (rustfs/backlog#1290): an external JSON rule file
({schema_version: 1, rules: [Rule...]}, the exact serde shape of the built-in
rules) merges over the seed library, with same-id rules replacing built-ins so
the support team can hotfix a misfiring rule without a release. The merged set
validates as a whole and any problem (bad regex, duplicate id, empty matcher
group, wrong schema version) prints every error and exits 2 — analysis never
runs on a half-broken set. External anchors are exempt from the CI anchor
guard, documented as author-owned quality. Adds the custom-rules section to
docs/operations/log-diagnose.md.
* fix(log-analyzer): address PR #4876 review (redaction coverage, order-independence, guards)
Redaction (--redact) now honours its "forwardable" intent across every report surface instead of a 15-name field whitelist applied over a subset:
- redact_event scrubs the full fields map (sensitive names hashed whole, every other value run through redact_text) plus provenance, so the JSON/Markdown full-sample dump no longer leaks non-whitelisted fields (client_ip, url, user, ...).
- node labels are hashed once at ingestion, so summary.nodes, per-node timeline ranges, samples and timeline anomalies stay consistent and correlatable under one stable hash.
- evidence values, unmatched-cluster templates and skipped-input paths are now redacted; peer/disk/drive/volume/node/user added to the sensitive set; IPv6 literals are hashed (without touching `rust::paths` or HH:MM:SS clocks); provenance keeps the leaf filename and hashes the customer directory/archive prefix.
- redact.rs and docs/operations/log-diagnose.md reworded to "best-effort identifier scrubbing", not an anonymization guarantee.
Order-independence (the crate's headline contract):
- the evidence value cap keeps the lexicographically smallest N distinct values instead of the first-N-by-arrival (previously order-dependent).
- first_seen/last_seen break equal-instant ties on the offset, so the serialized RFC3339 offset no longer depends on input order.
Parsing:
- a new-format panic header no longer swallows the line immediately after it when that line is itself a JSON event or a second panic header (previously dropped an interleaved ERROR in merged stdout/stderr, or merged a panic-during-panic); trailing "note: ..." backtrace lines now fold into the block.
CLI:
- diagnose --since/--until reject absurd relative amounts via checked_sub_signed instead of panicking; the exit-code doc now matches actual behaviour.
CI:
- check_log_analyzer_rules.sh is wired into the ci.yml test-and-lint job (it was only in make pre-pr, so anchor drift from other PRs could merge green).
Markdown table cells escape '|' so customer log text cannot break the table structure.
---------
Co-authored-by: houseme <housemecn@gmail.com>
4.8 KiB
4.8 KiB
日志故障诊断(rustfs diagnose)
对客户/现场提供的日志文件做离线故障归因:解析 RustFS 的 JSON 日志
(含 kubectl logs / docker compose / journald 采集前缀与 stderr panic 块),
匹配内置故障规则库,输出按严重度排序的诊断报告。设计与规则清单见
rustfs/backlog#1281(总纲)。
不启动存储、不联网、跑完即退,任何装有 rustfs 二进制的机器都可用。
用法
# 分析一个目录(自动处理 .zst/.gz 轮转归档)
rustfs diagnose /var/log/rustfs/
# 客户打包的多节点日志(zip/tar.gz 自动递归展开,第一层目录名作为节点标签)
rustfs diagnose customer-logs.zip
# 从 stdin 读(容器场景)
kubectl logs rustfs-0 | rustfs diagnose -
# 只看最近 24 小时,输出 Markdown 直接贴工单
rustfs diagnose logs.tar.gz --since 24h --format md > report.md
常用参数:--format text|json|md(默认 text;JSON 带稳定 schema_version)、
--since/--until(RFC-3339 或相对时间 30m/24h/7d)、--min-level、
--top N(未识别模式条数)、--samples N(每个 finding 的样本行数)。
报告解读
- 发现(按严重度):P0 数据风险 → P1 服务不可用 → P2 降级 → P3 客户端侧 → P4 提示。每条带诊断结论、建议动作、证据字段与样本行。
- 因果折叠:当症状类发现(如 quorum 刷屏)在时间上跟随其已知根因
(如盘 faulty)出现时,报告把症状折叠进根因块的"级联症状"行,根因块
提升到两者中更高的严重度位置——报告首块直接回答"最可能的原因"。
JSON 输出保留全部 findings(
collapsed_into/caused字段标注关系)。 - 时间线异常(提示):三个确定性启发,均为提示不定罪——混合 UTC 偏移 (伴签名错误时点名时钟偏移)、节点时间范围完全不重叠、时间线断档 (断档后紧跟 startup 类发现时升级为"重启证据")。
- 低频提示:命中数低于规则阈值的匹配(如零星的签名错误),仅供参考。
- 未识别的高频错误:规则库未覆盖的 WARN/ERROR 消息模板聚类。这一节是
规则库迭代的输入——反复出现的新模板请提给维护者补规则
(
crates/log-analyzer/src/rules/seed/)。 - 跳过的输入与时区提示:所有被跳过的文件(二进制/超限)逐条披露;
混合 UTC 偏移会显式提醒(时钟偏移本身就是
SignatureDoesNotMatch的常见根因)。
--redact(报告需要转发时)
把 bucket/object/AK、IPv4/IPv6、peer 与磁盘路径、节点标签、来源文件路径等客户标识替换为稳定哈希(同值同哈希,保持可关联);规则 id、诊断文本、模块 target 与 panic 源码位置(RustFS 自身代码,非客户数据)保留。样本会连同其完整 fields 一起脱敏。
覆盖是尽力而为,不是绝对保证:结构化字段与 key=value/IP 形态的文本都会被处理,但散落在自由文本里、既非字段形态也非 IP 的标识(例如句子里顺带提到的一个 bucket 名)可能仍有残留。转发前建议再抽查一遍。
自定义规则(--rules <file.json>)
支持团队可以在不等发版的情况下补规则,或热修一条误报的内置规则:
rustfs diagnose customer.zip --rules extra-rules.json
文件格式(Rule 的 JSON 表示与内置规则完全一致):
{
"schema_version": 1,
"rules": [
{
"id": "custom-oom-killer",
"severity": "p1_unavailable",
"category": "process",
"title": "内核 OOM killer 终止了进程",
"matcher": { "message_contains": "Out of memory: Killed process" },
"diagnosis": "内核因内存不足杀掉了 rustfs 进程。",
"suggestion": "检查内存限制与其他驻留进程;考虑调大内存或加节点。"
}
]
}
severity:p0_data_risk | p1_unavailable | p2_degraded | p3_client_side | p4_info;matcher:message_prefix/message_contains/message_regex/field_equals {name, value}/target_prefix/is_panic/min_level/all [..]/any [..],与内置规则同一套类型;- 可选字段:
evidence_fields、min_count(默认 1)、implies_root_cause(参与因果折叠)、anchors; - 同 id 覆盖内置规则(用于热修误报);合并后的规则集整体校验,任何错误 (坏 regex、重复 id、空 matcher 组)会逐条打印并以退出码 2 失败——不会 带着半坏的规则集分析;
- 外部规则的
anchors不受 CI 锚点守卫约束,质量由文件作者自担。
已知边界
- panic 只出现在 stderr:若客户只采集了 stdout,panic 不会出现在日志里,
但
rwlock ... poisoned类发现会提示"曾发生 panic"。 - 审计日志(camelCase 外发流)不在本工具范围内。
- 规则锚点由 CI 守卫(rustfs/backlog#1289)保证与源码日志文案同步。