mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-25 13:36:50 +00:00
feat: offline log fault-analysis system (rustfs diagnose) (#4876)
* feat(log-analyzer): add crate skeleton and unified event model
Implements LA-1 (rustfs/backlog#1282) of the log fault-analysis system
(rustfs/backlog#1281): new synchronous rustfs-log-analyzer crate with the
LogEvent/LogLevel/SourceRef/EventKind/ParseStats model shared by all
later stages. No tokio, no rustfs-* internal deps by design.
Note: thiserror listed in the issue is deferred until a stage actually
defines error types (LA-3/LA-4) to avoid an unused dependency.
* feat(log-analyzer): add line parsing layer
Implements LA-2 (rustfs/backlog#1283): four parse channels tried in order
per line — native tracing JSON, container-prefix stripping (K8s CRI /
docker compose / journald) with JSON retry, multi-line Rust panic block
folding (both pre- and post-1.65 formats, stderr has no JSON logger), and
a plain-text fallback that never fails. Parse accounting feeds the report
parse-ratio disclosure.
* feat(log-analyzer): add ingest layer for directories and archives
Implements LA-3 (rustfs/backlog#1284): expands customer inputs (files,
directories, zip/tar/tar.gz/.zst/.gz, stdin-like readers) into parsed
events. Magic-byte detection with extension fallback, recursive archive
walking with depth/entry/byte/memory caps (every capped input disclosed
in IngestReport.skipped), first-level directory names become node labels,
and nothing is ever extracted to disk so hostile entry paths are inert.
Adds tar 0.4 to workspace deps (sync; the async astral-tokio-tar used by
rustfs-zip does not fit this crate's no-tokio contract).
* feat(log-analyzer): add rule model, matching engine, and finding aggregation
Implements LA-4 (rustfs/backlog#1285): owned serde-round-trippable Rule/
Matcher/Severity types (external JSON rules deserialize into the same
types later), fail-fast RuleSet validation that reports every problem at
once, a linear-scan engine with regexes compiled once, and an
order-independent FindingsCollector (commutative aggregates only; the
test asserts byte-identical output across shuffled input orders).
* feat(log-analyzer): add built-in seed rule library (68 rules, 12 categories)
Implements LA-5 (rustfs/backlog#1286): the 2026-07 repository-wide
failure-log survey distilled into rules across disk health, erasure/
bitrot, quorum, network/RPC, distributed locks, heal, scanner, IAM,
startup/config/TLS, capacity, decommission/rebalance, and process panics.
Every anchor was verified verbatim against the source tree (94/94 hits,
zero corrections needed). Quorum rules pre-fill implies_root_cause for
the Phase-2 folding (rustfs/backlog#1290); client-side rules carry burst
thresholds (min_count) so isolated client mistakes don't clutter reports.
Tests: one realistic positive sample per rule (table-driven), exact-set
smoke samples including the intentional internode/client signature
double-hit, and negative cases.
* feat(log-analyzer): add analysis orchestration, report rendering, and redaction
Implements LA-6 (rustfs/backlog#1287): a single-pass Analyzer that does
rule matching, minute-bucket timelines (gap-filled, merged to <=60
buckets), unmatched WARN/ERROR template clustering (placeholders for
numbers/uuids/paths/addresses/quotes, 5000-template cap disclosed as
<overflow>), mixed-UTC-offset detection, and below-min_count demotion to
a low-confidence section. Renderers: pipe-friendly terminal text, stable
JSON (schema_version=1), and ticket-pasteable Markdown. --redact hashes
customer identifiers (stable h:sha256[..8]) in samples/evidence/messages
while keeping rule ids, targets, and panic locations intact.
* feat(rustfs): add 'rustfs diagnose' subcommand for offline log fault analysis
Implements LA-7 (rustfs/backlog#1288): wires rustfs-log-analyzer into the
main binary as a diagnose subcommand that short-circuits before
observability/storage init (same pattern as 'info' / 'tls inspect') so
the report on stdout is never wrapped by the JSON logger.
rustfs diagnose <paths>... [--format text|json|md] [--since 24h]
[--until ...] [--min-level warn] [--redact] [--top N] [--samples N]
Accepts files, directories, archives (.zip/.tar/.tar.gz/.zst/.gz) and '-'
for stdin. Exit codes: 0 = diagnosis completed (findings never fail the
process), 2 = bad arguments / no readable input.
diagnose_e2e covers the six MVP acceptance scenarios from
rustfs/backlog#1281 (directory+zst archive, multi-node zip attribution,
CRI-prefixed kubectl logs, panic folding, stable JSON schema, CLI parsing
incl. the legacy 'rustfs <volume>' preprocessor regression); the
full-binary smoke test is #[ignore]d (run with -- --ignored).
Usage doc: docs/operations/log-diagnose.md.
* ci(log-analyzer): guard rule anchors against log-message drift
Implements LA-8 (rustfs/backlog#1289): every seed-rule anchor must exist
verbatim in the rustfs source tree, so changing a log message without
updating its rule fails the gate instead of silently killing the rule.
- la-dump-anchors bin emits 'rule_id<TAB>anchor' TSV;
- scripts/check_log_analyzer_rules.sh greps each anchor (fixed-string,
*.rs only, excluding crates/log-analyzer itself to avoid self-matches);
- RuleSet::new now rejects anchors that are blank, contain tab/newline,
or are shorter than 8 bytes (no discriminating power); the '[FATAL]'
anchor gained its trailing space to meet the floor while still matching
the emit_fatal_stderr format string;
- wired as log-analyzer-rules-check into the pre-pr gate (it compiles the
crate, so it stays out of the fast pre-commit set).
Negative self-test: breaking an anchor makes the script exit 1 naming the
rule ('MISSING anchor for rule inconsistent-drive: zzz-not-exist-anchor').
* refactor(log-analyzer): use root-relative provenance for directory inputs
Binary smoke run showed report samples citing full absolute paths, which
drowns the useful part. Directory inputs now label sources as
"<root-name>/<relative-path>" (e.g. "smoke-logs/node1/rustfs.log");
archives and single files keep their existing provenance.
* chore(log-analyzer): reword comment to satisfy the typos gate
* fix(log-analyzer): declare chrono serde feature locally after workspace feature localization
* fix(log-analyzer): bound line reads so a newline-less input cannot bypass the byte cap
read_until grew the line buffer with the entire remaining stream before the
max_total_bytes check ran, so a single multi-GB line (decompression bomb or
corrupt file) could allocate unboundedly. Replace it with a capped reader that
enforces the remaining global budget chunk-by-chunk and adds a per-line cap
(IngestOptions::max_line_bytes, default 1 MiB); over-cap tails are discarded
but still charged, and truncation is disclosed once per file as the new
line_too_long skip reason. Flagged by Codex review on #4876.
* fix(log-analyzer): redact field-shaped identifiers inside message text and widen the hash to 64 bits
--redact only hashed IPv4 literals in unstructured message text, so
bucket/object/access-key values embedded in messages (access_key=AK123,
'bucket: media, object: private/a.bin') leaked into reports documented as
safe to forward. Apply the SENSITIVE_FIELDS list to key=value / key: value
shapes in message text with the same hash as structured fields, and extend
the hash from 8 to 16 hex chars so cross-identifier collisions stay
negligible. Flagged by Codex and Copilot review on #4876.
* fix(log-analyzer): strip collector prefixes before panic-block absorption
An open panic block tested continuation lines against absorbs() before their
CRI/compose/journald prefix was stripped, so containerized panics stored the
prefix in the payload and split into a truncated panic plus text noise as soon
as the note/backtrace lines arrived. Stripping now happens once at the top of
feed() and every channel judges the payload. Flagged by Codex review on #4876.
* fix(log-analyzer): remove two input-order dependencies in representative selection
The unmatched-cluster target stayed pinned to the first-seen event while the
representative sample could be replaced, so sample and target could come from
different events and vary with input order; the pair now updates together by
lexicographic (sample, target) min. Sample selection tie-broke on line number
alone, which is only unique within one file; the key now includes the source
file. Flagged by Copilot review on #4876.
* fix(rustfs): reject negative relative times in diagnose --since/--until
parse_time_arg accepted "-24h" and produced a future timestamp, contradicting
the documented 'counted back from now' semantics. The amount now parses as
unsigned, so a leading '-' fails with the usual invalid-time error. Flagged by
Copilot review on #4876.
* feat(log-analyzer): Phase 2 — causal folding, timeline anomalies, external rules (LA-9) (#4942)
* feat(log-analyzer): collapse cascade symptoms under their root-cause finding
Phase-2 sub-item A (rustfs/backlog#1290): a finding whose rule declares
implies_root_cause edges folds under a qualifying root — root.first_seen <=
symptom.first_seen + 5min and root.last_seen >= symptom.first_seen - 30min,
existence-based when either side has no timestamps (pure stderr panics).
Findings gain collapsed_into/caused; text/markdown render the root block with
an indented cascade line and stop listing collapsed symptoms flat, while JSON
keeps every finding. Roots are promoted to the most severe position among
their block so the report top still answers 'the most likely cause'.
* feat(log-analyzer): detect timeline/clock anomalies (schema v2)
Phase-2 sub-item B (rustfs/backlog#1290): three deterministic hints rendered
between the summary and findings — mixed UTC offsets (with a clock-skew note
when signature-mismatch findings coexist), per-node time ranges that do not
overlap at all (both nodes >100 timestamped events), and log gaps of at least
max(15min, 3x bucket width) after >=3 consecutive active minutes, upgraded to
restart evidence when a startup-class finding begins within 5min after the
gap. JSON gains timeline_anomalies and schema_version bumps to 2.
* feat(diagnose): load external rules with --rules <file.json>
Phase-2 sub-item C (rustfs/backlog#1290): an external JSON rule file
({schema_version: 1, rules: [Rule...]}, the exact serde shape of the built-in
rules) merges over the seed library, with same-id rules replacing built-ins so
the support team can hotfix a misfiring rule without a release. The merged set
validates as a whole and any problem (bad regex, duplicate id, empty matcher
group, wrong schema version) prints every error and exits 2 — analysis never
runs on a half-broken set. External anchors are exempt from the CI anchor
guard, documented as author-owned quality. Adds the custom-rules section to
docs/operations/log-diagnose.md.
* fix(log-analyzer): address PR #4876 review (redaction coverage, order-independence, guards)
Redaction (--redact) now honours its "forwardable" intent across every report surface instead of a 15-name field whitelist applied over a subset:
- redact_event scrubs the full fields map (sensitive names hashed whole, every other value run through redact_text) plus provenance, so the JSON/Markdown full-sample dump no longer leaks non-whitelisted fields (client_ip, url, user, ...).
- node labels are hashed once at ingestion, so summary.nodes, per-node timeline ranges, samples and timeline anomalies stay consistent and correlatable under one stable hash.
- evidence values, unmatched-cluster templates and skipped-input paths are now redacted; peer/disk/drive/volume/node/user added to the sensitive set; IPv6 literals are hashed (without touching `rust::paths` or HH:MM:SS clocks); provenance keeps the leaf filename and hashes the customer directory/archive prefix.
- redact.rs and docs/operations/log-diagnose.md reworded to "best-effort identifier scrubbing", not an anonymization guarantee.
Order-independence (the crate's headline contract):
- the evidence value cap keeps the lexicographically smallest N distinct values instead of the first-N-by-arrival (previously order-dependent).
- first_seen/last_seen break equal-instant ties on the offset, so the serialized RFC3339 offset no longer depends on input order.
Parsing:
- a new-format panic header no longer swallows the line immediately after it when that line is itself a JSON event or a second panic header (previously dropped an interleaved ERROR in merged stdout/stderr, or merged a panic-during-panic); trailing "note: ..." backtrace lines now fold into the block.
CLI:
- diagnose --since/--until reject absurd relative amounts via checked_sub_signed instead of panicking; the exit-code doc now matches actual behaviour.
CI:
- check_log_analyzer_rules.sh is wired into the ci.yml test-and-lint job (it was only in make pre-pr, so anchor drift from other PRs could merge green).
Markdown table cells escape '|' so customer log text cannot break the table structure.
---------
Co-authored-by: houseme <housemecn@gmail.com>
This commit is contained in:
Zhengchao An
@@ -0,0 +1,93 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! Matching engine: a linear scan over all rules per event, with regexes
|
||||
//! compiled once. The seed set is ~60 shallow rules — no indexing until a
|
||||
//! profile says otherwise.
|
||||
|
||||
use super::model::{Matcher, Rule};
|
||||
use super::rule_set::RuleSet;
|
||||
use crate::model::{EventKind, LogEvent};
|
||||
use regex::Regex;
|
||||
|
||||
enum CompiledMatcher {
|
||||
MessagePrefix(String),
|
||||
MessageContains(String),
|
||||
MessageRegex(Regex),
|
||||
FieldEquals { name: String, value: String },
|
||||
TargetPrefix(String),
|
||||
IsPanic,
|
||||
MinLevel(crate::model::LogLevel),
|
||||
All(Vec<CompiledMatcher>),
|
||||
Any(Vec<CompiledMatcher>),
|
||||
}
|
||||
|
||||
fn compile(matcher: &Matcher) -> CompiledMatcher {
|
||||
match matcher {
|
||||
Matcher::MessagePrefix(p) => CompiledMatcher::MessagePrefix(p.clone()),
|
||||
Matcher::MessageContains(c) => CompiledMatcher::MessageContains(c.clone()),
|
||||
// RuleSet::new already verified the regex compiles; this cannot fail here.
|
||||
Matcher::MessageRegex(re) => CompiledMatcher::MessageRegex(Regex::new(re).expect("validated by RuleSet::new")),
|
||||
Matcher::FieldEquals { name, value } => CompiledMatcher::FieldEquals {
|
||||
name: name.clone(),
|
||||
value: value.clone(),
|
||||
},
|
||||
Matcher::TargetPrefix(p) => CompiledMatcher::TargetPrefix(p.clone()),
|
||||
Matcher::IsPanic => CompiledMatcher::IsPanic,
|
||||
Matcher::MinLevel(level) => CompiledMatcher::MinLevel(*level),
|
||||
Matcher::All(inner) => CompiledMatcher::All(inner.iter().map(compile).collect()),
|
||||
Matcher::Any(inner) => CompiledMatcher::Any(inner.iter().map(compile).collect()),
|
||||
}
|
||||
}
|
||||
|
||||
fn eval(matcher: &CompiledMatcher, event: &LogEvent) -> bool {
|
||||
match matcher {
|
||||
CompiledMatcher::MessagePrefix(p) => event.message.starts_with(p.as_str()),
|
||||
CompiledMatcher::MessageContains(c) => event.message.contains(c.as_str()),
|
||||
CompiledMatcher::MessageRegex(re) => re.is_match(&event.message),
|
||||
CompiledMatcher::FieldEquals { name, value } => event.field_display(name).as_deref() == Some(value.as_str()),
|
||||
CompiledMatcher::TargetPrefix(p) => event.target.as_deref().is_some_and(|t| t.starts_with(p.as_str())),
|
||||
CompiledMatcher::IsPanic => event.kind == EventKind::Panic,
|
||||
CompiledMatcher::MinLevel(level) => event.level.is_some_and(|l| l >= *level),
|
||||
CompiledMatcher::All(inner) => inner.iter().all(|m| eval(m, event)),
|
||||
CompiledMatcher::Any(inner) => inner.iter().any(|m| eval(m, event)),
|
||||
}
|
||||
}
|
||||
|
||||
pub struct RuleEngine {
|
||||
set: RuleSet,
|
||||
compiled: Vec<CompiledMatcher>,
|
||||
}
|
||||
|
||||
impl RuleEngine {
|
||||
pub fn new(set: RuleSet) -> Self {
|
||||
let compiled = set.rules().iter().map(|r| compile(&r.matcher)).collect();
|
||||
Self { set, compiled }
|
||||
}
|
||||
|
||||
/// Indexes (aligned with [`Self::rules`]) of every rule the event hits.
|
||||
/// An event may hit several rules; arbitration happens at report time.
|
||||
pub fn matches(&self, event: &LogEvent) -> Vec<usize> {
|
||||
self.compiled
|
||||
.iter()
|
||||
.enumerate()
|
||||
.filter(|(_, m)| eval(m, event))
|
||||
.map(|(idx, _)| idx)
|
||||
.collect()
|
||||
}
|
||||
|
||||
pub fn rules(&self) -> &[Rule] {
|
||||
self.set.rules()
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,190 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! External rule file loading (`--rules`, rustfs/backlog#1290 sub-item C):
|
||||
//! the support team ships new rules — or hotfixes a misfiring built-in one —
|
||||
//! without waiting for a release. External rules deserialize into the exact
|
||||
//! same [`Rule`] type as the seed library (design decision 3 of
|
||||
//! rustfs/backlog#1281); their anchors are NOT covered by the CI anchor
|
||||
//! guard, so their quality is on the file author.
|
||||
|
||||
use super::model::Rule;
|
||||
use super::rule_set::{RuleSet, RuleSetError};
|
||||
use super::seed::seed_rules;
|
||||
use serde::Deserialize;
|
||||
use thiserror::Error;
|
||||
|
||||
/// Supported `schema_version` of the external rules file.
|
||||
pub const EXTERNAL_RULES_SCHEMA_VERSION: u32 = 1;
|
||||
|
||||
#[derive(Debug, Deserialize)]
|
||||
struct ExternalRuleFile {
|
||||
schema_version: u32,
|
||||
rules: Vec<Rule>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Error)]
|
||||
pub enum ExternalRulesError {
|
||||
#[error("cannot parse rules file: {0}")]
|
||||
Parse(#[from] serde_json::Error),
|
||||
#[error("unsupported rules schema_version {found} (supported: {EXTERNAL_RULES_SCHEMA_VERSION})")]
|
||||
SchemaVersion { found: u32 },
|
||||
#[error("duplicate rule id '{0}' within the external rules file")]
|
||||
DuplicateInFile(String),
|
||||
#[error(transparent)]
|
||||
Invalid(#[from] RuleSetError),
|
||||
}
|
||||
|
||||
/// Parses an external rule file and merges it over the built-in seed rules.
|
||||
/// An external rule with the same id REPLACES the built-in one; the merged
|
||||
/// set is validated as a whole and any problem fails the load — analysis
|
||||
/// never runs with a half-broken rule set.
|
||||
pub fn seed_rules_with_external(json: &str) -> Result<RuleSet, ExternalRulesError> {
|
||||
let file: ExternalRuleFile = serde_json::from_str(json)?;
|
||||
if file.schema_version != EXTERNAL_RULES_SCHEMA_VERSION {
|
||||
return Err(ExternalRulesError::SchemaVersion {
|
||||
found: file.schema_version,
|
||||
});
|
||||
}
|
||||
// Same-id merging would silently swallow duplicates inside the file
|
||||
// itself, so reject those before merging.
|
||||
let mut seen = std::collections::BTreeSet::new();
|
||||
for rule in &file.rules {
|
||||
if !seen.insert(rule.id.clone()) {
|
||||
return Err(ExternalRulesError::DuplicateInFile(rule.id.clone()));
|
||||
}
|
||||
}
|
||||
|
||||
let mut rules = seed_rules();
|
||||
for external in file.rules {
|
||||
match rules.iter_mut().find(|rule| rule.id == external.id) {
|
||||
Some(existing) => *existing = external,
|
||||
None => rules.push(external),
|
||||
}
|
||||
}
|
||||
Ok(RuleSet::new(rules)?)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::model::{EventKind, LogEvent, LogLevel, SourceRef};
|
||||
use crate::rules::RuleEngine;
|
||||
use std::sync::Arc;
|
||||
|
||||
fn event(message: &str) -> LogEvent {
|
||||
LogEvent {
|
||||
timestamp: None,
|
||||
level: Some(LogLevel::Error),
|
||||
target: None,
|
||||
message: message.to_string(),
|
||||
fields: serde_json::Map::new(),
|
||||
source: SourceRef {
|
||||
file: Arc::from("rustfs.log"),
|
||||
line: 1,
|
||||
},
|
||||
node: None,
|
||||
kind: EventKind::Json,
|
||||
}
|
||||
}
|
||||
|
||||
/// The canonical external-file shape; also documents the on-disk format
|
||||
/// (kept in sync with docs/operations/log-diagnose.md).
|
||||
const EXTERNAL_FILE: &str = r#"{
|
||||
"schema_version": 1,
|
||||
"rules": [
|
||||
{
|
||||
"id": "custom-oom-killer",
|
||||
"severity": "p1_unavailable",
|
||||
"category": "process",
|
||||
"title": "内核 OOM killer 终止了进程",
|
||||
"matcher": { "message_contains": "Out of memory: Killed process" },
|
||||
"diagnosis": "内核因内存不足杀掉了 rustfs 进程。",
|
||||
"suggestion": "检查内存限制与其他驻留进程;考虑调大内存或加节点。"
|
||||
}
|
||||
]
|
||||
}"#;
|
||||
|
||||
#[test]
|
||||
fn external_rule_is_added_and_matches() {
|
||||
let set = seed_rules_with_external(EXTERNAL_FILE).expect("load");
|
||||
let seed_len = seed_rules().len();
|
||||
assert_eq!(set.rules().len(), seed_len + 1);
|
||||
let engine = RuleEngine::new(set);
|
||||
let hits = engine.matches(&event("Out of memory: Killed process 1234 (rustfs)"));
|
||||
assert_eq!(hits.len(), 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn same_id_overrides_the_builtin_rule() {
|
||||
let file = r#"{
|
||||
"schema_version": 1,
|
||||
"rules": [
|
||||
{
|
||||
"id": "ec-write-quorum",
|
||||
"severity": "p4_info",
|
||||
"category": "erasure",
|
||||
"title": "覆盖后的写仲裁规则",
|
||||
"matcher": { "message_contains": "OVERRIDDEN-ANCHOR-TEXT" },
|
||||
"diagnosis": "外部覆盖。",
|
||||
"suggestion": "外部覆盖。"
|
||||
}
|
||||
]
|
||||
}"#;
|
||||
let set = seed_rules_with_external(file).expect("load");
|
||||
assert_eq!(set.rules().len(), seed_rules().len(), "override must not grow the set");
|
||||
let engine = RuleEngine::new(set);
|
||||
// The built-in matcher no longer fires; the external one does.
|
||||
assert!(
|
||||
engine
|
||||
.matches(&event("erasure write quorum (required=8, achieved=5)"))
|
||||
.is_empty()
|
||||
);
|
||||
assert_eq!(engine.matches(&event("OVERRIDDEN-ANCHOR-TEXT hit")).len(), 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn bad_files_fail_fast_with_actionable_errors() {
|
||||
// Not JSON at all.
|
||||
assert!(matches!(seed_rules_with_external("nope"), Err(ExternalRulesError::Parse(_))));
|
||||
|
||||
// Wrong schema version.
|
||||
let err = seed_rules_with_external(r#"{"schema_version": 9, "rules": []}"#).expect_err("schema");
|
||||
assert!(err.to_string().contains("schema_version 9"), "{err}");
|
||||
|
||||
// Duplicate id inside the file.
|
||||
let dup = r#"{
|
||||
"schema_version": 1,
|
||||
"rules": [
|
||||
{"id": "x-dup", "severity": "p4_info", "category": "ops", "title": "a",
|
||||
"matcher": {"message_contains": "aaaaaaaaaa"}, "diagnosis": "d", "suggestion": "s"},
|
||||
{"id": "x-dup", "severity": "p4_info", "category": "ops", "title": "b",
|
||||
"matcher": {"message_contains": "bbbbbbbbbb"}, "diagnosis": "d", "suggestion": "s"}
|
||||
]
|
||||
}"#;
|
||||
let err = seed_rules_with_external(dup).expect_err("dup");
|
||||
assert!(err.to_string().contains("x-dup"), "{err}");
|
||||
|
||||
// Bad regex: the merged-set validation names the offending rule.
|
||||
let bad_regex = r#"{
|
||||
"schema_version": 1,
|
||||
"rules": [
|
||||
{"id": "x-bad-regex", "severity": "p4_info", "category": "ops", "title": "a",
|
||||
"matcher": {"message_regex": "[unclosed"}, "diagnosis": "d", "suggestion": "s"}
|
||||
]
|
||||
}"#;
|
||||
let err = seed_rules_with_external(bad_regex).expect_err("regex");
|
||||
assert!(err.to_string().contains("x-bad-regex"), "{err}");
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,201 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! Finding aggregation. Every accumulator is a commutative monoid
|
||||
//! (count/min/max/set-union), so results are byte-identical no matter the
|
||||
//! input order — the order-independence contract of rustfs/backlog#1281.
|
||||
|
||||
use super::model::{Rule, Severity};
|
||||
use crate::model::LogEvent;
|
||||
use chrono::{DateTime, FixedOffset};
|
||||
use serde::Serialize;
|
||||
use std::collections::{BTreeMap, BTreeSet};
|
||||
|
||||
const EVIDENCE_VALUES_CAP: usize = 10;
|
||||
|
||||
/// Deduplicated evidence values for one field, capped with an overflow count.
|
||||
#[derive(Debug, Clone, Default, Serialize)]
|
||||
pub struct EvidenceValues {
|
||||
pub values: BTreeSet<String>,
|
||||
/// Occurrences of values that did not fit the cap (not deduplicated —
|
||||
/// remembering overflow values would defeat the memory bound).
|
||||
pub overflow: u64,
|
||||
}
|
||||
|
||||
/// One rule's aggregated hits across the whole input.
|
||||
#[derive(Debug, Clone, Serialize)]
|
||||
pub struct Finding {
|
||||
pub rule_id: String,
|
||||
pub severity: Severity,
|
||||
pub category: String,
|
||||
pub title: String,
|
||||
pub diagnosis: String,
|
||||
pub suggestion: String,
|
||||
pub count: u64,
|
||||
/// Earliest / latest timestamps among hits that carry one.
|
||||
pub first_seen: Option<DateTime<FixedOffset>>,
|
||||
pub last_seen: Option<DateTime<FixedOffset>>,
|
||||
/// Node labels of hits; label-less events are recorded as "-".
|
||||
pub nodes: BTreeSet<String>,
|
||||
/// The earliest `max_samples` hit events (timestamp-less ones sort last).
|
||||
pub samples: Vec<LogEvent>,
|
||||
/// evidence field name -> deduplicated values.
|
||||
pub evidence: BTreeMap<String, EvidenceValues>,
|
||||
/// `count < rule.min_count`: the report demotes this to low-confidence.
|
||||
pub below_min_count: bool,
|
||||
/// Causal folding (rustfs/backlog#1290): the root-cause finding this one
|
||||
/// collapsed into. Renderers fold it under that root instead of listing
|
||||
/// it flat; JSON keeps the full finding either way.
|
||||
pub collapsed_into: Option<String>,
|
||||
/// The symptom finding ids collapsed under this root (other direction).
|
||||
pub caused: Vec<String>,
|
||||
/// The rule's causal edges, carried for `finalize` — not report data.
|
||||
#[serde(skip)]
|
||||
pub implies_root_cause: Vec<String>,
|
||||
}
|
||||
|
||||
struct FindingAcc {
|
||||
finding: Finding,
|
||||
min_count: u64,
|
||||
evidence_fields: Vec<String>,
|
||||
}
|
||||
|
||||
pub struct FindingsCollector {
|
||||
max_samples: usize,
|
||||
accs: BTreeMap<usize, FindingAcc>,
|
||||
}
|
||||
|
||||
/// Sort key for sample selection: earliest first, timestamp-less last,
|
||||
/// (file, line) as the deterministic tiebreaker — line numbers alone are only
|
||||
/// unique within one file, so ties across inputs would be order-dependent.
|
||||
fn sample_key(event: &LogEvent) -> (bool, Option<DateTime<FixedOffset>>, &str, u64) {
|
||||
(event.timestamp.is_none(), event.timestamp, &event.source.file, event.source.line)
|
||||
}
|
||||
|
||||
/// `DateTime<FixedOffset>` orders by instant, so two equal instants carrying
|
||||
/// different offsets compare equal and `min`/`max` keep whichever arrived
|
||||
/// first — an input-order dependency that then leaks into the serialized
|
||||
/// RFC3339 offset. Break the tie on the offset itself to stay deterministic.
|
||||
fn earliest(a: DateTime<FixedOffset>, b: DateTime<FixedOffset>) -> DateTime<FixedOffset> {
|
||||
match a.cmp(&b) {
|
||||
std::cmp::Ordering::Less => a,
|
||||
std::cmp::Ordering::Greater => b,
|
||||
std::cmp::Ordering::Equal if a.offset().local_minus_utc() <= b.offset().local_minus_utc() => a,
|
||||
std::cmp::Ordering::Equal => b,
|
||||
}
|
||||
}
|
||||
|
||||
fn latest(a: DateTime<FixedOffset>, b: DateTime<FixedOffset>) -> DateTime<FixedOffset> {
|
||||
match a.cmp(&b) {
|
||||
std::cmp::Ordering::Greater => a,
|
||||
std::cmp::Ordering::Less => b,
|
||||
std::cmp::Ordering::Equal if a.offset().local_minus_utc() >= b.offset().local_minus_utc() => a,
|
||||
std::cmp::Ordering::Equal => b,
|
||||
}
|
||||
}
|
||||
|
||||
impl FindingsCollector {
|
||||
pub fn new(max_samples: usize) -> Self {
|
||||
Self {
|
||||
max_samples,
|
||||
accs: BTreeMap::new(),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn observe(&mut self, rule: &Rule, rule_idx: usize, event: &LogEvent) {
|
||||
let acc = self.accs.entry(rule_idx).or_insert_with(|| FindingAcc {
|
||||
finding: Finding {
|
||||
rule_id: rule.id.clone(),
|
||||
severity: rule.severity,
|
||||
category: rule.category.clone(),
|
||||
title: rule.title.clone(),
|
||||
diagnosis: rule.diagnosis.clone(),
|
||||
suggestion: rule.suggestion.clone(),
|
||||
count: 0,
|
||||
first_seen: None,
|
||||
last_seen: None,
|
||||
nodes: BTreeSet::new(),
|
||||
samples: Vec::new(),
|
||||
evidence: BTreeMap::new(),
|
||||
below_min_count: false,
|
||||
collapsed_into: None,
|
||||
caused: Vec::new(),
|
||||
implies_root_cause: rule.implies_root_cause.clone(),
|
||||
},
|
||||
min_count: rule.min_count,
|
||||
evidence_fields: rule.evidence_fields.clone(),
|
||||
});
|
||||
|
||||
let finding = &mut acc.finding;
|
||||
finding.count += 1;
|
||||
|
||||
if let Some(ts) = event.timestamp {
|
||||
finding.first_seen = Some(finding.first_seen.map_or(ts, |cur| earliest(cur, ts)));
|
||||
finding.last_seen = Some(finding.last_seen.map_or(ts, |cur| latest(cur, ts)));
|
||||
}
|
||||
|
||||
finding.nodes.insert(event.node.as_deref().unwrap_or("-").to_string());
|
||||
|
||||
// Bounded, order-independent sample selection: keep the earliest
|
||||
// `max_samples` events by (has-ts, ts, line).
|
||||
if self.max_samples > 0 {
|
||||
let key = sample_key(event);
|
||||
let full = finding.samples.len() >= self.max_samples;
|
||||
if !full || key < sample_key(finding.samples.last().expect("non-empty when full")) {
|
||||
let pos = finding.samples.partition_point(|s| sample_key(s) <= key);
|
||||
finding.samples.insert(pos, event.clone());
|
||||
finding.samples.truncate(self.max_samples);
|
||||
}
|
||||
}
|
||||
|
||||
for field in &acc.evidence_fields {
|
||||
if let Some(value) = event.field_display(field) {
|
||||
let slot = finding.evidence.entry(field.clone()).or_default();
|
||||
if slot.values.contains(&value) {
|
||||
continue;
|
||||
}
|
||||
slot.values.insert(value);
|
||||
// Keep a deterministic subset — the lexicographically smallest
|
||||
// EVIDENCE_VALUES_CAP distinct values — so the evidence set is
|
||||
// order-independent (previously it kept the first-10-by-arrival).
|
||||
if slot.values.len() > EVIDENCE_VALUES_CAP {
|
||||
if let Some(max) = slot.values.iter().next_back().cloned() {
|
||||
slot.values.remove(&max);
|
||||
}
|
||||
slot.overflow += 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Sorted: severity ascending (P0 first) -> count descending -> id.
|
||||
pub fn into_findings(self) -> Vec<Finding> {
|
||||
let mut findings: Vec<Finding> = self
|
||||
.accs
|
||||
.into_values()
|
||||
.map(|acc| {
|
||||
let mut finding = acc.finding;
|
||||
finding.below_min_count = finding.count < acc.min_count;
|
||||
finding
|
||||
})
|
||||
.collect();
|
||||
findings.sort_by(|a, b| {
|
||||
a.severity
|
||||
.cmp(&b.severity)
|
||||
.then(b.count.cmp(&a.count))
|
||||
.then(a.rule_id.cmp(&b.rule_id))
|
||||
});
|
||||
findings
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,354 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! Rule model, matching engine, and finding aggregation
|
||||
//! (rustfs/backlog#1285). The built-in seed rule library is LA-5
|
||||
//! (rustfs/backlog#1286).
|
||||
|
||||
mod engine;
|
||||
mod external;
|
||||
mod findings;
|
||||
mod model;
|
||||
mod rule_set;
|
||||
mod seed;
|
||||
|
||||
pub use engine::RuleEngine;
|
||||
pub use external::{EXTERNAL_RULES_SCHEMA_VERSION, ExternalRulesError, seed_rules_with_external};
|
||||
pub use findings::{EvidenceValues, Finding, FindingsCollector};
|
||||
pub use model::{Matcher, Rule, Severity};
|
||||
pub use rule_set::{RuleSet, RuleSetError};
|
||||
pub use seed::seed_rules;
|
||||
|
||||
/// The built-in rule library as a validated set. Seed rules are validated
|
||||
/// by construction; a failure here is a bug in the seed tree itself.
|
||||
pub fn seed_rule_set() -> RuleSet {
|
||||
RuleSet::new(seed_rules()).expect("seed rules must be valid")
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::model::{EventKind, LogEvent, LogLevel, SourceRef};
|
||||
use std::sync::Arc;
|
||||
|
||||
fn event(message: &str, level: Option<LogLevel>, line: u64) -> LogEvent {
|
||||
LogEvent {
|
||||
timestamp: None,
|
||||
level,
|
||||
target: Some("rustfs::scanner::io".to_string()),
|
||||
message: message.to_string(),
|
||||
fields: serde_json::Map::new(),
|
||||
source: SourceRef {
|
||||
file: Arc::from("rustfs.log"),
|
||||
line,
|
||||
},
|
||||
node: None,
|
||||
kind: EventKind::Json,
|
||||
}
|
||||
}
|
||||
|
||||
fn event_at(message: &str, ts: &str, line: u64) -> LogEvent {
|
||||
LogEvent {
|
||||
timestamp: Some(chrono::DateTime::parse_from_rfc3339(ts).expect("ts")),
|
||||
..event(message, Some(LogLevel::Error), line)
|
||||
}
|
||||
}
|
||||
|
||||
fn rule(id: &str, severity: Severity, matcher: Matcher) -> Rule {
|
||||
Rule {
|
||||
id: id.to_string(),
|
||||
severity,
|
||||
category: "disk".to_string(),
|
||||
title: format!("title {id}"),
|
||||
matcher,
|
||||
diagnosis: "diag".to_string(),
|
||||
suggestion: "fix".to_string(),
|
||||
evidence_fields: Vec::new(),
|
||||
min_count: 1,
|
||||
implies_root_cause: Vec::new(),
|
||||
anchors: Vec::new(),
|
||||
}
|
||||
}
|
||||
|
||||
fn engine_of(rules: Vec<Rule>) -> RuleEngine {
|
||||
RuleEngine::new(RuleSet::new(rules).expect("valid rules"))
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn every_matcher_kind_has_positive_and_negative_cases() {
|
||||
let ev = {
|
||||
let mut ev = event("erasure write quorum (required=8, achieved=5)", Some(LogLevel::Error), 1);
|
||||
ev.fields
|
||||
.insert("reason".to_string(), serde_json::Value::String("faulty_disk".into()));
|
||||
ev.fields.insert("achieved".to_string(), serde_json::json!(5));
|
||||
ev
|
||||
};
|
||||
let panic_ev = LogEvent {
|
||||
kind: EventKind::Panic,
|
||||
..event("thread 'main' panicked at src/main.rs:3:5: boom", Some(LogLevel::Error), 2)
|
||||
};
|
||||
|
||||
let cases: Vec<(Matcher, bool, bool)> = vec![
|
||||
// (matcher, matches ev, matches panic_ev)
|
||||
(Matcher::MessagePrefix("erasure write quorum (".into()), true, false),
|
||||
(Matcher::MessageContains("achieved=5".into()), true, false),
|
||||
(Matcher::MessageRegex(r"required=\d+".into()), true, false),
|
||||
(
|
||||
Matcher::FieldEquals {
|
||||
name: "reason".into(),
|
||||
value: "faulty_disk".into(),
|
||||
},
|
||||
true,
|
||||
false,
|
||||
),
|
||||
(Matcher::TargetPrefix("rustfs::scanner".into()), true, true),
|
||||
(Matcher::IsPanic, false, true),
|
||||
(Matcher::MinLevel(LogLevel::Warn), true, true),
|
||||
(
|
||||
Matcher::All(vec![
|
||||
Matcher::MinLevel(LogLevel::Error),
|
||||
Matcher::Any(vec![Matcher::IsPanic, Matcher::MessageContains("quorum".into())]),
|
||||
]),
|
||||
true,
|
||||
true,
|
||||
),
|
||||
(
|
||||
Matcher::Any(vec![Matcher::IsPanic, Matcher::MessagePrefix("no such".into())]),
|
||||
false,
|
||||
true,
|
||||
),
|
||||
];
|
||||
for (matcher, expect_ev, expect_panic) in cases {
|
||||
let engine = engine_of(vec![rule("r", Severity::P2Degraded, matcher.clone())]);
|
||||
assert_eq!(engine.matches(&ev).len() == 1, expect_ev, "matcher {matcher:?} vs ev");
|
||||
assert_eq!(engine.matches(&panic_ev).len() == 1, expect_panic, "matcher {matcher:?} vs panic");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn field_equals_coerces_numbers_via_display() {
|
||||
let mut ev = event("x", Some(LogLevel::Error), 1);
|
||||
ev.fields.insert("achieved".to_string(), serde_json::json!(1));
|
||||
let engine = engine_of(vec![rule(
|
||||
"n",
|
||||
Severity::P1Unavailable,
|
||||
Matcher::FieldEquals {
|
||||
name: "achieved".into(),
|
||||
value: "1".into(),
|
||||
},
|
||||
)]);
|
||||
assert_eq!(engine.matches(&ev).len(), 1);
|
||||
|
||||
// A JSON string "1" also matches (field_display unquotes strings)…
|
||||
ev.fields.insert("achieved".to_string(), serde_json::json!("1"));
|
||||
assert_eq!(engine.matches(&ev).len(), 1);
|
||||
// …but a quoted value does not accidentally match.
|
||||
ev.fields.insert("achieved".to_string(), serde_json::json!("\"1\""));
|
||||
assert_eq!(engine.matches(&ev).len(), 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn min_level_requires_a_level() {
|
||||
let engine = engine_of(vec![rule("lvl", Severity::P4Info, Matcher::MinLevel(LogLevel::Warn))]);
|
||||
assert_eq!(engine.matches(&event("x", Some(LogLevel::Error), 1)).len(), 1);
|
||||
assert_eq!(engine.matches(&event("x", Some(LogLevel::Info), 1)).len(), 0);
|
||||
assert_eq!(engine.matches(&event("x", None, 1)).len(), 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rule_set_reports_all_problems_at_once() {
|
||||
let rules = vec![
|
||||
rule("dup", Severity::P4Info, Matcher::IsPanic),
|
||||
rule("dup", Severity::P4Info, Matcher::MessageRegex("[unclosed".into())),
|
||||
rule("", Severity::P4Info, Matcher::All(vec![])),
|
||||
];
|
||||
let err = RuleSet::new(rules).expect_err("invalid");
|
||||
let RuleSetError::Multiple(problems) = err else {
|
||||
panic!("expected Multiple");
|
||||
};
|
||||
assert_eq!(problems.len(), 4, "{problems:?}"); // dup id + bad regex + empty id + empty group
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn one_event_can_hit_several_rules() {
|
||||
let engine = engine_of(vec![
|
||||
rule("a", Severity::P2Degraded, Matcher::MessageContains("quorum".into())),
|
||||
rule("b", Severity::P1Unavailable, Matcher::MinLevel(LogLevel::Error)),
|
||||
]);
|
||||
let hits = engine.matches(&event("write quorum lost", Some(LogLevel::Error), 1));
|
||||
assert_eq!(hits, vec![0, 1]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn collector_is_order_independent() {
|
||||
let r = rule("q", Severity::P1Unavailable, Matcher::MessageContains("quorum".into()));
|
||||
let events = vec![
|
||||
event_at("quorum e1", "2026-07-15T03:00:00+08:00", 10),
|
||||
event_at("quorum e2", "2026-07-15T01:00:00+08:00", 20),
|
||||
event_at("quorum e3", "2026-07-15T02:00:00+08:00", 30),
|
||||
event_at("quorum e4", "2026-07-15T04:00:00+08:00", 40),
|
||||
event("quorum no-ts", Some(LogLevel::Error), 50),
|
||||
];
|
||||
|
||||
let collect = |order: &[usize]| {
|
||||
let mut collector = FindingsCollector::new(3);
|
||||
for &i in order {
|
||||
collector.observe(&r, 0, &events[i]);
|
||||
}
|
||||
serde_json::to_string(&collector.into_findings()).expect("json")
|
||||
};
|
||||
|
||||
let forward = collect(&[0, 1, 2, 3, 4]);
|
||||
let reverse = collect(&[4, 3, 2, 1, 0]);
|
||||
let shuffled = collect(&[2, 4, 0, 3, 1]);
|
||||
assert_eq!(forward, reverse);
|
||||
assert_eq!(forward, shuffled);
|
||||
|
||||
let findings: Vec<Finding> = {
|
||||
let mut collector = FindingsCollector::new(3);
|
||||
for ev in &events {
|
||||
collector.observe(&r, 0, ev);
|
||||
}
|
||||
collector.into_findings()
|
||||
};
|
||||
let f = &findings[0];
|
||||
assert_eq!(f.count, 5);
|
||||
assert_eq!(f.first_seen.expect("first").to_rfc3339(), "2026-07-15T01:00:00+08:00");
|
||||
assert_eq!(f.last_seen.expect("last").to_rfc3339(), "2026-07-15T04:00:00+08:00");
|
||||
let sample_messages: Vec<_> = f.samples.iter().map(|s| s.message.as_str()).collect();
|
||||
assert_eq!(sample_messages, vec!["quorum e2", "quorum e3", "quorum e1"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn sample_ties_across_files_are_order_independent() {
|
||||
let r = rule("q", Severity::P1Unavailable, Matcher::MessageContains("quorum".into()));
|
||||
// Same timestamp and same line number in two different files: only
|
||||
// the file name can break the tie deterministically.
|
||||
let mk = |file: &str, msg: &str| {
|
||||
let mut ev = event_at(msg, "2026-07-15T03:00:00+08:00", 1);
|
||||
ev.source.file = Arc::from(file);
|
||||
ev
|
||||
};
|
||||
let a = mk("node1/rustfs.log", "quorum from node1");
|
||||
let b = mk("node2/rustfs.log", "quorum from node2");
|
||||
|
||||
let selected = |events: &[&LogEvent]| {
|
||||
let mut collector = FindingsCollector::new(1);
|
||||
for ev in events {
|
||||
collector.observe(&r, 0, ev);
|
||||
}
|
||||
collector.into_findings()[0].samples[0].message.clone()
|
||||
};
|
||||
assert_eq!(selected(&[&a, &b]), selected(&[&b, &a]));
|
||||
assert_eq!(selected(&[&a, &b]), "quorum from node1");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn evidence_values_are_capped_with_overflow() {
|
||||
let mut r = rule("e", Severity::P2Degraded, Matcher::MessageContains("x".into()));
|
||||
r.evidence_fields = vec!["disk".to_string()];
|
||||
let mut collector = FindingsCollector::new(3);
|
||||
for i in 0..11 {
|
||||
let mut ev = event("x", Some(LogLevel::Error), i);
|
||||
ev.fields
|
||||
.insert("disk".to_string(), serde_json::Value::String(format!("/data/disk{i}")));
|
||||
collector.observe(&r, 0, &ev);
|
||||
}
|
||||
let findings = collector.into_findings();
|
||||
let evidence = &findings[0].evidence["disk"];
|
||||
assert_eq!(evidence.values.len(), 10);
|
||||
assert_eq!(evidence.overflow, 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn evidence_cap_keeps_smallest_subset_regardless_of_order() {
|
||||
let mut r = rule("e", Severity::P2Degraded, Matcher::MessageContains("x".into()));
|
||||
r.evidence_fields = vec!["disk".to_string()];
|
||||
let retained = |order: &[u64]| -> Vec<String> {
|
||||
let mut c = FindingsCollector::new(3);
|
||||
for &i in order {
|
||||
let mut ev = event("x", Some(LogLevel::Error), i);
|
||||
// Zero-padded so lexicographic order matches numeric order.
|
||||
ev.fields
|
||||
.insert("disk".to_string(), serde_json::Value::String(format!("/d/{i:02}")));
|
||||
c.observe(&r, 0, &ev);
|
||||
}
|
||||
c.into_findings()[0].evidence["disk"].values.iter().cloned().collect()
|
||||
};
|
||||
let forward: Vec<u64> = (0..15).collect();
|
||||
let reverse: Vec<u64> = (0..15).rev().collect();
|
||||
let shuffled: Vec<u64> = vec![7, 3, 14, 0, 9, 2, 11, 5, 13, 1, 8, 4, 12, 6, 10];
|
||||
let a = retained(&forward);
|
||||
assert_eq!(a, retained(&reverse), "reverse order changed the retained set");
|
||||
assert_eq!(a, retained(&shuffled), "shuffled order changed the retained set");
|
||||
// The kept subset is the lexicographically smallest 10, deterministically.
|
||||
assert_eq!(a, (0..10).map(|i| format!("/d/{i:02}")).collect::<Vec<_>>());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn below_min_count_is_flagged() {
|
||||
let mut r = rule("m", Severity::P3ClientSide, Matcher::MessageContains("denied".into()));
|
||||
r.min_count = 5;
|
||||
let mut collector = FindingsCollector::new(3);
|
||||
for i in 0..3 {
|
||||
collector.observe(&r, 0, &event("denied", Some(LogLevel::Warn), i));
|
||||
}
|
||||
assert!(collector.into_findings()[0].below_min_count);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn findings_sort_by_severity_then_count() {
|
||||
let rules = [
|
||||
rule("p2", Severity::P2Degraded, Matcher::MessageContains("a".into())),
|
||||
rule("p0", Severity::P0DataRisk, Matcher::MessageContains("b".into())),
|
||||
rule("p1-low", Severity::P1Unavailable, Matcher::MessageContains("c".into())),
|
||||
rule("p1-high", Severity::P1Unavailable, Matcher::MessageContains("d".into())),
|
||||
];
|
||||
let mut collector = FindingsCollector::new(3);
|
||||
let feed = |collector: &mut FindingsCollector, idx: usize, msg: &str, times: u64| {
|
||||
for i in 0..times {
|
||||
collector.observe(&rules[idx], idx, &event(msg, Some(LogLevel::Error), i));
|
||||
}
|
||||
};
|
||||
feed(&mut collector, 0, "a", 100);
|
||||
feed(&mut collector, 1, "b", 1);
|
||||
feed(&mut collector, 2, "c", 2);
|
||||
feed(&mut collector, 3, "d", 9);
|
||||
let ids: Vec<_> = collector.into_findings().into_iter().map(|f| f.rule_id).collect();
|
||||
assert_eq!(ids, vec!["p0", "p1-high", "p1-low", "p2"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rule_json_round_trip_and_schema_shape() {
|
||||
let json = r#"{
|
||||
"id": "demo-rule",
|
||||
"severity": "p1_unavailable",
|
||||
"category": "disk",
|
||||
"title": "示例",
|
||||
"matcher": { "all": [ { "message_prefix": "erasure write quorum (" }, { "min_level": "WARN" } ] },
|
||||
"diagnosis": "…",
|
||||
"suggestion": "…",
|
||||
"evidence_fields": ["required", "achieved"],
|
||||
"anchors": ["erasure write quorum ("]
|
||||
}"#;
|
||||
let parsed: Rule = serde_json::from_str(json).expect("hand-written JSON stays parseable");
|
||||
assert_eq!(parsed.id, "demo-rule");
|
||||
assert_eq!(parsed.severity, Severity::P1Unavailable);
|
||||
assert_eq!(parsed.min_count, 1, "min_count defaults");
|
||||
assert!(matches!(&parsed.matcher, Matcher::All(inner) if inner.len() == 2));
|
||||
|
||||
let round: Rule = serde_json::from_str(&serde_json::to_string(&parsed).expect("ser")).expect("de");
|
||||
assert_eq!(serde_json::to_value(&round).expect("v"), serde_json::to_value(&parsed).expect("v"));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,120 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! Rule data model.
|
||||
//!
|
||||
//! Rules are owned values (not `&'static` graphs) on purpose: the Phase-2
|
||||
//! external rule file deserializes into these exact types with zero
|
||||
//! impedance (see rustfs/backlog#1281 design decision 3).
|
||||
|
||||
use crate::model::LogLevel;
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
/// Severity == what the operator should do about it. `Ord` puts the most
|
||||
/// severe first, which is the report order.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum Severity {
|
||||
/// P0: possible data loss / unreconstructable objects.
|
||||
P0DataRisk,
|
||||
/// P1: service unavailable — cannot write, cannot start, process crash.
|
||||
P1Unavailable,
|
||||
/// P2: degraded — self-healing, partial disk/node failure, still serving.
|
||||
P2Degraded,
|
||||
/// P3: client/environment side — credentials, quota, policy.
|
||||
P3ClientSide,
|
||||
/// P4: informational — state-machine conflicts, ignorable noise.
|
||||
P4Info,
|
||||
}
|
||||
|
||||
impl Severity {
|
||||
/// Human label used by report renderers.
|
||||
pub fn label(&self) -> &'static str {
|
||||
match self {
|
||||
Severity::P0DataRisk => "P0 数据风险",
|
||||
Severity::P1Unavailable => "P1 服务不可用",
|
||||
Severity::P2Degraded => "P2 降级",
|
||||
Severity::P3ClientSide => "P3 客户端侧",
|
||||
Severity::P4Info => "P4 提示",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Match condition tree. Semantics (see rustfs/backlog#1285):
|
||||
///
|
||||
/// - level `None` never satisfies [`Matcher::MinLevel`];
|
||||
/// - target `None` never satisfies [`Matcher::TargetPrefix`];
|
||||
/// - [`Matcher::FieldEquals`] compares via `LogEvent::field_display`, so
|
||||
/// the numeric field `{"n":3}` equals the string value `"3"`.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum Matcher {
|
||||
/// `event.message` starts with the given text.
|
||||
MessagePrefix(String),
|
||||
/// `event.message` contains the given text.
|
||||
MessageContains(String),
|
||||
/// Regex over `event.message`. Compilation is validated at
|
||||
/// `RuleSet::new` time (fail fast), not per event.
|
||||
MessageRegex(String),
|
||||
/// `fields[name]` rendered via `field_display` equals `value`.
|
||||
FieldEquals {
|
||||
name: String,
|
||||
value: String,
|
||||
},
|
||||
/// `event.target` starts with the given prefix.
|
||||
TargetPrefix(String),
|
||||
/// `event.kind == EventKind::Panic`.
|
||||
IsPanic,
|
||||
/// `event.level >= level` (`None` level never matches).
|
||||
MinLevel(LogLevel),
|
||||
All(Vec<Matcher>),
|
||||
Any(Vec<Matcher>),
|
||||
}
|
||||
|
||||
fn default_min_count() -> u64 {
|
||||
1
|
||||
}
|
||||
|
||||
/// One diagnosis rule. Text fields are operator-facing Chinese; `id` and
|
||||
/// `category` are stable kebab-case identifiers.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct Rule {
|
||||
/// Unique kebab-case id, e.g. "ec-write-quorum".
|
||||
pub id: String,
|
||||
pub severity: Severity,
|
||||
/// Grouping: "disk" | "erasure" | "quorum" | "network" | "lock" | "heal"
|
||||
/// | "scanner" | "iam" | "startup" | "capacity" | "ops" | "process".
|
||||
pub category: String,
|
||||
/// One-line report title.
|
||||
pub title: String,
|
||||
pub matcher: Matcher,
|
||||
/// What this pattern usually means (1-3 sentences).
|
||||
pub diagnosis: String,
|
||||
/// Actionable advice (1-3 sentences).
|
||||
pub suggestion: String,
|
||||
/// Field names collected from matching events as evidence.
|
||||
#[serde(default)]
|
||||
pub evidence_fields: Vec<String>,
|
||||
/// Findings with fewer hits are demoted to the low-confidence section.
|
||||
#[serde(default = "default_min_count")]
|
||||
pub min_count: u64,
|
||||
/// Phase-2 root-cause folding (rustfs/backlog#1290); stored only for now.
|
||||
#[serde(default)]
|
||||
pub implies_root_cause: Vec<String>,
|
||||
/// CI guard (rustfs/backlog#1289): these texts must exist verbatim in
|
||||
/// the rustfs source tree. Rules matching on message text carry at
|
||||
/// least one anchor; pure field/panic rules may have none.
|
||||
#[serde(default)]
|
||||
pub anchors: Vec<String>,
|
||||
}
|
||||
@@ -0,0 +1,119 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! Validated rule collection. Validation reports *all* problems at once,
|
||||
//! not just the first — a rule author fixing an external file should not
|
||||
//! have to iterate error by error.
|
||||
|
||||
use super::model::{Matcher, Rule};
|
||||
use std::collections::BTreeSet;
|
||||
use thiserror::Error;
|
||||
|
||||
#[derive(Debug, Error)]
|
||||
pub enum RuleSetError {
|
||||
#[error("duplicate rule id '{0}'")]
|
||||
DuplicateId(String),
|
||||
#[error("rule has an empty id")]
|
||||
EmptyId,
|
||||
#[error("rule '{rule}': bad regex: {source}")]
|
||||
BadRegex {
|
||||
rule: String,
|
||||
#[source]
|
||||
source: regex::Error,
|
||||
},
|
||||
#[error("rule '{0}': empty all/any matcher group")]
|
||||
EmptyGroup(String),
|
||||
#[error("rule '{rule}': bad anchor {anchor:?}: {reason}")]
|
||||
BadAnchor {
|
||||
rule: String,
|
||||
anchor: String,
|
||||
reason: &'static str,
|
||||
},
|
||||
#[error("{} rule set problem(s):\n{}", .0.len(), .0.iter().map(|e| format!(" - {e}")).collect::<Vec<_>>().join("\n"))]
|
||||
Multiple(Vec<RuleSetError>),
|
||||
}
|
||||
|
||||
#[derive(Debug)]
|
||||
pub struct RuleSet {
|
||||
rules: Vec<Rule>,
|
||||
}
|
||||
|
||||
impl RuleSet {
|
||||
/// Validates: ids unique and non-empty, every `MessageRegex` compiles,
|
||||
/// every `All`/`Any` group is non-empty. Returns all violations.
|
||||
pub fn new(rules: Vec<Rule>) -> Result<Self, RuleSetError> {
|
||||
let mut problems = Vec::new();
|
||||
let mut seen = BTreeSet::new();
|
||||
for rule in &rules {
|
||||
if rule.id.trim().is_empty() {
|
||||
problems.push(RuleSetError::EmptyId);
|
||||
} else if !seen.insert(rule.id.clone()) {
|
||||
problems.push(RuleSetError::DuplicateId(rule.id.clone()));
|
||||
}
|
||||
validate_matcher(&rule.matcher, &rule.id, &mut problems);
|
||||
for anchor in &rule.anchors {
|
||||
// Anchors feed a TSV dump consumed line-by-line by the CI
|
||||
// guard (rustfs/backlog#1289); short anchors have no
|
||||
// discriminating power and would always grep-match.
|
||||
let reason = if anchor.contains('\t') || anchor.contains('\n') {
|
||||
Some("must not contain tabs or newlines")
|
||||
} else if anchor.trim().is_empty() {
|
||||
Some("must not be blank")
|
||||
} else if anchor.len() < 8 {
|
||||
Some("must be at least 8 bytes")
|
||||
} else {
|
||||
None
|
||||
};
|
||||
if let Some(reason) = reason {
|
||||
problems.push(RuleSetError::BadAnchor {
|
||||
rule: rule.id.clone(),
|
||||
anchor: anchor.clone(),
|
||||
reason,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
if problems.is_empty() {
|
||||
Ok(Self { rules })
|
||||
} else {
|
||||
Err(RuleSetError::Multiple(problems))
|
||||
}
|
||||
}
|
||||
|
||||
pub fn rules(&self) -> &[Rule] {
|
||||
&self.rules
|
||||
}
|
||||
}
|
||||
|
||||
fn validate_matcher(matcher: &Matcher, rule_id: &str, problems: &mut Vec<RuleSetError>) {
|
||||
match matcher {
|
||||
Matcher::MessageRegex(re) => {
|
||||
if let Err(source) = regex::Regex::new(re) {
|
||||
problems.push(RuleSetError::BadRegex {
|
||||
rule: rule_id.to_string(),
|
||||
source,
|
||||
});
|
||||
}
|
||||
}
|
||||
Matcher::All(inner) | Matcher::Any(inner) => {
|
||||
if inner.is_empty() {
|
||||
problems.push(RuleSetError::EmptyGroup(rule_id.to_string()));
|
||||
}
|
||||
for m in inner {
|
||||
validate_matcher(m, rule_id, problems);
|
||||
}
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,90 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! Capacity rules.
|
||||
|
||||
use super::super::model::{Rule, Severity::*};
|
||||
use super::{any, base, contains, prefix, strings};
|
||||
|
||||
pub(super) fn rules() -> Vec<Rule> {
|
||||
vec![
|
||||
Rule {
|
||||
evidence_fields: strings(["disk"]),
|
||||
anchors: strings(["Disk full"]),
|
||||
..base(
|
||||
"disk-full",
|
||||
P1Unavailable,
|
||||
"capacity",
|
||||
"盘写满",
|
||||
any([
|
||||
contains("Disk full"),
|
||||
contains("drive path full"),
|
||||
contains("No space left on device"),
|
||||
]),
|
||||
"盘写满。",
|
||||
"清理/扩容;检查 ILM 是否按预期转储;结合 heal-orphan 泄漏 finding。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
anchors: strings(["Storage reached its minimum free drive threshold"]),
|
||||
..base(
|
||||
"min-free-threshold",
|
||||
P1Unavailable,
|
||||
"capacity",
|
||||
"触及最小空闲阈值,拒绝新写",
|
||||
contains("Storage reached its minimum free drive threshold"),
|
||||
"触及最小空闲阈值,拒绝新写(保护机制,先于物理满盘)。",
|
||||
"扩容或清理;这解释「df 有空间但写入报满」。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
anchors: strings(["Storage resources are insufficient for the"]),
|
||||
..base(
|
||||
"insufficient-storage",
|
||||
P1Unavailable,
|
||||
"capacity",
|
||||
"存储资源不足",
|
||||
prefix("Storage resources are insufficient for the"),
|
||||
"存储资源不足(读/写)。",
|
||||
"同 disk-full 排查。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
evidence_fields: strings(["current", "limit", "operation"]),
|
||||
anchors: strings(["Bucket quota exceeded"]),
|
||||
..base(
|
||||
"bucket-quota-exceeded",
|
||||
P3ClientSide,
|
||||
"capacity",
|
||||
"bucket 配额超限",
|
||||
prefix("Bucket quota exceeded"),
|
||||
"bucket 配额超限(管理面设置,非集群故障)。",
|
||||
"调整配额或清理该桶。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
evidence_fields: strings(["required", "target_free"]),
|
||||
anchors: strings(["insufficient target pool capacity"]),
|
||||
..base(
|
||||
"decom-capacity-insufficient",
|
||||
P1Unavailable,
|
||||
"capacity",
|
||||
"pool 下线目标容量不足",
|
||||
contains("insufficient target pool capacity"),
|
||||
"pool 下线迁移目标容量不足,decommission 无法开始。",
|
||||
"先扩容目标 pool。",
|
||||
)
|
||||
},
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,175 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! Disk health rules.
|
||||
|
||||
use super::super::model::{Matcher, Rule, Severity::*};
|
||||
use super::{all, any, base, contains, field, strings};
|
||||
|
||||
pub(super) fn rules() -> Vec<Rule> {
|
||||
vec![
|
||||
Rule {
|
||||
evidence_fields: strings(["disk", "endpoint"]),
|
||||
anchors: strings(["Disk health check marked disk faulty"]),
|
||||
..base(
|
||||
"disk-marked-faulty",
|
||||
P2Degraded,
|
||||
"disk",
|
||||
"磁盘被标记为 faulty",
|
||||
any([
|
||||
field("reason", "faulty_disk"),
|
||||
contains("Disk health check marked disk faulty"),
|
||||
]),
|
||||
"某盘连续健康检查失败被标记 faulty,后续请求短路避开。",
|
||||
"检查该盘 SMART/内核日志(dmesg)与挂载状态;确认是否伴随 quorum 类 finding(级联)。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
evidence_fields: strings(["disk"]),
|
||||
anchors: strings(["rejected operation because disk is marked faulty"]),
|
||||
..base(
|
||||
"disk-faulty-rejected",
|
||||
P2Degraded,
|
||||
"disk",
|
||||
"请求被 faulty 盘保护性拒绝",
|
||||
any([
|
||||
field("reason", "disk_marked_faulty"),
|
||||
contains("rejected operation because disk is marked faulty"),
|
||||
]),
|
||||
"请求命中已标记 faulty 的盘被拒(保护性短路)。",
|
||||
"与 disk-marked-faulty 同源,处理根因盘。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
evidence_fields: strings(["peer", "host"]),
|
||||
anchors: strings(["Remote peer marked faulty", "Remote peer health check failed"]),
|
||||
..base(
|
||||
"remote-peer-faulty",
|
||||
P2Degraded,
|
||||
"disk",
|
||||
"远端节点被标记 faulty",
|
||||
any([
|
||||
contains("Remote peer marked faulty"),
|
||||
contains("Remote peer health check failed"),
|
||||
]),
|
||||
"远端节点网络不可达/健康检查失败,被标记 faulty。",
|
||||
"检查节点间网络连通与目标节点进程存活。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
anchors: strings(["reporting peer disks offline after consecutive storage_info failures"]),
|
||||
..base(
|
||||
"peer-disks-offline",
|
||||
P2Degraded,
|
||||
"disk",
|
||||
"peer 磁盘被整体判定离线",
|
||||
contains("reporting peer disks offline after consecutive storage_info failures"),
|
||||
"对某 peer 连续 storage_info 失败,判定其磁盘整体离线。",
|
||||
"检查该 peer 节点存活与 RPC 端口可达。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
anchors: strings(["drive is faulty"]),
|
||||
..base(
|
||||
"drive-faulty-error",
|
||||
P2Degraded,
|
||||
"disk",
|
||||
"存储路径出现盘级 faulty 错误",
|
||||
any([
|
||||
contains("Faulty disk"),
|
||||
contains("drive is faulty"),
|
||||
contains("Faulty remote disk"),
|
||||
contains("remote drive is faulty"),
|
||||
]),
|
||||
"盘级 faulty 错误出现在存储路径错误链。",
|
||||
"定位具体盘(结合 samples 的 fields),检查硬件。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
anchors: strings(["Unformatted disk"]),
|
||||
..base(
|
||||
"unformatted-disk",
|
||||
P2Degraded,
|
||||
"disk",
|
||||
"存在未格式化的盘",
|
||||
contains("Unformatted disk"),
|
||||
"盘未格式化(新盘/format.json 缺失)。",
|
||||
"确认是否新换盘待 heal;检查该盘挂载点是否指向了空目录。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
anchors: strings(["disk access denied"]),
|
||||
..base(
|
||||
"disk-access-denied",
|
||||
P1Unavailable,
|
||||
"disk",
|
||||
"盘目录权限不可访问",
|
||||
all([
|
||||
Matcher::MinLevel(crate::model::LogLevel::Warn),
|
||||
any([contains("disk access denied"), contains("Disk access denied")]),
|
||||
]),
|
||||
"盘目录权限不可访问,该盘等效离线。",
|
||||
"检查数据目录属主/权限与 SELinux/AppArmor。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
anchors: strings(["inconsistent drive found"]),
|
||||
..base(
|
||||
"inconsistent-drive",
|
||||
P2Degraded,
|
||||
"disk",
|
||||
"盘上格式与集群预期不一致",
|
||||
contains("inconsistent drive found"),
|
||||
"盘上格式/成员信息与集群预期不一致(错插盘/复用旧盘)。",
|
||||
"核对该盘 format.json 与集群拓扑。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
anchors: strings(["too many open files"]),
|
||||
..base(
|
||||
"fd-exhausted",
|
||||
P1Unavailable,
|
||||
"disk",
|
||||
"文件描述符耗尽",
|
||||
contains("too many open files"),
|
||||
"文件描述符耗尽,读写随机失败。",
|
||||
"提高 `ulimit -n`(systemd 需配 LimitNOFILE),排查 fd 泄漏。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
anchors: strings(["does not support O_DIRECT"]),
|
||||
..base(
|
||||
"odirect-unsupported",
|
||||
P1Unavailable,
|
||||
"disk",
|
||||
"后端文件系统不支持 O_DIRECT",
|
||||
contains("does not support O_DIRECT"),
|
||||
"后端文件系统不支持 O_DIRECT(常见:tmpfs/部分网络盘)。",
|
||||
"更换数据目录所在文件系统。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
anchors: strings(["Rename across devices not allowed"]),
|
||||
..base(
|
||||
"rename-across-devices",
|
||||
P1Unavailable,
|
||||
"disk",
|
||||
"数据目录跨设备,原子 rename 不可用",
|
||||
contains("Rename across devices not allowed"),
|
||||
"数据目录跨设备(挂载布局错误),原子 rename 不可用。",
|
||||
"确保每个盘路径整体位于单一挂载点。",
|
||||
)
|
||||
},
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,113 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! Erasure-coding / bitrot / data-integrity rules.
|
||||
|
||||
use super::super::model::{Rule, Severity::*};
|
||||
use super::{any, base, contains, prefix, strings};
|
||||
|
||||
const QUORUM_ROOT_CAUSES: [&str; 4] = ["disk-marked-faulty", "remote-peer-faulty", "peer-disks-offline", "disk-full"];
|
||||
|
||||
pub(super) fn rules() -> Vec<Rule> {
|
||||
vec![
|
||||
Rule {
|
||||
evidence_fields: strings(["bucket", "object", "disk"]),
|
||||
anchors: strings(["bitrot hash mismatch", "bitrot shard file size mismatch"]),
|
||||
..base(
|
||||
"bitrot-detected",
|
||||
P1Unavailable,
|
||||
"erasure",
|
||||
"检测到分片位腐烂(bitrot)",
|
||||
any([
|
||||
contains("bitrot hash mismatch"),
|
||||
contains("bitrot checksum verification failed"),
|
||||
contains("bitrot shard file size mismatch"),
|
||||
]),
|
||||
"分片位腐烂(校验和/长度不符),读到损坏数据。",
|
||||
"触发 heal;若集中于单盘,按坏盘处理。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
anchors: strings(["short shard read: got"]),
|
||||
..base(
|
||||
"short-shard-read",
|
||||
P2Degraded,
|
||||
"erasure",
|
||||
"分片读取长度不足",
|
||||
contains("short shard read: got"),
|
||||
"分片读取长度不足(截断/损坏)。",
|
||||
"结合 bitrot 与盘健康 finding 定位坏盘。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
evidence_fields: strings(["bucket", "object"]),
|
||||
anchors: strings(["cannot reconstruct with available shards"]),
|
||||
..base(
|
||||
"heal-cannot-reconstruct",
|
||||
P0DataRisk,
|
||||
"erasure",
|
||||
"对象无法用现存分片重建",
|
||||
contains("cannot reconstruct with available shards"),
|
||||
"存活分片少于数据分片,对象不可重建,存在数据丢失风险。",
|
||||
"立即停止换盘/清盘类操作,清点离线盘并尽量恢复上线,再评估受影响对象清单。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
evidence_fields: strings(["required", "achieved", "failed", "offline-disks", "dominant-error"]),
|
||||
anchors: strings(["erasure write quorum"]),
|
||||
implies_root_cause: strings(QUORUM_ROOT_CAUSES),
|
||||
..base(
|
||||
"ec-write-quorum",
|
||||
P1Unavailable,
|
||||
"erasure",
|
||||
"写仲裁不足,写入被拒",
|
||||
contains("erasure write quorum"),
|
||||
"在线盘数低于写仲裁阈值,写入被拒。",
|
||||
"此为级联症状,排查同时段 disk/peer 类 P2 finding 定位根因盘或节点。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
evidence_fields: strings(["bucket", "object"]),
|
||||
anchors: strings(["erasure read quorum", "reduce_read_quorum_errs"]),
|
||||
implies_root_cause: strings(QUORUM_ROOT_CAUSES),
|
||||
..base(
|
||||
"ec-read-quorum",
|
||||
P1Unavailable,
|
||||
"erasure",
|
||||
"读仲裁不足,读取失败",
|
||||
any([contains("erasure read quorum"), prefix("reduce_read_quorum_errs:")]),
|
||||
"读仲裁不足,对象读取失败。",
|
||||
"同 ec-write-quorum,先恢复离线盘/节点。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
evidence_fields: strings(["bucket", "object"]),
|
||||
anchors: strings(["part missing or corrupt"]),
|
||||
..base(
|
||||
"file-corrupted",
|
||||
P1Unavailable,
|
||||
"erasure",
|
||||
"对象数据或 xl.meta 损坏",
|
||||
any([
|
||||
contains("part missing or corrupt"),
|
||||
contains("file is corrupted"),
|
||||
contains("File is corrupted"),
|
||||
contains("outdated XL meta"),
|
||||
]),
|
||||
"对象数据或 xl.meta 损坏/过期。",
|
||||
"对受影响对象触发 heal;批量出现时按 bitrot/坏盘路径排查。",
|
||||
)
|
||||
},
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,116 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! Self-heal rules.
|
||||
|
||||
use super::super::model::{Rule, Severity::*};
|
||||
use super::{any, base, contains, prefix, strings};
|
||||
|
||||
pub(super) fn rules() -> Vec<Rule> {
|
||||
vec![
|
||||
Rule {
|
||||
evidence_fields: strings(["bucket", "object"]),
|
||||
anchors: strings(["has no data_dir, cannot heal object data"]),
|
||||
..base(
|
||||
"heal-no-datadir",
|
||||
P0DataRisk,
|
||||
"heal",
|
||||
"元数据缺 data_dir,对象数据无法自愈",
|
||||
contains("has no data_dir, cannot heal object data"),
|
||||
"最新元数据缺 data_dir,对象数据无法自愈,存在数据不可恢复风险。",
|
||||
"保留现场,收集对象 versionId 与 xl.meta 上报。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
anchors: strings(["all drives had write errors, unable to heal"]),
|
||||
..base(
|
||||
"heal-all-writes-failed",
|
||||
P0DataRisk,
|
||||
"heal",
|
||||
"heal 结果无法写入任何盘",
|
||||
contains("all drives had write errors, unable to heal"),
|
||||
"heal 结果无法写入任何盘(全盘故障/满盘)。",
|
||||
"先解决容量/盘故障 finding,再重跑 heal。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
anchors: strings(["all healed data rename attempts failed"]),
|
||||
..base(
|
||||
"heal-rename-failed",
|
||||
P1Unavailable,
|
||||
"heal",
|
||||
"heal 数据落位失败",
|
||||
contains("all healed data rename attempts failed"),
|
||||
"heal 数据落位(rename)全部失败。",
|
||||
"检查盘写权限与空间。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
anchors: strings(["failed to regenerate recoverable xl.meta"]),
|
||||
..base(
|
||||
"heal-xlmeta-regen-failed",
|
||||
P1Unavailable,
|
||||
"heal",
|
||||
"可恢复 xl.meta 重建失败",
|
||||
contains("failed to regenerate recoverable xl.meta"),
|
||||
"可恢复 xl.meta 重建失败。",
|
||||
"收集对象路径上报;检查该盘可写性。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
anchors: strings(["create_bitrot_writer"]),
|
||||
min_count: 3,
|
||||
..base(
|
||||
"heal-writer-create-failed",
|
||||
P2Degraded,
|
||||
"heal",
|
||||
"heal 写入器创建失败",
|
||||
contains("create_bitrot_writer"),
|
||||
"heal 写入器创建失败并跳过部分盘。",
|
||||
"检查对应盘状态。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
anchors: strings(["orphan data-dir reclaim failed"]),
|
||||
..base(
|
||||
"heal-orphan-reclaim-failed",
|
||||
P3ClientSide,
|
||||
"heal",
|
||||
"孤儿数据目录清理失败",
|
||||
any([
|
||||
contains("orphan data-dir reclaim failed"),
|
||||
contains("Heal remote data-dir cleanup failed"),
|
||||
]),
|
||||
"孤儿数据目录清理失败(空间泄漏隐患,非紧急)。",
|
||||
"观察容量;持续出现时排查对应盘。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
anchors: strings(["Heal task execution failed", "Heal manager is not running"]),
|
||||
..base(
|
||||
"heal-task-failure",
|
||||
P2Degraded,
|
||||
"heal",
|
||||
"heal 任务调度/执行失败",
|
||||
any([
|
||||
prefix("Heal task timeout"),
|
||||
prefix("Heal task execution failed"),
|
||||
contains("Heal manager is not running"),
|
||||
]),
|
||||
"heal 任务调度/执行层故障。",
|
||||
"检查 heal 后台服务状态与资源压力。",
|
||||
)
|
||||
},
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,124 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! IAM / auth / signature rules — mostly client-side (P3) with burst
|
||||
//! thresholds so isolated client mistakes don't clutter the report.
|
||||
|
||||
use super::super::model::{Matcher, Rule, Severity::*};
|
||||
use super::{all, any, base, contains, prefix, strings};
|
||||
use crate::model::LogLevel;
|
||||
|
||||
pub(super) fn rules() -> Vec<Rule> {
|
||||
vec![
|
||||
Rule {
|
||||
anchors: strings(["SignatureDoesNotMatch"]),
|
||||
min_count: 10,
|
||||
..base(
|
||||
"client-signature-mismatch",
|
||||
P3ClientSide,
|
||||
"iam",
|
||||
"客户端签名不匹配",
|
||||
any([
|
||||
contains("SignatureDoesNotMatch"),
|
||||
contains("request signature we calculated does not match"),
|
||||
]),
|
||||
"客户端签名不匹配:SK 错误、客户端与服务端时钟偏移过大、或代理改写了 Host/Authorization 头。",
|
||||
"核对凭证;检查 NTP;若走反代,对照 docs/operations/reverse-proxy.md 检查头透传。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
anchors: strings(["Access Key Id you provided does not exist"]),
|
||||
min_count: 5,
|
||||
..base(
|
||||
"unknown-access-key",
|
||||
P3ClientSide,
|
||||
"iam",
|
||||
"Access Key 不存在",
|
||||
contains("Access Key Id you provided does not exist"),
|
||||
"AK 不存在(凭证配错/已删除/打到错误集群)。",
|
||||
"核对客户端配置与目标端点。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
evidence_fields: strings(["access_key"]),
|
||||
anchors: strings(["authenticate_request: authentication failed"]),
|
||||
min_count: 5,
|
||||
..base(
|
||||
"admin-auth-failed",
|
||||
P3ClientSide,
|
||||
"iam",
|
||||
"管理接口鉴权失败",
|
||||
prefix("authenticate_request: authentication failed"),
|
||||
"管理接口鉴权失败。",
|
||||
"核对 console/admin 凭证;高频出现需警惕爆破(看来源 IP)。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
anchors: strings(["action not allowed"]),
|
||||
min_count: 50,
|
||||
..base(
|
||||
"access-denied-burst",
|
||||
P3ClientSide,
|
||||
"iam",
|
||||
"大量授权拒绝",
|
||||
all([
|
||||
Matcher::MinLevel(LogLevel::Warn),
|
||||
any([contains("Access Denied"), contains("action not allowed")]),
|
||||
]),
|
||||
"大量授权拒绝(策略不匹配或客户端行为异常)。",
|
||||
"抽样 samples 核对 bucket 策略与用户策略。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
anchors: strings(["invalid access key length"]),
|
||||
..base(
|
||||
"credential-format-invalid",
|
||||
P3ClientSide,
|
||||
"iam",
|
||||
"凭证格式非法",
|
||||
any([
|
||||
contains("invalid access key length"),
|
||||
contains("invalid secret key length"),
|
||||
contains("malformed credential"),
|
||||
]),
|
||||
"凭证格式非法(常见:环境变量带引号/空格/换行)。",
|
||||
"检查凭证注入方式。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
anchors: strings(["Invalid Keystone token"]),
|
||||
..base(
|
||||
"keystone-auth-failed",
|
||||
P3ClientSide,
|
||||
"iam",
|
||||
"Keystone 集成认证失败",
|
||||
any([prefix("Invalid Keystone token"), contains("Keystone authentication requires")]),
|
||||
"Keystone 集成认证失败。",
|
||||
"检查 Keystone 服务与 token 有效期。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
anchors: strings(["AssumeRole get policy failed"]),
|
||||
..base(
|
||||
"assume-role-failed",
|
||||
P3ClientSide,
|
||||
"iam",
|
||||
"STS AssumeRole 取策略失败",
|
||||
prefix("AssumeRole get policy failed"),
|
||||
"STS AssumeRole 取策略失败。",
|
||||
"核对角色策略配置。",
|
||||
)
|
||||
},
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,102 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! Distributed-lock rules.
|
||||
|
||||
use super::super::model::{Rule, Severity::*};
|
||||
use super::{any, base, contains, prefix, strings};
|
||||
|
||||
pub(super) fn rules() -> Vec<Rule> {
|
||||
vec![
|
||||
Rule {
|
||||
evidence_fields: strings(["resource"]),
|
||||
anchors: strings(["Lock acquisition timeout for resource"]),
|
||||
min_count: 3,
|
||||
..base(
|
||||
"lock-acquire-timeout",
|
||||
P2Degraded,
|
||||
"lock",
|
||||
"锁获取超时",
|
||||
prefix("Lock acquisition timeout for resource"),
|
||||
"锁获取超时(热点对象/慢盘/死锁的症状)。",
|
||||
"看 resource 集中度:集中单对象=热点;广泛分布=盘慢或节点失联。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
evidence_fields: strings(["required", "achieved", "available"]),
|
||||
anchors: strings(["Insufficient nodes for quorum", "Quorum not reached"]),
|
||||
..base(
|
||||
"lock-quorum-nodes",
|
||||
P1Unavailable,
|
||||
"lock",
|
||||
"锁子系统节点数不足仲裁",
|
||||
any([prefix("Insufficient nodes for quorum:"), prefix("Quorum not reached:")]),
|
||||
"锁子系统节点数不足仲裁。",
|
||||
"恢复失联节点;确认部署节点数为奇数且过半存活。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
anchors: strings(["Not the lock owner"]),
|
||||
..base(
|
||||
"lock-owner-mismatch",
|
||||
P2Degraded,
|
||||
"lock",
|
||||
"锁 owner 不匹配",
|
||||
prefix("Not the lock owner"),
|
||||
"释放/续约了非自己持有的锁,锁状态错乱(通常伴随超时后重试)。",
|
||||
"结合 lock-acquire-timeout 判断;孤立出现可忽略。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
anchors: strings(["distributed unlock failed on client"]),
|
||||
..base(
|
||||
"dist-unlock-failed",
|
||||
P2Degraded,
|
||||
"lock",
|
||||
"分布式加/解锁在部分节点失败",
|
||||
any([
|
||||
contains("distributed unlock failed on client"),
|
||||
contains("Failed to acquire lock on client"),
|
||||
]),
|
||||
"分布式加/解锁在部分节点失败(peer 不可达)。",
|
||||
"检查该节点连通性。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
anchors: strings(["Atomic state inconsistency during exclusive lock release"]),
|
||||
..base(
|
||||
"lock-state-inconsistent",
|
||||
P2Degraded,
|
||||
"lock",
|
||||
"锁内部原子状态不一致",
|
||||
contains("Atomic state inconsistency during exclusive lock release"),
|
||||
"锁内部原子状态不一致(并发缺陷级信号)。",
|
||||
"收集完整日志上报研发,附 samples。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
anchors: strings(["poisoned, recovering"]),
|
||||
implies_root_cause: strings(["process-panic"]),
|
||||
..base(
|
||||
"rwlock-poisoned",
|
||||
P1Unavailable,
|
||||
"lock",
|
||||
"RwLock 中毒(进程曾发生 panic 的间接证据)",
|
||||
contains("poisoned, recovering"),
|
||||
"某线程曾持锁 panic 导致 RwLock 中毒——这是进程发生过 panic 的间接证据。",
|
||||
"在同批日志中查 process-panic finding(stderr 段);无 panic 块时说明 stderr 未被采集,建议客户补采。",
|
||||
)
|
||||
},
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,112 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! Built-in seed rule library (rustfs/backlog#1286), distilled from the
|
||||
//! 2026-07 repository-wide failure-log survey.
|
||||
//!
|
||||
//! Every anchor string is verbatim source text (thiserror `#[error]`
|
||||
//! displays or `error!`/`warn!` literals); the CI guard
|
||||
//! (rustfs/backlog#1289) fails when a log message drifts away from its
|
||||
//! rule. Titles/diagnoses/suggestions are operator-facing Chinese.
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests;
|
||||
|
||||
mod capacity;
|
||||
mod disk;
|
||||
mod erasure;
|
||||
mod heal;
|
||||
mod iam;
|
||||
mod lock;
|
||||
mod network;
|
||||
mod ops;
|
||||
mod process;
|
||||
mod quorum;
|
||||
mod scanner;
|
||||
mod startup;
|
||||
|
||||
use super::model::{Matcher, Rule, Severity};
|
||||
|
||||
/// All built-in rules, grouped by category.
|
||||
pub fn seed_rules() -> Vec<Rule> {
|
||||
let groups: [Vec<Rule>; 12] = [
|
||||
disk::rules(),
|
||||
erasure::rules(),
|
||||
quorum::rules(),
|
||||
network::rules(),
|
||||
lock::rules(),
|
||||
heal::rules(),
|
||||
scanner::rules(),
|
||||
iam::rules(),
|
||||
startup::rules(),
|
||||
capacity::rules(),
|
||||
ops::rules(),
|
||||
process::rules(),
|
||||
];
|
||||
groups.into_iter().flatten().collect()
|
||||
}
|
||||
|
||||
// -- shared construction helpers (pub(crate) within the seed tree) --
|
||||
|
||||
pub(crate) fn strings<const N: usize>(items: [&str; N]) -> Vec<String> {
|
||||
items.into_iter().map(String::from).collect()
|
||||
}
|
||||
|
||||
pub(crate) fn prefix(text: &str) -> Matcher {
|
||||
Matcher::MessagePrefix(text.to_string())
|
||||
}
|
||||
|
||||
pub(crate) fn contains(text: &str) -> Matcher {
|
||||
Matcher::MessageContains(text.to_string())
|
||||
}
|
||||
|
||||
pub(crate) fn field(name: &str, value: &str) -> Matcher {
|
||||
Matcher::FieldEquals {
|
||||
name: name.to_string(),
|
||||
value: value.to_string(),
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn any<const N: usize>(matchers: [Matcher; N]) -> Matcher {
|
||||
Matcher::Any(matchers.into())
|
||||
}
|
||||
|
||||
pub(crate) fn all<const N: usize>(matchers: [Matcher; N]) -> Matcher {
|
||||
Matcher::All(matchers.into())
|
||||
}
|
||||
|
||||
/// Base rule with empty extras; callers extend via struct update syntax.
|
||||
pub(crate) fn base(
|
||||
id: &str,
|
||||
severity: Severity,
|
||||
category: &str,
|
||||
title: &str,
|
||||
matcher: Matcher,
|
||||
diagnosis: &str,
|
||||
suggestion: &str,
|
||||
) -> Rule {
|
||||
Rule {
|
||||
id: id.to_string(),
|
||||
severity,
|
||||
category: category.to_string(),
|
||||
title: title.to_string(),
|
||||
matcher,
|
||||
diagnosis: diagnosis.to_string(),
|
||||
suggestion: suggestion.to_string(),
|
||||
evidence_fields: Vec::new(),
|
||||
min_count: 1,
|
||||
implies_root_cause: Vec::new(),
|
||||
anchors: Vec::new(),
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,100 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! Inter-node network / RPC rules.
|
||||
|
||||
use super::super::model::{Rule, Severity::*};
|
||||
use super::{all, any, base, contains, field, prefix, strings};
|
||||
|
||||
pub(super) fn rules() -> Vec<Rule> {
|
||||
vec![
|
||||
Rule {
|
||||
evidence_fields: strings(["peer"]),
|
||||
anchors: strings(["Remote peer operation timeout after"]),
|
||||
..base(
|
||||
"peer-rpc-timeout",
|
||||
P2Degraded,
|
||||
"network",
|
||||
"节点间 RPC 超时",
|
||||
prefix("Remote peer operation timeout after"),
|
||||
"节点间 RPC 超时。",
|
||||
"检查网络延迟/丢包与目标节点负载。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
anchors: strings(["server_info timed out after retry"]),
|
||||
..base(
|
||||
"peer-probe-timeout",
|
||||
P2Degraded,
|
||||
"network",
|
||||
"peer 探活重试后仍超时",
|
||||
contains("server_info timed out after retry"),
|
||||
"peer 探活重试后仍超时。",
|
||||
"检查网络延迟/丢包与目标节点负载。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
anchors: strings(["peer request failed"]),
|
||||
..base(
|
||||
"internode-signature-mismatch",
|
||||
P1Unavailable,
|
||||
"network",
|
||||
"节点间 RPC 鉴权失败",
|
||||
all([contains("peer request failed"), contains("SignatureDoesNotMatch")]),
|
||||
"节点间 RPC 鉴权失败——各节点 RUSTFS_ACCESS_KEY/SECRET_KEY 不一致,或节点间时钟偏移过大。",
|
||||
"核对各节点凭证环境变量一致;检查 NTP 同步。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
anchors: strings(["RPC auth secret resolution failed"]),
|
||||
..base(
|
||||
"rpc-secret-resolution",
|
||||
P1Unavailable,
|
||||
"network",
|
||||
"内部 RPC 密钥解析失败",
|
||||
prefix("RPC auth secret resolution failed"),
|
||||
"内部 RPC 密钥解析失败,节点间调用将全部失败。",
|
||||
"检查凭证配置来源(env/文件)完整性。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
evidence_fields: strings(["peer", "host"]),
|
||||
anchors: strings(["peer_connection_marked_offline"]),
|
||||
..base(
|
||||
"peer-connection-offline",
|
||||
P2Degraded,
|
||||
"network",
|
||||
"peer 连接被标记离线",
|
||||
any([
|
||||
contains("peer_connection_marked_offline"),
|
||||
field("event", "peer_connection_marked_offline"),
|
||||
]),
|
||||
"peer 连接被判定离线。",
|
||||
"检查目标节点与网络。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
anchors: strings(["Expected number of all hosts"]),
|
||||
..base(
|
||||
"topology-mismatch",
|
||||
P1Unavailable,
|
||||
"network",
|
||||
"集群成员拓扑不一致",
|
||||
prefix("Expected number of all hosts"),
|
||||
"集群成员拓扑与配置不一致(各节点 volumes 参数不一致/DNS 漂移)。",
|
||||
"逐节点比对启动参数中的 endpoint 列表。",
|
||||
)
|
||||
},
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,81 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! Decommission / rebalance operational rules.
|
||||
|
||||
use super::super::model::{Matcher, Rule, Severity::*};
|
||||
use super::{all, any, base, contains, prefix, strings};
|
||||
use crate::model::LogLevel;
|
||||
|
||||
pub(super) fn rules() -> Vec<Rule> {
|
||||
vec![
|
||||
Rule {
|
||||
anchors: strings(["decommission_object err"]),
|
||||
min_count: 3,
|
||||
..base(
|
||||
"decom-object-failed",
|
||||
P2Degraded,
|
||||
"ops",
|
||||
"下线迁移部分对象失败",
|
||||
any([
|
||||
contains("decommission_object err"),
|
||||
contains("get_object_reader err"),
|
||||
contains("decommission_entry failed"),
|
||||
]),
|
||||
"下线迁移部分对象失败(会重试;持续失败需关注)。",
|
||||
"看失败对象是否集中(坏对象/坏盘)。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
anchors: strings(["Rebalance worker"]),
|
||||
..base(
|
||||
"rebalance-worker-error",
|
||||
P2Degraded,
|
||||
"ops",
|
||||
"再平衡 worker 失败",
|
||||
all([prefix("Rebalance worker"), Matcher::MinLevel(LogLevel::Error)]),
|
||||
"再平衡 worker 失败。",
|
||||
"看 samples 内层错误;结合盘/网络 finding。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
anchors: strings(["source and destination pool are the same"]),
|
||||
..base(
|
||||
"datamove-same-pool",
|
||||
P1Unavailable,
|
||||
"ops",
|
||||
"数据迁移源目标同 pool",
|
||||
contains("source and destination pool are the same"),
|
||||
"数据迁移源目标同 pool(配置错误)。",
|
||||
"核对 decommission/rebalance 目标参数。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
anchors: strings(["Decommission already running"]),
|
||||
..base(
|
||||
"ops-state-conflict",
|
||||
P4Info,
|
||||
"ops",
|
||||
"运维操作状态机冲突",
|
||||
any([
|
||||
contains("Decommission already running"),
|
||||
contains("Decommission not started"),
|
||||
contains("Rebalance already running"),
|
||||
]),
|
||||
"运维操作状态机冲突(重复触发,通常无害)。",
|
||||
"确认是否有并行运维脚本。",
|
||||
)
|
||||
},
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! Process-level rules.
|
||||
|
||||
use super::super::model::{Matcher, Rule, Severity::*};
|
||||
use super::{base, strings};
|
||||
|
||||
pub(super) fn rules() -> Vec<Rule> {
|
||||
vec![Rule {
|
||||
evidence_fields: strings(["panic_location", "panic_thread"]),
|
||||
// Structural matcher — no message anchor to guard.
|
||||
..base(
|
||||
"process-panic",
|
||||
P1Unavailable,
|
||||
"process",
|
||||
"进程发生 panic",
|
||||
Matcher::IsPanic,
|
||||
"进程发生 panic(可能随后重启/触发锁中毒)。",
|
||||
"将 panic_location 与 panic_full 上报研发;检查是否伴随 rwlock-poisoned 与重启痕迹(时间线断档)。",
|
||||
)
|
||||
}]
|
||||
}
|
||||
@@ -0,0 +1,73 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! Read/write quorum rules.
|
||||
|
||||
use super::super::model::{Rule, Severity::*};
|
||||
use super::{base, contains, prefix, strings};
|
||||
|
||||
pub(super) fn rules() -> Vec<Rule> {
|
||||
vec![
|
||||
Rule {
|
||||
evidence_fields: strings(["bucket", "object"]),
|
||||
anchors: strings(["Namespace lock quorum unavailable"]),
|
||||
..base(
|
||||
"nslock-quorum",
|
||||
P1Unavailable,
|
||||
"quorum",
|
||||
"命名空间锁仲裁不可用",
|
||||
prefix("Namespace lock quorum unavailable"),
|
||||
"命名空间锁拿不到仲裁,写路径被阻断(节点半数以上不可达的典型症状)。",
|
||||
"检查节点间连通与存活节点数是否过半。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
anchors: strings(["below write quorum"]),
|
||||
implies_root_cause: strings(["disk-marked-faulty", "remote-peer-faulty", "peer-disks-offline", "disk-full"]),
|
||||
..base(
|
||||
"below-write-quorum",
|
||||
P1Unavailable,
|
||||
"quorum",
|
||||
"在线盘数低于写仲裁",
|
||||
contains("below write quorum"),
|
||||
"在线盘快照低于写仲裁。",
|
||||
"同 ec-write-quorum。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
anchors: strings(["reduce_write_quorum_errs"]),
|
||||
..base(
|
||||
"bucket-op-quorum",
|
||||
P1Unavailable,
|
||||
"quorum",
|
||||
"bucket 级操作写仲裁失败",
|
||||
contains("reduce_write_quorum_errs"),
|
||||
"bucket 级操作(建桶/heal/list)跨 pool 写仲裁失败。",
|
||||
"检查各 pool 节点在线状态。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
anchors: strings(["object_quorum_from_meta"]),
|
||||
..base(
|
||||
"quorum-meta-derive",
|
||||
P1Unavailable,
|
||||
"quorum",
|
||||
"从元数据推导仲裁参数失败",
|
||||
contains("object_quorum_from_meta"),
|
||||
"从对象元数据推导仲裁参数失败,元数据可能损坏。",
|
||||
"对样本对象做 xl.meta 检查(参见 docs/operations/tier-ilm-debugging.md 的 xl.meta 工具)。",
|
||||
)
|
||||
},
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,64 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! Scanner rules. The scanner logs under `target: "rustfs::scanner*"` with
|
||||
//! structured fields, so the burst rule keys on the target prefix.
|
||||
|
||||
use super::super::model::{Matcher, Rule, Severity::*};
|
||||
use super::{all, base, contains, prefix, strings};
|
||||
use crate::model::LogLevel;
|
||||
|
||||
pub(super) fn rules() -> Vec<Rule> {
|
||||
vec![
|
||||
Rule {
|
||||
anchors: strings(["Scanner stopped with partial data usage cache"]),
|
||||
..base(
|
||||
"scanner-partial-cache",
|
||||
P3ClientSide,
|
||||
"scanner",
|
||||
"扫描中断,容量统计不完整",
|
||||
contains("Scanner stopped with partial data usage cache"),
|
||||
"扫描中断,容量统计不完整(显示值可能偏低,非数据问题)。",
|
||||
"确认 scanner 是否被频繁重启打断。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
anchors: strings(["invalid scanner config value for"]),
|
||||
..base(
|
||||
"scanner-config-invalid",
|
||||
P3ClientSide,
|
||||
"scanner",
|
||||
"scanner 运行时配置非法",
|
||||
prefix("invalid scanner config value for"),
|
||||
"scanner 运行时配置非法,回落默认值。",
|
||||
"修正对应环境变量(见 docs/operations/scanner-runtime-controls.md)。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
min_count: 10,
|
||||
..base(
|
||||
"scanner-error-burst",
|
||||
P4Info,
|
||||
"scanner",
|
||||
"scanner 持续报错",
|
||||
all([
|
||||
Matcher::TargetPrefix("rustfs::scanner".to_string()),
|
||||
Matcher::MinLevel(LogLevel::Error),
|
||||
]),
|
||||
"scanner 持续报错(聚合信号,具体原因看 samples)。",
|
||||
"结合盘健康 finding 判断是否坏盘引起。",
|
||||
)
|
||||
},
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,143 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! Startup / configuration / TLS rules.
|
||||
|
||||
use super::super::model::{Rule, Severity::*};
|
||||
use super::{all, any, base, contains, field, prefix, strings};
|
||||
|
||||
pub(super) fn rules() -> Vec<Rule> {
|
||||
vec![
|
||||
Rule {
|
||||
evidence_fields: strings(["host"]),
|
||||
anchors: strings(["Create pool endpoints host"]),
|
||||
..base(
|
||||
"endpoint-resolve-failed",
|
||||
P1Unavailable,
|
||||
"startup",
|
||||
"启动期端点主机名解析失败",
|
||||
all([contains("Create pool endpoints host"), contains("not found")]),
|
||||
"启动期端点主机名解析失败(DNS 未就绪/拼写错误/hosts 缺失)。",
|
||||
"容器/K8s 场景确认 DNS 就绪顺序;核对 volumes 参数拼写。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
anchors: strings(["Console and endpoint should use different ports"]),
|
||||
..base(
|
||||
"listen-config-invalid",
|
||||
P1Unavailable,
|
||||
"startup",
|
||||
"监听地址/端口配置错误",
|
||||
any([
|
||||
contains("Invalid endpoint port"),
|
||||
contains("Invalid console port"),
|
||||
contains("Console and endpoint should use different ports"),
|
||||
contains("Failed to parse with address args"),
|
||||
]),
|
||||
"监听地址/端口配置错误。",
|
||||
"核对 --address/--console-address。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
anchors: strings([
|
||||
"persisted server config cannot be decoded",
|
||||
"server config is corrupt after heal",
|
||||
]),
|
||||
..base(
|
||||
"server-config-corrupt",
|
||||
P1Unavailable,
|
||||
"startup",
|
||||
"持久化服务配置损坏或解密失败",
|
||||
any([
|
||||
contains("persisted server config cannot be decoded"),
|
||||
prefix("server config corrupt:"),
|
||||
contains("server config is corrupt after heal"),
|
||||
]),
|
||||
"持久化服务配置损坏或解密失败;若提示 fallback 被禁用则启动会失败。",
|
||||
"按日志指引评估 RUSTFS_CONFIG_RECOVER_ON_CORRUPTION;排查 KMS/密钥变更历史。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
anchors: strings(["unable to load the certificate for"]),
|
||||
..base(
|
||||
"tls-cert-load-failed",
|
||||
P1Unavailable,
|
||||
"startup",
|
||||
"TLS 证书加载失败",
|
||||
any([
|
||||
prefix("unable to load the certificate for"),
|
||||
prefix("unable to load root directory certificate"),
|
||||
]),
|
||||
"TLS 证书加载失败(路径/权限/格式)。",
|
||||
"用 `rustfs tls inspect --path <dir>` 现场检查证书目录。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
anchors: strings(["client_cert and client_key must be specified as a pair"]),
|
||||
..base(
|
||||
"tls-config-invalid",
|
||||
P2Degraded,
|
||||
"startup",
|
||||
"TLS 目标配置组合非法",
|
||||
any([
|
||||
contains("client_cert and client_key must be specified as a pair"),
|
||||
contains("skipTlsVerify and caCertPem cannot be enabled together"),
|
||||
contains("caCertPem requires an HTTPS remote target"),
|
||||
]),
|
||||
"TLS 目标配置组合非法。",
|
||||
"按报错修正配置对。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
anchors: strings(["Global server configuration not loaded"]),
|
||||
..base(
|
||||
"subsystem-init-order",
|
||||
P1Unavailable,
|
||||
"startup",
|
||||
"全局配置未加载导致子系统初始化失败",
|
||||
any([
|
||||
contains("Global server configuration not loaded"),
|
||||
contains("Global server config not loaded"),
|
||||
]),
|
||||
"全局配置未加载导致子系统初始化失败(启动顺序/更早的配置错误)。",
|
||||
"向前查同批日志更早的 startup 类 finding。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
anchors: strings(["[FATAL] "]),
|
||||
..base(
|
||||
"startup-fatal",
|
||||
P1Unavailable,
|
||||
"startup",
|
||||
"进程启动期致命错误",
|
||||
prefix("[FATAL]"),
|
||||
"进程启动期致命错误(observability/预检失败),进程未能起来。",
|
||||
"按 message 内层错误处理;这是「服务起不来」最直接的证据。",
|
||||
)
|
||||
},
|
||||
Rule {
|
||||
evidence_fields: strings(["error"]),
|
||||
anchors: strings(["server_runtime_failed"]),
|
||||
..base(
|
||||
"runtime-failed",
|
||||
P1Unavailable,
|
||||
"startup",
|
||||
"服务运行时整体退出",
|
||||
any([field("event", "server_runtime_failed"), contains("Server runtime failed")]),
|
||||
"服务运行时整体退出。",
|
||||
"看 error 字段与临近 finding。",
|
||||
)
|
||||
},
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,343 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! Seed-library tests: one realistic positive sample per rule, exact-set
|
||||
//! smoke samples, and cross-rule negative cases.
|
||||
|
||||
use super::super::{RuleEngine, seed_rule_set};
|
||||
use crate::model::{EventKind, LogEvent, LogLevel, SourceRef};
|
||||
use std::sync::Arc;
|
||||
|
||||
fn engine() -> RuleEngine {
|
||||
RuleEngine::new(seed_rule_set())
|
||||
}
|
||||
|
||||
struct Sample {
|
||||
message: &'static str,
|
||||
level: Option<LogLevel>,
|
||||
target: &'static str,
|
||||
kind: EventKind,
|
||||
fields: &'static [(&'static str, &'static str)],
|
||||
}
|
||||
|
||||
impl Default for Sample {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
message: "",
|
||||
level: Some(LogLevel::Error),
|
||||
target: "rustfs::server::http",
|
||||
kind: EventKind::Json,
|
||||
fields: &[],
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn event(sample: &Sample) -> LogEvent {
|
||||
let mut fields = serde_json::Map::new();
|
||||
for (k, v) in sample.fields {
|
||||
fields.insert((*k).to_string(), serde_json::Value::String((*v).to_string()));
|
||||
}
|
||||
LogEvent {
|
||||
timestamp: None,
|
||||
level: sample.level,
|
||||
target: Some(sample.target.to_string()),
|
||||
message: sample.message.to_string(),
|
||||
fields,
|
||||
source: SourceRef {
|
||||
file: Arc::from("rustfs.log"),
|
||||
line: 1,
|
||||
},
|
||||
node: None,
|
||||
kind: sample.kind,
|
||||
}
|
||||
}
|
||||
|
||||
fn msg(message: &'static str) -> Sample {
|
||||
Sample {
|
||||
message,
|
||||
..Default::default()
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn seed_rule_set_is_valid_and_complete() {
|
||||
let set = seed_rule_set();
|
||||
assert_eq!(set.rules().len(), 68);
|
||||
// Every message-matching rule carries at least one anchor for the CI
|
||||
// guard; the only anchor-less rules are the two structural matchers
|
||||
// (panic kind, scanner target+level).
|
||||
let anchorless: Vec<_> = set
|
||||
.rules()
|
||||
.iter()
|
||||
.filter(|r| r.anchors.is_empty())
|
||||
.map(|r| r.id.as_str())
|
||||
.collect();
|
||||
assert_eq!(anchorless, vec!["scanner-error-burst", "process-panic"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn every_rule_has_a_positive_sample() {
|
||||
// (rule id, sample). Kept in seed-table order (rustfs/backlog#1286).
|
||||
let table: Vec<(&str, Sample)> = vec![
|
||||
// disk
|
||||
(
|
||||
"disk-marked-faulty",
|
||||
Sample {
|
||||
message: "health state changed",
|
||||
fields: &[("reason", "faulty_disk")],
|
||||
..Default::default()
|
||||
},
|
||||
),
|
||||
(
|
||||
"disk-faulty-rejected",
|
||||
Sample {
|
||||
message: "operation rejected",
|
||||
fields: &[("reason", "disk_marked_faulty")],
|
||||
..Default::default()
|
||||
},
|
||||
),
|
||||
("remote-peer-faulty", msg("Remote peer health check failed for node2: marking as faulty")),
|
||||
(
|
||||
"peer-disks-offline",
|
||||
msg("reporting peer disks offline after consecutive storage_info failures"),
|
||||
),
|
||||
("drive-faulty-error", msg("remote drive is faulty")),
|
||||
("unformatted-disk", msg("Unformatted disk found")),
|
||||
("disk-access-denied", msg("disk access denied: /data/disk1")),
|
||||
("inconsistent-drive", msg("inconsistent drive found")),
|
||||
("fd-exhausted", msg("too many open files, please increase 'ulimit -n'")),
|
||||
("odirect-unsupported", msg("drive does not support O_DIRECT")),
|
||||
(
|
||||
"rename-across-devices",
|
||||
msg("Rename across devices not allowed, please fix your backend configuration"),
|
||||
),
|
||||
// erasure
|
||||
("bitrot-detected", msg("bitrot hash mismatch on shard 3")),
|
||||
("short-shard-read", msg("bitrot reader short shard read: got 100 of 200 bytes")),
|
||||
("heal-cannot-reconstruct", msg("Heal object cannot reconstruct with available shards")),
|
||||
("ec-write-quorum", msg("erasure write quorum (required=8, achieved=5, failed=3)")),
|
||||
("ec-read-quorum", msg("reduce_read_quorum_errs: [..], bucket: media, object: a.bin")),
|
||||
("file-corrupted", msg("part missing or corrupt")),
|
||||
// quorum
|
||||
(
|
||||
"nslock-quorum",
|
||||
msg("Namespace lock quorum unavailable for write lock on b/o: required 3, achieved 1"),
|
||||
),
|
||||
(
|
||||
"below-write-quorum",
|
||||
msg("online disk snapshot 3 below write quorum 4 for b/o; returning error"),
|
||||
),
|
||||
("bucket-op-quorum", msg("heal_bucket reduce_write_quorum_errs: pool error")),
|
||||
("quorum-meta-derive", msg("object_quorum_from_meta: parity_blocks < 0, errs=[..]")),
|
||||
// network
|
||||
("peer-rpc-timeout", msg("Remote peer operation timeout after 30s")),
|
||||
("peer-probe-timeout", msg("peer node2 server_info timed out after retry (10s)")),
|
||||
(
|
||||
"internode-signature-mismatch",
|
||||
msg("peer request failed with 403 Forbidden: SignatureDoesNotMatch"),
|
||||
),
|
||||
(
|
||||
"rpc-secret-resolution",
|
||||
msg("RPC auth secret resolution failed: missing secret; source=env"),
|
||||
),
|
||||
("peer-connection-offline", msg("peer_connection_marked_offline")),
|
||||
("topology-mismatch", msg("Expected number of all hosts (4) to be remote +1 (3)")),
|
||||
// lock
|
||||
("lock-acquire-timeout", msg("Lock acquisition timeout for resource 'b/o' after 30s")),
|
||||
("lock-quorum-nodes", msg("Quorum not reached: required 3, achieved 1")),
|
||||
("lock-owner-mismatch", msg("Not the lock owner: lock_id abc, owner node1")),
|
||||
("dist-unlock-failed", msg("distributed unlock failed on client: node2")),
|
||||
(
|
||||
"lock-state-inconsistent",
|
||||
msg("Atomic state inconsistency during exclusive lock release: owner=x, atomic_state=101"),
|
||||
),
|
||||
("rwlock-poisoned", msg("bucket monitor measurement rwlock read poisoned, recovering")),
|
||||
// heal
|
||||
(
|
||||
"heal-no-datadir",
|
||||
msg("heal: latest metadata for b/o has no data_dir, cannot heal object data"),
|
||||
),
|
||||
("heal-all-writes-failed", msg("all drives had write errors, unable to heal b/o")),
|
||||
("heal-rename-failed", msg("all healed data rename attempts failed for b/o")),
|
||||
(
|
||||
"heal-xlmeta-regen-failed",
|
||||
msg("heal_object: failed to regenerate recoverable xl.meta on disk"),
|
||||
),
|
||||
(
|
||||
"heal-writer-create-failed",
|
||||
msg("create_bitrot_writer disk d1, err timeout, skipping operation"),
|
||||
),
|
||||
("heal-orphan-reclaim-failed", msg("heal_object: orphan data-dir reclaim failed")),
|
||||
("heal-task-failure", msg("Heal task execution failed: worker died")),
|
||||
// scanner
|
||||
("scanner-partial-cache", msg("Scanner stopped with partial data usage cache")),
|
||||
(
|
||||
"scanner-config-invalid",
|
||||
msg("invalid scanner config value for max_wait: -1 (must be positive)"),
|
||||
),
|
||||
(
|
||||
"scanner-error-burst",
|
||||
Sample {
|
||||
message: "scan cycle failed",
|
||||
target: "rustfs::scanner::io",
|
||||
..Default::default()
|
||||
},
|
||||
),
|
||||
// iam
|
||||
("client-signature-mismatch", msg("SignatureDoesNotMatch")),
|
||||
("unknown-access-key", msg("The Access Key Id you provided does not exist in our records.")),
|
||||
(
|
||||
"admin-auth-failed",
|
||||
msg("authenticate_request: authentication failed - access_key=AK123, error=denied"),
|
||||
),
|
||||
("access-denied-burst", msg("action not allowed for user x")),
|
||||
("credential-format-invalid", msg("invalid access key length")),
|
||||
("keystone-auth-failed", msg("Invalid Keystone token: expired")),
|
||||
("assume-role-failed", msg("AssumeRole get policy failed, err: NotFound, access_key: AK1")),
|
||||
// startup
|
||||
(
|
||||
"endpoint-resolve-failed",
|
||||
msg("Create pool endpoints host node5 not found, error: dns failure"),
|
||||
),
|
||||
("listen-config-invalid", msg("Console and endpoint should use different ports")),
|
||||
(
|
||||
"server-config-corrupt",
|
||||
msg("persisted server config cannot be decoded, object is corrupt"),
|
||||
),
|
||||
(
|
||||
"tls-cert-load-failed",
|
||||
msg("unable to load the certificate for example.com domain name: bad path"),
|
||||
),
|
||||
("tls-config-invalid", msg("client_cert and client_key must be specified as a pair")),
|
||||
(
|
||||
"subsystem-init-order",
|
||||
msg("Audit system initialization failed: Global server configuration not loaded."),
|
||||
),
|
||||
(
|
||||
"startup-fatal",
|
||||
Sample {
|
||||
message: "[FATAL] Command parse failed: bad flag",
|
||||
level: None,
|
||||
kind: EventKind::Text,
|
||||
..Default::default()
|
||||
},
|
||||
),
|
||||
(
|
||||
"runtime-failed",
|
||||
Sample {
|
||||
message: "Server runtime failed",
|
||||
fields: &[("event", "server_runtime_failed")],
|
||||
..Default::default()
|
||||
},
|
||||
),
|
||||
// capacity
|
||||
("disk-full", msg("Disk full")),
|
||||
("min-free-threshold", msg("Storage reached its minimum free drive threshold.")),
|
||||
(
|
||||
"insufficient-storage",
|
||||
msg("Storage resources are insufficient for the write operation: 2/4"),
|
||||
),
|
||||
(
|
||||
"bucket-quota-exceeded",
|
||||
msg("Bucket quota exceeded: current=100, limit=50, operation=PutObject"),
|
||||
),
|
||||
(
|
||||
"decom-capacity-insufficient",
|
||||
msg("failed to start decommission: insufficient target pool capacity: required 100 bytes available 50 bytes"),
|
||||
),
|
||||
// ops
|
||||
("decom-object-failed", msg("decommission_pool: decommission_object err timeout")),
|
||||
("rebalance-worker-error", msg("Rebalance worker 3 error: disk gone")),
|
||||
(
|
||||
"datamove-same-pool",
|
||||
msg("invalid data movement operation, source and destination pool are the same for : b/o-1"),
|
||||
),
|
||||
("ops-state-conflict", msg("Decommission already running")),
|
||||
// process
|
||||
(
|
||||
"process-panic",
|
||||
Sample {
|
||||
message: "thread 'main' panicked at src/main.rs:3:5: boom",
|
||||
kind: EventKind::Panic,
|
||||
..Default::default()
|
||||
},
|
||||
),
|
||||
];
|
||||
|
||||
let engine = engine();
|
||||
let rule_ids: Vec<&str> = engine.rules().iter().map(|r| r.id.as_str()).collect();
|
||||
assert_eq!(table.len(), rule_ids.len(), "one sample per rule");
|
||||
|
||||
for (id, sample) in &table {
|
||||
assert!(rule_ids.contains(id), "sample references unknown rule '{id}'");
|
||||
let hits = engine.matches(&event(sample));
|
||||
let hit_ids: Vec<&str> = hits.iter().map(|&i| engine.rules()[i].id.as_str()).collect();
|
||||
assert!(hit_ids.contains(id), "rule '{id}' did not match its sample; hits: {hit_ids:?}");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn smoke_samples_hit_exact_rule_sets() {
|
||||
let engine = engine();
|
||||
let exact = |sample: &Sample, expected: &[&str]| {
|
||||
let hits = engine.matches(&event(sample));
|
||||
let mut hit_ids: Vec<&str> = hits.iter().map(|&i| engine.rules()[i].id.as_str()).collect();
|
||||
hit_ids.sort_unstable();
|
||||
let mut expected = expected.to_vec();
|
||||
expected.sort_unstable();
|
||||
assert_eq!(hit_ids, expected, "sample: {}", sample.message);
|
||||
};
|
||||
|
||||
exact(
|
||||
&Sample {
|
||||
message: "Disk health check marked disk faulty",
|
||||
fields: &[("reason", "faulty_disk")],
|
||||
..Default::default()
|
||||
},
|
||||
&["disk-marked-faulty"],
|
||||
);
|
||||
exact(&msg("erasure write quorum (required=8, achieved=5)"), &["ec-write-quorum"]);
|
||||
// Internode auth failures legitimately hit both the internode rule and
|
||||
// the generic client signature rule.
|
||||
exact(
|
||||
&msg("peer request failed with 403 Forbidden: SignatureDoesNotMatch"),
|
||||
&["internode-signature-mismatch", "client-signature-mismatch"],
|
||||
);
|
||||
exact(
|
||||
&Sample {
|
||||
message: "thread 'main' panicked at src/main.rs:3:5: boom",
|
||||
kind: EventKind::Panic,
|
||||
..Default::default()
|
||||
},
|
||||
&["process-panic"],
|
||||
);
|
||||
// Healthy INFO lines and plain text hit nothing.
|
||||
exact(
|
||||
&Sample {
|
||||
message: "request completed in 12ms",
|
||||
level: Some(LogLevel::Info),
|
||||
..Default::default()
|
||||
},
|
||||
&[],
|
||||
);
|
||||
exact(
|
||||
&Sample {
|
||||
message: "some ordinary text trailer",
|
||||
level: None,
|
||||
kind: EventKind::Text,
|
||||
..Default::default()
|
||||
},
|
||||
&[],
|
||||
);
|
||||
}
|
||||
Reference in New Issue
Block a user