mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-23 12:49:04 +00:00
refactor(logging): standardize concurrency and trusted proxy events (#3417)
* refactor(logging): standardize concurrency and proxy events * chore(logging): extend guardrails for concurrency and proxies * feat(skill): add rustfs logging governance skill
This commit is contained in:
houseme
@@ -0,0 +1,107 @@
|
||||
---
|
||||
name: rustfs-logging-governance
|
||||
description: Standardize and review RustFS logging with structured `tracing` events, lower noise on hot paths, preserve security-sensitive diagnostics, and extend guardrails to prevent legacy logging patterns from returning. Use when editing or reviewing RustFS logs, startup/config diagnostics, cloud metadata logs, request validation logs, or `scripts/check_logging_guardrails.sh`.
|
||||
---
|
||||
|
||||
# RustFS Logging Governance
|
||||
|
||||
Use this skill when RustFS logging needs to be added, cleaned up, reviewed, or protected against regressions.
|
||||
|
||||
## Quick Start
|
||||
|
||||
1. Identify the files whose logs are changing.
|
||||
2. Scan current `tracing` or `log` macros before editing.
|
||||
3. Convert sentence-style logs to short event-style logs.
|
||||
4. Demote hot-path success logs unless operators truly need them at `info`.
|
||||
5. Preserve failure, fallback, and security-relevant diagnostics.
|
||||
6. Update `scripts/check_logging_guardrails.sh` when a broad cleanup removes a legacy pattern class.
|
||||
7. Validate with formatting, targeted checks/tests, and the logging guardrail script.
|
||||
|
||||
## Core Workflow
|
||||
|
||||
### 1. Scope the logging surface
|
||||
|
||||
- Read the changed module in full before touching log lines.
|
||||
- Classify the log site:
|
||||
- lifecycle/startup
|
||||
- request or validation path
|
||||
- background loop or hot path
|
||||
- fallback/degraded behavior
|
||||
- cloud metadata or external fetch path
|
||||
- metrics/config summary
|
||||
- Do not rewrite business logic to make logging easier.
|
||||
|
||||
### 2. Use the RustFS event shape
|
||||
|
||||
- Prefer fields first, message second.
|
||||
- Use short labels, not prose paragraphs.
|
||||
- Default field shape:
|
||||
- `event`
|
||||
- `component`
|
||||
- `subsystem`
|
||||
- `state` or `result`
|
||||
- key context fields
|
||||
- Reuse stable field names and avoid inventing near-duplicates.
|
||||
|
||||
See `references/logging-governance.md` for the event model, level policy, and anti-pattern list.
|
||||
|
||||
### 3. Choose the right level
|
||||
|
||||
- `error`: operation failure that affects behavior or security guarantees.
|
||||
- `warn`: degraded path, fallback, suspicious input, or operator-actionable misconfiguration.
|
||||
- `info`: low-frequency lifecycle or mode selection.
|
||||
- `debug`: targeted diagnostics and low-volume detail.
|
||||
- `trace`: hot-path and repetitive success-path events.
|
||||
|
||||
When in doubt, lower the verbosity of normal success paths and keep structured detail in fields.
|
||||
|
||||
### 4. Preserve security and privacy boundaries
|
||||
|
||||
- Do not log secrets, tokens, auth headers, raw credential payloads, or merged config dumps.
|
||||
- Avoid logging raw forwarded headers or full trusted network inventories above `debug`.
|
||||
- Keep warning/error logs useful without echoing attacker-controlled payloads unnecessarily.
|
||||
|
||||
### 5. Keep summaries aggregated
|
||||
|
||||
- Replace multi-line startup banners or checklist logs with one structured event.
|
||||
- If metrics already express a concept, avoid duplicating it with many `info!` lines.
|
||||
- Prefer counts, modes, and sources over inventories unless debug detail is truly needed.
|
||||
|
||||
### 6. Update guardrails when needed
|
||||
|
||||
- Broad logging cleanup should usually extend `scripts/check_logging_guardrails.sh`.
|
||||
- Add forbidden patterns only for styles the repo has intentionally retired:
|
||||
- sentence-style lifecycle logs
|
||||
- noisy hot-path `info!`
|
||||
- checklist-style summary logs
|
||||
- legacy fallback wording that has been replaced by structured fields
|
||||
- Keep guardrails concrete and grep-friendly.
|
||||
|
||||
### 7. Validate manually
|
||||
|
||||
Use the smallest relevant set:
|
||||
|
||||
```bash
|
||||
cargo fmt --all --check
|
||||
./scripts/check_logging_guardrails.sh
|
||||
cargo check -p <affected-crate>
|
||||
cargo test -p <affected-crate>
|
||||
```
|
||||
|
||||
For broader Rust changes, add:
|
||||
|
||||
```bash
|
||||
./scripts/check_unsafe_code_allowances.sh
|
||||
./scripts/check_architecture_migration_rules.sh
|
||||
cargo clippy -p <affected-crates> --all-targets -- -D warnings
|
||||
```
|
||||
|
||||
## RustFS-Specific Notes
|
||||
|
||||
- The durable RustFS logging direction is `event + component + subsystem + state/result + key context fields`.
|
||||
- `crates/concurrency` and `crates/trusted-proxies` are examples of this style for lifecycle, fallback, and cloud metadata logs.
|
||||
- `scripts/check_logging_guardrails.sh` is the enforcement point for preventing removed log styles from returning.
|
||||
|
||||
## References
|
||||
|
||||
- Read `references/logging-governance.md` when you need the detailed field set, anti-pattern examples, or guardrail update checklist.
|
||||
@@ -0,0 +1,4 @@
|
||||
interface:
|
||||
display_name: "RustFS Logging Governance"
|
||||
short_description: "Standardize RustFS logs with structured events and guardrails."
|
||||
default_prompt: "Use $rustfs-logging-governance to standardize or review RustFS logging, reduce noise, and update guardrails."
|
||||
@@ -0,0 +1,285 @@
|
||||
# RustFS Logging Governance Reference
|
||||
|
||||
## Workspace Scope Map
|
||||
|
||||
Use `Cargo.toml` `[workspace].members` as the source of truth for crate membership. When doing a broad logging sweep, classify crates by operational role so logs stay consistent within each role.
|
||||
|
||||
### Core Server And Request Handling
|
||||
|
||||
- `rustfs`
|
||||
- Role: top-level server, startup, auth, admin wiring, S3 request handling.
|
||||
- Logging focus: startup lifecycle, config summaries, authn/authz failures, protocol entrypoints, degraded subsystems.
|
||||
- `crates/protocols`
|
||||
- Role: protocol integrations such as FTP, SFTP, WebDAV, and related server-side protocol layers.
|
||||
- Logging focus: listener lifecycle, per-protocol enablement/disablement, request bridge failures.
|
||||
- `crates/madmin`
|
||||
- Role: admin API contracts and management interfaces.
|
||||
- Logging focus: admin action boundaries, validation failures, compatibility warnings.
|
||||
- `crates/trusted-proxies`
|
||||
- Role: forwarded IP trust, proxy chain validation, cloud metadata sources.
|
||||
- Logging focus: direct/trusted/fallback decisions, degraded metadata fetches, aggregated config summaries.
|
||||
- `crates/keystone`
|
||||
- Role: Keystone auth integration.
|
||||
- Logging focus: integration enablement, upstream auth failures, config safety without credential leakage.
|
||||
|
||||
### Storage, Healing, And Data Plane
|
||||
|
||||
- `crates/ecstore`
|
||||
- Role: erasure-coded storage implementation and peer/store initialization.
|
||||
- Logging focus: disk/peer lifecycle, storage fallback, object I/O failures, avoid per-object noise.
|
||||
- `crates/heal`
|
||||
- Role: healing orchestration and repair workflows.
|
||||
- Logging focus: scheduler lifecycle, repair decisions, backlog or skipped work summaries, avoid repetitive task spam at `info`.
|
||||
- `crates/scanner`
|
||||
- Role: data integrity scanning and health monitoring.
|
||||
- Logging focus: scan lifecycle, compaction/deep-heal transitions, lag/backlog, noisy folder iteration should stay at `debug/trace`.
|
||||
- `crates/object-capacity`
|
||||
- Role: capacity scan and refresh core.
|
||||
- Logging focus: refresh lifecycle, degraded capacity sources, aggregate stats rather than per-object chatter.
|
||||
- `crates/filemeta`
|
||||
- Role: file metadata parsing and helpers.
|
||||
- Logging focus: parse failures, schema/format mismatch, avoid dumping raw metadata payloads.
|
||||
- `crates/storage-api`
|
||||
- Role: storage contracts and shared data plane interfaces.
|
||||
- Logging focus: contract mismatch and boundary diagnostics, usually low-volume.
|
||||
- `crates/checksums`
|
||||
- Role: checksum helpers and validation.
|
||||
- Logging focus: integrity failures and compatibility mismatches, not per-chunk success logs.
|
||||
- `crates/zip`
|
||||
- Role: ZIP handling and compression helpers.
|
||||
- Logging focus: parse/extract failures, archive path safety issues, avoid verbose file-by-file success logs.
|
||||
|
||||
### Security, Identity, And Policy
|
||||
|
||||
- `crates/iam`
|
||||
- Role: identity and access management.
|
||||
- Logging focus: authz decision boundaries, imported payload safety, do not leak principals, secrets, or claims.
|
||||
- `crates/policy`
|
||||
- Role: policy modeling and evaluation.
|
||||
- Logging focus: deny/allow decision context, parser/validation failures, no raw secret-bearing request dumps.
|
||||
- `crates/credentials`
|
||||
- Role: credential handling.
|
||||
- Logging focus: never log secrets or tokens; only safe identifiers and redacted states.
|
||||
- `crates/kms`
|
||||
- Role: key management service integration.
|
||||
- Logging focus: init/health/fallback, key-source availability, never log key material.
|
||||
- `crates/crypto`
|
||||
- Role: cryptographic helpers and security primitives.
|
||||
- Logging focus: only algorithm or mode state, not plaintext, ciphertext, or secret-derived material.
|
||||
- `crates/security-governance`
|
||||
- Role: security governance contracts.
|
||||
- Logging focus: policy/state transitions and enforcement diagnostics.
|
||||
- `crates/signer`
|
||||
- Role: request signing helpers.
|
||||
- Logging focus: signature validation failures without expected-signature leakage.
|
||||
|
||||
### Notifications, Audit, And Targets
|
||||
|
||||
- `crates/notify`
|
||||
- Role: notification dispatch, runtime facade, notifier implementations.
|
||||
- Logging focus: target lifecycle, dispatch summaries, stream lag/backpressure, avoid per-event success spam.
|
||||
- `crates/audit`
|
||||
- Role: audit target fan-out and audit pipeline management.
|
||||
- Logging focus: pipeline lifecycle, target availability, batch dispatch summaries, avoid noisy "started successfully" prose.
|
||||
- `crates/targets`
|
||||
- Role: target-specific configuration and utilities used by fan-out style systems.
|
||||
- Logging focus: target selection, config validation, per-target degraded state.
|
||||
- `crates/s3-types`
|
||||
- Role: S3 event and type definitions.
|
||||
- Logging focus: usually minimal; keep logging at integration boundaries rather than low-level type crates.
|
||||
- `crates/s3-ops`
|
||||
- Role: S3 operation definitions and mapping.
|
||||
- Logging focus: mapping/contract failures, unsupported combinations, not normal-path request spam.
|
||||
|
||||
### Concurrency, Locking, And Runtime Foundations
|
||||
|
||||
- `crates/concurrency`
|
||||
- Role: timeout, locking, backpressure, and I/O scheduling facade.
|
||||
- Logging focus: lifecycle transitions and degraded states, not high-frequency worker/permit churn at `info`.
|
||||
- `crates/lock`
|
||||
- Role: distributed locking implementation.
|
||||
- Logging focus: lock lifecycle, contention anomalies, lock ordering or timeout diagnostics.
|
||||
- `crates/tls-runtime`
|
||||
- Role: shared TLS runtime foundation.
|
||||
- Logging focus: certificate lifecycle, reload/fallback, validation failures without sensitive dumps.
|
||||
- `crates/obs`
|
||||
- Role: observability helpers.
|
||||
- Logging focus: this crate shapes other crates' telemetry conventions; avoid recursive or redundant summaries.
|
||||
- `crates/io-core`
|
||||
- Role: zero-copy I/O core primitives.
|
||||
- Logging focus: keep very sparse; prefer metrics unless failures are actionable.
|
||||
- `crates/io-metrics`
|
||||
- Role: I/O metrics collection.
|
||||
- Logging focus: typically minimal; metrics should carry the hot-path signal.
|
||||
- `crates/rio`
|
||||
- Role: Rust I/O utility layer.
|
||||
- Logging focus: compatibility or runtime boundary failures, not fast-path internals.
|
||||
- `crates/rio-v2`
|
||||
- Role: next-generation I/O compatibility layer.
|
||||
- Logging focus: migration/feature-mode differences and degraded fallback between I/O paths.
|
||||
- `crates/utils`
|
||||
- Role: shared helpers.
|
||||
- Logging focus: usually avoid direct logging in generic helpers unless the helper is itself an operational boundary.
|
||||
- `crates/common`
|
||||
- Role: shared data structures and helpers.
|
||||
- Logging focus: same principle as `utils`; prefer callers to log context-rich events.
|
||||
- `crates/config`
|
||||
- Role: configuration management.
|
||||
- Logging focus: config source, fallback, validation, and summary aggregation; avoid dumping merged configs.
|
||||
- `crates/data-usage`
|
||||
- Role: shared data usage models and algorithms.
|
||||
- Logging focus: refresh lifecycle, summary stats, and degraded reads.
|
||||
|
||||
### Schema, Contracts, And API Support
|
||||
|
||||
- `crates/protos`
|
||||
- Role: protobuf definitions.
|
||||
- Logging focus: usually none inside the crate; emit logs at decode/use boundaries.
|
||||
- `crates/extension-schema`
|
||||
- Role: extension schema contracts.
|
||||
- Logging focus: schema validation and compatibility mismatches.
|
||||
- `crates/s3select-api`
|
||||
- Role: S3 Select API interfaces.
|
||||
- Logging focus: request validation and unsupported feature boundaries.
|
||||
- `crates/s3select-query`
|
||||
- Role: S3 Select query engine.
|
||||
- Logging focus: query parse/planning/execution failures, avoid row-level spam.
|
||||
- `crates/protocols`
|
||||
- Role: non-S3 protocol support.
|
||||
- Logging focus: see core server section; keep per-request verbosity below `info`.
|
||||
|
||||
### Testing And Non-Production Crates
|
||||
|
||||
- `crates/e2e_test`
|
||||
- Role: end-to-end tests.
|
||||
- Logging focus: test clarity matters more than production governance, but avoid copying test-only logging style into production crates.
|
||||
|
||||
## Current Guardrail Coverage Map
|
||||
|
||||
`scripts/check_logging_guardrails.sh` currently enforces retired patterns in these high-signal areas:
|
||||
|
||||
- `rustfs/src/main.rs`
|
||||
- `rustfs/src/startup_iam.rs`
|
||||
- `rustfs/src/auth.rs`
|
||||
- `rustfs/src/protocols/client.rs`
|
||||
- `crates/audit/src/pipeline.rs`
|
||||
- `crates/audit/src/system.rs`
|
||||
- `crates/audit/src/global.rs`
|
||||
- `crates/notify/src/config_manager.rs`
|
||||
- `crates/notify/src/runtime_facade.rs`
|
||||
- `crates/notify/src/notifier.rs`
|
||||
- `crates/ecstore/src/store/peer.rs`
|
||||
- `crates/ecstore/src/store/init.rs`
|
||||
- `crates/ecstore/src/tier/tier.rs`
|
||||
- `crates/concurrency/src/workers.rs`
|
||||
- `crates/concurrency/src/manager.rs`
|
||||
- `crates/concurrency/src/lock.rs`
|
||||
- `crates/concurrency/src/deadlock.rs`
|
||||
- `crates/trusted-proxies/src/global.rs`
|
||||
- `crates/trusted-proxies/src/config/loader.rs`
|
||||
- `crates/trusted-proxies/src/proxy/metrics.rs`
|
||||
- `crates/trusted-proxies/src/proxy/validator.rs`
|
||||
- `crates/trusted-proxies/src/proxy/chain.rs`
|
||||
- `crates/trusted-proxies/src/middleware/service.rs`
|
||||
- `crates/trusted-proxies/src/cloud/detector.rs`
|
||||
- `crates/trusted-proxies/src/cloud/ranges.rs`
|
||||
- `crates/trusted-proxies/src/cloud/metadata/aws.rs`
|
||||
- `crates/trusted-proxies/src/cloud/metadata/azure.rs`
|
||||
- `crates/trusted-proxies/src/cloud/metadata/gcp.rs`
|
||||
|
||||
When expanding coverage, prefer crates with:
|
||||
|
||||
- repeated sentence-style lifecycle logs
|
||||
- high-frequency success-path `info!`
|
||||
- startup/config checklist banners
|
||||
- security-sensitive fallback wording
|
||||
- external fetch/retry/fallback flows
|
||||
|
||||
That typically means the next broad candidates are `rustfs`, `crates/notify`, `crates/audit`, `crates/targets`, `crates/heal`, and `crates/scanner`.
|
||||
|
||||
## Event Model
|
||||
|
||||
Prefer this structure when the fields are available:
|
||||
|
||||
- `event`
|
||||
- `component`
|
||||
- `subsystem`
|
||||
- `state` or `result`
|
||||
- stable context fields such as:
|
||||
- `enabled`
|
||||
- `implementation`
|
||||
- `validation_mode`
|
||||
- `peer_ip`
|
||||
- `client_ip`
|
||||
- `proxy_hops`
|
||||
- `duration_ms`
|
||||
- `fallback`
|
||||
- `reason`
|
||||
- `range_count`
|
||||
- `hold_time_ms`
|
||||
- `available_slots`
|
||||
- `total_slots`
|
||||
- `permits_in_use`
|
||||
|
||||
## Level Policy
|
||||
|
||||
- `error`: the operation fails and callers or security guarantees are affected.
|
||||
- `warn`: a degraded path, fallback, suspicious request, or operator-actionable config issue occurs.
|
||||
- `info`: a low-frequency lifecycle or mode transition occurs.
|
||||
- `debug`: useful diagnostics exist but normal operators do not need them all the time.
|
||||
- `trace`: hot-path and repetitive success-path details occur.
|
||||
|
||||
## Preferred Patterns
|
||||
|
||||
- Use a short message label:
|
||||
- `"trusted proxy validation failed"`
|
||||
- `"concurrency manager state changed"`
|
||||
- `"trusted proxy cloud metadata loaded"`
|
||||
- Put key meaning into fields, not only the message text.
|
||||
- Aggregate config or metrics summaries into one log event.
|
||||
|
||||
## Retired Patterns
|
||||
|
||||
These should usually be removed or replaced:
|
||||
|
||||
- Sentence-style lifecycle logs:
|
||||
- `info!("Concurrency manager stopped")`
|
||||
- `info!("Trusted Proxies module initialized")`
|
||||
- Checklist or banner logs:
|
||||
- `info!("=== Application Configuration ===")`
|
||||
- `info!("Available metrics:")`
|
||||
- Hot-path noise:
|
||||
- `info!("worker take, {}", *available)`
|
||||
- `debug!("Proxy validation successful in {:?}", duration)`
|
||||
- Legacy fallback prose:
|
||||
- `"Request from private network but not trusted: ..."`
|
||||
- `"Cloud metadata fetching is disabled"`
|
||||
|
||||
## Guardrail Update Checklist
|
||||
|
||||
When extending `scripts/check_logging_guardrails.sh`:
|
||||
|
||||
1. Add the touched files to `checked_files`.
|
||||
2. Add only legacy patterns that have been intentionally retired.
|
||||
3. Keep patterns literal and grep-friendly.
|
||||
4. Run the guardrail script after changes.
|
||||
5. Avoid adding patterns for logs that are still valid elsewhere in the repo.
|
||||
|
||||
## Validation Checklist
|
||||
|
||||
For logging-only changes:
|
||||
|
||||
```bash
|
||||
cargo fmt --all --check
|
||||
./scripts/check_logging_guardrails.sh
|
||||
cargo check -p <affected-crate>
|
||||
cargo test -p <affected-crate>
|
||||
```
|
||||
|
||||
For broader Rust changes:
|
||||
|
||||
```bash
|
||||
./scripts/check_unsafe_code_allowances.sh
|
||||
./scripts/check_architecture_migration_rules.sh
|
||||
cargo clippy -p <affected-crates> --all-targets -- -D warnings
|
||||
```
|
||||
Reference in New Issue
Block a user