mirror of
https://github.com/rustfs/rustfs.git
synced 2026-07-26 16:28:15 +00:00
00536da80c
* refactor(obs): make dial9 telemetry opt-in and actually record events
The dial9 Tokio-runtime profiler was disabled by default, yet every build
paid for it, and enabling it produced trace files with no events in them.
Recorded empty traces
---------------------
`build_traced_runtime` called `TracedRuntime::builder()...build(..)`, but dial9
only starts recording in `build_and_start*`. `build` still returns a live guard
whose `is_enabled()` reports true, and still creates and seals segment files —
they just contain a header and no events. It also skipped `with_trace_path`, so
the background worker driving the segment pipeline was never spawned.
Measured on the new smoke example: 310 bytes of bare segment header, against
5640 bytes for the same workload once recording actually starts.
Switch to `with_trace_path(..).build_and_start(..)`.
Cost was unconditional
----------------------
`--cfg tokio_unstable` was a global `[build] rustflags` entry and `rustfs-obs`
depended on `dial9-tokio-telemetry` unconditionally, so all builds depended on
Tokio's non-semver API. Worse, an environment `RUSTFLAGS` replaces (never
appends to) the config-file value, so any caller exporting their own RUSTFLAGS
silently dropped the flag — the long comment in build.yml was a scar from that.
dial9 is now an opt-in feature (`dial9`, plus `dial9-s3` and `dial9-taskdump`),
the global rustflag is gone, and `crates/obs/build.rs` fails the compile if the
feature is on without the flag. Telemetry builds go through `make build-profiling`.
Metrics that could not lie
--------------------------
`rustfs_dial9_{events_total,bytes_written_total,rotations_total,cpu_overhead_percent}`
were hard-coded to zero — a Counter pinned at 0 reads as "nothing happened".
Removed. `rustfs_dial9_enabled` was sourced from the environment, so it read 1
even when the traced runtime failed and the process fell back to a standard
runtime; it is replaced by `rustfs_dial9_supported` (compile-time),
`rustfs_dial9_configured` (intent) and `rustfs_dial9_active_sessions` (reality).
No `writer_healthy` gauge is exported: dial9's `RotatingWriter` can enter its
`Finished` state and stop writing, but exposes no way to observe that, so the
gauge could only ever be hard-coded to 1. Documented as a known gap instead.
Final events were lost
----------------------
The `TelemetryGuard` lived in a `static OnceLock`, which is never dropped, so
buffered events were never flushed at exit. `build_tokio_runtime` now returns
the guard and `run_process` drops it before any exit path.
Also
----
- `disk_usage_bytes` was a `read_dir` + per-file `stat` on the metrics
collection path. It is now sampled by a background task into an atomic.
- `SAMPLING_RATE`/`S3_BUCKET`/`S3_PREFIX` were parsed, warned about, and
discarded. S3 upload is now wired to dial9's `with_s3_uploader` behind
`dial9-s3`; `SAMPLING_RATE` has no upstream equivalent and is removed.
- Wire `with_task_dumps` (async backtraces of stalled tasks), configurable via
`RUSTFS_RUNTIME_DIAL9_TASK_DUMP_{ENABLED,IDLE_THRESHOLD_MS}`.
- Split `telemetry/dial9.rs` into `config`/`state`/`enabled`/`disabled`; the
stub keeps the public API identical so callers need no `#[cfg]`.
- Drop four print-only examples and the manual test bin that exercised the
removed `init_session` scaffolding.
Verified: cargo check/clippy/test across default, `dial9`, and `dial9-s3`;
build.rs correctly rejects `dial9` without `--cfg tokio_unstable`;
`make pre-commit` passes.
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(obs): document dial9 as an on-demand profiler
scripts/run.sh advertised a `SAMPLING_RATE` knob that was never passed to dial9,
and claimed "CPU overhead < 5% (with sampling rate 1.0)" and "lower values reduce
CPU overhead" on the strength of it. The knob is gone; the guidance built on it
had to go too.
Replace it with what is actually true: dial9 needs a `make build-profiling`
binary, its disk budget evicts oldest-first (so a high poll rate can overwrite
the incident you are chasing), and it cannot be toggled without a restart.
Add docs/operations/dial9-runtime-profiling.md covering the build variants, an
investigation walkthrough, the configuration table, how to read the three
supported/configured/active_sessions gauges against each other, and the upstream
gap that makes writer death only indirectly observable.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(obs): add a dial9 smoke example that proves events are recorded
The bug this guards against is invisible to every existing signal: with
`build` instead of `build_and_start`, dial9 creates the trace file, seals
segments, and reports `TelemetryGuard::is_enabled() == true` — it simply
records no events. Only the segment's byte count tells the two apart.
Measured on this workload: 5640 bytes when recording, 310 bytes (a bare
segment header) when not. The example asserts >= 2048 bytes, and was verified
to fail with the `build` call restored.
Also correct the comment on the `is_enabled` check in `finish_traced_runtime`.
It claimed to catch "recording silently off"; it does not. It only rejects the
inert guard a lenient config yields after a build failure. Recording is
guaranteed by `build_and_start`, not by that check.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(rustfs): accept Unsupported runtime telemetry capability
A binary built without the `dial9` feature now reports the runtime-telemetry
capability as `Unsupported` rather than `Disabled`. The distinction matters to
operators: `Disabled` implies the capability can be switched on by setting an
environment variable, which is not true here — telemetry needs a rebuild.
Widen the assertion and pin the new semantics: when `dial9::is_supported()` is
false, the state must be exactly `Unsupported`.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs): drop the dial9-s3 feature, its TLS stack is vulnerable
CI's Dependency Review and `cargo deny` both reject the branch: dial9's
`worker-s3` feature depends on aws-sdk-s3-transfer-manager 0.1.3, which pins
aws-smithy-http-client onto hyper-rustls 0.24 and rustls-webpki 0.101.7. That
webpki carries RUSTSEC-2026-0098, -0099 and -0104.
0.1.3 is the latest release of the transfer manager, and 1.2.0 the latest of the
smithy client, so there is nothing to upgrade to. Cargo's feature unification can
add features but cannot drop a transitive dependency, so it cannot be worked
around from here either — the rest of the workspace already resolves to the safe
rustls-webpki 0.103 / hyper-rustls 0.27.
Remove the `dial9-s3` feature and the `with_s3_uploader` wiring. The two S3
environment variables stay parsed and warned about, now naming the real reason
rather than a missing build feature. Trace segments are collected from the output
directory instead. Tracked as D9-14 in rustfs/backlog#1157.
With this, Cargo.lock is byte-identical to main: the PR no longer touches the
dependency graph at all.
Also correct the `dial9-taskdump` documentation. It claimed the feature "compiles
to a no-op elsewhere"; in fact `tokio/taskdump` raises a `compile_error!` on any
target other than linux/{aarch64,x86,x86_64}. Verified by trying to build it on
macOS, which is how the claim was found to be wrong.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
657 lines
33 KiB
Bash
Executable File
657 lines
33 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
set -euo pipefail
|
|
|
|
repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
|
cd "$repo_root"
|
|
|
|
checked_files=(
|
|
"rustfs/src/main.rs"
|
|
"rustfs/src/init.rs"
|
|
"rustfs/src/profiling.rs"
|
|
"rustfs/src/startup_iam.rs"
|
|
"rustfs/src/auth.rs"
|
|
"rustfs/src/protocols/client.rs"
|
|
"rustfs/src/admin/router.rs"
|
|
"rustfs/src/admin/handlers/table_catalog.rs"
|
|
"rustfs/src/admin/handlers/service_account.rs"
|
|
"rustfs/src/admin/handlers/kms_dynamic.rs"
|
|
"rustfs/src/admin/handlers/site_replication.rs"
|
|
"rustfs/src/admin/handlers/group.rs"
|
|
"rustfs/src/admin/handlers/quota.rs"
|
|
"rustfs/src/admin/handlers/rebalance.rs"
|
|
"rustfs/src/admin/handlers/tier.rs"
|
|
"rustfs/src/admin/handlers/event.rs"
|
|
"rustfs/src/admin/handlers/audit.rs"
|
|
"rustfs/src/admin/handlers/pools.rs"
|
|
"rustfs/src/admin/handlers/system.rs"
|
|
"rustfs/src/storage/rpc/http_service.rs"
|
|
"rustfs/src/storage/rpc/node_service.rs"
|
|
"crates/audit/src/pipeline.rs"
|
|
"crates/audit/src/system.rs"
|
|
"crates/audit/src/global.rs"
|
|
"crates/notify/src/config_manager.rs"
|
|
"crates/notify/src/runtime_facade.rs"
|
|
"crates/notify/src/integration.rs"
|
|
"crates/notify/src/notifier.rs"
|
|
"crates/notify/src/bucket_config_manager.rs"
|
|
"crates/notify/src/rule_engine.rs"
|
|
"crates/notify/src/global.rs"
|
|
"crates/audit/src/registry.rs"
|
|
"crates/audit/src/observability.rs"
|
|
"crates/targets/src/store.rs"
|
|
"crates/targets/src/target/mqtt.rs"
|
|
"crates/targets/src/target/webhook.rs"
|
|
"crates/ecstore/src/store/peer.rs"
|
|
"crates/ecstore/src/store/init.rs"
|
|
"crates/ecstore/src/services/tier/tier.rs"
|
|
"crates/heal/src/heal/manager.rs"
|
|
"crates/heal/src/heal/storage.rs"
|
|
"crates/heal/src/heal/task.rs"
|
|
"crates/heal/src/heal/erasure_healer.rs"
|
|
"crates/heal/src/heal/resume.rs"
|
|
"crates/heal/src/heal/channel.rs"
|
|
"crates/scanner/src/scanner.rs"
|
|
"crates/scanner/src/scanner_io.rs"
|
|
"crates/scanner/src/scanner_folder.rs"
|
|
"crates/concurrency/src/workers.rs"
|
|
"crates/concurrency/src/deadlock.rs"
|
|
"crates/trusted-proxies/src/global.rs"
|
|
"crates/trusted-proxies/src/config/loader.rs"
|
|
"crates/trusted-proxies/src/proxy/metrics.rs"
|
|
"crates/trusted-proxies/src/proxy/validator.rs"
|
|
"crates/trusted-proxies/src/proxy/chain.rs"
|
|
"crates/trusted-proxies/src/middleware/service.rs"
|
|
"crates/trusted-proxies/src/cloud/detector.rs"
|
|
"crates/trusted-proxies/src/cloud/ranges.rs"
|
|
"crates/trusted-proxies/src/cloud/metadata/aws.rs"
|
|
"crates/trusted-proxies/src/cloud/metadata/azure.rs"
|
|
"crates/trusted-proxies/src/cloud/metadata/gcp.rs"
|
|
"crates/protocols/src/webdav/server.rs"
|
|
"crates/protocols/src/webdav/driver.rs"
|
|
"crates/protocols/src/ftps/server.rs"
|
|
"crates/protocols/src/ftps/driver.rs"
|
|
"crates/protocols/src/sftp/server.rs"
|
|
"crates/protocols/src/sftp/driver.rs"
|
|
"crates/protocols/src/sftp/write.rs"
|
|
"crates/protocols/src/sftp/config.rs"
|
|
"crates/protocols/src/sftp/errors.rs"
|
|
"crates/protocols/src/sftp/dir.rs"
|
|
"crates/protocols/src/sftp/fallback_watchdog.rs"
|
|
"crates/protocols/src/sftp/wedge_watchdog.rs"
|
|
"crates/protocols/src/swift/expiration_worker.rs"
|
|
"crates/protocols/src/swift/versioning.rs"
|
|
"crates/protocols/src/swift/bulk.rs"
|
|
"crates/protocols/src/swift/handler.rs"
|
|
"crates/protocols/src/swift/staticweb.rs"
|
|
"crates/protocols/src/swift/acl.rs"
|
|
"crates/protocols/src/swift/sync.rs"
|
|
"crates/protocols/src/swift/container.rs"
|
|
"crates/protocols/src/swift/object.rs"
|
|
"crates/protocols/src/swift/formpost.rs"
|
|
"crates/protocols/src/swift/ratelimit.rs"
|
|
"crates/protocols/src/swift/expiration.rs"
|
|
"crates/protocols/src/swift/cors.rs"
|
|
"crates/protocols/src/swift/quota.rs"
|
|
"crates/protocols/src/swift/symlink.rs"
|
|
"crates/protocols/src/common/gateway.rs"
|
|
"crates/obs/src/telemetry/dial9/config.rs"
|
|
"crates/obs/src/telemetry/dial9/enabled.rs"
|
|
"crates/obs/src/telemetry/local.rs"
|
|
"crates/obs/src/metrics/scheduler.rs"
|
|
"crates/obs/src/cleaner/core.rs"
|
|
"crates/obs/src/cleaner/compress.rs"
|
|
"crates/obs/src/cleaner/scanner.rs"
|
|
"crates/obs/src/metrics/stats_collector.rs"
|
|
"crates/obs/src/global.rs"
|
|
"crates/obs/src/telemetry/recorder.rs"
|
|
"crates/obs/src/metrics/collectors/system_gpu.rs"
|
|
)
|
|
|
|
missing_files=()
|
|
for file in "${checked_files[@]}"; do
|
|
if [[ ! -f "$file" ]]; then
|
|
missing_files+=("$file")
|
|
fi
|
|
done
|
|
|
|
if ((${#missing_files[@]} > 0)); then
|
|
echo "❌ logging guardrail checked file is missing:" >&2
|
|
printf ' %s\n' "${missing_files[@]}" >&2
|
|
exit 1
|
|
fi
|
|
|
|
forbidden_patterns=(
|
|
'access_key={}'
|
|
'secret_key={}'
|
|
'Authorization={}'
|
|
'token={}'
|
|
'debug!("config: {:?}"'
|
|
'warn!("No audit targets configured for dispatch"'
|
|
'warn!("No audit targets configured for batch dispatch"'
|
|
'info!("Event stream processing for target {} is started successfully"'
|
|
'info!("Target {} has no replay worker to start"'
|
|
'info!("Sending event to targets: {:?}"'
|
|
'info!("Event processing initiated for {} targets for bucket: {}"'
|
|
'warn!("{}", notify_configuration_hint())'
|
|
'info!("Available ARNs: {:?}"'
|
|
'info!("Loaded notification config for bucket: {}"'
|
|
'info!("Updated notification rules for bucket: {}"'
|
|
'info!("Removed all notification rules for bucket: {}"'
|
|
'info!(event = EVENT_NOTIFY_RUNTIME_LIFECYCLE,'
|
|
'info!("Notification system instance is being dropped"'
|
|
'info!("Notification shutdown metric snapshot"'
|
|
'info!("Notification system status snapshot"'
|
|
'debug!("Audit replay stream skipped"'
|
|
'warn!("Dropped queued audit payload"'
|
|
'error!("Queued audit payload failed permanently"'
|
|
'info!(target_id = %target_id.id, "Webhook target created")'
|
|
'info!("Webhook target {} initialized"'
|
|
'info!("Webhook sending queued payload to target: {}"'
|
|
'debug!("Event saved to store for target: {}"'
|
|
'debug!("Event sent from store and deleted for target: {}"'
|
|
'info!("Webhook target closed: {}"'
|
|
'info!(target_id = %target_id, "MQTT target created")'
|
|
'info!(target_id = %self.id, "MQTT target initialized and connected.")'
|
|
'info!(target_id = %target_id, "MQTT event loop task started.")'
|
|
'info!(target_id = %self.id, "MQTT target close method finished.")'
|
|
'debug!("Wrote event to store: {}"'
|
|
'debug!("Deleted event from store: {}"'
|
|
'info!("Audit configuration reloaded"'
|
|
'info!("Audit system started"'
|
|
'info!("Audit metrics reset"'
|
|
'error!("Failed to set global observability guard: {}"'
|
|
'error!("Failed to initialize TLS from {}: {}"'
|
|
'error!("Server encountered an error and is shutting down: {}"'
|
|
'error!("Failed to initialize Keystone authentication: {}"'
|
|
'error!("new_global_notification_sys failed {:?}"'
|
|
'warn!(access_key = %new_cred.access_key, error = ?err, "site replication add service account hook failed")'
|
|
'warn!(access_key = %query.access_key, error = ?err, "site replication delete service account hook failed")'
|
|
'debug!("check key failed: {e:?}")'
|
|
'debug!("list service account failed: {e:?}")'
|
|
'debug!("list sts account failed: {e:?}")'
|
|
'error!("license check failed due to unexpected error: {er}")'
|
|
'info!("Created KMS key: {}"'
|
|
'info!("Successfully deleted KMS key: {}"'
|
|
'info!("Cancelled deletion for KMS key: {}"'
|
|
'info!("Listed {} KMS keys"'
|
|
'info!("Described KMS key: {}"'
|
|
'error!("Failed to create KMS key: {}"'
|
|
'error!("Failed to list KMS keys: {}"'
|
|
'error!("Failed to describe KMS key {}: {}"'
|
|
'warn!(access_key = %ak, error = ?err, "site replication create user hook failed")'
|
|
'info!("OIDC authorize redirect for provider'
|
|
'warn!("OIDC callback received error from IdP:'
|
|
'error!("OIDC code exchange failed: {}"'
|
|
'warn!("OIDC logout fallback triggered: {}"'
|
|
'warn!("get body failed, e: {:?}")'
|
|
'warn!("file path is invalid: {}")'
|
|
'warn!("bucket metadata not found: {e}")'
|
|
'warn!("get bucket metadata failed: {e}")'
|
|
'warn!("create bucket failed: {e}")'
|
|
'warn!("deserialize config failed: {e}")'
|
|
'warn!(policy = %query.name, error = ?err, "site replication policy add hook failed")'
|
|
'warn!(policy = %query.name, error = ?err, "site replication policy delete hook failed")'
|
|
'warn!(target = %query.user_or_group, error = ?err, "site replication policy mapping hook failed")'
|
|
'warn!(access_key = %ak, error = ?err, "site replication delete user hook failed")'
|
|
'warn!(target = %target_name, error = ?err, "site replication policy association hook failed")'
|
|
'error!("Failed to serialize response: {}"'
|
|
'warn!(peer = %peer.endpoint, error = ?err, "site replication peer metainfo fetch failed")'
|
|
'warn!(error = ?err, "site replication backfill: failed to list buckets")'
|
|
'warn!(bucket = %name, error = ?err, "site replication backfill: versioning setup failed")'
|
|
'warn!(bucket = %name, error = ?err, "site replication backfill: targets setup failed")'
|
|
'warn!(bucket = %name, error = ?err, "site replication backfill: replication config setup failed")'
|
|
'warn!(bucket = %name, error = ?err, "site replication backfill: failed to read bucket metadata, assuming lock_enabled=false")'
|
|
'warn!(bucket = %name, error = ?err, "site replication backfill: make-bucket broadcast failed")'
|
|
'warn!(bucket = %name, peer = %peer.endpoint, detail = %result.err_detail,'
|
|
'warn!(peer = %peer.endpoint, error = %err_detail, "site replication peer remove notification failed")'
|
|
'warn!(peer = %peer.endpoint, error = %detail, "site replication service account rotation failed for peer")'
|
|
'warn!("handle ListGroups")'
|
|
'warn!("handle GetGroup")'
|
|
'warn!("handle DeleteGroup")'
|
|
'warn!("handle SetGroupStatus")'
|
|
'warn!("handle UpdateGroupMembers")'
|
|
'warn!("list groups failed, e: {:?}")'
|
|
'warn!("get group failed, e: {:?}")'
|
|
'warn!("delete group failed, e: {:?}")'
|
|
'warn!("site replication group delete hook failed, err: {err}")'
|
|
'warn!("enable group failed, e: {:?}")'
|
|
'warn!("UpdateGroupMembers args {:?}")'
|
|
'warn!("remove group members")'
|
|
'warn!("remove group members failed, e: {:?}")'
|
|
'warn!("add group members")'
|
|
'warn!("add group members failed, e: {:?}")'
|
|
'warn!("site replication group membership hook failed, err: {err}")'
|
|
'warn!("handle SetBucketQuota")'
|
|
'warn!("handle GetBucketQuota")'
|
|
'warn!("handle ClearBucketQuota")'
|
|
'warn!("handle GetBucketQuotaStats")'
|
|
'warn!("handle CheckBucketQuota")'
|
|
'info!("Clearing quota for bucket: {}")'
|
|
'info!("Successfully cleared quota for bucket: {}")'
|
|
'debug!("Checking quota for bucket: {}, operation: {}, size: {}")'
|
|
'warn!("handle RebalanceStart")'
|
|
'warn!("Rebalance started with id: {}")'
|
|
'warn!("RebalanceStart Loading rebalance meta start")'
|
|
'warn!("rebalance start propagation failed after local state update: {err}")'
|
|
'warn!("RebalanceStart Loading rebalance meta done")'
|
|
'warn!("handle RebalanceStatus")'
|
|
'warn!("handle RebalanceStop")'
|
|
'warn!("handle RebalanceStop save_rebalance_stats done ")'
|
|
'warn!("handle RebalanceStop notification_sys load_rebalance_meta")'
|
|
'warn!("rebalance stop propagation failed after local state update: {err}")'
|
|
'warn!("handle RebalanceStop notification_sys load_rebalance_meta done")'
|
|
'warn!("handle ListPools")'
|
|
'warn!("handle StatusPool")'
|
|
'warn!("handle StartDecommission")'
|
|
'warn!("handle CancelDecommission")'
|
|
'warn!("specified pool {} not found, please specify a valid pool", &query.pool)'
|
|
'warn!("handle ServiceHandle")'
|
|
'warn!("handle InspectDataHandler")'
|
|
'warn!("handle StorageInfoHandler")'
|
|
'warn!("handle DataUsageInfoHandler")'
|
|
'warn!("failed to read request body: {:?}")'
|
|
'info!("Setting target config for type '\''{}'\'', name '\''{}'\''", target_type, target_name)'
|
|
'info!("Removing target config for type '\''{}'\'', name '\''{}'\''", target_type, target_name)'
|
|
'warn!("{message}")'
|
|
'warn!(error = %e, "walk_dir failed")'
|
|
'info!("handling load_rebalance_meta request")'
|
|
'info!("load_rebalance_meta completed")'
|
|
'info!(start_rebalance, "spawning background rebalance task")'
|
|
'error!(error = %message, "background rebalance start failed")'
|
|
'warn!("get body failed, e: {:?}")'
|
|
'debug!("add tier args {:?}")'
|
|
'warn!("parse force failed, e: {:?}")'
|
|
'warn!("tier reserved name, args.name: {}")'
|
|
'warn!("tier_config_mgr reload failed, e: {:?}")'
|
|
'warn!("tier_config_mgr add failed, e: {:?}")'
|
|
'debug!("edit tier args {:?}")'
|
|
'warn!("tier_config_mgr edit failed, e: {:?}")'
|
|
'warn!("tier_config_mgr remove failed, e: {:?}")'
|
|
'warn!("tier_config_mgr rand: {}")'
|
|
'warn!("tier_config_mgr clear failed, e: {:?}")'
|
|
'warn!("tier_config_mgr save failed, e: {:?}")'
|
|
'event = "object_lambda_tls_verification_disabled"'
|
|
'info!("FTP system initialized successfully"'
|
|
'debug!("FTP system is disabled")'
|
|
'info!("FTP system disabled"'
|
|
'info!("FTP server shutdown completed")'
|
|
'error!("Failed to initialize FTP system: {}"'
|
|
'info!("FTPS system initialized successfully"'
|
|
'debug!("FTPS system is disabled")'
|
|
'info!("FTPS system disabled"'
|
|
'info!("FTPS server shutdown completed")'
|
|
'error!("Failed to initialize FTPS system: {}"'
|
|
'info!("WebDAV system initialized successfully"'
|
|
'debug!("WebDAV system is disabled")'
|
|
'info!("WebDAV system disabled"'
|
|
'info!("WebDAV server shutdown completed")'
|
|
'error!("Failed to initialize WebDAV system: {}"'
|
|
'info!("SFTP system initialized successfully"'
|
|
'debug!("SFTP system is disabled")'
|
|
'info!("SFTP system disabled"'
|
|
'info!("SFTP server shutdown completed")'
|
|
'error!("Failed to initialize SFTP system: {}"'
|
|
'warn!("prewarm_local_disk_id_map: failed to load disk id for {}: {}"'
|
|
'info!("retrying get formats after {:?}"'
|
|
'info!("📝 Release notes: {}"'
|
|
'info!("🔗 Download URL: {}"'
|
|
'debug!("✅ Version check: Current version is up to date: {}"'
|
|
'warn!("get_notification_config err {:?}"'
|
|
'error!("ecstore config::init_global_config_sys failed {:?}"'
|
|
'info!(target: "rustfs::main::startup", "RustFS API: {api_endpoints} {localhost_endpoint}")'
|
|
'info!("virtual-hosted-style requests are enabled use domain_name {:?}"'
|
|
'info!("HTTP response compression enabled: extensions={:?}, mime_patterns={:?}, min_size={} bytes"'
|
|
'warn!("handle ListCannedPolicies")'
|
|
'warn!("handle AddCannedPolicy")'
|
|
'warn!("handle InfoCannedPolicy")'
|
|
'warn!("handle RemoveCannedPolicy")'
|
|
'warn!("handle SetPolicyForUserOrGroup")'
|
|
'warn!("handle Trace")'
|
|
'warn!("handle AddServiceAccount ")'
|
|
'warn!("handle UpdateServiceAccount")'
|
|
'warn!("handle InfoServiceAccount")'
|
|
'warn!("handle TemporaryAccountInfo")'
|
|
'warn!("handle InfoAccessKey")'
|
|
'warn!("handle ListServiceAccount")'
|
|
'warn!("handle ListAccessKeysBulk")'
|
|
'warn!("handle DeleteServiceAccount")'
|
|
'debug!("{action}, e: {:?}")'
|
|
'warn!("list policies failed, e: {:?}")'
|
|
'error!("Invalid JSON in configure request: {}")'
|
|
'warn!("get_sse_config err {:?}")'
|
|
'warn!("get_cors_config err {:?}")'
|
|
'warn!("get_notification_config err {:?}")'
|
|
'warn!("get_tagging_config err {:?}")'
|
|
'warn!("get_object_lock_config err {:?}")'
|
|
'info!("Starting get_object_tagging for bucket: {}, object: {}")'
|
|
'debug!("bucket_sse_config_result={:?}")'
|
|
'debug!("TDD: bucket_sse_config={:?}")'
|
|
'info!("Keystone authentication disabled")'
|
|
'info!("Initializing Keystone authentication...")'
|
|
'info!("Keystone authentication initialized successfully")'
|
|
'info!("Created TLS acceptor with root single certificate")'
|
|
'info!("TLS certificate hot reload enabled, checking every {}s")'
|
|
'info!("Initializing capacity management system...")'
|
|
'info!("Capacity management system initialized successfully")'
|
|
'info!("CPU profile exported: {}")'
|
|
'warn!("failed to reload current server config for object lambda request: {err}")'
|
|
'warn!("failed to fetch live events from peer {}: {err}")'
|
|
'warn!("timed out fetching live events from peer {}")'
|
|
'warn!("failed to serialize remote listen notification event: {err}")'
|
|
'warn!("failed to decode live events from peer {}: {err}")'
|
|
'warn!("failed to serialize listen notification event: {err}")'
|
|
'warn!("listen notification stream lagged and skipped {skipped} events")'
|
|
'debug!("Ignoring crypto provider installation error: {err:?}")'
|
|
'warn!("KMS initialization skipped: {e}")'
|
|
'warn!("Audit system: {e}")'
|
|
'warn!("notification system: {e}")'
|
|
'info!("jemalloc profiling controller is NOT available")'
|
|
'warn!("periodic CPU profiler create failed: {e}")'
|
|
'warn!("write periodic CPU pprof failed: {e}")'
|
|
'warn!("periodic CPU report build failed: {e}")'
|
|
'debug!("profiling: disabled by env")'
|
|
'debug!("profiling: CPU mode off")'
|
|
'info!("Starting HealManager")'
|
|
'info!("Stopping HealManager")'
|
|
'info!("HealManager started successfully")'
|
|
'info!("HealManager stopped successfully")'
|
|
'info!("Healing MRF: {}")'
|
|
'info!("Step 1: Performing MRF heal using ecstore")'
|
|
'info!("Skipping initial data scanner delay because persisted data usage cache is cold")'
|
|
'error!("Failed to run data scanner: {e}")'
|
|
'warn!("Failed to read background heal info from {}: {}")'
|
|
'error!("Failed to marshal background heal info: {}")'
|
|
'warn!("Failed to save background heal info to {}: {}")'
|
|
'debug!("nsscanner_cache: no online disks available for set")'
|
|
'debug!("scan_folder: Preemptively compacting: {}, entries: {}")'
|
|
'warn!("scan_folder: failed to scan child folder {}: {}")'
|
|
'error!("scan_folder: failed to list path: {}/{}: {}")'
|
|
'info!("Cancelled active heal task: {}")'
|
|
'info!("Cancelled queued heal task: {}")'
|
|
'info!("Cancelled {} heal task(s) for path: {}")'
|
|
'info!("Heal scheduler received shutdown signal")'
|
|
'info!("Heal queue has {} pending requests, {} tasks active")'
|
|
'error!("Heal channel processor failed: {}")'
|
|
'info!("Heal manager with channel processor initialized successfully")'
|
|
'"Heal object workflow started"'
|
|
'"Heal object recovery started"'
|
|
'"Heal object completed"'
|
|
'"Heal object cleanup completed"'
|
|
'"Heal metadata workflow started"'
|
|
'"Heal metadata completed"'
|
|
'"Heal MRF workflow started"'
|
|
'"Heal MRF completed"'
|
|
'"Heal EC decode workflow started"'
|
|
'"Heal EC decode completed"'
|
|
'"Heal erasure set workflow started"'
|
|
'"Heal erasure set format repair completed"'
|
|
'"Heal erasure set completed"'
|
|
'"Heal storage repair finished"'
|
|
'"Heal storage admin operation finished"'
|
|
'"Heal storage request finished"'
|
|
'"Erasure set object heal completed"'
|
|
'"Erasure set heal resumed from checkpoint"'
|
|
'"Scanner disk bucket state updated"'
|
|
'"Scanner lifecycle action evaluation started"'
|
|
'"Scanner state persisted"'
|
|
'"scanner deep heal downgraded to normal during new-object cooldown"'
|
|
'"scanner detected folder with excessive direct subfolders"'
|
|
'"scan_folder: failed to get size for item {}: {}"'
|
|
'info!("worker take, {}", *available)'
|
|
'info!("worker give, {}", *available)'
|
|
'info!("worker wait end")'
|
|
'"Concurrency manager started (timeout={}, lock={}, deadlock={}, backpressure={}, scheduler={})"'
|
|
'info!("Concurrency manager stopped")'
|
|
'info!("Deadlock detection started")'
|
|
'info!("Deadlock detection stopped")'
|
|
'"Lock released early (optimization active)"'
|
|
'"Lock released on drop (normal release)"'
|
|
'info!("Trusted Proxies module is disabled via configuration")'
|
|
'info!("Trusted Proxies module initialized")'
|
|
'info!("Configuration loaded successfully from environment variables")'
|
|
'warn!("Failed to load configuration from environment: {}. Using defaults", e)'
|
|
'info!("=== Application Configuration ===")'
|
|
'info!("Server: {}", config.server_addr)'
|
|
'info!("Trusted Proxies: {}", config.proxy.proxies.len())'
|
|
'info!("Validation Mode: {:?}", config.proxy.validation_mode)'
|
|
'info!("Cache Capacity: {}", config.cache.capacity)'
|
|
'info!("Metrics Enabled: {}", config.monitoring.metrics_enabled)'
|
|
'info!("Cloud Metadata: {}", config.cloud.metadata_enabled)'
|
|
'info!("Failed validations will be logged")'
|
|
'debug!("Trusted networks: {:?}", config.proxy.get_network_strings())'
|
|
'info!("Metrics collection is disabled")'
|
|
'info!("Proxy metrics enabled for application: {}", self.app_name)'
|
|
'info!("Available metrics:")'
|
|
'debug!("SocketAddr extension is missing; skipping trusted proxy evaluation")'
|
|
'debug!("Peer address is unspecified; skipping trusted proxy evaluation")'
|
|
'debug!("Request received from trusted proxy: {}", peer_ip)'
|
|
'"Request from private network but not trusted: {}. This might indicate a configuration issue."'
|
|
'debug!("No Tokio runtime available; trusted proxy cache maintenance is disabled")'
|
|
'trace!("Analyzing proxy chain: {:?} with current proxy: {}", proxy_chain, current_proxy_ip)'
|
|
'debug!("Trusted proxy middleware is disabled")'
|
|
'debug!("Proxy validation successful in {:?}", duration)'
|
|
'debug!("Unrecoverable proxy validation error: {}", err)'
|
|
'debug!("Cloud metadata fetching is disabled")'
|
|
'info!("Detected AWS environment, fetching metadata")'
|
|
'info!("Detected Azure environment, fetching metadata")'
|
|
'info!("Detected GCP environment, fetching metadata")'
|
|
'info!("Detected Cloudflare environment")'
|
|
'info!("Detected DigitalOcean environment")'
|
|
'warn!("Unknown cloud provider detected: {}", name)'
|
|
'debug!("No cloud provider detected")'
|
|
'debug!("Trying to fetch metadata from {}", provider_name)'
|
|
'info!("Fetched {} IP ranges from {}", ranges.len(), provider_name)'
|
|
'debug!("Failed to fetch metadata from {}: {}", provider_name, e)'
|
|
'warn!("Failed to fetch network CIDRs from {}: {}", self.provider_name(), e)'
|
|
'warn!("Failed to fetch public IP ranges from {}: {}", self.provider_name(), e)'
|
|
'info!("Loaded {} static Cloudflare IP ranges", networks.len())'
|
|
'debug!("Fetched {} IP ranges from {}", networks.len(), url)'
|
|
'debug!("Failed to parse IP ranges from {}: {}", url, e)'
|
|
'debug!("Failed to fetch IP ranges from {}: HTTP {}", url, response.status())'
|
|
'debug!("Failed to fetch from {}: {}", url, e)'
|
|
'info!("Successfully fetched {} Cloudflare IP ranges from API", all_ranges.len())'
|
|
'info!("Loaded {} static DigitalOcean IP ranges", networks.len())'
|
|
'info!("Successfully fetched {} Google Cloud IP ranges from API", networks.len())'
|
|
'debug!("Failed to fetch Google IP ranges: HTTP {}", response.status())'
|
|
'debug!("Failed to fetch Google IP ranges: {}", e)'
|
|
'debug!("IMDSv2 token request failed with status: {}", response.status())'
|
|
'debug!("IMDSv2 token request failed: {}", e)'
|
|
'debug!("Using default AWS VPC network ranges")'
|
|
'info!("Successfully fetched {} AWS public IP ranges", networks.len())'
|
|
'debug!("Failed to fetch AWS IP ranges: HTTP {}", response.status())'
|
|
'debug!("Failed to fetch AWS IP ranges: {}", e)'
|
|
'debug!("Fetching Azure metadata from: {}", url)'
|
|
'debug!("Azure metadata request failed with status: {}", response.status())'
|
|
'debug!("Azure metadata request failed: {}", e)'
|
|
'debug!("Fetching Azure IP ranges from: {}", url)'
|
|
'info!("Successfully fetched {} Azure public IP ranges", networks.len())'
|
|
'debug!("Failed to fetch Azure IP ranges: HTTP {}", response.status())'
|
|
'debug!("Failed to fetch Azure IP ranges: {}", e)'
|
|
'debug!("Using default Azure public IP ranges")'
|
|
'info!("Successfully fetched {} network CIDRs from Azure metadata", cidrs.len())'
|
|
'debug!("No network CIDRs found in Azure metadata, falling back to defaults")'
|
|
'warn!("Failed to fetch Azure network metadata: {}", e)'
|
|
'debug!("Using default Azure VNet network ranges")'
|
|
'debug!("Fetching GCP metadata from: {}", url)'
|
|
'debug!("GCP metadata request failed with status: {}", response.status())'
|
|
'debug!("GCP metadata request failed: {}", e)'
|
|
'warn!("No network interfaces found in GCP metadata")'
|
|
'debug!("Failed to get IP/mask for GCP interface {}: {}", index, e)'
|
|
'warn!("Could not determine network CIDRs from GCP metadata, falling back to defaults")'
|
|
'info!("Successfully fetched {} network CIDRs from GCP metadata", cidrs.len())'
|
|
'warn!("Failed to fetch GCP network metadata: {}", e)'
|
|
'debug!("Fetching GCP IP ranges from: {}", url)'
|
|
'info!("Successfully fetched {} GCP public IP ranges", networks.len())'
|
|
'debug!("Failed to fetch GCP IP ranges: HTTP {}", response.status())'
|
|
'debug!("Failed to fetch GCP IP ranges: {}", e)'
|
|
'debug!("Using default GCP public IP ranges")'
|
|
'debug!("Using default GCP VPC network ranges")'
|
|
'info!("Initializing WebDAV server on {}"'
|
|
'info!("WebDAV server listening on {}"'
|
|
'debug!("Enabling WebDAV TLS with certificates from: {}"'
|
|
'warn!("Request body too large: {} > {}"'
|
|
'error!("IAM system unavailable during WebDAV auth: {}"'
|
|
'error!("IAM check_key failed for {}: {}"'
|
|
'warn!("WebDAV login failed: Invalid access key '\''{}'\''"'
|
|
'warn!("WebDAV login failed: Invalid secret key for '\''{}'\''"'
|
|
'info!("WebDAV user '\''{}'\'' authenticated successfully"'
|
|
'info!("Initializing FTPS server on {}"'
|
|
'info!("Configuring FTPS passive ports range: {:?} ({})"'
|
|
'warn!("No passive ports configured, using system-assigned ports"'
|
|
'info!("Configuring FTPS external IP for passive mode: {}"'
|
|
'info!("FTPS server configured for both active and passive mode support"'
|
|
'info!("FTPS is explicitly required for all connections"'
|
|
'info!("TLS disabled, running in plain FTP mode"'
|
|
'error!("IAM system unavailable during FTPS auth: {}"'
|
|
'error!("IAM check_key failed for {}: {}"'
|
|
'warn!("FTPS login failed: Invalid access key '\''{}'\''"'
|
|
'warn!("FTPS login failed: Invalid secret key for '\''{}'\''"'
|
|
'info!("FTPS user '\''{}'\'' authenticated successfully"'
|
|
'warn!("Expiration worker already running"'
|
|
'info!("Starting expiration worker (scan_interval={}s, worker_id={}/{})"'
|
|
'info!("Expiration worker stopped"'
|
|
'error!("Expiration cleanup iteration failed: {}"'
|
|
'info!("Stopping expiration worker"'
|
|
'debug!("Skipping object {} (handled by different worker)"'
|
|
'debug!("Tracking object {} for expiration at {}"'
|
|
'debug!("Untracking object {} from expiration"'
|
|
'info!("Starting expiration cleanup iteration (worker_id={})"'
|
|
'warn!("Invalid expiration entry path: {}"'
|
|
'info!("Deleted expired object: {}"'
|
|
'debug!("Object {} no longer exists or expiration removed"'
|
|
'error!("Failed to delete expired object {}: {}"'
|
|
'info!("Expiration cleanup iteration complete: scanned={}, deleted={}, duration={}ms, queue_size={}"'
|
|
'debug!("Would delete expired object: {}/{}/{} (expires_at={})"'
|
|
'info!("Starting full scan of objects with expiration (worker_id={})"'
|
|
'warn!("Full object scan not yet implemented - requires storage layer integration"'
|
|
'debug!("Bulk delete request for account: {}"'
|
|
'debug!("Processing {} delete requests"'
|
|
'error!("Error deleting {}: {}"'
|
|
'error!("Invalid path {}: {}"'
|
|
'debug!("Bulk extract request for container: {}, format: {:?}"'
|
|
'debug!("Extracted: {}"'
|
|
'error!("Failed to upload {}: {}"'
|
|
'debug!("Skipping directory: {}"'
|
|
'error!("Failed to read tar entry {}: {}"'
|
|
'debug!("Client explicitly disabled encryption"'
|
|
'debug!("Encrypting {} bytes with {}"'
|
|
'warn!("Encryption not yet implemented - returning plaintext with metadata"'
|
|
'debug!("Decrypting {} bytes with {}"'
|
|
'warn!("Decryption not yet implemented - returning data as-is"'
|
|
'debug!("Swift route matched: {:?}"'
|
|
'debug!("No Swift route matched, delegating to S3 service"'
|
|
'debug!("TempURL detected for {}/{}/{}"'
|
|
'debug!("TempURL validated successfully"'
|
|
'warn!("Failed to restore version after delete: {}"'
|
|
'debug!("Static web request: container={}, path={}, config={:?}"'
|
|
'debug!("Generating directory listing for path: {}"'
|
|
'debug!("Attempting to serve object: {}"'
|
|
'debug!("Serving error document: {}"'
|
|
'debug!("Error document not found, returning standard 404"'
|
|
'debug!("Read access granted: public read enabled"'
|
|
'debug!("Read access granted: referrer matches pattern {}"'
|
|
'debug!("Read access granted: account {} matches"'
|
|
'debug!("Read access granted: user {}:{} matches"'
|
|
'debug!("Read access denied: no matching ACL grant"'
|
|
'debug!("Write access granted: account {} matches"'
|
|
'debug!("Write access granted: user {}:{} matches"'
|
|
'debug!("Write access denied: no matching ACL grant"'
|
|
'warn!("Container sync using unencrypted HTTP - consider using HTTPS"'
|
|
'warn!("Container sync key is short (<16 chars) - recommend longer key"'
|
|
'debug!("Scheduled retry #{} for '\''{}'\'' at +{}s"'
|
|
'error!("Storage operation '\''{}'\'' failed: {}"'
|
|
'debug!("Creating symlink to target: {}"'
|
|
'debug!("FormPost signature mismatch: expected={}, got={}"'
|
|
'debug!("FormPost uploaded: {}/{}/{}"'
|
|
'debug!("Rate limit OK for {}: {} remaining"'
|
|
'debug!("Rate limit exceeded for {}: retry after {} seconds"'
|
|
'debug!("X-Delete-After: {} seconds -> X-Delete-At: {}"'
|
|
'debug!("X-Delete-At: {}"'
|
|
'debug!("X-Delete-At timestamp is more than 10 years in the future: {}"'
|
|
'debug!("Extracted symlink target: container={:?}, object={}"'
|
|
'warn!("Circular symlink reference detected"'
|
|
'debug!("CORS preflight request for container: {}"'
|
|
'debug!("Quota check passed: {}/{:?} bytes, {}/{:?} objects"'
|
|
'warn!("SFTP backend error"'
|
|
'warn!("SFTP authorisation rejected because the IAM system was unreachable"'
|
|
'warn!("fallback watchdog cancelling session: silence exceeded fallback threshold"'
|
|
'warn!("wedge watchdog cancelling session: russh select! parked outside its arms"'
|
|
'warn!("RUSTFS_SFTP_HANDLES_PER_SESSION out of range. Falling back to the default."'
|
|
'warn!("RUSTFS_SFTP_BACKEND_OP_TIMEOUT_SECS out of range. Falling back to the default."'
|
|
'warn!("RUSTFS_SFTP_READ_CACHE_WINDOW_BYTES out of range. Set to 0 to disable the cache, or to a value between the named bounds. Falling back to the default."'
|
|
'warn!("RUSTFS_SFTP_READ_CACHE_TOTAL_MEM_BYTES below minimum. Falling back to the default."'
|
|
'warn!("cannot stat file, skipping"'
|
|
'debug!("skipping file: size outside valid key range"'
|
|
'warn!("cannot read file, skipping"'
|
|
'info!("loaded host key"'
|
|
'warn!("file looks like a private key but failed to decode (passphrase-protected keys are not supported)"'
|
|
'debug!("not a valid private key, skipping"'
|
|
'info!("host key loading complete"'
|
|
'warn!("SFTP host key file permission enforcement is not active on Windows.'
|
|
'info!("Dial9 telemetry disabled"'
|
|
'info!("Validating dial9 telemetry configuration"'
|
|
'warn!("Failed to create dial9 output directory '\''{}'\'': {}"'
|
|
'warn!("Continuing without dial9 telemetry"'
|
|
'info!("Dial9 telemetry configuration validated successfully"'
|
|
'info!("Dial9 telemetry data will be flushed on drop"'
|
|
'info!("Dial9 telemetry guard dropped, data flushed"'
|
|
'info!("Initialized local logging"'
|
|
'info!("log cleaner compression profile configured"'
|
|
'warn!("Log cleanup failed: {}"'
|
|
'warn!("Log cleanup task panicked: {}"'
|
|
'warn!("Metrics collection for cluster stats cancelled."'
|
|
'warn!("Metrics collection for supplementary cluster stats cancelled."'
|
|
'warn!("Metrics collection for bucket stats cancelled."'
|
|
'warn!("Metrics collection for node/disk stats cancelled."'
|
|
'warn!("Bucket monitor unavailable; skip replication bandwidth key-state transition this cycle."'
|
|
'warn!("Metrics collection for bucket replication bandwidth stats cancelled."'
|
|
'warn!("Metrics collection for audit target stats cancelled."'
|
|
'warn!("Metrics collection for notification stats cancelled."'
|
|
'warn!("Metrics collection for background workflow stats cancelled."'
|
|
'warn!("Failed to get current PID for system monitoring: {}"'
|
|
'warn!("GPU metrics collection failed: {}"'
|
|
'warn!("GPU collector initialization failed: {}"'
|
|
'warn!("Process metrics collection cancelled."'
|
|
'warn!("Metrics collection for internode network stats cancelled."'
|
|
'warn!("Failed to collect process attributes for metrics labels: {}"'
|
|
'debug!("Log directory does not exist: {:?}"'
|
|
'info!("Found {} regular log files, total size: {} bytes ({:.2} MB)"'
|
|
'info!("Cleanup completed: deleted {} files, freed {} bytes ({:.2} MB)"'
|
|
'warn!(file = ?file.path, error = %err, "parallel compression failed"'
|
|
'warn!("parallel compression worker panicked, falling back to serial path"'
|
|
'info!("parallel cleanup finished"'
|
|
'warn!(file = ?file.path, error = %err, "serial compression failed, source kept"'
|
|
'info!("[DRY RUN] Would delete: {:?} ({} bytes)"'
|
|
'debug!("Deleted: {:?}"'
|
|
'error!("Failed to delete {:?}: {}"'
|
|
'warn!("zstd compression failed, fallback to gzip"'
|
|
'debug!(file = ?archive_path, "compressed archive already exists, skipping"'
|
|
'info!("[DRY RUN] Would compress file: {:?} -> {:?}"'
|
|
'debug!("compression finished"'
|
|
'debug!("Excluding file from cleanup: {:?}"'
|
|
'tracing::warn!("Failed to delete empty file {:?}: {}"'
|
|
'debug!("Deleted empty file: {:?}"'
|
|
'tracing::info!("[DRY RUN] Would delete empty file: {:?}"'
|
|
'warn!("Failed to load data usage from backend: {}"'
|
|
'warn!("Failed to list buckets for cluster metrics: {}"'
|
|
'warn!("Failed to load data usage for bucket metrics: {}"'
|
|
'warn!("Failed to list buckets for bucket metrics: {}"'
|
|
'warn!("Invalid bandwidth limit value for target {:?}: {}"'
|
|
'warn!("OBSERVABILITY_METRIC_ENABLED was already initialized; keeping original value"'
|
|
'info!("Initializing global guard"'
|
|
'error!("{} cache read lock poisoned: {}"'
|
|
'error!("{} cache write lock poisoned: {}"'
|
|
'error!("metrics_metadata lock poisoned: {}"'
|
|
'warn!("Could not get GPU stats, recording 0 for GPU memory usage"'
|
|
)
|
|
|
|
for pattern in "${forbidden_patterns[@]}"; do
|
|
if rg -n -F -- "$pattern" "${checked_files[@]}" >/dev/null; then
|
|
echo "❌ logging guardrail violation: found forbidden pattern '$pattern'" >&2
|
|
rg -n -F -- "$pattern" "${checked_files[@]}" >&2
|
|
exit 1
|
|
fi
|
|
done
|
|
|
|
echo "✅ Logging guardrails check passed"
|