obs: mirror log attributes into loki lines (#5776)

Co-authored-by: heihutu <heihutu@gmail.com>
This commit is contained in:
houseme
2026-08-06 21:59:59 +08:00
committed by GitHub
parent fc0de983d8
commit 4855095446
3 changed files with 42 additions and 2 deletions
+15
View File
@@ -170,6 +170,10 @@ Important behavior notes:
- Logs and metrics usually appear during startup, so seeing those two signals
first is expected.
- The OpenTelemetry bridge sends `tracing` fields as log attributes. Loki stores
those attributes as structured metadata, and the Collector also mirrors the
common troubleshooting fields into the log line so simple line filters can
find them.
- Visible trace data usually requires real HTTP/S3/gRPC request traffic after
startup, because request-path spans are created on demand.
- `RUSTFS_OBS_LOGGER_LEVEL=info` keeps the top-level request span but filters
@@ -195,6 +199,17 @@ curl -I http://127.0.0.1:9000/health/ready
# Jaeger: http://localhost:16686
```
For a structured RustFS log such as an inter-node RPC authentication failure,
the Loki line now includes fields such as `event`, `component`, `subsystem`,
`failure_reason`, `rpc_service`, `rpc_method`, and `expected_audience`. Useful
LogQL checks:
```logql
{service_name="RustFS"} |= "RPC signature verification failed"
{service_name="RustFS"} |= "failure_reason="
{service_name="RustFS"} | failure_reason != ""
```
If logs and metrics are present but traces are sparse, the most common cause is
"no real request traffic yet" or "`info` level filtered nested spans", not an
OTLP routing failure.