build: update docker config and refine s3s region handling (#1976)

Co-authored-by: houseme <housemecn@gmail.com>
2026-08-19 19:16:17 +00:00 · 2026-02-27 01:21:12 +08:00
parent eb07f084cb
commit d983638391
43 changed files with 1009 additions and 852 deletions
@@ -0,0 +1,5 @@
+jaeger-data/*
+loki-data/*
+prometheus-data/*
+tempo-data/*
+grafana-data/*
@@ -1,109 +1,85 @@
-# Observability
+# RustFS Observability Stack

-This directory contains the observability stack for the application. The stack is composed of the following components:
+This directory contains the comprehensive observability stack for RustFS, designed to provide deep insights into application performance, logs, and traces.

- Prometheus v3.2.1
- Grafana 11.6.0
- Loki 3.4.2
- Jaeger 2.4.0
- Otel Collector 0.120.0 # 0.121.0 remove loki
+## Components

-## Prometheus
+The stack is composed of the following best-in-class open-source components:

-Prometheus is a monitoring and alerting toolkit. It scrapes metrics from instrumented jobs, either directly or via an
-intermediary push gateway for short-lived jobs. It stores all scraped samples locally and runs rules over this data to
-either aggregate and record new time series from existing data or generate alerts. Grafana or other API consumers can be
-used to visualize the collected data.
+- **Prometheus** (v2.53.1): The industry standard for metric collection and alerting.
+- **Grafana** (v11.1.0): The leading platform for observability visualization.
+- **Loki** (v3.1.0): A horizontally-scalable, highly-available, multi-tenant log aggregation system.
+- **Tempo** (v2.5.0): A high-volume, minimal dependency distributed tracing backend.
+- **Jaeger** (v1.59.0): Distributed tracing system (configured as a secondary UI/storage).
+- **OpenTelemetry Collector** (v0.104.0): A vendor-agnostic implementation for receiving, processing, and exporting telemetry data.

-## Grafana
+## Architecture

-Grafana is a multi-platform open-source analytics and interactive visualization web application. It provides charts,
-graphs, and alerts for the web when connected to supported data sources.
+1.  **Telemetry Collection**: Applications send OTLP (OpenTelemetry Protocol) data (Metrics, Logs, Traces) to the **OpenTelemetry Collector**.
+2.  **Processing & Exporting**: The Collector processes the data (batching, memory limiting) and exports it to the respective backends:
+    -   **Traces** -> **Tempo** (Primary) & **Jaeger** (Secondary/Optional)
+    -   **Metrics** -> **Prometheus** (via scraping the Collector's exporter)
+    -   **Logs** -> **Loki**
+3.  **Visualization**: **Grafana** connects to all backends (Prometheus, Tempo, Loki, Jaeger) to provide a unified dashboard experience.

-## Loki
+## Features

-Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. It is
-designed to be very cost-effective and easy to operate. It does not index the contents of the logs, but rather a set of
-labels for each log stream.
+-   **Full Persistence**: All data (Metrics, Logs, Traces) is persisted to Docker volumes, ensuring no data loss on restart.
+-   **Correlation**: Seamless navigation between Metrics, Logs, and Traces in Grafana.
+    -   Jump from a Metric spike to relevant Traces.
+    -   Jump from a Trace to relevant Logs.
+-   **High Performance**: Optimized configurations for batching, compression, and memory management.
+-   **Standardized Protocols**: Built entirely on OpenTelemetry standards.

-## Jaeger
+## Quick Start

-Jaeger is a distributed tracing system released as open source by Uber Technologies. It is used for monitoring and
-troubleshooting microservices-based distributed systems, including:
+### Prerequisites

- Distributed context propagation
- Distributed transaction monitoring
- Root cause analysis
- Service dependency analysis
- Performance / latency optimization
+-   Docker
+-   Docker Compose

-## Otel Collector
+### Deploy

-The OpenTelemetry Collector offers a vendor-agnostic implementation on how to receive, process, and export telemetry
-data. It removes the need to run, operate, and maintain multiple agents/collectors in order to support open-source
-observability data formats (e.g. Jaeger, Prometheus, etc.) sending to one or more open-source or commercial back-ends.
-
-## How to use
-
-To deploy the observability stack, run the following command:
-
- docker latest version
+Run the following command to start the entire stack:

 ```bash
-docker compose -f docker-compose.yml -f docker-compose.override.yml up -d
+docker compose up -d
 ```

- docker compose v2.0.0 or before
+### Access Dashboards
+
+| Service | URL | Credentials | Description |
+| :--- | :--- | :--- | :--- |
+| **Grafana** | [http://localhost:3000](http://localhost:3000) | `admin` / `admin` | Main visualization hub. |
+| **Prometheus** | [http://localhost:9090](http://localhost:9090) | - | Metric queries and status. |
+| **Jaeger UI** | [http://localhost:16686](http://localhost:16686) | - | Secondary trace visualization. |
+| **Tempo** | [http://localhost:3200](http://localhost:3200) | - | Tempo status/metrics. |
+
+## Configuration
+
+### Data Persistence
+
+Data is stored in the following Docker volumes:
+
+-   `prometheus-data`: Prometheus metrics
+-   `tempo-data`: Tempo traces (WAL and Blocks)
+-   `loki-data`: Loki logs (Chunks and Rules)
+-   `jaeger-data`: Jaeger traces (Badger DB)
+
+To clear all data:

 ```bash
-docker-compose -f docker-compose.yml -f docker-compose.override.yml up -d
+docker compose down -v
 ```

-To access the Grafana dashboard, navigate to `http://localhost:3000` in your browser. The default username and password
-are `admin` and `admin`, respectively.
-
-To access the Jaeger dashboard, navigate to `http://localhost:16686` in your browser.
-
-To access the Prometheus dashboard, navigate to `http://localhost:9090` in your browser.
-
-## How to stop
-
-To stop the observability stack, run the following command:
-
-```bash
-docker compose -f docker-compose.yml -f docker-compose.override.yml down
-```
-
-## How to remove data
-
-To remove the data generated by the observability stack, run the following command:
-
-```bash
-docker compose -f docker-compose.yml -f docker-compose.override.yml down -v
-```
-
-## How to configure
-
-To configure the observability stack, modify the `docker-compose.override.yml` file. The file contains the following
-
-```yaml
-services:
-  prometheus:
-    environment:
-      - PROMETHEUS_CONFIG_FILE=/etc/prometheus/prometheus.yml
-    volumes:
-      - ./prometheus.yml:/etc/prometheus/prometheus.yml
-
-  grafana:
-    environment:
-      - GF_SECURITY_ADMIN_PASSWORD=admin
-    volumes:
-      - ./grafana/provisioning:/etc/grafana/provisioning
-```
-
-The `prometheus` service mounts the `prometheus.yml` file to `/etc/prometheus/prometheus.yml`. The `grafana` service
-mounts the `grafana/provisioning` directory to `/etc/grafana/provisioning`. You can modify these files to configure the
-observability stack.
+### Customization

+-   **Prometheus**: Edit `prometheus.yml` to add scrape targets or alerting rules.
+-   **Grafana**: Dashboards and datasources are provisioned from the `grafana/` directory.
+-   **Collector**: Edit `otel-collector-config.yaml` to modify pipelines, processors, or exporters.

+## Troubleshooting

+-   **Service Health**: Check the health of services using `docker compose ps`.
+-   **Logs**: View logs for a specific service using `docker compose logs -f <service_name>`.
+-   **Otel Collector**: Check `http://localhost:13133` for health status and `http://localhost:1888/debug/pprof/` for profiling.
@@ -1,27 +1,85 @@
-## 部署可观测性系统
+# RustFS 可观测性技术栈

-OpenTelemetry Collector 提供了一个厂商中立的遥测数据处理方案，用于接收、处理和导出遥测数据。它消除了为支持多种开源可观测性数据格式（如
-Jaeger、Prometheus 等）而需要运行和维护多个代理/收集器的必要性。
+本目录包含 RustFS 的全面可观测性技术栈，旨在提供对应用程序性能、日志和追踪的深入洞察。

-### 快速部署
+## 组件

-1. 进入 `.docker/observability` 目录
-2. 执行以下命令启动服务：
+该技术栈由以下一流的开源组件组成：
+
+- **Prometheus** (v2.53.1): 行业标准的指标收集和告警工具。
+- **Grafana** (v11.1.0): 领先的可观测性可视化平台。
+- **Loki** (v3.1.0): 水平可扩展、高可用、多租户的日志聚合系统。
+- **Tempo** (v2.5.0): 高吞吐量、最小依赖的分布式追踪后端。
+- **Jaeger** (v1.59.0): 分布式追踪系统（配置为辅助 UI/存储）。
+- **OpenTelemetry Collector** (v0.104.0): 接收、处理和导出遥测数据的供应商无关实现。
+
+## 架构
+
+1.  **遥测收集**: 应用程序将 OTLP (OpenTelemetry Protocol) 数据（指标、日志、追踪）发送到 **OpenTelemetry Collector**。
+2.  **处理与导出**: Collector 处理数据（批处理、内存限制）并将其导出到相应的后端：
+    -   **追踪** -> **Tempo** (主要) & **Jaeger** (辅助/可选)
+    -   **指标** -> **Prometheus** (通过抓取 Collector 的导出器)
+    -   **日志** -> **Loki**
+3.  **可视化**: **Grafana** 连接到所有后端（Prometheus, Tempo, Loki, Jaeger），提供统一的仪表盘体验。
+
+## 特性
+
+-   **完全持久化**: 所有数据（指标、日志、追踪）都持久化到 Docker 卷，确保重启后无数据丢失。
+-   **关联性**: 在 Grafana 中实现指标、日志和追踪之间的无缝导航。
+    -   从指标峰值跳转到相关追踪。
+    -   从追踪跳转到相关日志。
+-   **高性能**: 针对批处理、压缩和内存管理进行了优化配置。
+-   **标准化协议**: 完全基于 OpenTelemetry 标准构建。
+
+## 快速开始
+
+### 前置条件
+
+-   Docker
+-   Docker Compose
+
+### 部署
+
+运行以下命令启动整个技术栈：

 ```bash
-docker compose -f docker-compose.yml  up -d
+docker compose up -d
 ```

-### 访问监控面板
+### 访问仪表盘

-服务启动后，可通过以下地址访问各个监控面板：
+| 服务 | URL | 凭据 | 描述 |
+| :--- | :--- | :--- | :--- |
+| **Grafana** | [http://localhost:3000](http://localhost:3000) | `admin` / `admin` | 主要可视化中心。 |
+| **Prometheus** | [http://localhost:9090](http://localhost:9090) | - | 指标查询和状态。 |
+| **Jaeger UI** | [http://localhost:16686](http://localhost:16686) | - | 辅助追踪可视化。 |
+| **Tempo** | [http://localhost:3200](http://localhost:3200) | - | Tempo 状态/指标。 |

- Grafana: `http://localhost:3000` (默认账号/密码：`admin`/`admin`)
- Jaeger: `http://localhost:16686`
- Prometheus: `http://localhost:9090`
+## 配置

-## 配置可观测性
+### 数据持久化

-```shell
-export RUSTFS_OBS_ENDPOINT="http://localhost:4317" # OpenTelemetry Collector 地址
+数据存储在以下 Docker 卷中：
+
+-   `prometheus-data`: Prometheus 指标
+-   `tempo-data`: Tempo 追踪 (WAL 和 Blocks)
+-   `loki-data`: Loki 日志 (Chunks 和 Rules)
+-   `jaeger-data`: Jaeger 追踪 (Badger DB)
+
+要清除所有数据：
+
+```bash
+docker compose down -v
 ```
+
+### 自定义
+
+-   **Prometheus**: 编辑 `prometheus.yml` 以添加抓取目标或告警规则。
+-   **Grafana**: 仪表盘和数据源从 `grafana/` 目录预置。
+-   **Collector**: 编辑 `otel-collector-config.yaml` 以修改管道、处理器或导出器。
+
+## 故障排除
+
+-   **服务健康**: 使用 `docker compose ps` 检查服务健康状况。
+-   **日志**: 使用 `docker compose logs -f <service_name>` 查看特定服务的日志。
+-   **Otel Collector**: 检查 `http://localhost:13133` 获取健康状态，检查 `http://localhost:1888/debug/pprof/` 进行性能分析。
@@ -14,6 +14,8 @@

 services:

+  # --- Tracing ---
+
  tempo-init:
    image: busybox:latest
    command: [ "sh", "-c", "chown -R 10001:10001 /var/tempo" ]
@@ -26,74 +28,52 @@ services:

  tempo:
    image: grafana/tempo:latest
-    user: "10001" # The container must be started with root to execute chown in the script
-    command: [ "-config.file=/etc/tempo.yaml" ] # This is passed as a parameter to the entry point script
+    user: "10001"
+    command: [ "-config.file=/etc/tempo.yaml" ]
    volumes:
      - ./tempo.yaml:/etc/tempo.yaml:ro
      - ./tempo-data:/var/tempo
    ports:
      - "3200:3200" # tempo
-      - "24317:4317" # otlp grpc
-      - "24318:4318" # otlp http
+      - "4317"      # otlp grpc
+      - "4318"      # otlp http
    restart: unless-stopped
    networks:
      - otel-network
    healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3200/metrics" ]
+      test: [ "CMD-SHELL", "wget --spider -q http://localhost:3200/metrics || exit 1" ]
      interval: 10s
      timeout: 5s
-      retries: 3
-      start_period: 15s
-
-  otel-collector:
-    image: otel/opentelemetry-collector-contrib:latest
-    environment:
-      - TZ=Asia/Shanghai
-    volumes:
-      - ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro
-    ports:
-      - "1888:1888" # pprof
-      - "8888:8888" # Prometheus metrics for Collector
-      - "8889:8889" # Prometheus metrics for application indicators
-      - "13133:13133" # health check
-      - "4317:4317" # OTLP gRPC
-      - "4318:4318" # OTLP HTTP
-      - "55679:55679" # zpages
-    networks:
-      - otel-network
-    depends_on:
-      jaeger:
-        condition: service_started
-      tempo:
-        condition: service_started
-      prometheus:
-        condition: service_started
-      loki:
-        condition: service_started
-    healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:13133" ]
-      interval: 10s
-      timeout: 5s
-      retries: 3
+      retries: 5
+      start_period: 40s

  jaeger:
    image: jaegertracing/jaeger:latest
    environment:
      - TZ=Asia/Shanghai
-      - SPAN_STORAGE_TYPE=memory
+      - SPAN_STORAGE_TYPE=badger
+      - BADGER_EPHEMERAL=false
+      - BADGER_DIRECTORY_VALUE=/badger/data
+      - BADGER_DIRECTORY_KEY=/badger/key
      - COLLECTOR_OTLP_ENABLED=true
+    volumes:
+      - ./jaeger-data:/badger
    ports:
      - "16686:16686" # Web UI
-      - "14317:4317" # OTLP gRPC
-      - "14318:4318" # OTLP HTTP
-      - "18888:8888" # collector
+      - "14269:14269" # Admin/Metrics
+      - "4317"
+      - "4318"
    networks:
      - otel-network
    healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:16686" ]
+      test: [ "CMD-SHELL", "wget --spider -q http://localhost:14269 || exit 1" ]
      interval: 10s
      timeout: 5s
-      retries: 3
+      retries: 5
+      start_period: 20s
+
+  # --- Metrics ---
+
  prometheus:
    image: prom/prometheus:latest
    environment:
@@ -105,11 +85,11 @@ services:
      - "9090:9090"
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
-      - '--web.enable-otlp-receiver' # Enable OTLP
-      - '--web.enable-remote-write-receiver' # Enable remote write
-      - '--enable-feature=promql-experimental-functions' # Enable info()
-      - '--storage.tsdb.min-block-duration=15m' # Minimum block duration
-      - '--storage.tsdb.max-block-duration=1h'  # Maximum block duration
+      - '--web.enable-otlp-receiver'
+      - '--web.enable-remote-write-receiver'
+      - '--enable-feature=promql-experimental-functions'
+      - '--storage.tsdb.min-block-duration=2h'
+      - '--storage.tsdb.max-block-duration=2h'
      - '--log.level=info'
      - '--storage.tsdb.retention.time=30d'
      - '--storage.tsdb.path=/prometheus'
@@ -119,37 +99,78 @@ services:
    networks:
      - otel-network
    healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:9090/-/healthy" ]
+      test: [ "CMD-SHELL", "wget --spider -q http://localhost:9090/-/healthy || exit 1" ]
      interval: 10s
      timeout: 5s
      retries: 3
+
+  # --- Logging ---
+
  loki:
    image: grafana/loki:latest
    environment:
      - TZ=Asia/Shanghai
    volumes:
      - ./loki-config.yaml:/etc/loki/local-config.yaml:ro
+      - ./loki-data:/loki
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml
    networks:
      - otel-network
    healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3100/ready" ]
+      test: [ "CMD-SHELL", "wget --spider -q http://localhost:3100/metrics || exit 1" ]
+      interval: 15s
+      timeout: 10s
+      retries: 5
+      start_period: 60s
+
+  # --- Collection ---
+
+  otel-collector:
+    image: otel/opentelemetry-collector-contrib:latest
+    environment:
+      - TZ=Asia/Shanghai
+    volumes:
+      - ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro
+    ports:
+      - "1888:1888"   # pprof
+      - "8888:8888"   # Prometheus metrics for Collector
+      - "8889:8889"   # Prometheus metrics for application indicators
+      - "13133:13133" # health check
+      - "4317:4317"   # OTLP gRPC
+      - "4318:4318"   # OTLP HTTP
+      - "55679:55679" # zpages
+    networks:
+      - otel-network
+    depends_on:
+      - tempo
+      - jaeger
+      - prometheus
+      - loki
+    healthcheck:
+      test: [ "CMD-SHELL", "wget --spider -q http://localhost:13133 || exit 1" ]
      interval: 10s
      timeout: 5s
      retries: 3
+      start_period: 20s
+
+  # --- Visualization ---
+
  grafana:
    image: grafana/grafana:latest
    ports:
-      - "3000:3000"  # Web UI
+      - "3000:3000"
    volumes:
-      - ./grafana-datasources.yaml:/etc/grafana/provisioning/datasources/datasources.yaml
+      - ./grafana/provisioning:/etc/grafana/provisioning
+      - ./grafana/dashboards:/var/lib/grafana/dashboards
+      - ./grafana-data:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
      - GF_SECURITY_ADMIN_USER=admin
      - TZ=Asia/Shanghai
      - GF_INSTALL_PLUGINS=grafana-pyroscope-datasource
+      - GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH=/var/lib/grafana/dashboards/home.json
    restart: unless-stopped
    networks:
      - otel-network
@@ -158,7 +179,7 @@ services:
      - tempo
      - loki
    healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3000/api/health" ]
+      test: [ "CMD-SHELL", "wget --spider -q http://localhost:3000/api/health || exit 1" ]
      interval: 10s
      timeout: 5s
      retries: 3
@@ -166,11 +187,14 @@ services:
 volumes:
  prometheus-data:
  tempo-data:
+  loki-data:
+  jaeger-data:
+  grafana-data:

 networks:
  otel-network:
    driver: bridge
-    name: "network_otel_config"
+    name: "network_otel"
    ipam:
      config:
        - subnet: 172.28.0.0/16
@@ -0,0 +1 @@
+*
@@ -1,3 +1,17 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 apiVersion: 1

 datasources:
@@ -7,102 +21,77 @@ datasources:
    access: proxy
    orgId: 1
    url: http://prometheus:9090
-    basicAuth: false
-    isDefault: false
+    isDefault: true
    version: 1
    editable: false
    jsonData:
      httpMethod: GET
+      exemplarTraceIdDestinations:
+        - name: trace_id
+          datasourceUid: tempo
+
  - name: Tempo
    type: tempo
+    uid: tempo
    access: proxy
    orgId: 1
    url: http://tempo:3200
-    basicAuth: false
-    isDefault: true
+    isDefault: false
    version: 1
    editable: false
-    apiVersion: 1
-    uid: tempo
    jsonData:
      httpMethod: GET
      serviceMap:
        datasourceUid: prometheus
-      streamingEnabled:
-        search: true
-      tracesToLogsV2:
-        # Field with an internal link pointing to a logs data source in Grafana.
-        # datasourceUid value must match the uid value of the logs data source.
-        datasourceUid: 'loki'
-        spanStartTimeShift: '-1h'
-        spanEndTimeShift: '1h'
-        tags: [ 'job', 'instance', 'pod', 'namespace' ]
-        filterByTraceID: false
+      tracesToLogs:
+        datasourceUid: loki
+        tags: [ 'job', 'instance', 'pod', 'namespace', 'service.name' ]
+        mappedTags: [ { key: 'service.name', value: 'app' } ]
+        spanStartTimeShift: '1s'
+        spanEndTimeShift: '-1s'
+        filterByTraceID: true
        filterBySpanID: false
-        customQuery: true
-        query: 'method="$${__span.tags.method}"'
-        tracesToMetrics:
-          datasourceUid: 'prometheus'
-          spanStartTimeShift: '-1h'
-          spanEndTimeShift: '1h'
-          tags: [ { key: 'service.name', value: 'service' }, { key: 'job' } ]
-          queries:
-            - name: 'Sample query'
-              query: 'sum(rate(traces_spanmetrics_latency_bucket{$$__tags}[5m]))'
-        tracesToProfiles:
-          datasourceUid: 'grafana-pyroscope-datasource'
-          tags: [ 'job', 'instance', 'pod', 'namespace' ]
-          profileTypeId: 'process_cpu:cpu:nanoseconds:cpu:nanoseconds'
-          customQuery: true
-          query: 'method="$${__span.tags.method}"'
-        serviceMap:
-          datasourceUid: 'prometheus'
-        nodeGraph:
-          enabled: true
-        search:
-          hide: false
-        traceQuery:
-          timeShiftEnabled: true
-          spanStartTimeShift: '-1h'
-          spanEndTimeShift: '1h'
-        spanBar:
-          type: 'Tag'
-          tag: 'http.path'
-        streamingEnabled:
-          search: true
-  - name: Jaeger
-    type: jaeger
-    uid: Jaeger
-    url: http://jaeger:16686
-    basicAuth: false
-    access: proxy
-    readOnly: false
-    isDefault: false
-    jsonData:
-      tracesToLogsV2:
-        # Field with an internal link pointing to a logs data source in Grafana.
-        # datasourceUid value must match the uid value of the logs data source.
-        datasourceUid: 'loki'
-        spanStartTimeShift: '1h'
-        spanEndTimeShift: '-1h'
-        tags: [ 'job', 'instance', 'pod', 'namespace' ]
-        filterByTraceID: false
-        filterBySpanID: false
-        customQuery: true
-        query: 'method="$${__span.tags.method}"'
      tracesToMetrics:
-        datasourceUid: 'Prometheus'
-        spanStartTimeShift: '1h'
-        spanEndTimeShift: '-1h'
-        tags: [ { key: 'service.name', value: 'service' }, { key: 'job' } ]
+        datasourceUid: prometheus
+        tags: [ { key: 'service.name' }, { key: 'job' } ]
        queries:
-          - name: 'Sample query'
-            query: 'sum(rate(traces_spanmetrics_latency_bucket{$$__tags}[5m]))'
+          - name: 'Service-Level Latency'
+            query: 'sum(rate(traces_spanmetrics_latency_bucket{$$__tags}[5m])) by (le)'
+          - name: 'Service-Level Calls'
+            query: 'sum(rate(traces_spanmetrics_calls_total{$$__tags}[5m]))'
+          - name: 'Service-Level Errors'
+            query: 'sum(rate(traces_spanmetrics_calls_total{status_code="ERROR", $$__tags}[5m]))'
      nodeGraph:
        enabled: true
-      traceQuery:
-        timeShiftEnabled: true
-        spanStartTimeShift: '1h'
-        spanEndTimeShift: '-1h'
-      spanBar:
-        type: 'None'
+
+  - name: Loki
+    type: loki
+    uid: loki
+    orgId: 1
+    url: http://loki:3100
+    isDefault: false
+    version: 1
+    editable: false
+    jsonData:
+      derivedFields:
+        - datasourceUid: tempo
+          matcherRegex: 'trace_id=(\w+)'
+          name: 'TraceID'
+          url: '$${__value.raw}'
+
+  - name: Jaeger
+    type: jaeger
+    uid: jaeger
+    url: http://jaeger:16686
+    access: proxy
+    isDefault: false
+    editable: false
+    jsonData:
+      tracesToLogs:
+        datasourceUid: loki
+        tags: [ 'job', 'instance', 'pod', 'namespace', 'service.name' ]
+        mappedTags: [ { key: 'service.name', value: 'app' } ]
+        spanStartTimeShift: '1s'
+        spanEndTimeShift: '-1s'
+        filterByTraceID: true
+        filterBySpanID: false
@@ -31,29 +31,19 @@ service:
                host: 0.0.0.0
                port: 8888
    logs:
-      level: debug
-    # TODO Initialize telemetry tracer once OTEL released new feature.
-    # https://github.com/open-telemetry/opentelemetry-collector/issues/10663
+      level: info

 extensions:
  healthcheckv2:
    use_v2: true
    http:

-  # pprof:
-  #   endpoint: 0.0.0.0:1777
-  # zpages:
-  #   endpoint: 0.0.0.0:55679
-
  jaeger_query:
    storage:
-      traces: some_store
-      traces_archive: another_store
+      traces: badger_store
    ui:
      config_file: ./cmd/jaeger/config-ui.json
      log_access: true
-    # The maximum duration that is considered for clock skew adjustments.
-    # Defaults to 0 seconds, which means it's disabled.
    max_clock_skew_adjust: 0s
    grpc:
      endpoint: 0.0.0.0:16685
@@ -62,26 +52,16 @@ extensions:

  jaeger_storage:
    backends:
-      some_store:
-        memory:
-          max_traces: 1000000
-          max_events: 100000
-      another_store:
-        memory:
-          max_traces: 1000000
-    metric_backends:
-      some_metrics_storage:
-        prometheus:
-          endpoint: http://prometheus:9090
-          normalize_calls: true
-          normalize_duration: true
+      badger_store:
+        badger:
+          ephemeral: false
+          directory_key: /badger/key
+          directory_value: /badger/data
+          span_store_ttl: 72h

  remote_sampling:
-    # You can either use file or adaptive sampling strategy in remote_sampling
-    # file:
-    #   path: ./cmd/jaeger/sampling-strategies.json
    adaptive:
-      sampling_store: some_store
+      sampling_store: badger_store
      initial_sampling_probability: 0.1
    http:
    grpc:
@@ -103,12 +83,8 @@ receivers:

 processors:
  batch:
-  metadata_keys: [ "span.kind", "http.method", "http.status_code", "db.system", "db.statement", "messaging.system", "messaging.destination", "messaging.operation","span.events","span.links" ]
-  # Adaptive Sampling Processor is required to support adaptive sampling.
-  # It expects remote_sampling extension with `adaptive:` config to be enabled.
  adaptive_sampling:

 exporters:
  jaeger_storage_exporter:
-    trace_storage: some_store
-
+    trace_storage: badger_store
@@ -0,0 +1 @@
+*
@@ -17,16 +17,16 @@ auth_enabled: false
 server:
  http_listen_port: 3100
  grpc_listen_port: 9096
-  log_level: debug
+  log_level: info
  grpc_server_max_concurrent_streams: 1000

 common:
  instance_addr: 127.0.0.1
-  path_prefix: /tmp/loki
+  path_prefix: /loki
  storage:
    filesystem:
-      chunks_directory: /tmp/loki/chunks
-      rules_directory: /tmp/loki/rules
+      chunks_directory: /loki/chunks
+      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
@@ -66,17 +66,3 @@ ruler:

 frontend:
  encoding: protobuf
-
-
-# By default, Loki will send anonymous, but uniquely-identifiable usage and configuration
-# analytics to Grafana Labs. These statistics are sent to https://stats.grafana.org/
-#
-# Statistics help us better understand how Loki is used, and they show us performance
-# levels for most users. This helps us prioritize features and documentation.
-# For more information on what's sent, look at
-# https://github.com/grafana/loki/blob/main/pkg/analytics/stats.go
-# Refer to the buildReport method to see what goes into a report.
-#
-# If you would like to disable reporting, uncomment the following lines:
-#analytics:
-#  reporting_enabled: false
@@ -0,0 +1 @@
+*
@@ -15,69 +15,70 @@
 receivers:
  otlp:
    protocols:
-      grpc: # OTLP gRPC receiver
+      grpc:
        endpoint: 0.0.0.0:4317
-      http: # OTLP HTTP receiver
+      http:
        endpoint: 0.0.0.0:4318

 processors:
-  batch: # Batch processor to improve throughput
-    timeout: 5s
-    send_batch_size: 1000
-    metadata_keys: [ ]
-    metadata_cardinality_limit: 1000
+  batch:
+    timeout: 1s
+    send_batch_size: 1024
  memory_limiter:
    check_interval: 1s
-    limit_mib: 512
+    limit_mib: 1024
+    spike_limit_mib: 256
  transform/logs:
    log_statements:
      - context: log
        statements:
-          # Extract Body as attribute "message"
          - set(attributes["message"], body.string)
-          # Retain the original Body
          - set(attributes["log.body"], body.string)

 exporters:
-  otlp/traces: # OTLP exporter for trace data
-    endpoint: "http://jaeger:4317"  # OTLP gRPC endpoint for Jaeger
+  otlp/tempo:
+    endpoint: "tempo:4317"
    tls:
-      insecure: true  # TLS is disabled in the development environment and a certificate needs to be configured in the production environment.
-    compression: gzip  # Enable compression to reduce network bandwidth
+      insecure: true
+    compression: gzip
    retry_on_failure:
-      enabled: true  # Enable retry on failure
-      initial_interval: 1s  # Initial interval for retry
-      max_interval: 30s  # Maximum interval for retry
-      max_elapsed_time: 300s  # Maximum elapsed time for retry
+      enabled: true
+      initial_interval: 1s
+      max_interval: 30s
+      max_elapsed_time: 300s
    sending_queue:
-      enabled: true  # Enable sending queue
-      num_consumers: 10  # Number of consumers
-      queue_size: 5000  # Queue size
-  otlp/tempo: # OTLP exporter for trace data
-    endpoint: "http://tempo:4317"  # OTLP gRPC endpoint for tempo
+      enabled: true
+      num_consumers: 10
+      queue_size: 5000
+
+  otlp/jaeger:
+    endpoint: "jaeger:4317"
    tls:
-      insecure: true  # TLS is disabled in the development environment and a certificate needs to be configured in the production environment.
-    compression: gzip  # Enable compression to reduce network bandwidth
+      insecure: true
+    compression: gzip
    retry_on_failure:
-      enabled: true  # Enable retry on failure
-      initial_interval: 1s  # Initial interval for retry
-      max_interval: 30s  # Maximum interval for retry
-      max_elapsed_time: 300s  # Maximum elapsed time for retry
+      enabled: true
+      initial_interval: 1s
+      max_interval: 30s
+      max_elapsed_time: 300s
    sending_queue:
-      enabled: true  # Enable sending queue
-      num_consumers: 10  # Number of consumers
-      queue_size: 5000  # Queue size
-  prometheus: # Prometheus exporter for metrics data
-    endpoint: "0.0.0.0:8889"  # Prometheus scraping endpoint
-    send_timestamps: true  # Send timestamp
-    metric_expiration: 5m  # Metric expiration time
+      enabled: true
+      num_consumers: 10
+      queue_size: 5000
+
+  prometheus:
+    endpoint: "0.0.0.0:8889"
+    send_timestamps: true
+    metric_expiration: 5m
    resource_to_telemetry_conversion:
-      enabled: true  # Enable resource to telemetry conversion
-  otlphttp/loki: # Loki exporter for log data
+      enabled: true
+
+  otlphttp/loki:
    endpoint: "http://loki:3100/otlp"
    tls:
      insecure: true
-    compression: gzip  # Enable compression to reduce network bandwidth
+    compression: gzip
+
 extensions:
  health_check:
    endpoint: 0.0.0.0:13133
@@ -85,13 +86,14 @@ extensions:
    endpoint: 0.0.0.0:1888
  zpages:
    endpoint: 0.0.0.0:55679
+
 service:
-  extensions: [ health_check, pprof, zpages ]  # Enable extension
+  extensions: [ health_check, pprof, zpages ]
  pipelines:
    traces:
      receivers: [ otlp ]
      processors: [ memory_limiter, batch ]
-      exporters: [ otlp/traces, otlp/tempo ]
+      exporters: [ otlp/tempo, otlp/jaeger ]
    metrics:
      receivers: [ otlp ]
      processors: [ batch ]
@@ -102,20 +104,13 @@ service:
      exporters: [ otlphttp/loki ]
  telemetry:
    logs:
-      level: "debug"  # Collector log level
-      encoding: "json"  # Log encoding: console or json
+      level: "info"
+      encoding: "json"
    metrics:
-      level: "detailed" # Can be basic, normal, detailed
+      level: "normal"
      readers:
-        - periodic:
-            exporter:
-              otlp:
-                protocol: http/protobuf
-                endpoint: http://otel-collector:4318
        - pull:
            exporter:
              prometheus:
                host: '0.0.0.0'
                port: 8888
-
-
@@ -17,27 +17,40 @@ global:
  evaluation_interval: 15s
  external_labels:
    cluster: 'rustfs-dev'  # Label to identify the cluster
-    relica: '1'  # Replica identifier
+    replica: '1'  # Replica identifier

 scrape_configs:
-  - job_name: 'otel-collector-internal'
+  - job_name: 'otel-collector'
    static_configs:
      - targets: [ 'otel-collector:8888' ]  # Scrape metrics from Collector
    scrape_interval: 10s
+
  - job_name: 'rustfs-app-metrics'
    static_configs:
      - targets: [ 'otel-collector:8889' ]  # Application indicators
    scrape_interval: 15s
    metric_relabel_configs:
+      - source_labels: [ __name__ ]
+        regex: 'go_.*'
+        action: drop  # Drop Go runtime metrics if not needed
+
  - job_name: 'tempo'
    static_configs:
      - targets: [ 'tempo:3200' ]  # Scrape metrics from Tempo
+
  - job_name: 'jaeger'
    static_configs:
-      - targets: [ 'jaeger:8888' ]  # Jaeger admin port
+      - targets: [ 'jaeger:14269' ]  # Jaeger admin port (14269 is standard for admin/metrics)
+
+  - job_name: 'loki'
+    static_configs:
+      - targets: [ 'loki:3100' ]
+
+  - job_name: 'prometheus'
+    static_configs:
+      - targets: [ 'localhost:9090' ]

 otlp:
-  # Recommended attributes to be promoted to labels.
  promote_resource_attributes:
    - service.instance.id
    - service.name
@@ -56,10 +69,8 @@ otlp:
    - k8s.pod.name
    - k8s.replicaset.name
    - k8s.statefulset.name
-  # Ingest OTLP data keeping all characters in metric/label names.
  translation_strategy: NoUTF8EscapingWithSuffixes

 storage:
-  # OTLP is a push-based protocol, Out of order samples is a common scenario.
  tsdb:
-    out_of_order_time_window: 30m
+    out_of_order_time_window: 30m
@@ -1,18 +1,21 @@
-stream_over_http_enabled: true
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 server:
  http_listen_port: 3200
  log_level: info

-query_frontend:
-  search:
-    duration_slo: 5s
-    throughput_bytes_slo: 1.073741824e+09
-    metadata_slo:
-      duration_slo: 5s
-      throughput_bytes_slo: 1.073741824e+09
-  trace_by_id:
-    duration_slo: 5s
-
 distributor:
  receivers:
    otlp:
@@ -25,10 +28,6 @@ distributor:
 ingester:
  max_block_duration: 5m # cut the headblock when this much time passes. this is being set for demo purposes and should probably be left alone normally

-compactor:
-  compaction:
-    block_retention: 1h # overall Tempo trace retention. set for demo purposes
-
 metrics_generator:
  registry:
    external_labels:
@@ -49,9 +48,3 @@ storage:
      path: /var/tempo/wal # where to store the wal locally
    local:
      path: /var/tempo/blocks
-
-overrides:
-  defaults:
-    metrics_generator:
-      processors: [ service-graphs, span-metrics, local-blocks ] # enables metrics generator
-      generate_native_histograms: both