# Specialized Docker Compose Configurations

This directory contains specialized Docker Compose configurations for specific testing scenarios.

## ⚠️ Important Note

**For Observability:**
We **strongly recommend** using the new, fully integrated observability stack located in `../observability/`. It provides a production-ready setup with Prometheus, Grafana, Tempo, Loki, and OpenTelemetry Collector, all with persistent storage and optimized configurations.

The `docker-compose.observability.yaml` in this directory is kept for legacy reference or specific minimal testing needs but is **not** the primary recommended setup.

## 📁 Configuration Files

### Cluster Testing

- **`docker-compose.cluster.yaml`**
  - **Purpose**: Simulates a 4-node RustFS distributed cluster.
  - **Use Case**: Testing distributed storage logic, consensus, and failover.
  - **Nodes**: 4 RustFS instances.
  - **Storage**: Uses local HTTP endpoints.

### Legacy / Minimal Observability

- **`docker-compose.observability.yaml`**
  - **Purpose**: A minimal observability setup.
  - **Status**: **Deprecated**. Please use `../observability/docker-compose.yml` instead.

## 🚀 Usage Examples

### Cluster Testing

To start a 4-node cluster for distributed testing:

```bash
# From project root
docker compose -f .docker/compose/docker-compose.cluster.yaml up -d
```

### Script-Based 4-Node Validation (Recommended)

Use the local validation script when you need local-source image build, failover checks,
and benchmark workflow in one command:

```bash
# Default mode: WAIT_PROBE_MODE=service
# This avoids false negatives where /health/ready remains 503 locally
# while the service path is already available.
./scripts/run_four_node_cluster_failover_bench.sh
```

Strict mode is available when you explicitly want `/health/ready == 200` as the gate:

```bash
WAIT_PROBE_MODE=ready ./scripts/run_four_node_cluster_failover_bench.sh
```

### Profiling + Trace Validation

The profiling-focused 4-node compose keeps profiling enabled and points RustFS
to an OTLP/HTTP collector endpoint:

```bash
docker compose -f .docker/compose/docker-compose.cluster.local-build.profiling-amd64.yml up -d
```

Important behavior notes:

- `RUSTFS_OBS_ENDPOINT` is the OTLP/HTTP base URL. RustFS automatically sends
  traces to `/v1/traces`, metrics to `/v1/metrics`, and logs to `/v1/logs`.
- Startup usually produces logs and metrics first. That does not guarantee
  visible traces yet.
- Trace data becomes obvious only after real HTTP/S3/gRPC requests hit RustFS.
- `RUSTFS_OBS_LOGGER_LEVEL=info` keeps the top-level request span but filters
  many nested `debug` spans. If Tempo/Jaeger looks sparse, retry with
  `RUSTFS_OBS_LOGGER_LEVEL=debug` before suspecting the collector.

Minimal trace verification flow:

```bash
# 1. Start the profiling compose with richer span visibility.
RUSTFS_OBS_LOGGER_LEVEL=debug \
docker compose -f .docker/compose/docker-compose.cluster.local-build.profiling-amd64.yml up -d

# 2. Generate real request traffic after startup.
curl -I http://127.0.0.1:9000/health
curl -I http://127.0.0.1:9000/health/ready

# 3. Then inspect Tempo or Jaeger.
# Grafana: http://localhost:3000
# Jaeger:  http://localhost:16686
```

If logs and metrics are present but traces are sparse, the most common cause is
"no real request traffic yet" or "`info` level filtered nested spans", not an
OTLP routing failure.

### (Deprecated) Minimal Observability

```bash
# From project root
docker compose -f .docker/compose/docker-compose.observability.yaml up -d
```