houseme
088c4bda43
* fix(ecstore): harden issue3031 multipart validation path - clear stale multipart part destinations before rename fan-out - add repeated part overwrite regression coverage - reduce remote disk startup false-fault escalation to suspect-first - refine remote locker diagnostics and lower scanner leader-lock log noise - add a dedicated 4-node issue3031 docker validation script * refactor(admin): inline console version json macro - drop the unused serde_json::json import in admin console - call serde_json::json! inline in version_handler - keep the console version response behavior unchanged * fix(remote-disk): recover suspect health on probe success - record probe success during remote disk health checks so suspect drives recover - use async_with_vars for the remote disk health probe test - make the missing-listener test assert the state transition more robustly
Specialized Docker Compose Configurations
This directory contains specialized Docker Compose configurations for specific testing scenarios.
⚠️ Important Note
For Observability:
We strongly recommend using the new, fully integrated observability stack located in ../observability/. It provides a production-ready setup with Prometheus, Grafana, Tempo, Loki, and OpenTelemetry Collector, all with persistent storage and optimized configurations.
The docker-compose.observability.yaml in this directory is kept for legacy reference or specific minimal testing needs but is not the primary recommended setup.
📁 Configuration Files
Cluster Testing
docker-compose.cluster.yaml- Purpose: Simulates a 4-node RustFS distributed cluster.
- Use Case: Testing distributed storage logic, consensus, and failover.
- Nodes: 4 RustFS instances.
- Storage: Uses local HTTP endpoints.
Legacy / Minimal Observability
docker-compose.observability.yaml- Purpose: A minimal observability setup.
- Status: Deprecated. Please use
../observability/docker-compose.ymlinstead.
🚀 Usage Examples
Cluster Testing
To start a 4-node cluster for distributed testing:
# From project root
docker compose -f .docker/compose/docker-compose.cluster.yaml up -d
Script-Based 4-Node Validation (Recommended)
Use the local validation script when you need local-source image build, failover checks, and benchmark workflow in one command:
# Default mode: WAIT_PROBE_MODE=service
# This avoids false negatives where /health/ready remains 503 locally
# while the service path is already available.
./scripts/run_four_node_cluster_failover_bench.sh
Strict mode is available when you explicitly want /health/ready == 200 as the gate:
WAIT_PROBE_MODE=ready ./scripts/run_four_node_cluster_failover_bench.sh
Profiling + Trace Validation
The profiling-focused 4-node compose keeps profiling enabled and points RustFS to an OTLP/HTTP collector endpoint:
docker compose -f .docker/compose/docker-compose.cluster.local-build.profiling-amd64.yml up -d
Important behavior notes:
RUSTFS_OBS_ENDPOINTis the OTLP/HTTP base URL. RustFS automatically sends traces to/v1/traces, metrics to/v1/metrics, and logs to/v1/logs.- Startup usually produces logs and metrics first. That does not guarantee visible traces yet.
- Trace data becomes obvious only after real HTTP/S3/gRPC requests hit RustFS.
RUSTFS_OBS_LOGGER_LEVEL=infokeeps the top-level request span but filters many nesteddebugspans. If Tempo/Jaeger looks sparse, retry withRUSTFS_OBS_LOGGER_LEVEL=debugbefore suspecting the collector.
Minimal trace verification flow:
# 1. Start the profiling compose with richer span visibility.
RUSTFS_OBS_LOGGER_LEVEL=debug \
docker compose -f .docker/compose/docker-compose.cluster.local-build.profiling-amd64.yml up -d
# 2. Generate real request traffic after startup.
curl -I http://127.0.0.1:9000/health
curl -I http://127.0.0.1:9000/health/ready
# 3. Then inspect Tempo or Jaeger.
# Grafana: http://localhost:3000
# Jaeger: http://localhost:16686
If logs and metrics are present but traces are sparse, the most common cause is
"no real request traffic yet" or "info level filtered nested spans", not an
OTLP routing failure.
(Deprecated) Minimal Observability
# From project root
docker compose -f .docker/compose/docker-compose.observability.yaml up -d