rustfs/ARCHITECTURE.md

ARCHITECTURE.md

Last updated: 2026-08-12 · Revision: 3

This document describes the high-level architecture of RustFS. If you want to familiarize yourself with the code base, you are in the right place!

See also CONTRIBUTING.md for development workflow. See also docs/architecture for active architecture migration guardrails.

Bird's Eye View

RustFS is a high-performance, S3-compatible distributed object storage system written in Rust. It uses erasure coding for data durability, supports multi-tenancy through IAM/STS, and provides a web-based admin console.

A running RustFS node exposes:

• S3 API (port 9000) — the primary data path for object CRUD
• Admin API (port 9000, /minio/ prefix) — cluster management, IAM, metrics
• Console (port 9001) — web UI backed by the Admin API
• Inter-node RPC (gRPC/tonic) — cluster communication for distributed mode

The core data flow for a PUT request looks like:

HTTP request
  → server (TLS, auth, routing, compression)
    → app/object_usecase (validation, policy, lifecycle)
      → storage/ecfs (erasure coding, encryption, checksums)
        → ecstore (disk pool selection, data distribution)
          → rio (reader pipeline: encrypt → compress → hash → write)
            → io-core (buffer pool, storage profiling, admission control)
              → local disk / remote disk via RPC

Code Map

The repository is a Cargo workspace with a flat crates/ layout:

rustfs/                      # Workspace root (virtual manifest)
├── rustfs/                  # Main binary + library crate
│   └── src/
│       ├── main.rs          # Entry point, startup sequence
│       ├── lib.rs           # Module tree root
│       ├── server/          # HTTP server, TLS, routing, middleware
│       ├── admin/           # Admin API handlers and console
│       ├── app/             # Use-case layer (object, bucket, multipart)
│       ├── storage/         # Storage engine interface and implementation
│       ├── auth.rs          # S3 request authentication
│       ├── config/          # CLI args, config parsing, workload profiles
│       └── ...
├── crates/                  # library crates (authoritative list: Cargo.toml [workspace].members)
│   ├── ecstore/             # Erasure-coded storage engine
│   ├── rio/                 # Reader I/O pipeline (encrypt, compress, hash)
│   ├── io-core/             # Buffer pool, storage profiling, backpressure/deadlock policy, lock optimizer, operation progress
│   ├── io-metrics/          # I/O metrics collection
│   ├── common/              # Shared runtime state, globals, data usage types
│   ├── config/              # Configuration types and parsing
│   ├── utils/               # Pure utility functions
│   ├── ...                  # (see "Crate Reference" below)
│   └── e2e_test/            # End-to-end integration tests
└── docs/                    # Design documents and analysis

Main Crate Layers (rustfs/src/)

The main crate is organized in layers, top to bottom:

Layer	Directory	Responsibility
Server	server/	HTTP listener, TLS, CORS, compression, middleware, graceful shutdown
Admin	admin/	Admin API routing, 30+ handler modules, web console
App	app/	Use-case orchestration: object (per-operation modules under app/object/, re-exported as object_usecase), bucket_usecase, multipart_usecase
Storage	storage/	S3 API translation, erasure-coded FS, SSE encryption, RPC, concurrency
Auth	auth.rs	S3 signature verification, credential validation
Config	config/	CLI parsing, config struct, workload profiles

A request flows downward through the layers. No layer should reach upward (e.g., storage must not import from admin).

Crate Reference

Cargo.toml is the authoritative workspace membership and cargo tree is the authoritative dependency graph. This overview deliberately avoids line-count and dependency-depth snapshots because both quickly become stale during refactors.

By Domain

Domain	Current workspace crates	Responsibility
Foundation	checksums, common, config, data-usage, heal-contracts, scanner-contracts, utils	Shared configuration, data-usage models, heal/scanner domain contracts, utilities, and checksums.
I/O and storage	concurrency, ecstore, filemeta, heal, io-core, io-metrics, lifecycle, lock, object-capacity, object-data-cache, replication, rio, rio-v2, s3-client, scanner, storage-api	Erasure-coded object storage, metadata, recovery, lifecycle, replication, locking, cache, I/O pipelines, and the engine-side S3 client for remote tier/transition targets.
Security and identity	credentials, crypto, iam, keystone, kms, policy, security-governance, signer, tls-runtime, trusted-proxies	Credentials, authentication, authorization, encryption, key management, TLS, and security contracts.
Protocols and contracts	extension-schema, madmin, protos, protocols, s3-ops, s3-types, s3select-api, s3select-query	Admin, inter-node, S3, S3 Select, and optional protocol contracts.
Operations and integration	audit, notify, obs, targets, zip	Auditing, observability, event delivery, notification targets, and archive support.
Test support	e2e_test, test-utils	End-to-end validation and shared test bootstrap utilities.

The rustfs binary crate composes these libraries into the running server. ecstore remains the storage engine at the architectural center; its internal module split is tracked under docs/architecture/. rio-v2 is the feature-gated MinIO on-disk format compatibility I/O layer; it ships in no default build (lifecycle: docs/architecture/minio-file-format-compat.md).

Architecture Invariants

These are rules that the codebase should follow. Some are currently violated (marked with ⚠️). Documenting them here makes the violations explicit and trackable.

1. Layers flow downward. Server → Admin/App → Storage → ecstore → rio/io-core. No upward imports.

2. Leaf crates depend only on external crates, with adjudicated exceptions pinned by a guard. config, credentials, and crypto take no internal dependency. io-metrics takes exactly rustfs-s3-ops (transitively rustfs-s3-types), a pure contract crate with no I/O and no global state — adjudicated in rustfs/backlog#1834. madmin left the leaf set when #6166 made it the SigV4-signed admin SDK client; its internal dependency surface is pinned to exactly rustfs-signer. Both pins live in the leaf allowlist in scripts/check_architecture_migration_rules.sh; any other internal dependency fails the guard (crate boundaries).

   • ✅ RESOLVED: the historical utils → config and common → filemeta/madmin edges were removed; do not reintroduce them (see Known Structural Issues).

3. Each type has exactly one definition. Types shared across crates must be defined in one crate and re-exported or imported by others.

   • ⚠️ VIOLATED: ReplicationStats names three unrelated types (crates/data-usage/src/data_usage.rs, crates/obs/src/metrics/collectors/replication.rs, crates/ecstore/src/bucket/replication/replication_state.rs) — a naming collision, not copies; renaming is tracked in rustfs/backlog#1847.
   • LastMinuteLatency has two deliberately different implementations: the per-second bucketed accumulator in crates/scanner-contracts/src/last_minute.rs and the in-memory endpoint-health sample tracker in crates/ecstore/src/bucket/bucket_target_sys.rs (its doc comment explains why it stays local).
   • ✅ RESOLVED: BackpressureConfig and DataUsageInfo each have exactly one definition (crates/io-core/src/backpressure.rs, crates/data-usage/src/data_usage.rs). The zero-consumer BackpressureSettings copy that lingered in io-metrics was removed (rustfs/backlog#1833).

4. ecstore does not serve HTTP or the S3 wire protocol. It operates on storage-level abstractions (objects, buckets, disks, pools) and holds no wire or DTO types of the serving surface. Consuming remote S3-compatible endpoints (ILM tier warm backends, transition targets) is a legitimate engine capability, but it lives in the dedicated rustfs-s3-client crate (crates/s3-client, extracted from the formerly embedded crates/ecstore/src/client/ by rustfs/backlog#1842), not inside ecstore.

   • ⚠️ PARTIALLY VIOLATED: serving-side s3s references remain in ecstore (bucket metadata/replication/lifecycle DTOs and error mapping). The count is ratcheted shrink-only by scripts/check_s3s_footprint.sh (S3S_ECSTORE_FILES_BASELINE; the object_lock module was converted to storage-level types as the first ratchet step). Target state: the baseline reaches zero and ecstore's Cargo.toml drops s3s.

5. The rustfs binary crate is the only place that wires everything together. Individual crates should be testable in isolation.

6. Error types use thiserror with descriptive names (e.g., StorageError, not bare Error).

   • ✅ RESOLVED (strategy): snafu is gone from source (rg -l snafu crates/ rustfs/ is empty) and library code no longer uses anyhow (remaining hits are test code and the e2e_test crate; heal uses thiserror).
   • ⚠️ VIOLATED (naming): 6 crates still export a bare pub enum Error: crypto, filemeta, heal, iam, policy, and replication (src/resync.rs) — all thiserror-derived.

Known Structural Issues

This section documents known problems in the current architecture. It exists so the team can track and address them deliberately.

Critical

• scanner/data-usage duplicate .usage-cache.bin serialization types. The original finding ("common/scanner code duplication, ~3K lines") is resolved: scanner imports the shared data-usage types from rustfs-data-usage (see the pub use rustfs_data_usage::… re-exports at the top of crates/scanner/src/data_usage_define.rs). What remains: scanner and data-usage each hold their own serialization types for the scanner cache file (DataUsageCacheInfo/DataUsageEntryInfo in crates/scanner/src/data_usage_define.rs vs DataUsageCacheInfo/DataUsageEntry in crates/data-usage/src/data_usage.rs); convergence is tracked in rustfs/backlog#1828.

• ecstore is a monolith (265 files, ~288K lines — roughly half is inline #[cfg(test)] code). Measured with find crates/ecstore/src -name '*.rs' | xargs wc -l. It contains disk management, bucket management, erasure coding, replication, lifecycle, RPC, and configuration — all in one crate. It should be decomposed along its existing subdirectories; the split plan lives in docs/architecture/ecstore-module-split-plan.md.

High

• Dependency inversions. Historical utils → config and common → filemeta/madmin edges must stay removed so leaf/helper crates do not regain upward dependencies.

• Three-layer backpressure/deadlock policy bridging across io-core, concurrency, and rustfs/src/storage. The config types are no longer duplicated (BackpressureConfig and DeadlockDetectorConfig are each defined once, in io-core). Storage policies expose and consume explicit projections into the concurrency/io-core policy shapes, and workload admission snapshots are composed through provider registries; later work should use those bridges before deleting compatibility wrappers.

Medium

• Bare Error naming. Error-handling strategy has converged on thiserror (no snafu, no anyhow in library code); the remaining inconsistency is the bare pub enum Error naming in the 6 crates listed under Invariant 6.

• common is mostly parked domain code, not shared utilities. Of its 6,724 lines, ~83% is scanner/heal domain code stranded there to break dependency cycles (metrics.rs, ~4,810 lines of scanner-domain metrics; heal_channel.rs, ~776 lines of heal-domain channel types). The "common vs utils" naming ambiguity is secondary to moving that code to its domain owners.

Cross-Cutting Concerns

Error Handling

The project convention is thiserror for typed errors with descriptive names. See AGENTS.md: "Prefer thiserror for library-facing error types."

// GOOD
#[derive(Debug, thiserror::Error)]
pub enum StorageError {
    #[error("disk not found: {0}")]
    DiskNotFound(String),
}

// AVOID
pub enum Error { ... }        // too generic
anyhow::Result<T>             // in library code (OK in tests/CLI)

Logging & Tracing

• Use tracing crate (info!, warn!, error!, debug!, trace!)
• Structured fields: tracing::info!(bucket = %name, "created bucket")
• Spans for request-scoped context

Metrics

• Prometheus-style metrics via rustfs-obs runtime and schema
• I/O-specific counters via rustfs-io-metrics
• Registration happens at crate level, collection/reporting in rustfs-obs

Testing

• Unit tests: #[cfg(test)] mod tests in the same file
• Integration tests: inside respective crates (not top-level tests/)
• E2E tests: crates/e2e_test/ — tests against a running server
• Run all: make test or cargo nextest run

Startup Sequence

The binary (main.rs) boots in this order:

1. Environment variable compatibility (MINIO_* → RUSTFS_*)
2. Tokio runtime construction
3. CLI argument parsing
4. License, observability, TLS, trusted proxies initialization
5. Config parsing, server address resolution
6. Credentials, endpoints, local disks, lock client initialization
7. Capacity management initialization
8. HTTP server start (S3 API + optional console)
9. ECStore initialization (erasure coding storage engine)
10. Global config, background replication, KMS
11. Optional: FTP/FTPS/WebDAV servers
12. Event notifier, audit system, deadlock detector
13. Bucket metadata, IAM, Keystone, OIDC
14. Scanner and heal manager
15. Metrics system, mark FullReady
16. Wait for shutdown signal → graceful shutdown

Dependency Diagram (Simplified)

                            ┌─────────┐
                            │  rustfs │  (binary + lib)
                            │  main   │
                            └────┬────┘
                                 │
                 ┌───────────────┼───────────────┐
                 │               │               │
            ┌────▼────┐     ┌────▼────┐   ┌──────▼─────┐
            │ server  │     │  admin  │   │    app     │
            │ (HTTP)  │     │(console)│   │(use-cases) │
            └────┬────┘     └────┬────┘   └──────┬─────┘
                 │               │               │
                 └───────────────┼───────────────┘
                                 │
                          ┌──────▼──────┐
                          │   storage   │
                          │ (ecfs, SSE, │
                          │  RPC, ACL)  │
                          └──────┬──────┘
                                 │
              ┌──────────────────┼──────────────────┐
              │                  │                  │
        ┌─────▼──────┐    ┌──────▼──────┐    ┌──────▼──────┐
        │  ecstore   │    │     rio     │    │   io-core   │
        │   (core)   │    │  (readers)  │    │ (buffers)   │
        └─────┬──────┘    └─────────────┘    └─────────────┘
              │
     ┌─────┬──┼──┬─────┬──────┐
     │     │  │  │     │      │
 common utils config policy filemeta ...

How to Navigate

• "Where does S3 PutObject go?" server/ routes → app/object_usecase validates → storage/ecfs encodes → ecstore distributes → rio encrypts/compresses → io-core supplies buffers

• "Where are bucket policies enforced?" app/bucket_usecase calls into crates/policy/

• "Where is replication configured?" admin/handlers/replication.rs and admin/handlers/site_replication.rs for API, rustfs/src/site_replication/ for the site-replication service subsystem (state, peer transport, retry queue, repair, hooks), ecstore/src/bucket/replication/ for engine

• "Where do I add a new admin endpoint?" Add handler in admin/handlers/, register in admin/router.rs

• "Where do I add a new metric?" Define descriptor/collector in crates/obs/src/metrics/, expose via /minio/v2/metrics

---

Inspired by matklad's ARCHITECTURE.md and rust-analyzer's architecture.md.