gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-07-26 08:18:18 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(architecture): prevent workspace overview from drifting (#4983)

Browse Source 
Replace stale dependency-depth and line-count snapshots with a domain overview based on the current Cargo workspace. This preserves the high-level architecture while avoiding references to removed crates and fragile size estimates.

Constraint: Workspace membership changes independently of architecture documentation updates
Rejected: Refresh the old numeric snapshots | they would immediately become stale again
Confidence: high
Scope-risk: narrow
Reversibility: clean
Directive: Keep Cargo.toml as the source of truth for workspace membership
Tested: git diff --check; scripts/check_doc_paths.sh; cargo metadata --no-deps --format-version 1
Not-tested: make pre-commit (documentation-only change)
This commit is contained in:
darion-yaphet
2026-07-18 10:49:05 +08:00
committed by GitHub
parent 906805568b
commit 5ec124bf23
1 changed files with 18 additions and 117 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ARCHITECTURE.md
+18 -117
View File
@@ -41,7 +41,7 @@ The repository is a Cargo workspace with a flat `crates/` layout:
``` ```
rustfs/ # Workspace root (virtual manifest) rustfs/ # Workspace root (virtual manifest)
├── rustfs/ # Main binary + library crate (75K lines) ├── rustfs/ # Main binary + library crate
│ └── src/ │ └── src/
│ ├── main.rs # Entry point, startup sequence │ ├── main.rs # Entry point, startup sequence
│ ├── lib.rs # Module tree root │ ├── lib.rs # Module tree root
@@ -53,7 +53,7 @@ rustfs/ # Workspace root (virtual manifest)
│ ├── config/ # CLI args, config parsing, workload profiles │ ├── config/ # CLI args, config parsing, workload profiles
│ └── ... │ └── ...
├── crates/ # library crates (authoritative list: Cargo.toml [workspace].members) ├── crates/ # library crates (authoritative list: Cargo.toml [workspace].members)
│ ├── ecstore/ # Erasure-coded storage engine (⚠️ 87K lines) │ ├── ecstore/ # Erasure-coded storage engine
│ ├── rio/ # Reader I/O pipeline (encrypt, compress, hash) │ ├── rio/ # Reader I/O pipeline (encrypt, compress, hash)
│ ├── io-core/ # Zero-copy I/O, scheduling, buffer pool │ ├── io-core/ # Zero-copy I/O, scheduling, buffer pool
│ ├── io-metrics/ # I/O metrics collection │ ├── io-metrics/ # I/O metrics collection
@@ -83,124 +83,25 @@ A request flows **downward** through the layers. No layer should reach upward
### Crate Reference ### Crate Reference
> Depth levels, line counts, and crate counts in this section are a `Cargo.toml` is the authoritative workspace membership and `cargo tree` is the
> point-in-time snapshot and drift with refactors. Treat them as orders of authoritative dependency graph. This overview deliberately avoids line-count
> magnitude; `Cargo.toml` and `cargo tree` are the source of truth. and dependency-depth snapshots because both quickly become stale during
refactors.
Crates are organized in a dependency DAG with 9 depth levels (0 = leaf, 8 = top):
```
Depth 0 — LEAF (no internal deps):
appauth, checksums, config, credentials, crypto, io-metrics,
madmin, s3-common, workers, zip
Depth 1:
io-core (→ io-metrics)
policy (→ config, credentials, crypto)
utils (historical → config edge removed; now effectively leaf)
Depth 2:
concurrency, filemeta, keystone, kms, lock, obs,
signer, targets, trusted-proxies
Depth 3:
common (historical → filemeta/madmin edges removed; now effectively leaf)
Depth 4:
object-capacity, protos, rio
Depth 5 — CORE:
ecstore (16 internal deps, 11 dependents — the architectural heart)
Depth 6:
audit, heal, iam, metrics, notify, s3select-api, scanner
Depth 7:
object-io, protocols, s3select-query
Depth 8 — TOP:
rustfs (35 internal deps — the binary, depends on almost everything)
```
#### By Domain #### By Domain
**Core Infrastructure:** | Domain | Current workspace crates | Responsibility |
|--------|--------------------------|----------------|
| Foundation | `checksums`, `common`, `config`, `data-usage`, `utils` | Shared configuration, data-usage models, utilities, and checksums. |
| I/O and storage | `concurrency`, `ecstore`, `filemeta`, `heal`, `io-core`, `io-metrics`, `lifecycle`, `lock`, `object-capacity`, `object-data-cache`, `replication`, `rio`, `rio-v2`, `scanner`, `storage-api` | Erasure-coded object storage, metadata, recovery, lifecycle, replication, locking, cache, and I/O pipelines. |
| Security and identity | `credentials`, `crypto`, `iam`, `keystone`, `kms`, `policy`, `security-governance`, `signer`, `tls-runtime`, `trusted-proxies` | Credentials, authentication, authorization, encryption, key management, TLS, and security contracts. |
| Protocols and contracts | `extension-schema`, `madmin`, `protos`, `protocols`, `s3-ops`, `s3-types`, `s3select-api`, `s3select-query` | Admin, inter-node, S3, S3 Select, and optional protocol contracts. |
| Operations and integration | `audit`, `notify`, `obs`, `targets`, `zip` | Auditing, observability, event delivery, notification targets, and archive support. |
| Test support | `e2e_test`, `test-utils` | End-to-end validation and shared test bootstrap utilities. |
| Crate | Lines | Purpose | The `rustfs` binary crate composes these libraries into the running server.
|-------|-------|---------| `ecstore` remains the storage engine at the architectural center; its internal
| `config` | 3.3K | Configuration types and environment parsing | module split is tracked under `docs/architecture/`.
| `utils` | 8.7K | Pure utilities (paths, compression, network, retry) |
| `common` | 4.4K | Shared runtime state, globals, data usage types, metrics |
| `madmin` | 5.5K | Admin API request/response types |
**I/O Pipeline:**
| Crate | Lines | Purpose |
|-------|-------|---------|
| `io-core` | 6.5K | Zero-copy I/O, buffer pool, direct I/O, scheduling, backpressure |
| `io-metrics` | 4.5K | I/O operation metrics and counters |
| `rio` | 6.9K | Composable reader chain (encrypt → compress → hash → limit) |
| `object-io` | 2.4K | High-level object read/write using rio + ecstore |
| `concurrency` | 0.8K | Shared concurrency contract types: workload admission snapshots, worker-slot pool, policy types (runtime control lives in `rustfs/src/storage`) |
**Storage Engine:**
| Crate | Lines | Purpose |
|-------|-------|---------|
| `ecstore` | 87K | ⚠️ Erasure-coded storage: disks, pools, buckets, replication, lifecycle |
| `filemeta` | 10K | File/object metadata types and versioning |
| `checksums` | 732 | Checksum computation |
| `lock` | 7.1K | Distributed lock manager |
| `heal` | 5.9K | Data healing / bitrot repair |
| `scanner` | 5.4K | Background data usage scanner |
| `object-capacity` | 2.5K | Capacity tracking and management |
**Security & Auth:**
| Crate | Lines | Purpose |
|-------|-------|---------|
| `crypto` | 1.6K | Encryption primitives |
| `credentials` | 713 | Credential types (access key / secret key) |
| `signer` | 1.4K | S3 v4 request signing |
| `iam` | 9.0K | Identity and access management |
| `policy` | 8.8K | Policy engine (S3 bucket/IAM policies) |
| `kms` | 8.1K | Key management service integration |
| `keystone` | 1.9K | OpenStack Keystone auth |
| `appauth` | 143 | Application-level auth tokens |
**Protocol & API:**
| Crate | Lines | Purpose |
|-------|-------|---------|
| `protos` | 5.7K | Protobuf/gRPC definitions for inter-node RPC |
| `protocols` | 18K | FTP/FTPS, WebDAV, Swift API support |
| `s3-common` | 738 | Shared S3 types |
| `s3select-api` | 1.9K | S3 Select interface |
| `s3select-query` | 3.6K | S3 Select query engine |
**Observability:**
| Crate | Lines | Purpose |
|-------|-------|---------|
| `metrics` | 8.4K | Prometheus metric collectors |
| `io-metrics` | 4.5K | I/O-specific metrics |
| `obs` | 5.6K | OpenTelemetry tracing and telemetry |
| `audit` | 2.4K | Audit logging |
**Events:**
| Crate | Lines | Purpose |
|-------|-------|---------|
| `notify` | 5.5K | Event notification system |
| `targets` | 3.2K | Notification targets (Kafka, AMQP, webhook, etc.) |
**Other:**
| Crate | Lines | Purpose |
|-------|-------|---------|
| `trusted-proxies` | 4.0K | Trusted proxy / IP forwarding |
| `zip` | 986 | ZIP archive support for bulk downloads |
| `workers` | 136 | Simple worker abstraction |
## Architecture Invariants ## Architecture Invariants
@@ -212,7 +113,7 @@ Depth 8 — TOP:
No upward imports. No upward imports.
2. **Leaf crates have zero internal dependencies.** `config`, `credentials`, `crypto`, 2. **Leaf crates have zero internal dependencies.** `config`, `credentials`, `crypto`,
`io-metrics`, `madmin`, `s3-common` should depend only on external crates. `io-metrics`, and `madmin` should depend only on external crates.
- ✅ RESOLVED: the historical `utils → config` and `common → filemeta`/`madmin` - ✅ RESOLVED: the historical `utils → config` and `common → filemeta`/`madmin`
edges were removed; do not reintroduce them (see Known Structural Issues). edges were removed; do not reintroduce them (see Known Structural Issues).