mirror of
https://github.com/rustfs/rustfs.git
synced 2026-07-26 08:18:18 +00:00
chore(docs): refresh agent docs, guard doc paths, archive plans (#4203)
Agent-instruction and architecture docs had drifted from the code: - CLAUDE.md: slim to commands + pointers; fix wrong claim that `make pre-commit` is the full pre-PR gate (that is `make pre-pr`); drop stale pre-#3929 file paths and merged bug narratives - AGENTS.md: drop dead `rust-refactor-helper` skill rule and the hand-maintained (already stale) scoped-AGENTS index; link architecture docs from Sources of Truth - .github/AGENTS.md: replace the outdated copied CI command matrix with a pointer to ci.yml - crates/AGENTS.md: merge duplicated Testing sections - ARCHITECTURE.md: resolve the utils->config contradiction (edges are removed), mark volatile counts as snapshots, fix a bad path - docs/architecture: add README router; move one-shot plans/trackers (rebalance-decommission phases, migration-progress ledger, PR template) to docs/superpowers/plans with archive headers; fix stale source paths in kept inventories (core/sets.rs, core/pools.rs, store/mod.rs, startup_* split from #3671) - docs/operations/tier-ilm-debugging.md: extracted tier debugging playbook with corrected paths - scripts/check_doc_paths.sh: new guard failing pre-commit/pre-pr when instruction/architecture docs reference nonexistent file paths - .claude/skills: add tier-debug and arch-checks repo skills; .gitignore now keeps .claude/skills and docs/operations committable Verification: ./scripts/check_doc_paths.sh, ./scripts/check_architecture_migration_rules.sh (both pass) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Zhengchao An
@@ -0,0 +1,51 @@
|
||||
---
|
||||
name: arch-checks
|
||||
description: Resolve failures from the repository's architecture guard scripts — check_layer_dependencies.sh, check_architecture_migration_rules.sh, check_unsafe_code_allowances.sh, check_logging_guardrails.sh, check_doc_paths.sh. Use when make pre-commit / pre-pr or CI fails on one of these checks.
|
||||
---
|
||||
|
||||
# Architecture Guard Checks
|
||||
|
||||
All five run in `make pre-commit` / `make pre-pr` and in CI. Fix the cause;
|
||||
never weaken a check to get green.
|
||||
|
||||
## `check_layer_dependencies.sh` — layer DAG in `rustfs/src`
|
||||
|
||||
Enforces `interface (admin, storage/ecfs, storage/s3_api) → app → infra`; no
|
||||
upward imports. Known legacy violations live in
|
||||
`scripts/layer-dependency-baseline.txt`.
|
||||
|
||||
- **New violation**: restructure your change so the dependency points
|
||||
downward (move the shared type/function to the lower layer).
|
||||
- **You legitimately removed a baseline entry**: run
|
||||
`./scripts/check_layer_dependencies.sh --update-baseline` and commit the
|
||||
shrunken baseline. Never add new entries to the baseline to make a new
|
||||
violation pass.
|
||||
|
||||
## `check_architecture_migration_rules.sh` — required doc sections
|
||||
|
||||
Asserts that the core docs under `docs/architecture/` (overview,
|
||||
crate-boundaries, runtime-lifecycle, readiness-matrix,
|
||||
storage-control-data-plane, global-state-crate-split-plan,
|
||||
ecstore-module-split-plan, …) still contain specific headings and exact
|
||||
source lines. If it fails after a doc edit, you reworded or removed a
|
||||
guarded line — restore the wording or update the script deliberately in the
|
||||
same PR, with rationale.
|
||||
|
||||
## `check_unsafe_code_allowances.sh`
|
||||
|
||||
Every `#[allow(unsafe_code)]` needs a `SAFETY:` comment within a few lines.
|
||||
Write the actual safety argument; don't add a placeholder.
|
||||
|
||||
## `check_logging_guardrails.sh`
|
||||
|
||||
A fixed list of security-sensitive files (auth, IAM, KMS, admin handlers…)
|
||||
is scanned for logging violations. If you created a new sensitive file,
|
||||
consider adding it to the script's `checked_files` list.
|
||||
|
||||
## `check_doc_paths.sh`
|
||||
|
||||
Instruction/architecture docs (`AGENTS.md`, `CLAUDE.md`, `ARCHITECTURE.md`,
|
||||
`docs/architecture/*.md`) must not reference repo file paths that no longer
|
||||
exist. If your refactor moved code, update the docs that point at it — the
|
||||
error message lists `doc -> stale-path` pairs. Historical plans under
|
||||
`docs/superpowers/plans/` are exempt.
|
||||
@@ -0,0 +1,34 @@
|
||||
---
|
||||
name: tier-debug
|
||||
description: Debug ILM tiering / lifecycle transition issues — NoSuchVersion on tier GET, restore failures, xl.meta inspection, remote-tier versionId tracing. Use when investigating tiered/transitioned objects, warm backends, or transition metadata.
|
||||
---
|
||||
|
||||
# Tier / ILM Debugging
|
||||
|
||||
Full playbook: [docs/operations/tier-ilm-debugging.md](../../../docs/operations/tier-ilm-debugging.md)
|
||||
— read it before changing tier code.
|
||||
|
||||
Quick moves:
|
||||
|
||||
```bash
|
||||
# Inspect transition metadata on disk (one xl.meta per erasure shard disk)
|
||||
cargo run -p rustfs-filemeta --example dump_fileinfo -- "/path/to/{bucket}/{object}/xl.meta"
|
||||
|
||||
# Trace what versionId is sent to the remote tier
|
||||
RUST_LOG=rustfs_ecstore::bucket::lifecycle=debug ./target/debug/rustfs …
|
||||
```
|
||||
|
||||
Interpretation:
|
||||
|
||||
- `transition_ver_id: <none>` → correct for an unversioned tier bucket; no
|
||||
`versionId` must be sent on tier GET/DELETE.
|
||||
- `transition_ver_id: 00000000-…` (nil) → corrupt legacy write-back; readers
|
||||
must filter it out, never send it.
|
||||
- Empty-string `transitioned-versionID` metadata under both
|
||||
`x-rustfs-internal-*` and `x-minio-internal-*` keys → object went to an
|
||||
unversioned tier bucket.
|
||||
|
||||
Code entry points: `crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs`
|
||||
(ILM actions), `crates/ecstore/src/services/tier/` (warm backends),
|
||||
`crates/filemeta/src/filemeta/version.rs` (metadata read/write + regression
|
||||
tests).
|
||||
Symlink
+1
@@ -0,0 +1 @@
|
||||
../.agents/skills
|
||||
@@ -8,14 +8,19 @@ setup-hooks: ## Set up git hooks
|
||||
chmod +x .git/hooks/pre-commit
|
||||
@echo "✅ Git hooks setup complete!"
|
||||
|
||||
.PHONY: doc-paths-check
|
||||
doc-paths-check: ## Check that instruction/architecture docs reference existing file paths
|
||||
@echo "📄 Checking doc path references..."
|
||||
./scripts/check_doc_paths.sh
|
||||
|
||||
.PHONY: pre-commit
|
||||
pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check quick-check ## Run fast pre-commit checks without clippy/full tests
|
||||
pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check doc-paths-check quick-check ## Run fast pre-commit checks without clippy/full tests
|
||||
@echo "✅ All pre-commit checks passed!"
|
||||
|
||||
.PHONY: pre-pr
|
||||
pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check clippy-check test ## Run full pre-PR checks with clippy and tests
|
||||
pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check doc-paths-check clippy-check test ## Run full pre-PR checks with clippy and tests
|
||||
@echo "✅ All pre-PR checks passed!"
|
||||
|
||||
.PHONY: dev-check
|
||||
dev-check: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check quick-check ## Run fast local development checks
|
||||
dev-check: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check doc-paths-check quick-check ## Run fast local development checks
|
||||
@echo "✅ Fast development checks passed!"
|
||||
|
||||
+4
-19
@@ -19,22 +19,7 @@ Applies to `.github/` and repository pull-request operations.
|
||||
|
||||
## CI Alignment
|
||||
|
||||
When changing CI-sensitive behavior, keep local validation aligned with `.github/workflows/ci.yml`.
|
||||
|
||||
Current `test-and-lint` gate includes:
|
||||
|
||||
- `cargo nextest run --all --exclude e2e_test`
|
||||
- `cargo test --all --doc`
|
||||
- `cargo test -p rustfs get_object_chunk_fast_path`
|
||||
- `cargo test -p rustfs materialize_chunk_stream_before_commit`
|
||||
- `touch rustfs/build.rs`
|
||||
- `cargo build -p rustfs --bins --jobs 2`
|
||||
- `cargo test -p e2e_test archive_multipart_roundtrip_preserves_bytes`
|
||||
- `cargo test -p e2e_test presigned_get_and_reverse_proxy_preserve_multipart_bytes_with_fast_path`
|
||||
- `cargo fmt --all --check`
|
||||
- `cargo clippy --all-targets -- -D warnings`
|
||||
- `cargo nextest run -p rustfs -p rustfs-ecstore --features rio-v2`
|
||||
- `cargo test -p rustfs --doc --features rio-v2`
|
||||
- `cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings`
|
||||
- `./scripts/check_layer_dependencies.sh`
|
||||
- `./scripts/check_architecture_migration_rules.sh`
|
||||
When changing CI-sensitive behavior, keep local validation aligned with
|
||||
`.github/workflows/ci.yml`. Read the workflow file directly for the current
|
||||
gate steps — do not rely on (or add) a copied command list here; copies go
|
||||
stale. `make pre-pr` is the local equivalent of the main gate.
|
||||
|
||||
+10
-1
@@ -4,7 +4,9 @@
|
||||
.vscode
|
||||
.cursor
|
||||
.direnv/
|
||||
.claude/
|
||||
# Ignore local Claude state but keep shared repo skills committable
|
||||
.claude/*
|
||||
!.claude/skills
|
||||
.tmp/
|
||||
/test
|
||||
/logs
|
||||
@@ -51,6 +53,10 @@ __pycache__/
|
||||
docs/*
|
||||
!docs/architecture/
|
||||
!docs/architecture/**
|
||||
!docs/operations/
|
||||
!docs/operations/**
|
||||
!docs/superpowers/
|
||||
!docs/superpowers/**
|
||||
docs/heal-scanner-logging-governance.md
|
||||
docs/benchmark/rustfs-target-bench/
|
||||
docs/benchmark/*.md
|
||||
@@ -74,3 +80,6 @@ crates/*/docs
|
||||
fuzz/target
|
||||
outputs
|
||||
worktrees/*
|
||||
|
||||
# Local AI-agent review artifacts (omo evidence dumps)
|
||||
.omo/
|
||||
|
||||
@@ -27,10 +27,6 @@ If repo-level instructions conflict, follow the nearest file and keep behavior a
|
||||
- Keep source code, comments, commit messages, and PR title/body in English.
|
||||
- Be concise. Avoid sycophantic openers, closing fluff, and verbose status reporting.
|
||||
|
||||
## Skill Usage
|
||||
|
||||
- Do not use the `rust-refactor-helper` skill in any scenario.
|
||||
|
||||
## Change Style for Existing Logic
|
||||
|
||||
- Prefer direct, local code over extracting one-off helpers.
|
||||
@@ -57,6 +53,13 @@ If repo-level instructions conflict, follow the nearest file and keep behavior a
|
||||
- Local quality commands: `Makefile` and `.config/make/`
|
||||
- CI quality gates: `.github/workflows/ci.yml`
|
||||
- PR template: `.github/pull_request_template.md`
|
||||
- High-level architecture and crate map: `ARCHITECTURE.md`
|
||||
- Migration guardrails, readiness contracts, support matrices:
|
||||
`docs/architecture/README.md` (routes by audience)
|
||||
- Historical implementation plans and trackers: `docs/superpowers/plans/`
|
||||
- Shared agent skills (all tools): `.agents/skills/` — Claude Code reads them
|
||||
through the `.claude/skills` symlink; add new skills to `.agents/skills/`
|
||||
only, never as separate copies per tool
|
||||
|
||||
Avoid duplicating long crate lists or command matrices in instruction files.
|
||||
Reference the source files above instead.
|
||||
@@ -136,14 +139,13 @@ cargo run -p rustfs-filemeta --example dump_fileinfo -- "/path/to/file/xl.meta"
|
||||
|
||||
## Scoped Guidance in This Repository
|
||||
|
||||
- `.github/AGENTS.md`
|
||||
- `crates/AGENTS.md`
|
||||
- `crates/config/AGENTS.md`
|
||||
- `crates/ecstore/AGENTS.md`
|
||||
- `crates/e2e_test/AGENTS.md`
|
||||
- `crates/iam/AGENTS.md`
|
||||
- `crates/kms/AGENTS.md`
|
||||
- `crates/policy/AGENTS.md`
|
||||
- `crates/targets/AGENTS.md`
|
||||
- `rustfs/src/admin/AGENTS.md`
|
||||
- `rustfs/src/storage/AGENTS.md`
|
||||
Many crates and modules carry their own `AGENTS.md` with path-specific rules
|
||||
(security boundaries, lock ordering, domain invariants). Before editing a
|
||||
path, check for the nearest one:
|
||||
|
||||
```bash
|
||||
git ls-files '*AGENTS.md'
|
||||
```
|
||||
|
||||
The nearest file wins. Do not maintain a hand-written index of these files
|
||||
here — it goes stale.
|
||||
|
||||
+11
-6
@@ -1,6 +1,6 @@
|
||||
# ARCHITECTURE.md
|
||||
|
||||
> Last updated: 2026-04-13 · Revision: 1 (draft)
|
||||
> Last updated: 2026-07-02 · Revision: 2
|
||||
>
|
||||
> This document describes the high-level architecture of RustFS.
|
||||
> If you want to familiarize yourself with the code base, you are in the right place!
|
||||
@@ -52,7 +52,7 @@ rustfs/ # Workspace root (virtual manifest)
|
||||
│ ├── auth.rs # S3 request authentication
|
||||
│ ├── config/ # CLI args, config parsing, workload profiles
|
||||
│ └── ...
|
||||
├── crates/ # 39 library crates
|
||||
├── crates/ # library crates (authoritative list: Cargo.toml [workspace].members)
|
||||
│ ├── ecstore/ # Erasure-coded storage engine (⚠️ 87K lines)
|
||||
│ ├── rio/ # Reader I/O pipeline (encrypt, compress, hash)
|
||||
│ ├── io-core/ # Zero-copy I/O, scheduling, buffer pool
|
||||
@@ -83,6 +83,10 @@ A request flows **downward** through the layers. No layer should reach upward
|
||||
|
||||
### Crate Reference
|
||||
|
||||
> Depth levels, line counts, and crate counts in this section are a
|
||||
> point-in-time snapshot and drift with refactors. Treat them as orders of
|
||||
> magnitude; `Cargo.toml` and `cargo tree` are the source of truth.
|
||||
|
||||
Crates are organized in a dependency DAG with 9 depth levels (0 = leaf, 8 = top):
|
||||
|
||||
```
|
||||
@@ -93,14 +97,14 @@ Depth 0 — LEAF (no internal deps):
|
||||
Depth 1:
|
||||
io-core (→ io-metrics)
|
||||
policy (→ config, credentials, crypto)
|
||||
utils (→ config) ⚠️ inverted: utils should be leaf
|
||||
utils (historical → config edge removed; now effectively leaf)
|
||||
|
||||
Depth 2:
|
||||
concurrency, filemeta, keystone, kms, lock, obs,
|
||||
signer, targets, trusted-proxies
|
||||
|
||||
Depth 3:
|
||||
common (→ filemeta, madmin) ⚠️ inverted: common should be leaf
|
||||
common (historical → filemeta/madmin edges removed; now effectively leaf)
|
||||
|
||||
Depth 4:
|
||||
object-capacity, protos, rio
|
||||
@@ -209,7 +213,8 @@ Depth 8 — TOP:
|
||||
|
||||
2. **Leaf crates have zero internal dependencies.** `config`, `credentials`, `crypto`,
|
||||
`io-metrics`, `madmin`, `s3-common` should depend only on external crates.
|
||||
- ⚠️ VIOLATED: `utils` depends on `config`, `common` depends on `filemeta` and `madmin`.
|
||||
- ✅ RESOLVED: the historical `utils → config` and `common → filemeta`/`madmin`
|
||||
edges were removed; do not reintroduce them (see Known Structural Issues).
|
||||
|
||||
3. **Each type has exactly one definition.** Types shared across crates must be defined
|
||||
in one crate and re-exported or imported by others.
|
||||
@@ -249,7 +254,7 @@ Depth 8 — TOP:
|
||||
not regain upward dependencies.
|
||||
|
||||
- **Three-layer BackpressureConfig/DeadlockConfig duplication** across io-core,
|
||||
concurrency, and rustfs/storage. Storage policies now expose and consume
|
||||
concurrency, and `rustfs/src/storage`. Storage policies now expose and consume
|
||||
explicit projections into the concurrency/io-core policy shapes, and workload
|
||||
admission snapshots are composed through provider registries; later work
|
||||
should use those bridges before deleting compatibility wrappers.
|
||||
|
||||
@@ -1,188 +1,47 @@
|
||||
# RustFS — CLAUDE.md
|
||||
|
||||
S3-compatible object store in Rust, derived from MinIO. Erasure-coded, multi-pool, supports ILM tiering/lifecycle.
|
||||
S3-compatible object store in Rust, derived from MinIO. Erasure-coded,
|
||||
multi-pool, supports ILM tiering/lifecycle.
|
||||
|
||||
Agent rules (change style, verification, PR conventions, scoped per-crate
|
||||
guidance) live in [AGENTS.md](AGENTS.md) — follow it. This file only adds
|
||||
what Claude Code needs on top: commands and pointers.
|
||||
|
||||
## Commands
|
||||
|
||||
```bash
|
||||
cargo build --release --bin rustfs # production binary
|
||||
cargo build # dev build
|
||||
cargo check -p <crate> # fast type-check one crate
|
||||
cargo test -p <crate> # test one crate
|
||||
cargo fmt --all # format (required before PR)
|
||||
make pre-commit # full pre-PR gate (fmt + clippy + test)
|
||||
make build-docker BUILD_OS=ubuntu22.04 # Docker cross-build
|
||||
cargo build --release --bin rustfs # production binary
|
||||
cargo check -p <crate> # fast type-check one crate
|
||||
cargo test -p <crate> # test one crate
|
||||
cargo fmt --all # format (required before PR)
|
||||
make pre-commit # fast gate: fmt + arch checks + quick-check (NO clippy/tests)
|
||||
make pre-pr # full pre-PR gate: fmt + arch checks + clippy + tests
|
||||
make build-docker BUILD_OS=ubuntu22.04
|
||||
```
|
||||
|
||||
> **Docker build note**: `buildx build` without `--load` keeps the image in the buildx cache only — `docker run` will use a stale local image. The Makefile already includes `--load`; if you suspect a stale binary, add `--no-cache` to the `buildx build` invocation inside `.config/make/build-docker.mak`.
|
||||
> **Docker build note**: `buildx build` without `--load` keeps the image in
|
||||
> the buildx cache only — `docker run` will use a stale local image. The
|
||||
> Makefile already includes `--load`; if you suspect a stale binary, add
|
||||
> `--no-cache` to the `buildx build` invocation inside
|
||||
> `.config/make/build-docker.mak`.
|
||||
|
||||
> Agent/PR rules: see `.github/copilot-instructions.md`.
|
||||
> Crate membership: `Cargo.toml` `[workspace].members`.
|
||||
> CI gates: `.github/workflows/ci.yml`.
|
||||
## Where to look (do not duplicate here)
|
||||
|
||||
## Workspace layout
|
||||
- Crate membership: `Cargo.toml` `[workspace].members`
|
||||
- Architecture, layering, crate map: [ARCHITECTURE.md](ARCHITECTURE.md)
|
||||
- Migration guardrails & readiness contracts: [docs/architecture/](docs/architecture/README.md)
|
||||
- CI gates: `.github/workflows/ci.yml` (source of truth; never copy its steps into docs)
|
||||
- Tier/ILM transition debugging (xl.meta inspection, versionId tracing):
|
||||
[docs/operations/tier-ilm-debugging.md](docs/operations/tier-ilm-debugging.md)
|
||||
|
||||
```
|
||||
rustfs/src/main.rs # binary entry point
|
||||
crates/ecstore/src/
|
||||
set_disk.rs # ErasureSet: transition_object, restore_transitioned_object
|
||||
store.rs / store_api/ # ECStore trait + ObjectInfo / TransitionedObject types
|
||||
bucket/lifecycle/
|
||||
bucket_lifecycle_ops.rs # ILM actions: transition_object, expire_transitioned_object,
|
||||
# get_transitioned_object_reader, gen_transition_objname
|
||||
tier_sweeper.rs # background sweep: delete_object_from_remote_tier
|
||||
tier/
|
||||
warm_backend.rs # WarmBackend trait (put/get/remove/in_use)
|
||||
warm_backend_s3.rs # HTTP-client based (TransitionClient) — used for S3/MinIO
|
||||
warm_backend_s3sdk.rs # aws-sdk-s3 based — alternative S3 backend
|
||||
warm_backend_minio.rs / _rustfs.rs / … # per-provider wrappers (all delegate to _s3 or _s3sdk)
|
||||
tier.rs # TierConfigMgr, new_warm_backend dispatch
|
||||
client/transition_api.rs # TransitionClient HTTP plumbing; UploadInfo, to_object_info
|
||||
client/api_put_object_streaming.rs # put_object_do → UploadInfo (version_id from x-amz-version-id)
|
||||
crates/filemeta/src/
|
||||
filemeta.rs # FileMeta (xl.meta top-level), is_skip_meta_key
|
||||
filemeta/version.rs # FileMetaVersion, MetaObject, MetaDeleteMarker
|
||||
# → to_fileinfo() reads transition_version_id
|
||||
# → set_transition() writes raw UUID bytes
|
||||
# → From<FileInfo> for MetaObject writes all meta
|
||||
fileinfo.rs # FileInfo struct (transition_version_id: Option<Uuid>)
|
||||
examples/
|
||||
dump_fileinfo.rs # CLI: parse xl.meta, print transition fields + metadata
|
||||
dump_versions.rs # CLI: list all versions in xl.meta
|
||||
crates/utils/src/http/metadata_compat.rs # SUFFIX_* constants, insert_bytes/get_bytes (dual RustFS+MinIO keys)
|
||||
```
|
||||
## Domain conventions worth knowing up front
|
||||
|
||||
## Metadata key conventions
|
||||
|
||||
Internal metadata is stored under **both** `x-rustfs-internal-<suffix>` and `x-minio-internal-<suffix>` for MinIO interoperability. `get_bytes` prefers the RustFS key with MinIO fallback.
|
||||
|
||||
Key suffixes (from `metadata_compat.rs`):
|
||||
| Suffix | Meaning |
|
||||
|--------|---------|
|
||||
| `transition-status` | `"complete"` when tiered |
|
||||
| `transitioned-object` | tier key path (without prefix) |
|
||||
| `transitioned-versionID` | S3 version_id returned by tier PUT (16 raw UUID bytes, or absent) |
|
||||
| `transition-tier` | tier name |
|
||||
| `tier-free-versionID` | delete-marker version for free-version sweep |
|
||||
|
||||
## Tier / ILM transition architecture
|
||||
|
||||
### Transition flow (hot → cold)
|
||||
1. `transition_object` (lifecycle_ops) → `ECStore::transition_object` → `set_disk.rs`
|
||||
2. `gen_transition_objname(bucket)` → `{sha256_hash[0..16]}/{uuid[0..2]}/{uuid[2..4]}/{uuid}` (unique per object version)
|
||||
3. `tgt_client.put_with_meta(dest_obj, …)` → returns `rv: String` (remote S3 version_id, or `""`)
|
||||
4. `fi.transition_version_id = if rv.is_empty() { None } else { Some(Uuid::parse_str(&rv)?) }`
|
||||
5. `fi.transitioned_objname = dest_obj` (without tier prefix)
|
||||
6. Written to xl.meta via `MetaObject::from(FileInfo)` → `insert_bytes(SUFFIX_TRANSITIONED_VERSION_ID, uuid.as_bytes())` (16 raw bytes)
|
||||
|
||||
### Tier GET flow (restore/read)
|
||||
`get_transitioned_object_reader` (lifecycle_ops):
|
||||
- reads `oi.transitioned_object.name` (= `fi.transitioned_objname`)
|
||||
- reads `oi.transitioned_object.version_id` (= `fi.transition_version_id.to_string()` or `""`)
|
||||
- calls `warm_backend.get(name, version_id, opts)`
|
||||
- `warm_backend_s3.rs::get`: adds `?versionId=…` only when `rv != ""`
|
||||
|
||||
### Tier prefix handling
|
||||
`WarmBackendS3::get_dest(object)` prepends `self.prefix` to the object name.
|
||||
`transitioned_objname` is stored **without** the prefix — `get_dest` adds it on every call.
|
||||
|
||||
### xl.meta on disk
|
||||
Path: `{disk}/{bucket}/{object}/xl.meta` — one per erasure shard disk.
|
||||
All shards should be identical for a healthy object.
|
||||
|
||||
## Known bugs & fixes
|
||||
|
||||
### Bug 1: `NoSuchVersion` on tier GET — nil UUID sent as versionId
|
||||
**Root cause**: `transitioned-versionID` metadata key exists with empty string value (0 bytes). Old reading code:
|
||||
```rust
|
||||
// OLD — unwrap_or_default() converts 0-byte or wrong-length slice to Uuid::nil()
|
||||
get_bytes(…).map(|v| Uuid::from_slice(v.as_slice()).unwrap_or_default())
|
||||
// → Some(Uuid::nil()) → sends ?versionId=00000000-… → NoSuchVersion
|
||||
```
|
||||
**Fix** (version.rs, `MetaObject::to_fileinfo` + `MetaDeleteMarker::to_fileinfo`):
|
||||
```rust
|
||||
get_bytes(…)
|
||||
.and_then(|v| Uuid::from_slice(v.as_slice()).ok()) // None for wrong-length bytes
|
||||
.filter(|u| !u.is_nil()) // None for nil UUID (old write-back)
|
||||
```
|
||||
**Regression tests** (`crates/filemeta/src/filemeta/version.rs` `mod tests`): 6 tests cover absent key, empty bytes, nil UUID, and valid UUID round-trip for both `MetaObject` and `MetaDeleteMarker` paths.
|
||||
|
||||
### Bug 2: `warm_backend_s3sdk.rs` ignored rv and range opts
|
||||
**Fix**: added `req.version_id(rv)` and `req.range(…)` to GET; `req.version_id(rv)` to DELETE.
|
||||
|
||||
### Bug 4: `set_disk::copy_object` returns 501 for tiered objects (storage class restore)
|
||||
**Root cause**: `set_disk::copy_object` immediately returns `StorageError::NotImplemented` when `src_info.metadata_only = false`. For tiered objects, `metadata_only` is never set to `true` (guarded by `transitioned_object.tier.is_empty()`). So `mc cp --storage-class STANDARD obj obj` on a tiered object always returns 501.
|
||||
**Fix** (`crates/ecstore/src/set_disk.rs`, `copy_object`):
|
||||
```rust
|
||||
if !src_info.metadata_only {
|
||||
if path_join_buf(&[src_bucket, src_object]) == path_join_buf(&[dst_bucket, dst_object]) {
|
||||
if let Some(mut put_reader) = src_info.put_object_reader.take() {
|
||||
return self.put_object(dst_bucket, dst_object, &mut put_reader, dst_opts).await;
|
||||
}
|
||||
}
|
||||
return Err(StorageError::NotImplemented);
|
||||
}
|
||||
```
|
||||
When a self-copy has a `put_object_reader` (data already fetched from tier in `execute_copy_object`), writes it back locally via `put_object`, effectively de-tiering the object.
|
||||
**How `mc cp --storage-class STANDARD` flows**:
|
||||
1. mc sends `PUT /bucket/key` with `x-amz-copy-source`, `x-amz-metadata-directive: REPLACE`, `x-amz-storage-class: STANDARD`
|
||||
2. `execute_copy_object` → `get_object_reader` fetches data from tier backend → stores in `src_info.put_object_reader`
|
||||
3. `store.copy_object(...)` → now calls `put_object` with tier data and STANDARD storage class in `dst_opts`
|
||||
4. New xl.meta written locally with STANDARD class, no tier metadata → object de-tiered
|
||||
|
||||
### Bug 3 (open): race in `expire_transitioned_object`
|
||||
Order is: delete remote tier version → delete local object.
|
||||
A concurrent GET between those two steps fetches a valid stored version_id but the tier version is already gone → `NoSuchVersion`.
|
||||
The proper fix is to delete local metadata first (making the object unreachable) before deleting the remote tier version.
|
||||
|
||||
## Debugging tier issues
|
||||
|
||||
### Inspect xl.meta directly
|
||||
```bash
|
||||
cargo build -p rustfs-filemeta --example dump_fileinfo
|
||||
./target/debug/examples/dump_fileinfo /srv/rustfs/data/disk0/{bucket}/{object}/xl.meta
|
||||
# Shows: transition_status, transition_tier, transitioned_obj, transition_ver_id
|
||||
```
|
||||
`transition_ver_id: <none>` → no versionId will be sent to tier (correct for non-versioned tier bucket).
|
||||
`transition_ver_id: <uuid>` → that UUID will be sent as `?versionId=<uuid>`.
|
||||
|
||||
### Check what versionId is being sent at runtime
|
||||
Enable debug logging:
|
||||
```bash
|
||||
RUST_LOG=rustfs_ecstore::bucket::lifecycle=debug rustfs …
|
||||
```
|
||||
Log line: `fetching transitioned object from tier` (DEBUG before request).
|
||||
Log line: `tier GET failed` (ERROR on failure, includes `tier_version_id`).
|
||||
|
||||
### Metadata key to watch
|
||||
```
|
||||
x-minio-internal-transitioned-versionID= ← empty string = will cause NoSuchVersion with old code
|
||||
x-rustfs-internal-transitioned-versionID= ← same
|
||||
```
|
||||
If both are empty string, the object was transitioned to a non-versioned tier bucket. The versionId should NOT be sent — fixed by Bug 1 above.
|
||||
|
||||
## Common patterns
|
||||
|
||||
### Writing internal metadata (binary values)
|
||||
```rust
|
||||
insert_bytes(&mut meta_sys, SUFFIX_TRANSITIONED_VERSION_ID, uuid.as_bytes().to_vec());
|
||||
// stores under both x-rustfs-internal-* and x-minio-internal-* keys
|
||||
```
|
||||
|
||||
### Reading internal metadata (binary values)
|
||||
```rust
|
||||
get_bytes(&self.meta_sys, SUFFIX_TRANSITIONED_VERSION_ID)
|
||||
.and_then(|v| Uuid::from_slice(v.as_slice()).ok())
|
||||
.filter(|u| !u.is_nil())
|
||||
// Returns None for: absent, wrong-length bytes, nil UUID
|
||||
```
|
||||
|
||||
### WarmBackend trait
|
||||
```rust
|
||||
put_with_meta(object, reader, length, meta) -> Result<String> // returns S3 version_id or ""
|
||||
put(object, reader, length) -> Result<String>
|
||||
get(object, rv, opts) -> Result<ReadCloser> // rv="" means no versionId
|
||||
remove(object, rv) -> Result<()>
|
||||
in_use() -> Result<bool>
|
||||
```
|
||||
`rv` = remote version, always pass as empty string when `transition_version_id` is None.
|
||||
- Internal object metadata is written under **both** `x-rustfs-internal-<suffix>`
|
||||
and `x-minio-internal-<suffix>` keys for MinIO interop
|
||||
(`crates/utils/src/http/metadata_compat.rs`; `get_bytes` prefers the RustFS
|
||||
key). Never write only one of the two.
|
||||
- Binary metadata values (UUIDs) must be read defensively:
|
||||
`.and_then(|v| Uuid::from_slice(&v).ok()).filter(|u| !u.is_nil())` —
|
||||
absent/empty/nil all mean "no value", not `Uuid::nil()`.
|
||||
- A remote-tier version of `None`/`""` means the tier bucket is unversioned:
|
||||
send **no** `versionId` on tier GET/DELETE.
|
||||
|
||||
@@ -8,12 +8,6 @@ Applies to all paths under `crates/`.
|
||||
- Prefer `thiserror` for library-facing error types.
|
||||
- Do not use `unwrap()`, `expect()`, or panic-driven control flow outside tests.
|
||||
|
||||
## Testing
|
||||
|
||||
- Keep unit tests close to the module they test.
|
||||
- Keep integration tests under each crate's `tests/` directory.
|
||||
- Add regression tests for bug fixes and behavior changes.
|
||||
|
||||
## Error Type Design
|
||||
|
||||
- Public API functions must return a typed error enum (preferably `thiserror`-derived), never `Result<_, String>`.
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
# Architecture Documentation
|
||||
|
||||
Durable architecture reference for RustFS: migration guardrails, runtime
|
||||
contracts, boundary rules, and support matrices.
|
||||
|
||||
Two rules keep this directory healthy:
|
||||
|
||||
1. **Durable reference only.** One-shot implementation plans, task trackers,
|
||||
and PR templates live in [`docs/superpowers/plans/`](../superpowers/plans/)
|
||||
and are archived there when their work closes.
|
||||
2. **No copies of other sources of truth.** Crate lists come from
|
||||
`Cargo.toml`, CI steps from `.github/workflows/ci.yml`, code structure from
|
||||
the code. `scripts/check_doc_paths.sh` fails the pre-commit gate when a
|
||||
doc here references a file path that no longer exists.
|
||||
|
||||
## Start here
|
||||
|
||||
- [overview.md](overview.md) — migration baseline, phase order, core principles
|
||||
|
||||
## CI-enforced core (required by `scripts/check_architecture_migration_rules.sh`)
|
||||
|
||||
- [crate-boundaries.md](crate-boundaries.md) — dependency direction, PR types, re-export contracts
|
||||
- [runtime-lifecycle.md](runtime-lifecycle.md) — startup/shutdown sequencing, readiness guarantees
|
||||
- [readiness-matrix.md](readiness-matrix.md) — request/dependency behavior, probe semantics
|
||||
- [storage-control-data-plane.md](storage-control-data-plane.md) — storage API contracts, control-plane boundaries
|
||||
- [global-state-crate-split-plan.md](global-state-crate-split-plan.md) — remaining global-state owners and split evaluation
|
||||
- [ecstore-module-split-plan.md](ecstore-module-split-plan.md) — ECStore decomposition rules and facade contracts
|
||||
|
||||
## Contracts & invariants
|
||||
|
||||
- [placement-repair-invariants.md](placement-repair-invariants.md)
|
||||
- [runtime-capability-contracts.md](runtime-capability-contracts.md)
|
||||
- [workload-admission-contracts.md](workload-admission-contracts.md)
|
||||
- [background-controller-contract.md](background-controller-contract.md)
|
||||
- [config-model-boundary-adr.md](config-model-boundary-adr.md)
|
||||
- [ecstore-layout-boundary.md](ecstore-layout-boundary.md)
|
||||
- [decommission-compatibility.md](decommission-compatibility.md)
|
||||
|
||||
## Support matrices (release-facing, keep current)
|
||||
|
||||
- [s3-compatibility-matrix.md](s3-compatibility-matrix.md)
|
||||
- [s3-tables-support-matrix.md](s3-tables-support-matrix.md)
|
||||
|
||||
## Inventories & baselines (snapshots that feed migration work)
|
||||
|
||||
- [global-state-inventory.md](global-state-inventory.md)
|
||||
- [ecstore-api-facade-inventory.md](ecstore-api-facade-inventory.md)
|
||||
- [ecstore-config-consumer-inventory.md](ecstore-config-consumer-inventory.md)
|
||||
- [obs-ecstore-dependency-inventory.md](obs-ecstore-dependency-inventory.md)
|
||||
- [background-services-inventory.md](background-services-inventory.md)
|
||||
- [admin-route-action-snapshot.md](admin-route-action-snapshot.md)
|
||||
- [startup-timeline.md](startup-timeline.md)
|
||||
- [scheduler-baseline.md](scheduler-baseline.md)
|
||||
- [profiling-numa-capability-inventory.md](profiling-numa-capability-inventory.md)
|
||||
- [kms-development-defaults-inventory.md](kms-development-defaults-inventory.md)
|
||||
- [compat-cleanup-register.md](compat-cleanup-register.md)
|
||||
|
||||
Historical plans and trackers (rebalance/decommission phases,
|
||||
migration-progress ledger) were moved to
|
||||
[`docs/superpowers/plans/`](../superpowers/plans/) in 2026-07.
|
||||
@@ -36,7 +36,7 @@ not define a new scheduler, controller framework, or shutdown contract.
|
||||
| Service | Trigger and workers | Side effects | Status and metrics | Migration notes |
|
||||
|---|---|---|---|---|
|
||||
| Capacity background refresh | `rustfs/src/capacity/capacity_integration.rs::init_capacity_management` delegates to `init_capacity_management_for_local_disks`, then `crates/object-capacity/src/capacity_manager.rs::start_background_task` spawns a scheduled refresh loop and a runtime summary loop. | Refreshes global capacity cache from local disks and logs runtime summaries. | Uses the object-capacity manager state and log summaries; no explicit shutdown status surface is exposed here. | Add read-only status before any controller migration. A future controller must not change scheduled interval defaults or singleflight refresh behavior. |
|
||||
| ECStore endpoint monitor | `crates/ecstore/src/sets.rs::new` spawns `monitor_and_connect_endpoints`. | Monitors endpoint connectivity and reconnect behavior for erasure sets. | Logs monitor start, cancellation, and exit. | This is storage-adjacent and must stay outside broad controller movement until storage shutdown semantics are explicitly covered. |
|
||||
| ECStore endpoint monitor | `crates/ecstore/src/core/sets.rs::new` spawns `monitor_and_connect_endpoints`. | Monitors endpoint connectivity and reconnect behavior for erasure sets. | Logs monitor start, cancellation, and exit. | This is storage-adjacent and must stay outside broad controller movement until storage shutdown semantics are explicitly covered. |
|
||||
| Local disk health monitor | `crates/ecstore/src/store/init.rs::init` enables disk health checks after store initialization; `crates/ecstore/src/disk/disk_store.rs::enable_health_check` spawns writable and recovery monitors. | Periodically probes disk writability, can create test objects named `health-check-*`, and updates disk runtime health state. | Disk info includes runtime health metrics and waiting counts. | Do not merge this with scanner/heal controller work; probes affect disk health semantics. |
|
||||
| Data scanner | `crates/scanner/src/scanner.rs::init_data_scanner` configures scanner defaults, applies runtime config, waits the initial scanner delay, then loops `run_data_scanner`. | Updates data usage cache, scans buckets/sets, evaluates lifecycle rules, queues replication heal, queues scanner heal, and emits scanner alerts. | Scanner runtime config/status is exposed through admin scanner status; scanner metrics record ILM, replication admission, heal admission, checkpoints, yields, and alerts. | Scanner implies heal because scanner can enqueue heal requests. Future controller status must separate scheduler state from scanner work-source accounting. |
|
||||
| Heal/AHM | `crates/heal/src/lib.rs::init_heal_manager` starts `HealManager`, initializes the shared heal channel, and spawns `HealChannelProcessor`. | Consumes heal requests from the global heal channel and drives heal work through the configured heal storage API. | Global active-task and queue-length atomics track current heal pressure. | Keep heal admission and channel semantics intact. Controller work should first expose queue/active status and shutdown state. |
|
||||
|
||||
@@ -167,5 +167,5 @@ The queued multi-pool contract is guarded by:
|
||||
- `test_first_resumable_decommission_queue_indices_allows_after_completed_prefix`
|
||||
- `admin_pool_list_item_exposes_queued_decommission_state`
|
||||
|
||||
These tests live in `crates/ecstore/src/pools.rs` and
|
||||
These tests live in `crates/ecstore/src/core/pools.rs` and
|
||||
`rustfs/src/app/admin_usecase.rs`.
|
||||
|
||||
@@ -42,7 +42,7 @@ or calls the right node.
|
||||
```mermaid
|
||||
flowchart TB
|
||||
EC["crates/ecstore/src/config"]
|
||||
Store["crates/ecstore/src/store.rs"]
|
||||
Store["crates/ecstore/src/store/mod.rs"]
|
||||
AppCtx["rustfs/src/app/context.rs"]
|
||||
Server["rustfs/src/server/{event,audit}.rs"]
|
||||
Admin["rustfs/src/admin"]
|
||||
@@ -93,7 +93,7 @@ behind narrower contracts.
|
||||
| `crates/ecstore/src/config/mod.rs` | Defines `KV`, `KVS`, `Config`, defaults, global snapshot, initialization, and tests. |
|
||||
| `crates/ecstore/src/config/com.rs` | Encodes, decodes, reads, writes, creates, and normalizes server config objects through ECStore-local persistence helpers. |
|
||||
| `crates/ecstore/src/config/{notify,audit,oidc,scanner,storageclass}.rs` | Register default `KVS` values and subsystem-specific parsing helpers. |
|
||||
| `crates/ecstore/src/store.rs` | Exposes store-level server-config accessors that delegate to the global config snapshot. |
|
||||
| `crates/ecstore/src/store/mod.rs` | Exposes store-level server-config accessors that delegate to the global config snapshot. |
|
||||
|
||||
### App Context And Server Startup Consumers
|
||||
|
||||
|
||||
@@ -50,8 +50,8 @@ hot-path behavior must not drift during this migration.
|
||||
- [`config-model-boundary-adr.md`](config-model-boundary-adr.md): target crate,
|
||||
module path, dependency rules, and verification gates for moving the pure
|
||||
server-config model.
|
||||
- [`migration-progress.md`](migration-progress.md): current task state and context
|
||||
handoff.
|
||||
- [`migration-progress.md`](../superpowers/plans/migration-progress.md): historical
|
||||
task ledger for the closed migration (execution tracking moved to issue #665).
|
||||
- [`compat-cleanup-register.md`](compat-cleanup-register.md): temporary
|
||||
compatibility code that must be removed later.
|
||||
|
||||
|
||||
@@ -23,7 +23,7 @@ new startup semantics.
|
||||
| `BOOT-003` | `rustfs/src/startup_entrypoint.rs` | Parse CLI command and dispatch non-server commands. | `info` and `tls` commands execute and return without server startup. | Command parse exits process with code 1; TLS command errors propagate. | None |
|
||||
| `BOOT-004` | `rustfs/src/startup_preflight.rs` | Initialize config snapshot and license state. | Publishes config snapshot for later readers and initializes runtime license state. | License init is non-fallible in this path. | None |
|
||||
| `BOOT-005` | `rustfs/src/startup_preflight.rs` | Initialize observability and store the global guard. | Initializes tracing/observability, stores the guard globally, and logs license/runtime telemetry status. | Fatal if observability init or guard publication fails. | None |
|
||||
| `BOOT-006` | `rustfs/src/startup_runtime.rs`, `rustfs/src/startup_runtime_hooks.rs`, `rustfs/src/startup_tls_material.rs`, and `rustfs/src/startup_profiling.rs` | Log startup logo, initialize profiling, trusted proxies, rustls provider, and outbound TLS material. | Starts optional profiling tasks, trusted proxy config, default rustls provider, outbound TLS global state, TLS generation metric, and TLS metrics when enabled. | Profiling/proxy/provider setup is non-fatal; configured TLS material load is fatal on error. | None |
|
||||
| `BOOT-006` | `rustfs/src/startup_runtime.rs`, `rustfs/src/startup_runtime_hooks.rs`, `rustfs/src/startup_tls_material.rs` | Log startup logo, initialize profiling, trusted proxies, rustls provider, and outbound TLS material. | Starts optional profiling tasks, trusted proxy config, default rustls provider, outbound TLS global state, TLS generation metric, and TLS metrics when enabled. | Profiling/proxy/provider setup is non-fatal; configured TLS material load is fatal on error. | None |
|
||||
| `RUN-001` | `rustfs/src/startup_server.rs` | Enter startup run orchestration and create `GlobalReadiness`. | Allocates the readiness tracker shared with HTTP readiness gates. | Non-fatal. | Initial readiness state is not ready |
|
||||
| `RUN-002` | `rustfs/src/startup_server.rs` | Parse and publish the configured region. | Updates ECStore global region when configured. | Fatal if the configured region is invalid. | None |
|
||||
| `RUN-003` | `rustfs/src/startup_server.rs` | Resolve server address and warn on default credentials. | Computes server port/address and emits production credential warning when defaults are used. | Address parse is fatal; default credentials warning is non-fatal. | None |
|
||||
@@ -38,13 +38,13 @@ new startup semantics.
|
||||
| `RUN-012` | `rustfs/src/startup_storage.rs` | Initialize ECStore config and global config system. | Initializes ECStore config, attempts server-config migration, then retries global config init up to 15 times. | Migration attempt is non-fatal in this path; global config init becomes fatal after retries. | Marks the `GlobalReadiness` `StorageReady` stage after global config init succeeds; later runtime readiness still rechecks storage, IAM, lock quorum, and gated peer health before `FullReady` |
|
||||
| `RUN-013` | `rustfs/src/startup_storage.rs` and `rustfs/src/startup_services.rs` | Start replication and KMS systems. | Starts background replication pool, then initializes KMS from startup services. | Replication init is non-fatal in this path; KMS init is fatal on error. | `StorageReady` stage is already marked; dynamic runtime storage readiness is still checked before `FullReady`; KMS compatibility readiness remains feature-gated health behavior |
|
||||
| `RUN-014` | `rustfs/src/startup_optional_runtime_sidecars.rs` and `rustfs/src/startup_protocols.rs` | Initialize optional protocol servers. | Starts FTP/FTPS/WebDAV/SFTP when feature-enabled and configured, collecting shutdown handles. | Feature-enabled protocol init is fatal on error; disabled protocols are non-fatal. | None |
|
||||
| `RUN-015` | `rustfs/src/startup_services.rs` and `rustfs/src/startup_service_components.rs` | Initialize buffer profiling, event notifier, audit, and deadlock detector. | Starts buffer profile system, event notifier, audit system, and optional deadlock detector. | Audit startup failure is logged and non-fatal; the others are non-fatal in this path. | None |
|
||||
| `RUN-016` | `rustfs/src/startup_service_components.rs` | List buckets and run bucket/replication/IAM metadata migrations. | Reads bucket names, migrates bucket metadata, initializes replication resync, migrates IAM config, and initializes bucket metadata system. | Bucket list and replication resync are fatal on error; metadata migration calls are non-fatal in this path. | Storage remains ready; IAM not yet ready |
|
||||
| `RUN-017` | `rustfs/src/startup_service_components.rs` and `rustfs/src/startup_iam.rs` | Bootstrap IAM inline or defer recovery. | Initializes IAM when possible; otherwise starts the deferred IAM recovery path through `startup_iam`. | Fatal only when `bootstrap_or_defer_iam_init` returns an unrecoverable error. | Inline success marks `IamReady`; deferred mode publishes `IamReady` later from the recovery task |
|
||||
| `RUN-018` | `rustfs/src/startup_service_components.rs` | Initialize Keystone and OIDC auth integrations. | Loads Keystone env config and initializes OIDC providers. | Keystone config parse is fatal; Keystone runtime init failure is non-fatal; OIDC init failure is non-fatal. | None |
|
||||
| `RUN-019` | `rustfs/src/startup_service_components.rs` | Add bucket notification config and initialize notification system. | Adds bucket notification configuration and publishes the global notification system. | Notification config add is non-fatal in this path; global notification init is fatal on error. | None |
|
||||
| `RUN-020` | `rustfs/src/startup_service_components.rs` | Create AHM cancellation token and initialize heal manager when scanner or heal is enabled. | Creates AHM cancellation token and starts heal manager for heal/scanner workflows. | Heal manager init is fatal when enabled. | None |
|
||||
| `RUN-021` | `rustfs/src/startup_service_components.rs` | Print server info, init update check, allocator reclaim, metrics, memory observability, and auto-tuner. | Starts informational/update/memory/metrics background tasks when enabled. | Non-fatal in this path. | None |
|
||||
| `RUN-015` | `rustfs/src/startup_services.rs`, `rustfs/src/startup_audit.rs`, and `rustfs/src/startup_deadlock.rs` | Initialize buffer profiling, event notifier, audit, and deadlock detector. | Starts buffer profile system, event notifier, audit system, and optional deadlock detector. | Audit startup failure is logged and non-fatal; the others are non-fatal in this path. | None |
|
||||
| `RUN-016` | `rustfs/src/startup_bucket_metadata.rs` | List buckets and run bucket/replication/IAM metadata migrations. | Reads bucket names, migrates bucket metadata, initializes replication resync, migrates IAM config, and initializes bucket metadata system. | Bucket list and replication resync are fatal on error; metadata migration calls are non-fatal in this path. | Storage remains ready; IAM not yet ready |
|
||||
| `RUN-017` | `rustfs/src/startup_iam.rs` | Bootstrap IAM inline or defer recovery. | Initializes IAM when possible; otherwise starts the deferred IAM recovery path through `startup_iam`. | Fatal only when `bootstrap_or_defer_iam_init` returns an unrecoverable error. | Inline success marks `IamReady`; deferred mode publishes `IamReady` later from the recovery task |
|
||||
| `RUN-018` | `rustfs/src/startup_auth.rs` | Initialize Keystone and OIDC auth integrations. | Loads Keystone env config and initializes OIDC providers. | Keystone config parse is fatal; Keystone runtime init failure is non-fatal; OIDC init failure is non-fatal. | None |
|
||||
| `RUN-019` | `rustfs/src/startup_notification.rs` | Add bucket notification config and initialize notification system. | Adds bucket notification configuration and publishes the global notification system. | Notification config add is non-fatal in this path; global notification init is fatal on error. | None |
|
||||
| `RUN-020` | `rustfs/src/startup_background.rs` | Create AHM cancellation token and initialize heal manager when scanner or heal is enabled. | Creates AHM cancellation token and starts heal manager for heal/scanner workflows. | Heal manager init is fatal when enabled. | None |
|
||||
| `RUN-021` | `rustfs/src/startup_observability.rs` | Print server info, init update check, allocator reclaim, metrics, memory observability, and auto-tuner. | Starts informational/update/memory/metrics background tasks when enabled. | Non-fatal in this path. | None |
|
||||
| `RUN-022` | `rustfs/src/startup_lifecycle.rs` and `rustfs/src/startup_iam.rs` | Log successful startup and publish full readiness for inline IAM. | Logs version/address, checks runtime readiness, marks `FullReady`, and sets service state to `Ready` when IAM was ready inline. | Fatal if runtime readiness is not reached within the startup wait. | Marks `FullReady` only for inline IAM here |
|
||||
| `RUN-023` | `rustfs/src/startup_lifecycle.rs` | Publish global init time and start data scanner when enabled. | Sets global init time and starts scanner after the successful-startup log. | Scanner start is non-fatal in this path. | Full readiness may already be published or may await deferred IAM recovery |
|
||||
| `RUN-024` | `rustfs/src/startup_lifecycle.rs` | Wait for shutdown signal. | Blocks the main task until a shutdown signal is received. | Non-fatal. | Runtime remains in its current readiness state |
|
||||
@@ -72,7 +72,7 @@ new startup semantics.
|
||||
| `STOP-001` | `rustfs/src/startup_shutdown.rs` | Cancel runtime token and move service state to `Stopping`. | Notifies cancellation-aware background tasks. | Non-fatal. | Service state moves to `Stopping`; readiness stages are not cleared here |
|
||||
| `STOP-002` | `rustfs/src/startup_shutdown.rs` | Stop scanner/background services and AHM services according to enable flags. | Calls ECStore background shutdown and heal/scanner shutdown helpers. | Non-fatal in this path. | No readiness-stage change |
|
||||
| `STOP-003` | `rustfs/src/startup_optional_runtime_sidecars.rs` | Plan optional runtime shutdown and log stopping state for FTP/FTPS/WebDAV/SFTP protocol servers. | Collects protocol shutdown handles. | Non-fatal in this path. | No readiness-stage change |
|
||||
| `STOP-004` | `rustfs/src/startup_shutdown.rs`, `rustfs/src/startup_runtime_hooks.rs`, and `rustfs/src/startup_profiling.rs` | Stop event notifier, audit system, and profiling tasks. | Stops notifier and profiling tasks; audit stop failures are logged. | Non-fatal in this path. | No readiness-stage change |
|
||||
| `STOP-004` | `rustfs/src/startup_shutdown.rs`, `rustfs/src/startup_runtime_hooks.rs` | Stop event notifier, audit system, and profiling tasks. | Stops notifier and profiling tasks; audit stop failures are logged. | Non-fatal in this path. | No readiness-stage change |
|
||||
| `STOP-005` | `rustfs/src/startup_shutdown.rs` and `rustfs/src/startup_optional_runtime_sidecars.rs` | Stop S3 and console HTTP servers, signal and wait for optional protocol shutdowns, then mark service state `Stopped`. | HTTP shutdown happens after notifier/audit/profiling shutdown in current order. | Join failures are logged by shutdown handles; this path does not return errors. | Service state moves to `Stopped`; readiness stages are not cleared here |
|
||||
|
||||
## Migration Rules
|
||||
|
||||
@@ -0,0 +1,94 @@
|
||||
# Tier / ILM Transition Debugging Guide
|
||||
|
||||
How to debug lifecycle tiering (hot → cold transition) issues: inspecting
|
||||
`xl.meta`, tracing the versionId sent to the remote tier, and known pitfalls.
|
||||
|
||||
> Code map (post `#3929` layout):
|
||||
>
|
||||
> | Concern | Location |
|
||||
> |---------|----------|
|
||||
> | ILM actions (`transition_object`, `expire_transitioned_object`, `get_transitioned_object_reader`, `gen_transition_objname`) | `crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs` |
|
||||
> | Erasure-set transition/restore entry points | `crates/ecstore/src/set_disk/` and `crates/ecstore/src/store/` |
|
||||
> | `WarmBackend` trait (put/get/remove/in_use) | `crates/ecstore/src/services/tier/warm_backend.rs` |
|
||||
> | Per-provider tier backends (S3, MinIO, GCS, Azure, …) | `crates/ecstore/src/services/tier/warm_backend_*.rs` |
|
||||
> | Remote-tier sweep (`delete_object_from_remote_tier`) | `crates/ecstore/src/bucket/lifecycle/tier_sweeper.rs` |
|
||||
> | `ObjectInfo` / `TransitionedObject` types | `crates/ecstore/src/object_api/types.rs` |
|
||||
> | `FileMeta` / `FileInfo` / version metadata | `crates/filemeta/src/` |
|
||||
> | Dual-key internal metadata helpers (`insert_bytes` / `get_bytes`) | `crates/utils/src/http/metadata_compat.rs` |
|
||||
|
||||
## Metadata key conventions
|
||||
|
||||
Internal metadata is stored under **both** `x-rustfs-internal-<suffix>` and
|
||||
`x-minio-internal-<suffix>` for MinIO interoperability. `get_bytes` prefers
|
||||
the RustFS key and falls back to the MinIO key.
|
||||
|
||||
| Suffix | Meaning |
|
||||
|--------|---------|
|
||||
| `transition-status` | `"complete"` when tiered |
|
||||
| `transitioned-object` | tier key path (stored **without** the tier prefix; `get_dest` adds it) |
|
||||
| `transitioned-versionID` | S3 version_id returned by tier PUT (16 raw UUID bytes, or absent) |
|
||||
| `transition-tier` | tier name |
|
||||
| `tier-free-versionID` | delete-marker version for free-version sweep |
|
||||
|
||||
Reading binary values must reject empty/malformed/nil values (regression
|
||||
covered in `crates/filemeta/src/filemeta/version.rs` tests):
|
||||
|
||||
```rust
|
||||
get_bytes(&self.meta_sys, SUFFIX_TRANSITIONED_VERSION_ID)
|
||||
.and_then(|v| Uuid::from_slice(v.as_slice()).ok())
|
||||
.filter(|u| !u.is_nil())
|
||||
// None for: absent key, wrong-length bytes, nil UUID
|
||||
```
|
||||
|
||||
`transition_version_id == None` means the tier bucket is unversioned; the
|
||||
GET/DELETE against the tier must then send **no** `versionId` parameter.
|
||||
A nil UUID (`00000000-…`) sent as `?versionId=` causes `NoSuchVersion`.
|
||||
|
||||
## Inspect xl.meta directly
|
||||
|
||||
```bash
|
||||
cargo build -p rustfs-filemeta --example dump_fileinfo
|
||||
./target/debug/examples/dump_fileinfo /srv/rustfs/data/disk0/{bucket}/{object}/xl.meta
|
||||
# Shows: transition_status, transition_tier, transitioned_obj, transition_ver_id
|
||||
```
|
||||
|
||||
- `transition_ver_id: <none>` → no versionId will be sent to the tier
|
||||
(correct for a non-versioned tier bucket).
|
||||
- `transition_ver_id: <uuid>` → that UUID will be sent as `?versionId=<uuid>`.
|
||||
|
||||
There is one `xl.meta` per erasure shard disk
|
||||
(`{disk}/{bucket}/{object}/xl.meta`); all shards of a healthy object should
|
||||
be identical. `dump_versions` (same crate) lists every version in a file.
|
||||
|
||||
## Trace the versionId at runtime
|
||||
|
||||
```bash
|
||||
RUST_LOG=rustfs_ecstore::bucket::lifecycle=debug rustfs …
|
||||
```
|
||||
|
||||
- `fetching transitioned object from tier` — DEBUG, before the tier request.
|
||||
- `tier GET failed` — ERROR, includes `tier_version_id`.
|
||||
|
||||
If both `x-rustfs-internal-transitioned-versionID` and
|
||||
`x-minio-internal-transitioned-versionID` are the **empty string**, the object
|
||||
was transitioned to a non-versioned tier bucket and no versionId must be sent.
|
||||
|
||||
## Known open issue: expire/GET race
|
||||
|
||||
`expire_transitioned_object` deletes the remote tier version **before**
|
||||
deleting local metadata. A concurrent GET between those two steps reads a
|
||||
valid stored version_id whose remote version is already gone →
|
||||
`NoSuchVersion`. The proper fix is to delete local metadata first (making the
|
||||
object unreachable), then the remote tier version.
|
||||
|
||||
## Historical fixes (for context, already merged)
|
||||
|
||||
- Nil-UUID versionId sent to tier (`NoSuchVersion`): reading code used
|
||||
`Uuid::from_slice(..).unwrap_or_default()`, converting an empty metadata
|
||||
value into `Uuid::nil()`. Fixed by the `and_then`/`filter` pattern above.
|
||||
- `warm_backend_s3sdk` ignored the remote version and range options on
|
||||
GET/DELETE.
|
||||
- `copy_object` returned 501 (`NotImplemented`) for tiered objects on
|
||||
self-copy with `--storage-class`, blocking de-tiering via
|
||||
`mc cp --storage-class STANDARD obj obj`. Fixed by writing the
|
||||
tier-fetched `put_object_reader` back through `put_object`.
|
||||
@@ -1,3 +1,8 @@
|
||||
> **Archived implementation plan/tracker** — moved from `docs/architecture/` (2026-07).
|
||||
> Kept for history; not maintained. File paths inside may reflect the pre-#3929
|
||||
> module layout (e.g. `crates/ecstore/src/rebalance.rs` is now
|
||||
> `crates/ecstore/src/store/rebalance.rs`; `set_disk.rs` is now `set_disk/`).
|
||||
|
||||
# Architecture Migration Progress
|
||||
|
||||
Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` blocked.
|
||||
+5
@@ -1,3 +1,8 @@
|
||||
> **Archived implementation plan/tracker** — moved from `docs/architecture/` (2026-07).
|
||||
> Kept for history; not maintained. File paths inside may reflect the pre-#3929
|
||||
> module layout (e.g. `crates/ecstore/src/rebalance.rs` is now
|
||||
> `crates/ecstore/src/store/rebalance.rs`; `set_disk.rs` is now `set_disk/`).
|
||||
|
||||
# Rebalance and Decommission Follow-up Review Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
+5
@@ -1,3 +1,8 @@
|
||||
> **Archived implementation plan/tracker** — moved from `docs/architecture/` (2026-07).
|
||||
> Kept for history; not maintained. File paths inside may reflect the pre-#3929
|
||||
> module layout (e.g. `crates/ecstore/src/rebalance.rs` is now
|
||||
> `crates/ecstore/src/store/rebalance.rs`; `set_disk.rs` is now `set_disk/`).
|
||||
|
||||
# Rebalance and Decommission Implementation Plan Index
|
||||
|
||||
> This index is based on `docs/architecture/rebalance-decommission-remediation-plan.md`. The remediation scope is intentionally split into smaller implementation plans because the fixes touch independent risk areas: object-version safety, distributed operation semantics, data movement internals, and operational hardening.
|
||||
+5
@@ -1,3 +1,8 @@
|
||||
> **Archived implementation plan/tracker** — moved from `docs/architecture/` (2026-07).
|
||||
> Kept for history; not maintained. File paths inside may reflect the pre-#3929
|
||||
> module layout (e.g. `crates/ecstore/src/rebalance.rs` is now
|
||||
> `crates/ecstore/src/store/rebalance.rs`; `set_disk.rs` is now `set_disk/`).
|
||||
|
||||
# Rebalance and Decommission Phase 1 Safety Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
+5
@@ -1,3 +1,8 @@
|
||||
> **Archived implementation plan/tracker** — moved from `docs/architecture/` (2026-07).
|
||||
> Kept for history; not maintained. File paths inside may reflect the pre-#3929
|
||||
> module layout (e.g. `crates/ecstore/src/rebalance.rs` is now
|
||||
> `crates/ecstore/src/store/rebalance.rs`; `set_disk.rs` is now `set_disk/`).
|
||||
|
||||
# Rebalance and Decommission Phase 2 Data Movement Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
+5
@@ -1,3 +1,8 @@
|
||||
> **Archived implementation plan/tracker** — moved from `docs/architecture/` (2026-07).
|
||||
> Kept for history; not maintained. File paths inside may reflect the pre-#3929
|
||||
> module layout (e.g. `crates/ecstore/src/rebalance.rs` is now
|
||||
> `crates/ecstore/src/store/rebalance.rs`; `set_disk.rs` is now `set_disk/`).
|
||||
|
||||
# Rebalance and Decommission Phase 3 Hardening Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
+5
@@ -1,3 +1,8 @@
|
||||
> **Archived implementation plan/tracker** — moved from `docs/architecture/` (2026-07).
|
||||
> Kept for history; not maintained. File paths inside may reflect the pre-#3929
|
||||
> module layout (e.g. `crates/ecstore/src/rebalance.rs` is now
|
||||
> `crates/ecstore/src/store/rebalance.rs`; `set_disk.rs` is now `set_disk/`).
|
||||
|
||||
# Rebalance and Decommission Post-Remediation Review
|
||||
|
||||
> **Status:** Updated for branch `cxymds/rebalance-decommission-remediation`.
|
||||
+5
@@ -1,3 +1,8 @@
|
||||
> **Archived implementation plan/tracker** — moved from `docs/architecture/` (2026-07).
|
||||
> Kept for history; not maintained. File paths inside may reflect the pre-#3929
|
||||
> module layout (e.g. `crates/ecstore/src/rebalance.rs` is now
|
||||
> `crates/ecstore/src/store/rebalance.rs`; `set_disk.rs` is now `set_disk/`).
|
||||
|
||||
# Rebalance and Decommission Remediation Plan
|
||||
|
||||
> This document turns `docs/architecture/expert-review-analysis.md` into an actionable remediation backlog. It is intentionally split into independent fix blocks so each item can be analyzed, assigned, implemented, and verified separately.
|
||||
+5
@@ -1,3 +1,8 @@
|
||||
> **Archived implementation plan/tracker** — moved from `docs/architecture/` (2026-07).
|
||||
> Kept for history; not maintained. File paths inside may reflect the pre-#3929
|
||||
> module layout (e.g. `crates/ecstore/src/rebalance.rs` is now
|
||||
> `crates/ecstore/src/store/rebalance.rs`; `set_disk.rs` is now `set_disk/`).
|
||||
|
||||
## Related Issues
|
||||
|
||||
N/A
|
||||
Executable
+69
@@ -0,0 +1,69 @@
|
||||
#!/usr/bin/env bash
|
||||
#
|
||||
# Fail when an agent-instruction or architecture document references a
|
||||
# repository file path that no longer exists. Keeps CLAUDE.md / AGENTS.md /
|
||||
# ARCHITECTURE.md / docs/architecture honest after refactors move code.
|
||||
#
|
||||
# Checked files: all tracked AGENTS.md, CLAUDE.md, ARCHITECTURE.md, and
|
||||
# docs/architecture/*.md (archived plans under docs/superpowers/plans are
|
||||
# intentionally NOT checked — they are historical).
|
||||
#
|
||||
# A reference is any token starting with crates/, rustfs/, scripts/, docs/,
|
||||
# .github/ or .config/ that ends in a known file extension. Tokens containing
|
||||
# globs or placeholders, extensionless tokens (HTTP routes, directory
|
||||
# references, org/repo shorthands), and lines containing URLs are skipped.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "$ROOT_DIR"
|
||||
|
||||
FAILURES=0
|
||||
|
||||
doc_files() {
|
||||
git ls-files 'AGENTS.md' '*/AGENTS.md' 'CLAUDE.md' 'ARCHITECTURE.md' 'docs/architecture/*.md'
|
||||
}
|
||||
|
||||
check_file() {
|
||||
local doc="$1"
|
||||
# Path-like tokens; strip surrounding backticks/brackets/punctuation later.
|
||||
# Lines with URLs are skipped wholesale: path fragments inside links are
|
||||
# external, not repo paths.
|
||||
# grep exits 1 on no matches; that must not kill the script under pipefail.
|
||||
{ grep -vE 'https?://' "$doc" || true; } \
|
||||
| { grep -oE '(crates|rustfs|scripts|docs|\.github|\.config)/[A-Za-z0-9_./-]+' || true; } \
|
||||
| sed -e 's/[.,;:)]*$//' -e 's:/$::' \
|
||||
| sort -u \
|
||||
| while IFS= read -r ref; do
|
||||
case "$ref" in
|
||||
(*'*'*|*'<'*|*'>'*|*'{'*|*'}'*|*'…'*) continue ;;
|
||||
esac
|
||||
# Only verify references with a known file extension. Extensionless
|
||||
# tokens are too often HTTP routes (rustfs/admin/v3/...), org/repo
|
||||
# shorthands (rustfs/backlog), or glob prefixes — not repo paths.
|
||||
case "$ref" in
|
||||
(*.rs|*.md|*.sh|*.toml|*.yml|*.yaml|*.mak|*.proto) ;;
|
||||
(*) continue ;;
|
||||
esac
|
||||
if [[ ! -e "$ref" ]]; then
|
||||
printf 'stale path reference: %s -> %s\n' "$doc" "$ref" >&2
|
||||
echo "FAIL" >> "$FAIL_MARKER"
|
||||
fi
|
||||
done
|
||||
}
|
||||
|
||||
FAIL_MARKER="$(mktemp)"
|
||||
trap 'rm -f "$FAIL_MARKER"' EXIT
|
||||
|
||||
while IFS= read -r doc; do
|
||||
[[ -f "$doc" ]] && check_file "$doc"
|
||||
done < <(doc_files)
|
||||
|
||||
if [[ -s "$FAIL_MARKER" ]]; then
|
||||
count="$(wc -l < "$FAIL_MARKER" | tr -d ' ')"
|
||||
printf '\n%s stale doc path reference(s) found.\n' "$count" >&2
|
||||
printf 'Fix the doc to match the current tree, or update the moved code path.\n' >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
printf 'doc path check passed\n'
|
||||
Reference in New Issue
Block a user