chore(docs): refresh agent docs, guard doc paths, archive plans (#4203)

Agent-instruction and architecture docs had drifted from the code:

- CLAUDE.md: slim to commands + pointers; fix wrong claim that
  `make pre-commit` is the full pre-PR gate (that is `make pre-pr`);
  drop stale pre-#3929 file paths and merged bug narratives
- AGENTS.md: drop dead `rust-refactor-helper` skill rule and the
  hand-maintained (already stale) scoped-AGENTS index; link
  architecture docs from Sources of Truth
- .github/AGENTS.md: replace the outdated copied CI command matrix
  with a pointer to ci.yml
- crates/AGENTS.md: merge duplicated Testing sections
- ARCHITECTURE.md: resolve the utils->config contradiction (edges are
  removed), mark volatile counts as snapshots, fix a bad path
- docs/architecture: add README router; move one-shot plans/trackers
  (rebalance-decommission phases, migration-progress ledger, PR
  template) to docs/superpowers/plans with archive headers; fix stale
  source paths in kept inventories (core/sets.rs, core/pools.rs,
  store/mod.rs, startup_* split from #3671)
- docs/operations/tier-ilm-debugging.md: extracted tier debugging
  playbook with corrected paths
- scripts/check_doc_paths.sh: new guard failing pre-commit/pre-pr when
  instruction/architecture docs reference nonexistent file paths
- .claude/skills: add tier-debug and arch-checks repo skills;
  .gitignore now keeps .claude/skills and docs/operations committable

Verification: ./scripts/check_doc_paths.sh,
./scripts/check_architecture_migration_rules.sh (both pass)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Zhengchao An
2026-07-02 23:30:13 +08:00
committed by GitHub
parent 29899f4731
commit d1db9a10cd
27 changed files with 454 additions and 241 deletions
+51
View File
@@ -0,0 +1,51 @@
---
name: arch-checks
description: Resolve failures from the repository's architecture guard scripts — check_layer_dependencies.sh, check_architecture_migration_rules.sh, check_unsafe_code_allowances.sh, check_logging_guardrails.sh, check_doc_paths.sh. Use when make pre-commit / pre-pr or CI fails on one of these checks.
---
# Architecture Guard Checks
All five run in `make pre-commit` / `make pre-pr` and in CI. Fix the cause;
never weaken a check to get green.
## `check_layer_dependencies.sh` — layer DAG in `rustfs/src`
Enforces `interface (admin, storage/ecfs, storage/s3_api) → app → infra`; no
upward imports. Known legacy violations live in
`scripts/layer-dependency-baseline.txt`.
- **New violation**: restructure your change so the dependency points
downward (move the shared type/function to the lower layer).
- **You legitimately removed a baseline entry**: run
`./scripts/check_layer_dependencies.sh --update-baseline` and commit the
shrunken baseline. Never add new entries to the baseline to make a new
violation pass.
## `check_architecture_migration_rules.sh` — required doc sections
Asserts that the core docs under `docs/architecture/` (overview,
crate-boundaries, runtime-lifecycle, readiness-matrix,
storage-control-data-plane, global-state-crate-split-plan,
ecstore-module-split-plan, …) still contain specific headings and exact
source lines. If it fails after a doc edit, you reworded or removed a
guarded line — restore the wording or update the script deliberately in the
same PR, with rationale.
## `check_unsafe_code_allowances.sh`
Every `#[allow(unsafe_code)]` needs a `SAFETY:` comment within a few lines.
Write the actual safety argument; don't add a placeholder.
## `check_logging_guardrails.sh`
A fixed list of security-sensitive files (auth, IAM, KMS, admin handlers…)
is scanned for logging violations. If you created a new sensitive file,
consider adding it to the script's `checked_files` list.
## `check_doc_paths.sh`
Instruction/architecture docs (`AGENTS.md`, `CLAUDE.md`, `ARCHITECTURE.md`,
`docs/architecture/*.md`) must not reference repo file paths that no longer
exist. If your refactor moved code, update the docs that point at it — the
error message lists `doc -> stale-path` pairs. Historical plans under
`docs/superpowers/plans/` are exempt.
+34
View File
@@ -0,0 +1,34 @@
---
name: tier-debug
description: Debug ILM tiering / lifecycle transition issues — NoSuchVersion on tier GET, restore failures, xl.meta inspection, remote-tier versionId tracing. Use when investigating tiered/transitioned objects, warm backends, or transition metadata.
---
# Tier / ILM Debugging
Full playbook: [docs/operations/tier-ilm-debugging.md](../../../docs/operations/tier-ilm-debugging.md)
— read it before changing tier code.
Quick moves:
```bash
# Inspect transition metadata on disk (one xl.meta per erasure shard disk)
cargo run -p rustfs-filemeta --example dump_fileinfo -- "/path/to/{bucket}/{object}/xl.meta"
# Trace what versionId is sent to the remote tier
RUST_LOG=rustfs_ecstore::bucket::lifecycle=debug ./target/debug/rustfs …
```
Interpretation:
- `transition_ver_id: <none>` → correct for an unversioned tier bucket; no
`versionId` must be sent on tier GET/DELETE.
- `transition_ver_id: 00000000-…` (nil) → corrupt legacy write-back; readers
must filter it out, never send it.
- Empty-string `transitioned-versionID` metadata under both
`x-rustfs-internal-*` and `x-minio-internal-*` keys → object went to an
unversioned tier bucket.
Code entry points: `crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs`
(ILM actions), `crates/ecstore/src/services/tier/` (warm backends),
`crates/filemeta/src/filemeta/version.rs` (metadata read/write + regression
tests).
+1
View File
@@ -0,0 +1 @@
../.agents/skills
+8 -3
View File
@@ -8,14 +8,19 @@ setup-hooks: ## Set up git hooks
chmod +x .git/hooks/pre-commit chmod +x .git/hooks/pre-commit
@echo "✅ Git hooks setup complete!" @echo "✅ Git hooks setup complete!"
.PHONY: doc-paths-check
doc-paths-check: ## Check that instruction/architecture docs reference existing file paths
@echo "📄 Checking doc path references..."
./scripts/check_doc_paths.sh
.PHONY: pre-commit .PHONY: pre-commit
pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check quick-check ## Run fast pre-commit checks without clippy/full tests pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check doc-paths-check quick-check ## Run fast pre-commit checks without clippy/full tests
@echo "✅ All pre-commit checks passed!" @echo "✅ All pre-commit checks passed!"
.PHONY: pre-pr .PHONY: pre-pr
pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check clippy-check test ## Run full pre-PR checks with clippy and tests pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check doc-paths-check clippy-check test ## Run full pre-PR checks with clippy and tests
@echo "✅ All pre-PR checks passed!" @echo "✅ All pre-PR checks passed!"
.PHONY: dev-check .PHONY: dev-check
dev-check: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check quick-check ## Run fast local development checks dev-check: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check doc-paths-check quick-check ## Run fast local development checks
@echo "✅ Fast development checks passed!" @echo "✅ Fast development checks passed!"
+4 -19
View File
@@ -19,22 +19,7 @@ Applies to `.github/` and repository pull-request operations.
## CI Alignment ## CI Alignment
When changing CI-sensitive behavior, keep local validation aligned with `.github/workflows/ci.yml`. When changing CI-sensitive behavior, keep local validation aligned with
`.github/workflows/ci.yml`. Read the workflow file directly for the current
Current `test-and-lint` gate includes: gate steps — do not rely on (or add) a copied command list here; copies go
stale. `make pre-pr` is the local equivalent of the main gate.
- `cargo nextest run --all --exclude e2e_test`
- `cargo test --all --doc`
- `cargo test -p rustfs get_object_chunk_fast_path`
- `cargo test -p rustfs materialize_chunk_stream_before_commit`
- `touch rustfs/build.rs`
- `cargo build -p rustfs --bins --jobs 2`
- `cargo test -p e2e_test archive_multipart_roundtrip_preserves_bytes`
- `cargo test -p e2e_test presigned_get_and_reverse_proxy_preserve_multipart_bytes_with_fast_path`
- `cargo fmt --all --check`
- `cargo clippy --all-targets -- -D warnings`
- `cargo nextest run -p rustfs -p rustfs-ecstore --features rio-v2`
- `cargo test -p rustfs --doc --features rio-v2`
- `cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings`
- `./scripts/check_layer_dependencies.sh`
- `./scripts/check_architecture_migration_rules.sh`
+10 -1
View File
@@ -4,7 +4,9 @@
.vscode .vscode
.cursor .cursor
.direnv/ .direnv/
.claude/ # Ignore local Claude state but keep shared repo skills committable
.claude/*
!.claude/skills
.tmp/ .tmp/
/test /test
/logs /logs
@@ -51,6 +53,10 @@ __pycache__/
docs/* docs/*
!docs/architecture/ !docs/architecture/
!docs/architecture/** !docs/architecture/**
!docs/operations/
!docs/operations/**
!docs/superpowers/
!docs/superpowers/**
docs/heal-scanner-logging-governance.md docs/heal-scanner-logging-governance.md
docs/benchmark/rustfs-target-bench/ docs/benchmark/rustfs-target-bench/
docs/benchmark/*.md docs/benchmark/*.md
@@ -74,3 +80,6 @@ crates/*/docs
fuzz/target fuzz/target
outputs outputs
worktrees/* worktrees/*
# Local AI-agent review artifacts (omo evidence dumps)
.omo/
+17 -15
View File
@@ -27,10 +27,6 @@ If repo-level instructions conflict, follow the nearest file and keep behavior a
- Keep source code, comments, commit messages, and PR title/body in English. - Keep source code, comments, commit messages, and PR title/body in English.
- Be concise. Avoid sycophantic openers, closing fluff, and verbose status reporting. - Be concise. Avoid sycophantic openers, closing fluff, and verbose status reporting.
## Skill Usage
- Do not use the `rust-refactor-helper` skill in any scenario.
## Change Style for Existing Logic ## Change Style for Existing Logic
- Prefer direct, local code over extracting one-off helpers. - Prefer direct, local code over extracting one-off helpers.
@@ -57,6 +53,13 @@ If repo-level instructions conflict, follow the nearest file and keep behavior a
- Local quality commands: `Makefile` and `.config/make/` - Local quality commands: `Makefile` and `.config/make/`
- CI quality gates: `.github/workflows/ci.yml` - CI quality gates: `.github/workflows/ci.yml`
- PR template: `.github/pull_request_template.md` - PR template: `.github/pull_request_template.md`
- High-level architecture and crate map: `ARCHITECTURE.md`
- Migration guardrails, readiness contracts, support matrices:
`docs/architecture/README.md` (routes by audience)
- Historical implementation plans and trackers: `docs/superpowers/plans/`
- Shared agent skills (all tools): `.agents/skills/` — Claude Code reads them
through the `.claude/skills` symlink; add new skills to `.agents/skills/`
only, never as separate copies per tool
Avoid duplicating long crate lists or command matrices in instruction files. Avoid duplicating long crate lists or command matrices in instruction files.
Reference the source files above instead. Reference the source files above instead.
@@ -136,14 +139,13 @@ cargo run -p rustfs-filemeta --example dump_fileinfo -- "/path/to/file/xl.meta"
## Scoped Guidance in This Repository ## Scoped Guidance in This Repository
- `.github/AGENTS.md` Many crates and modules carry their own `AGENTS.md` with path-specific rules
- `crates/AGENTS.md` (security boundaries, lock ordering, domain invariants). Before editing a
- `crates/config/AGENTS.md` path, check for the nearest one:
- `crates/ecstore/AGENTS.md`
- `crates/e2e_test/AGENTS.md` ```bash
- `crates/iam/AGENTS.md` git ls-files '*AGENTS.md'
- `crates/kms/AGENTS.md` ```
- `crates/policy/AGENTS.md`
- `crates/targets/AGENTS.md` The nearest file wins. Do not maintain a hand-written index of these files
- `rustfs/src/admin/AGENTS.md` here — it goes stale.
- `rustfs/src/storage/AGENTS.md`
+11 -6
View File
@@ -1,6 +1,6 @@
# ARCHITECTURE.md # ARCHITECTURE.md
> Last updated: 2026-04-13 · Revision: 1 (draft) > Last updated: 2026-07-02 · Revision: 2
> >
> This document describes the high-level architecture of RustFS. > This document describes the high-level architecture of RustFS.
> If you want to familiarize yourself with the code base, you are in the right place! > If you want to familiarize yourself with the code base, you are in the right place!
@@ -52,7 +52,7 @@ rustfs/ # Workspace root (virtual manifest)
│ ├── auth.rs # S3 request authentication │ ├── auth.rs # S3 request authentication
│ ├── config/ # CLI args, config parsing, workload profiles │ ├── config/ # CLI args, config parsing, workload profiles
│ └── ... │ └── ...
├── crates/ # 39 library crates ├── crates/ # library crates (authoritative list: Cargo.toml [workspace].members)
│ ├── ecstore/ # Erasure-coded storage engine (⚠️ 87K lines) │ ├── ecstore/ # Erasure-coded storage engine (⚠️ 87K lines)
│ ├── rio/ # Reader I/O pipeline (encrypt, compress, hash) │ ├── rio/ # Reader I/O pipeline (encrypt, compress, hash)
│ ├── io-core/ # Zero-copy I/O, scheduling, buffer pool │ ├── io-core/ # Zero-copy I/O, scheduling, buffer pool
@@ -83,6 +83,10 @@ A request flows **downward** through the layers. No layer should reach upward
### Crate Reference ### Crate Reference
> Depth levels, line counts, and crate counts in this section are a
> point-in-time snapshot and drift with refactors. Treat them as orders of
> magnitude; `Cargo.toml` and `cargo tree` are the source of truth.
Crates are organized in a dependency DAG with 9 depth levels (0 = leaf, 8 = top): Crates are organized in a dependency DAG with 9 depth levels (0 = leaf, 8 = top):
``` ```
@@ -93,14 +97,14 @@ Depth 0 — LEAF (no internal deps):
Depth 1: Depth 1:
io-core (→ io-metrics) io-core (→ io-metrics)
policy (→ config, credentials, crypto) policy (→ config, credentials, crypto)
utils (→ config) ⚠️ inverted: utils should be leaf utils (historical → config edge removed; now effectively leaf)
Depth 2: Depth 2:
concurrency, filemeta, keystone, kms, lock, obs, concurrency, filemeta, keystone, kms, lock, obs,
signer, targets, trusted-proxies signer, targets, trusted-proxies
Depth 3: Depth 3:
common (→ filemeta, madmin) ⚠️ inverted: common should be leaf common (historical → filemeta/madmin edges removed; now effectively leaf)
Depth 4: Depth 4:
object-capacity, protos, rio object-capacity, protos, rio
@@ -209,7 +213,8 @@ Depth 8 — TOP:
2. **Leaf crates have zero internal dependencies.** `config`, `credentials`, `crypto`, 2. **Leaf crates have zero internal dependencies.** `config`, `credentials`, `crypto`,
`io-metrics`, `madmin`, `s3-common` should depend only on external crates. `io-metrics`, `madmin`, `s3-common` should depend only on external crates.
- ⚠️ VIOLATED: `utils` depends on `config`, `common` depends on `filemeta` and `madmin`. - ✅ RESOLVED: the historical `utils → config` and `commonfilemeta`/`madmin`
edges were removed; do not reintroduce them (see Known Structural Issues).
3. **Each type has exactly one definition.** Types shared across crates must be defined 3. **Each type has exactly one definition.** Types shared across crates must be defined
in one crate and re-exported or imported by others. in one crate and re-exported or imported by others.
@@ -249,7 +254,7 @@ Depth 8 — TOP:
not regain upward dependencies. not regain upward dependencies.
- **Three-layer BackpressureConfig/DeadlockConfig duplication** across io-core, - **Three-layer BackpressureConfig/DeadlockConfig duplication** across io-core,
concurrency, and rustfs/storage. Storage policies now expose and consume concurrency, and `rustfs/src/storage`. Storage policies now expose and consume
explicit projections into the concurrency/io-core policy shapes, and workload explicit projections into the concurrency/io-core policy shapes, and workload
admission snapshots are composed through provider registries; later work admission snapshots are composed through provider registries; later work
should use those bridges before deleting compatibility wrappers. should use those bridges before deleting compatibility wrappers.
+35 -176
View File
@@ -1,188 +1,47 @@
# RustFS — CLAUDE.md # RustFS — CLAUDE.md
S3-compatible object store in Rust, derived from MinIO. Erasure-coded, multi-pool, supports ILM tiering/lifecycle. S3-compatible object store in Rust, derived from MinIO. Erasure-coded,
multi-pool, supports ILM tiering/lifecycle.
Agent rules (change style, verification, PR conventions, scoped per-crate
guidance) live in [AGENTS.md](AGENTS.md) — follow it. This file only adds
what Claude Code needs on top: commands and pointers.
## Commands ## Commands
```bash ```bash
cargo build --release --bin rustfs # production binary cargo build --release --bin rustfs # production binary
cargo build # dev build cargo check -p <crate> # fast type-check one crate
cargo check -p <crate> # fast type-check one crate cargo test -p <crate> # test one crate
cargo test -p <crate> # test one crate cargo fmt --all # format (required before PR)
cargo fmt --all # format (required before PR) make pre-commit # fast gate: fmt + arch checks + quick-check (NO clippy/tests)
make pre-commit # full pre-PR gate (fmt + clippy + test) make pre-pr # full pre-PR gate: fmt + arch checks + clippy + tests
make build-docker BUILD_OS=ubuntu22.04 # Docker cross-build make build-docker BUILD_OS=ubuntu22.04
``` ```
> **Docker build note**: `buildx build` without `--load` keeps the image in the buildx cache only — `docker run` will use a stale local image. The Makefile already includes `--load`; if you suspect a stale binary, add `--no-cache` to the `buildx build` invocation inside `.config/make/build-docker.mak`. > **Docker build note**: `buildx build` without `--load` keeps the image in
> the buildx cache only — `docker run` will use a stale local image. The
> Makefile already includes `--load`; if you suspect a stale binary, add
> `--no-cache` to the `buildx build` invocation inside
> `.config/make/build-docker.mak`.
> Agent/PR rules: see `.github/copilot-instructions.md`. ## Where to look (do not duplicate here)
> Crate membership: `Cargo.toml` `[workspace].members`.
> CI gates: `.github/workflows/ci.yml`.
## Workspace layout - Crate membership: `Cargo.toml` `[workspace].members`
- Architecture, layering, crate map: [ARCHITECTURE.md](ARCHITECTURE.md)
- Migration guardrails & readiness contracts: [docs/architecture/](docs/architecture/README.md)
- CI gates: `.github/workflows/ci.yml` (source of truth; never copy its steps into docs)
- Tier/ILM transition debugging (xl.meta inspection, versionId tracing):
[docs/operations/tier-ilm-debugging.md](docs/operations/tier-ilm-debugging.md)
``` ## Domain conventions worth knowing up front
rustfs/src/main.rs # binary entry point
crates/ecstore/src/
set_disk.rs # ErasureSet: transition_object, restore_transitioned_object
store.rs / store_api/ # ECStore trait + ObjectInfo / TransitionedObject types
bucket/lifecycle/
bucket_lifecycle_ops.rs # ILM actions: transition_object, expire_transitioned_object,
# get_transitioned_object_reader, gen_transition_objname
tier_sweeper.rs # background sweep: delete_object_from_remote_tier
tier/
warm_backend.rs # WarmBackend trait (put/get/remove/in_use)
warm_backend_s3.rs # HTTP-client based (TransitionClient) — used for S3/MinIO
warm_backend_s3sdk.rs # aws-sdk-s3 based — alternative S3 backend
warm_backend_minio.rs / _rustfs.rs / … # per-provider wrappers (all delegate to _s3 or _s3sdk)
tier.rs # TierConfigMgr, new_warm_backend dispatch
client/transition_api.rs # TransitionClient HTTP plumbing; UploadInfo, to_object_info
client/api_put_object_streaming.rs # put_object_do → UploadInfo (version_id from x-amz-version-id)
crates/filemeta/src/
filemeta.rs # FileMeta (xl.meta top-level), is_skip_meta_key
filemeta/version.rs # FileMetaVersion, MetaObject, MetaDeleteMarker
# → to_fileinfo() reads transition_version_id
# → set_transition() writes raw UUID bytes
# → From<FileInfo> for MetaObject writes all meta
fileinfo.rs # FileInfo struct (transition_version_id: Option<Uuid>)
examples/
dump_fileinfo.rs # CLI: parse xl.meta, print transition fields + metadata
dump_versions.rs # CLI: list all versions in xl.meta
crates/utils/src/http/metadata_compat.rs # SUFFIX_* constants, insert_bytes/get_bytes (dual RustFS+MinIO keys)
```
## Metadata key conventions - Internal object metadata is written under **both** `x-rustfs-internal-<suffix>`
and `x-minio-internal-<suffix>` keys for MinIO interop
Internal metadata is stored under **both** `x-rustfs-internal-<suffix>` and `x-minio-internal-<suffix>` for MinIO interoperability. `get_bytes` prefers the RustFS key with MinIO fallback. (`crates/utils/src/http/metadata_compat.rs`; `get_bytes` prefers the RustFS
key). Never write only one of the two.
Key suffixes (from `metadata_compat.rs`): - Binary metadata values (UUIDs) must be read defensively:
| Suffix | Meaning | `.and_then(|v| Uuid::from_slice(&v).ok()).filter(|u| !u.is_nil())`
|--------|---------| absent/empty/nil all mean "no value", not `Uuid::nil()`.
| `transition-status` | `"complete"` when tiered | - A remote-tier version of `None`/`""` means the tier bucket is unversioned:
| `transitioned-object` | tier key path (without prefix) | send **no** `versionId` on tier GET/DELETE.
| `transitioned-versionID` | S3 version_id returned by tier PUT (16 raw UUID bytes, or absent) |
| `transition-tier` | tier name |
| `tier-free-versionID` | delete-marker version for free-version sweep |
## Tier / ILM transition architecture
### Transition flow (hot → cold)
1. `transition_object` (lifecycle_ops) → `ECStore::transition_object``set_disk.rs`
2. `gen_transition_objname(bucket)``{sha256_hash[0..16]}/{uuid[0..2]}/{uuid[2..4]}/{uuid}` (unique per object version)
3. `tgt_client.put_with_meta(dest_obj, …)` → returns `rv: String` (remote S3 version_id, or `""`)
4. `fi.transition_version_id = if rv.is_empty() { None } else { Some(Uuid::parse_str(&rv)?) }`
5. `fi.transitioned_objname = dest_obj` (without tier prefix)
6. Written to xl.meta via `MetaObject::from(FileInfo)``insert_bytes(SUFFIX_TRANSITIONED_VERSION_ID, uuid.as_bytes())` (16 raw bytes)
### Tier GET flow (restore/read)
`get_transitioned_object_reader` (lifecycle_ops):
- reads `oi.transitioned_object.name` (= `fi.transitioned_objname`)
- reads `oi.transitioned_object.version_id` (= `fi.transition_version_id.to_string()` or `""`)
- calls `warm_backend.get(name, version_id, opts)`
- `warm_backend_s3.rs::get`: adds `?versionId=…` only when `rv != ""`
### Tier prefix handling
`WarmBackendS3::get_dest(object)` prepends `self.prefix` to the object name.
`transitioned_objname` is stored **without** the prefix — `get_dest` adds it on every call.
### xl.meta on disk
Path: `{disk}/{bucket}/{object}/xl.meta` — one per erasure shard disk.
All shards should be identical for a healthy object.
## Known bugs & fixes
### Bug 1: `NoSuchVersion` on tier GET — nil UUID sent as versionId
**Root cause**: `transitioned-versionID` metadata key exists with empty string value (0 bytes). Old reading code:
```rust
// OLD — unwrap_or_default() converts 0-byte or wrong-length slice to Uuid::nil()
get_bytes().map(|v| Uuid::from_slice(v.as_slice()).unwrap_or_default())
// → Some(Uuid::nil()) → sends ?versionId=00000000-… → NoSuchVersion
```
**Fix** (version.rs, `MetaObject::to_fileinfo` + `MetaDeleteMarker::to_fileinfo`):
```rust
get_bytes()
.and_then(|v| Uuid::from_slice(v.as_slice()).ok()) // None for wrong-length bytes
.filter(|u| !u.is_nil()) // None for nil UUID (old write-back)
```
**Regression tests** (`crates/filemeta/src/filemeta/version.rs` `mod tests`): 6 tests cover absent key, empty bytes, nil UUID, and valid UUID round-trip for both `MetaObject` and `MetaDeleteMarker` paths.
### Bug 2: `warm_backend_s3sdk.rs` ignored rv and range opts
**Fix**: added `req.version_id(rv)` and `req.range(…)` to GET; `req.version_id(rv)` to DELETE.
### Bug 4: `set_disk::copy_object` returns 501 for tiered objects (storage class restore)
**Root cause**: `set_disk::copy_object` immediately returns `StorageError::NotImplemented` when `src_info.metadata_only = false`. For tiered objects, `metadata_only` is never set to `true` (guarded by `transitioned_object.tier.is_empty()`). So `mc cp --storage-class STANDARD obj obj` on a tiered object always returns 501.
**Fix** (`crates/ecstore/src/set_disk.rs`, `copy_object`):
```rust
if !src_info.metadata_only {
if path_join_buf(&[src_bucket, src_object]) == path_join_buf(&[dst_bucket, dst_object]) {
if let Some(mut put_reader) = src_info.put_object_reader.take() {
return self.put_object(dst_bucket, dst_object, &mut put_reader, dst_opts).await;
}
}
return Err(StorageError::NotImplemented);
}
```
When a self-copy has a `put_object_reader` (data already fetched from tier in `execute_copy_object`), writes it back locally via `put_object`, effectively de-tiering the object.
**How `mc cp --storage-class STANDARD` flows**:
1. mc sends `PUT /bucket/key` with `x-amz-copy-source`, `x-amz-metadata-directive: REPLACE`, `x-amz-storage-class: STANDARD`
2. `execute_copy_object``get_object_reader` fetches data from tier backend → stores in `src_info.put_object_reader`
3. `store.copy_object(...)` → now calls `put_object` with tier data and STANDARD storage class in `dst_opts`
4. New xl.meta written locally with STANDARD class, no tier metadata → object de-tiered
### Bug 3 (open): race in `expire_transitioned_object`
Order is: delete remote tier version → delete local object.
A concurrent GET between those two steps fetches a valid stored version_id but the tier version is already gone → `NoSuchVersion`.
The proper fix is to delete local metadata first (making the object unreachable) before deleting the remote tier version.
## Debugging tier issues
### Inspect xl.meta directly
```bash
cargo build -p rustfs-filemeta --example dump_fileinfo
./target/debug/examples/dump_fileinfo /srv/rustfs/data/disk0/{bucket}/{object}/xl.meta
# Shows: transition_status, transition_tier, transitioned_obj, transition_ver_id
```
`transition_ver_id: <none>` → no versionId will be sent to tier (correct for non-versioned tier bucket).
`transition_ver_id: <uuid>` → that UUID will be sent as `?versionId=<uuid>`.
### Check what versionId is being sent at runtime
Enable debug logging:
```bash
RUST_LOG=rustfs_ecstore::bucket::lifecycle=debug rustfs …
```
Log line: `fetching transitioned object from tier` (DEBUG before request).
Log line: `tier GET failed` (ERROR on failure, includes `tier_version_id`).
### Metadata key to watch
```
x-minio-internal-transitioned-versionID= ← empty string = will cause NoSuchVersion with old code
x-rustfs-internal-transitioned-versionID= ← same
```
If both are empty string, the object was transitioned to a non-versioned tier bucket. The versionId should NOT be sent — fixed by Bug 1 above.
## Common patterns
### Writing internal metadata (binary values)
```rust
insert_bytes(&mut meta_sys, SUFFIX_TRANSITIONED_VERSION_ID, uuid.as_bytes().to_vec());
// stores under both x-rustfs-internal-* and x-minio-internal-* keys
```
### Reading internal metadata (binary values)
```rust
get_bytes(&self.meta_sys, SUFFIX_TRANSITIONED_VERSION_ID)
.and_then(|v| Uuid::from_slice(v.as_slice()).ok())
.filter(|u| !u.is_nil())
// Returns None for: absent, wrong-length bytes, nil UUID
```
### WarmBackend trait
```rust
put_with_meta(object, reader, length, meta) -> Result<String> // returns S3 version_id or ""
put(object, reader, length) -> Result<String>
get(object, rv, opts) -> Result<ReadCloser> // rv="" means no versionId
remove(object, rv) -> Result<()>
in_use() -> Result<bool>
```
`rv` = remote version, always pass as empty string when `transition_version_id` is None.
-6
View File
@@ -8,12 +8,6 @@ Applies to all paths under `crates/`.
- Prefer `thiserror` for library-facing error types. - Prefer `thiserror` for library-facing error types.
- Do not use `unwrap()`, `expect()`, or panic-driven control flow outside tests. - Do not use `unwrap()`, `expect()`, or panic-driven control flow outside tests.
## Testing
- Keep unit tests close to the module they test.
- Keep integration tests under each crate's `tests/` directory.
- Add regression tests for bug fixes and behavior changes.
## Error Type Design ## Error Type Design
- Public API functions must return a typed error enum (preferably `thiserror`-derived), never `Result<_, String>`. - Public API functions must return a typed error enum (preferably `thiserror`-derived), never `Result<_, String>`.
+60
View File
@@ -0,0 +1,60 @@
# Architecture Documentation
Durable architecture reference for RustFS: migration guardrails, runtime
contracts, boundary rules, and support matrices.
Two rules keep this directory healthy:
1. **Durable reference only.** One-shot implementation plans, task trackers,
and PR templates live in [`docs/superpowers/plans/`](../superpowers/plans/)
and are archived there when their work closes.
2. **No copies of other sources of truth.** Crate lists come from
`Cargo.toml`, CI steps from `.github/workflows/ci.yml`, code structure from
the code. `scripts/check_doc_paths.sh` fails the pre-commit gate when a
doc here references a file path that no longer exists.
## Start here
- [overview.md](overview.md) — migration baseline, phase order, core principles
## CI-enforced core (required by `scripts/check_architecture_migration_rules.sh`)
- [crate-boundaries.md](crate-boundaries.md) — dependency direction, PR types, re-export contracts
- [runtime-lifecycle.md](runtime-lifecycle.md) — startup/shutdown sequencing, readiness guarantees
- [readiness-matrix.md](readiness-matrix.md) — request/dependency behavior, probe semantics
- [storage-control-data-plane.md](storage-control-data-plane.md) — storage API contracts, control-plane boundaries
- [global-state-crate-split-plan.md](global-state-crate-split-plan.md) — remaining global-state owners and split evaluation
- [ecstore-module-split-plan.md](ecstore-module-split-plan.md) — ECStore decomposition rules and facade contracts
## Contracts & invariants
- [placement-repair-invariants.md](placement-repair-invariants.md)
- [runtime-capability-contracts.md](runtime-capability-contracts.md)
- [workload-admission-contracts.md](workload-admission-contracts.md)
- [background-controller-contract.md](background-controller-contract.md)
- [config-model-boundary-adr.md](config-model-boundary-adr.md)
- [ecstore-layout-boundary.md](ecstore-layout-boundary.md)
- [decommission-compatibility.md](decommission-compatibility.md)
## Support matrices (release-facing, keep current)
- [s3-compatibility-matrix.md](s3-compatibility-matrix.md)
- [s3-tables-support-matrix.md](s3-tables-support-matrix.md)
## Inventories & baselines (snapshots that feed migration work)
- [global-state-inventory.md](global-state-inventory.md)
- [ecstore-api-facade-inventory.md](ecstore-api-facade-inventory.md)
- [ecstore-config-consumer-inventory.md](ecstore-config-consumer-inventory.md)
- [obs-ecstore-dependency-inventory.md](obs-ecstore-dependency-inventory.md)
- [background-services-inventory.md](background-services-inventory.md)
- [admin-route-action-snapshot.md](admin-route-action-snapshot.md)
- [startup-timeline.md](startup-timeline.md)
- [scheduler-baseline.md](scheduler-baseline.md)
- [profiling-numa-capability-inventory.md](profiling-numa-capability-inventory.md)
- [kms-development-defaults-inventory.md](kms-development-defaults-inventory.md)
- [compat-cleanup-register.md](compat-cleanup-register.md)
Historical plans and trackers (rebalance/decommission phases,
migration-progress ledger) were moved to
[`docs/superpowers/plans/`](../superpowers/plans/) in 2026-07.
@@ -36,7 +36,7 @@ not define a new scheduler, controller framework, or shutdown contract.
| Service | Trigger and workers | Side effects | Status and metrics | Migration notes | | Service | Trigger and workers | Side effects | Status and metrics | Migration notes |
|---|---|---|---|---| |---|---|---|---|---|
| Capacity background refresh | `rustfs/src/capacity/capacity_integration.rs::init_capacity_management` delegates to `init_capacity_management_for_local_disks`, then `crates/object-capacity/src/capacity_manager.rs::start_background_task` spawns a scheduled refresh loop and a runtime summary loop. | Refreshes global capacity cache from local disks and logs runtime summaries. | Uses the object-capacity manager state and log summaries; no explicit shutdown status surface is exposed here. | Add read-only status before any controller migration. A future controller must not change scheduled interval defaults or singleflight refresh behavior. | | Capacity background refresh | `rustfs/src/capacity/capacity_integration.rs::init_capacity_management` delegates to `init_capacity_management_for_local_disks`, then `crates/object-capacity/src/capacity_manager.rs::start_background_task` spawns a scheduled refresh loop and a runtime summary loop. | Refreshes global capacity cache from local disks and logs runtime summaries. | Uses the object-capacity manager state and log summaries; no explicit shutdown status surface is exposed here. | Add read-only status before any controller migration. A future controller must not change scheduled interval defaults or singleflight refresh behavior. |
| ECStore endpoint monitor | `crates/ecstore/src/sets.rs::new` spawns `monitor_and_connect_endpoints`. | Monitors endpoint connectivity and reconnect behavior for erasure sets. | Logs monitor start, cancellation, and exit. | This is storage-adjacent and must stay outside broad controller movement until storage shutdown semantics are explicitly covered. | | ECStore endpoint monitor | `crates/ecstore/src/core/sets.rs::new` spawns `monitor_and_connect_endpoints`. | Monitors endpoint connectivity and reconnect behavior for erasure sets. | Logs monitor start, cancellation, and exit. | This is storage-adjacent and must stay outside broad controller movement until storage shutdown semantics are explicitly covered. |
| Local disk health monitor | `crates/ecstore/src/store/init.rs::init` enables disk health checks after store initialization; `crates/ecstore/src/disk/disk_store.rs::enable_health_check` spawns writable and recovery monitors. | Periodically probes disk writability, can create test objects named `health-check-*`, and updates disk runtime health state. | Disk info includes runtime health metrics and waiting counts. | Do not merge this with scanner/heal controller work; probes affect disk health semantics. | | Local disk health monitor | `crates/ecstore/src/store/init.rs::init` enables disk health checks after store initialization; `crates/ecstore/src/disk/disk_store.rs::enable_health_check` spawns writable and recovery monitors. | Periodically probes disk writability, can create test objects named `health-check-*`, and updates disk runtime health state. | Disk info includes runtime health metrics and waiting counts. | Do not merge this with scanner/heal controller work; probes affect disk health semantics. |
| Data scanner | `crates/scanner/src/scanner.rs::init_data_scanner` configures scanner defaults, applies runtime config, waits the initial scanner delay, then loops `run_data_scanner`. | Updates data usage cache, scans buckets/sets, evaluates lifecycle rules, queues replication heal, queues scanner heal, and emits scanner alerts. | Scanner runtime config/status is exposed through admin scanner status; scanner metrics record ILM, replication admission, heal admission, checkpoints, yields, and alerts. | Scanner implies heal because scanner can enqueue heal requests. Future controller status must separate scheduler state from scanner work-source accounting. | | Data scanner | `crates/scanner/src/scanner.rs::init_data_scanner` configures scanner defaults, applies runtime config, waits the initial scanner delay, then loops `run_data_scanner`. | Updates data usage cache, scans buckets/sets, evaluates lifecycle rules, queues replication heal, queues scanner heal, and emits scanner alerts. | Scanner runtime config/status is exposed through admin scanner status; scanner metrics record ILM, replication admission, heal admission, checkpoints, yields, and alerts. | Scanner implies heal because scanner can enqueue heal requests. Future controller status must separate scheduler state from scanner work-source accounting. |
| Heal/AHM | `crates/heal/src/lib.rs::init_heal_manager` starts `HealManager`, initializes the shared heal channel, and spawns `HealChannelProcessor`. | Consumes heal requests from the global heal channel and drives heal work through the configured heal storage API. | Global active-task and queue-length atomics track current heal pressure. | Keep heal admission and channel semantics intact. Controller work should first expose queue/active status and shutdown state. | | Heal/AHM | `crates/heal/src/lib.rs::init_heal_manager` starts `HealManager`, initializes the shared heal channel, and spawns `HealChannelProcessor`. | Consumes heal requests from the global heal channel and drives heal work through the configured heal storage API. | Global active-task and queue-length atomics track current heal pressure. | Keep heal admission and channel semantics intact. Controller work should first expose queue/active status and shutdown state. |
@@ -167,5 +167,5 @@ The queued multi-pool contract is guarded by:
- `test_first_resumable_decommission_queue_indices_allows_after_completed_prefix` - `test_first_resumable_decommission_queue_indices_allows_after_completed_prefix`
- `admin_pool_list_item_exposes_queued_decommission_state` - `admin_pool_list_item_exposes_queued_decommission_state`
These tests live in `crates/ecstore/src/pools.rs` and These tests live in `crates/ecstore/src/core/pools.rs` and
`rustfs/src/app/admin_usecase.rs`. `rustfs/src/app/admin_usecase.rs`.
@@ -42,7 +42,7 @@ or calls the right node.
```mermaid ```mermaid
flowchart TB flowchart TB
EC["crates/ecstore/src/config"] EC["crates/ecstore/src/config"]
Store["crates/ecstore/src/store.rs"] Store["crates/ecstore/src/store/mod.rs"]
AppCtx["rustfs/src/app/context.rs"] AppCtx["rustfs/src/app/context.rs"]
Server["rustfs/src/server/{event,audit}.rs"] Server["rustfs/src/server/{event,audit}.rs"]
Admin["rustfs/src/admin"] Admin["rustfs/src/admin"]
@@ -93,7 +93,7 @@ behind narrower contracts.
| `crates/ecstore/src/config/mod.rs` | Defines `KV`, `KVS`, `Config`, defaults, global snapshot, initialization, and tests. | | `crates/ecstore/src/config/mod.rs` | Defines `KV`, `KVS`, `Config`, defaults, global snapshot, initialization, and tests. |
| `crates/ecstore/src/config/com.rs` | Encodes, decodes, reads, writes, creates, and normalizes server config objects through ECStore-local persistence helpers. | | `crates/ecstore/src/config/com.rs` | Encodes, decodes, reads, writes, creates, and normalizes server config objects through ECStore-local persistence helpers. |
| `crates/ecstore/src/config/{notify,audit,oidc,scanner,storageclass}.rs` | Register default `KVS` values and subsystem-specific parsing helpers. | | `crates/ecstore/src/config/{notify,audit,oidc,scanner,storageclass}.rs` | Register default `KVS` values and subsystem-specific parsing helpers. |
| `crates/ecstore/src/store.rs` | Exposes store-level server-config accessors that delegate to the global config snapshot. | | `crates/ecstore/src/store/mod.rs` | Exposes store-level server-config accessors that delegate to the global config snapshot. |
### App Context And Server Startup Consumers ### App Context And Server Startup Consumers
+2 -2
View File
@@ -50,8 +50,8 @@ hot-path behavior must not drift during this migration.
- [`config-model-boundary-adr.md`](config-model-boundary-adr.md): target crate, - [`config-model-boundary-adr.md`](config-model-boundary-adr.md): target crate,
module path, dependency rules, and verification gates for moving the pure module path, dependency rules, and verification gates for moving the pure
server-config model. server-config model.
- [`migration-progress.md`](migration-progress.md): current task state and context - [`migration-progress.md`](../superpowers/plans/migration-progress.md): historical
handoff. task ledger for the closed migration (execution tracking moved to issue #665).
- [`compat-cleanup-register.md`](compat-cleanup-register.md): temporary - [`compat-cleanup-register.md`](compat-cleanup-register.md): temporary
compatibility code that must be removed later. compatibility code that must be removed later.
+9 -9
View File
@@ -23,7 +23,7 @@ new startup semantics.
| `BOOT-003` | `rustfs/src/startup_entrypoint.rs` | Parse CLI command and dispatch non-server commands. | `info` and `tls` commands execute and return without server startup. | Command parse exits process with code 1; TLS command errors propagate. | None | | `BOOT-003` | `rustfs/src/startup_entrypoint.rs` | Parse CLI command and dispatch non-server commands. | `info` and `tls` commands execute and return without server startup. | Command parse exits process with code 1; TLS command errors propagate. | None |
| `BOOT-004` | `rustfs/src/startup_preflight.rs` | Initialize config snapshot and license state. | Publishes config snapshot for later readers and initializes runtime license state. | License init is non-fallible in this path. | None | | `BOOT-004` | `rustfs/src/startup_preflight.rs` | Initialize config snapshot and license state. | Publishes config snapshot for later readers and initializes runtime license state. | License init is non-fallible in this path. | None |
| `BOOT-005` | `rustfs/src/startup_preflight.rs` | Initialize observability and store the global guard. | Initializes tracing/observability, stores the guard globally, and logs license/runtime telemetry status. | Fatal if observability init or guard publication fails. | None | | `BOOT-005` | `rustfs/src/startup_preflight.rs` | Initialize observability and store the global guard. | Initializes tracing/observability, stores the guard globally, and logs license/runtime telemetry status. | Fatal if observability init or guard publication fails. | None |
| `BOOT-006` | `rustfs/src/startup_runtime.rs`, `rustfs/src/startup_runtime_hooks.rs`, `rustfs/src/startup_tls_material.rs`, and `rustfs/src/startup_profiling.rs` | Log startup logo, initialize profiling, trusted proxies, rustls provider, and outbound TLS material. | Starts optional profiling tasks, trusted proxy config, default rustls provider, outbound TLS global state, TLS generation metric, and TLS metrics when enabled. | Profiling/proxy/provider setup is non-fatal; configured TLS material load is fatal on error. | None | | `BOOT-006` | `rustfs/src/startup_runtime.rs`, `rustfs/src/startup_runtime_hooks.rs`, `rustfs/src/startup_tls_material.rs` | Log startup logo, initialize profiling, trusted proxies, rustls provider, and outbound TLS material. | Starts optional profiling tasks, trusted proxy config, default rustls provider, outbound TLS global state, TLS generation metric, and TLS metrics when enabled. | Profiling/proxy/provider setup is non-fatal; configured TLS material load is fatal on error. | None |
| `RUN-001` | `rustfs/src/startup_server.rs` | Enter startup run orchestration and create `GlobalReadiness`. | Allocates the readiness tracker shared with HTTP readiness gates. | Non-fatal. | Initial readiness state is not ready | | `RUN-001` | `rustfs/src/startup_server.rs` | Enter startup run orchestration and create `GlobalReadiness`. | Allocates the readiness tracker shared with HTTP readiness gates. | Non-fatal. | Initial readiness state is not ready |
| `RUN-002` | `rustfs/src/startup_server.rs` | Parse and publish the configured region. | Updates ECStore global region when configured. | Fatal if the configured region is invalid. | None | | `RUN-002` | `rustfs/src/startup_server.rs` | Parse and publish the configured region. | Updates ECStore global region when configured. | Fatal if the configured region is invalid. | None |
| `RUN-003` | `rustfs/src/startup_server.rs` | Resolve server address and warn on default credentials. | Computes server port/address and emits production credential warning when defaults are used. | Address parse is fatal; default credentials warning is non-fatal. | None | | `RUN-003` | `rustfs/src/startup_server.rs` | Resolve server address and warn on default credentials. | Computes server port/address and emits production credential warning when defaults are used. | Address parse is fatal; default credentials warning is non-fatal. | None |
@@ -38,13 +38,13 @@ new startup semantics.
| `RUN-012` | `rustfs/src/startup_storage.rs` | Initialize ECStore config and global config system. | Initializes ECStore config, attempts server-config migration, then retries global config init up to 15 times. | Migration attempt is non-fatal in this path; global config init becomes fatal after retries. | Marks the `GlobalReadiness` `StorageReady` stage after global config init succeeds; later runtime readiness still rechecks storage, IAM, lock quorum, and gated peer health before `FullReady` | | `RUN-012` | `rustfs/src/startup_storage.rs` | Initialize ECStore config and global config system. | Initializes ECStore config, attempts server-config migration, then retries global config init up to 15 times. | Migration attempt is non-fatal in this path; global config init becomes fatal after retries. | Marks the `GlobalReadiness` `StorageReady` stage after global config init succeeds; later runtime readiness still rechecks storage, IAM, lock quorum, and gated peer health before `FullReady` |
| `RUN-013` | `rustfs/src/startup_storage.rs` and `rustfs/src/startup_services.rs` | Start replication and KMS systems. | Starts background replication pool, then initializes KMS from startup services. | Replication init is non-fatal in this path; KMS init is fatal on error. | `StorageReady` stage is already marked; dynamic runtime storage readiness is still checked before `FullReady`; KMS compatibility readiness remains feature-gated health behavior | | `RUN-013` | `rustfs/src/startup_storage.rs` and `rustfs/src/startup_services.rs` | Start replication and KMS systems. | Starts background replication pool, then initializes KMS from startup services. | Replication init is non-fatal in this path; KMS init is fatal on error. | `StorageReady` stage is already marked; dynamic runtime storage readiness is still checked before `FullReady`; KMS compatibility readiness remains feature-gated health behavior |
| `RUN-014` | `rustfs/src/startup_optional_runtime_sidecars.rs` and `rustfs/src/startup_protocols.rs` | Initialize optional protocol servers. | Starts FTP/FTPS/WebDAV/SFTP when feature-enabled and configured, collecting shutdown handles. | Feature-enabled protocol init is fatal on error; disabled protocols are non-fatal. | None | | `RUN-014` | `rustfs/src/startup_optional_runtime_sidecars.rs` and `rustfs/src/startup_protocols.rs` | Initialize optional protocol servers. | Starts FTP/FTPS/WebDAV/SFTP when feature-enabled and configured, collecting shutdown handles. | Feature-enabled protocol init is fatal on error; disabled protocols are non-fatal. | None |
| `RUN-015` | `rustfs/src/startup_services.rs` and `rustfs/src/startup_service_components.rs` | Initialize buffer profiling, event notifier, audit, and deadlock detector. | Starts buffer profile system, event notifier, audit system, and optional deadlock detector. | Audit startup failure is logged and non-fatal; the others are non-fatal in this path. | None | | `RUN-015` | `rustfs/src/startup_services.rs`, `rustfs/src/startup_audit.rs`, and `rustfs/src/startup_deadlock.rs` | Initialize buffer profiling, event notifier, audit, and deadlock detector. | Starts buffer profile system, event notifier, audit system, and optional deadlock detector. | Audit startup failure is logged and non-fatal; the others are non-fatal in this path. | None |
| `RUN-016` | `rustfs/src/startup_service_components.rs` | List buckets and run bucket/replication/IAM metadata migrations. | Reads bucket names, migrates bucket metadata, initializes replication resync, migrates IAM config, and initializes bucket metadata system. | Bucket list and replication resync are fatal on error; metadata migration calls are non-fatal in this path. | Storage remains ready; IAM not yet ready | | `RUN-016` | `rustfs/src/startup_bucket_metadata.rs` | List buckets and run bucket/replication/IAM metadata migrations. | Reads bucket names, migrates bucket metadata, initializes replication resync, migrates IAM config, and initializes bucket metadata system. | Bucket list and replication resync are fatal on error; metadata migration calls are non-fatal in this path. | Storage remains ready; IAM not yet ready |
| `RUN-017` | `rustfs/src/startup_service_components.rs` and `rustfs/src/startup_iam.rs` | Bootstrap IAM inline or defer recovery. | Initializes IAM when possible; otherwise starts the deferred IAM recovery path through `startup_iam`. | Fatal only when `bootstrap_or_defer_iam_init` returns an unrecoverable error. | Inline success marks `IamReady`; deferred mode publishes `IamReady` later from the recovery task | | `RUN-017` | `rustfs/src/startup_iam.rs` | Bootstrap IAM inline or defer recovery. | Initializes IAM when possible; otherwise starts the deferred IAM recovery path through `startup_iam`. | Fatal only when `bootstrap_or_defer_iam_init` returns an unrecoverable error. | Inline success marks `IamReady`; deferred mode publishes `IamReady` later from the recovery task |
| `RUN-018` | `rustfs/src/startup_service_components.rs` | Initialize Keystone and OIDC auth integrations. | Loads Keystone env config and initializes OIDC providers. | Keystone config parse is fatal; Keystone runtime init failure is non-fatal; OIDC init failure is non-fatal. | None | | `RUN-018` | `rustfs/src/startup_auth.rs` | Initialize Keystone and OIDC auth integrations. | Loads Keystone env config and initializes OIDC providers. | Keystone config parse is fatal; Keystone runtime init failure is non-fatal; OIDC init failure is non-fatal. | None |
| `RUN-019` | `rustfs/src/startup_service_components.rs` | Add bucket notification config and initialize notification system. | Adds bucket notification configuration and publishes the global notification system. | Notification config add is non-fatal in this path; global notification init is fatal on error. | None | | `RUN-019` | `rustfs/src/startup_notification.rs` | Add bucket notification config and initialize notification system. | Adds bucket notification configuration and publishes the global notification system. | Notification config add is non-fatal in this path; global notification init is fatal on error. | None |
| `RUN-020` | `rustfs/src/startup_service_components.rs` | Create AHM cancellation token and initialize heal manager when scanner or heal is enabled. | Creates AHM cancellation token and starts heal manager for heal/scanner workflows. | Heal manager init is fatal when enabled. | None | | `RUN-020` | `rustfs/src/startup_background.rs` | Create AHM cancellation token and initialize heal manager when scanner or heal is enabled. | Creates AHM cancellation token and starts heal manager for heal/scanner workflows. | Heal manager init is fatal when enabled. | None |
| `RUN-021` | `rustfs/src/startup_service_components.rs` | Print server info, init update check, allocator reclaim, metrics, memory observability, and auto-tuner. | Starts informational/update/memory/metrics background tasks when enabled. | Non-fatal in this path. | None | | `RUN-021` | `rustfs/src/startup_observability.rs` | Print server info, init update check, allocator reclaim, metrics, memory observability, and auto-tuner. | Starts informational/update/memory/metrics background tasks when enabled. | Non-fatal in this path. | None |
| `RUN-022` | `rustfs/src/startup_lifecycle.rs` and `rustfs/src/startup_iam.rs` | Log successful startup and publish full readiness for inline IAM. | Logs version/address, checks runtime readiness, marks `FullReady`, and sets service state to `Ready` when IAM was ready inline. | Fatal if runtime readiness is not reached within the startup wait. | Marks `FullReady` only for inline IAM here | | `RUN-022` | `rustfs/src/startup_lifecycle.rs` and `rustfs/src/startup_iam.rs` | Log successful startup and publish full readiness for inline IAM. | Logs version/address, checks runtime readiness, marks `FullReady`, and sets service state to `Ready` when IAM was ready inline. | Fatal if runtime readiness is not reached within the startup wait. | Marks `FullReady` only for inline IAM here |
| `RUN-023` | `rustfs/src/startup_lifecycle.rs` | Publish global init time and start data scanner when enabled. | Sets global init time and starts scanner after the successful-startup log. | Scanner start is non-fatal in this path. | Full readiness may already be published or may await deferred IAM recovery | | `RUN-023` | `rustfs/src/startup_lifecycle.rs` | Publish global init time and start data scanner when enabled. | Sets global init time and starts scanner after the successful-startup log. | Scanner start is non-fatal in this path. | Full readiness may already be published or may await deferred IAM recovery |
| `RUN-024` | `rustfs/src/startup_lifecycle.rs` | Wait for shutdown signal. | Blocks the main task until a shutdown signal is received. | Non-fatal. | Runtime remains in its current readiness state | | `RUN-024` | `rustfs/src/startup_lifecycle.rs` | Wait for shutdown signal. | Blocks the main task until a shutdown signal is received. | Non-fatal. | Runtime remains in its current readiness state |
@@ -72,7 +72,7 @@ new startup semantics.
| `STOP-001` | `rustfs/src/startup_shutdown.rs` | Cancel runtime token and move service state to `Stopping`. | Notifies cancellation-aware background tasks. | Non-fatal. | Service state moves to `Stopping`; readiness stages are not cleared here | | `STOP-001` | `rustfs/src/startup_shutdown.rs` | Cancel runtime token and move service state to `Stopping`. | Notifies cancellation-aware background tasks. | Non-fatal. | Service state moves to `Stopping`; readiness stages are not cleared here |
| `STOP-002` | `rustfs/src/startup_shutdown.rs` | Stop scanner/background services and AHM services according to enable flags. | Calls ECStore background shutdown and heal/scanner shutdown helpers. | Non-fatal in this path. | No readiness-stage change | | `STOP-002` | `rustfs/src/startup_shutdown.rs` | Stop scanner/background services and AHM services according to enable flags. | Calls ECStore background shutdown and heal/scanner shutdown helpers. | Non-fatal in this path. | No readiness-stage change |
| `STOP-003` | `rustfs/src/startup_optional_runtime_sidecars.rs` | Plan optional runtime shutdown and log stopping state for FTP/FTPS/WebDAV/SFTP protocol servers. | Collects protocol shutdown handles. | Non-fatal in this path. | No readiness-stage change | | `STOP-003` | `rustfs/src/startup_optional_runtime_sidecars.rs` | Plan optional runtime shutdown and log stopping state for FTP/FTPS/WebDAV/SFTP protocol servers. | Collects protocol shutdown handles. | Non-fatal in this path. | No readiness-stage change |
| `STOP-004` | `rustfs/src/startup_shutdown.rs`, `rustfs/src/startup_runtime_hooks.rs`, and `rustfs/src/startup_profiling.rs` | Stop event notifier, audit system, and profiling tasks. | Stops notifier and profiling tasks; audit stop failures are logged. | Non-fatal in this path. | No readiness-stage change | | `STOP-004` | `rustfs/src/startup_shutdown.rs`, `rustfs/src/startup_runtime_hooks.rs` | Stop event notifier, audit system, and profiling tasks. | Stops notifier and profiling tasks; audit stop failures are logged. | Non-fatal in this path. | No readiness-stage change |
| `STOP-005` | `rustfs/src/startup_shutdown.rs` and `rustfs/src/startup_optional_runtime_sidecars.rs` | Stop S3 and console HTTP servers, signal and wait for optional protocol shutdowns, then mark service state `Stopped`. | HTTP shutdown happens after notifier/audit/profiling shutdown in current order. | Join failures are logged by shutdown handles; this path does not return errors. | Service state moves to `Stopped`; readiness stages are not cleared here | | `STOP-005` | `rustfs/src/startup_shutdown.rs` and `rustfs/src/startup_optional_runtime_sidecars.rs` | Stop S3 and console HTTP servers, signal and wait for optional protocol shutdowns, then mark service state `Stopped`. | HTTP shutdown happens after notifier/audit/profiling shutdown in current order. | Join failures are logged by shutdown handles; this path does not return errors. | Service state moves to `Stopped`; readiness stages are not cleared here |
## Migration Rules ## Migration Rules
+94
View File
@@ -0,0 +1,94 @@
# Tier / ILM Transition Debugging Guide
How to debug lifecycle tiering (hot → cold transition) issues: inspecting
`xl.meta`, tracing the versionId sent to the remote tier, and known pitfalls.
> Code map (post `#3929` layout):
>
> | Concern | Location |
> |---------|----------|
> | ILM actions (`transition_object`, `expire_transitioned_object`, `get_transitioned_object_reader`, `gen_transition_objname`) | `crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs` |
> | Erasure-set transition/restore entry points | `crates/ecstore/src/set_disk/` and `crates/ecstore/src/store/` |
> | `WarmBackend` trait (put/get/remove/in_use) | `crates/ecstore/src/services/tier/warm_backend.rs` |
> | Per-provider tier backends (S3, MinIO, GCS, Azure, …) | `crates/ecstore/src/services/tier/warm_backend_*.rs` |
> | Remote-tier sweep (`delete_object_from_remote_tier`) | `crates/ecstore/src/bucket/lifecycle/tier_sweeper.rs` |
> | `ObjectInfo` / `TransitionedObject` types | `crates/ecstore/src/object_api/types.rs` |
> | `FileMeta` / `FileInfo` / version metadata | `crates/filemeta/src/` |
> | Dual-key internal metadata helpers (`insert_bytes` / `get_bytes`) | `crates/utils/src/http/metadata_compat.rs` |
## Metadata key conventions
Internal metadata is stored under **both** `x-rustfs-internal-<suffix>` and
`x-minio-internal-<suffix>` for MinIO interoperability. `get_bytes` prefers
the RustFS key and falls back to the MinIO key.
| Suffix | Meaning |
|--------|---------|
| `transition-status` | `"complete"` when tiered |
| `transitioned-object` | tier key path (stored **without** the tier prefix; `get_dest` adds it) |
| `transitioned-versionID` | S3 version_id returned by tier PUT (16 raw UUID bytes, or absent) |
| `transition-tier` | tier name |
| `tier-free-versionID` | delete-marker version for free-version sweep |
Reading binary values must reject empty/malformed/nil values (regression
covered in `crates/filemeta/src/filemeta/version.rs` tests):
```rust
get_bytes(&self.meta_sys, SUFFIX_TRANSITIONED_VERSION_ID)
.and_then(|v| Uuid::from_slice(v.as_slice()).ok())
.filter(|u| !u.is_nil())
// None for: absent key, wrong-length bytes, nil UUID
```
`transition_version_id == None` means the tier bucket is unversioned; the
GET/DELETE against the tier must then send **no** `versionId` parameter.
A nil UUID (`00000000-…`) sent as `?versionId=` causes `NoSuchVersion`.
## Inspect xl.meta directly
```bash
cargo build -p rustfs-filemeta --example dump_fileinfo
./target/debug/examples/dump_fileinfo /srv/rustfs/data/disk0/{bucket}/{object}/xl.meta
# Shows: transition_status, transition_tier, transitioned_obj, transition_ver_id
```
- `transition_ver_id: <none>` → no versionId will be sent to the tier
(correct for a non-versioned tier bucket).
- `transition_ver_id: <uuid>` → that UUID will be sent as `?versionId=<uuid>`.
There is one `xl.meta` per erasure shard disk
(`{disk}/{bucket}/{object}/xl.meta`); all shards of a healthy object should
be identical. `dump_versions` (same crate) lists every version in a file.
## Trace the versionId at runtime
```bash
RUST_LOG=rustfs_ecstore::bucket::lifecycle=debug rustfs …
```
- `fetching transitioned object from tier` — DEBUG, before the tier request.
- `tier GET failed` — ERROR, includes `tier_version_id`.
If both `x-rustfs-internal-transitioned-versionID` and
`x-minio-internal-transitioned-versionID` are the **empty string**, the object
was transitioned to a non-versioned tier bucket and no versionId must be sent.
## Known open issue: expire/GET race
`expire_transitioned_object` deletes the remote tier version **before**
deleting local metadata. A concurrent GET between those two steps reads a
valid stored version_id whose remote version is already gone →
`NoSuchVersion`. The proper fix is to delete local metadata first (making the
object unreachable), then the remote tier version.
## Historical fixes (for context, already merged)
- Nil-UUID versionId sent to tier (`NoSuchVersion`): reading code used
`Uuid::from_slice(..).unwrap_or_default()`, converting an empty metadata
value into `Uuid::nil()`. Fixed by the `and_then`/`filter` pattern above.
- `warm_backend_s3sdk` ignored the remote version and range options on
GET/DELETE.
- `copy_object` returned 501 (`NotImplemented`) for tiered objects on
self-copy with `--storage-class`, blocking de-tiering via
`mc cp --storage-class STANDARD obj obj`. Fixed by writing the
tier-fetched `put_object_reader` back through `put_object`.
@@ -1,3 +1,8 @@
> **Archived implementation plan/tracker** — moved from `docs/architecture/` (2026-07).
> Kept for history; not maintained. File paths inside may reflect the pre-#3929
> module layout (e.g. `crates/ecstore/src/rebalance.rs` is now
> `crates/ecstore/src/store/rebalance.rs`; `set_disk.rs` is now `set_disk/`).
# Architecture Migration Progress # Architecture Migration Progress
Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` blocked. Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` blocked.
@@ -1,3 +1,8 @@
> **Archived implementation plan/tracker** — moved from `docs/architecture/` (2026-07).
> Kept for history; not maintained. File paths inside may reflect the pre-#3929
> module layout (e.g. `crates/ecstore/src/rebalance.rs` is now
> `crates/ecstore/src/store/rebalance.rs`; `set_disk.rs` is now `set_disk/`).
# Rebalance and Decommission Follow-up Review Implementation Plan # Rebalance and Decommission Follow-up Review Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
@@ -1,3 +1,8 @@
> **Archived implementation plan/tracker** — moved from `docs/architecture/` (2026-07).
> Kept for history; not maintained. File paths inside may reflect the pre-#3929
> module layout (e.g. `crates/ecstore/src/rebalance.rs` is now
> `crates/ecstore/src/store/rebalance.rs`; `set_disk.rs` is now `set_disk/`).
# Rebalance and Decommission Implementation Plan Index # Rebalance and Decommission Implementation Plan Index
> This index is based on `docs/architecture/rebalance-decommission-remediation-plan.md`. The remediation scope is intentionally split into smaller implementation plans because the fixes touch independent risk areas: object-version safety, distributed operation semantics, data movement internals, and operational hardening. > This index is based on `docs/architecture/rebalance-decommission-remediation-plan.md`. The remediation scope is intentionally split into smaller implementation plans because the fixes touch independent risk areas: object-version safety, distributed operation semantics, data movement internals, and operational hardening.
@@ -1,3 +1,8 @@
> **Archived implementation plan/tracker** — moved from `docs/architecture/` (2026-07).
> Kept for history; not maintained. File paths inside may reflect the pre-#3929
> module layout (e.g. `crates/ecstore/src/rebalance.rs` is now
> `crates/ecstore/src/store/rebalance.rs`; `set_disk.rs` is now `set_disk/`).
# Rebalance and Decommission Phase 1 Safety Plan # Rebalance and Decommission Phase 1 Safety Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
@@ -1,3 +1,8 @@
> **Archived implementation plan/tracker** — moved from `docs/architecture/` (2026-07).
> Kept for history; not maintained. File paths inside may reflect the pre-#3929
> module layout (e.g. `crates/ecstore/src/rebalance.rs` is now
> `crates/ecstore/src/store/rebalance.rs`; `set_disk.rs` is now `set_disk/`).
# Rebalance and Decommission Phase 2 Data Movement Plan # Rebalance and Decommission Phase 2 Data Movement Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
@@ -1,3 +1,8 @@
> **Archived implementation plan/tracker** — moved from `docs/architecture/` (2026-07).
> Kept for history; not maintained. File paths inside may reflect the pre-#3929
> module layout (e.g. `crates/ecstore/src/rebalance.rs` is now
> `crates/ecstore/src/store/rebalance.rs`; `set_disk.rs` is now `set_disk/`).
# Rebalance and Decommission Phase 3 Hardening Plan # Rebalance and Decommission Phase 3 Hardening Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
@@ -1,3 +1,8 @@
> **Archived implementation plan/tracker** — moved from `docs/architecture/` (2026-07).
> Kept for history; not maintained. File paths inside may reflect the pre-#3929
> module layout (e.g. `crates/ecstore/src/rebalance.rs` is now
> `crates/ecstore/src/store/rebalance.rs`; `set_disk.rs` is now `set_disk/`).
# Rebalance and Decommission Post-Remediation Review # Rebalance and Decommission Post-Remediation Review
> **Status:** Updated for branch `cxymds/rebalance-decommission-remediation`. > **Status:** Updated for branch `cxymds/rebalance-decommission-remediation`.
@@ -1,3 +1,8 @@
> **Archived implementation plan/tracker** — moved from `docs/architecture/` (2026-07).
> Kept for history; not maintained. File paths inside may reflect the pre-#3929
> module layout (e.g. `crates/ecstore/src/rebalance.rs` is now
> `crates/ecstore/src/store/rebalance.rs`; `set_disk.rs` is now `set_disk/`).
# Rebalance and Decommission Remediation Plan # Rebalance and Decommission Remediation Plan
> This document turns `docs/architecture/expert-review-analysis.md` into an actionable remediation backlog. It is intentionally split into independent fix blocks so each item can be analyzed, assigned, implemented, and verified separately. > This document turns `docs/architecture/expert-review-analysis.md` into an actionable remediation backlog. It is intentionally split into independent fix blocks so each item can be analyzed, assigned, implemented, and verified separately.
@@ -1,3 +1,8 @@
> **Archived implementation plan/tracker** — moved from `docs/architecture/` (2026-07).
> Kept for history; not maintained. File paths inside may reflect the pre-#3929
> module layout (e.g. `crates/ecstore/src/rebalance.rs` is now
> `crates/ecstore/src/store/rebalance.rs`; `set_disk.rs` is now `set_disk/`).
## Related Issues ## Related Issues
N/A N/A
+69
View File
@@ -0,0 +1,69 @@
#!/usr/bin/env bash
#
# Fail when an agent-instruction or architecture document references a
# repository file path that no longer exists. Keeps CLAUDE.md / AGENTS.md /
# ARCHITECTURE.md / docs/architecture honest after refactors move code.
#
# Checked files: all tracked AGENTS.md, CLAUDE.md, ARCHITECTURE.md, and
# docs/architecture/*.md (archived plans under docs/superpowers/plans are
# intentionally NOT checked — they are historical).
#
# A reference is any token starting with crates/, rustfs/, scripts/, docs/,
# .github/ or .config/ that ends in a known file extension. Tokens containing
# globs or placeholders, extensionless tokens (HTTP routes, directory
# references, org/repo shorthands), and lines containing URLs are skipped.
set -euo pipefail
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$ROOT_DIR"
FAILURES=0
doc_files() {
git ls-files 'AGENTS.md' '*/AGENTS.md' 'CLAUDE.md' 'ARCHITECTURE.md' 'docs/architecture/*.md'
}
check_file() {
local doc="$1"
# Path-like tokens; strip surrounding backticks/brackets/punctuation later.
# Lines with URLs are skipped wholesale: path fragments inside links are
# external, not repo paths.
# grep exits 1 on no matches; that must not kill the script under pipefail.
{ grep -vE 'https?://' "$doc" || true; } \
| { grep -oE '(crates|rustfs|scripts|docs|\.github|\.config)/[A-Za-z0-9_./-]+' || true; } \
| sed -e 's/[.,;:)]*$//' -e 's:/$::' \
| sort -u \
| while IFS= read -r ref; do
case "$ref" in
(*'*'*|*'<'*|*'>'*|*'{'*|*'}'*|*'…'*) continue ;;
esac
# Only verify references with a known file extension. Extensionless
# tokens are too often HTTP routes (rustfs/admin/v3/...), org/repo
# shorthands (rustfs/backlog), or glob prefixes — not repo paths.
case "$ref" in
(*.rs|*.md|*.sh|*.toml|*.yml|*.yaml|*.mak|*.proto) ;;
(*) continue ;;
esac
if [[ ! -e "$ref" ]]; then
printf 'stale path reference: %s -> %s\n' "$doc" "$ref" >&2
echo "FAIL" >> "$FAIL_MARKER"
fi
done
}
FAIL_MARKER="$(mktemp)"
trap 'rm -f "$FAIL_MARKER"' EXIT
while IFS= read -r doc; do
[[ -f "$doc" ]] && check_file "$doc"
done < <(doc_files)
if [[ -s "$FAIL_MARKER" ]]; then
count="$(wc -l < "$FAIL_MARKER" | tr -d ' ')"
printf '\n%s stale doc path reference(s) found.\n' "$count" >&2
printf 'Fix the doc to match the current tree, or update the moved code path.\n' >&2
exit 1
fi
printf 'doc path check passed\n'