From d1db9a10cd8ff655746f2f536d595084036d7aed Mon Sep 17 00:00:00 2001 From: Zhengchao An Date: Thu, 2 Jul 2026 23:30:13 +0800 Subject: [PATCH] chore(docs): refresh agent docs, guard doc paths, archive plans (#4203) Agent-instruction and architecture docs had drifted from the code: - CLAUDE.md: slim to commands + pointers; fix wrong claim that `make pre-commit` is the full pre-PR gate (that is `make pre-pr`); drop stale pre-#3929 file paths and merged bug narratives - AGENTS.md: drop dead `rust-refactor-helper` skill rule and the hand-maintained (already stale) scoped-AGENTS index; link architecture docs from Sources of Truth - .github/AGENTS.md: replace the outdated copied CI command matrix with a pointer to ci.yml - crates/AGENTS.md: merge duplicated Testing sections - ARCHITECTURE.md: resolve the utils->config contradiction (edges are removed), mark volatile counts as snapshots, fix a bad path - docs/architecture: add README router; move one-shot plans/trackers (rebalance-decommission phases, migration-progress ledger, PR template) to docs/superpowers/plans with archive headers; fix stale source paths in kept inventories (core/sets.rs, core/pools.rs, store/mod.rs, startup_* split from #3671) - docs/operations/tier-ilm-debugging.md: extracted tier debugging playbook with corrected paths - scripts/check_doc_paths.sh: new guard failing pre-commit/pre-pr when instruction/architecture docs reference nonexistent file paths - .claude/skills: add tier-debug and arch-checks repo skills; .gitignore now keeps .claude/skills and docs/operations committable Verification: ./scripts/check_doc_paths.sh, ./scripts/check_architecture_migration_rules.sh (both pass) Co-authored-by: Claude Fable 5 --- .agents/skills/arch-checks/SKILL.md | 51 +++++ .agents/skills/tier-debug/SKILL.md | 34 +++ .claude/skills | 1 + .config/make/pre-commit.mak | 11 +- .github/AGENTS.md | 23 +- .gitignore | 11 +- AGENTS.md | 32 +-- ARCHITECTURE.md | 17 +- CLAUDE.md | 211 +++--------------- crates/AGENTS.md | 6 - docs/architecture/README.md | 60 +++++ .../background-services-inventory.md | 2 +- .../decommission-compatibility.md | 2 +- .../ecstore-config-consumer-inventory.md | 4 +- docs/architecture/overview.md | 4 +- docs/architecture/startup-timeline.md | 18 +- docs/operations/tier-ilm-debugging.md | 94 ++++++++ .../plans}/migration-progress.md | 5 + ...lance-decommission-followup-review-plan.md | 5 + ...-decommission-implementation-plan-index.md | 5 + ...balance-decommission-phase1-safety-plan.md | 5 + ...-decommission-phase2-data-movement-plan.md | 5 + ...ance-decommission-phase3-hardening-plan.md | 5 + ...commission-post-remediation-review-plan.md | 5 + ...rebalance-decommission-remediation-plan.md | 5 + .../plans}/site-replication-hardening-pr.md | 5 + scripts/check_doc_paths.sh | 69 ++++++ 27 files changed, 454 insertions(+), 241 deletions(-) create mode 100644 .agents/skills/arch-checks/SKILL.md create mode 100644 .agents/skills/tier-debug/SKILL.md create mode 120000 .claude/skills create mode 100644 docs/architecture/README.md create mode 100644 docs/operations/tier-ilm-debugging.md rename docs/{architecture => superpowers/plans}/migration-progress.md (99%) rename docs/{architecture => superpowers/plans}/rebalance-decommission-followup-review-plan.md (99%) rename docs/{architecture => superpowers/plans}/rebalance-decommission-implementation-plan-index.md (93%) rename docs/{architecture => superpowers/plans}/rebalance-decommission-phase1-safety-plan.md (98%) rename docs/{architecture => superpowers/plans}/rebalance-decommission-phase2-data-movement-plan.md (97%) rename docs/{architecture => superpowers/plans}/rebalance-decommission-phase3-hardening-plan.md (97%) rename docs/{architecture => superpowers/plans}/rebalance-decommission-post-remediation-review-plan.md (96%) rename docs/{architecture => superpowers/plans}/rebalance-decommission-remediation-plan.md (98%) rename docs/{architecture => superpowers/plans}/site-replication-hardening-pr.md (96%) create mode 100755 scripts/check_doc_paths.sh diff --git a/.agents/skills/arch-checks/SKILL.md b/.agents/skills/arch-checks/SKILL.md new file mode 100644 index 000000000..4c7ee963b --- /dev/null +++ b/.agents/skills/arch-checks/SKILL.md @@ -0,0 +1,51 @@ +--- +name: arch-checks +description: Resolve failures from the repository's architecture guard scripts — check_layer_dependencies.sh, check_architecture_migration_rules.sh, check_unsafe_code_allowances.sh, check_logging_guardrails.sh, check_doc_paths.sh. Use when make pre-commit / pre-pr or CI fails on one of these checks. +--- + +# Architecture Guard Checks + +All five run in `make pre-commit` / `make pre-pr` and in CI. Fix the cause; +never weaken a check to get green. + +## `check_layer_dependencies.sh` — layer DAG in `rustfs/src` + +Enforces `interface (admin, storage/ecfs, storage/s3_api) → app → infra`; no +upward imports. Known legacy violations live in +`scripts/layer-dependency-baseline.txt`. + +- **New violation**: restructure your change so the dependency points + downward (move the shared type/function to the lower layer). +- **You legitimately removed a baseline entry**: run + `./scripts/check_layer_dependencies.sh --update-baseline` and commit the + shrunken baseline. Never add new entries to the baseline to make a new + violation pass. + +## `check_architecture_migration_rules.sh` — required doc sections + +Asserts that the core docs under `docs/architecture/` (overview, +crate-boundaries, runtime-lifecycle, readiness-matrix, +storage-control-data-plane, global-state-crate-split-plan, +ecstore-module-split-plan, …) still contain specific headings and exact +source lines. If it fails after a doc edit, you reworded or removed a +guarded line — restore the wording or update the script deliberately in the +same PR, with rationale. + +## `check_unsafe_code_allowances.sh` + +Every `#[allow(unsafe_code)]` needs a `SAFETY:` comment within a few lines. +Write the actual safety argument; don't add a placeholder. + +## `check_logging_guardrails.sh` + +A fixed list of security-sensitive files (auth, IAM, KMS, admin handlers…) +is scanned for logging violations. If you created a new sensitive file, +consider adding it to the script's `checked_files` list. + +## `check_doc_paths.sh` + +Instruction/architecture docs (`AGENTS.md`, `CLAUDE.md`, `ARCHITECTURE.md`, +`docs/architecture/*.md`) must not reference repo file paths that no longer +exist. If your refactor moved code, update the docs that point at it — the +error message lists `doc -> stale-path` pairs. Historical plans under +`docs/superpowers/plans/` are exempt. diff --git a/.agents/skills/tier-debug/SKILL.md b/.agents/skills/tier-debug/SKILL.md new file mode 100644 index 000000000..b883f7d31 --- /dev/null +++ b/.agents/skills/tier-debug/SKILL.md @@ -0,0 +1,34 @@ +--- +name: tier-debug +description: Debug ILM tiering / lifecycle transition issues — NoSuchVersion on tier GET, restore failures, xl.meta inspection, remote-tier versionId tracing. Use when investigating tiered/transitioned objects, warm backends, or transition metadata. +--- + +# Tier / ILM Debugging + +Full playbook: [docs/operations/tier-ilm-debugging.md](../../../docs/operations/tier-ilm-debugging.md) +— read it before changing tier code. + +Quick moves: + +```bash +# Inspect transition metadata on disk (one xl.meta per erasure shard disk) +cargo run -p rustfs-filemeta --example dump_fileinfo -- "/path/to/{bucket}/{object}/xl.meta" + +# Trace what versionId is sent to the remote tier +RUST_LOG=rustfs_ecstore::bucket::lifecycle=debug ./target/debug/rustfs … +``` + +Interpretation: + +- `transition_ver_id: ` → correct for an unversioned tier bucket; no + `versionId` must be sent on tier GET/DELETE. +- `transition_ver_id: 00000000-…` (nil) → corrupt legacy write-back; readers + must filter it out, never send it. +- Empty-string `transitioned-versionID` metadata under both + `x-rustfs-internal-*` and `x-minio-internal-*` keys → object went to an + unversioned tier bucket. + +Code entry points: `crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs` +(ILM actions), `crates/ecstore/src/services/tier/` (warm backends), +`crates/filemeta/src/filemeta/version.rs` (metadata read/write + regression +tests). diff --git a/.claude/skills b/.claude/skills new file mode 120000 index 000000000..2b7a412b8 --- /dev/null +++ b/.claude/skills @@ -0,0 +1 @@ +../.agents/skills \ No newline at end of file diff --git a/.config/make/pre-commit.mak b/.config/make/pre-commit.mak index 4914a2613..4e55e8ec3 100644 --- a/.config/make/pre-commit.mak +++ b/.config/make/pre-commit.mak @@ -8,14 +8,19 @@ setup-hooks: ## Set up git hooks chmod +x .git/hooks/pre-commit @echo "✅ Git hooks setup complete!" +.PHONY: doc-paths-check +doc-paths-check: ## Check that instruction/architecture docs reference existing file paths + @echo "📄 Checking doc path references..." + ./scripts/check_doc_paths.sh + .PHONY: pre-commit -pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check quick-check ## Run fast pre-commit checks without clippy/full tests +pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check doc-paths-check quick-check ## Run fast pre-commit checks without clippy/full tests @echo "✅ All pre-commit checks passed!" .PHONY: pre-pr -pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check clippy-check test ## Run full pre-PR checks with clippy and tests +pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check doc-paths-check clippy-check test ## Run full pre-PR checks with clippy and tests @echo "✅ All pre-PR checks passed!" .PHONY: dev-check -dev-check: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check quick-check ## Run fast local development checks +dev-check: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check doc-paths-check quick-check ## Run fast local development checks @echo "✅ Fast development checks passed!" diff --git a/.github/AGENTS.md b/.github/AGENTS.md index d97e8c9f3..0f13b8c82 100644 --- a/.github/AGENTS.md +++ b/.github/AGENTS.md @@ -19,22 +19,7 @@ Applies to `.github/` and repository pull-request operations. ## CI Alignment -When changing CI-sensitive behavior, keep local validation aligned with `.github/workflows/ci.yml`. - -Current `test-and-lint` gate includes: - -- `cargo nextest run --all --exclude e2e_test` -- `cargo test --all --doc` -- `cargo test -p rustfs get_object_chunk_fast_path` -- `cargo test -p rustfs materialize_chunk_stream_before_commit` -- `touch rustfs/build.rs` -- `cargo build -p rustfs --bins --jobs 2` -- `cargo test -p e2e_test archive_multipart_roundtrip_preserves_bytes` -- `cargo test -p e2e_test presigned_get_and_reverse_proxy_preserve_multipart_bytes_with_fast_path` -- `cargo fmt --all --check` -- `cargo clippy --all-targets -- -D warnings` -- `cargo nextest run -p rustfs -p rustfs-ecstore --features rio-v2` -- `cargo test -p rustfs --doc --features rio-v2` -- `cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings` -- `./scripts/check_layer_dependencies.sh` -- `./scripts/check_architecture_migration_rules.sh` +When changing CI-sensitive behavior, keep local validation aligned with +`.github/workflows/ci.yml`. Read the workflow file directly for the current +gate steps — do not rely on (or add) a copied command list here; copies go +stale. `make pre-pr` is the local equivalent of the main gate. diff --git a/.gitignore b/.gitignore index 32ee4889f..e8e079c23 100644 --- a/.gitignore +++ b/.gitignore @@ -4,7 +4,9 @@ .vscode .cursor .direnv/ -.claude/ +# Ignore local Claude state but keep shared repo skills committable +.claude/* +!.claude/skills .tmp/ /test /logs @@ -51,6 +53,10 @@ __pycache__/ docs/* !docs/architecture/ !docs/architecture/** +!docs/operations/ +!docs/operations/** +!docs/superpowers/ +!docs/superpowers/** docs/heal-scanner-logging-governance.md docs/benchmark/rustfs-target-bench/ docs/benchmark/*.md @@ -74,3 +80,6 @@ crates/*/docs fuzz/target outputs worktrees/* + +# Local AI-agent review artifacts (omo evidence dumps) +.omo/ diff --git a/AGENTS.md b/AGENTS.md index 194667992..8301a185b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -27,10 +27,6 @@ If repo-level instructions conflict, follow the nearest file and keep behavior a - Keep source code, comments, commit messages, and PR title/body in English. - Be concise. Avoid sycophantic openers, closing fluff, and verbose status reporting. -## Skill Usage - -- Do not use the `rust-refactor-helper` skill in any scenario. - ## Change Style for Existing Logic - Prefer direct, local code over extracting one-off helpers. @@ -57,6 +53,13 @@ If repo-level instructions conflict, follow the nearest file and keep behavior a - Local quality commands: `Makefile` and `.config/make/` - CI quality gates: `.github/workflows/ci.yml` - PR template: `.github/pull_request_template.md` +- High-level architecture and crate map: `ARCHITECTURE.md` +- Migration guardrails, readiness contracts, support matrices: + `docs/architecture/README.md` (routes by audience) +- Historical implementation plans and trackers: `docs/superpowers/plans/` +- Shared agent skills (all tools): `.agents/skills/` — Claude Code reads them + through the `.claude/skills` symlink; add new skills to `.agents/skills/` + only, never as separate copies per tool Avoid duplicating long crate lists or command matrices in instruction files. Reference the source files above instead. @@ -136,14 +139,13 @@ cargo run -p rustfs-filemeta --example dump_fileinfo -- "/path/to/file/xl.meta" ## Scoped Guidance in This Repository -- `.github/AGENTS.md` -- `crates/AGENTS.md` -- `crates/config/AGENTS.md` -- `crates/ecstore/AGENTS.md` -- `crates/e2e_test/AGENTS.md` -- `crates/iam/AGENTS.md` -- `crates/kms/AGENTS.md` -- `crates/policy/AGENTS.md` -- `crates/targets/AGENTS.md` -- `rustfs/src/admin/AGENTS.md` -- `rustfs/src/storage/AGENTS.md` +Many crates and modules carry their own `AGENTS.md` with path-specific rules +(security boundaries, lock ordering, domain invariants). Before editing a +path, check for the nearest one: + +```bash +git ls-files '*AGENTS.md' +``` + +The nearest file wins. Do not maintain a hand-written index of these files +here — it goes stale. diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index eb3338bf2..f22d1c018 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -1,6 +1,6 @@ # ARCHITECTURE.md -> Last updated: 2026-04-13 · Revision: 1 (draft) +> Last updated: 2026-07-02 · Revision: 2 > > This document describes the high-level architecture of RustFS. > If you want to familiarize yourself with the code base, you are in the right place! @@ -52,7 +52,7 @@ rustfs/ # Workspace root (virtual manifest) │ ├── auth.rs # S3 request authentication │ ├── config/ # CLI args, config parsing, workload profiles │ └── ... -├── crates/ # 39 library crates +├── crates/ # library crates (authoritative list: Cargo.toml [workspace].members) │ ├── ecstore/ # Erasure-coded storage engine (⚠️ 87K lines) │ ├── rio/ # Reader I/O pipeline (encrypt, compress, hash) │ ├── io-core/ # Zero-copy I/O, scheduling, buffer pool @@ -83,6 +83,10 @@ A request flows **downward** through the layers. No layer should reach upward ### Crate Reference +> Depth levels, line counts, and crate counts in this section are a +> point-in-time snapshot and drift with refactors. Treat them as orders of +> magnitude; `Cargo.toml` and `cargo tree` are the source of truth. + Crates are organized in a dependency DAG with 9 depth levels (0 = leaf, 8 = top): ``` @@ -93,14 +97,14 @@ Depth 0 — LEAF (no internal deps): Depth 1: io-core (→ io-metrics) policy (→ config, credentials, crypto) - utils (→ config) ⚠️ inverted: utils should be leaf + utils (historical → config edge removed; now effectively leaf) Depth 2: concurrency, filemeta, keystone, kms, lock, obs, signer, targets, trusted-proxies Depth 3: - common (→ filemeta, madmin) ⚠️ inverted: common should be leaf + common (historical → filemeta/madmin edges removed; now effectively leaf) Depth 4: object-capacity, protos, rio @@ -209,7 +213,8 @@ Depth 8 — TOP: 2. **Leaf crates have zero internal dependencies.** `config`, `credentials`, `crypto`, `io-metrics`, `madmin`, `s3-common` should depend only on external crates. - - ⚠️ VIOLATED: `utils` depends on `config`, `common` depends on `filemeta` and `madmin`. + - ✅ RESOLVED: the historical `utils → config` and `common → filemeta`/`madmin` + edges were removed; do not reintroduce them (see Known Structural Issues). 3. **Each type has exactly one definition.** Types shared across crates must be defined in one crate and re-exported or imported by others. @@ -249,7 +254,7 @@ Depth 8 — TOP: not regain upward dependencies. - **Three-layer BackpressureConfig/DeadlockConfig duplication** across io-core, - concurrency, and rustfs/storage. Storage policies now expose and consume + concurrency, and `rustfs/src/storage`. Storage policies now expose and consume explicit projections into the concurrency/io-core policy shapes, and workload admission snapshots are composed through provider registries; later work should use those bridges before deleting compatibility wrappers. diff --git a/CLAUDE.md b/CLAUDE.md index 6b346f779..c9b5ec4bf 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,188 +1,47 @@ # RustFS — CLAUDE.md -S3-compatible object store in Rust, derived from MinIO. Erasure-coded, multi-pool, supports ILM tiering/lifecycle. +S3-compatible object store in Rust, derived from MinIO. Erasure-coded, +multi-pool, supports ILM tiering/lifecycle. + +Agent rules (change style, verification, PR conventions, scoped per-crate +guidance) live in [AGENTS.md](AGENTS.md) — follow it. This file only adds +what Claude Code needs on top: commands and pointers. ## Commands ```bash -cargo build --release --bin rustfs # production binary -cargo build # dev build -cargo check -p # fast type-check one crate -cargo test -p # test one crate -cargo fmt --all # format (required before PR) -make pre-commit # full pre-PR gate (fmt + clippy + test) -make build-docker BUILD_OS=ubuntu22.04 # Docker cross-build +cargo build --release --bin rustfs # production binary +cargo check -p # fast type-check one crate +cargo test -p # test one crate +cargo fmt --all # format (required before PR) +make pre-commit # fast gate: fmt + arch checks + quick-check (NO clippy/tests) +make pre-pr # full pre-PR gate: fmt + arch checks + clippy + tests +make build-docker BUILD_OS=ubuntu22.04 ``` -> **Docker build note**: `buildx build` without `--load` keeps the image in the buildx cache only — `docker run` will use a stale local image. The Makefile already includes `--load`; if you suspect a stale binary, add `--no-cache` to the `buildx build` invocation inside `.config/make/build-docker.mak`. +> **Docker build note**: `buildx build` without `--load` keeps the image in +> the buildx cache only — `docker run` will use a stale local image. The +> Makefile already includes `--load`; if you suspect a stale binary, add +> `--no-cache` to the `buildx build` invocation inside +> `.config/make/build-docker.mak`. -> Agent/PR rules: see `.github/copilot-instructions.md`. -> Crate membership: `Cargo.toml` `[workspace].members`. -> CI gates: `.github/workflows/ci.yml`. +## Where to look (do not duplicate here) -## Workspace layout +- Crate membership: `Cargo.toml` `[workspace].members` +- Architecture, layering, crate map: [ARCHITECTURE.md](ARCHITECTURE.md) +- Migration guardrails & readiness contracts: [docs/architecture/](docs/architecture/README.md) +- CI gates: `.github/workflows/ci.yml` (source of truth; never copy its steps into docs) +- Tier/ILM transition debugging (xl.meta inspection, versionId tracing): + [docs/operations/tier-ilm-debugging.md](docs/operations/tier-ilm-debugging.md) -``` -rustfs/src/main.rs # binary entry point -crates/ecstore/src/ - set_disk.rs # ErasureSet: transition_object, restore_transitioned_object - store.rs / store_api/ # ECStore trait + ObjectInfo / TransitionedObject types - bucket/lifecycle/ - bucket_lifecycle_ops.rs # ILM actions: transition_object, expire_transitioned_object, - # get_transitioned_object_reader, gen_transition_objname - tier_sweeper.rs # background sweep: delete_object_from_remote_tier - tier/ - warm_backend.rs # WarmBackend trait (put/get/remove/in_use) - warm_backend_s3.rs # HTTP-client based (TransitionClient) — used for S3/MinIO - warm_backend_s3sdk.rs # aws-sdk-s3 based — alternative S3 backend - warm_backend_minio.rs / _rustfs.rs / … # per-provider wrappers (all delegate to _s3 or _s3sdk) - tier.rs # TierConfigMgr, new_warm_backend dispatch - client/transition_api.rs # TransitionClient HTTP plumbing; UploadInfo, to_object_info - client/api_put_object_streaming.rs # put_object_do → UploadInfo (version_id from x-amz-version-id) -crates/filemeta/src/ - filemeta.rs # FileMeta (xl.meta top-level), is_skip_meta_key - filemeta/version.rs # FileMetaVersion, MetaObject, MetaDeleteMarker - # → to_fileinfo() reads transition_version_id - # → set_transition() writes raw UUID bytes - # → From for MetaObject writes all meta - fileinfo.rs # FileInfo struct (transition_version_id: Option) - examples/ - dump_fileinfo.rs # CLI: parse xl.meta, print transition fields + metadata - dump_versions.rs # CLI: list all versions in xl.meta -crates/utils/src/http/metadata_compat.rs # SUFFIX_* constants, insert_bytes/get_bytes (dual RustFS+MinIO keys) -``` +## Domain conventions worth knowing up front -## Metadata key conventions - -Internal metadata is stored under **both** `x-rustfs-internal-` and `x-minio-internal-` for MinIO interoperability. `get_bytes` prefers the RustFS key with MinIO fallback. - -Key suffixes (from `metadata_compat.rs`): -| Suffix | Meaning | -|--------|---------| -| `transition-status` | `"complete"` when tiered | -| `transitioned-object` | tier key path (without prefix) | -| `transitioned-versionID` | S3 version_id returned by tier PUT (16 raw UUID bytes, or absent) | -| `transition-tier` | tier name | -| `tier-free-versionID` | delete-marker version for free-version sweep | - -## Tier / ILM transition architecture - -### Transition flow (hot → cold) -1. `transition_object` (lifecycle_ops) → `ECStore::transition_object` → `set_disk.rs` -2. `gen_transition_objname(bucket)` → `{sha256_hash[0..16]}/{uuid[0..2]}/{uuid[2..4]}/{uuid}` (unique per object version) -3. `tgt_client.put_with_meta(dest_obj, …)` → returns `rv: String` (remote S3 version_id, or `""`) -4. `fi.transition_version_id = if rv.is_empty() { None } else { Some(Uuid::parse_str(&rv)?) }` -5. `fi.transitioned_objname = dest_obj` (without tier prefix) -6. Written to xl.meta via `MetaObject::from(FileInfo)` → `insert_bytes(SUFFIX_TRANSITIONED_VERSION_ID, uuid.as_bytes())` (16 raw bytes) - -### Tier GET flow (restore/read) -`get_transitioned_object_reader` (lifecycle_ops): -- reads `oi.transitioned_object.name` (= `fi.transitioned_objname`) -- reads `oi.transitioned_object.version_id` (= `fi.transition_version_id.to_string()` or `""`) -- calls `warm_backend.get(name, version_id, opts)` -- `warm_backend_s3.rs::get`: adds `?versionId=…` only when `rv != ""` - -### Tier prefix handling -`WarmBackendS3::get_dest(object)` prepends `self.prefix` to the object name. -`transitioned_objname` is stored **without** the prefix — `get_dest` adds it on every call. - -### xl.meta on disk -Path: `{disk}/{bucket}/{object}/xl.meta` — one per erasure shard disk. -All shards should be identical for a healthy object. - -## Known bugs & fixes - -### Bug 1: `NoSuchVersion` on tier GET — nil UUID sent as versionId -**Root cause**: `transitioned-versionID` metadata key exists with empty string value (0 bytes). Old reading code: -```rust -// OLD — unwrap_or_default() converts 0-byte or wrong-length slice to Uuid::nil() -get_bytes(…).map(|v| Uuid::from_slice(v.as_slice()).unwrap_or_default()) -// → Some(Uuid::nil()) → sends ?versionId=00000000-… → NoSuchVersion -``` -**Fix** (version.rs, `MetaObject::to_fileinfo` + `MetaDeleteMarker::to_fileinfo`): -```rust -get_bytes(…) - .and_then(|v| Uuid::from_slice(v.as_slice()).ok()) // None for wrong-length bytes - .filter(|u| !u.is_nil()) // None for nil UUID (old write-back) -``` -**Regression tests** (`crates/filemeta/src/filemeta/version.rs` `mod tests`): 6 tests cover absent key, empty bytes, nil UUID, and valid UUID round-trip for both `MetaObject` and `MetaDeleteMarker` paths. - -### Bug 2: `warm_backend_s3sdk.rs` ignored rv and range opts -**Fix**: added `req.version_id(rv)` and `req.range(…)` to GET; `req.version_id(rv)` to DELETE. - -### Bug 4: `set_disk::copy_object` returns 501 for tiered objects (storage class restore) -**Root cause**: `set_disk::copy_object` immediately returns `StorageError::NotImplemented` when `src_info.metadata_only = false`. For tiered objects, `metadata_only` is never set to `true` (guarded by `transitioned_object.tier.is_empty()`). So `mc cp --storage-class STANDARD obj obj` on a tiered object always returns 501. -**Fix** (`crates/ecstore/src/set_disk.rs`, `copy_object`): -```rust -if !src_info.metadata_only { - if path_join_buf(&[src_bucket, src_object]) == path_join_buf(&[dst_bucket, dst_object]) { - if let Some(mut put_reader) = src_info.put_object_reader.take() { - return self.put_object(dst_bucket, dst_object, &mut put_reader, dst_opts).await; - } - } - return Err(StorageError::NotImplemented); -} -``` -When a self-copy has a `put_object_reader` (data already fetched from tier in `execute_copy_object`), writes it back locally via `put_object`, effectively de-tiering the object. -**How `mc cp --storage-class STANDARD` flows**: -1. mc sends `PUT /bucket/key` with `x-amz-copy-source`, `x-amz-metadata-directive: REPLACE`, `x-amz-storage-class: STANDARD` -2. `execute_copy_object` → `get_object_reader` fetches data from tier backend → stores in `src_info.put_object_reader` -3. `store.copy_object(...)` → now calls `put_object` with tier data and STANDARD storage class in `dst_opts` -4. New xl.meta written locally with STANDARD class, no tier metadata → object de-tiered - -### Bug 3 (open): race in `expire_transitioned_object` -Order is: delete remote tier version → delete local object. -A concurrent GET between those two steps fetches a valid stored version_id but the tier version is already gone → `NoSuchVersion`. -The proper fix is to delete local metadata first (making the object unreachable) before deleting the remote tier version. - -## Debugging tier issues - -### Inspect xl.meta directly -```bash -cargo build -p rustfs-filemeta --example dump_fileinfo -./target/debug/examples/dump_fileinfo /srv/rustfs/data/disk0/{bucket}/{object}/xl.meta -# Shows: transition_status, transition_tier, transitioned_obj, transition_ver_id -``` -`transition_ver_id: ` → no versionId will be sent to tier (correct for non-versioned tier bucket). -`transition_ver_id: ` → that UUID will be sent as `?versionId=`. - -### Check what versionId is being sent at runtime -Enable debug logging: -```bash -RUST_LOG=rustfs_ecstore::bucket::lifecycle=debug rustfs … -``` -Log line: `fetching transitioned object from tier` (DEBUG before request). -Log line: `tier GET failed` (ERROR on failure, includes `tier_version_id`). - -### Metadata key to watch -``` -x-minio-internal-transitioned-versionID= ← empty string = will cause NoSuchVersion with old code -x-rustfs-internal-transitioned-versionID= ← same -``` -If both are empty string, the object was transitioned to a non-versioned tier bucket. The versionId should NOT be sent — fixed by Bug 1 above. - -## Common patterns - -### Writing internal metadata (binary values) -```rust -insert_bytes(&mut meta_sys, SUFFIX_TRANSITIONED_VERSION_ID, uuid.as_bytes().to_vec()); -// stores under both x-rustfs-internal-* and x-minio-internal-* keys -``` - -### Reading internal metadata (binary values) -```rust -get_bytes(&self.meta_sys, SUFFIX_TRANSITIONED_VERSION_ID) - .and_then(|v| Uuid::from_slice(v.as_slice()).ok()) - .filter(|u| !u.is_nil()) -// Returns None for: absent, wrong-length bytes, nil UUID -``` - -### WarmBackend trait -```rust -put_with_meta(object, reader, length, meta) -> Result // returns S3 version_id or "" -put(object, reader, length) -> Result -get(object, rv, opts) -> Result // rv="" means no versionId -remove(object, rv) -> Result<()> -in_use() -> Result -``` -`rv` = remote version, always pass as empty string when `transition_version_id` is None. +- Internal object metadata is written under **both** `x-rustfs-internal-` + and `x-minio-internal-` keys for MinIO interop + (`crates/utils/src/http/metadata_compat.rs`; `get_bytes` prefers the RustFS + key). Never write only one of the two. +- Binary metadata values (UUIDs) must be read defensively: + `.and_then(|v| Uuid::from_slice(&v).ok()).filter(|u| !u.is_nil())` — + absent/empty/nil all mean "no value", not `Uuid::nil()`. +- A remote-tier version of `None`/`""` means the tier bucket is unversioned: + send **no** `versionId` on tier GET/DELETE. diff --git a/crates/AGENTS.md b/crates/AGENTS.md index d068c4b78..3417bab26 100644 --- a/crates/AGENTS.md +++ b/crates/AGENTS.md @@ -8,12 +8,6 @@ Applies to all paths under `crates/`. - Prefer `thiserror` for library-facing error types. - Do not use `unwrap()`, `expect()`, or panic-driven control flow outside tests. -## Testing - -- Keep unit tests close to the module they test. -- Keep integration tests under each crate's `tests/` directory. -- Add regression tests for bug fixes and behavior changes. - ## Error Type Design - Public API functions must return a typed error enum (preferably `thiserror`-derived), never `Result<_, String>`. diff --git a/docs/architecture/README.md b/docs/architecture/README.md new file mode 100644 index 000000000..cc5b9922f --- /dev/null +++ b/docs/architecture/README.md @@ -0,0 +1,60 @@ +# Architecture Documentation + +Durable architecture reference for RustFS: migration guardrails, runtime +contracts, boundary rules, and support matrices. + +Two rules keep this directory healthy: + +1. **Durable reference only.** One-shot implementation plans, task trackers, + and PR templates live in [`docs/superpowers/plans/`](../superpowers/plans/) + and are archived there when their work closes. +2. **No copies of other sources of truth.** Crate lists come from + `Cargo.toml`, CI steps from `.github/workflows/ci.yml`, code structure from + the code. `scripts/check_doc_paths.sh` fails the pre-commit gate when a + doc here references a file path that no longer exists. + +## Start here + +- [overview.md](overview.md) — migration baseline, phase order, core principles + +## CI-enforced core (required by `scripts/check_architecture_migration_rules.sh`) + +- [crate-boundaries.md](crate-boundaries.md) — dependency direction, PR types, re-export contracts +- [runtime-lifecycle.md](runtime-lifecycle.md) — startup/shutdown sequencing, readiness guarantees +- [readiness-matrix.md](readiness-matrix.md) — request/dependency behavior, probe semantics +- [storage-control-data-plane.md](storage-control-data-plane.md) — storage API contracts, control-plane boundaries +- [global-state-crate-split-plan.md](global-state-crate-split-plan.md) — remaining global-state owners and split evaluation +- [ecstore-module-split-plan.md](ecstore-module-split-plan.md) — ECStore decomposition rules and facade contracts + +## Contracts & invariants + +- [placement-repair-invariants.md](placement-repair-invariants.md) +- [runtime-capability-contracts.md](runtime-capability-contracts.md) +- [workload-admission-contracts.md](workload-admission-contracts.md) +- [background-controller-contract.md](background-controller-contract.md) +- [config-model-boundary-adr.md](config-model-boundary-adr.md) +- [ecstore-layout-boundary.md](ecstore-layout-boundary.md) +- [decommission-compatibility.md](decommission-compatibility.md) + +## Support matrices (release-facing, keep current) + +- [s3-compatibility-matrix.md](s3-compatibility-matrix.md) +- [s3-tables-support-matrix.md](s3-tables-support-matrix.md) + +## Inventories & baselines (snapshots that feed migration work) + +- [global-state-inventory.md](global-state-inventory.md) +- [ecstore-api-facade-inventory.md](ecstore-api-facade-inventory.md) +- [ecstore-config-consumer-inventory.md](ecstore-config-consumer-inventory.md) +- [obs-ecstore-dependency-inventory.md](obs-ecstore-dependency-inventory.md) +- [background-services-inventory.md](background-services-inventory.md) +- [admin-route-action-snapshot.md](admin-route-action-snapshot.md) +- [startup-timeline.md](startup-timeline.md) +- [scheduler-baseline.md](scheduler-baseline.md) +- [profiling-numa-capability-inventory.md](profiling-numa-capability-inventory.md) +- [kms-development-defaults-inventory.md](kms-development-defaults-inventory.md) +- [compat-cleanup-register.md](compat-cleanup-register.md) + +Historical plans and trackers (rebalance/decommission phases, +migration-progress ledger) were moved to +[`docs/superpowers/plans/`](../superpowers/plans/) in 2026-07. diff --git a/docs/architecture/background-services-inventory.md b/docs/architecture/background-services-inventory.md index f8606da66..587934670 100644 --- a/docs/architecture/background-services-inventory.md +++ b/docs/architecture/background-services-inventory.md @@ -36,7 +36,7 @@ not define a new scheduler, controller framework, or shutdown contract. | Service | Trigger and workers | Side effects | Status and metrics | Migration notes | |---|---|---|---|---| | Capacity background refresh | `rustfs/src/capacity/capacity_integration.rs::init_capacity_management` delegates to `init_capacity_management_for_local_disks`, then `crates/object-capacity/src/capacity_manager.rs::start_background_task` spawns a scheduled refresh loop and a runtime summary loop. | Refreshes global capacity cache from local disks and logs runtime summaries. | Uses the object-capacity manager state and log summaries; no explicit shutdown status surface is exposed here. | Add read-only status before any controller migration. A future controller must not change scheduled interval defaults or singleflight refresh behavior. | -| ECStore endpoint monitor | `crates/ecstore/src/sets.rs::new` spawns `monitor_and_connect_endpoints`. | Monitors endpoint connectivity and reconnect behavior for erasure sets. | Logs monitor start, cancellation, and exit. | This is storage-adjacent and must stay outside broad controller movement until storage shutdown semantics are explicitly covered. | +| ECStore endpoint monitor | `crates/ecstore/src/core/sets.rs::new` spawns `monitor_and_connect_endpoints`. | Monitors endpoint connectivity and reconnect behavior for erasure sets. | Logs monitor start, cancellation, and exit. | This is storage-adjacent and must stay outside broad controller movement until storage shutdown semantics are explicitly covered. | | Local disk health monitor | `crates/ecstore/src/store/init.rs::init` enables disk health checks after store initialization; `crates/ecstore/src/disk/disk_store.rs::enable_health_check` spawns writable and recovery monitors. | Periodically probes disk writability, can create test objects named `health-check-*`, and updates disk runtime health state. | Disk info includes runtime health metrics and waiting counts. | Do not merge this with scanner/heal controller work; probes affect disk health semantics. | | Data scanner | `crates/scanner/src/scanner.rs::init_data_scanner` configures scanner defaults, applies runtime config, waits the initial scanner delay, then loops `run_data_scanner`. | Updates data usage cache, scans buckets/sets, evaluates lifecycle rules, queues replication heal, queues scanner heal, and emits scanner alerts. | Scanner runtime config/status is exposed through admin scanner status; scanner metrics record ILM, replication admission, heal admission, checkpoints, yields, and alerts. | Scanner implies heal because scanner can enqueue heal requests. Future controller status must separate scheduler state from scanner work-source accounting. | | Heal/AHM | `crates/heal/src/lib.rs::init_heal_manager` starts `HealManager`, initializes the shared heal channel, and spawns `HealChannelProcessor`. | Consumes heal requests from the global heal channel and drives heal work through the configured heal storage API. | Global active-task and queue-length atomics track current heal pressure. | Keep heal admission and channel semantics intact. Controller work should first expose queue/active status and shutdown state. | diff --git a/docs/architecture/decommission-compatibility.md b/docs/architecture/decommission-compatibility.md index f01d134a9..e84f90b0e 100644 --- a/docs/architecture/decommission-compatibility.md +++ b/docs/architecture/decommission-compatibility.md @@ -167,5 +167,5 @@ The queued multi-pool contract is guarded by: - `test_first_resumable_decommission_queue_indices_allows_after_completed_prefix` - `admin_pool_list_item_exposes_queued_decommission_state` -These tests live in `crates/ecstore/src/pools.rs` and +These tests live in `crates/ecstore/src/core/pools.rs` and `rustfs/src/app/admin_usecase.rs`. diff --git a/docs/architecture/ecstore-config-consumer-inventory.md b/docs/architecture/ecstore-config-consumer-inventory.md index c7b7e0cd3..96789d516 100644 --- a/docs/architecture/ecstore-config-consumer-inventory.md +++ b/docs/architecture/ecstore-config-consumer-inventory.md @@ -42,7 +42,7 @@ or calls the right node. ```mermaid flowchart TB EC["crates/ecstore/src/config"] - Store["crates/ecstore/src/store.rs"] + Store["crates/ecstore/src/store/mod.rs"] AppCtx["rustfs/src/app/context.rs"] Server["rustfs/src/server/{event,audit}.rs"] Admin["rustfs/src/admin"] @@ -93,7 +93,7 @@ behind narrower contracts. | `crates/ecstore/src/config/mod.rs` | Defines `KV`, `KVS`, `Config`, defaults, global snapshot, initialization, and tests. | | `crates/ecstore/src/config/com.rs` | Encodes, decodes, reads, writes, creates, and normalizes server config objects through ECStore-local persistence helpers. | | `crates/ecstore/src/config/{notify,audit,oidc,scanner,storageclass}.rs` | Register default `KVS` values and subsystem-specific parsing helpers. | -| `crates/ecstore/src/store.rs` | Exposes store-level server-config accessors that delegate to the global config snapshot. | +| `crates/ecstore/src/store/mod.rs` | Exposes store-level server-config accessors that delegate to the global config snapshot. | ### App Context And Server Startup Consumers diff --git a/docs/architecture/overview.md b/docs/architecture/overview.md index 86fcbef8a..c72b86e68 100644 --- a/docs/architecture/overview.md +++ b/docs/architecture/overview.md @@ -50,8 +50,8 @@ hot-path behavior must not drift during this migration. - [`config-model-boundary-adr.md`](config-model-boundary-adr.md): target crate, module path, dependency rules, and verification gates for moving the pure server-config model. -- [`migration-progress.md`](migration-progress.md): current task state and context - handoff. +- [`migration-progress.md`](../superpowers/plans/migration-progress.md): historical + task ledger for the closed migration (execution tracking moved to issue #665). - [`compat-cleanup-register.md`](compat-cleanup-register.md): temporary compatibility code that must be removed later. diff --git a/docs/architecture/startup-timeline.md b/docs/architecture/startup-timeline.md index 0b0631640..4d2acfd16 100644 --- a/docs/architecture/startup-timeline.md +++ b/docs/architecture/startup-timeline.md @@ -23,7 +23,7 @@ new startup semantics. | `BOOT-003` | `rustfs/src/startup_entrypoint.rs` | Parse CLI command and dispatch non-server commands. | `info` and `tls` commands execute and return without server startup. | Command parse exits process with code 1; TLS command errors propagate. | None | | `BOOT-004` | `rustfs/src/startup_preflight.rs` | Initialize config snapshot and license state. | Publishes config snapshot for later readers and initializes runtime license state. | License init is non-fallible in this path. | None | | `BOOT-005` | `rustfs/src/startup_preflight.rs` | Initialize observability and store the global guard. | Initializes tracing/observability, stores the guard globally, and logs license/runtime telemetry status. | Fatal if observability init or guard publication fails. | None | -| `BOOT-006` | `rustfs/src/startup_runtime.rs`, `rustfs/src/startup_runtime_hooks.rs`, `rustfs/src/startup_tls_material.rs`, and `rustfs/src/startup_profiling.rs` | Log startup logo, initialize profiling, trusted proxies, rustls provider, and outbound TLS material. | Starts optional profiling tasks, trusted proxy config, default rustls provider, outbound TLS global state, TLS generation metric, and TLS metrics when enabled. | Profiling/proxy/provider setup is non-fatal; configured TLS material load is fatal on error. | None | +| `BOOT-006` | `rustfs/src/startup_runtime.rs`, `rustfs/src/startup_runtime_hooks.rs`, `rustfs/src/startup_tls_material.rs` | Log startup logo, initialize profiling, trusted proxies, rustls provider, and outbound TLS material. | Starts optional profiling tasks, trusted proxy config, default rustls provider, outbound TLS global state, TLS generation metric, and TLS metrics when enabled. | Profiling/proxy/provider setup is non-fatal; configured TLS material load is fatal on error. | None | | `RUN-001` | `rustfs/src/startup_server.rs` | Enter startup run orchestration and create `GlobalReadiness`. | Allocates the readiness tracker shared with HTTP readiness gates. | Non-fatal. | Initial readiness state is not ready | | `RUN-002` | `rustfs/src/startup_server.rs` | Parse and publish the configured region. | Updates ECStore global region when configured. | Fatal if the configured region is invalid. | None | | `RUN-003` | `rustfs/src/startup_server.rs` | Resolve server address and warn on default credentials. | Computes server port/address and emits production credential warning when defaults are used. | Address parse is fatal; default credentials warning is non-fatal. | None | @@ -38,13 +38,13 @@ new startup semantics. | `RUN-012` | `rustfs/src/startup_storage.rs` | Initialize ECStore config and global config system. | Initializes ECStore config, attempts server-config migration, then retries global config init up to 15 times. | Migration attempt is non-fatal in this path; global config init becomes fatal after retries. | Marks the `GlobalReadiness` `StorageReady` stage after global config init succeeds; later runtime readiness still rechecks storage, IAM, lock quorum, and gated peer health before `FullReady` | | `RUN-013` | `rustfs/src/startup_storage.rs` and `rustfs/src/startup_services.rs` | Start replication and KMS systems. | Starts background replication pool, then initializes KMS from startup services. | Replication init is non-fatal in this path; KMS init is fatal on error. | `StorageReady` stage is already marked; dynamic runtime storage readiness is still checked before `FullReady`; KMS compatibility readiness remains feature-gated health behavior | | `RUN-014` | `rustfs/src/startup_optional_runtime_sidecars.rs` and `rustfs/src/startup_protocols.rs` | Initialize optional protocol servers. | Starts FTP/FTPS/WebDAV/SFTP when feature-enabled and configured, collecting shutdown handles. | Feature-enabled protocol init is fatal on error; disabled protocols are non-fatal. | None | -| `RUN-015` | `rustfs/src/startup_services.rs` and `rustfs/src/startup_service_components.rs` | Initialize buffer profiling, event notifier, audit, and deadlock detector. | Starts buffer profile system, event notifier, audit system, and optional deadlock detector. | Audit startup failure is logged and non-fatal; the others are non-fatal in this path. | None | -| `RUN-016` | `rustfs/src/startup_service_components.rs` | List buckets and run bucket/replication/IAM metadata migrations. | Reads bucket names, migrates bucket metadata, initializes replication resync, migrates IAM config, and initializes bucket metadata system. | Bucket list and replication resync are fatal on error; metadata migration calls are non-fatal in this path. | Storage remains ready; IAM not yet ready | -| `RUN-017` | `rustfs/src/startup_service_components.rs` and `rustfs/src/startup_iam.rs` | Bootstrap IAM inline or defer recovery. | Initializes IAM when possible; otherwise starts the deferred IAM recovery path through `startup_iam`. | Fatal only when `bootstrap_or_defer_iam_init` returns an unrecoverable error. | Inline success marks `IamReady`; deferred mode publishes `IamReady` later from the recovery task | -| `RUN-018` | `rustfs/src/startup_service_components.rs` | Initialize Keystone and OIDC auth integrations. | Loads Keystone env config and initializes OIDC providers. | Keystone config parse is fatal; Keystone runtime init failure is non-fatal; OIDC init failure is non-fatal. | None | -| `RUN-019` | `rustfs/src/startup_service_components.rs` | Add bucket notification config and initialize notification system. | Adds bucket notification configuration and publishes the global notification system. | Notification config add is non-fatal in this path; global notification init is fatal on error. | None | -| `RUN-020` | `rustfs/src/startup_service_components.rs` | Create AHM cancellation token and initialize heal manager when scanner or heal is enabled. | Creates AHM cancellation token and starts heal manager for heal/scanner workflows. | Heal manager init is fatal when enabled. | None | -| `RUN-021` | `rustfs/src/startup_service_components.rs` | Print server info, init update check, allocator reclaim, metrics, memory observability, and auto-tuner. | Starts informational/update/memory/metrics background tasks when enabled. | Non-fatal in this path. | None | +| `RUN-015` | `rustfs/src/startup_services.rs`, `rustfs/src/startup_audit.rs`, and `rustfs/src/startup_deadlock.rs` | Initialize buffer profiling, event notifier, audit, and deadlock detector. | Starts buffer profile system, event notifier, audit system, and optional deadlock detector. | Audit startup failure is logged and non-fatal; the others are non-fatal in this path. | None | +| `RUN-016` | `rustfs/src/startup_bucket_metadata.rs` | List buckets and run bucket/replication/IAM metadata migrations. | Reads bucket names, migrates bucket metadata, initializes replication resync, migrates IAM config, and initializes bucket metadata system. | Bucket list and replication resync are fatal on error; metadata migration calls are non-fatal in this path. | Storage remains ready; IAM not yet ready | +| `RUN-017` | `rustfs/src/startup_iam.rs` | Bootstrap IAM inline or defer recovery. | Initializes IAM when possible; otherwise starts the deferred IAM recovery path through `startup_iam`. | Fatal only when `bootstrap_or_defer_iam_init` returns an unrecoverable error. | Inline success marks `IamReady`; deferred mode publishes `IamReady` later from the recovery task | +| `RUN-018` | `rustfs/src/startup_auth.rs` | Initialize Keystone and OIDC auth integrations. | Loads Keystone env config and initializes OIDC providers. | Keystone config parse is fatal; Keystone runtime init failure is non-fatal; OIDC init failure is non-fatal. | None | +| `RUN-019` | `rustfs/src/startup_notification.rs` | Add bucket notification config and initialize notification system. | Adds bucket notification configuration and publishes the global notification system. | Notification config add is non-fatal in this path; global notification init is fatal on error. | None | +| `RUN-020` | `rustfs/src/startup_background.rs` | Create AHM cancellation token and initialize heal manager when scanner or heal is enabled. | Creates AHM cancellation token and starts heal manager for heal/scanner workflows. | Heal manager init is fatal when enabled. | None | +| `RUN-021` | `rustfs/src/startup_observability.rs` | Print server info, init update check, allocator reclaim, metrics, memory observability, and auto-tuner. | Starts informational/update/memory/metrics background tasks when enabled. | Non-fatal in this path. | None | | `RUN-022` | `rustfs/src/startup_lifecycle.rs` and `rustfs/src/startup_iam.rs` | Log successful startup and publish full readiness for inline IAM. | Logs version/address, checks runtime readiness, marks `FullReady`, and sets service state to `Ready` when IAM was ready inline. | Fatal if runtime readiness is not reached within the startup wait. | Marks `FullReady` only for inline IAM here | | `RUN-023` | `rustfs/src/startup_lifecycle.rs` | Publish global init time and start data scanner when enabled. | Sets global init time and starts scanner after the successful-startup log. | Scanner start is non-fatal in this path. | Full readiness may already be published or may await deferred IAM recovery | | `RUN-024` | `rustfs/src/startup_lifecycle.rs` | Wait for shutdown signal. | Blocks the main task until a shutdown signal is received. | Non-fatal. | Runtime remains in its current readiness state | @@ -72,7 +72,7 @@ new startup semantics. | `STOP-001` | `rustfs/src/startup_shutdown.rs` | Cancel runtime token and move service state to `Stopping`. | Notifies cancellation-aware background tasks. | Non-fatal. | Service state moves to `Stopping`; readiness stages are not cleared here | | `STOP-002` | `rustfs/src/startup_shutdown.rs` | Stop scanner/background services and AHM services according to enable flags. | Calls ECStore background shutdown and heal/scanner shutdown helpers. | Non-fatal in this path. | No readiness-stage change | | `STOP-003` | `rustfs/src/startup_optional_runtime_sidecars.rs` | Plan optional runtime shutdown and log stopping state for FTP/FTPS/WebDAV/SFTP protocol servers. | Collects protocol shutdown handles. | Non-fatal in this path. | No readiness-stage change | -| `STOP-004` | `rustfs/src/startup_shutdown.rs`, `rustfs/src/startup_runtime_hooks.rs`, and `rustfs/src/startup_profiling.rs` | Stop event notifier, audit system, and profiling tasks. | Stops notifier and profiling tasks; audit stop failures are logged. | Non-fatal in this path. | No readiness-stage change | +| `STOP-004` | `rustfs/src/startup_shutdown.rs`, `rustfs/src/startup_runtime_hooks.rs` | Stop event notifier, audit system, and profiling tasks. | Stops notifier and profiling tasks; audit stop failures are logged. | Non-fatal in this path. | No readiness-stage change | | `STOP-005` | `rustfs/src/startup_shutdown.rs` and `rustfs/src/startup_optional_runtime_sidecars.rs` | Stop S3 and console HTTP servers, signal and wait for optional protocol shutdowns, then mark service state `Stopped`. | HTTP shutdown happens after notifier/audit/profiling shutdown in current order. | Join failures are logged by shutdown handles; this path does not return errors. | Service state moves to `Stopped`; readiness stages are not cleared here | ## Migration Rules diff --git a/docs/operations/tier-ilm-debugging.md b/docs/operations/tier-ilm-debugging.md new file mode 100644 index 000000000..71dc1e46d --- /dev/null +++ b/docs/operations/tier-ilm-debugging.md @@ -0,0 +1,94 @@ +# Tier / ILM Transition Debugging Guide + +How to debug lifecycle tiering (hot → cold transition) issues: inspecting +`xl.meta`, tracing the versionId sent to the remote tier, and known pitfalls. + +> Code map (post `#3929` layout): +> +> | Concern | Location | +> |---------|----------| +> | ILM actions (`transition_object`, `expire_transitioned_object`, `get_transitioned_object_reader`, `gen_transition_objname`) | `crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs` | +> | Erasure-set transition/restore entry points | `crates/ecstore/src/set_disk/` and `crates/ecstore/src/store/` | +> | `WarmBackend` trait (put/get/remove/in_use) | `crates/ecstore/src/services/tier/warm_backend.rs` | +> | Per-provider tier backends (S3, MinIO, GCS, Azure, …) | `crates/ecstore/src/services/tier/warm_backend_*.rs` | +> | Remote-tier sweep (`delete_object_from_remote_tier`) | `crates/ecstore/src/bucket/lifecycle/tier_sweeper.rs` | +> | `ObjectInfo` / `TransitionedObject` types | `crates/ecstore/src/object_api/types.rs` | +> | `FileMeta` / `FileInfo` / version metadata | `crates/filemeta/src/` | +> | Dual-key internal metadata helpers (`insert_bytes` / `get_bytes`) | `crates/utils/src/http/metadata_compat.rs` | + +## Metadata key conventions + +Internal metadata is stored under **both** `x-rustfs-internal-` and +`x-minio-internal-` for MinIO interoperability. `get_bytes` prefers +the RustFS key and falls back to the MinIO key. + +| Suffix | Meaning | +|--------|---------| +| `transition-status` | `"complete"` when tiered | +| `transitioned-object` | tier key path (stored **without** the tier prefix; `get_dest` adds it) | +| `transitioned-versionID` | S3 version_id returned by tier PUT (16 raw UUID bytes, or absent) | +| `transition-tier` | tier name | +| `tier-free-versionID` | delete-marker version for free-version sweep | + +Reading binary values must reject empty/malformed/nil values (regression +covered in `crates/filemeta/src/filemeta/version.rs` tests): + +```rust +get_bytes(&self.meta_sys, SUFFIX_TRANSITIONED_VERSION_ID) + .and_then(|v| Uuid::from_slice(v.as_slice()).ok()) + .filter(|u| !u.is_nil()) +// None for: absent key, wrong-length bytes, nil UUID +``` + +`transition_version_id == None` means the tier bucket is unversioned; the +GET/DELETE against the tier must then send **no** `versionId` parameter. +A nil UUID (`00000000-…`) sent as `?versionId=` causes `NoSuchVersion`. + +## Inspect xl.meta directly + +```bash +cargo build -p rustfs-filemeta --example dump_fileinfo +./target/debug/examples/dump_fileinfo /srv/rustfs/data/disk0/{bucket}/{object}/xl.meta +# Shows: transition_status, transition_tier, transitioned_obj, transition_ver_id +``` + +- `transition_ver_id: ` → no versionId will be sent to the tier + (correct for a non-versioned tier bucket). +- `transition_ver_id: ` → that UUID will be sent as `?versionId=`. + +There is one `xl.meta` per erasure shard disk +(`{disk}/{bucket}/{object}/xl.meta`); all shards of a healthy object should +be identical. `dump_versions` (same crate) lists every version in a file. + +## Trace the versionId at runtime + +```bash +RUST_LOG=rustfs_ecstore::bucket::lifecycle=debug rustfs … +``` + +- `fetching transitioned object from tier` — DEBUG, before the tier request. +- `tier GET failed` — ERROR, includes `tier_version_id`. + +If both `x-rustfs-internal-transitioned-versionID` and +`x-minio-internal-transitioned-versionID` are the **empty string**, the object +was transitioned to a non-versioned tier bucket and no versionId must be sent. + +## Known open issue: expire/GET race + +`expire_transitioned_object` deletes the remote tier version **before** +deleting local metadata. A concurrent GET between those two steps reads a +valid stored version_id whose remote version is already gone → +`NoSuchVersion`. The proper fix is to delete local metadata first (making the +object unreachable), then the remote tier version. + +## Historical fixes (for context, already merged) + +- Nil-UUID versionId sent to tier (`NoSuchVersion`): reading code used + `Uuid::from_slice(..).unwrap_or_default()`, converting an empty metadata + value into `Uuid::nil()`. Fixed by the `and_then`/`filter` pattern above. +- `warm_backend_s3sdk` ignored the remote version and range options on + GET/DELETE. +- `copy_object` returned 501 (`NotImplemented`) for tiered objects on + self-copy with `--storage-class`, blocking de-tiering via + `mc cp --storage-class STANDARD obj obj`. Fixed by writing the + tier-fetched `put_object_reader` back through `put_object`. diff --git a/docs/architecture/migration-progress.md b/docs/superpowers/plans/migration-progress.md similarity index 99% rename from docs/architecture/migration-progress.md rename to docs/superpowers/plans/migration-progress.md index da95ed759..a7721b79b 100644 --- a/docs/architecture/migration-progress.md +++ b/docs/superpowers/plans/migration-progress.md @@ -1,3 +1,8 @@ +> **Archived implementation plan/tracker** — moved from `docs/architecture/` (2026-07). +> Kept for history; not maintained. File paths inside may reflect the pre-#3929 +> module layout (e.g. `crates/ecstore/src/rebalance.rs` is now +> `crates/ecstore/src/store/rebalance.rs`; `set_disk.rs` is now `set_disk/`). + # Architecture Migration Progress Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` blocked. diff --git a/docs/architecture/rebalance-decommission-followup-review-plan.md b/docs/superpowers/plans/rebalance-decommission-followup-review-plan.md similarity index 99% rename from docs/architecture/rebalance-decommission-followup-review-plan.md rename to docs/superpowers/plans/rebalance-decommission-followup-review-plan.md index cebed5b81..514972ca8 100644 --- a/docs/architecture/rebalance-decommission-followup-review-plan.md +++ b/docs/superpowers/plans/rebalance-decommission-followup-review-plan.md @@ -1,3 +1,8 @@ +> **Archived implementation plan/tracker** — moved from `docs/architecture/` (2026-07). +> Kept for history; not maintained. File paths inside may reflect the pre-#3929 +> module layout (e.g. `crates/ecstore/src/rebalance.rs` is now +> `crates/ecstore/src/store/rebalance.rs`; `set_disk.rs` is now `set_disk/`). + # Rebalance and Decommission Follow-up Review Implementation Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. diff --git a/docs/architecture/rebalance-decommission-implementation-plan-index.md b/docs/superpowers/plans/rebalance-decommission-implementation-plan-index.md similarity index 93% rename from docs/architecture/rebalance-decommission-implementation-plan-index.md rename to docs/superpowers/plans/rebalance-decommission-implementation-plan-index.md index f3a7262f5..19a73692d 100644 --- a/docs/architecture/rebalance-decommission-implementation-plan-index.md +++ b/docs/superpowers/plans/rebalance-decommission-implementation-plan-index.md @@ -1,3 +1,8 @@ +> **Archived implementation plan/tracker** — moved from `docs/architecture/` (2026-07). +> Kept for history; not maintained. File paths inside may reflect the pre-#3929 +> module layout (e.g. `crates/ecstore/src/rebalance.rs` is now +> `crates/ecstore/src/store/rebalance.rs`; `set_disk.rs` is now `set_disk/`). + # Rebalance and Decommission Implementation Plan Index > This index is based on `docs/architecture/rebalance-decommission-remediation-plan.md`. The remediation scope is intentionally split into smaller implementation plans because the fixes touch independent risk areas: object-version safety, distributed operation semantics, data movement internals, and operational hardening. diff --git a/docs/architecture/rebalance-decommission-phase1-safety-plan.md b/docs/superpowers/plans/rebalance-decommission-phase1-safety-plan.md similarity index 98% rename from docs/architecture/rebalance-decommission-phase1-safety-plan.md rename to docs/superpowers/plans/rebalance-decommission-phase1-safety-plan.md index aca9bd8e6..bc6707806 100644 --- a/docs/architecture/rebalance-decommission-phase1-safety-plan.md +++ b/docs/superpowers/plans/rebalance-decommission-phase1-safety-plan.md @@ -1,3 +1,8 @@ +> **Archived implementation plan/tracker** — moved from `docs/architecture/` (2026-07). +> Kept for history; not maintained. File paths inside may reflect the pre-#3929 +> module layout (e.g. `crates/ecstore/src/rebalance.rs` is now +> `crates/ecstore/src/store/rebalance.rs`; `set_disk.rs` is now `set_disk/`). + # Rebalance and Decommission Phase 1 Safety Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. diff --git a/docs/architecture/rebalance-decommission-phase2-data-movement-plan.md b/docs/superpowers/plans/rebalance-decommission-phase2-data-movement-plan.md similarity index 97% rename from docs/architecture/rebalance-decommission-phase2-data-movement-plan.md rename to docs/superpowers/plans/rebalance-decommission-phase2-data-movement-plan.md index 0e4ccb0e8..ef631683e 100644 --- a/docs/architecture/rebalance-decommission-phase2-data-movement-plan.md +++ b/docs/superpowers/plans/rebalance-decommission-phase2-data-movement-plan.md @@ -1,3 +1,8 @@ +> **Archived implementation plan/tracker** — moved from `docs/architecture/` (2026-07). +> Kept for history; not maintained. File paths inside may reflect the pre-#3929 +> module layout (e.g. `crates/ecstore/src/rebalance.rs` is now +> `crates/ecstore/src/store/rebalance.rs`; `set_disk.rs` is now `set_disk/`). + # Rebalance and Decommission Phase 2 Data Movement Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. diff --git a/docs/architecture/rebalance-decommission-phase3-hardening-plan.md b/docs/superpowers/plans/rebalance-decommission-phase3-hardening-plan.md similarity index 97% rename from docs/architecture/rebalance-decommission-phase3-hardening-plan.md rename to docs/superpowers/plans/rebalance-decommission-phase3-hardening-plan.md index dda5ca2f3..d15b67fa2 100644 --- a/docs/architecture/rebalance-decommission-phase3-hardening-plan.md +++ b/docs/superpowers/plans/rebalance-decommission-phase3-hardening-plan.md @@ -1,3 +1,8 @@ +> **Archived implementation plan/tracker** — moved from `docs/architecture/` (2026-07). +> Kept for history; not maintained. File paths inside may reflect the pre-#3929 +> module layout (e.g. `crates/ecstore/src/rebalance.rs` is now +> `crates/ecstore/src/store/rebalance.rs`; `set_disk.rs` is now `set_disk/`). + # Rebalance and Decommission Phase 3 Hardening Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. diff --git a/docs/architecture/rebalance-decommission-post-remediation-review-plan.md b/docs/superpowers/plans/rebalance-decommission-post-remediation-review-plan.md similarity index 96% rename from docs/architecture/rebalance-decommission-post-remediation-review-plan.md rename to docs/superpowers/plans/rebalance-decommission-post-remediation-review-plan.md index ae8b94516..eaf195955 100644 --- a/docs/architecture/rebalance-decommission-post-remediation-review-plan.md +++ b/docs/superpowers/plans/rebalance-decommission-post-remediation-review-plan.md @@ -1,3 +1,8 @@ +> **Archived implementation plan/tracker** — moved from `docs/architecture/` (2026-07). +> Kept for history; not maintained. File paths inside may reflect the pre-#3929 +> module layout (e.g. `crates/ecstore/src/rebalance.rs` is now +> `crates/ecstore/src/store/rebalance.rs`; `set_disk.rs` is now `set_disk/`). + # Rebalance and Decommission Post-Remediation Review > **Status:** Updated for branch `cxymds/rebalance-decommission-remediation`. diff --git a/docs/architecture/rebalance-decommission-remediation-plan.md b/docs/superpowers/plans/rebalance-decommission-remediation-plan.md similarity index 98% rename from docs/architecture/rebalance-decommission-remediation-plan.md rename to docs/superpowers/plans/rebalance-decommission-remediation-plan.md index 4d60b041f..cfd2b9300 100644 --- a/docs/architecture/rebalance-decommission-remediation-plan.md +++ b/docs/superpowers/plans/rebalance-decommission-remediation-plan.md @@ -1,3 +1,8 @@ +> **Archived implementation plan/tracker** — moved from `docs/architecture/` (2026-07). +> Kept for history; not maintained. File paths inside may reflect the pre-#3929 +> module layout (e.g. `crates/ecstore/src/rebalance.rs` is now +> `crates/ecstore/src/store/rebalance.rs`; `set_disk.rs` is now `set_disk/`). + # Rebalance and Decommission Remediation Plan > This document turns `docs/architecture/expert-review-analysis.md` into an actionable remediation backlog. It is intentionally split into independent fix blocks so each item can be analyzed, assigned, implemented, and verified separately. diff --git a/docs/architecture/site-replication-hardening-pr.md b/docs/superpowers/plans/site-replication-hardening-pr.md similarity index 96% rename from docs/architecture/site-replication-hardening-pr.md rename to docs/superpowers/plans/site-replication-hardening-pr.md index 8fba3d134..ea22a64c5 100644 --- a/docs/architecture/site-replication-hardening-pr.md +++ b/docs/superpowers/plans/site-replication-hardening-pr.md @@ -1,3 +1,8 @@ +> **Archived implementation plan/tracker** — moved from `docs/architecture/` (2026-07). +> Kept for history; not maintained. File paths inside may reflect the pre-#3929 +> module layout (e.g. `crates/ecstore/src/rebalance.rs` is now +> `crates/ecstore/src/store/rebalance.rs`; `set_disk.rs` is now `set_disk/`). + ## Related Issues N/A diff --git a/scripts/check_doc_paths.sh b/scripts/check_doc_paths.sh new file mode 100755 index 000000000..ac4d1fcd4 --- /dev/null +++ b/scripts/check_doc_paths.sh @@ -0,0 +1,69 @@ +#!/usr/bin/env bash +# +# Fail when an agent-instruction or architecture document references a +# repository file path that no longer exists. Keeps CLAUDE.md / AGENTS.md / +# ARCHITECTURE.md / docs/architecture honest after refactors move code. +# +# Checked files: all tracked AGENTS.md, CLAUDE.md, ARCHITECTURE.md, and +# docs/architecture/*.md (archived plans under docs/superpowers/plans are +# intentionally NOT checked — they are historical). +# +# A reference is any token starting with crates/, rustfs/, scripts/, docs/, +# .github/ or .config/ that ends in a known file extension. Tokens containing +# globs or placeholders, extensionless tokens (HTTP routes, directory +# references, org/repo shorthands), and lines containing URLs are skipped. + +set -euo pipefail + +ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +cd "$ROOT_DIR" + +FAILURES=0 + +doc_files() { + git ls-files 'AGENTS.md' '*/AGENTS.md' 'CLAUDE.md' 'ARCHITECTURE.md' 'docs/architecture/*.md' +} + +check_file() { + local doc="$1" + # Path-like tokens; strip surrounding backticks/brackets/punctuation later. + # Lines with URLs are skipped wholesale: path fragments inside links are + # external, not repo paths. + # grep exits 1 on no matches; that must not kill the script under pipefail. + { grep -vE 'https?://' "$doc" || true; } \ + | { grep -oE '(crates|rustfs|scripts|docs|\.github|\.config)/[A-Za-z0-9_./-]+' || true; } \ + | sed -e 's/[.,;:)]*$//' -e 's:/$::' \ + | sort -u \ + | while IFS= read -r ref; do + case "$ref" in + (*'*'*|*'<'*|*'>'*|*'{'*|*'}'*|*'…'*) continue ;; + esac + # Only verify references with a known file extension. Extensionless + # tokens are too often HTTP routes (rustfs/admin/v3/...), org/repo + # shorthands (rustfs/backlog), or glob prefixes — not repo paths. + case "$ref" in + (*.rs|*.md|*.sh|*.toml|*.yml|*.yaml|*.mak|*.proto) ;; + (*) continue ;; + esac + if [[ ! -e "$ref" ]]; then + printf 'stale path reference: %s -> %s\n' "$doc" "$ref" >&2 + echo "FAIL" >> "$FAIL_MARKER" + fi + done +} + +FAIL_MARKER="$(mktemp)" +trap 'rm -f "$FAIL_MARKER"' EXIT + +while IFS= read -r doc; do + [[ -f "$doc" ]] && check_file "$doc" +done < <(doc_files) + +if [[ -s "$FAIL_MARKER" ]]; then + count="$(wc -l < "$FAIL_MARKER" | tr -d ' ')" + printf '\n%s stale doc path reference(s) found.\n' "$count" >&2 + printf 'Fix the doc to match the current tree, or update the moved code path.\n' >&2 + exit 1 +fi + +printf 'doc path check passed\n'