mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-03 03:47:42 +00:00
da531c8a9719eba696fe7d7eb045bd9735586dc6
7 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Zhengchao An
|
9644064e57 |
docs(kms): define the bulk object rekey job contract (#5605)
* docs(kms): add the object-side bulk rekey job contract Define the design contract for the bulk DEK re-wrap job before any of it is built: work unit granularity, idempotency model, failure semantics, the exclusion list, the metadata write constraints, and the ownership model. The job re-wraps envelopes only and never rewrites object bodies, and it never destroys a superseded key version: a half-finished run is a fully serviceable state precisely because the old version still decrypts, so folding destruction into the job would turn a resumable action into irreversible loss on partial failure. Records two positions that diverge from the originating request. Pause is not provided; cancel plus cursor restart plus rate control covers what pause is actually asked for, and none of the seven existing long-job frameworks has a pause state. And the KEK version a given envelope was sealed under is not observable today, from object metadata or from the decrypt response, so recognizing the target state requires the re-wrap primitive to record a version witness - stated here as the one interface both sides must agree on. Refs rustfs/backlog#1642 (part of rustfs/backlog#1562) * docs(kms): correct how the wrapping KEK version is recovered The first draft claimed the wrapping KEK version was not observable at all, and built the idempotent-skip and completion-evidence conclusions on that. It is observable for every backend that rotates, so that framing was wrong. The sealed blob under x-rustfs-encryption-key is the base64 of the backend ciphertext, which is DataKeyEnvelope JSON; the read path in sse.rs already discriminates on it via is_data_key_envelope. Vault KV2 records the version in the envelope's master_key_version, and Transit leaves that field None on purpose because its ciphertext self-describes as vault:vN:. Local and Static hardcode None because neither rotates. Only AWS is genuinely opaque. So the skip is achievable, and the honest statement is its cost: a base64 decode plus a JSON parse per scanned work unit, which belongs in the rate budget. AWS becomes a scope-admission refusal instead of a silent every-object rewrite on re-run. Two traps replace the old over-broad claim. A bare None means three different things across backends, so version extraction must be dispatched by backend. And Local omits the version precisely because rotation is rejected there, which couples it to backlog#1565: whichever change gives Local a rotation history must start recording the version in the same change, or Local joins AWS in the unreadable column. The requirement left on the re-wrap primitive shrinks accordingly, from "record a version" to exposing one backend-dispatched accessor and reporting "already at target state" as a distinct outcome. Refs rustfs/backlog#1642 (part of rustfs/backlog#1562) |
||
|
Zhengchao An
|
2c113542f8 |
docs(architecture): normative erasure-coding algorithm and on-disk compatibility contract (#4999)
* docs(architecture): add normative erasure-coding algorithm and on-disk compatibility contract Adds docs/architecture/erasure-coding.md as the source of truth for how RustFS erasure-codes, stores, reads, reconstructs, and heals user data, and the on-disk (xl.meta) / decode compatibility contract every future change must preserve. Grounded in the baseline (main) implementation with file:line anchors; cross-links (does not duplicate) the existing placement, MinIO-format-compat, layout-boundary, decommission, and tier-ILM docs, and the AGENTS.md cross-cutting invariants. Covers: Reed-Solomon over GF(2^8) modern vs GF(2^16) legacy backends and how each is selected; pool/set/drive geometry with the 2..=16 set-size and per-pool parity invariants; the key-derived distribution permutation (1..=N); 1 MiB block size and the modern/legacy shard-size formulas; HighwayHash256S interleaved bitrot layout; the full xl.meta container/header/version-body schema and internal dual-key convention; write/read/heal quorum rules; and, newly codified as a first-class contract, the decode-tolerance invariants (nil-UUID/epoch-mod_time to None, skip unknown fields, hard-guard only length-critical arrays, tolerate the negative actual_size compressed sentinel, tolerate a malformed transitioned-versionID). Linked from the architecture README under "Contracts & invariants". Docs-only; check_doc_paths.sh passes. * docs(architecture): anchor erasure spec by symbol name, not line numbers Line numbers rot as code changes and are not validated by check_doc_paths.sh, so they would silently mislead the very code changes this normative spec is meant to guide. Replace all file:line citations with file-path + symbol-name references (functions/consts/types are greppable and rename only on deliberate changes; format byte offsets are kept). Add an explicit "references work here" note and a §13 rule that governed changes must update this spec in the same PR. * docs(architecture): correct erasure spec after multi-expert adversarial review Four independent adversarial reviewers fact-checked every claim against the code. Fixes: - CRITICAL (found independently by two reviewers): §1 mislabeled the GF(2^16) reed-solomon-simd "legacy" backend as the reader for "older MinIO-lineage format". It is the opposite — that backend serves RustFS's own older main-branch (rmp_serde, uses_legacy_checksum) objects; MinIO-migrated data uses the same rs-vandermonde GF(2^8) scheme and is decoded by the modern backend. The old wording contradicted §1's own MinIO-interop line, §12, minio-file-format-compat.md, and the source comments, and would have misled the highest-stakes decode-routing decision. - §2.1/§12: set size 2..=16 holds for multi-drive layouts; single-drive deployments run at N=1 (parity 0) outside is_valid_set_size. - §2.2: validate_parity_inner enforces parity <= N/2 only for N > 2 (user storage-class parity flows through it); the standalone validate_parity is unconditional but only applied to the resolved default. - §6.2/§12: header format is dispatched by header_ver; array length (4/5/7) is a per-version validation, not the discriminator. - §6.3: the part-array length guard applies on the all_parts decode path. - §6.5: get_bytes matches only the two canonical lowercase keys; only is_internal_key/get_str are case-insensitive. - §6.5: transitioned-versionID — state the load-bearing invariant (non-16-byte decodes to None, never fatal); string-form recovery is optional, and transitioned-xl.meta interop is out of scope. - §11: typo RustSF -> RustFS. All other claims across §1-§14 were verified accurate against code (distribution formula, quorum formulas, on-disk key set/endianness/markers, decode-tolerance invariants, version anchors, standard references). |
||
|
Zhengchao An
|
f40abbb9f2 |
docs(architecture): unify per-object generation authority (#4912)
Add the shared design/contract document for backlog #1326: a single per-object generation authority (the #1312 fencing epoch) that spans commit fencing, read lease, prepared pool read, quota reservation, and old-dir GC. Pins the five cross-cutting constraints once - RPC signature binding with server-side nonce enforcement, xl.meta encoding contract (no meta_ver bump, no positional FileInfo field, internal metadata map under the dual-key contract), proto3 optional presence, mixed-version fallback direction, and cluster-level capability negotiation - so the implementation sub-issues follow them rather than each re-deciding. No product code changes. |
||
|
Zhengchao An
|
c4c198670d |
docs: remove agent-generated planning docs and forbid committing them (#4771)
docs: remove agent-generated planning docs, forbid committing them Delete one-shot planning/progress artifacts that were checked into the tree: the 14 superpowers plan/tracker docs under docs/superpowers/plans/, plus issue-scoped implementation plans, optimization conclusions, and dated benchmark-result snapshots under docs/ (issue-4003 ListObjectsV2 plans, get-small-file conclusion, issue824/issue829 benchmark results, issue-713 >1GiB GET baseline summary and ops guide). Codify the rule so they do not come back: - .gitignore drops the docs/superpowers whitelist, so anything new under docs/ stays ignored unless force-added. - AGENTS.md gains an explicit 'do not commit planning-type documents' rule scoping version control to the durable architecture/operations/testing sets. - docs/architecture/README.md, overview.md, arch-checks SKILL.md, and check_doc_paths.sh drop their references to the removed archive. |
||
|
Zhengchao An
|
f63af3df63 |
chore: retire completed-migration scaffolding, wire orphaned boundary check (#4719)
The ecstore/global-state migrations are done (backlog#815, #939, #1052 all closed). Review of every migration-era test/gate measure found three things actually retirable or broken — everything else is a live anti-regression guard and stays. Remove: - scripts/check_metrics_migration_refs.sh — guards a migration that finished: rustfs_metrics:: has zero hits, the metrics crate no longer exists, and the script was never wired into CI or make (only reference was one line in config-model-boundary-adr.md, also removed). - crates/obs init_metrics_collectors — the "backward-compatible alias kept during migration" the removed script was guarding. Zero callers; pure delegate to init_metrics_runtime. Archive (docs/superpowers/plans/, continuing the 2026-07 convention, with the standard archived banner): - startup-timeline.md, scheduler-baseline.md, profiling-numa-capability-inventory.md, kms-development-defaults-inventory.md — one-shot snapshots whose only consumer is the already-archived migration-progress ledger (their same-dir links there start resolving again after the move); zero script pins; fed the closed backlog#660/#665 architecture-review ledger. Fixed the one outbound link (startup-timeline -> readiness-matrix) that the move would have broken — check_doc_paths.sh deliberately does not scan plans/, so nothing else would have caught it. Wire (found orphaned by the same review): - scripts/check_extension_schema_boundaries.sh guards a live contract crate but was never invoked anywhere. Add lint-fmt.mak target, include in pre-commit/pre-pr/dev-check, add ci.yml Quick Checks step (job already installs ripgrep), sync the CONTRIBUTING.md enumerated list, and harden the script against a silently-passing rg probe when src/ is missing. Keep (verified live, documented so the next cleanup pass does not repeat this analysis): - scripts/check_architecture_migration_rules.sh — added a header stating it is a permanent boundary guard, not retirable migration scaffolding; 'migration' in the name is historical. - check_migration_gate_count.sh + floor, delete-marker e2e proof, all pinned docs, compat-cleanup-register sync, remaining inventories (referenced by live docs). Verification: all 7 guard scripts pass, actionlint clean, cargo check --workspace (excl e2e) clean, cargo fmt --check clean. Adversarially reviewed by two independent skeptic passes; their 7 findings (alias left behind, broken outbound link, missing banners, wrong backlog attribution, CONTRIBUTING drift, rg exit-2 hole, missing header rationale) are all folded in. |
||
|
Zhengchao An
|
0271abc14d |
docs: add MinIO compatibility router + file-format docs (#4328)
Add two durable architecture references grounded in current code: - minio-rustfs-router-compatibility.md: MinIO cmd/api-router.go (S3 object/bucket) and cmd/admin-router.go (admin /v3, /v4) vs RustFS implementation status, with per-row landing points and a gaps-only checklist of still-missing admin endpoints (profiling, healthinfo, LDAP IDP config, replication diff, MRF, batch, locks, speedtest, log stream, top, trace). - minio-file-format-compat.md: xl.meta (meta_ver 3 write, <=3 read, XL2 magic, rs-vandermonde, HighwayHash256 bitrot, inline data) and .metadata.bin bucket-metadata interop matrix for the backlog#580 items, plus old-RustFS -> new migration and a phased plan. Cross-links s3-compatibility-matrix.md and admin-route-action-snapshot.md and indexes both docs in docs/architecture/README.md. Refs rustfs/backlog#596 rustfs/backlog#603 rustfs/backlog#580 |
||
|
Zhengchao An
|
d1db9a10cd |
chore(docs): refresh agent docs, guard doc paths, archive plans (#4203)
Agent-instruction and architecture docs had drifted from the code: - CLAUDE.md: slim to commands + pointers; fix wrong claim that `make pre-commit` is the full pre-PR gate (that is `make pre-pr`); drop stale pre-#3929 file paths and merged bug narratives - AGENTS.md: drop dead `rust-refactor-helper` skill rule and the hand-maintained (already stale) scoped-AGENTS index; link architecture docs from Sources of Truth - .github/AGENTS.md: replace the outdated copied CI command matrix with a pointer to ci.yml - crates/AGENTS.md: merge duplicated Testing sections - ARCHITECTURE.md: resolve the utils->config contradiction (edges are removed), mark volatile counts as snapshots, fix a bad path - docs/architecture: add README router; move one-shot plans/trackers (rebalance-decommission phases, migration-progress ledger, PR template) to docs/superpowers/plans with archive headers; fix stale source paths in kept inventories (core/sets.rs, core/pools.rs, store/mod.rs, startup_* split from #3671) - docs/operations/tier-ilm-debugging.md: extracted tier debugging playbook with corrected paths - scripts/check_doc_paths.sh: new guard failing pre-commit/pre-pr when instruction/architecture docs reference nonexistent file paths - .claude/skills: add tier-debug and arch-checks repo skills; .gitignore now keeps .claude/skills and docs/operations committable Verification: ./scripts/check_doc_paths.sh, ./scripts/check_architecture_migration_rules.sh (both pass) Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |