mirror of
https://github.com/rustfs/rustfs.git
synced 2026-11-01 00:32:21 +00:00
Chris An
7a3bf8cdbd
* fix(ecstore): select the latest version at read quorum like MinIO GET and HEAD escalated latest-version selection to write quorum when enough metadata answered and skipped a newer version below write quorum on degraded reads. LIST and heal already pick the newer side of a split write at read quorum, so such keys listed but answered 503 until heal. Select with MinIO's getObjectFileInfo rule: the most common mod time that reaches the object's data blocks wins, ties go to the newer time, and one content identity at that time must reach the same quorum. Two identities tied at the selected time fail instead of resolving by hash order, and the selected identity's version-stack shape follows disk order. * fix(ecstore): resolve selection ties by disk order and guard writes Review of the read-quorum selection found three gaps: - Rejecting a same-time half split made explicit-version reads of a partially applied tagging update fail and blocked the retry that repairs it. Resolve ties to the identity on the lowest-index disk, for reads and heal alike, instead of rejecting or using map order. - Delete markers and empty versions can lower the common read quorum below the selected version's data blocks, so HEAD could report a version GET cannot decode. Require that many copies, as MinIO's getLatestFileInfo does. - A selection can now return exactly read-quorum disks, and partial metadata writes are not rolled back. Refuse an update before writing when its target disks cannot reach write quorum. * fix(ecstore): count identity copies in the data-blocks guard The first-candidate shortcut counts byte-identical copies, so copies that differ only in identity-neutral fields such as replication status were missing from the data-blocks check. When the shortcut's count falls short, count every copy of the selected content identity before refusing the version. * fix(ecstore): keep commit-proof selection for system-bucket reads Pool metadata's PREPARE/COMMIT recovery and identity epochs treat an ordinary read of pool.bin as proof that the record committed. Under the read-quorum rule, a COMMIT that reached only half of a 2+2 set was served as committed, and the pool_meta_native_physical_* integration tests failed. Route reads through select_fileinfo_for_read: S3 buckets keep MinIO's read-quorum rule, while is_meta_bucketname reads use the previous algorithm unchanged as select_committed_fileinfo, so a split stays unreadable until native recovery resolves it. * fix(ecstore): serve the latest readable version past dangling versions A failed versioned PUT whose rollback also failed can leave the new version on top of fewer drives than its data blocks. Reading only each drive's top version then left the committed version below it short of read quorum, so GET and HEAD of the latest version answered 503 although that version is intact in every stack. MinIO resolves the stacks in pickLatestQuorumFilesInfo. When every drive answered and an S3 latest read still cannot select a version, walk the newest version headers of each stack through exact-version reads: serve the readable version with the newest selected modification time, skip versions whose exact read proves them dangling (the NoSuchVersion an explicit GET reports) and queue heal for them, and keep the read-quorum error on any other outcome. Candidates are ordered by each version's newest modification time across drives, and at most eight exact reads are spent. Test PUTs now wait for every drive so tests that edit one drive's metadata no longer race the PUT's background tail writes. * fix(ecstore): resolve dangling latest versions in metadata updates A failed versioned PUT whose rollback also failed can leave the new version on fewer drives than its data blocks. GET and HEAD of the latest version already look past such versions, but read-before-write callers without a version id still answered 503: object tagging read each drive's top version with the quorum-error read, and put_object_metadata and the CopyObject source read selected from read_all_xl, whose merged latest version is missing on the other drives, so object_quorum_from_meta failed before selection. Object tagging now reads through a variant that runs the same walk past dangling versions as a latest GET while keeping the quorum error for a dangling key. put_object_metadata and the CopyObject source try the walk when their quorum or selection step fails with read quorum, before the existing dangling cleanup could report a key with a committed version absent. A version-only copy takes the served version's own copies and write quorum. The walk still applies only to S3 buckets, latest reads, and sets where every drive answered, and the update still targets only the served version's disks, so a version below write quorum is refused before writing. * fix(ecstore): refuse version-only copies below write quorum up front Review of the dangling-latest metadata updates found three gaps: - A version-only CopyObject wrote the new version and reverted on a write-quorum miss; refuse it before writing when the served version's disks cannot reach its write quorum, like update_object_meta_with_opts. - Replication status write-back, including the zero-write ValidateOnly mode, now keeps its read-quorum error and never walks past dangling versions, since the walk queues heal for the versions it skips. - The erasure-coding doc overstated that these callers never report a dangling key as absent; put_object_metadata and CopyObject keep their existing dangling cleanup when the walk finds no readable version.
Architecture Documentation
Use this when: you need the contract, invariant, or boundary rule that governs a change, and you want the one document that owns it.
Source of truth: the code and the guards. scripts/check_architecture_migration_rules.sh enforces the CI-anchored documents below; scripts/check_doc_paths.sh fails the pre-commit gate when any doc under docs/ cites a repository path that no longer exists.
Two rules keep this directory healthy:
- Durable reference only. One-shot plans, task trackers, dated analyses, status snapshots, and PR-scoped notes do not belong in the repository; keep them in the issue tracker or a local worktree and delete them when the work closes.
- No copies of other sources of truth. Crate lists come from
Cargo.toml, CI steps from.github/workflows/, code structure from the code. Cite a file path plus a symbol name, never a line number, and never paste counts or tables that a command can regenerate.
Every document starts with a **Use this when:** line so an agent can decide in one glance whether to read further. The index below repeats those lines.
CI-anchored core
Required headings and strings in these files are asserted by scripts/check_architecture_migration_rules.sh; rename a heading only together with the guard.
| Document | Use this when |
|---|---|
| crate-boundaries.md | you add a crate dependency, move code across crates, touch a storage_api.rs boundary file, or need the change-type vocabulary the architecture guard enforces |
| runtime-lifecycle.md | moving or reordering anything in rustfs/src/startup_*.rs, changing readiness publication, or touching shutdown ordering |
| readiness-matrix.md | changing what a request surface does before storage or IAM is ready, changing probe semantics, or adding a runtime dependency that readiness must wait for |
| storage-control-data-plane.md | adding a storage API surface, a cluster read model, or a background-service status/reconcile surface, and you need to know which layer owns it |
| global-state-crate-split-plan.md | business logic needs runtime state (object store, endpoints, lock clients, lifecycle state, config) and you must pick the right boundary, or you are evaluating a crate split out of ECStore |
| global-state-inventory.md | you meet a GLOBAL_* static or an OnceLock and need to know whether it is a runtime ownership handle, an owner-local static, or process-global by design |
| ecstore-module-split-plan.md | you add lifecycle or replication logic and need to know which crate it belongs in, plan to move an operation family out of SetDisks, or the guard fails on one of the split rules |
| ecstore-api-facade-inventory.md | you need something from rustfs_ecstore in another crate, you are narrowing a rustfs_ecstore::api facade group, or the guard reports a facade bypass |
| obs-ecstore-dependency-inventory.md | adding, removing, or moving any rustfs_ecstore or rustfs_storage_api reference inside crates/obs |
| compat-cleanup-register.md | you add, review, or remove a temporary compatibility path and need the RUSTFS_COMPAT_TODO marker format and its removal condition |
| overview.md | you need the historical framing of the architecture-migration program or the phase names that other contracts refer to |
Contracts and invariants
| Document | Use this when |
|---|---|
| local-descriptor-cache.md | changing descriptor cache keys/open modes or invalidation after local mutations |
| erasure-coding.md | changing anything under crates/ecstore/src/erasure/, crates/filemeta/, crates/ecstore/src/set_disk/, storage-class or layout code, or any decode, quorum, or heal boundary (normative spec) |
| placement-repair-invariants.md | changing anything that resolves an object to a pool, set, or disk, or that admits scanner or heal work |
| heal-concurrency-model.md | changing heal, PUT/multipart commit, delete, lifecycle expiry, or data-movement code that shares the (bucket, object) commit surface, or asking whether RustFS needs a persistent healing marker |
| heal-terminal-reports.md | changing retained admin heal outcomes, terminal report publication, cancellation refinement, or report upgrade/rollback behavior |
| unified-object-generation.md | adding or changing anything that fences a commit, scopes a read lease, gates old-directory cleanup, binds prepared pool reads, or settles quota against the current object version |
| ilm-tiering-persistence-contracts.md | changing an ILM transition, tier configuration mutation, manual transition job, tier-delete recovery path, pool decommission, or any code that can create, transfer, or destroy ownership of a remote-tier object |
| decommission-compatibility.md | changing pool decommission or rebalance behavior, its admin API shape, the persisted PoolMeta fields, or how tier free versions move between pools |
| ecstore-layout-boundary.md | touching endpoint expansion, FormatV3, pool/set layout, or moving files between ECStore's internal directories |
| runtime-capability-contracts.md | changing the read-only observability or topology snapshot contracts in rustfs-storage-api, their providers, or the storage_classes payload of GET /rustfs/admin/v4/runtime/capabilities |
| workload-admission-contracts.md | adding a workload class or snapshot provider, or consuming admission state from a background job |
| background-controller-contract.md | adding a status snapshot or reconcile surface for a background service, or being tempted to fold several services into a generic controller |
| background-services-inventory.md | you need one audited background service's desired source, current-status inputs, status surface, and declared side effects |
| scanner-usage-publication.md | changing scanner data-usage cache publication, quota-visible usage snapshots, scanner cycle recovery, or the persisted scanner usage artifacts |
| scanner-dirty-journal-source-contract.md | designing storage-verified provenance, completion receipts, repair responsibility or replay for the node-owned scanner dirty-producer journal (proposed; not runtime implemented) |
| scanner-usage-authority-decision.md | deciding whether quota admission depends on scanner data usage, removing scanner publication layers, or designing a scanner storage boundary |
| scanner-per-bucket-convergence.md | changing what makes a scanner cycle Complete, Superseded, or converged, adding or re-classifying a dirty-usage producer, deciding whether a background writeback may invalidate a cycle, changing scanner worker concurrency or throttling, or activating scoped dirty acknowledgements |
| config-model-boundary-adr.md | touching the server-config model (Config, KV, KVS) or its persistence, or asking which crate owns which part of server configuration |
| admin-route-action-snapshot.md | adding, moving, or re-authorizing an admin route and needing to know where the route → handler → AdminAction contract is enforced |
| kms-bulk-rekey-contract.md | changing the bulk envelope re-wrap sweep, its admin endpoints, the re-wrap primitive, or which objects a rekey may touch |
| remote-credential-sealing-adr.md | adding, reading, or persisting a stored remote credential (replication target, remote tier, on-demand migration source), or needing the sealed-envelope format, the mixed-version rules, or the reason this is worth doing in one deployment and not in another |
| tier-stats-contract.md | changing what GET /rustfs/admin/v3/tier-stats returns, adding a tier accounting source, or wiring a metric to a remote tier request |
Support and compatibility matrices (release-facing, keep current)
| Document | Use this when |
|---|---|
| s3-compatibility-matrix.md | writing or checking a user-facing S3 compatibility claim, or moving a Ceph s3tests case between lists |
| s3-tables-support-matrix.md | writing a release note or client-compatibility statement about S3 Tables / Iceberg REST Catalog (cutover procedure: ../operations/s3-tables-cutover-runbook.md) |
| minio-rustfs-router-compatibility.md | a client or mc call that works against MinIO fails against RustFS and you need to know whether the endpoint is missing, stubbed, or deliberately different |
| minio-file-format-compat.md | deciding whether a MinIO drive set, bucket-metadata blob, or SSE object can be read or imported by a given RustFS build, or before touching a listed version anchor |
Operations runbooks are registered in the documentation operations index, and testing references live in ../testing/README.md.
For replication operations, start with site replication operations, replication target check, replication object size limits, and replication outbound transport.
For per-node HTTP failure ratios and cached storage probe provenance, see S3 write failure diagnostics.