Files
rustfs/docs/architecture
Hauser Zhi 571d100a27 perf(ecstore): align inline placement and batch deletion codecs (#9449)
* feat(ecstore): align inline budgets and binary batch decoding

Use a fixed 128 KiB per-shard inline budget with versioned admission and
preserve historical configuration markers and persisted object layouts.

Decode binary delete batches independently of the legacy JSON list while
retaining JSON-only requests and dual-write senders. Reject malformed
binary entries before disk mutation and preallocate the selected batch.

Cover budget boundaries, layout coexistence, MinIO metadata fixtures,
binary-only batches and JSON compatibility. Document geometry and
rollback boundaries.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): add isolated Docker inline geometry validation

Build clean-source Linux validation images from a frozen Git bundle.
Pin running image identities and binary hashes, and inspect persisted
object geometry through the existing filemeta decoder.

Provide an owned four-node Docker adapter for the existing ABBA runner.
Guard resource ownership, isolate loopback access, serialize state
mutations, and verify HEAD, full GET and Range contents.

Keep PUT content probes available through an isolated Linux mc adapter
when a host client cannot initialize. Cover ownership and failure paths
with script tests wired into the existing runner.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): reuse bounded verified inline bodies

Share the materialization contract between metadata pre-verification
and buffered inline GETs. Bound responses to 2 MiB and a limited frame
count, and reuse an already verified body within the selected snapshot.

Route proof-protected parts through the readers that bind PartProofReader.
Cover larger single-block bodies, allocation bounds and proof validation
after a shard's bitrot checksum has been recomputed.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* feat(ecstore): support bounded multi-block inline objects

Keep the default EC block and accumulated shard budget while bounding inline frame storage and verified-body reuse. Preserve transformed and protected write admission and existing metadata formats.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): isolate inline transport comparison modes

Pin and verify both MessagePack-only gates for each Docker cell, preserve an explicit JSON compatibility cell and probe exact accumulated shard boundaries.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): cover inline rollback and mixed-reader fleets

Add reusable compatibility fixtures for null and versioned objects, cross-block ranges, unfinished multipart uploads and batch version deletion. Record coordinator identity and per-version geometry, preserve retained-data manifests and archive manifests on dataset resets.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(ecstore): unify bounded inline read and write paths

Share checked frame decoding across metadata preverification and GET, resolve write admission once, and use one checked physical shard length for inline and disk writers. Preserve proof-bound readers and erasure recovery for damaged or missing shards.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(testing): persist inline compatibility mutation receipts

Save successful multipart and delete receipts before fallible content and geometry checks. Resume completed mutations through verification without repeating writes, and cover read failures, geometry failures, and command retries.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(gateway): isolate capability probes from multipart listings

Give successful capability probes their own buckets so CreateMultipartUpload does not change the exact two-upload pagination fixtures. Preserve all listing-boundary assertions on both HTTP stacks.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(rpc): honor fleet-confirmed codecs for batch deletion

Build DeleteVersions requests with one codec policy for the complete batch. Emit JSON compatibility fields by default and binary-only batches after both existing fleet gates pass. Cover request fields, canonical binding and actual remote sender, handler and disk deletion in both modes.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(ecstore): verify inline shards through borrowed frames

Share borrowed frame verification across buffered GET, bitrot readers and deep inline healing. Validate exact physical framing before scanning memory slices. Cover short prefixes, legacy padded frames, damage and trailing data, and provide a reproducible manual ABBA kernel comparison.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ci): account for manual inline verification benchmark

Raise the ECStore ignored-test ceiling from 2 to 3 for the opt-in
ABBA comparison of borrowed and streaming bitrot verification.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): reject inexact inline shard framing

Validate the complete encoded length of every selected inline data shard
before decoding or caching a verified object body. Return to the erasure
reader on missing, truncated, or trailing payload bytes.

Extend decoder fixtures across single and multi-block geometry, legacy
padding, unhashed payloads, and explicit bitrot verification bypass. Cover
metadata early-stop rejection and GET recovery for trailing shard bytes.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(ecstore): share inline shard reader construction

Build owned-metadata and borrowed-metadata readers through one Send iterator-based factory. Preserve shard ordering, empty slots, bitrot settings and caller-bound integrity verification.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(rpc): move endpoint buffers into batch delete requests

Transfer the prepared endpoint String into the protobuf request and assert allocation identity across every batch codec mode. This removes the intermediate endpoint copy during request construction.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(gateway): remove superseded pagination isolation

Use the upstream capability probes that clean up their own multipart sessions. Run probes against the seeded pagination bucket while keeping all exact page and cursor assertions.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(table-catalog): recognize borrowed multipart guard calls

Count owned and borrowed bucket/key guard forms while requiring all four production calls. Cover mixed split-file calls and reject wrong resources, missing error propagation and test-only mentions.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): size heal cleanup fixtures above inline budgets

Use an environment-independent per-pool fixture policy and payload sizes above the aggregate inline budget. Assert external placement before pausing old-data cleanup, rebuilding the committed generation and checking physical shards and the final payload.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(table-catalog): count borrowed object metadata guards

Recognize the borrowed bucket/key guards in migrated legal-hold and retention entrypoints. Validate two object and two key guards and cover incorrect resources, missing propagation and test-only mentions.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-16 10:11:19 +00:00
..

Architecture Documentation

Use this when: you need the contract, invariant, or boundary rule that governs a change, and you want the one document that owns it. Source of truth: the code and the guards. scripts/check_architecture_migration_rules.sh enforces the CI-anchored documents below; scripts/check_doc_paths.sh fails the pre-commit gate when any doc under docs/ cites a repository path that no longer exists.

Two rules keep this directory healthy:

  1. Durable reference only. One-shot plans, task trackers, dated analyses, status snapshots, and PR-scoped notes do not belong in the repository; keep them in the issue tracker or a local worktree and delete them when the work closes.
  2. No copies of other sources of truth. Crate lists come from Cargo.toml, CI steps from .github/workflows/, code structure from the code. Cite a file path plus a symbol name, never a line number, and never paste counts or tables that a command can regenerate.

Every document starts with a **Use this when:** line so an agent can decide in one glance whether to read further. The index below repeats those lines.

CI-anchored core

Required headings and strings in these files are asserted by scripts/check_architecture_migration_rules.sh; rename a heading only together with the guard.

Document Use this when
crate-boundaries.md you add a crate dependency, move code across crates, touch a storage_api.rs boundary file, or need the change-type vocabulary the architecture guard enforces
runtime-lifecycle.md moving or reordering anything in rustfs/src/startup_*.rs, changing readiness publication, or touching shutdown ordering
readiness-matrix.md changing what a request surface does before storage or IAM is ready, changing probe semantics, or adding a runtime dependency that readiness must wait for
storage-control-data-plane.md adding a storage API surface, a cluster read model, or a background-service status/reconcile surface, and you need to know which layer owns it
global-state-crate-split-plan.md business logic needs runtime state (object store, endpoints, lock clients, lifecycle state, config) and you must pick the right boundary, or you are evaluating a crate split out of ECStore
global-state-inventory.md you meet a GLOBAL_* static or an OnceLock and need to know whether it is a runtime ownership handle, an owner-local static, or process-global by design
ecstore-module-split-plan.md you add lifecycle or replication logic and need to know which crate it belongs in, plan to move an operation family out of SetDisks, or the guard fails on one of the split rules
ecstore-api-facade-inventory.md you need something from rustfs_ecstore in another crate, you are narrowing a rustfs_ecstore::api facade group, or the guard reports a facade bypass
obs-ecstore-dependency-inventory.md adding, removing, or moving any rustfs_ecstore or rustfs_storage_api reference inside crates/obs
compat-cleanup-register.md you add, review, or remove a temporary compatibility path and need the RUSTFS_COMPAT_TODO marker format and its removal condition
overview.md you need the historical framing of the architecture-migration program or the phase names that other contracts refer to

Contracts and invariants

Document Use this when
local-descriptor-cache.md changing descriptor cache keys/open modes or invalidation after local mutations
erasure-coding.md changing anything under crates/ecstore/src/erasure/, crates/filemeta/, crates/ecstore/src/set_disk/, storage-class or layout code, or any decode, quorum, or heal boundary (normative spec)
placement-repair-invariants.md changing anything that resolves an object to a pool, set, or disk, or that admits scanner or heal work
heal-concurrency-model.md changing heal, PUT/multipart commit, delete, lifecycle expiry, or data-movement code that shares the (bucket, object) commit surface, or asking whether RustFS needs a persistent healing marker
heal-terminal-reports.md changing retained admin heal outcomes, terminal report publication, cancellation refinement, or report upgrade/rollback behavior
mrf-management-contract.md changing cluster MRF observations, responsibility actions, or legacy integrity audit evidence
unified-object-generation.md adding or changing anything that fences a commit, scopes a read lease, gates old-directory cleanup, binds prepared pool reads, or settles quota against the current object version
ilm-tiering-persistence-contracts.md changing an ILM transition, tier configuration mutation, manual transition job, tier-delete recovery path, pool decommission, or any code that can create, transfer, or destroy ownership of a remote-tier object
decommission-compatibility.md changing pool decommission or rebalance behavior, its admin API shape, the persisted PoolMeta fields, or how tier free versions move between pools
ecstore-layout-boundary.md touching endpoint expansion, FormatV3, pool/set layout, or moving files between ECStore's internal directories
runtime-capability-contracts.md changing the read-only observability or topology snapshot contracts in rustfs-storage-api, their providers, or the storage_classes payload of GET /rustfs/admin/v4/runtime/capabilities
workload-admission-contracts.md adding a workload class or snapshot provider, or consuming admission state from a background job
background-controller-contract.md adding a status snapshot or reconcile surface for a background service, or being tempted to fold several services into a generic controller
background-services-inventory.md you need one audited background service's desired source, current-status inputs, status surface, and declared side effects
scanner-usage-publication.md changing scanner data-usage cache publication, quota-visible usage snapshots, scanner cycle recovery, or the persisted scanner usage artifacts
scanner-usage-watch.md changing scanner reuse across cycles, adding an object mutation producer, moving a physical commit into detached work, or changing the bucket watch RPC and its compatibility fallback
scanner-prefix-statistics-contract.md designing, implementing, or reviewing scanner-derived prefix statistics for the object browser or another authorized consumer (proposed; not runtime implemented)
scanner-dirty-journal-source-contract.md designing storage-verified provenance, completion receipts, repair responsibility or replay for the node-owned scanner dirty-producer journal (proposed; not runtime implemented)
scanner-usage-authority-decision.md deciding whether quota admission depends on scanner data usage, removing scanner publication layers, or designing a scanner storage boundary
scanner-per-bucket-convergence.md changing what makes a scanner cycle Complete, Superseded, or converged, adding or re-classifying a dirty-usage producer, deciding whether a background writeback may invalidate a cycle, changing scanner worker concurrency or throttling, or activating scoped dirty acknowledgements
config-model-boundary-adr.md touching the server-config model (Config, KV, KVS) or its persistence, or asking which crate owns which part of server configuration
plugin-delegation.md changing original-user plugin delegation, attempt credentials, or Runner task APIs
admin-route-action-snapshot.md adding, moving, or re-authorizing an admin route and needing to know where the route → handler → AdminAction contract is enforced
kms-bulk-rekey-contract.md changing the bulk envelope re-wrap sweep, its admin endpoints, the re-wrap primitive, or which objects a rekey may touch
remote-credential-sealing-adr.md adding, reading, or persisting a stored remote credential (replication target, remote tier, on-demand migration source), or needing the sealed-envelope format, the mixed-version rules, or the reason this is worth doing in one deployment and not in another
tier-stats-contract.md changing what GET /rustfs/admin/v3/tier-stats returns, adding a tier accounting source, or wiring a metric to a remote tier request
s3-over-rdma-llm-serving.md you change the RDMA or NIXL data path, the large-PUT path, or a per-bucket ETag setting, or you build or run a client that loads model weights or offloads KV cache to RustFS over RDMA, such as the SGLang weight loader and HiCache backend

Support and compatibility matrices (release-facing, keep current)

Document Use this when
s3-compatibility-matrix.md writing or checking a user-facing S3 compatibility claim, or moving a Ceph s3tests case between lists
s3-tables-support-matrix.md writing a release note or client-compatibility statement about S3 Tables / Iceberg REST Catalog
minio-rustfs-router-compatibility.md a client or mc call that works against MinIO fails against RustFS and you need to know whether the endpoint is missing, stubbed, or deliberately different
minio-file-format-compat.md deciding whether a MinIO drive set, bucket-metadata blob, or SSE object can be read or imported by a given RustFS build, or before touching a listed version anchor

Operations runbooks are registered in the documentation operations index, and testing references live in ../testing/README.md.

For replication operations, start with site replication operations, replication target check, replication object size limits, and replication outbound transport.

For per-node HTTP failure ratios and cached storage probe provenance, see S3 write failure diagnostics.