mirror of
https://github.com/rustfs/rustfs.git
synced 2026-11-07 11:10:47 +00:00
Chris An
d6210ac17b
* fix(ecstore,heal,select): bound storage-layer work driven by client input Scan findings in the erasure store, heal, file metadata and S3 Select layers, grouped because they share the disk walk, the multipart staging area, delete recovery and the select snapshot lease: - Multipart: upload ids are canonicalized before they name a staging path or a lock key (backlog#3027); trailing-slash keys are encoded like every other handler (backlog#3055); unlisted parts are removed at completion (backlog#3106); per-bucket uploads are found through a signed per-disk index instead of a scan of the shared staging area (backlog#3121); encrypted part ETags carry a per-upload secret so a write-only user cannot confirm plaintext guesses (backlog#3044). - Delete and recovery: recovery walks are bounded per request (backlog#3038); a delete marker on a key whose xl.meta path is a directory, and sidecar names used as child keys, no longer wedge the journal (backlog#3035, backlog#3105); over-quota PUTs no longer list the bucket under the ledger lock (backlog#3128). - Listing: scan_dir sends metadata without inline data (backlog#3052), counts delete markers against max-keys (backlog#3108), and rejects a marker that carries the cursor tag (backlog#3059). - Heal: bucket metadata and heal records live in separate namespaces so DeleteBucket cannot erase a drive rebuild (backlog#3039), a deleted bucket is skipped instead of failing the rebuild (backlog#3040), and directory objects are encoded on readback (backlog#3061). - Restore and tiering: RestoreObject preserves tags and the version's modification time (backlog#3073, backlog#3074); Snowball transport headers are not forwarded to the tier (backlog#3090); an epoch modification time round-trips through xl.meta (backlog#3063). - Locks: PutObject takes its bucket locks after the body is staged (backlog#3131); SelectObjectContent releases its namespace locks once the snapshot is taken (backlog#3132). - RPC and peers: the v3 signature check consumes the v2 nonce (backlog#3129); a bucket-metadata reload timeout no longer marks the peer offline (backlog#3124); replication target requests carry an operation deadline (backlog#3102); the scanner resumes its interrupted wait after a lifecycle write (backlog#3118). - S3 Select: bounded schema inference, column cap and budgeted CSV batches, Parquet dictionary bounds and pre-planner projection checks (backlog#3075, backlog#3076, backlog#3077, backlog#3078). * test(e2e): seed legacy duplicate bucket tags offline for the OPA contract * fix(s3): authorize what handlers store and bound request-driven work Scan findings in the S3 request layer (access hooks, object use cases, HTTP server layers), grouped because they share the authorization condition builder, the PutObject admission path and the response layers. Builds on the storage-layer batch for the bounded recovery and multipart APIs it calls. Authorization and policy conditions: - copy source is percent-decoded and version-canonical before the s3:x-amz-copy-source condition (backlog#3135); CopyObject authorizes the tag set it copies (backlog#3034); HeadObject with the replication check header still authorizes s3:GetObject (backlog#3047); the x-rustfs-source-version-id header feeds s3:versionid (backlog#3072); recursive force-delete with a REPLICA header authorizes every descendant as a delete (backlog#3070); include-deleted listings require s3:ListBucketVersions (backlog#3099); the metadata listing route keeps table isolation and uses the same prefix it authorizes (backlog#3053, backlog#3054); If-Match probing needs read permission (backlog#3097); Object Lambda checks s3:GetObjectVersion (backlog#3058); replication proxy fallback keeps ExistingObjectTag denies (backlog#3089); HeadObject returns lock and tag headers only with permission (backlog#3113); IPv4 clients on the dual-stack listener match IPv4 SourceIp conditions (backlog#3049); bucket policies are parsed once and cached (backlog#3032); keystone: access keys no longer verify with an empty secret (backlog#3043). Admission and resource bounds: - GetObject releases disk-read permits when the client stops reading (backlog#3023); PutObject takes buffer and foreground admission only after body bytes arrive and bounds the wait (backlog#3031, backlog#3084); xl.meta size is bounded on the write path (backlog#3100); XML compat layers stream instead of buffering whole bodies (backlog#3094); tier GETs stream the remote body (backlog#3046); on-demand migration pulls release permits on unread GETs (backlog#3110); SSE-KMS key policy covers UploadPart and completion (backlog#3130); MinIO admin decryption runs Argon2id off the async worker (backlog#3024). Input validation and leaks: - replication headers are validated (source ETag shape, original size) (backlog#3092, backlog#3093); control characters in keys and lifecycle rule IDs are rejected (backlog#3101, backlog#3117); Snowball member mtime needs s3:ReplicateObject (backlog#3083); DeleteObjects checks Content-MD5 and admin/STS reject SigV2 replays with altered query or body (backlog#3037, backlog#3026); request logs redact presigned queries and handler spans skip credentials (backlog#3068, backlog#3107); SSE-C listings and completions withhold plaintext metadata and checksums (backlog#3120, backlog#3050); CORS resolves the bucket like s3s does (backlog#3104); PutBucketTagging rejects duplicate keys (backlog#3126); notifications fire after a client disconnect (backlog#3127); ListenNotification streams re-check access (backlog#3119). * fix(api): drop imports left unused by the base merge * test(s3): move the KMS multipart denial test to the app layer The test drives app-layer multipart use cases, so it belongs with them; living in the SSE module it imported app code from the infra layer and failed the layer dependency guard. The SSE test helpers it borrows are exposed through the storage and app facades. * fix(s3): keep the SSE-C key fingerprint on keyless multipart completion The backlog#3050 fix withheld both the plaintext checksums and the customer-key MD5 header from a CompleteMultipartUpload that did not present the key. The checksums are a digest of the plaintext and stay hidden; the key MD5 identifies the key only, and the S3 conformance suite (test_copy_part_enc) expects it on a keyless completion, so it is echoed again. * fix(tier): stream GET bodies and document replication budgets * fix(ecstore,s3): keep the multipart index and cold fill off disk lifetimes Two lifecycle regressions surfaced once the storage and request batches ran together against main: - The multipart index owner cloned the disk's pinned publication root and bootstrapped on a task of its own. That kept the mount alive past the last LocalDisk owner (the disk lifecycle allows exactly three holders of the root FD) and left a stray task behind, which the local-disk and read-repair tests detect. The owner now keeps only the root path and reopens it for identity checks, and the bootstrap scan rides the disk's existing cleanup task, where stopping the disk abandons it and a reconnect resumes it. - The disk-read admission wrapper defers releasing its permit until an in-flight read completes, so a cancelled client cannot widen disk concurrency. The cold-fill producer, however, drops a read it has already classified as stalled when its deadline fires, and that permit stayed pinned for the lifetime of the stuck read. The producer now abandons the admission explicitly on timeout. * test(ecstore): pass the index owner to the Linux cleanup-loop test * fix(s3): release cold-fill admission after watchdog handoff * fix(s3,ecstore): close the round-1 adversarial review findings Adversarial review of the combined request- and storage-layer batch found fixes that fell short of their issues, two behaviour regressions for legitimate clients and several unrequested costs: - backlog#3038: a bulk DeleteObjects reserved the whole per-disk inspection allowance up front for every key in flight and exhausted the 16 Mi request budget on any set wider than eight drives, failing healthy keys with SlowDown. Calls are now admitted while budget remains and charged for the entries they actually visited. Object-scoped discovery charges plain child keys at one entry per 64 names instead of one each, so deleting a missing key that names a folder prefix completes instead of answering 503 forever while a hostile prefix stays bounded. A peer still on the previous binary is reported as an unsupported participant, the repository's rolling-upgrade contract, instead of as a budget verdict. - backlog#3037/#3026: a SigV2 configuration PUT without Content-MD5 was still replayable with another body; it is refused with MissingContentMD5 while SigV4 keeps the optional-MD5 contract. - backlog#3072: DeleteObject now resolves x-rustfs-source-version-id before authorization like PUT and COPY, so a replica delete cannot remove a version the policy never saw. - backlog#3121: the multipart index no longer fsyncs a marker and four directories per multipart metadata write (it is rebuilt on every start), and ListMultipartUploads discovers roots concurrently. - backlog#3077: Parquet SelectObjectContent decodes budget-sized batches instead of one row per batch. - backlog#3031: a full upload buffer pool hands out an unpooled buffer instead of refusing the upload; the progress floor already bounds stalls. - backlog#3023: the GET disk-read idle cut-off is configurable (RUSTFS_GET_DISK_READ_IDLE_TIMEOUT_SECS, default 120 s) so a paused client is not truncated after 30 s. - backlog#3075: JSON Lines schema inference keeps the sample collected at the 4 MiB bound instead of failing the query. - backlog#3024: compat admin decryption queues briefly for a slot instead of refusing the third concurrent request. - backlog#3113: HEAD skips its permission probes for the bucket owner. - backlog#3117: control characters in a lifecycle rule ID are rejected on PutBucketLifecycle as well. The lock-crate publication FIFO serves backlog#3111 and moves to the table-catalog PR. Also fixed: a lossy admin error message, an unused s3-client function, two unused test-util features, a duplicated key validation, the s3s footprint baseline, and a stale log event name in the reverse-proxy runbook. The ODM multipart write-back test runs on the large test stack like its sibling. * fix(table-catalog): bound maintenance work and isolate table namespaces (#9174) * fix(table-catalog): bound maintenance work and isolate table namespaces Scan findings in the Iceberg table catalog: - Metadata maintenance deleted every unreferenced *.metadata.json in a table's directory without checking whether a renamed table in another namespace still owned those files (backlog#3060); it now resolves the owning table through the directory index before deleting. - Manifest lists and manifests were decoded with the embedded Avro writer schema and no recursion depth limit; a self-referencing record schema overflowed the stack (backlog#3080). Schemas are validated for depth before decoding. - Compaction decoded every input Parquet file into Arrow batches held in memory; a decompression bomb OOM-killed the server (backlog#3085). Row groups are streamed through a byte budget and oversized inputs are refused. - An S3 request for a key outside an indexed table prefix listed the bucket's whole namespaces/ catalog prefix and failed once it held more than 4096 objects (backlog#3086); a per-bucket directory index answers index misses without the listing. - The maintenance planner read whole orphan files to obtain their modification time (backlog#3087); it reads metadata only. - A client disconnect between persisting active_rename_id and clearing it left the bucket's catalog entry locked at 503 (backlog#3088); the rename runs detached from the request future and clears the flag on every exit. - Table commits held the bucket publication fence until publication, starving every object write (backlog#3111); commits now hold it under a deadline and object writes are admitted between commits. The directory index is managed through three new admin routes under {warehouse}/catalog/directory, registered in the admin route policy. The durable-strong store is left as is (its snapshot partitioning for backlog#3109 and the strong-path deadline are not ported): that backing is being retired separately, so the new tests cover the object backing. * fix(table-catalog): close the review gaps in the scan-fix batch Adversarial review of the table-catalog batch found the fixes narrower than the issues they close and two cost regressions: - backlog#3085: the compaction footer check bounded nothing a page header could not override; parquet sizes its page buffers from the header. Walk every column chunk's page headers (Thrift compact, no decompression) and hold each page and the chunk total to the budget before decoding. The stored-input budget is per rewrite group, so a multi-group job no longer fails after writing earlier outputs. - backlog#3086: the default index-miss scan still counted maintenance reports toward its 4096-object limit; only table entries count now, and the listing itself is bounded to 64 pages. The directory index remains an optional accelerator. - backlog#3111: a create wrote its metadata file under the fence as the first durable mutation, which disarmed the deadline before the unbounded registration inventory. The write precedes publication now. The deadline gets its own knob (RUSTFS_TABLE_CATALOG_PUBLICATION_PREPARATION_TIMEOUT_SECS, default 30 s, never below the lock-acquire timeout) instead of reusing the 5 s lock timeout, and the ownership inventory reads entries concurrently. - Directory candidates are written only while a directory exists; the backfill covers earlier entries. A directory on the migration-fence path is reported as a conflict instead of a parse error. - Dead test hooks, a stale comment and the support-matrix text that did not match the code are removed or corrected. * fix(select): preserve complete samples and Parquet batch limits * fix(s3,ecstore): close the round-2 adversarial review findings DeleteObject resolved x-rustfs-source-version-id for authorization on every request while storage consumes it only for a REPLICA request, so a non-replica caller with s3:DeleteObjectVersion could trigger a plain delete of the current version. The hook now resolves the header under the same predicate as storage, and the regression test covers the non-replica case. HEAD no longer skips its permission probes for the bucket owner; an explicit bucket-policy Deny on the metadata actions binds the owner too. Multipart index pages verify each root against the staging area and prune markers whose staging root is gone, under the marker's own lease with a re-check, so the per-drive index stays proportional to live uploads instead of every key ever multipart-initiated on the drive. SigV2 configuration PUTs that declare an empty body need no Content-MD5: there is no body to replay. The minimum-progress floor stays independent of RUSTFS_HTTP_REQUEST_BODY_READ_TIMEOUT; the knob's documentation now says so instead of claiming 0 disables it. The quota bootstrap window (503 until the background scan publishes the ledger) is kept, because ledger admission must not come from scanner reporting; the compatibility matrix now documents it. The S3 compatibility matrix gains a section for the client-visible validation bounds, and the drive-timeout tuning guide tabulates RUSTFS_GET_DISK_READ_IDLE_TIMEOUT_SECS. * fix(lock): queue table-bucket publication writers fairly Table commits take the bucket publication fence as a write lock; under a stream of commits the lock's try-and-retry acquisition let a new commit jump ahead of object writers that had been waiting longer, which is part of what starved them (backlog#3111). Requests for the publication lock path now take a FIFO ticket per shard: a waiter is admitted only when it is at the head of the queue, cancelled waiters release their turn, and batch acquisitions join the same queue. Other lock paths are unchanged. This moved here from the request-layer batch, where no cited issue covered it; it serves the table-catalog issue this PR closes. * fix(table-catalog): close the round-2 adversarial review findings Parquet page-header walker: read the size fields by field id as parquet does, accepting I16/I32/I64 wire kinds and rejecting any other kind, so a header encoding uncompressed_page_size as I64 can no longer bypass the budget while parquet allocates it. Bound every list, set and map count by the bytes remaining in the chunk, and reject bool collection elements (parquet skips them with zero width, so either width would desynchronize the two parsers); this removes the 2^63-iteration spin on LIST<BOOL>. Start each column-chunk walk at the dictionary page offset whenever the footer declares one, exactly as parquet's byte_range does, instead of min(dictionary, data). Footer pre-walk: before parquet decodes a footer, walk FileMetaData with the hardened cursor and charge the declared row_groups count times the RowGroupMetaData size to the decode budget, because parquet 59 reserves that Vec before reading one element. Preparation deadline: clamp the RUSTFS_TABLE_CATALOG_PUBLICATION_PREPARATION_TIMEOUT_SECS knob to 86,400 seconds so Instant + Duration cannot overflow. Index-miss scan: after each listing page, jump the cursor past the maintenance subtree the page ended in, and allow one extra page per table entry already listed, so job-report history scales with the catalog while a report flood under one table stays bounded. Migration fence: read the fence-path object once and decide from its stored kind whether it is directory state or a fence. Expired or failed create: remove the initial metadata file when the registration fails before any catalog write may have begun; nothing else references it. Correct the comment and support matrix that claimed maintenance reclaims it. Tests cover each walker divergence, the footer row-group reservation, the deadline ceiling, the paging budget for honest catalogs and for a report flood, and the removal of the unpublished metadata file. * test(admin): register the catalog directory routes in the route matrix The table-catalog batch added GET, POST and DELETE {warehouse}/catalog/directory on both the Iceberg and the compat admin prefixes without listing them in the admin route matrix, so test_admin_route_matrix_matches_registered_routes failed on CI. * fix(s3,ecstore): close the round-3 adversarial review findings Multipart index pages now treat the cursor as the last examined marker rather than the last root, so a page that pruned a stale marker still validates and ListMultipartUploads keeps using the index. A marker is written before its staging root is published, so pruning spares markers younger than a five-minute grace period and an existing marker is touched on every new upload; the prune re-checks both under the marker lease. Bounded delete-recovery inspection rides the durable-mutation lane like the unbounded path instead of the one-worker foreground lane. The on-demand-migration inline body budget scales with the object size (at least 64 KiB/s) instead of a flat 120 s. The control-body digest service honors RUSTFS_HTTP_REQUEST_BODY_READ_TIMEOUT=0 as "no inactivity timeout" like every other reader of that knob. Quota reconciliation throttle objects are deleted together with a retired generation's ledger, so bucket turnover leaves no residue. The PUT guard test shim is gone; tests call the shared guard. Bounded DiskAPI defaults answer MethodNotAllowed like their siblings. Stale budget docs, an unreachable CSV arm, an always-Some limit parameter, commented-out constants and a misplaced doc comment are cleaned up; weak test assertions name the expected error; lifecycle rule-ID control characters gain a test. The compatibility matrix records the CopyObject tag-copy permission, the exact SigV2 empty-body predicate, replica and migration writes of control-character keys, and the SigV2 DeleteObjects digest rule. * fix(table-catalog,heal): close the round-3 review findings A. Pack compaction rewrite groups to min(max-rewrite-bytes-per-job, TABLE_COMPACTION_MAX_STORED_BYTES), reject an over-ceiling group before any output is written, and name the ceiling in the commit-path errors; document the effective cap in the support matrix and scheduler runbook. B. Keep TaskCancelled/TaskTimeout classification on replacement passes: the namespace verification runs only for completed or storage-failed passes, in the erasure healer and the task bucket prepass. C. Compare the replacement bucket pass per bucket: a recorded identity that is gone or replaced still errors or restarts the pass, buckets created mid-pass are appended to the identities, pending_buckets and a persisted positional order without rewinding progress or the checkpoint; verification ignores additions. D. Read the migration-fence object once in the object-backed write permit and dispatch on its kind. E. Drop the strong store's copy of maintenance_cleanup_roots. F. Remove the unused test hooks fail_put_prefix, corrupt_put_prefix and pause_after_metadata; make compact_parquet_data_files and PublicationTicket.state private. G. Document the body-revision identity, the legacy revalidation budget and its HTTP 503 in the support matrix. H. Define HEAL_REPLACEMENT_RECOVERY_DIR once in ecstore and consume it from heal through the storage_api facade. * fix(ecstore): cap the index pages one disk may answer Pruned multipart index pages may carry few roots, so the page cursor no longer bounds how many pages a peer can make a listing fetch. Cut a disk off after 4096 pages (about one million examined markers) so it contributes no coverage and the listing falls back to the staging scan. * fix(ecstore,table-catalog): close the round-4 adversarial review findings Multipart index markers dated in the future (a backwards clock step between recording a marker and publishing its staging root) were judged old and could be pruned while their upload was in flight; a future-dated marker is now kept. The marker touch on EEXIST is best effort, so a filesystem that refuses to set times no longer fails the upload commit. Pruning runs on the durable-mutation lane instead of the one-worker foreground lane. Quota reconciliation keeps one throttle object per bucket incarnation (a quota revision change replaces it instead of orphaning the previous revision's object), the retirement cleanup deletes it ahead of the ledger and also when the ledger is already gone, and a scan the retirement already refused writes nothing back. Bounded DiskAPI defaults answer UnsupportedDisk, the verdict the aggregate gives a peer without the protocol. The compaction ceiling pre-check runs over every rewrite group before any output is written, and the worker's ceiling error no longer names a placeholder group. Docs: the inline on-demand-migration client budget, the protocol-2 downgrade conflict of replacement passes, the exact footprint numbers.
Architecture Documentation
Use this when: you need the contract, invariant, or boundary rule that governs a change, and you want the one document that owns it.
Source of truth: the code and the guards. scripts/check_architecture_migration_rules.sh enforces the CI-anchored documents below; scripts/check_doc_paths.sh fails the pre-commit gate when any doc under docs/ cites a repository path that no longer exists.
Two rules keep this directory healthy:
- Durable reference only. One-shot plans, task trackers, dated analyses, status snapshots, and PR-scoped notes do not belong in the repository; keep them in the issue tracker or a local worktree and delete them when the work closes.
- No copies of other sources of truth. Crate lists come from
Cargo.toml, CI steps from.github/workflows/, code structure from the code. Cite a file path plus a symbol name, never a line number, and never paste counts or tables that a command can regenerate.
Every document starts with a **Use this when:** line so an agent can decide in one glance whether to read further. The index below repeats those lines.
CI-anchored core
Required headings and strings in these files are asserted by scripts/check_architecture_migration_rules.sh; rename a heading only together with the guard.
| Document | Use this when |
|---|---|
| crate-boundaries.md | you add a crate dependency, move code across crates, touch a storage_api.rs boundary file, or need the change-type vocabulary the architecture guard enforces |
| runtime-lifecycle.md | moving or reordering anything in rustfs/src/startup_*.rs, changing readiness publication, or touching shutdown ordering |
| readiness-matrix.md | changing what a request surface does before storage or IAM is ready, changing probe semantics, or adding a runtime dependency that readiness must wait for |
| storage-control-data-plane.md | adding a storage API surface, a cluster read model, or a background-service status/reconcile surface, and you need to know which layer owns it |
| global-state-crate-split-plan.md | business logic needs runtime state (object store, endpoints, lock clients, lifecycle state, config) and you must pick the right boundary, or you are evaluating a crate split out of ECStore |
| global-state-inventory.md | you meet a GLOBAL_* static or an OnceLock and need to know whether it is a runtime ownership handle, an owner-local static, or process-global by design |
| ecstore-module-split-plan.md | you add lifecycle or replication logic and need to know which crate it belongs in, plan to move an operation family out of SetDisks, or the guard fails on one of the split rules |
| ecstore-api-facade-inventory.md | you need something from rustfs_ecstore in another crate, you are narrowing a rustfs_ecstore::api facade group, or the guard reports a facade bypass |
| obs-ecstore-dependency-inventory.md | adding, removing, or moving any rustfs_ecstore or rustfs_storage_api reference inside crates/obs |
| compat-cleanup-register.md | you add, review, or remove a temporary compatibility path and need the RUSTFS_COMPAT_TODO marker format and its removal condition |
| overview.md | you need the historical framing of the architecture-migration program or the phase names that other contracts refer to |
Contracts and invariants
| Document | Use this when |
|---|---|
| local-descriptor-cache.md | changing descriptor cache keys/open modes or invalidation after local mutations |
| erasure-coding.md | changing anything under crates/ecstore/src/erasure/, crates/filemeta/, crates/ecstore/src/set_disk/, storage-class or layout code, or any decode, quorum, or heal boundary (normative spec) |
| placement-repair-invariants.md | changing anything that resolves an object to a pool, set, or disk, or that admits scanner or heal work |
| heal-concurrency-model.md | changing heal, PUT/multipart commit, delete, lifecycle expiry, or data-movement code that shares the (bucket, object) commit surface, or asking whether RustFS needs a persistent healing marker |
| heal-terminal-reports.md | changing retained admin heal outcomes, terminal report publication, cancellation refinement, or report upgrade/rollback behavior |
| unified-object-generation.md | adding or changing anything that fences a commit, scopes a read lease, gates old-directory cleanup, binds prepared pool reads, or settles quota against the current object version |
| ilm-tiering-persistence-contracts.md | changing an ILM transition, tier configuration mutation, manual transition job, tier-delete recovery path, pool decommission, or any code that can create, transfer, or destroy ownership of a remote-tier object |
| decommission-compatibility.md | changing pool decommission or rebalance behavior, its admin API shape, the persisted PoolMeta fields, or how tier free versions move between pools |
| ecstore-layout-boundary.md | touching endpoint expansion, FormatV3, pool/set layout, or moving files between ECStore's internal directories |
| runtime-capability-contracts.md | changing the read-only observability or topology snapshot contracts in rustfs-storage-api, their providers, or the storage_classes payload of GET /rustfs/admin/v4/runtime/capabilities |
| workload-admission-contracts.md | adding a workload class or snapshot provider, or consuming admission state from a background job |
| background-controller-contract.md | adding a status snapshot or reconcile surface for a background service, or being tempted to fold several services into a generic controller |
| background-services-inventory.md | you need one audited background service's desired source, current-status inputs, status surface, and declared side effects |
| scanner-usage-publication.md | changing scanner data-usage cache publication, quota-visible usage snapshots, scanner cycle recovery, or the persisted scanner usage artifacts |
| scanner-prefix-statistics-contract.md | designing, implementing, or reviewing scanner-derived prefix statistics for the object browser or another authorized consumer (proposed; not runtime implemented) |
| scanner-dirty-journal-source-contract.md | designing storage-verified provenance, completion receipts, repair responsibility or replay for the node-owned scanner dirty-producer journal (proposed; not runtime implemented) |
| scanner-usage-authority-decision.md | deciding whether quota admission depends on scanner data usage, removing scanner publication layers, or designing a scanner storage boundary |
| scanner-per-bucket-convergence.md | changing what makes a scanner cycle Complete, Superseded, or converged, adding or re-classifying a dirty-usage producer, deciding whether a background writeback may invalidate a cycle, changing scanner worker concurrency or throttling, or activating scoped dirty acknowledgements |
| config-model-boundary-adr.md | touching the server-config model (Config, KV, KVS) or its persistence, or asking which crate owns which part of server configuration |
| plugin-delegation.md | changing original-user plugin delegation, attempt credentials, or Runner task APIs |
| admin-route-action-snapshot.md | adding, moving, or re-authorizing an admin route and needing to know where the route → handler → AdminAction contract is enforced |
| kms-bulk-rekey-contract.md | changing the bulk envelope re-wrap sweep, its admin endpoints, the re-wrap primitive, or which objects a rekey may touch |
| remote-credential-sealing-adr.md | adding, reading, or persisting a stored remote credential (replication target, remote tier, on-demand migration source), or needing the sealed-envelope format, the mixed-version rules, or the reason this is worth doing in one deployment and not in another |
| tier-stats-contract.md | changing what GET /rustfs/admin/v3/tier-stats returns, adding a tier accounting source, or wiring a metric to a remote tier request |
Support and compatibility matrices (release-facing, keep current)
| Document | Use this when |
|---|---|
| s3-compatibility-matrix.md | writing or checking a user-facing S3 compatibility claim, or moving a Ceph s3tests case between lists |
| s3-tables-support-matrix.md | writing a release note or client-compatibility statement about S3 Tables / Iceberg REST Catalog (cutover procedure: ../operations/s3-tables-cutover-runbook.md) |
| minio-rustfs-router-compatibility.md | a client or mc call that works against MinIO fails against RustFS and you need to know whether the endpoint is missing, stubbed, or deliberately different |
| minio-file-format-compat.md | deciding whether a MinIO drive set, bucket-metadata blob, or SSE object can be read or imported by a given RustFS build, or before touching a listed version anchor |
Operations runbooks are registered in the documentation operations index, and testing references live in ../testing/README.md.
For replication operations, start with site replication operations, replication target check, replication object size limits, and replication outbound transport.
For per-node HTTP failure ratios and cached storage probe provenance, see S3 write failure diagnostics.