mirror of
https://github.com/rustfs/rustfs.git
synced 2026-11-10 12:19:07 +00:00
fbc7af7df05ecd16c4fce845fcf66cb096491c21
619 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris An
|
d6210ac17b |
fix(s3,ecstore,table-catalog): bound work driven by client input (#9172)
* fix(ecstore,heal,select): bound storage-layer work driven by client input Scan findings in the erasure store, heal, file metadata and S3 Select layers, grouped because they share the disk walk, the multipart staging area, delete recovery and the select snapshot lease: - Multipart: upload ids are canonicalized before they name a staging path or a lock key (backlog#3027); trailing-slash keys are encoded like every other handler (backlog#3055); unlisted parts are removed at completion (backlog#3106); per-bucket uploads are found through a signed per-disk index instead of a scan of the shared staging area (backlog#3121); encrypted part ETags carry a per-upload secret so a write-only user cannot confirm plaintext guesses (backlog#3044). - Delete and recovery: recovery walks are bounded per request (backlog#3038); a delete marker on a key whose xl.meta path is a directory, and sidecar names used as child keys, no longer wedge the journal (backlog#3035, backlog#3105); over-quota PUTs no longer list the bucket under the ledger lock (backlog#3128). - Listing: scan_dir sends metadata without inline data (backlog#3052), counts delete markers against max-keys (backlog#3108), and rejects a marker that carries the cursor tag (backlog#3059). - Heal: bucket metadata and heal records live in separate namespaces so DeleteBucket cannot erase a drive rebuild (backlog#3039), a deleted bucket is skipped instead of failing the rebuild (backlog#3040), and directory objects are encoded on readback (backlog#3061). - Restore and tiering: RestoreObject preserves tags and the version's modification time (backlog#3073, backlog#3074); Snowball transport headers are not forwarded to the tier (backlog#3090); an epoch modification time round-trips through xl.meta (backlog#3063). - Locks: PutObject takes its bucket locks after the body is staged (backlog#3131); SelectObjectContent releases its namespace locks once the snapshot is taken (backlog#3132). - RPC and peers: the v3 signature check consumes the v2 nonce (backlog#3129); a bucket-metadata reload timeout no longer marks the peer offline (backlog#3124); replication target requests carry an operation deadline (backlog#3102); the scanner resumes its interrupted wait after a lifecycle write (backlog#3118). - S3 Select: bounded schema inference, column cap and budgeted CSV batches, Parquet dictionary bounds and pre-planner projection checks (backlog#3075, backlog#3076, backlog#3077, backlog#3078). * test(e2e): seed legacy duplicate bucket tags offline for the OPA contract * fix(s3): authorize what handlers store and bound request-driven work Scan findings in the S3 request layer (access hooks, object use cases, HTTP server layers), grouped because they share the authorization condition builder, the PutObject admission path and the response layers. Builds on the storage-layer batch for the bounded recovery and multipart APIs it calls. Authorization and policy conditions: - copy source is percent-decoded and version-canonical before the s3:x-amz-copy-source condition (backlog#3135); CopyObject authorizes the tag set it copies (backlog#3034); HeadObject with the replication check header still authorizes s3:GetObject (backlog#3047); the x-rustfs-source-version-id header feeds s3:versionid (backlog#3072); recursive force-delete with a REPLICA header authorizes every descendant as a delete (backlog#3070); include-deleted listings require s3:ListBucketVersions (backlog#3099); the metadata listing route keeps table isolation and uses the same prefix it authorizes (backlog#3053, backlog#3054); If-Match probing needs read permission (backlog#3097); Object Lambda checks s3:GetObjectVersion (backlog#3058); replication proxy fallback keeps ExistingObjectTag denies (backlog#3089); HeadObject returns lock and tag headers only with permission (backlog#3113); IPv4 clients on the dual-stack listener match IPv4 SourceIp conditions (backlog#3049); bucket policies are parsed once and cached (backlog#3032); keystone: access keys no longer verify with an empty secret (backlog#3043). Admission and resource bounds: - GetObject releases disk-read permits when the client stops reading (backlog#3023); PutObject takes buffer and foreground admission only after body bytes arrive and bounds the wait (backlog#3031, backlog#3084); xl.meta size is bounded on the write path (backlog#3100); XML compat layers stream instead of buffering whole bodies (backlog#3094); tier GETs stream the remote body (backlog#3046); on-demand migration pulls release permits on unread GETs (backlog#3110); SSE-KMS key policy covers UploadPart and completion (backlog#3130); MinIO admin decryption runs Argon2id off the async worker (backlog#3024). Input validation and leaks: - replication headers are validated (source ETag shape, original size) (backlog#3092, backlog#3093); control characters in keys and lifecycle rule IDs are rejected (backlog#3101, backlog#3117); Snowball member mtime needs s3:ReplicateObject (backlog#3083); DeleteObjects checks Content-MD5 and admin/STS reject SigV2 replays with altered query or body (backlog#3037, backlog#3026); request logs redact presigned queries and handler spans skip credentials (backlog#3068, backlog#3107); SSE-C listings and completions withhold plaintext metadata and checksums (backlog#3120, backlog#3050); CORS resolves the bucket like s3s does (backlog#3104); PutBucketTagging rejects duplicate keys (backlog#3126); notifications fire after a client disconnect (backlog#3127); ListenNotification streams re-check access (backlog#3119). * fix(api): drop imports left unused by the base merge * test(s3): move the KMS multipart denial test to the app layer The test drives app-layer multipart use cases, so it belongs with them; living in the SSE module it imported app code from the infra layer and failed the layer dependency guard. The SSE test helpers it borrows are exposed through the storage and app facades. * fix(s3): keep the SSE-C key fingerprint on keyless multipart completion The backlog#3050 fix withheld both the plaintext checksums and the customer-key MD5 header from a CompleteMultipartUpload that did not present the key. The checksums are a digest of the plaintext and stay hidden; the key MD5 identifies the key only, and the S3 conformance suite (test_copy_part_enc) expects it on a keyless completion, so it is echoed again. * fix(tier): stream GET bodies and document replication budgets * fix(ecstore,s3): keep the multipart index and cold fill off disk lifetimes Two lifecycle regressions surfaced once the storage and request batches ran together against main: - The multipart index owner cloned the disk's pinned publication root and bootstrapped on a task of its own. That kept the mount alive past the last LocalDisk owner (the disk lifecycle allows exactly three holders of the root FD) and left a stray task behind, which the local-disk and read-repair tests detect. The owner now keeps only the root path and reopens it for identity checks, and the bootstrap scan rides the disk's existing cleanup task, where stopping the disk abandons it and a reconnect resumes it. - The disk-read admission wrapper defers releasing its permit until an in-flight read completes, so a cancelled client cannot widen disk concurrency. The cold-fill producer, however, drops a read it has already classified as stalled when its deadline fires, and that permit stayed pinned for the lifetime of the stuck read. The producer now abandons the admission explicitly on timeout. * test(ecstore): pass the index owner to the Linux cleanup-loop test * fix(s3): release cold-fill admission after watchdog handoff * fix(s3,ecstore): close the round-1 adversarial review findings Adversarial review of the combined request- and storage-layer batch found fixes that fell short of their issues, two behaviour regressions for legitimate clients and several unrequested costs: - backlog#3038: a bulk DeleteObjects reserved the whole per-disk inspection allowance up front for every key in flight and exhausted the 16 Mi request budget on any set wider than eight drives, failing healthy keys with SlowDown. Calls are now admitted while budget remains and charged for the entries they actually visited. Object-scoped discovery charges plain child keys at one entry per 64 names instead of one each, so deleting a missing key that names a folder prefix completes instead of answering 503 forever while a hostile prefix stays bounded. A peer still on the previous binary is reported as an unsupported participant, the repository's rolling-upgrade contract, instead of as a budget verdict. - backlog#3037/#3026: a SigV2 configuration PUT without Content-MD5 was still replayable with another body; it is refused with MissingContentMD5 while SigV4 keeps the optional-MD5 contract. - backlog#3072: DeleteObject now resolves x-rustfs-source-version-id before authorization like PUT and COPY, so a replica delete cannot remove a version the policy never saw. - backlog#3121: the multipart index no longer fsyncs a marker and four directories per multipart metadata write (it is rebuilt on every start), and ListMultipartUploads discovers roots concurrently. - backlog#3077: Parquet SelectObjectContent decodes budget-sized batches instead of one row per batch. - backlog#3031: a full upload buffer pool hands out an unpooled buffer instead of refusing the upload; the progress floor already bounds stalls. - backlog#3023: the GET disk-read idle cut-off is configurable (RUSTFS_GET_DISK_READ_IDLE_TIMEOUT_SECS, default 120 s) so a paused client is not truncated after 30 s. - backlog#3075: JSON Lines schema inference keeps the sample collected at the 4 MiB bound instead of failing the query. - backlog#3024: compat admin decryption queues briefly for a slot instead of refusing the third concurrent request. - backlog#3113: HEAD skips its permission probes for the bucket owner. - backlog#3117: control characters in a lifecycle rule ID are rejected on PutBucketLifecycle as well. The lock-crate publication FIFO serves backlog#3111 and moves to the table-catalog PR. Also fixed: a lossy admin error message, an unused s3-client function, two unused test-util features, a duplicated key validation, the s3s footprint baseline, and a stale log event name in the reverse-proxy runbook. The ODM multipart write-back test runs on the large test stack like its sibling. * fix(table-catalog): bound maintenance work and isolate table namespaces (#9174) * fix(table-catalog): bound maintenance work and isolate table namespaces Scan findings in the Iceberg table catalog: - Metadata maintenance deleted every unreferenced *.metadata.json in a table's directory without checking whether a renamed table in another namespace still owned those files (backlog#3060); it now resolves the owning table through the directory index before deleting. - Manifest lists and manifests were decoded with the embedded Avro writer schema and no recursion depth limit; a self-referencing record schema overflowed the stack (backlog#3080). Schemas are validated for depth before decoding. - Compaction decoded every input Parquet file into Arrow batches held in memory; a decompression bomb OOM-killed the server (backlog#3085). Row groups are streamed through a byte budget and oversized inputs are refused. - An S3 request for a key outside an indexed table prefix listed the bucket's whole namespaces/ catalog prefix and failed once it held more than 4096 objects (backlog#3086); a per-bucket directory index answers index misses without the listing. - The maintenance planner read whole orphan files to obtain their modification time (backlog#3087); it reads metadata only. - A client disconnect between persisting active_rename_id and clearing it left the bucket's catalog entry locked at 503 (backlog#3088); the rename runs detached from the request future and clears the flag on every exit. - Table commits held the bucket publication fence until publication, starving every object write (backlog#3111); commits now hold it under a deadline and object writes are admitted between commits. The directory index is managed through three new admin routes under {warehouse}/catalog/directory, registered in the admin route policy. The durable-strong store is left as is (its snapshot partitioning for backlog#3109 and the strong-path deadline are not ported): that backing is being retired separately, so the new tests cover the object backing. * fix(table-catalog): close the review gaps in the scan-fix batch Adversarial review of the table-catalog batch found the fixes narrower than the issues they close and two cost regressions: - backlog#3085: the compaction footer check bounded nothing a page header could not override; parquet sizes its page buffers from the header. Walk every column chunk's page headers (Thrift compact, no decompression) and hold each page and the chunk total to the budget before decoding. The stored-input budget is per rewrite group, so a multi-group job no longer fails after writing earlier outputs. - backlog#3086: the default index-miss scan still counted maintenance reports toward its 4096-object limit; only table entries count now, and the listing itself is bounded to 64 pages. The directory index remains an optional accelerator. - backlog#3111: a create wrote its metadata file under the fence as the first durable mutation, which disarmed the deadline before the unbounded registration inventory. The write precedes publication now. The deadline gets its own knob (RUSTFS_TABLE_CATALOG_PUBLICATION_PREPARATION_TIMEOUT_SECS, default 30 s, never below the lock-acquire timeout) instead of reusing the 5 s lock timeout, and the ownership inventory reads entries concurrently. - Directory candidates are written only while a directory exists; the backfill covers earlier entries. A directory on the migration-fence path is reported as a conflict instead of a parse error. - Dead test hooks, a stale comment and the support-matrix text that did not match the code are removed or corrected. * fix(select): preserve complete samples and Parquet batch limits * fix(s3,ecstore): close the round-2 adversarial review findings DeleteObject resolved x-rustfs-source-version-id for authorization on every request while storage consumes it only for a REPLICA request, so a non-replica caller with s3:DeleteObjectVersion could trigger a plain delete of the current version. The hook now resolves the header under the same predicate as storage, and the regression test covers the non-replica case. HEAD no longer skips its permission probes for the bucket owner; an explicit bucket-policy Deny on the metadata actions binds the owner too. Multipart index pages verify each root against the staging area and prune markers whose staging root is gone, under the marker's own lease with a re-check, so the per-drive index stays proportional to live uploads instead of every key ever multipart-initiated on the drive. SigV2 configuration PUTs that declare an empty body need no Content-MD5: there is no body to replay. The minimum-progress floor stays independent of RUSTFS_HTTP_REQUEST_BODY_READ_TIMEOUT; the knob's documentation now says so instead of claiming 0 disables it. The quota bootstrap window (503 until the background scan publishes the ledger) is kept, because ledger admission must not come from scanner reporting; the compatibility matrix now documents it. The S3 compatibility matrix gains a section for the client-visible validation bounds, and the drive-timeout tuning guide tabulates RUSTFS_GET_DISK_READ_IDLE_TIMEOUT_SECS. * fix(lock): queue table-bucket publication writers fairly Table commits take the bucket publication fence as a write lock; under a stream of commits the lock's try-and-retry acquisition let a new commit jump ahead of object writers that had been waiting longer, which is part of what starved them (backlog#3111). Requests for the publication lock path now take a FIFO ticket per shard: a waiter is admitted only when it is at the head of the queue, cancelled waiters release their turn, and batch acquisitions join the same queue. Other lock paths are unchanged. This moved here from the request-layer batch, where no cited issue covered it; it serves the table-catalog issue this PR closes. * fix(table-catalog): close the round-2 adversarial review findings Parquet page-header walker: read the size fields by field id as parquet does, accepting I16/I32/I64 wire kinds and rejecting any other kind, so a header encoding uncompressed_page_size as I64 can no longer bypass the budget while parquet allocates it. Bound every list, set and map count by the bytes remaining in the chunk, and reject bool collection elements (parquet skips them with zero width, so either width would desynchronize the two parsers); this removes the 2^63-iteration spin on LIST<BOOL>. Start each column-chunk walk at the dictionary page offset whenever the footer declares one, exactly as parquet's byte_range does, instead of min(dictionary, data). Footer pre-walk: before parquet decodes a footer, walk FileMetaData with the hardened cursor and charge the declared row_groups count times the RowGroupMetaData size to the decode budget, because parquet 59 reserves that Vec before reading one element. Preparation deadline: clamp the RUSTFS_TABLE_CATALOG_PUBLICATION_PREPARATION_TIMEOUT_SECS knob to 86,400 seconds so Instant + Duration cannot overflow. Index-miss scan: after each listing page, jump the cursor past the maintenance subtree the page ended in, and allow one extra page per table entry already listed, so job-report history scales with the catalog while a report flood under one table stays bounded. Migration fence: read the fence-path object once and decide from its stored kind whether it is directory state or a fence. Expired or failed create: remove the initial metadata file when the registration fails before any catalog write may have begun; nothing else references it. Correct the comment and support matrix that claimed maintenance reclaims it. Tests cover each walker divergence, the footer row-group reservation, the deadline ceiling, the paging budget for honest catalogs and for a report flood, and the removal of the unpublished metadata file. * test(admin): register the catalog directory routes in the route matrix The table-catalog batch added GET, POST and DELETE {warehouse}/catalog/directory on both the Iceberg and the compat admin prefixes without listing them in the admin route matrix, so test_admin_route_matrix_matches_registered_routes failed on CI. * fix(s3,ecstore): close the round-3 adversarial review findings Multipart index pages now treat the cursor as the last examined marker rather than the last root, so a page that pruned a stale marker still validates and ListMultipartUploads keeps using the index. A marker is written before its staging root is published, so pruning spares markers younger than a five-minute grace period and an existing marker is touched on every new upload; the prune re-checks both under the marker lease. Bounded delete-recovery inspection rides the durable-mutation lane like the unbounded path instead of the one-worker foreground lane. The on-demand-migration inline body budget scales with the object size (at least 64 KiB/s) instead of a flat 120 s. The control-body digest service honors RUSTFS_HTTP_REQUEST_BODY_READ_TIMEOUT=0 as "no inactivity timeout" like every other reader of that knob. Quota reconciliation throttle objects are deleted together with a retired generation's ledger, so bucket turnover leaves no residue. The PUT guard test shim is gone; tests call the shared guard. Bounded DiskAPI defaults answer MethodNotAllowed like their siblings. Stale budget docs, an unreachable CSV arm, an always-Some limit parameter, commented-out constants and a misplaced doc comment are cleaned up; weak test assertions name the expected error; lifecycle rule-ID control characters gain a test. The compatibility matrix records the CopyObject tag-copy permission, the exact SigV2 empty-body predicate, replica and migration writes of control-character keys, and the SigV2 DeleteObjects digest rule. * fix(table-catalog,heal): close the round-3 review findings A. Pack compaction rewrite groups to min(max-rewrite-bytes-per-job, TABLE_COMPACTION_MAX_STORED_BYTES), reject an over-ceiling group before any output is written, and name the ceiling in the commit-path errors; document the effective cap in the support matrix and scheduler runbook. B. Keep TaskCancelled/TaskTimeout classification on replacement passes: the namespace verification runs only for completed or storage-failed passes, in the erasure healer and the task bucket prepass. C. Compare the replacement bucket pass per bucket: a recorded identity that is gone or replaced still errors or restarts the pass, buckets created mid-pass are appended to the identities, pending_buckets and a persisted positional order without rewinding progress or the checkpoint; verification ignores additions. D. Read the migration-fence object once in the object-backed write permit and dispatch on its kind. E. Drop the strong store's copy of maintenance_cleanup_roots. F. Remove the unused test hooks fail_put_prefix, corrupt_put_prefix and pause_after_metadata; make compact_parquet_data_files and PublicationTicket.state private. G. Document the body-revision identity, the legacy revalidation budget and its HTTP 503 in the support matrix. H. Define HEAL_REPLACEMENT_RECOVERY_DIR once in ecstore and consume it from heal through the storage_api facade. * fix(ecstore): cap the index pages one disk may answer Pruned multipart index pages may carry few roots, so the page cursor no longer bounds how many pages a peer can make a listing fetch. Cut a disk off after 4096 pages (about one million examined markers) so it contributes no coverage and the listing falls back to the staging scan. * fix(ecstore,table-catalog): close the round-4 adversarial review findings Multipart index markers dated in the future (a backwards clock step between recording a marker and publishing its staging root) were judged old and could be pruned while their upload was in flight; a future-dated marker is now kept. The marker touch on EEXIST is best effort, so a filesystem that refuses to set times no longer fails the upload commit. Pruning runs on the durable-mutation lane instead of the one-worker foreground lane. Quota reconciliation keeps one throttle object per bucket incarnation (a quota revision change replaces it instead of orphaning the previous revision's object), the retirement cleanup deletes it ahead of the ledger and also when the ledger is already gone, and a scan the retirement already refused writes nothing back. Bounded DiskAPI defaults answer UnsupportedDisk, the verdict the aggregate gives a peer without the protocol. The compaction ceiling pre-check runs over every rewrite group before any output is written, and the worker's ceiling error no longer names a placeholder group. Docs: the inline on-demand-migration client budget, the protocol-2 downgrade conflict of replacement passes, the exact footprint numbers. |
||
|
Henry Guo
|
00a238f838 |
fix(table-catalog): resolve superseded staged commits (#9124)
fix(table-catalog): diagnose superseded staged commits Co-authored-by: Chris An <anzhengchao@gmail.com> |
||
|
Hauser Zhi
|
ed4099ebe1 |
chore(deps): update crates and fix storage CI regressions (#9111)
* chore(deps): refresh crates and adopt openidconnect 4.1 Use published OIDC/OAuth2 releases and reject malformed HMAC signature lengths before upstream verification. Refresh workspace and fuzz locks, and advance the local zstd compatibility patch to 0.13.4 while retaining the fixed 0.14.3 codec backend. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * chore(deps): refresh compatible lockfile resolutions Resolve Prost build and derive dependencies with itertools 0.14 and prefer windows-sys 0.61.2 within the existing supported version ranges. Retain all package versions, sources and checksums. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(connect): separate cleanup checks from short probe windows Use the functional measurement budget for the cleanup HTTP fixture so setup and measured I/O can finish before checking delete status handling. Exercise both normal scheduling and an in-flight PUT clock advance, and include target classification in unexpected-failure diagnostics. Retain the cleanup deadline, outer timeout and delete-stage assertions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): isolate stalled multipart disk timeouts Apply the short injected timeout only to the deliberately stalled disk. Healthy disks retain their normal operation budgets, while logical lock, physical owner, immediate CopyObject and final shard assertions remain. Keep overrides scoped across cloned and reconnected handles, preserve shorter caller deadlines and explicit zero timeouts, and report drive health when the real CopyObject assertion fails. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(rebalance): retry metadata write lock quorum admission Recognize typed write-lock quorum failures for rebalance.bin within the existing bounded metadata admission retry policy. Each attempt must acquire quorum before loading or saving metadata; data writes are not replayed by this retry. Cover recovered and persistent quorum failures, cancellation after a failed attempt, and rejection of unrelated resource or read-lock errors. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser Zhi
|
ecbddc188a |
docs(scanner): define prefix statistics contract (#9116)
Define the scope, authority, coverage, persistence, and resource guarantees needed before scanner-derived current-object statistics can be enabled. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser
|
e0444b519d |
feat(ecstore): add sampled batch CPU diagnostics (#9019)
* feat(ecstore): add sampled batch CPU diagnostics Keep timing and request correlation in a separate default-off feature. Preserve the existing positioned-read backend, byte and error contracts. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): route batch diagnostics through storage facade Keep the diagnostic implementation private to its owner and expose explicit feature-gated helpers through the existing storage boundary. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): retain diagnostic IDs across GET producer tasks Capture request context before spawning the legacy producer. Keep the existing producer, cancellation and read-lock lifetime intact. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): avoid unused batch read index Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(obs): isolate local batch diagnostic JSON events Omit span metadata and inferred parent IDs for diagnostic root events. Verify per-shard index order and document existing production metric timing. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): accelerate receipt fixture in CI Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(heal): allocate stack for bucket recovery fixture Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(s3): allocate stack for conditional GET fixture Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
cxymds
|
4a63e53161 |
test(validation): run canonical cases and build pinned Heal tools (#8455)
* test(validation): add pinned auto-testing manual lane * Install pinned ILM client for canonical validation lane * Prepare Python client for pool validation cases * Install validation client for replication cases * test(validation): provide the Python client for Heal cases * test(validation): build the candidate-bound Heal inspector artifact * test(validation): expose tool artifacts through the existing manual lane * test(validation): verify pinned Heal tools before fleet execution * test(validation): verify Heal tools against released shard captures * ci: audit retained validation environments before finalization * test(validation): add optional native Heal regression tooling * test(validation): route native tools through the existing driver * Run validation tool resolution on hosted Linux without acquiring the fleet lock * test(validation): bind C07 to pinned native authority evidence * test(validation): build optional C10 fixture codec on hosted Linux * test(ci): bind C10 execution to the verified shared codec * feat(ci): expose read-only validation fleet inventory * docs(ci): describe supported validation recovery * test(ci): add isolated manager native validation builds * test(ci): expose manager regressions through tools-only dispatch * ci(validation): consume pinned manager native prerequisites * fix(heal): preserve typed bucket errors for bounded retries * fix(heal): preserve typed errors and SSE-C response headers * ci(validation): publish isolated immutable candidate packages * test(validation): enforce private candidate manifest schema * test(validation): verify isolated package publication failures * ci(validation): dispatch isolated packages through the driver * fix(heal): deeply verify read repair payload failures * test(ecstore): verify incarnation fence at repair commit * ci(validation): verify C11 native storage evidence * fix(ci): isolate validation package publication credentials * test(ci): archive scoped F07 directory fsync evidence * fix(ecstore): retain typed quorum errors for missing parts * chore(ecstore): shrink typed-error ratchet baseline * fix(ecstore): retain below-quorum data during no-remove heal * fix(ci): allow recovery of completed cancelled validation runs * fix(heal): retain exhausted admin batch repair receipts * fix(observability): retain typed object error diagnostics * fix(s3): map typed RPC unavailability to service refusal * fix(filemeta): classify supported inline framing damage as corrupt * test(ecstore): exercise heal during committed overwrite cleanup * ci(validation): admit source-bound G01 native evidence before fleet execution * test(ecstore): fence version races and bind cleanup slots * ci(validation): require complete race native runtime preflight * test(ecstore): fence marker and batch deletes during heal * ci(validation): require G04 native and runtime admission * test(ecstore): fence multipart continue complete and abort during heal * test(ecstore): cover default heal owners and copy metadata races * ci(validation): admit source-bound owner and copy race proofs * test(scanner): assert typed corruption after metadata truncation * fix(ci): require complete native admission for multipart races * test(validation): wire I05 queued-successor native prerequisites * fix(scanner): bind object heals to the scanning disk scope * test(scanner): reuse storage boundary for scope regression * style: format RPC digest precedence assertion * fix(heal): query automatic tasks on their owning node * test(admin): reuse error facade in heal query regression * fix(heal): retain cancellation after successful bucket storage I/O * fix(ci): retain validation evidence outside runner temp * fix(heal): retain resolved object identity on terminal failures * test(scanner): adapt scope regression to admission receipts * ci(validation): inspect shared runner without fleet access * ci(validation): prepare Python before creating client environments * ci(validation): isolate each client virtual environment * ci(validation): retain interrupted evidence without fleet recovery * ci(validation): bind recovery to retained runner evidence * ci(validation): wire persistence and lock source plans * fix(validation): align read-only dispatch defaults --------- Co-authored-by: Chris <anzhengchao@gmail.com> |
||
|
Hauser
|
a9d3022f37 |
fix(ecstore): prove multipart part sizes before cleanup (#9037)
* fix(ecstore): prove multipart part sizes before cleanup Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(ecstore): merge strict multipart observation passes Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test: respect lifecycle architecture boundaries Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test: fix multipart E2E clippy warnings Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
RustFS
|
7581a70412 |
feat(s3): serve Veeam SOSAPI system and capacity XML (#9007)
* feat(s3): serve Veeam SOSAPI system and capacity XML Authenticated GET and HEAD of the two well-known SOSAPI keys return generated XML, and read preconditions use that generated ETag and Last-Modified. Mutations of those keys are rejected while the feature is on. Co-authored-by: RustFS <hello@rustfs.com> * fix(s3): simplify Veeam quota metadata lookup Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(s3): harden Veeam SOSAPI capacity and reads Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(s3): bind Veeam capacity to bucket identity Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(s3): protect virtual objects and coalesce refreshes Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): refresh Veeam profile selections Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): bound pagination fixture writes Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: Hauser <housemecn@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Chris An
|
f6ad79dd58 |
feat(plugins): add minimal original-user runner integration (#9017)
* feat(plugins): add minimal original-user runner integration * test(plugins): use the main storage fixture * fix(plugins): reuse runner connections across requests * test(plugins): allocate stack for delegation fixture |
||
|
Chris An
|
7a3bf8cdbd |
fix(ecstore): resolve dangling latest versions in metadata updates (#9009)
* fix(ecstore): select the latest version at read quorum like MinIO GET and HEAD escalated latest-version selection to write quorum when enough metadata answered and skipped a newer version below write quorum on degraded reads. LIST and heal already pick the newer side of a split write at read quorum, so such keys listed but answered 503 until heal. Select with MinIO's getObjectFileInfo rule: the most common mod time that reaches the object's data blocks wins, ties go to the newer time, and one content identity at that time must reach the same quorum. Two identities tied at the selected time fail instead of resolving by hash order, and the selected identity's version-stack shape follows disk order. * fix(ecstore): resolve selection ties by disk order and guard writes Review of the read-quorum selection found three gaps: - Rejecting a same-time half split made explicit-version reads of a partially applied tagging update fail and blocked the retry that repairs it. Resolve ties to the identity on the lowest-index disk, for reads and heal alike, instead of rejecting or using map order. - Delete markers and empty versions can lower the common read quorum below the selected version's data blocks, so HEAD could report a version GET cannot decode. Require that many copies, as MinIO's getLatestFileInfo does. - A selection can now return exactly read-quorum disks, and partial metadata writes are not rolled back. Refuse an update before writing when its target disks cannot reach write quorum. * fix(ecstore): count identity copies in the data-blocks guard The first-candidate shortcut counts byte-identical copies, so copies that differ only in identity-neutral fields such as replication status were missing from the data-blocks check. When the shortcut's count falls short, count every copy of the selected content identity before refusing the version. * fix(ecstore): keep commit-proof selection for system-bucket reads Pool metadata's PREPARE/COMMIT recovery and identity epochs treat an ordinary read of pool.bin as proof that the record committed. Under the read-quorum rule, a COMMIT that reached only half of a 2+2 set was served as committed, and the pool_meta_native_physical_* integration tests failed. Route reads through select_fileinfo_for_read: S3 buckets keep MinIO's read-quorum rule, while is_meta_bucketname reads use the previous algorithm unchanged as select_committed_fileinfo, so a split stays unreadable until native recovery resolves it. * fix(ecstore): serve the latest readable version past dangling versions A failed versioned PUT whose rollback also failed can leave the new version on top of fewer drives than its data blocks. Reading only each drive's top version then left the committed version below it short of read quorum, so GET and HEAD of the latest version answered 503 although that version is intact in every stack. MinIO resolves the stacks in pickLatestQuorumFilesInfo. When every drive answered and an S3 latest read still cannot select a version, walk the newest version headers of each stack through exact-version reads: serve the readable version with the newest selected modification time, skip versions whose exact read proves them dangling (the NoSuchVersion an explicit GET reports) and queue heal for them, and keep the read-quorum error on any other outcome. Candidates are ordered by each version's newest modification time across drives, and at most eight exact reads are spent. Test PUTs now wait for every drive so tests that edit one drive's metadata no longer race the PUT's background tail writes. * fix(ecstore): resolve dangling latest versions in metadata updates A failed versioned PUT whose rollback also failed can leave the new version on fewer drives than its data blocks. GET and HEAD of the latest version already look past such versions, but read-before-write callers without a version id still answered 503: object tagging read each drive's top version with the quorum-error read, and put_object_metadata and the CopyObject source read selected from read_all_xl, whose merged latest version is missing on the other drives, so object_quorum_from_meta failed before selection. Object tagging now reads through a variant that runs the same walk past dangling versions as a latest GET while keeping the quorum error for a dangling key. put_object_metadata and the CopyObject source try the walk when their quorum or selection step fails with read quorum, before the existing dangling cleanup could report a key with a committed version absent. A version-only copy takes the served version's own copies and write quorum. The walk still applies only to S3 buckets, latest reads, and sets where every drive answered, and the update still targets only the served version's disks, so a version below write quorum is refused before writing. * fix(ecstore): refuse version-only copies below write quorum up front Review of the dangling-latest metadata updates found three gaps: - A version-only CopyObject wrote the new version and reverted on a write-quorum miss; refuse it before writing when the served version's disks cannot reach its write quorum, like update_object_meta_with_opts. - Replication status write-back, including the zero-write ValidateOnly mode, now keeps its read-quorum error and never walks past dangling versions, since the walk queues heal for the versions it skips. - The erasure-coding doc overstated that these callers never report a dangling key as absent; put_object_metadata and CopyObject keep their existing dangling cleanup when the walk finds no readable version. |
||
|
Chris An
|
f879946ead | chore: remove redundant review skills (#9022) | ||
|
Chris An
|
d88f524d9f |
fix(ecstore): serve the latest readable version past dangling versions (#9004)
* fix(ecstore): select the latest version at read quorum like MinIO GET and HEAD escalated latest-version selection to write quorum when enough metadata answered and skipped a newer version below write quorum on degraded reads. LIST and heal already pick the newer side of a split write at read quorum, so such keys listed but answered 503 until heal. Select with MinIO's getObjectFileInfo rule: the most common mod time that reaches the object's data blocks wins, ties go to the newer time, and one content identity at that time must reach the same quorum. Two identities tied at the selected time fail instead of resolving by hash order, and the selected identity's version-stack shape follows disk order. * fix(ecstore): resolve selection ties by disk order and guard writes Review of the read-quorum selection found three gaps: - Rejecting a same-time half split made explicit-version reads of a partially applied tagging update fail and blocked the retry that repairs it. Resolve ties to the identity on the lowest-index disk, for reads and heal alike, instead of rejecting or using map order. - Delete markers and empty versions can lower the common read quorum below the selected version's data blocks, so HEAD could report a version GET cannot decode. Require that many copies, as MinIO's getLatestFileInfo does. - A selection can now return exactly read-quorum disks, and partial metadata writes are not rolled back. Refuse an update before writing when its target disks cannot reach write quorum. * fix(ecstore): count identity copies in the data-blocks guard The first-candidate shortcut counts byte-identical copies, so copies that differ only in identity-neutral fields such as replication status were missing from the data-blocks check. When the shortcut's count falls short, count every copy of the selected content identity before refusing the version. * fix(ecstore): keep commit-proof selection for system-bucket reads Pool metadata's PREPARE/COMMIT recovery and identity epochs treat an ordinary read of pool.bin as proof that the record committed. Under the read-quorum rule, a COMMIT that reached only half of a 2+2 set was served as committed, and the pool_meta_native_physical_* integration tests failed. Route reads through select_fileinfo_for_read: S3 buckets keep MinIO's read-quorum rule, while is_meta_bucketname reads use the previous algorithm unchanged as select_committed_fileinfo, so a split stays unreadable until native recovery resolves it. * fix(ecstore): serve the latest readable version past dangling versions A failed versioned PUT whose rollback also failed can leave the new version on top of fewer drives than its data blocks. Reading only each drive's top version then left the committed version below it short of read quorum, so GET and HEAD of the latest version answered 503 although that version is intact in every stack. MinIO resolves the stacks in pickLatestQuorumFilesInfo. When every drive answered and an S3 latest read still cannot select a version, walk the newest version headers of each stack through exact-version reads: serve the readable version with the newest selected modification time, skip versions whose exact read proves them dangling (the NoSuchVersion an explicit GET reports) and queue heal for them, and keep the read-quorum error on any other outcome. Candidates are ordered by each version's newest modification time across drives, and at most eight exact reads are spent. Test PUTs now wait for every drive so tests that edit one drive's metadata no longer race the PUT's background tail writes. * chore(ci): shrink the error(format!) ratchet baseline for rebalance The rebalance control and meta modules now have fewer Error::other(format!) call sites than the baseline records, which fails the shrink-only ratchet in Quick Checks. Record the smaller counts. * fix(ecstore): preserve canonical order in latest-version fallback |
||
|
Chris An
|
f96c94f950 |
fix(listing): treat key markers as literal keys (#8978)
* fix(listing): treat version markers as literal keys * fix(listing): keep start-after and V1 markers literal * docs(listing): state cursor compatibility removal condition * fix(listing): resume protected table pages from raw keys * test(listing): keep protected pagination coverage in app layer |
||
|
Chris An
|
acebee4ea3 |
fix(ecstore): select the latest version at read quorum like MinIO (#8982)
* fix(ecstore): select the latest version at read quorum like MinIO GET and HEAD escalated latest-version selection to write quorum when enough metadata answered and skipped a newer version below write quorum on degraded reads. LIST and heal already pick the newer side of a split write at read quorum, so such keys listed but answered 503 until heal. Select with MinIO's getObjectFileInfo rule: the most common mod time that reaches the object's data blocks wins, ties go to the newer time, and one content identity at that time must reach the same quorum. Two identities tied at the selected time fail instead of resolving by hash order, and the selected identity's version-stack shape follows disk order. * fix(ecstore): resolve selection ties by disk order and guard writes Review of the read-quorum selection found three gaps: - Rejecting a same-time half split made explicit-version reads of a partially applied tagging update fail and blocked the retry that repairs it. Resolve ties to the identity on the lowest-index disk, for reads and heal alike, instead of rejecting or using map order. - Delete markers and empty versions can lower the common read quorum below the selected version's data blocks, so HEAD could report a version GET cannot decode. Require that many copies, as MinIO's getLatestFileInfo does. - A selection can now return exactly read-quorum disks, and partial metadata writes are not rolled back. Refuse an update before writing when its target disks cannot reach write quorum. * fix(ecstore): count identity copies in the data-blocks guard The first-candidate shortcut counts byte-identical copies, so copies that differ only in identity-neutral fields such as replication status were missing from the data-blocks check. When the shortcut's count falls short, count every copy of the selected content identity before refusing the version. * fix(ecstore): keep commit-proof selection for system-bucket reads Pool metadata's PREPARE/COMMIT recovery and identity epochs treat an ordinary read of pool.bin as proof that the record committed. Under the read-quorum rule, a COMMIT that reached only half of a 2+2 set was served as committed, and the pool_meta_native_physical_* integration tests failed. Route reads through select_fileinfo_for_read: S3 buckets keep MinIO's read-quorum rule, while is_meta_bucketname reads use the previous algorithm unchanged as select_committed_fileinfo, so a split stays unreadable until native recovery resolves it. |
||
|
Chris An
|
e452d837dd | docs(scanner): clarify incomplete census diagnostics (#9002) | ||
|
Hauser
|
cf5a9e0ba3 |
docs: define the dirty-journal metadata source contract (#8984)
* docs: define the dirty-journal metadata source contract Specify the single node-owned journal scope: verified bootstrap, physical fencing and tail completion, reader-first replay, exact new responsibility settlement and retained legacy unknowns. Explicitly distinguish the proposed contract from the existing asynchronous hint implementation. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * docs: require receiving peers to revalidate source claims Keep user metadata, decoded records and authenticated transport distinct from storage-verified dirty-journal authority. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser
|
d4d416ed47 |
fix(deps): upgrade russh and replace vulnerable zstd codecs (#8969)
* chore(deps): upgrade russh to 0.65.0 and refresh crates Run two cargo update and cargo upgrade rounds from origin/main. Adapt SFTP public-key callbacks to PublicKeyOffer and VerifiedPublicKey, preserving explicit rejection and password-only authentication. Cover the rejection policy with a pinned-host-key SSH handshake test. Refresh Arrow, Parquet, Brotli, ZIP, and compatible dependencies. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(deps): route legacy zstd consumers to fixed codecs Replace the old registry implementation with a private version-constraint adapter backed only by zstd 0.14.3 and zstd-safe 8.2.0. Preserve legacy features and Avro, Arrow IPC, Parquet, and Kafka formats. Cover abandoned contexts, legacy decode errors, seekable bounds, stored frames, consumer interoperability, and resolved implementation versions. Record the adapter removal condition in the compatibility register. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Chris An
|
fca714efe8 |
fix(listing): omit sub-quorum versions from ListObjectVersions (#8964)
Version listings bypassed listing quorum: SetDisks::list_path returned early into list_versions_authoritatively, whose resolver seeds unread slots as DiskNotFound. With any one drive of the set offline, every version held by fewer than ceil(n/2) drives (for example a deleted version kept by drives that were offline during the delete) failed the whole page with ErasureReadQuorum, surfaced as S3 503. Route merged version pages through the listing-quorum branch of SetDisks::list_path, as MinIO's listPath does with requestedVersions = 0, so such versions are omitted instead of failing the page. Keep the set-level floor: fewer than ceil(n/2) online drives still fails the page. Set walks (tier reference proofs, tier-delete journals, pool maintenance) and the recursive prefix-delete scan act on absence, so they keep the authoritative per-version scan and fail closed. |
||
|
Chris An
|
65bd2d575b |
refactor(listing): make cursors plain keys and opaque V2 tokens (#8950)
* refactor(listing): make cursors plain keys and opaque V2 tokens RustFS inherited MinIO's metacache cursor tag and appended `[rustfs_cache:v2,id:<uuid>,src:walker,gen:live]` to every internal continuation marker, although nothing ever consumed the id: four parsers had drifted apart, V1 NextMarker had to be stripped on the way out, and ListObjectVersions leaked the tag onto the wire as NextKeyMarker. ecstore stops assigning `list_id` in `ECStore::list_path` and `SetDisks::list_path_result`, drops `append_list_cache_id_to_marker` and the `cache_id` parameters of the pagination helpers, and removes the unused `ListPathOptions::encode_marker`; internal markers are the plain last key, last common prefix or last raw scanned key. The API layer builds the V2 NextContinuationToken as base64 of the server-side cursor in one place and decodes it in one place; V1 NextMarker is the decoded cursor and the `public_list_marker` tag stripper is gone; NextKeyMarker is the plain key. Base64 of the plain key is what nodes from earlier releases already read, so a rolling upgrade needs no switch. Compatibility, registered as `listing-cursor-legacy-tag`: `parse_marker` and `list_marker_key` still strip a trailing legacy tag from incoming markers so a client that paginates across the upgrade keeps its position. * fix(listing): preserve literal cache-tag cursor boundaries |
||
|
Hauser
|
fea5a584c3 |
fix(storage): fence multipart and metadata recovery (#8919)
* perf(ilm): back off idle tier-delete journal recovery Use bounded idle backoff only after complete empty journal and dispatch scans, while preserving immediate notifications and the retry cadence for observed work. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): reconcile interrupted pool metadata publication Bind pool metadata CAS and Undo to exact native predecessors. Recover an interrupted physical publication only from an ordinary committed authority and complete verified identity, current and backup evidence, then require normal V3 CAS and quorum readback. Preserve known-conflict gates, startup and heal lock ordering, targeted admission and the original kill-recovery deadline. Wake proof-ready recovery immediately and bound parallel capacity sampling without removing mutation fences. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(ecstore): reuse proven pool metadata durability Publish native rollback backups directly in their owned UUID directories. Reuse a one-shot data-durability receipt only for an exact hardlink to the pinned, fully synchronized head and root. Preserve new-head and directory synchronization, native predecessor CAS, conditional Undo and uncertainty gates. Keep unknown, copied, raw-parent, cancelled and failed operations cold. Add bounded diagnostics, sync-call regressions and path-isolated test hooks. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(multipart): exercise selective RPC delay and peer loss Add authenticated method-entry barriers for metadata and lifecycle locks under e2e-test-hooks. Verify real dispatch, write quorum, prior part payloads, lock drain and incarnation isolation without changing production budgets. Record cancellation separately from normal exits so late filesystem completion cannot overwrite Drop evidence. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * docs: define multipart delay gates and receipt invariants Document RPC entry evidence, peer-loss and cancellation oracles, remaining incident attribution limits, and immutable head inodes required for durability receipts. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(readiness): expose missing runtime drive slots Report configured slots without a runtime handle separately from unavailable inventoried drives. Preserve quorum and probe status semantics. Wait for every coordinator to regain its complete live inventory before multipart RPC fault windows and single-attempt recovery oracles. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): retry committed promotion convergence safely Require peer reload convergence on every worker attempt, including unchanged retries. Recheck durable identity, replicas, namespace fence, generation and cancellation before allowing any data movement. Keep known committed promotions active on a verified propagation pause. Release precise owners, retain queued successors and reuse the existing watchdog. Preserve uncertainty gates and terminal operation behavior. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): guard native metadata mutation aliases Reject protected RenamePart source and destination paths before I/O. Classify the protected namespace through the native path resolver, and require canonical conditional RenameData and Undo keys so aliases cannot create separate lease domains. Preserve ordinary multipart and canonical CAS behavior, warm durability receipts and zero-allocation canonical fast paths. Cover real alias, byte identity, receipt and normal mutation behavior. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): register Darwin multipart RPC delay gates Update distributed membership from the actual Darwin listing. Preserve the full-profile digest and leave Linux unchanged until its own listing is produced, as required by the contributor guide. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(multipart): fence native publication and recovery Publish multipart data through private hardlinks and atomically consume the prepared source head. Bind exact compensation capabilities and native recovery to the physical cohort, source generation and durable publication state. Require fresh terminal confirmation for native replay and preserve indeterminate data when no safe recovery witness exists. Release logical guards only after validated irreversible quorum success while retaining physical mutation leases, accounting and resource ownership through actual callbacks. Preserve local instance proof authority and historical Copy metadata without granting healed donors fresh publication authority. Keep general movement identity strict while preserving directory capacity compatibility in a separate checked comparison. Cover native fault and cancellation boundaries, zero-byte objects, real Copy, heal and migration, exact physical readback and distributed RPC delay. Retain all quorum and durability requirements and document rollout limitations. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): fix ratchet scan and Windows fault hooks Move native pool metadata inventory tests after production items so the error-format ratchet sees the existing call site. Preserve the baseline instead of accepting a false count decrease. Make the directory-barrier failure hook available on every test platform and exercise the parent-barrier failure/retry cases in Windows CI. Keep production synchronization behavior and all existing assertions intact. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(protos): remove redundant recovery request clone Move the baseline into its final struct update while retaining the earlier clone needed to construct the first request variant. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): align RPC delay gates with native abort * fix(storage): validate native source recovery and CI gates Preserve wire layouts while bounding native response sizes and fix the last-use recovery request clone. Avoid destination replay reads only for fully observed initial native sources; keep upgraded and recovered sources conservative. Cover real legacy publication after a cold restart. Align upgrade acceptance with the native mixed-version mutation pause and use exact upload-scoped test completion evidence before advancing time. Keep all original quorum, ownership, byte and deadline assertions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): verify native abort under partial admission Delay native cleanup at the exact upload root after all-participant closure. Verify quorum-only admissions retain every source byte on refused Abort and complete the same upload without creating a missing original participant. Keep sub-quorum orphans and their discoverable identities isolated while a fresh all-participant upload proves service recovery. Route shared constants through the existing E2E storage facade. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): avoid formatting a fixed invalid upload path Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): align native MPU fault recovery assertions Verify real source participants, preserved orphan bytes and same-upload completion after quorum-only admission. Require full physical inventories so an expected read-quorum error cannot hide unrelated leakage. Preserve ordinary quorum and recovery budgets, and document listing availability through the affected set. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): keep MPU tail deadlines on the wall clock Global Tokio time advancement can expire drive-health probes while real filesystem callbacks are pending. Use a short real drive deadline with the existing physical-entry and cleanup barriers, retaining the byte, lock, resource-permit and final-shard assertions. Fixes #8937 Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(storage): align native fixtures with instance ownership Open independent disks for the simulated capacity-loss instance, scope four coordinator journal disks without rebinding source authority, and probe configured namespace keys while retaining physical I/O hooks. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): isolate native multipart future construction Construct optional native publication futures behind synchronous heap boundaries so ordinary rename callers do not carry their state machines. Preserve native guards and ownership without adding allocations to ordinary PUT, copy or heal branches. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(heal): assert coordinator journal scope after rejoin Check the exact original four-member journal map after every rejoin and manager stop while preserving all sixteen native data slots. Reconnect expansion fails the test rather than being filtered away. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(heal): bound object repair future construction Construct the normal and FileCorrupt deep-retry state machines behind a synchronous heap boundary. Preserve guards and cancellation while bounding the shared rename and deletion call stacks without increasing test stacks. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): qualify fixture disk identity checks Call the same DiskAPI method explicitly so the independently reopened disk identity assertion compiles under every target and feature combination. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): isolate native inspection from mutation state Select a dedicated read-only inspection future behind the existing single Box allocation. Preserve validation, source ownership and the canonical inspection body while keeping mutation state out of inline read callers. Reject misrouted operations with typed errors and retain source evidence. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): pause abort time after native cleanup Wait for durable source deletion on the wall clock before advancing the capacity lease clock at the post-delete barrier. Paused time can otherwise expire the barrier while native filesystem cleanup is still pending. Retain lease-loss injection, durable inflight-byte accounting and restart reconciliation assertions. Document the test-util feature required to select the native decommission fixtures in package-scoped regressions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): reclaim data from owned multipart undo Retain exact runtime-only Undo ownership through deferred snapshot cleanup. Revalidate every target metadata version and the retirement anchor, keep the delete claim through the physical callback, and remove the anchor after the payload barrier. Preserve generic cleanup refusal when metadata is missing. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): preserve unformatted bootstrap routes Keep explicitly unformatted local paths reachable by an in-flight elected formatter while quarantining their service slots and disk-ID membership. Own bounded legacy-read handles so the actual asynchronous creator future remains Send. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(s3): bound CopyObject tracing poll frames Construct the business future behind an outlined single-Box boundary and lazily instrument its handle. Preserve Send, Unpin, span timing and cancellation without embedding concrete business-state construction in the tracing poll. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(mpu): observe recovery after direct legacy bootstrap Seed all sixteen real previous-release sources before attaching the same-port method observers. Require exact per-peer disk-route Unimplemented responses with zero publication calls, retain byte and cleanup assertions, and remove superseded raw transport. Register the two new regressions from actual Linux and macOS listings. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(mpu): prepare the homogeneous previous-release fleet Reuse the existing release baseline before creating the genuine legacy multipart sources. Restart old processes with their normal budgets and require every drive through the reduced-redundancy probes before installing the mixed-version observers and current fault budgets. Keep standard EC:4, single-attempt calls, all mixed-version byte and reclamation assertions, and their original deadlines unchanged. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(mpu): isolate release compatibility from fault profiles Keep normal component configuration for the observed mixed-release rollout, which injects no RPC delay or loss. Retain the short component profile in actual method-fault scenarios and preserve the single-attempt 45-second probe and every typed error, routing, source byte and reclamation assertion. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(mpu): use normal shutdown for planned release replacements Use the existing bounded SIGTERM path during homogeneous release preparation and the observed mixed-version rollout. Preserve every multipart inventory, routing, quorum and byte assertion, and keep actual crash recovery on its original hard-kill path. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * revert: retain the verified release compatibility fixture Restore the published fixture after the local configuration and lifecycle experiments failed to resolve the macOS startup failures. Keep their diagnostic records separate from the delivered implementation. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(copy): bound inherent handler polling stack Build the inherent CopyObject future through a synchronous noinline factory at the existing trait await. This keeps its large state out of the trait wrapper's active polling frame during historical self-copy. Preserve cancellation, tracing and post-write notifications, with one additional allocation restricted to COPY. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): validate authoritative quota and native pool heals Require actual authoritative quota usage in the existing readiness loop instead of treating HTTP success or unknown usage as a numeric baseline. Keep the original quota assertions and convergence budgets. Allow the owning pool metadata CAS to republish into a fresh valid data directory while retaining exact version, layout, completeness and shard fingerprints. Require every node's real metadata/read/write readiness before accepting recovery. Keep ordinary object manifests and the original 180-second recovery deadline unchanged, with negative oracle regressions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): distinguish certified native Abort recovery Require native participants with a persisted full closure certificate to retire their heads and reclaim every surviving source on retry. Keep uncertified legacy authority below identity quorum conservative, while retaining actual partial-delete and committed-object byte checks. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> Co-authored-by: overtrue <anzhengchao@gmail.com> |
||
|
Hauser
|
c610d11e8f |
fix(storage): guard MPU recovery and startup RPC admission (#8934)
* fix(ecstore): preserve uncertainty after partial multipart abort Keep recovery callers from reporting NoSuchUpload while minority primary metadata remains. Retain all quorum and deletion checks. Exercise actual partial Abort and explicitly retained late RPC packets after caller retirement, staging cleanup and bucket recreation. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): prevent retired multipart staging resurrection Require the upload root to exist before preparing recovery publication. Preserve rooted directory walks for nested bases on non-Linux Unix and cover missing-source native work, configured aliases and pinned mixed-version coordinator recovery. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): use IPv4 peer aliases for pinned mixed recovery Keep direct IPv6 S3 listeners behind IPv4 RPC proxies for the pinned previous release, whose peer health resolver rejects bracketed IPv6 hosts. Preserve signed authorities, ports and strict recovery assertions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): separate pinned startup from MPU fault budgets Use the old fleet defaults during bootstrap, then apply the original short fault budgets to the current replacement. Keep direct S3 URLs aligned with IPv6 listeners and preserve all quorum and physical cleanup assertions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): seed legacy MPU through transparent transport Retire raw proxy connections before enabling method faults so legacy streaming setup stays byte-transparent and every later scoped mutation must traverse the fault barrier. Verify raw and adapted protocol authority, bodies and trailers. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): retain direct listeners for legacy MPU peers Advertise same-port localhost peers for the pinned resolver while retaining direct IPv4 S3 listeners. Exact scoped RPC hit counts still reject bypassed injection and all physical authority assertions remain unchanged. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(server): keep startup gRPC on the authenticated router Use the hybrid protocol predicate for startup RPC admission so early control responses retain production authentication and owner validation. Keep REST FullReady gating and pinned replay capability rejection intact. Add deterministic regressions, scoped diagnostics and bounded legacy fixture resources. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ci): verify MPU recovery across upgrade source versions Run the exact sixteen-drive mixed MPU case for every checksum-verified upgrade source, independently of generic compatibility failures. Require one listed test and retain per-version logs while preserving the historical distributed source pin and every recovery assertion. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ci): trigger upgrade MPU checks for fixture changes Include distributed fixtures and the RPC fault proxy in the upgrade workflow paths so test-only changes schedule the strict recovery gate. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(storage): verify resend owners across MPU finalization Hold real part-transaction RPCs while another coordinator completes or aborts the upload, with both live and cancelled client response waiters. Require full object bytes, terminal upload state, and all staging roots to remain consistent after the part owner drains. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser
|
299609bb83 |
fix(quota): keep reads available without snapshot-wide copies (#8928)
* fix(quota): keep quota reads available during scans Use optional scanner usage for read-only quota statistics across all quota configurations. Preserve strict legacy write admission and durable reservation enforcement while keeping unknown usage distinct from zero. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(usage): avoid snapshot copies for single-bucket reads Project owned snapshots, scalar bucket sizes and refresh-only results through the same TTL-bounded cache loader. Move loaded snapshots into the cache without changing refresh serialization or generation fencing. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser
|
017b85a5bf |
fix(ecstore): preserve inactive multipart abort authority (#8920)
Retain per-disk recovery metadata through partial publication and require write-quorum proof for completion replay. Keep completed uploads terminal and strengthen RPC recovery acceptance. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Chris An
|
b23723a51f |
feat(heal): add live node admission feedback (#8897)
* feat(heal): add live node admission feedback * test(heal): use storage facade in admission regression * fix(heal): preserve retryable node admission deferrals |
||
|
Chris
|
da7dc74147 |
fix(quota): retire degraded floors from durable usage reports (#8832)
* fix(quota): retire degraded floors from durable usage reports * docs(quota): clarify retained usage cache refresh behavior * fix(admin): report unknown bucket usage as null in GetBucketQuota GetBucketQuota serialized missing usage as size 0, which contradicted the D3 rule that unknown usage is never reported as zero and, after retiring degraded floors, made a bucket near its limit look empty. Make size optional and stop collapsing a missing or unreadable usage snapshot to 0. |
||
|
Chris
|
db55098aa6 |
fix(scanner): recover dual corruption with retained epoch fencing (#8833)
* fix(scanner): recover dual corruption with retained epoch fencing * fix(scanner): preserve recovery markers and legacy quota floors * test(scanner): expect retained usage recovery marker state * docs(scanner): register dual-corruption recovery evidence objects Add the .bloomcycle.bin.dual-recovery/ evidence family to the persisted objects table and describe its owner, write-once lifecycle and recovery rule in Deletion And Recovery Rules. |
||
|
Chris
|
47e1850d7e |
fix(scanner): preserve remote checkpoint coverage identity (#8827)
* fix(scanner): preserve remote checkpoint coverage identity * test(scanner): verify remote plan identity propagation * test(rpc): disambiguate scanner capability body collection |
||
|
Chris
|
4e88ebcf30 |
fix(heal): preserve typed replacement readiness deferrals (#8823)
* fix(heal): preserve typed replacement readiness deferrals * docs(heal): align retry compatibility markers with guards |
||
|
Chris
|
8af4070d19 |
fix(decommission): move the free version of an expired tier version (#8777)
* fix(decommission): move the free version of an expired tier version Decommission only queued the lifecycle expiry of a transitioned version, so a source cleanup that ran first deleted the version itself and appended its tier free version after the entry's last pass. The post-check then failed the decommission until an operator re-ran it. Apply the expiry of a transitioned version synchronously, as rebalance does. Its free version is on the source before the cleanup preflight, which reports the entry changed, and the retried entry moves the record to a target in the same pass. An expiry that cannot be applied leaves the version live, so it moves like any other version. Refs rustfs/backlog#2873 * fix(decommission): move a delete-all expired tier version live Applying a delete-all expiry inline made its prefix delete wait for the bucket lifecycle write lock behind the entry's own read lock until the lock timeout, blocking every write in the bucket meanwhile. Move such a transitioned version live and leave the delete-all to lifecycle on the target. Cover a genuinely unapplied expiry and a free-version worker that consumes the source record before the entry moves it. Refs rustfs/backlog#2873 |
||
|
Hauser
|
c1de42af2c |
fix(heal): type legacy MRF decisions and defer diagnostics (#8778)
* fix(heal): explain unverified legacy heal outcomes Preserve storage diagnostics in canonical unknown outcomes and explain legacy payload identity limits through the shared local/peer status path. Borrow expected receipt identities while preserving all proof checks. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(heal): keep diagnostics from pausing merged MRF repairs Require the actual MRF executor before publishing legacy pause events. An ordinary queued owner can retain a merged durable MRF notice without having performed the specialized deep check. Preserve its diagnostic without granting it authority to park the repair responsibility. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(heal): type MRF outcomes and defer status diagnostics Carry storage-owned legacy observations in process and return a typed terminal result only after the qualified MRF Deep check. Publish pause events from that result and exact identities instead of source or text. Keep object names out of retry classification and compute only the status diagnostic selected by the canonical execution state. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Chris
|
d2eda195e6 |
fix(decommission): defer source cleanup quorum failures (#8760)
* fix(decommission): defer source cleanup quorum failures A quorum failure while deleting the source of an already copied entry was fatal and marked the whole pool failed, although the copied source is still in place and an uncommitted delete transaction is settled by delete recovery. Carry the cleanup error typed and resolve a quorum failure as Deferred: the entry is replayed at the end of the round, and a deferral that persists takes the existing capacity-blocked pause path instead of a terminal failure. Other cleanup errors and quorum errors from other stages stay terminal. Add set-level regressions for parent and child keys (`a/b`, `a/b/c`): parent cleanup leaves every child file untouched, and parent cleanup reaches quorum while the child is cleaned up or rewritten concurrently. * test(ecstore): keep source cleanup errors typed in prefix cleanup tests Return SourceCleanupError from the test helper instead of formatting it into Error::other, and shrink the error(format!) ratchet baseline for core/pools.rs, which lost one call site in the cleanup failure change. * fix(decommission): retry cleanup quorum failures and count replays once Retry a source cleanup quorum failure in place, up to three attempts with the existing source-cleanup backoff, before deferring the entry. Carry the versions an entry already counted on its deferral, so the end-of-round replay does not add a fully counted entry to items_decommissioned and bytes_done again. A deferral that persists pauses the pool through the existing deferred drain path. When source cleanup caused it, persist a reason led by "source cleanup blocked" without the carrier error's free-drive wording, and report waitingReason "source_cleanup" instead of "capacity". Add a source_cleanup test fault stage and a routine test that drives a cleanup quorum fault through deferral and replay to completion. * fix(decommission): wait seconds between cleanup quorum retries Give the in-place source cleanup quorum retry a fixed floor of one and then two seconds, with the existing jitter on top, so it can ride out a drive reconnect or health-check blip instead of retrying within a few hundred milliseconds. The wait stays bounded and cancel-safe. Find a source cleanup pause reason through stage wrappers, as the capacity-blocked check does, and add a routine test where cleanup keeps losing quorum: the decommission pauses with a source cleanup reason, the bucket stays queued and the source stays in place. Document the additive waitingReason value source_cleanup and its compatibility with older nodes reading the same pool.bin. * fix(decommission): never retry a lost source cleanup fence The cleanup retry classifier used is_quorum_error(), which also matches NamespaceLockQuorumUnavailable. The delete commit fence reports a lost namespace lock or mutation fence with that variant, so a lost fence was retried in place and only stopped later at the pre-cleanup fence check. A lost fence means this worker no longer owns the key; it must stop without retrying. Retry and defer only drive quorum failures: ErasureReadQuorum, ErasureWriteQuorum, InsufficientReadQuorum and InsufficientWriteQuorum. --------- Co-authored-by: Hauser <housemecn@gmail.com> |
||
|
Chris
|
77a36d92a3 |
fix(decommission): let a failed start roll back its own activation (#8752)
* fix(decommission): let a failed start roll back its own activation A fresh decommission start whose peer reload failed could not roll back: the pool metadata stale-write guard from #6604 refuses any save that clears a persisted active or queued decommission, so the pool stayed active with no worker until the watchdog or a leader restart picked it up. The rollback save now carries the activation that start committed. The guard exempts a pool only while its persisted record still equals that activation field for field, so a worker checkpoint or a later start keeps the rejection. A canceled record is not an active decommission and is cleared by the rollback as well. Every other save path passes no activation and is guarded as before. A peer may already have reloaded the activation and started a worker before the rollback. Saves are now fenced by decommission generation: a write carrying a start_time is rejected when the persisted record belongs to a later start, or has no decommission state and was updated after that start_time, so such a worker cannot make the pool active again. Its queued promotion and progress checkpoint also adopt the persisted record and stop the worker. Refs rustfs/backlog#2873 * fix(decommission): fence terminal writes of a withdrawn decommission Terminal records carry no start_time, so the generation fence did not cover them: after a start rollback, a peer worker of the rolled back generation that failed (for example on a bucket completion) could still persist a failed record over the restored pool. The merge now rejects a failed, canceled or complete record over a persisted record without decommission state; every attempt persists decommission state first, so no legitimate path writes a terminal record there. Peers also drop a withdrawn decommission sooner. A pool metadata reload replaces a running decommission with a newer record that withdrew it, even under an active worker; bucket completion adopts such a record like promotion and progress checkpoints do; and a withdrawn progress checkpoint stops the worker instead of logging a failed save and moving more entries. Tests cover the peer failure mark, a rollback retry that sees a worker checkpoint land between attempts, and the reload withdrawal. A heal fixture that wrote complete records over a fresh pool.bin now persists the active attempt first. Refs rustfs/backlog#2873 * fix(decommission): fence peer writes after a retry start rollback A retry start replaces a failed or canceled record that keeps its unresolved ledger, so its rollback restores that terminal record rather than clearing the pool. The withdrawal checks only recognized a record without decommission state, and the generation fence ignored a persisted record without a start_time. A peer worker of the rolled back retry could therefore checkpoint the pool back to active. The generation fence now compares the persisted last_update whenever the persisted record has no start_time of its own (cleared, terminal or queued). The progress checkpoint also checks the generation against the record read under the pool metadata lock, as bucket completion already does, and the save-path withdrawal check treats a terminal record newer than the running view as a withdrawal, so the peer adopts it and stops. A reload still never lets a newer terminal record replace an active worker's view, so after a retry's rollback a peer worker stops at its next bucket completion or progress checkpoint rather than on reload. Tests cover the retry rollback against a peer checkpoint and direct write, and a peer bucket completion after a fresh start's rollback. Refs rustfs/backlog#2873 * fix(decommission): keep a retry rollback's restored record intact After a retry start rolls back, the restored record is failed, so a peer worker's own failure mark passed every merge guard and replaced it. A failure mark now saves through save_current_pool_meta_checked, which runs the withdrawal check against the record read under the pool metadata lock and adopts that record instead of writing. The withdrawal check compared the restored record with the worker's in-memory last_update, which a rejected save had already raised, so the peer never adopted the rollback and the watchdog kept resuming it. It now compares against the running generation (start_time), and a rejected unresolved-entry save undoes its in-memory change. save_current_pool_meta_for_decommission_start is also compiled for the test-util feature again; a test-util helper still calls it, which broke the workspace clippy and Connect builds. Refs rustfs/backlog#2873 * fix(decommission): keep an unresolved entry when its own save fails The rollback of a rejected unresolved-entry save also dropped the entry on transient failures such as a quorum or lock-fence error, so the following failure mark persisted a failed record without it and an operator could clear a decommission that skipped an unreadable object. Keep the entry and undo only the last_update bump, which is all a withdrawn worker needs to still see the withdrawal. Refs rustfs/backlog#2873 |
||
|
唐小鸭
|
444eb5ddba |
fix(scanner): admit dirty scope across the bucket-listing window and pace full-walk fallbacks (#8690)
Under sustained writes nearly every cycle fell back to a full walk: scope admission required the global dirty generation to stay unchanged across the bucket listing, and after a Complete cycle the dirty-usage wake ran the next full walk immediately, back to back. - Track dirty-map removals with a process-local removal generation and admit a scoped walk when no entry was removed or replayed during the listing (`covers_listed_pending`). Writes to listed buckets in the window join the start snapshot; unlisted buckets dirtied after the cutoff stay pending and are never cleared by compare-and-remove certification. The strict `covers_all_pending` still gates segment-reuse evidence. - Classify scope fallbacks (removal during listing, generation exhausted, dirty-bucket overflow, no listed dirty bucket, baseline unavailable, peer scope unavailable), log `scoped_scan_fallback`, and report them in usage_freshness as additive serde(default) fields. - Pace the cycle after a Complete fallback while dirty buckets remain on the capped retry schedule clamped to the cycle interval, with the dirty wake disabled; the paced timer wake still requests a scope. - Update the per-bucket convergence design doc. No persisted format or internode protocol change. Co-authored-by: Chris <anzhengchao@gmail.com> |
||
|
GatewayJ
|
4dba635d48 | fix(table-catalog): authorize warehouse location claims (#8736) | ||
|
Chris
|
7cc4a9dbd0 |
fix(ecstore): leave no source free version for a moved tier version (#8732)
* fix(ecstore): leave no source free version for a moved tier version Data-movement source cleanup deletes each source version separately and never set skip_tier_free_version, so every transitioned version that decommission or rebalance had already copied to a target pool left a new tier free version on the source. Decommission then failed its post-check for any pool with tiered objects; rebalance left the record behind. The record claims the remote object the target copy still reads and stays blocked behind that live reference. Decommission and rebalance now pass the identities of transitioned versions whose copy committed on a target, and the cleanup skips the free version for exactly those. A transitioned version that did not move, such as one lifecycle expired during the move, still gets its free version because that record is its only remote-delete owner. Refs rustfs/backlog#2873, rustfs/backlog#2872 * fix(ecstore): keep the free version of a superseded moved source Data movement treats a source null version as complete when a newer unversioned object already sits on the target, without copying it. That outcome returned the same Ok as a committed copy, so a restored transitioned null version superseded this way was recorded as transferred and lost its source free version, leaving its remote object with no owner. The copy paths now return DataMovementCopy::Committed or ::Superseded, and decommission and rebalance record only committed copies of transitioned versions. A superseded source keeps its free version. Tests cover the superseded restored null version for rebalance and decommission, a restored and a null transitioned version moving by rebalance, an entry with one lifecycle-expired and one moved version, the per-version skip with two versions of one key, and a directory object. The tiered stale-target routine test from #8724 now expects the decommission to complete. The ILM ownership contract and the decommission notes describe the moved, superseded, expired and legacy cases. Refs rustfs/backlog#2873, rustfs/backlog#2872 * test(ecstore): build the decommission ILM fixtures without s3s The superseded and expiry-mix decommission tests named s3s request and lifecycle types directly, which added pools_test.rs to the ecstore files that reference s3s and failed the footprint ratchet. Set the restore request through ObjectOptions, and store the lifecycle rule as bucket metadata XML and load it with get_expiry_configs, as the decommission routine does. * test(ecstore): pin restored decommission copies and multipart supersession Decommission recorded a committed data copy of a restored transitioned version without any test failing if the record were dropped; add a routine-level decommission of a restored version that checks the post- check passes, the source keeps nothing, and the target keeps the full remote tuple. Add the multipart variant of the superseded restored null version (rebalance), and a unit test that separates committed, superseded and mismatched targets in the overwrite resume adjudication. |
||
|
Chris
|
5313b72baa |
fix(decommission): run the leader-side start to completion (#8723)
* fix(decommission): run the leader-side start to completion A forwarded decommission start awaits `ECStore::decommission` inline in the RPC handler. When the forwarding node's RPC times out, the stream is reset and the handler future is dropped wherever it is. Dropped after `pool.bin` records the pool as active but before the peers reload and the worker is spawned, the start left the pool active with no worker and peers on the old metadata, and a retry was rejected as already active. An admin client that disconnects can do the same. Run the start in its own task and let the caller await only its result. The start then always reaches its end: it fails before persisting anything, or persists the pool, has the peers reload and spawns the local workers, rolling back if the reload fails. A retry still cannot start twice: it waits on the start gate held by the running start and either reports the busy gate or is rejected because the pool is active. `decommission` now takes `self: &Arc<Self>` and spawns workers on that store instead of looking up the global object store, which is the same store in production and makes the path testable; the unused `require_decommission_store` helper is removed. Refs rustfs/backlog#2873 * fix(decommission): detach the leader-side start from its caller |
||
|
overtrue
|
a55f2f671b |
fix(decommission): keep status reads and starts off the movement gate
The data-movement operation gate is a write-preferring RwLock. A scanner publication lease stores a read guard on the leasing node, and the coordinator's lease-scoped rename, delete and validation on that node queued a second read. A movement writer queued in between (a decommission start, a status refresh, a peer reload) then waited for the lease while the coordinator's second read waited for the writer, so both stalled until the 60 s lease TTL. Admin status reads also took the movement writer, so every status call waited for the slowest in-flight copy or cleanup and stalled new movement reads while it queued. - Lease-scoped target admission shares the read admission the lease already holds instead of queueing a new read. - Pool and rebalance status reads publish under the gates only when they are free; otherwise they merge the persisted metadata into a private copy and publish nothing. - A decommission start bounds its start, activation and movement gate waits with one 30 s budget and fails with a retryable lock timeout before persisting anything; the admin handler maps it to 503 with Retry-After. Refs rustfs/backlog#2873 |
||
|
Chris
|
6f582b0548 |
fix(heal): decide object absence from storage evidence, not source (#8697)
* test(heal): add failing regressions for a source-independent absent verdict Add the regression tests for backlog#2900 that fail on the current code, where the same object-level not-found is treated as deleted on the plain heal path but as recreate_failed on the recreate path, and where the erasure-set healer, the task predicates and the error helpers disagree on wrapped not-found and transient inputs. Also add the per-object disposition cache module with its unit tests, the manager admission tests for it, the compat register entry for the display boundary, and the admission-invariant notes. The module and the manager tests are not declared yet, so nothing is wired into the build. * fix(heal): decide object absence from storage evidence, not request source Replace the five heal error classifiers (erasure-set healer outcome mapping, task-level not-found and lock predicates, error.rs recovery helpers and the storage existence-check matcher) with one typed classify_heal_error. An object-level not-found from the locked storage read is Absent for every source and every recreate_missing value; the recreate path, the usage-observation and scanner object-dir whitelists are gone, and display-text matching survives only in the marked classify_error_display boundary. Add a bounded, expiring per-object disposition memory consulted before queue admission so a best-effort producer cannot requeue an absent or terminally failed object on every tick; durable MRF, admin, forced, high-priority and dry-run requests bypass it. The read-repair producer no longer forces recreate_missing. * fix(heal): keep rendered grace windows out of task retries A grace window rendered into a task message belongs to an attempt that already finished; its owner schedules the next one. Only typed grace errors replay the task, as before the classifier merge. Update the ecstore read-repair black-box test: the producer no longer forces recreate_missing. * fix(heal): classify ambiguous legacy delete residue as permanent explicitly * fix(heal): scope disposition memory and bound absence parsing |
||
|
Chris
|
f063a8e862 |
fix(readiness): hold startup readiness until peers re-admit the node (#8704)
A restarted node published FullReady from its own view while survivors still counted its drives unavailable (stale DNS for the old pod IP, then the Offline->Returning->Online hysteresis). A rolling update could then stop the next pod and cost a set its read quorum (#8669). Add a read-only PeerDriveAdmission internode RPC that reports how a peer sees another host's drives, and gate the startup FullReady transition until a strict majority of each erasure set's peers count this node's drives online. The gate shares the startup budget, releases after 15s of unreachable peers, skips immediately when older peers cannot answer, and can be disabled with RUSTFS_STARTUP_PEER_READMISSION_ENABLE=false. |
||
|
Chris
|
9a1f3d2c4c |
fix(quota): keep ledger-admitted writes independent of scanner usage (#8699)
Hard quota admission is owned by the durable reservation ledger (decision D3, rustfs/backlog#2872), but the request-path quota check still read scanner usage on every write and mapped `UsageUnavailable` to a retryable 503 even for buckets the ledger admits. For a durable quota the checker now treats scanner usage as reporting input: an unavailable lookup yields `current_usage = None` and `allowed = true`, counted and logged at debug level, and admin quota stats report the usage as null instead of zero. Buckets without a reservation protocol keep the fail-closed snapshot path. Quotas saved before the reservation protocols are re-saved under protocol v2 with the same limit, type and created_at. The write goes through a new guarded save that keeps the admin save's lifecycle/transaction guards, incarnation check and fleet capability proof and additionally refuses when the quota revision moved since the cached read, so a concurrent quota edit wins. It is triggered lazily from the quota read path (background, one attempt per bucket with a retry interval, never blocking the read) and in bulk by `POST /rustfs/admin/v3/quota-migration`, which reports every bucket. The ADR's Consequences section inventories the remaining scanner-usage "authority" consumers with their meaning after D3, retirement condition and compatibility window; the operations guide documents the migration and the report-only usage semantics. Refs rustfs/backlog#2904, rustfs/backlog#2872, rustfs/rustfs#8678, rustfs/rustfs#5716, rustfs/rustfs#7350 Co-authored-by: Hauser <housemecn@gmail.com> |
||
|
Chris
|
7c75f319bc |
fix(scanner): stop treating an ordinary PUT as data movement in cycle termination (#8696)
* fix(scanner): stop treating an ordinary PUT as data movement in cycle termination A continuously written deployment never reached a `Complete` scanner cycle (#8494, #8305, #7108). Three places in the termination path vetoed the cycle for an ordinary PUT; each now lets the write shade its bucket, the path per-bucket certification (#8542) built, and keeps the data-movement fences. 1. Usage publication is fenced by data movement only. `scanner_data_usage_publication_blocked` no longer includes `namespace_commits_pending()`, so the post-walk probe, the local defer decision, the post-lease recheck and the usage-store route probes stop superseding or deferring a cycle for a pending rename fanout. The scanner publication leases (acquire, validate, target guard) drop the pending-commit rejection and the namespace-generation staleness check and keep the movement generation, the operation gate, generation exhaustion and the purpose/target restriction. A test-util probe lets fixtures wait for tails directly. 2. The plan identity is split into a coverage identity and a reuse identity. `scanner_activity_coverage_digest` (peer set, protocol, topology, movement, and a remote peer's generations) is what a bucket checkpoint is started and resumed under, through `ScannerBucketScanPlan::checkpoint_coverage_digest`, `DataUsageCacheReuseOptions::checkpoint_plan_digest` and `prepare_bucket_checkpoint_with_coverage`. The reuse identity (`scanner_activity_structural_digest`) keeps `instance_id` and `maintenance_generation`, so a restart or a maintenance change still invalidates the baseline and the set caches; a completed sweep is stored under it. `publication_blocked` leaves every digest and stays a guardrail veto. A remote worker still resumes under the single digest on the wire. 3. A bucket walk that stops early no longer makes the cycle `Incomplete` when the bucket has a certified value. The set scan stands in with the bucket's flattened value from its previous complete set cache (`carry_partial_bucket_values`), the cycle reports it pending (`ScannerBucketScanStatus::PartialCarried`), keeps it in the dirty map, and the next scoped cycle walks it because the baseline's pending list joins the dirty buckets. A bucket with no certified value keeps the cycle `Incomplete`; failed buckets are unchanged. No persisted file, generation, receipt or environment variable is added. Refs #8494, #8305, #7108, rustfs/backlog#2903. * chore(ecstore): gate the namespace commit probe to test builds The scanner publication barrier no longer consults InstanceContext::namespace_commits_pending, so it has no production caller left; keep it for tests and the test-util facade only. * test(rustfs): observe namespace commit ownership directly in node_service tests The scanner usage publication barrier is movement-only now, so it no longer reports a pending rename. The node_service tests that assert a cancelled handler keeps its physical owner, or that wait for rename tails to drain, read the namespace commit state through a test-only ECStore probe instead. |
||
|
Hauser
|
05b483b31a |
fix(storage): preserve scanner checkpoints and bound MRF journal I/O (#8692)
* fix(scanner): preserve checkpoints on identity read failures Distinguish confirmed missing incarnation sidecars from lookup errors before cache preparation. Preserve durable checkpoint progress when identity reads are unavailable and report post-scan uncertainty separately from replacement. Reuse the transaction-fenced reader for optional and required lookups while retaining existing missing, malformed, system-bucket and lock-loss behavior. Verify preservation and recovery through a real coordinator with the identity sidecar write-locked, including unchanged cache revisions and dirty work. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(ecstore): share required incarnation conversion Keep the missing-incarnation error contract in one conversion used by both required lookup entry points. This removes a duplicate formatted error site and satisfies the existing error-format ratchet without changing its baseline. Pin the full missing-identity error text in the real SetDisks regression. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(heal): bound cross-disk MRF checkpoint I/O Run independent journal disks with bounded concurrency while retaining each replica's payload, companion and manifest ordering and the mirror phases. Share encoded bodies, deduplicate local roots and collect every result without changing receipt deadlines or persistence failure semantics. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Chris
|
f3cef98afc |
fix(scanner): anchor the bitrot cadence and budget per-disk I/O (#8701)
* test(scanner): pin Normal mode when no bitrot start is recorded Regression for rustfs/backlog#2905: three consecutive cycles that find no recorded bitrot start currently run Deep through the selection window, and a Normal cycle never records the cadence start. * fix(scanner): anchor the bitrot cadence and budget per-disk I/O Decide the scan mode from a recorded Deep window start only. A cycle that finds no recorded start runs Normal and anchors the bitrot cadence in process memory instead of opening the selection window; the Deep cycle that follows the elapsed cadence records the window start exactly as before. A missing record used to be read as a window that started at cycle 0, which ran Deep for deep_window_cycles cycles whenever the start could not be read back (rustfs/rustfs#8015). The recorded start is also remembered across a skipped or failed save, and /v3/scanner/status reports the cadence state under a new bitrot object. Add a per-local-disk background I/O budget (operations and bytes per second, defaults by detected media, auto|off|<rate>) that DynamicSleeper charges before directory listings and object metadata reads as a hard ceiling layered on the foreground-priority disk admission gate, with runtime config keys, per-disk status, cycle metrics, pacing pressure, usage_freshness, Prometheus series, Helm values, admin config help, and operator docs. Refs rustfs/backlog#2905, rustfs/rustfs#8015, rustfs/rustfs#8377. * test(scanner): expect Normal retries on erasure sets without a bitrot start The pending-PUT retry fixture asserted that a 4-drive set revisits every bucket because the cycle ran Deep. Without a recorded Deep window the cycle now runs Normal on every layout and reuses the cold inventory. |
||
|
Chris
|
2534b6175b |
fix(scanner): startup gates back off, rebuild and run degraded (#8695)
* fix(scanner): give startup gates automatic exits instead of a pause Every fail-closed scanner gate now has a bounded automatic exit without deleting quarantine evidence or relaxing the leader, CAS, movement or usage-floor fences (rustfs/backlog#2901). - Recovery retries back off instead of pausing: after the short budget the observation reads `backoff`, stays retryable, names the next attempt and is capped at ten minutes; one success resets the count and a 24 h window restarts the budget (rustfs/rustfs#8297, #8670). The usage-floor failure no longer carries the retry count across attempts. - A corrupt `.bloomcycle.bin` is rebuilt on the iteration after its quarantine marker is durable: the fenced leadership claim CAS-replaces the quarantined revision under a higher epoch and the marker stays as evidence. `future_schema` and `cleanup-pending` remain blocked. - An unclaimable pause ledger runs the leader degraded: cycles run untracked behind the data-movement fence, `persistence_state` reports `degraded`, and the claim is retried at most every five minutes (rustfs/rustfs#8251). - The usage-state full-rebuild reset starts from the empty baseline when every usage slot is corrupt and no cycle state exists (rustfs/rustfs#8670, #6951). - A cycle-state save reconciles non-conflict failures against durable state and retries under the same epoch; an exhausted save reloads durable state in place. Only a newer owner or an unreadable primary ends the iteration, so peers see a changed fence only when ownership changed (rustfs/rustfs#8141). * test(scanner): align usage-state reset tests with the empty-baseline rebuild The full-rebuild reset no longer refuses corrupt or identity-less usage slots when no cycle state exists (rustfs/rustfs#8670). The two tests that encoded that refusal now check the new contract: the reset starts from the empty baseline with a bootstrap-pending marker, never an authoritative zero floor, and a valid backup still supplies the recovery floor. --------- Co-authored-by: cxymds <cxymds@gmail.com> |
||
|
Chris
|
90749fbb06 | ci: run feature lanes nightly and warm caches on a schedule (#8683) | ||
|
Chris
|
22740876b3 |
docs(architecture): record scanner usage authority and PUT-tail decisions (#8678)
Supersede the 2026-09-03 decision: hard quota admission is owned by the durable reservation ledger, and scanner data usage is best-effort reporting input. Record that a pending ordinary PUT commit is a write-barrier event rather than a usage-publication veto, and split the scanner plan identity into a coverage identity and a reuse identity. Refs rustfs/backlog#2872, rustfs/backlog#2903, rustfs/backlog#2904. |
||
|
Hauser
|
6681dd12e3 |
fix(scanner): expose mixed sweep checkpoint transitions (#8657)
* fix(scanner): expose mixed sweep checkpoint transitions Correlate checkpoint admission, incomplete sweeps, and successful terminal Partial persistence by bucket/source and cycle/leader epoch. Share the traversal-position predicate and local/remote save diagnostic without changing certification, acknowledgement, or persistence contracts. Extend real-disk cache regressions across cycles and leader changes and verify that failed terminal saves retain an older periodic frontier until a successful terminal save permits fresh certification. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(scanner): report the aggregate incomplete sweep gate Expose the existing unresolved_objects decision in the incomplete-sweep event. Nested cache-entry failures can keep a sweep partial while both the root counter and the capped diagnostic failure map are empty. Extend the real-disk regression with diagnostics disabled and distinguish this case from mixed-only coverage without adding another cache walk. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Henry Guo
|
ee534fa19b |
test(table-catalog): validate current backing contracts (#8559)
* test(table-catalog): qualify current backing contracts * fix(table-catalog): verify smoke cleanup after object rename * fix(ci): include the baseline writer test in full E2E selection * fix(ci): reserve test capacity for disk-backed lock probes --------- Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com> Co-authored-by: overtrue <anzhengchao@gmail.com> |
||
|
Hauser
|
453be3dd84 |
fix(scanner): attribute incomplete census and lease waits (#8643)
* fix(scanner): attribute incomplete census and lease waits Report bucket sweep blockers and correlate partial remote work and rejected roots with their set and cycle. Keep distinct accepted bucket names in the result collector so completion counts and missing-bucket attribution share one source of truth, including during cancellation drain. Distinguish worker completion from coverage, retain peer/purpose context in lease failures without changing retry suffixes, and avoid claiming that an unverified leader fence changed or a waiting scan was complete. Document the census stages and preserve publication, dirty acknowledgement, remote failure classification and checkpoint safety rules. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(scanner): move set caches through the result collector Give the sole result collector exclusive ownership of the aggregate cache and return it together with the accepted bucket names after channel drain. Move that cache into complete persistence or incomplete updates instead of sharing a mutex and cloning the terminal cache. Return each accepted result's owned name from cache application so coverage tracking also avoids an extra name allocation. Preserve worker joins, cancellation drain, publication metadata and all certification fences. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
唐小鸭
|
222beaba00 |
fix(scanner): certify dirty usage per bucket and complete cycles with pending buckets (#8542)
* fix(scanner): treat replication status writebacks as status-class hits
A replication status writeback (persist_replication_state_if_current in
Update mode, including the heals the scanner queues during its own walk)
marked the bucket dirty on the namespace path: it advanced the dirty-usage
generation, woke the scanner, reached the durable dirty journal and
entered the activity snapshot. Any replicated bucket therefore made every
scanner cycle Superseded, the authoritative usage baseline was never
certified and usage freshness stayed stale for as long as replication ran
(#8375).
The writeback is now status class: record_status_class_mutation records
only the process-local producer coverage bit and nothing else. There is
no rescan queue for it, because a deployment with an active replication
rule already walks every bucket on every cycle
(ScannerMaintenanceFeatures::needs_regular_cycle) and that walk refreshes
replication_stats. The routing lives in a named, unit-tested function in
rustfs/src/storage/storage_api.rs; tier-expiration cleanup removes data
and stays on the namespace path.
No persisted format, protocol, admin API or configuration changes.
This is phase 1 of docs/architecture/scanner-per-bucket-convergence.md,
which this change adds together with its index entry.
Refs #8375, #8494, rustfs/backlog#2816.
* fix(scanner): refresh replication statistics during maintenance
Refresh compacted prefixes and completed bucket caches during replication maintenance. Bind set execution reuse to the observed replication statistics so a same-cycle retry cannot restore older counters.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(scanner): avoid expanding the legacy S3 DTO surface
Decode the existing replication configuration type from a JSON fixture instead of adding direct S3 DTO references to a new file. Preserve the compacted-prefix and same-cycle assertions and the footprint ratchet.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* feat(scanner): attribute activity guardrail vetoes per field
The post-walk cluster activity proof compared the whole activity snapshot,
so any namespace or dirty-usage generation movement vetoed publication.
Veto only on the fields that change which heap the walk described
(peer set, instance id, protocol version, topology, maintenance
generation, data movement, movement generation, publication blocked) and
name the field and host in the log and in a new usage-freshness counter.
Generation movement is a shade signal consumed per bucket. A remote peer
whose dirty-usage state moved keeps the whole-cycle outcome until scoped
acknowledgements can name its buckets.
* feat(data-usage): carry pending-bucket certification in DataUsageInfo
Add the additive fields a per-bucket certified scanner snapshot needs:
the sorted list and count of buckets whose dirty-usage generation moved
during the walk, and an explicit observational marker for snapshots whose
authoritative publication was vetoed. All three default when absent and
are skipped when empty, so older readers see the existing snapshot shape.
* feat(scanner): certify dirty usage per bucket and complete cycles with pending buckets
A structurally complete walk that the guardrail did not veto is now
`Complete` even when the dirty-usage generation moved during the walk.
Certification keeps the compare-and-remove semantics of
clear_dirty_usage_buckets: buckets whose generation still equals the start
snapshot are cleared, buckets written during the walk stay in the dirty map
and are published as the pending set, so usage_snapshot_converged becomes
`pending == 0`. The authoritative object is written for such cycles; the
observed route is selected by the explicit observational mark that vetoed
cycles carry instead of by the convergence flag, and the root publication
proof is read for every authoritative save. The post-lease activity recheck
compares the guardrail fields, with the peer dirty-usage fallback, instead
of the full snapshot digest.
* feat(scanner): accept unconverged complete publications as the cache baseline
complete_scanner_cache_baseline_plan_digest required the authoritative
snapshot to be converged, so a site whose cycles always left a bucket
pending could never scope a rescan and every cycle was a full walk. The
authoritative object is now a structural baseline whenever it carries a
convergence marker and no observational mark; bucket-cache reuse is then
governed by certification, because pending buckets stay in the dirty map
and are selected for the rescan while certified buckets reuse their set
cache. Observed candidates keep requiring the unconverged marker, and
segment reuse evidence keeps requiring a converged baseline.
Add the single-disk cycle test: a bucket written during the walk is the
only pending bucket, the others are certified, the publication is
authoritative and unconverged, and the next cycle is scoped to that
bucket; a held PUT tail still supersedes onto the observed route.
* feat(admin): expose scanner guardrail vetoes in usage freshness
Carry the attributed guardrail veto counter, timestamp and <field>@<host>
reason from the scanner metrics report into the madmin scanner status so
an operator can tell a rebalance or a PUT tail from a broken probe. The
fields are additive with serde defaults and merge like the deferral
fields.
* docs(scanner): record per-bucket certification in the design and changelog
Amend the convergence design where the implementation deviates: pending
buckets carry their walked value, maintenance_generation stays a veto,
guardrail localisation and the remaining phase-2 items ship separately,
last_cycle_result keeps its success label, and the admin surface gains the
guardrail veto fields. Update the publication contract for the post-lease
guardrail recheck and the explicit observational routing mark.
* refactor(scanner): drop the unreachable failed-bucket input from the pending set
A failed bucket never reaches a Complete cycle, so the pending set only
needs the start snapshot. Also remove a redundant clone in the data-usage
round-trip test.
* test(scanner): name the activity field mutation table type
* test(scanner): fill the post-walk gate in merged replication fixtures
The replication maintenance fixture merged from main builds a
ScannerCycleRequest literal and predates the test-only post_walk_gate
field; give it the default so the test target compiles on the rebased
branch.
* ci(nextest): run the admin target-repair fixtures on the ecstore base stack
repair_transaction_rejects_a_bucket_recreated_after_target_validation
overflowed the 2 MiB libtest default twice in a row on Linux CI for this
branch. Measured by sweeping RUST_MIN_STACK with the ci profile on macOS
arm64: main (
|