mirror of
https://github.com/rustfs/rustfs.git
synced 2026-11-07 11:10:47 +00:00
fbc7af7df05ecd16c4fce845fcf66cb096491c21
337 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 cxymds
|
fbc7af7df0 | fix(ecstore): preserve multipart shards within write budgets | ||
|
Hauser Zhi
|
32238a179c |
fix(server): bound rejoin waits and preserve recovery ownership (#9181)
* test(server): cover recovered-node admission boundaries Document backend readiness and write-availability checks during rejoin. Extend the existing regression through the production middleware order without changing runtime admission or cluster-quorum semantics. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(server): bound rejoin waits and preserve recovery ownership Enforce startup and unreachable-grace bounds while consuming peer replies. Release admission when every set has its required peer majority and cancel surplus read-only probes without carrying votes between rounds. Bound local readiness observations and polling sleeps. Retain initialized IAM across preparation failures, install the context once, and retry only publication after readiness timeouts. Share preparation with inline startup. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(server): make peer admission folding incremental Compile peer-to-set memberships once and retain compact per-edge votes. Consume each reply once, discard its payload, and update only affected set counters instead of rescanning all sets and retained responses. Recheck the captured deadline after polling and folding a reply so late work cannot admit the node or clear an expired unreachable grace period. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Chris An
|
d6210ac17b |
fix(s3,ecstore,table-catalog): bound work driven by client input (#9172)
* fix(ecstore,heal,select): bound storage-layer work driven by client input Scan findings in the erasure store, heal, file metadata and S3 Select layers, grouped because they share the disk walk, the multipart staging area, delete recovery and the select snapshot lease: - Multipart: upload ids are canonicalized before they name a staging path or a lock key (backlog#3027); trailing-slash keys are encoded like every other handler (backlog#3055); unlisted parts are removed at completion (backlog#3106); per-bucket uploads are found through a signed per-disk index instead of a scan of the shared staging area (backlog#3121); encrypted part ETags carry a per-upload secret so a write-only user cannot confirm plaintext guesses (backlog#3044). - Delete and recovery: recovery walks are bounded per request (backlog#3038); a delete marker on a key whose xl.meta path is a directory, and sidecar names used as child keys, no longer wedge the journal (backlog#3035, backlog#3105); over-quota PUTs no longer list the bucket under the ledger lock (backlog#3128). - Listing: scan_dir sends metadata without inline data (backlog#3052), counts delete markers against max-keys (backlog#3108), and rejects a marker that carries the cursor tag (backlog#3059). - Heal: bucket metadata and heal records live in separate namespaces so DeleteBucket cannot erase a drive rebuild (backlog#3039), a deleted bucket is skipped instead of failing the rebuild (backlog#3040), and directory objects are encoded on readback (backlog#3061). - Restore and tiering: RestoreObject preserves tags and the version's modification time (backlog#3073, backlog#3074); Snowball transport headers are not forwarded to the tier (backlog#3090); an epoch modification time round-trips through xl.meta (backlog#3063). - Locks: PutObject takes its bucket locks after the body is staged (backlog#3131); SelectObjectContent releases its namespace locks once the snapshot is taken (backlog#3132). - RPC and peers: the v3 signature check consumes the v2 nonce (backlog#3129); a bucket-metadata reload timeout no longer marks the peer offline (backlog#3124); replication target requests carry an operation deadline (backlog#3102); the scanner resumes its interrupted wait after a lifecycle write (backlog#3118). - S3 Select: bounded schema inference, column cap and budgeted CSV batches, Parquet dictionary bounds and pre-planner projection checks (backlog#3075, backlog#3076, backlog#3077, backlog#3078). * test(e2e): seed legacy duplicate bucket tags offline for the OPA contract * fix(s3): authorize what handlers store and bound request-driven work Scan findings in the S3 request layer (access hooks, object use cases, HTTP server layers), grouped because they share the authorization condition builder, the PutObject admission path and the response layers. Builds on the storage-layer batch for the bounded recovery and multipart APIs it calls. Authorization and policy conditions: - copy source is percent-decoded and version-canonical before the s3:x-amz-copy-source condition (backlog#3135); CopyObject authorizes the tag set it copies (backlog#3034); HeadObject with the replication check header still authorizes s3:GetObject (backlog#3047); the x-rustfs-source-version-id header feeds s3:versionid (backlog#3072); recursive force-delete with a REPLICA header authorizes every descendant as a delete (backlog#3070); include-deleted listings require s3:ListBucketVersions (backlog#3099); the metadata listing route keeps table isolation and uses the same prefix it authorizes (backlog#3053, backlog#3054); If-Match probing needs read permission (backlog#3097); Object Lambda checks s3:GetObjectVersion (backlog#3058); replication proxy fallback keeps ExistingObjectTag denies (backlog#3089); HeadObject returns lock and tag headers only with permission (backlog#3113); IPv4 clients on the dual-stack listener match IPv4 SourceIp conditions (backlog#3049); bucket policies are parsed once and cached (backlog#3032); keystone: access keys no longer verify with an empty secret (backlog#3043). Admission and resource bounds: - GetObject releases disk-read permits when the client stops reading (backlog#3023); PutObject takes buffer and foreground admission only after body bytes arrive and bounds the wait (backlog#3031, backlog#3084); xl.meta size is bounded on the write path (backlog#3100); XML compat layers stream instead of buffering whole bodies (backlog#3094); tier GETs stream the remote body (backlog#3046); on-demand migration pulls release permits on unread GETs (backlog#3110); SSE-KMS key policy covers UploadPart and completion (backlog#3130); MinIO admin decryption runs Argon2id off the async worker (backlog#3024). Input validation and leaks: - replication headers are validated (source ETag shape, original size) (backlog#3092, backlog#3093); control characters in keys and lifecycle rule IDs are rejected (backlog#3101, backlog#3117); Snowball member mtime needs s3:ReplicateObject (backlog#3083); DeleteObjects checks Content-MD5 and admin/STS reject SigV2 replays with altered query or body (backlog#3037, backlog#3026); request logs redact presigned queries and handler spans skip credentials (backlog#3068, backlog#3107); SSE-C listings and completions withhold plaintext metadata and checksums (backlog#3120, backlog#3050); CORS resolves the bucket like s3s does (backlog#3104); PutBucketTagging rejects duplicate keys (backlog#3126); notifications fire after a client disconnect (backlog#3127); ListenNotification streams re-check access (backlog#3119). * fix(api): drop imports left unused by the base merge * test(s3): move the KMS multipart denial test to the app layer The test drives app-layer multipart use cases, so it belongs with them; living in the SSE module it imported app code from the infra layer and failed the layer dependency guard. The SSE test helpers it borrows are exposed through the storage and app facades. * fix(s3): keep the SSE-C key fingerprint on keyless multipart completion The backlog#3050 fix withheld both the plaintext checksums and the customer-key MD5 header from a CompleteMultipartUpload that did not present the key. The checksums are a digest of the plaintext and stay hidden; the key MD5 identifies the key only, and the S3 conformance suite (test_copy_part_enc) expects it on a keyless completion, so it is echoed again. * fix(tier): stream GET bodies and document replication budgets * fix(ecstore,s3): keep the multipart index and cold fill off disk lifetimes Two lifecycle regressions surfaced once the storage and request batches ran together against main: - The multipart index owner cloned the disk's pinned publication root and bootstrapped on a task of its own. That kept the mount alive past the last LocalDisk owner (the disk lifecycle allows exactly three holders of the root FD) and left a stray task behind, which the local-disk and read-repair tests detect. The owner now keeps only the root path and reopens it for identity checks, and the bootstrap scan rides the disk's existing cleanup task, where stopping the disk abandons it and a reconnect resumes it. - The disk-read admission wrapper defers releasing its permit until an in-flight read completes, so a cancelled client cannot widen disk concurrency. The cold-fill producer, however, drops a read it has already classified as stalled when its deadline fires, and that permit stayed pinned for the lifetime of the stuck read. The producer now abandons the admission explicitly on timeout. * test(ecstore): pass the index owner to the Linux cleanup-loop test * fix(s3): release cold-fill admission after watchdog handoff * fix(s3,ecstore): close the round-1 adversarial review findings Adversarial review of the combined request- and storage-layer batch found fixes that fell short of their issues, two behaviour regressions for legitimate clients and several unrequested costs: - backlog#3038: a bulk DeleteObjects reserved the whole per-disk inspection allowance up front for every key in flight and exhausted the 16 Mi request budget on any set wider than eight drives, failing healthy keys with SlowDown. Calls are now admitted while budget remains and charged for the entries they actually visited. Object-scoped discovery charges plain child keys at one entry per 64 names instead of one each, so deleting a missing key that names a folder prefix completes instead of answering 503 forever while a hostile prefix stays bounded. A peer still on the previous binary is reported as an unsupported participant, the repository's rolling-upgrade contract, instead of as a budget verdict. - backlog#3037/#3026: a SigV2 configuration PUT without Content-MD5 was still replayable with another body; it is refused with MissingContentMD5 while SigV4 keeps the optional-MD5 contract. - backlog#3072: DeleteObject now resolves x-rustfs-source-version-id before authorization like PUT and COPY, so a replica delete cannot remove a version the policy never saw. - backlog#3121: the multipart index no longer fsyncs a marker and four directories per multipart metadata write (it is rebuilt on every start), and ListMultipartUploads discovers roots concurrently. - backlog#3077: Parquet SelectObjectContent decodes budget-sized batches instead of one row per batch. - backlog#3031: a full upload buffer pool hands out an unpooled buffer instead of refusing the upload; the progress floor already bounds stalls. - backlog#3023: the GET disk-read idle cut-off is configurable (RUSTFS_GET_DISK_READ_IDLE_TIMEOUT_SECS, default 120 s) so a paused client is not truncated after 30 s. - backlog#3075: JSON Lines schema inference keeps the sample collected at the 4 MiB bound instead of failing the query. - backlog#3024: compat admin decryption queues briefly for a slot instead of refusing the third concurrent request. - backlog#3113: HEAD skips its permission probes for the bucket owner. - backlog#3117: control characters in a lifecycle rule ID are rejected on PutBucketLifecycle as well. The lock-crate publication FIFO serves backlog#3111 and moves to the table-catalog PR. Also fixed: a lossy admin error message, an unused s3-client function, two unused test-util features, a duplicated key validation, the s3s footprint baseline, and a stale log event name in the reverse-proxy runbook. The ODM multipart write-back test runs on the large test stack like its sibling. * fix(table-catalog): bound maintenance work and isolate table namespaces (#9174) * fix(table-catalog): bound maintenance work and isolate table namespaces Scan findings in the Iceberg table catalog: - Metadata maintenance deleted every unreferenced *.metadata.json in a table's directory without checking whether a renamed table in another namespace still owned those files (backlog#3060); it now resolves the owning table through the directory index before deleting. - Manifest lists and manifests were decoded with the embedded Avro writer schema and no recursion depth limit; a self-referencing record schema overflowed the stack (backlog#3080). Schemas are validated for depth before decoding. - Compaction decoded every input Parquet file into Arrow batches held in memory; a decompression bomb OOM-killed the server (backlog#3085). Row groups are streamed through a byte budget and oversized inputs are refused. - An S3 request for a key outside an indexed table prefix listed the bucket's whole namespaces/ catalog prefix and failed once it held more than 4096 objects (backlog#3086); a per-bucket directory index answers index misses without the listing. - The maintenance planner read whole orphan files to obtain their modification time (backlog#3087); it reads metadata only. - A client disconnect between persisting active_rename_id and clearing it left the bucket's catalog entry locked at 503 (backlog#3088); the rename runs detached from the request future and clears the flag on every exit. - Table commits held the bucket publication fence until publication, starving every object write (backlog#3111); commits now hold it under a deadline and object writes are admitted between commits. The directory index is managed through three new admin routes under {warehouse}/catalog/directory, registered in the admin route policy. The durable-strong store is left as is (its snapshot partitioning for backlog#3109 and the strong-path deadline are not ported): that backing is being retired separately, so the new tests cover the object backing. * fix(table-catalog): close the review gaps in the scan-fix batch Adversarial review of the table-catalog batch found the fixes narrower than the issues they close and two cost regressions: - backlog#3085: the compaction footer check bounded nothing a page header could not override; parquet sizes its page buffers from the header. Walk every column chunk's page headers (Thrift compact, no decompression) and hold each page and the chunk total to the budget before decoding. The stored-input budget is per rewrite group, so a multi-group job no longer fails after writing earlier outputs. - backlog#3086: the default index-miss scan still counted maintenance reports toward its 4096-object limit; only table entries count now, and the listing itself is bounded to 64 pages. The directory index remains an optional accelerator. - backlog#3111: a create wrote its metadata file under the fence as the first durable mutation, which disarmed the deadline before the unbounded registration inventory. The write precedes publication now. The deadline gets its own knob (RUSTFS_TABLE_CATALOG_PUBLICATION_PREPARATION_TIMEOUT_SECS, default 30 s, never below the lock-acquire timeout) instead of reusing the 5 s lock timeout, and the ownership inventory reads entries concurrently. - Directory candidates are written only while a directory exists; the backfill covers earlier entries. A directory on the migration-fence path is reported as a conflict instead of a parse error. - Dead test hooks, a stale comment and the support-matrix text that did not match the code are removed or corrected. * fix(select): preserve complete samples and Parquet batch limits * fix(s3,ecstore): close the round-2 adversarial review findings DeleteObject resolved x-rustfs-source-version-id for authorization on every request while storage consumes it only for a REPLICA request, so a non-replica caller with s3:DeleteObjectVersion could trigger a plain delete of the current version. The hook now resolves the header under the same predicate as storage, and the regression test covers the non-replica case. HEAD no longer skips its permission probes for the bucket owner; an explicit bucket-policy Deny on the metadata actions binds the owner too. Multipart index pages verify each root against the staging area and prune markers whose staging root is gone, under the marker's own lease with a re-check, so the per-drive index stays proportional to live uploads instead of every key ever multipart-initiated on the drive. SigV2 configuration PUTs that declare an empty body need no Content-MD5: there is no body to replay. The minimum-progress floor stays independent of RUSTFS_HTTP_REQUEST_BODY_READ_TIMEOUT; the knob's documentation now says so instead of claiming 0 disables it. The quota bootstrap window (503 until the background scan publishes the ledger) is kept, because ledger admission must not come from scanner reporting; the compatibility matrix now documents it. The S3 compatibility matrix gains a section for the client-visible validation bounds, and the drive-timeout tuning guide tabulates RUSTFS_GET_DISK_READ_IDLE_TIMEOUT_SECS. * fix(lock): queue table-bucket publication writers fairly Table commits take the bucket publication fence as a write lock; under a stream of commits the lock's try-and-retry acquisition let a new commit jump ahead of object writers that had been waiting longer, which is part of what starved them (backlog#3111). Requests for the publication lock path now take a FIFO ticket per shard: a waiter is admitted only when it is at the head of the queue, cancelled waiters release their turn, and batch acquisitions join the same queue. Other lock paths are unchanged. This moved here from the request-layer batch, where no cited issue covered it; it serves the table-catalog issue this PR closes. * fix(table-catalog): close the round-2 adversarial review findings Parquet page-header walker: read the size fields by field id as parquet does, accepting I16/I32/I64 wire kinds and rejecting any other kind, so a header encoding uncompressed_page_size as I64 can no longer bypass the budget while parquet allocates it. Bound every list, set and map count by the bytes remaining in the chunk, and reject bool collection elements (parquet skips them with zero width, so either width would desynchronize the two parsers); this removes the 2^63-iteration spin on LIST<BOOL>. Start each column-chunk walk at the dictionary page offset whenever the footer declares one, exactly as parquet's byte_range does, instead of min(dictionary, data). Footer pre-walk: before parquet decodes a footer, walk FileMetaData with the hardened cursor and charge the declared row_groups count times the RowGroupMetaData size to the decode budget, because parquet 59 reserves that Vec before reading one element. Preparation deadline: clamp the RUSTFS_TABLE_CATALOG_PUBLICATION_PREPARATION_TIMEOUT_SECS knob to 86,400 seconds so Instant + Duration cannot overflow. Index-miss scan: after each listing page, jump the cursor past the maintenance subtree the page ended in, and allow one extra page per table entry already listed, so job-report history scales with the catalog while a report flood under one table stays bounded. Migration fence: read the fence-path object once and decide from its stored kind whether it is directory state or a fence. Expired or failed create: remove the initial metadata file when the registration fails before any catalog write may have begun; nothing else references it. Correct the comment and support matrix that claimed maintenance reclaims it. Tests cover each walker divergence, the footer row-group reservation, the deadline ceiling, the paging budget for honest catalogs and for a report flood, and the removal of the unpublished metadata file. * test(admin): register the catalog directory routes in the route matrix The table-catalog batch added GET, POST and DELETE {warehouse}/catalog/directory on both the Iceberg and the compat admin prefixes without listing them in the admin route matrix, so test_admin_route_matrix_matches_registered_routes failed on CI. * fix(s3,ecstore): close the round-3 adversarial review findings Multipart index pages now treat the cursor as the last examined marker rather than the last root, so a page that pruned a stale marker still validates and ListMultipartUploads keeps using the index. A marker is written before its staging root is published, so pruning spares markers younger than a five-minute grace period and an existing marker is touched on every new upload; the prune re-checks both under the marker lease. Bounded delete-recovery inspection rides the durable-mutation lane like the unbounded path instead of the one-worker foreground lane. The on-demand-migration inline body budget scales with the object size (at least 64 KiB/s) instead of a flat 120 s. The control-body digest service honors RUSTFS_HTTP_REQUEST_BODY_READ_TIMEOUT=0 as "no inactivity timeout" like every other reader of that knob. Quota reconciliation throttle objects are deleted together with a retired generation's ledger, so bucket turnover leaves no residue. The PUT guard test shim is gone; tests call the shared guard. Bounded DiskAPI defaults answer MethodNotAllowed like their siblings. Stale budget docs, an unreachable CSV arm, an always-Some limit parameter, commented-out constants and a misplaced doc comment are cleaned up; weak test assertions name the expected error; lifecycle rule-ID control characters gain a test. The compatibility matrix records the CopyObject tag-copy permission, the exact SigV2 empty-body predicate, replica and migration writes of control-character keys, and the SigV2 DeleteObjects digest rule. * fix(table-catalog,heal): close the round-3 review findings A. Pack compaction rewrite groups to min(max-rewrite-bytes-per-job, TABLE_COMPACTION_MAX_STORED_BYTES), reject an over-ceiling group before any output is written, and name the ceiling in the commit-path errors; document the effective cap in the support matrix and scheduler runbook. B. Keep TaskCancelled/TaskTimeout classification on replacement passes: the namespace verification runs only for completed or storage-failed passes, in the erasure healer and the task bucket prepass. C. Compare the replacement bucket pass per bucket: a recorded identity that is gone or replaced still errors or restarts the pass, buckets created mid-pass are appended to the identities, pending_buckets and a persisted positional order without rewinding progress or the checkpoint; verification ignores additions. D. Read the migration-fence object once in the object-backed write permit and dispatch on its kind. E. Drop the strong store's copy of maintenance_cleanup_roots. F. Remove the unused test hooks fail_put_prefix, corrupt_put_prefix and pause_after_metadata; make compact_parquet_data_files and PublicationTicket.state private. G. Document the body-revision identity, the legacy revalidation budget and its HTTP 503 in the support matrix. H. Define HEAL_REPLACEMENT_RECOVERY_DIR once in ecstore and consume it from heal through the storage_api facade. * fix(ecstore): cap the index pages one disk may answer Pruned multipart index pages may carry few roots, so the page cursor no longer bounds how many pages a peer can make a listing fetch. Cut a disk off after 4096 pages (about one million examined markers) so it contributes no coverage and the listing falls back to the staging scan. * fix(ecstore,table-catalog): close the round-4 adversarial review findings Multipart index markers dated in the future (a backwards clock step between recording a marker and publishing its staging root) were judged old and could be pruned while their upload was in flight; a future-dated marker is now kept. The marker touch on EEXIST is best effort, so a filesystem that refuses to set times no longer fails the upload commit. Pruning runs on the durable-mutation lane instead of the one-worker foreground lane. Quota reconciliation keeps one throttle object per bucket incarnation (a quota revision change replaces it instead of orphaning the previous revision's object), the retirement cleanup deletes it ahead of the ledger and also when the ledger is already gone, and a scan the retirement already refused writes nothing back. Bounded DiskAPI defaults answer UnsupportedDisk, the verdict the aggregate gives a peer without the protocol. The compaction ceiling pre-check runs over every rewrite group before any output is written, and the worker's ceiling error no longer names a placeholder group. Docs: the inline on-demand-migration client budget, the protocol-2 downgrade conflict of replacement passes, the exact footprint numbers. |
||
|
Chris An
|
158aa276e9 |
fix(storage): bound heal retries and recover pool metadata (#9164)
* fix(ecstore): retry metadata writes rejected before staging * fix(ecstore): reuse registered disks during startup * fix(ecstore): retain remote health monitors during reconnect * test(heal): release the commit barrier during graceful shutdown * test(e2e): register graceful shutdown barrier regressions * fix(ecstore): roll back interrupted pool metadata commits * test(upgrade): honor the 1.0.4 delete recovery protocol * fix(heal): bound admission during object rate pressure * test(e2e): handle rebalance completion before stop * test(e2e): observe rebalance activity before the start reply * test(rebalance): hold the first run until stop cancels admission |
||
|
Chris An
|
a97c65895c |
fix(proxies,targets,obs): bound untrusted input in leaf crates (#9166)
Scan findings in crates whose behavior is driven by untrusted input and that stand alone from the storage core: - trusted-proxies: the simple implementation takes the client address from the right-most untrusted X-Forwarded-For hop instead of the left-most (backlog#3082); the legacy validator never falls back to the proxy identity when the chain is rejected and reads X-Forwarded-For unless RFC 7239 is explicitly selected (backlog#3051). - targets: webhook and similar outbound clients drain responses up to a byte limit instead of buffering arbitrary bodies (backlog#3062); the on-disk notification queue has a shared byte budget in addition to the entry count, configurable via RUSTFS_TARGET_STORE_MAX_BYTES (backlog#3122); AMQP and Redis manifests declare the broker url as a secret and bind credentials to their connection fields, and the plugin-instance PUT refuses to restore a redacted secret when the connection fields changed (backlog#3065, backlog#3066). - obs: the debug-level s3s response dump is filtered so STS and service-account credentials never reach the log (backlog#3029). |
||
|
唐小鸭
|
116a5299b2 |
fix(replication): replay the MRF journal periodically (#9139)
* fix(replication): replay the MRF journal periodically The persisted MRF journal was replayed only at startup. Delete-marker purges whose watch window drained with targets unreachable, and replications that overflowed the worker queues, reach the journal while the server runs, so they waited for a restart to converge. Move the startup pass into replay_mrf_journal and rerun it every five minutes from a background loop. The existing recovery leader lock keeps one pass at a time across the cluster, including the startup pass. * fix(replication): space periodic MRF replay across nodes Every node runs the MRF replay loop and the recovery leader lock only serializes passes, so an N-node cluster replayed the same retained backlog N times per interval while a target stayed down. Record the start time of a pass that finds entries in a shared replay stamp and skip a periodic pass when any node started one within the interval. The startup pass still always runs. |
||
|
Boyi (Elliot) Li
|
cc08e0d11b |
feat(fips): route internode RPC HMAC through aws-lc-rs FIPS (#9137)
* feat(fips): route internode RPC HMAC through aws-lc-rs FIPS * Revise KMS cryptographic compliance details Updated the KMS cryptographic compliance documentation to reflect changes in HMAC authentication and added details about the internode RPC HMAC authentication. Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Hauser Zhi <housemecn@gmail.com> --------- Signed-off-by: Hauser Zhi <housemecn@gmail.com> Co-authored-by: Hauser Zhi <housemecn@gmail.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Hauser Zhi
|
41b13aeb8b |
fix(tls): unify FIPS provider policy across clients (#9125)
* fix(tls): enforce active FIPS provider at startup Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * ci: provision FIPS validation runner Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(tls): validate FIPS before observability startup Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(tls): reuse the installed crypto provider Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): gate Linux-only commit helpers Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(tls): centralize FIPS provider policy Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> Co-authored-by: Chris An <anzhengchao@gmail.com> |
||
|
Boyi (Elliot) Li
|
3a1fac16e3 |
feat(tls): add opt-in fips feature for aws-lc-rs and rustls (#9084)
Add an opt-in `fips` Cargo feature to `rustfs` and `rustfs-tls-runtime` that enables `rustls/fips` and `tokio-rustls/fips` (activating `aws-lc-rs/fips` / `aws-lc-fips-sys` and `rustls-webpki/aws-lc-rs-fips`). When compiled with `--features fips`, `startup_runtime_hooks` installs `rustls::crypto::default_fips_provider()` as the process-wide default crypto provider and asserts `provider.fips()` at startup. Also lock `aws-lc-fips-sys v0.14.4` in `Cargo.lock`, allow the `OpenSSL` license on `aws-lc-fips-sys` in `deny.toml`, and update `docs/operations/kms-cryptographic-compliance.md` to document the scope of `-p rustfs --features fips` vs. `rustfs-crypto`. Co-authored-by: hector <42570491+majinghe@users.noreply.github.com> |
||
|
GatewayJ
|
04690f7da5 |
fix(s3-tables): clean empty catalogs and cache registration reads (#9063)
* fix(s3-tables): clean up empty tables and parallelize catalog reads * fix(s3-tables): bound catalog registration index reads * fix(s3-tables): cache validated registration ownership in memory |
||
|
Chris An
|
0e5fe8c658 |
feat(storage): share physical admission across scanner and heal (#8911)
* feat(scanner): share node admission with background heal * fix(scanner): drain candidates after listing completion * fix(scanner): yield cache leases for shared node admission * test(scanner): attest adaptive admission in process receipts * fix(concurrency): declare Tokio time dependency * feat(concurrency): retain bounded physical resource subsets * fix(scanner): retain resource ownership in native filesystem jobs * feat(concurrency): acquire physical groups before node work * feat(scanner): gate background reads by physical device * feat(concurrency): combine preallocated resource selections * feat(scanner): share physical topology across disk plans * fix(storage): retain only native disk resource subsets * feat(storage): admit auxiliary physical resources without waiting * feat(heal): admit physical resources before object repair * fix(storage): retain resources during native format probes * fix(storage): retain resources through direct writer I/O * fix(storage): retain resources through buffered writer I/O * fix(storage): retain physical resources through streamed reads * fix(storage): retain resources through native filesystem jobs * fix(storage): retain physical ownership during bitrot reads * fix(ecstore): retain resources during positioned reads * fix(ecstore): retain staged metadata write ownership * fix(ecstore): retain native io_uring read ownership * fix(ecstore): enforce receiver-side background admission * fix(ecstore): retain resources through whole-file native I/O * fix(ecstore): retain physical ownership during durable rename sync * fix(storage): account for retained background stream buffers * fix(storage): retain memory charges for background read results * fix(storage): account for native io_uring read allocations * fix(storage): retain memory charges for direct-write buffers * fix(ecstore): account for heal and RPC reader buffers * test(scanner): verify samples and clean concurrent collectors * fix(storage): retain inline and HTTP writer memory reservations * perf(rio): reuse reserved HTTP writer storage * fix(rpc): reserve background scanner pipe storage * perf(rpc): reuse reserved response reader storage * feat(scanner): measure cache copy allocation requests * fix(rpc): budget namespace scanner responses * fix(scanner): retain the enabled supervisor task handle * fix(scanner): budget namespace frame read staging * fix(scanner): account namespace envelope decode storage * fix(heal): retain and join the MRF consumer on shutdown * test(heal): isolate process-global MRF consumer fixture * fix(heal): join manager workers during shutdown * test(heal): verify shutdown at native read boundaries * fix(heal): drain channel commands before stopping MRF * fix(heal): admit resume scans and retain replacement workers * build: pin profiling sources and trust fixed ownership dependencies * fix(heal): yield node admission while accepted native work drains * fix(startup): retain shutdown work and preserve deadline failures * fix(heal): admit inventory probes and retain replacement work * fix(shutdown): preserve maintenance and frontend join failures * fix(shutdown): retain HTTP child tasks through real retirement * fix(shutdown): join actual TLS reload work before retirement * test(multipart): preserve failures before fault barriers * fix: keep physical admission inside repository guards Route the multipart lock diagnostics filter through the e2e storage compatibility module, reuse the parent module's s3s body aliases in the namespace memory test, and share one Reed-Solomon verify error mapping across the reserved-buffer and default paths. * fix: reconcile physical admission with main's new call sites Main added a source-admission metadata write layer, a CAS write fence, fixture writers and scanner handoff tests against the old borrowed-slice and node-permit shapes. Pass owned Bytes through the new metadata layer, answer the scanner handoff with ScannerNodePermit, gate the test-only disk lookup re-export, and clear the lint findings that stable 1.99 reports for the shutdown and admission code without suppressions. * test(concurrency): name reversed attachment groups clearly * fix(ecstore): box disk operation futures below admission layers Every LocalDiskWrapper operation reaches the LocalDisk future through the health, admission and timeout layers (track_cleanup_mutation, the track_disk_health family, the io-permit and native-resource admission split, and track_admitted_disk_operation). Each layer embedded that future, and unoptimized builds copy it into every awaiting frame, so one local delete cost about 1.3 MiB of stack and 16 tests overflowed libtest's default 2 MiB stack. Box the operation future once where track_admitted_disk_operation creates it, through a non-async helper so the temporary never stays in a polled frame. The layers above now hold only a pointer; the cost is one allocation per local disk operation. * fix(heal): box object heal futures below admission The admission wrappers took the object heal future by value, so unoptimized builds copied it into three awaiting frames, about 650 KiB per heal. Create it through a non-async helper instead, leaving only a pointer in those frames. * test(ecstore): count the I/O backend's root FD owner |
||
|
Chris An
|
3fc610c8a2 |
fix(scanner): track heal generation and payload observations (#9008)
* fix(scanner): distinguish missing heal identity fields * test(scanner): simplify identity case types * fix(scanner): track heal generation and payload observations * fix(scanner): treat heal results as observations * test(scanner): route heal integration through storage boundary * test(scanner): scope the heal generation fixture * test(heal): fill observation fields in the retained outcome fixture |
||
|
Chris An
|
7d45685d70 |
feat: classify scanner metadata calls by diagnostic prefix (#9052)
* feat: classify scanner metadata calls by diagnostic prefix * fix: export zero scanner visit counts for diagnostic scope |
||
|
Chris An
|
76bab81aa6 |
fix: reject unverified scanner performance evidence (#9049)
* fix: reject unverified scanner performance evidence * fix: keep unverified G14 descriptors unpublished * test: release scanner fixtures without signaling stale pids |
||
|
Mathew Beane
|
7e536843ea |
fix(audit): emit a bounded audit entry for authorization denials (#9073)
A request refused by the authorization layer was logged as s3_authorization_denied but never reached the audit stream. Emit an audit entry from DenialContext::deny with the account, bucket, object, denied action and denial reason. Entries are bounded by a per-class token bucket (lower for anonymous callers) with a suppressed count carried on the next entry, quiet probe denials are skipped, DeleteObjects audits its refused keys as one aggregate entry, and the S3 response is unchanged. |
||
|
唐小鸭
|
7672867476 |
fix(replication): count force-delete outcomes in replication stats (#9072)
A replicated force-delete that a target rejected (for example a peer whose credentials lack s3:ForceDeleteObject) was only logged; the per-target replication counters never saw it, so `replication status` reported zero failures while the sites had already diverged. Build a ReplicatedTargetInfo for every target outcome of replicate_force_delete_to_targets (missing client, offline target, rejected remove, success) with a Pending previous status, and feed each through ReplicationStats::update so rejections count as failures and accepted force-deletes count as replicated. Document that the managed site-replicator-0 policy must name s3:ForceDeleteObject explicitly and that a bucket-replication target with its own credentials needs the same grant. |
||
|
Hauser
|
e0444b519d |
feat(ecstore): add sampled batch CPU diagnostics (#9019)
* feat(ecstore): add sampled batch CPU diagnostics Keep timing and request correlation in a separate default-off feature. Preserve the existing positioned-read backend, byte and error contracts. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): route batch diagnostics through storage facade Keep the diagnostic implementation private to its owner and expose explicit feature-gated helpers through the existing storage boundary. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): retain diagnostic IDs across GET producer tasks Capture request context before spawning the legacy producer. Keep the existing producer, cancellation and read-lock lifetime intact. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): avoid unused batch read index Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(obs): isolate local batch diagnostic JSON events Omit span metadata and inferred parent IDs for diagnostic root events. Verify per-shard index order and document existing production metric timing. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): accelerate receipt fixture in CI Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(heal): allocate stack for bucket recovery fixture Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(s3): allocate stack for conditional GET fixture Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
cxymds
|
c90ba5922f |
fix(tiering): diagnose blocked tier deletion and retry cleanup (#9035)
* fix(tiering): diagnose blocked tier deletion and retry cleanup * fix(tiering): reuse admin errors for deletion diagnostics |
||
|
Hauser
|
a9d3022f37 |
fix(ecstore): prove multipart part sizes before cleanup (#9037)
* fix(ecstore): prove multipart part sizes before cleanup Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(ecstore): merge strict multipart observation passes Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test: respect lifecycle architecture boundaries Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test: fix multipart E2E clippy warnings Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Chris An
|
8e88419f3a |
feat(scanner): share node admission with background heal (#8903)
* feat(scanner): share node admission with background heal * fix(scanner): drain candidates after listing completion * fix(scanner): yield cache leases for shared node admission * test(scanner): attest adaptive admission in process receipts * fix(concurrency): declare Tokio time dependency * fix(scanner): route test disk access through owner facade * test(scanner): fix lint warnings |
||
|
Hauser
|
b4026b43b2 |
feat(observability): attribute local batch shard reads (#8991)
Record local batch eligibility, blocking queue and execution time, shard outcomes, and actual read bytes behind the existing GET-stage metrics gate. Keep captured metric handles with the blocking worker after cancellation and use scoped low-latency histogram buckets for the new duration series. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Chris An
|
b5c253f3ed | docs(listing): describe degraded strict quorum limits (#8981) | ||
|
Chris An
|
f96c94f950 |
fix(listing): treat key markers as literal keys (#8978)
* fix(listing): treat version markers as literal keys * fix(listing): keep start-after and V1 markers literal * docs(listing): state cursor compatibility removal condition * fix(listing): resume protected table pages from raw keys * test(listing): keep protected pagination coverage in app layer |
||
|
Hauser
|
b9f01162b2 |
feat(tools): inspect BTF scalar member layouts (#9003)
Report selected integer offsets and widths with strict member and payload validation. Cross-check compiler fixtures with bpftool and independent C offsetof/sizeof results. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Chris An
|
5ab1d5191e | fix(heal): pace and parallelize replacement recovery (#9005) | ||
|
Chris An
|
1451f3189b |
fix(listing): leave delete recovery out of list requests (#8980)
* fix(listing): leave delete recovery out of list requests * test(listing): expect residue preservation across walks |
||
|
Hauser
|
254592717a |
feat(tools): add read-only BTF prototype inspection (#8985)
* feat(tools): add read-only BTF prototype inspection Expose bounded declared prototype headers through a standard-library CLI, preserve explicit void and failure semantics, and wire local contracts plus an independent compiler/bpftool reference into CI. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(tools): retain native reference failure diagnostics Clear checkout credentials and capture bounded native stderr alongside stdout, preserving exact exit status and owned child cleanup. Keep the first reference failure unresolved rather than guessing its missing diagnostics. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * ci(tools): select a concrete bpftool file parser The hosted PATH wrapper can exist without tools for its Azure kernel. Select an executable ELF by availability/version before comparing the fixed fixture, installing generic tools only when needed. Data comparison errors remain fatal and never trigger tool switching. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Chris An
|
b8eef694c7 |
fix(heal): group newly attached replacement disks (#8973)
fix(heal): batch replacement targets before freezing recovery scope |
||
|
hector
|
26d55bb04e | ci: remove never-green Performance A/B workflow (#8979) | ||
|
Chris An
|
7ea3aa2a93 |
fix(listing): keep the V1 NextMarker escape and document optimal limits (#8976)
A cursor for a key that literally ends in a legacy-shaped tag is issued wrapped in one extra `[rustfs_cache:v2,return:]`, because the listing layer still strips one legacy tag from incoming markers. ListObjects (V1) unwrapped it in `calculate_next_marker`; the client then sent the bare key back as `marker`, the listing layer stripped the key's own tail, and the next page restarted at the stem and returned the same key forever. Keep the wrapper in V1 NextMarker and cover the V1 resume path (marker sent back as StartAfter) on a real store. Document that sampled list quorum modes can also omit committed objects whose copies mostly sit outside the sample, and recommend `strict` for backup, sync, replication and lifecycle tooling. |
||
|
Chris An
|
1c2f69eba9 |
fix(storage): 404 for dangling objects and strict list quorum default (#8963)
* fix(storage): 404 for dangling objects and strict list quorum default Stage F.0 of the ListObjects quorum plan. Once listing returns to MinIO's listing quorum, a PUT that reached half of the drives but not read quorum becomes visible in LIST, so GET and HEAD must not answer it with a retryable 503. GET, HEAD and prepared GET metadata now report a metadata read-quorum failure as ObjectNotFound or VersionNotFound when every drive answered with valid metadata of a single generation or not-found and is_object_dangling proves the object or version dangling, matching MinIO's getObjectFileInfo. Offline or unanswered drives, timeouts, corrupt xl.meta, two generations, remote versions and the meta bucket keep the read-quorum error. Nothing is deleted on the read path; heal keeps owning cleanup and its grace window. Read-before-write callers of get_object_fileinfo keep the quorum error. RUSTFS_LIST_OBJECTS_QUORUM now defaults to strict, matching MinIO, and the operations doc explains that optimal samples drives and can briefly show deleted objects. A new test pins the write-path precondition: a PUT that misses write quorum leaves no new version or data directory on any drive once its rollback completes. * test(listing): prove the list quorum env var is read with a non-default value |
||
|
Hauser
|
fea5a584c3 |
fix(storage): fence multipart and metadata recovery (#8919)
* perf(ilm): back off idle tier-delete journal recovery Use bounded idle backoff only after complete empty journal and dispatch scans, while preserving immediate notifications and the retry cadence for observed work. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): reconcile interrupted pool metadata publication Bind pool metadata CAS and Undo to exact native predecessors. Recover an interrupted physical publication only from an ordinary committed authority and complete verified identity, current and backup evidence, then require normal V3 CAS and quorum readback. Preserve known-conflict gates, startup and heal lock ordering, targeted admission and the original kill-recovery deadline. Wake proof-ready recovery immediately and bound parallel capacity sampling without removing mutation fences. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(ecstore): reuse proven pool metadata durability Publish native rollback backups directly in their owned UUID directories. Reuse a one-shot data-durability receipt only for an exact hardlink to the pinned, fully synchronized head and root. Preserve new-head and directory synchronization, native predecessor CAS, conditional Undo and uncertainty gates. Keep unknown, copied, raw-parent, cancelled and failed operations cold. Add bounded diagnostics, sync-call regressions and path-isolated test hooks. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(multipart): exercise selective RPC delay and peer loss Add authenticated method-entry barriers for metadata and lifecycle locks under e2e-test-hooks. Verify real dispatch, write quorum, prior part payloads, lock drain and incarnation isolation without changing production budgets. Record cancellation separately from normal exits so late filesystem completion cannot overwrite Drop evidence. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * docs: define multipart delay gates and receipt invariants Document RPC entry evidence, peer-loss and cancellation oracles, remaining incident attribution limits, and immutable head inodes required for durability receipts. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(readiness): expose missing runtime drive slots Report configured slots without a runtime handle separately from unavailable inventoried drives. Preserve quorum and probe status semantics. Wait for every coordinator to regain its complete live inventory before multipart RPC fault windows and single-attempt recovery oracles. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): retry committed promotion convergence safely Require peer reload convergence on every worker attempt, including unchanged retries. Recheck durable identity, replicas, namespace fence, generation and cancellation before allowing any data movement. Keep known committed promotions active on a verified propagation pause. Release precise owners, retain queued successors and reuse the existing watchdog. Preserve uncertainty gates and terminal operation behavior. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): guard native metadata mutation aliases Reject protected RenamePart source and destination paths before I/O. Classify the protected namespace through the native path resolver, and require canonical conditional RenameData and Undo keys so aliases cannot create separate lease domains. Preserve ordinary multipart and canonical CAS behavior, warm durability receipts and zero-allocation canonical fast paths. Cover real alias, byte identity, receipt and normal mutation behavior. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): register Darwin multipart RPC delay gates Update distributed membership from the actual Darwin listing. Preserve the full-profile digest and leave Linux unchanged until its own listing is produced, as required by the contributor guide. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(multipart): fence native publication and recovery Publish multipart data through private hardlinks and atomically consume the prepared source head. Bind exact compensation capabilities and native recovery to the physical cohort, source generation and durable publication state. Require fresh terminal confirmation for native replay and preserve indeterminate data when no safe recovery witness exists. Release logical guards only after validated irreversible quorum success while retaining physical mutation leases, accounting and resource ownership through actual callbacks. Preserve local instance proof authority and historical Copy metadata without granting healed donors fresh publication authority. Keep general movement identity strict while preserving directory capacity compatibility in a separate checked comparison. Cover native fault and cancellation boundaries, zero-byte objects, real Copy, heal and migration, exact physical readback and distributed RPC delay. Retain all quorum and durability requirements and document rollout limitations. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): fix ratchet scan and Windows fault hooks Move native pool metadata inventory tests after production items so the error-format ratchet sees the existing call site. Preserve the baseline instead of accepting a false count decrease. Make the directory-barrier failure hook available on every test platform and exercise the parent-barrier failure/retry cases in Windows CI. Keep production synchronization behavior and all existing assertions intact. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(protos): remove redundant recovery request clone Move the baseline into its final struct update while retaining the earlier clone needed to construct the first request variant. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): align RPC delay gates with native abort * fix(storage): validate native source recovery and CI gates Preserve wire layouts while bounding native response sizes and fix the last-use recovery request clone. Avoid destination replay reads only for fully observed initial native sources; keep upgraded and recovered sources conservative. Cover real legacy publication after a cold restart. Align upgrade acceptance with the native mixed-version mutation pause and use exact upload-scoped test completion evidence before advancing time. Keep all original quorum, ownership, byte and deadline assertions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): verify native abort under partial admission Delay native cleanup at the exact upload root after all-participant closure. Verify quorum-only admissions retain every source byte on refused Abort and complete the same upload without creating a missing original participant. Keep sub-quorum orphans and their discoverable identities isolated while a fresh all-participant upload proves service recovery. Route shared constants through the existing E2E storage facade. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): avoid formatting a fixed invalid upload path Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): align native MPU fault recovery assertions Verify real source participants, preserved orphan bytes and same-upload completion after quorum-only admission. Require full physical inventories so an expected read-quorum error cannot hide unrelated leakage. Preserve ordinary quorum and recovery budgets, and document listing availability through the affected set. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): keep MPU tail deadlines on the wall clock Global Tokio time advancement can expire drive-health probes while real filesystem callbacks are pending. Use a short real drive deadline with the existing physical-entry and cleanup barriers, retaining the byte, lock, resource-permit and final-shard assertions. Fixes #8937 Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(storage): align native fixtures with instance ownership Open independent disks for the simulated capacity-loss instance, scope four coordinator journal disks without rebinding source authority, and probe configured namespace keys while retaining physical I/O hooks. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): isolate native multipart future construction Construct optional native publication futures behind synchronous heap boundaries so ordinary rename callers do not carry their state machines. Preserve native guards and ownership without adding allocations to ordinary PUT, copy or heal branches. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(heal): assert coordinator journal scope after rejoin Check the exact original four-member journal map after every rejoin and manager stop while preserving all sixteen native data slots. Reconnect expansion fails the test rather than being filtered away. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(heal): bound object repair future construction Construct the normal and FileCorrupt deep-retry state machines behind a synchronous heap boundary. Preserve guards and cancellation while bounding the shared rename and deletion call stacks without increasing test stacks. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): qualify fixture disk identity checks Call the same DiskAPI method explicitly so the independently reopened disk identity assertion compiles under every target and feature combination. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): isolate native inspection from mutation state Select a dedicated read-only inspection future behind the existing single Box allocation. Preserve validation, source ownership and the canonical inspection body while keeping mutation state out of inline read callers. Reject misrouted operations with typed errors and retain source evidence. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): pause abort time after native cleanup Wait for durable source deletion on the wall clock before advancing the capacity lease clock at the post-delete barrier. Paused time can otherwise expire the barrier while native filesystem cleanup is still pending. Retain lease-loss injection, durable inflight-byte accounting and restart reconciliation assertions. Document the test-util feature required to select the native decommission fixtures in package-scoped regressions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): reclaim data from owned multipart undo Retain exact runtime-only Undo ownership through deferred snapshot cleanup. Revalidate every target metadata version and the retirement anchor, keep the delete claim through the physical callback, and remove the anchor after the payload barrier. Preserve generic cleanup refusal when metadata is missing. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): preserve unformatted bootstrap routes Keep explicitly unformatted local paths reachable by an in-flight elected formatter while quarantining their service slots and disk-ID membership. Own bounded legacy-read handles so the actual asynchronous creator future remains Send. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(s3): bound CopyObject tracing poll frames Construct the business future behind an outlined single-Box boundary and lazily instrument its handle. Preserve Send, Unpin, span timing and cancellation without embedding concrete business-state construction in the tracing poll. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(mpu): observe recovery after direct legacy bootstrap Seed all sixteen real previous-release sources before attaching the same-port method observers. Require exact per-peer disk-route Unimplemented responses with zero publication calls, retain byte and cleanup assertions, and remove superseded raw transport. Register the two new regressions from actual Linux and macOS listings. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(mpu): prepare the homogeneous previous-release fleet Reuse the existing release baseline before creating the genuine legacy multipart sources. Restart old processes with their normal budgets and require every drive through the reduced-redundancy probes before installing the mixed-version observers and current fault budgets. Keep standard EC:4, single-attempt calls, all mixed-version byte and reclamation assertions, and their original deadlines unchanged. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(mpu): isolate release compatibility from fault profiles Keep normal component configuration for the observed mixed-release rollout, which injects no RPC delay or loss. Retain the short component profile in actual method-fault scenarios and preserve the single-attempt 45-second probe and every typed error, routing, source byte and reclamation assertion. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(mpu): use normal shutdown for planned release replacements Use the existing bounded SIGTERM path during homogeneous release preparation and the observed mixed-version rollout. Preserve every multipart inventory, routing, quorum and byte assertion, and keep actual crash recovery on its original hard-kill path. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * revert: retain the verified release compatibility fixture Restore the published fixture after the local configuration and lifecycle experiments failed to resolve the macOS startup failures. Keep their diagnostic records separate from the delivered implementation. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(copy): bound inherent handler polling stack Build the inherent CopyObject future through a synchronous noinline factory at the existing trait await. This keeps its large state out of the trait wrapper's active polling frame during historical self-copy. Preserve cancellation, tracing and post-write notifications, with one additional allocation restricted to COPY. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): validate authoritative quota and native pool heals Require actual authoritative quota usage in the existing readiness loop instead of treating HTTP success or unknown usage as a numeric baseline. Keep the original quota assertions and convergence budgets. Allow the owning pool metadata CAS to republish into a fresh valid data directory while retaining exact version, layout, completeness and shard fingerprints. Require every node's real metadata/read/write readiness before accepting recovery. Keep ordinary object manifests and the original 180-second recovery deadline unchanged, with negative oracle regressions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): distinguish certified native Abort recovery Require native participants with a persisted full closure certificate to retire their heads and reclaim every surviving source on retry. Keep uncertified legacy authority below identity quorum conservative, while retaining actual partial-delete and committed-object byte checks. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> Co-authored-by: overtrue <anzhengchao@gmail.com> |
||
|
Chris An
|
4a96f06983 |
refactor(listing): remove env-gated ListObjects index provider (#8948)
* refactor(listing): keep paginated listing on ECStore only `Sets` and `SetDisks` carried their own copies of `inner_list_objects_v2`, `list_objects_generic`, `inner_list_object_versions`, `list_path` and `list_merged`, plus a third copy of the task that converts merged raw entries into the walk result stream. Only `ECStore` can answer a paginated listing, since a page merges every set of every pool; the copies drifted (the `Sets` copy still rejected `//` prefixes that #8605 made literal on `ECStore`) and were reachable only from tests and one decommission helper. Remove the `ListOperations` impls for `Sets` and `SetDisks` together with the duplicated pagination code. `SetDisks::walk` stays as an inherent method for set-scoped callers and now honors `versions_sort` like the store-wide walk; `SetDisks::list_path_result` stays for the recursive delete lister. The walk entry conversion moves into one `spawn_walk_entry_forwarder` shared by both walks. The decommission receipt lister walks its pool's sets instead of paginating through the removed `Sets` listing. Tests that exercised the removed layers now exercise `ECStore` for pagination or `SetDisks::walk` for per-pool version inspection. The architecture and logging guard rows that pinned the removed `SetDisks`/`Sets` listing entry points are dropped, and the error-other ratchet baseline shrinks by one. * fix(listing): exclude synthetic directories from receipt walks * refactor(listing): remove env-gated ListObjects index provider Delete the experimental, default-off key-only / verified-page / metadata-fast index provider from `ECStore::list_objects_generic` together with everything only it drove: the `RUSTFS_LIST_OBJECTS_INDEX_*`, `RUSTFS_LIST_OBJECTS_NAMESPACE_JOURNAL_*` and `RUSTFS_LIST_OBJECTS_METADATA_FAST_*` readers, the persistent key-only index and its on-disk format, the namespace mutation journal (state, chaos injector, system/local backends, degraded set, global lock), the index lifecycle/health/source-selection types, the verified-index candidate pager, the `list-chaos` cargo feature, the `rustfs_s3_list_objects_index_*` metrics and the archived bench script. `list_objects_generic` now always walks. The write-path hooks `observe_list_objects_mutation(s)` are gone; their callers bump the scanner namespace generation directly through `observe_scanner_namespace_mutations`, which is the sub-step the old hook performed before taking the journal lock, so the generation observed by the scanner, the node RPC service and the data-movement lease checks is unchanged while PUT/DELETE no longer take a global tokio RwLock or read environment variables. `ListSourceMode::{metadata_authority, requires_live_metadata_verification, can_satisfy_strong_listing}` and `ListMetadataAuthority` were only reachable from the provider and are removed as well; the enum, its variants, the cursor encoding and `ListContinuationV2` are left untouched. * docs(listing): define index retirement upgrade prerequisites * fix(ci): retire the removed listing index test requirement |
||
|
Chris An
|
197ab19f0c |
test(heal): verify backup recovery and tier owner handoff (#8935)
* test(heal): verify backup recovery and tier owner handoff * test: prepare tier mover before holding remote delete |
||
|
Chris An
|
85231b71e6 |
fix(heal): gate unresolved MRF lifecycle writes on reader support (#8918)
* feat(heal): read retained MRF lifecycle v2 checkpoints * feat(heal): advertise MRF lifecycle reader capability * fix(heal): gate unresolved MRF lifecycle writes on reader support * test(heal): use native path for CAS publication hook |
||
|
Chris An
|
7728eeab1a |
feat(heal): advertise MRF lifecycle reader capability (#8916)
* feat(heal): read retained MRF lifecycle v2 checkpoints * feat(heal): advertise MRF lifecycle reader capability |
||
|
Hauser
|
13dc635d51 |
feat(scanner): expose startup phases and attempt deadlines (#8931)
* feat(scanner): expose startup phases and attempt deadlines Share one node-local startup snapshot between native status and metrics. Track actual waits and first attempts without changing admission timing, coverage acknowledgement, or publication semantics. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(scanner): observe startup recovery before initialization Create the observation owner at enabled startup entry and move it through the existing initializer, preserving admission timing and disabled cleanup. Cover blocked recovery discovery and cancellation through the public entry. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser
|
299609bb83 |
fix(quota): keep reads available without snapshot-wide copies (#8928)
* fix(quota): keep quota reads available during scans Use optional scanner usage for read-only quota statistics across all quota configurations. Preserve strict legacy write admission and durable reservation enforcement while keeping unknown usage distinct from zero. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(usage): avoid snapshot copies for single-bucket reads Project owned snapshots, scalar bucket sizes and refresh-only results through the same TTL-bounded cache loader. Move loaded snapshots into the cache without changing refresh serialization or generation fencing. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Chris An
|
124cbfc00a | feat(heal): read retained MRF lifecycle v2 checkpoints (#8913) | ||
|
Hauser
|
0f24511535 |
test(e2e): harden multipart fault and TLS acceptance (#8907)
* test(e2e): tighten scoped multipart fault acceptance Target Abort root deletion instead of child transaction cleanup, require actual metadata peer hits, and keep faulted APIs single-attempt. Validate bounded ordinary recovery and safe quorum refusal for partial mutations, with the recovery enhancement tracked in backlog#2940. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(e2e): drive concurrent TLS reads independently Give each coordinator its own supervised task and cooperative polling budget while retaining the existing workload, deadline and byte/hash assertions. Preserve node logs after fixture teardown and report the operation, response phase and received bytes when a transfer times out. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser
|
af4ce32528 |
perf(ecstore): reduce scanner metadata path allocation overhead (#8899)
* fix(scripts): collect scanner host telemetry on macOS Use platform-supported iostat arguments and expose optional collector failures without interrupting scanner-status collection. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(ecstore): reuse validated metadata read path buffers Share the checked bucket/object tuple across metadata read consumers and borrow clean normalization inputs without skipping filesystem validation. Keep Windows reconstruction and the second bucket check intact. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(scripts): finalize scanner telemetry for failed runs Initialize host telemetry artifacts before external captures so reused output directories cannot retain prior successes. Share normal and error-path finalization, reap collectors on request failure, and preserve the primary exit status even if status recording fails. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): distinguish unreadable recovery inventories Treat only the exact system-prefix read-quorum result as incomplete during bounded polling. Preserve strict final reads and the shutdown/drain assertions, record failed observations, and cover interrupted empty streaks and unrelated errors. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Chris An
|
32c4593587 |
fix(heal): retain MRF state when bucket identity is unresolved (#8905)
* fix(heal): retain MRF state when bucket identity is unresolved * test(heal): give real-storage replay fixture a bounded stack |
||
|
Hauser
|
7130f25af5 |
fix(ecstore): unblock delete replay and parallelize preparation (#8896)
* perf(ecstore): avoid head-of-line blocking in delete recovery Recover independent keys through a bounded rolling window while preserving per-key hint generations, admission budgets, cancellation retry, and the existing transaction decision engine. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(ecstore): bound durable delete preparation and bootstrap recovery Prepare independent Linux delete transactions with bounded concurrency while retaining per-key result order and serial fallback for overlapping, rollback, and legacy requests. Preserve startup discovery until bucket metadata is initialized, and add release performance and process-crash acceptance runners. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): verify clean exits in recovery acceptance Reject forced or OOM exits when measuring clean startup and verify original ETags across mixed-layout crash recovery and a second restart. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): distinguish prepared and decided batch crashes Observe real preparation and receipt states separately, retain non-vacuous acknowledged-delete sentinels, and require read-only physical convergence before S3 reads can trigger cleanup. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): move final delete recovery options Move the options value into the batch-ordering regression's final case instead of cloning it. Keep the earlier clone needed for reuse. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): replay delete receipts after interrupted proof GC Retire only the exact empty transaction leaf before journal activation. Preserve occupied directories and prefix children, and cover partial cross-disk proof GC without a fresh delete. Tighten crash evidence to distinguish directory-only remnants from retained content. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser
|
900d082764 |
fix(scanner): make startup responsive and skip redundant config work (#8898)
* fix(scanner): bound implicit startup delay and publish config early Limit warm-cache startup to the smaller of the cycle interval and one minute before jitter, while preserving explicit startup settings. Publish configuration gauges when applying runtime settings and retain timeout diagnostics until an actual new cycle begins. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(scanner): unify startup waits and skip unchanged configuration Bound read-only startup inspection and make startup and pre-leadership retries respond to cancellation and configuration changes. Preserve one startup deadline and discover replication behind lifecycle-only buckets. Avoid irrelevant bucket listings and repeated per-disk reconfiguration. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(scanner): retain effective startup policy and fallback metrics Keep the sampled startup deadline when configuration changes leave its normalized policy unchanged. Initialize retained runtime metrics after an invalid first refresh without suppressing the validation error or replacing a newer accepted configuration. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Chris An
|
b23723a51f |
feat(heal): add live node admission feedback (#8897)
* feat(heal): add live node admission feedback * test(heal): use storage facade in admission regression * fix(heal): preserve retryable node admission deferrals |
||
|
Hauser
|
aaeedcd1d7 |
perf(rpc): remove stream boxing and verify distributed TLS reads (#8885)
* perf(rpc): remove redundant boxing from read streams Keep the capped read stream concrete until StreamingBlob owns its type erasure. Add verified four-node TLS coverage for full, unaligned range and concurrent reads, with bounded client memory and explicit integrity checks. Document protocol comparison commands and the old-node workaround during rolling upgrades. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): preserve TLS trust for alternate credentials Derive credential-specific clients from the cluster configuration so they retain the generated CA. Exercise successful trusted reads and reject an invalid replacement identity instead of inheriting root credentials. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): collapse acceptance report path guard Combine the optional report path and existing-file checks into a let chain without changing the report or PID sidecar collision behavior. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Chris
|
f279cefa9c |
feat(observability): track foreground response-body completion windows (#8894)
* feat(observability): track foreground response-body completion windows * docs: correct foreground ownership spelling |
||
|
Hauser
|
40837f67f8 |
perf(ecstore): avoid head-of-line blocking in delete recovery (#8884)
Recover independent keys through a bounded rolling window while preserving per-key hint generations, admission budgets, cancellation retry, and the existing transaction decision engine. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Chris
|
63eac7a387 | feat(observability): sample live cgroup resource limits (#8882) | ||
|
Chris
|
e13336a116 | feat(scanner): expose publication and heal observation status (#8878) |