mirror of
https://github.com/rustfs/rustfs.git
synced 2026-11-08 19:43:05 +00:00
fbc7af7df05ecd16c4fce845fcf66cb096491c21
1055 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 cxymds
|
fbc7af7df0 | fix(ecstore): preserve multipart shards within write budgets | ||
|
Hauser Zhi
|
32238a179c |
fix(server): bound rejoin waits and preserve recovery ownership (#9181)
* test(server): cover recovered-node admission boundaries Document backend readiness and write-availability checks during rejoin. Extend the existing regression through the production middleware order without changing runtime admission or cluster-quorum semantics. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(server): bound rejoin waits and preserve recovery ownership Enforce startup and unreachable-grace bounds while consuming peer replies. Release admission when every set has its required peer majority and cancel surplus read-only probes without carrying votes between rounds. Bound local readiness observations and polling sleeps. Retain initialized IAM across preparation failures, install the context once, and retry only publication after readiness timeouts. Share preparation with inline startup. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(server): make peer admission folding incremental Compile peer-to-set memberships once and retain compact per-edge votes. Consume each reply once, discard its payload, and update only affected set counters instead of rescanning all sets and retained responses. Recheck the captured deadline after polling and folding a reply so late work cannot admit the node or clear an expired unreachable grace period. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Chris An
|
d6210ac17b |
fix(s3,ecstore,table-catalog): bound work driven by client input (#9172)
* fix(ecstore,heal,select): bound storage-layer work driven by client input Scan findings in the erasure store, heal, file metadata and S3 Select layers, grouped because they share the disk walk, the multipart staging area, delete recovery and the select snapshot lease: - Multipart: upload ids are canonicalized before they name a staging path or a lock key (backlog#3027); trailing-slash keys are encoded like every other handler (backlog#3055); unlisted parts are removed at completion (backlog#3106); per-bucket uploads are found through a signed per-disk index instead of a scan of the shared staging area (backlog#3121); encrypted part ETags carry a per-upload secret so a write-only user cannot confirm plaintext guesses (backlog#3044). - Delete and recovery: recovery walks are bounded per request (backlog#3038); a delete marker on a key whose xl.meta path is a directory, and sidecar names used as child keys, no longer wedge the journal (backlog#3035, backlog#3105); over-quota PUTs no longer list the bucket under the ledger lock (backlog#3128). - Listing: scan_dir sends metadata without inline data (backlog#3052), counts delete markers against max-keys (backlog#3108), and rejects a marker that carries the cursor tag (backlog#3059). - Heal: bucket metadata and heal records live in separate namespaces so DeleteBucket cannot erase a drive rebuild (backlog#3039), a deleted bucket is skipped instead of failing the rebuild (backlog#3040), and directory objects are encoded on readback (backlog#3061). - Restore and tiering: RestoreObject preserves tags and the version's modification time (backlog#3073, backlog#3074); Snowball transport headers are not forwarded to the tier (backlog#3090); an epoch modification time round-trips through xl.meta (backlog#3063). - Locks: PutObject takes its bucket locks after the body is staged (backlog#3131); SelectObjectContent releases its namespace locks once the snapshot is taken (backlog#3132). - RPC and peers: the v3 signature check consumes the v2 nonce (backlog#3129); a bucket-metadata reload timeout no longer marks the peer offline (backlog#3124); replication target requests carry an operation deadline (backlog#3102); the scanner resumes its interrupted wait after a lifecycle write (backlog#3118). - S3 Select: bounded schema inference, column cap and budgeted CSV batches, Parquet dictionary bounds and pre-planner projection checks (backlog#3075, backlog#3076, backlog#3077, backlog#3078). * test(e2e): seed legacy duplicate bucket tags offline for the OPA contract * fix(s3): authorize what handlers store and bound request-driven work Scan findings in the S3 request layer (access hooks, object use cases, HTTP server layers), grouped because they share the authorization condition builder, the PutObject admission path and the response layers. Builds on the storage-layer batch for the bounded recovery and multipart APIs it calls. Authorization and policy conditions: - copy source is percent-decoded and version-canonical before the s3:x-amz-copy-source condition (backlog#3135); CopyObject authorizes the tag set it copies (backlog#3034); HeadObject with the replication check header still authorizes s3:GetObject (backlog#3047); the x-rustfs-source-version-id header feeds s3:versionid (backlog#3072); recursive force-delete with a REPLICA header authorizes every descendant as a delete (backlog#3070); include-deleted listings require s3:ListBucketVersions (backlog#3099); the metadata listing route keeps table isolation and uses the same prefix it authorizes (backlog#3053, backlog#3054); If-Match probing needs read permission (backlog#3097); Object Lambda checks s3:GetObjectVersion (backlog#3058); replication proxy fallback keeps ExistingObjectTag denies (backlog#3089); HeadObject returns lock and tag headers only with permission (backlog#3113); IPv4 clients on the dual-stack listener match IPv4 SourceIp conditions (backlog#3049); bucket policies are parsed once and cached (backlog#3032); keystone: access keys no longer verify with an empty secret (backlog#3043). Admission and resource bounds: - GetObject releases disk-read permits when the client stops reading (backlog#3023); PutObject takes buffer and foreground admission only after body bytes arrive and bounds the wait (backlog#3031, backlog#3084); xl.meta size is bounded on the write path (backlog#3100); XML compat layers stream instead of buffering whole bodies (backlog#3094); tier GETs stream the remote body (backlog#3046); on-demand migration pulls release permits on unread GETs (backlog#3110); SSE-KMS key policy covers UploadPart and completion (backlog#3130); MinIO admin decryption runs Argon2id off the async worker (backlog#3024). Input validation and leaks: - replication headers are validated (source ETag shape, original size) (backlog#3092, backlog#3093); control characters in keys and lifecycle rule IDs are rejected (backlog#3101, backlog#3117); Snowball member mtime needs s3:ReplicateObject (backlog#3083); DeleteObjects checks Content-MD5 and admin/STS reject SigV2 replays with altered query or body (backlog#3037, backlog#3026); request logs redact presigned queries and handler spans skip credentials (backlog#3068, backlog#3107); SSE-C listings and completions withhold plaintext metadata and checksums (backlog#3120, backlog#3050); CORS resolves the bucket like s3s does (backlog#3104); PutBucketTagging rejects duplicate keys (backlog#3126); notifications fire after a client disconnect (backlog#3127); ListenNotification streams re-check access (backlog#3119). * fix(api): drop imports left unused by the base merge * test(s3): move the KMS multipart denial test to the app layer The test drives app-layer multipart use cases, so it belongs with them; living in the SSE module it imported app code from the infra layer and failed the layer dependency guard. The SSE test helpers it borrows are exposed through the storage and app facades. * fix(s3): keep the SSE-C key fingerprint on keyless multipart completion The backlog#3050 fix withheld both the plaintext checksums and the customer-key MD5 header from a CompleteMultipartUpload that did not present the key. The checksums are a digest of the plaintext and stay hidden; the key MD5 identifies the key only, and the S3 conformance suite (test_copy_part_enc) expects it on a keyless completion, so it is echoed again. * fix(tier): stream GET bodies and document replication budgets * fix(ecstore,s3): keep the multipart index and cold fill off disk lifetimes Two lifecycle regressions surfaced once the storage and request batches ran together against main: - The multipart index owner cloned the disk's pinned publication root and bootstrapped on a task of its own. That kept the mount alive past the last LocalDisk owner (the disk lifecycle allows exactly three holders of the root FD) and left a stray task behind, which the local-disk and read-repair tests detect. The owner now keeps only the root path and reopens it for identity checks, and the bootstrap scan rides the disk's existing cleanup task, where stopping the disk abandons it and a reconnect resumes it. - The disk-read admission wrapper defers releasing its permit until an in-flight read completes, so a cancelled client cannot widen disk concurrency. The cold-fill producer, however, drops a read it has already classified as stalled when its deadline fires, and that permit stayed pinned for the lifetime of the stuck read. The producer now abandons the admission explicitly on timeout. * test(ecstore): pass the index owner to the Linux cleanup-loop test * fix(s3): release cold-fill admission after watchdog handoff * fix(s3,ecstore): close the round-1 adversarial review findings Adversarial review of the combined request- and storage-layer batch found fixes that fell short of their issues, two behaviour regressions for legitimate clients and several unrequested costs: - backlog#3038: a bulk DeleteObjects reserved the whole per-disk inspection allowance up front for every key in flight and exhausted the 16 Mi request budget on any set wider than eight drives, failing healthy keys with SlowDown. Calls are now admitted while budget remains and charged for the entries they actually visited. Object-scoped discovery charges plain child keys at one entry per 64 names instead of one each, so deleting a missing key that names a folder prefix completes instead of answering 503 forever while a hostile prefix stays bounded. A peer still on the previous binary is reported as an unsupported participant, the repository's rolling-upgrade contract, instead of as a budget verdict. - backlog#3037/#3026: a SigV2 configuration PUT without Content-MD5 was still replayable with another body; it is refused with MissingContentMD5 while SigV4 keeps the optional-MD5 contract. - backlog#3072: DeleteObject now resolves x-rustfs-source-version-id before authorization like PUT and COPY, so a replica delete cannot remove a version the policy never saw. - backlog#3121: the multipart index no longer fsyncs a marker and four directories per multipart metadata write (it is rebuilt on every start), and ListMultipartUploads discovers roots concurrently. - backlog#3077: Parquet SelectObjectContent decodes budget-sized batches instead of one row per batch. - backlog#3031: a full upload buffer pool hands out an unpooled buffer instead of refusing the upload; the progress floor already bounds stalls. - backlog#3023: the GET disk-read idle cut-off is configurable (RUSTFS_GET_DISK_READ_IDLE_TIMEOUT_SECS, default 120 s) so a paused client is not truncated after 30 s. - backlog#3075: JSON Lines schema inference keeps the sample collected at the 4 MiB bound instead of failing the query. - backlog#3024: compat admin decryption queues briefly for a slot instead of refusing the third concurrent request. - backlog#3113: HEAD skips its permission probes for the bucket owner. - backlog#3117: control characters in a lifecycle rule ID are rejected on PutBucketLifecycle as well. The lock-crate publication FIFO serves backlog#3111 and moves to the table-catalog PR. Also fixed: a lossy admin error message, an unused s3-client function, two unused test-util features, a duplicated key validation, the s3s footprint baseline, and a stale log event name in the reverse-proxy runbook. The ODM multipart write-back test runs on the large test stack like its sibling. * fix(table-catalog): bound maintenance work and isolate table namespaces (#9174) * fix(table-catalog): bound maintenance work and isolate table namespaces Scan findings in the Iceberg table catalog: - Metadata maintenance deleted every unreferenced *.metadata.json in a table's directory without checking whether a renamed table in another namespace still owned those files (backlog#3060); it now resolves the owning table through the directory index before deleting. - Manifest lists and manifests were decoded with the embedded Avro writer schema and no recursion depth limit; a self-referencing record schema overflowed the stack (backlog#3080). Schemas are validated for depth before decoding. - Compaction decoded every input Parquet file into Arrow batches held in memory; a decompression bomb OOM-killed the server (backlog#3085). Row groups are streamed through a byte budget and oversized inputs are refused. - An S3 request for a key outside an indexed table prefix listed the bucket's whole namespaces/ catalog prefix and failed once it held more than 4096 objects (backlog#3086); a per-bucket directory index answers index misses without the listing. - The maintenance planner read whole orphan files to obtain their modification time (backlog#3087); it reads metadata only. - A client disconnect between persisting active_rename_id and clearing it left the bucket's catalog entry locked at 503 (backlog#3088); the rename runs detached from the request future and clears the flag on every exit. - Table commits held the bucket publication fence until publication, starving every object write (backlog#3111); commits now hold it under a deadline and object writes are admitted between commits. The directory index is managed through three new admin routes under {warehouse}/catalog/directory, registered in the admin route policy. The durable-strong store is left as is (its snapshot partitioning for backlog#3109 and the strong-path deadline are not ported): that backing is being retired separately, so the new tests cover the object backing. * fix(table-catalog): close the review gaps in the scan-fix batch Adversarial review of the table-catalog batch found the fixes narrower than the issues they close and two cost regressions: - backlog#3085: the compaction footer check bounded nothing a page header could not override; parquet sizes its page buffers from the header. Walk every column chunk's page headers (Thrift compact, no decompression) and hold each page and the chunk total to the budget before decoding. The stored-input budget is per rewrite group, so a multi-group job no longer fails after writing earlier outputs. - backlog#3086: the default index-miss scan still counted maintenance reports toward its 4096-object limit; only table entries count now, and the listing itself is bounded to 64 pages. The directory index remains an optional accelerator. - backlog#3111: a create wrote its metadata file under the fence as the first durable mutation, which disarmed the deadline before the unbounded registration inventory. The write precedes publication now. The deadline gets its own knob (RUSTFS_TABLE_CATALOG_PUBLICATION_PREPARATION_TIMEOUT_SECS, default 30 s, never below the lock-acquire timeout) instead of reusing the 5 s lock timeout, and the ownership inventory reads entries concurrently. - Directory candidates are written only while a directory exists; the backfill covers earlier entries. A directory on the migration-fence path is reported as a conflict instead of a parse error. - Dead test hooks, a stale comment and the support-matrix text that did not match the code are removed or corrected. * fix(select): preserve complete samples and Parquet batch limits * fix(s3,ecstore): close the round-2 adversarial review findings DeleteObject resolved x-rustfs-source-version-id for authorization on every request while storage consumes it only for a REPLICA request, so a non-replica caller with s3:DeleteObjectVersion could trigger a plain delete of the current version. The hook now resolves the header under the same predicate as storage, and the regression test covers the non-replica case. HEAD no longer skips its permission probes for the bucket owner; an explicit bucket-policy Deny on the metadata actions binds the owner too. Multipart index pages verify each root against the staging area and prune markers whose staging root is gone, under the marker's own lease with a re-check, so the per-drive index stays proportional to live uploads instead of every key ever multipart-initiated on the drive. SigV2 configuration PUTs that declare an empty body need no Content-MD5: there is no body to replay. The minimum-progress floor stays independent of RUSTFS_HTTP_REQUEST_BODY_READ_TIMEOUT; the knob's documentation now says so instead of claiming 0 disables it. The quota bootstrap window (503 until the background scan publishes the ledger) is kept, because ledger admission must not come from scanner reporting; the compatibility matrix now documents it. The S3 compatibility matrix gains a section for the client-visible validation bounds, and the drive-timeout tuning guide tabulates RUSTFS_GET_DISK_READ_IDLE_TIMEOUT_SECS. * fix(lock): queue table-bucket publication writers fairly Table commits take the bucket publication fence as a write lock; under a stream of commits the lock's try-and-retry acquisition let a new commit jump ahead of object writers that had been waiting longer, which is part of what starved them (backlog#3111). Requests for the publication lock path now take a FIFO ticket per shard: a waiter is admitted only when it is at the head of the queue, cancelled waiters release their turn, and batch acquisitions join the same queue. Other lock paths are unchanged. This moved here from the request-layer batch, where no cited issue covered it; it serves the table-catalog issue this PR closes. * fix(table-catalog): close the round-2 adversarial review findings Parquet page-header walker: read the size fields by field id as parquet does, accepting I16/I32/I64 wire kinds and rejecting any other kind, so a header encoding uncompressed_page_size as I64 can no longer bypass the budget while parquet allocates it. Bound every list, set and map count by the bytes remaining in the chunk, and reject bool collection elements (parquet skips them with zero width, so either width would desynchronize the two parsers); this removes the 2^63-iteration spin on LIST<BOOL>. Start each column-chunk walk at the dictionary page offset whenever the footer declares one, exactly as parquet's byte_range does, instead of min(dictionary, data). Footer pre-walk: before parquet decodes a footer, walk FileMetaData with the hardened cursor and charge the declared row_groups count times the RowGroupMetaData size to the decode budget, because parquet 59 reserves that Vec before reading one element. Preparation deadline: clamp the RUSTFS_TABLE_CATALOG_PUBLICATION_PREPARATION_TIMEOUT_SECS knob to 86,400 seconds so Instant + Duration cannot overflow. Index-miss scan: after each listing page, jump the cursor past the maintenance subtree the page ended in, and allow one extra page per table entry already listed, so job-report history scales with the catalog while a report flood under one table stays bounded. Migration fence: read the fence-path object once and decide from its stored kind whether it is directory state or a fence. Expired or failed create: remove the initial metadata file when the registration fails before any catalog write may have begun; nothing else references it. Correct the comment and support matrix that claimed maintenance reclaims it. Tests cover each walker divergence, the footer row-group reservation, the deadline ceiling, the paging budget for honest catalogs and for a report flood, and the removal of the unpublished metadata file. * test(admin): register the catalog directory routes in the route matrix The table-catalog batch added GET, POST and DELETE {warehouse}/catalog/directory on both the Iceberg and the compat admin prefixes without listing them in the admin route matrix, so test_admin_route_matrix_matches_registered_routes failed on CI. * fix(s3,ecstore): close the round-3 adversarial review findings Multipart index pages now treat the cursor as the last examined marker rather than the last root, so a page that pruned a stale marker still validates and ListMultipartUploads keeps using the index. A marker is written before its staging root is published, so pruning spares markers younger than a five-minute grace period and an existing marker is touched on every new upload; the prune re-checks both under the marker lease. Bounded delete-recovery inspection rides the durable-mutation lane like the unbounded path instead of the one-worker foreground lane. The on-demand-migration inline body budget scales with the object size (at least 64 KiB/s) instead of a flat 120 s. The control-body digest service honors RUSTFS_HTTP_REQUEST_BODY_READ_TIMEOUT=0 as "no inactivity timeout" like every other reader of that knob. Quota reconciliation throttle objects are deleted together with a retired generation's ledger, so bucket turnover leaves no residue. The PUT guard test shim is gone; tests call the shared guard. Bounded DiskAPI defaults answer MethodNotAllowed like their siblings. Stale budget docs, an unreachable CSV arm, an always-Some limit parameter, commented-out constants and a misplaced doc comment are cleaned up; weak test assertions name the expected error; lifecycle rule-ID control characters gain a test. The compatibility matrix records the CopyObject tag-copy permission, the exact SigV2 empty-body predicate, replica and migration writes of control-character keys, and the SigV2 DeleteObjects digest rule. * fix(table-catalog,heal): close the round-3 review findings A. Pack compaction rewrite groups to min(max-rewrite-bytes-per-job, TABLE_COMPACTION_MAX_STORED_BYTES), reject an over-ceiling group before any output is written, and name the ceiling in the commit-path errors; document the effective cap in the support matrix and scheduler runbook. B. Keep TaskCancelled/TaskTimeout classification on replacement passes: the namespace verification runs only for completed or storage-failed passes, in the erasure healer and the task bucket prepass. C. Compare the replacement bucket pass per bucket: a recorded identity that is gone or replaced still errors or restarts the pass, buckets created mid-pass are appended to the identities, pending_buckets and a persisted positional order without rewinding progress or the checkpoint; verification ignores additions. D. Read the migration-fence object once in the object-backed write permit and dispatch on its kind. E. Drop the strong store's copy of maintenance_cleanup_roots. F. Remove the unused test hooks fail_put_prefix, corrupt_put_prefix and pause_after_metadata; make compact_parquet_data_files and PublicationTicket.state private. G. Document the body-revision identity, the legacy revalidation budget and its HTTP 503 in the support matrix. H. Define HEAL_REPLACEMENT_RECOVERY_DIR once in ecstore and consume it from heal through the storage_api facade. * fix(ecstore): cap the index pages one disk may answer Pruned multipart index pages may carry few roots, so the page cursor no longer bounds how many pages a peer can make a listing fetch. Cut a disk off after 4096 pages (about one million examined markers) so it contributes no coverage and the listing falls back to the staging scan. * fix(ecstore,table-catalog): close the round-4 adversarial review findings Multipart index markers dated in the future (a backwards clock step between recording a marker and publishing its staging root) were judged old and could be pruned while their upload was in flight; a future-dated marker is now kept. The marker touch on EEXIST is best effort, so a filesystem that refuses to set times no longer fails the upload commit. Pruning runs on the durable-mutation lane instead of the one-worker foreground lane. Quota reconciliation keeps one throttle object per bucket incarnation (a quota revision change replaces it instead of orphaning the previous revision's object), the retirement cleanup deletes it ahead of the ledger and also when the ledger is already gone, and a scan the retirement already refused writes nothing back. Bounded DiskAPI defaults answer UnsupportedDisk, the verdict the aggregate gives a peer without the protocol. The compaction ceiling pre-check runs over every rewrite group before any output is written, and the worker's ceiling error no longer names a placeholder group. Docs: the inline on-demand-migration client budget, the protocol-2 downgrade conflict of replacement passes, the exact footprint numbers. |
||
|
Chris An
|
158aa276e9 |
fix(storage): bound heal retries and recover pool metadata (#9164)
* fix(ecstore): retry metadata writes rejected before staging * fix(ecstore): reuse registered disks during startup * fix(ecstore): retain remote health monitors during reconnect * test(heal): release the commit barrier during graceful shutdown * test(e2e): register graceful shutdown barrier regressions * fix(ecstore): roll back interrupted pool metadata commits * test(upgrade): honor the 1.0.4 delete recovery protocol * fix(heal): bound admission during object rate pressure * test(e2e): handle rebalance completion before stop * test(e2e): observe rebalance activity before the start reply * test(rebalance): hold the first run until stop cancels admission |
||
|
Chris An
|
a97c65895c |
fix(proxies,targets,obs): bound untrusted input in leaf crates (#9166)
Scan findings in crates whose behavior is driven by untrusted input and that stand alone from the storage core: - trusted-proxies: the simple implementation takes the client address from the right-most untrusted X-Forwarded-For hop instead of the left-most (backlog#3082); the legacy validator never falls back to the proxy identity when the chain is rejected and reads X-Forwarded-For unless RFC 7239 is explicitly selected (backlog#3051). - targets: webhook and similar outbound clients drain responses up to a byte limit instead of buffering arbitrary bodies (backlog#3062); the on-disk notification queue has a shared byte budget in addition to the entry count, configurable via RUSTFS_TARGET_STORE_MAX_BYTES (backlog#3122); AMQP and Redis manifests declare the broker url as a secret and bind credentials to their connection fields, and the plugin-instance PUT refuses to restore a redacted secret when the connection fields changed (backlog#3065, backlog#3066). - obs: the debug-level s3s response dump is filtered so STS and service-account credentials never reach the log (backlog#3029). |
||
|
唐小鸭
|
116a5299b2 |
fix(replication): replay the MRF journal periodically (#9139)
* fix(replication): replay the MRF journal periodically The persisted MRF journal was replayed only at startup. Delete-marker purges whose watch window drained with targets unreachable, and replications that overflowed the worker queues, reach the journal while the server runs, so they waited for a restart to converge. Move the startup pass into replay_mrf_journal and rerun it every five minutes from a background loop. The existing recovery leader lock keeps one pass at a time across the cluster, including the startup pass. * fix(replication): space periodic MRF replay across nodes Every node runs the MRF replay loop and the recovery leader lock only serializes passes, so an N-node cluster replayed the same retained backlog N times per interval while a target stayed down. Record the start time of a pass that finds entries in a shared replay stamp and skip a periodic pass when any node started one within the interval. The startup pass still always runs. |
||
|
Boyi (Elliot) Li
|
cc08e0d11b |
feat(fips): route internode RPC HMAC through aws-lc-rs FIPS (#9137)
* feat(fips): route internode RPC HMAC through aws-lc-rs FIPS * Revise KMS cryptographic compliance details Updated the KMS cryptographic compliance documentation to reflect changes in HMAC authentication and added details about the internode RPC HMAC authentication. Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Hauser Zhi <housemecn@gmail.com> --------- Signed-off-by: Hauser Zhi <housemecn@gmail.com> Co-authored-by: Hauser Zhi <housemecn@gmail.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Hauser Zhi
|
41b13aeb8b |
fix(tls): unify FIPS provider policy across clients (#9125)
* fix(tls): enforce active FIPS provider at startup Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * ci: provision FIPS validation runner Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(tls): validate FIPS before observability startup Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(tls): reuse the installed crypto provider Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): gate Linux-only commit helpers Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(tls): centralize FIPS provider policy Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> Co-authored-by: Chris An <anzhengchao@gmail.com> |
||
|
Henry Guo
|
00a238f838 |
fix(table-catalog): resolve superseded staged commits (#9124)
fix(table-catalog): diagnose superseded staged commits Co-authored-by: Chris An <anzhengchao@gmail.com> |
||
|
hector
|
965d805a4f |
ci: run distributed and upgrade e2e on schedule only (#9126)
Remove pull_request from the distributed and upgrade workflows. Distributed stays on the nightly cron, and upgrade stays on the Monday cron plus version tags. Both remain manual-dispatch lanes and stay outside the required merge checks. Co-authored-by: majinghe <majinghe@users.noreply.github.com> |
||
|
Hauser Zhi
|
ed4099ebe1 |
chore(deps): update crates and fix storage CI regressions (#9111)
* chore(deps): refresh crates and adopt openidconnect 4.1 Use published OIDC/OAuth2 releases and reject malformed HMAC signature lengths before upstream verification. Refresh workspace and fuzz locks, and advance the local zstd compatibility patch to 0.13.4 while retaining the fixed 0.14.3 codec backend. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * chore(deps): refresh compatible lockfile resolutions Resolve Prost build and derive dependencies with itertools 0.14 and prefer windows-sys 0.61.2 within the existing supported version ranges. Retain all package versions, sources and checksums. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(connect): separate cleanup checks from short probe windows Use the functional measurement budget for the cleanup HTTP fixture so setup and measured I/O can finish before checking delete status handling. Exercise both normal scheduling and an in-flight PUT clock advance, and include target classification in unexpected-failure diagnostics. Retain the cleanup deadline, outer timeout and delete-stage assertions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): isolate stalled multipart disk timeouts Apply the short injected timeout only to the deliberately stalled disk. Healthy disks retain their normal operation budgets, while logical lock, physical owner, immediate CopyObject and final shard assertions remain. Keep overrides scoped across cloned and reconnected handles, preserve shorter caller deadlines and explicit zero timeouts, and report drive health when the real CopyObject assertion fails. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(rebalance): retry metadata write lock quorum admission Recognize typed write-lock quorum failures for rebalance.bin within the existing bounded metadata admission retry policy. Each attempt must acquire quorum before loading or saving metadata; data writes are not replayed by this retry. Cover recovered and persistent quorum failures, cancellation after a failed attempt, and rejection of unrelated resource or read-lock errors. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Boyi (Elliot) Li
|
3a1fac16e3 |
feat(tls): add opt-in fips feature for aws-lc-rs and rustls (#9084)
Add an opt-in `fips` Cargo feature to `rustfs` and `rustfs-tls-runtime` that enables `rustls/fips` and `tokio-rustls/fips` (activating `aws-lc-rs/fips` / `aws-lc-fips-sys` and `rustls-webpki/aws-lc-rs-fips`). When compiled with `--features fips`, `startup_runtime_hooks` installs `rustls::crypto::default_fips_provider()` as the process-wide default crypto provider and asserts `provider.fips()` at startup. Also lock `aws-lc-fips-sys v0.14.4` in `Cargo.lock`, allow the `OpenSSL` license on `aws-lc-fips-sys` in `deny.toml`, and update `docs/operations/kms-cryptographic-compliance.md` to document the scope of `-p rustfs --features fips` vs. `rustfs-crypto`. Co-authored-by: hector <42570491+majinghe@users.noreply.github.com> |
||
|
Hauser Zhi
|
ecbddc188a |
docs(scanner): define prefix statistics contract (#9116)
Define the scope, authority, coverage, persistence, and resource guarantees needed before scanner-derived current-object statistics can be enabled. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
GatewayJ
|
04690f7da5 |
fix(s3-tables): clean empty catalogs and cache registration reads (#9063)
* fix(s3-tables): clean up empty tables and parallelize catalog reads * fix(s3-tables): bound catalog registration index reads * fix(s3-tables): cache validated registration ownership in memory |
||
|
Chris An
|
1cb4c0ebe2 |
ci: stamp the plan contract and rename the workspace lint-and-build key (#9095)
* ci: gate the plan on a contract marker A base planner predating the NIGHTLY_JOBS split still emits the old required set, and the strict verifier would reject it. Select now stamps plan_contract=2 and the workflow refuses any plan without the current marker, falling back to the full matrix, so a stale base planner can neither authorize the new narrower selection nor force a job the new workflow no longer schedules. * ci: rename the workspace lint-and-build job key and evidence path The job key test-and-lint named a job that runs no tests: it does Clippy, the test-binary archive, doctests and the log-rule anchors, while the tests live in workspace-tests. Anyone reading needs: [test-and-lint] would conclude tests had already gated that dependency. The key becomes workspace-lint-and-build to match its display name, and the shared evidence directory becomes artifacts/workspace so the path stops asserting the same thing. The aggregate's display name stays Test and Lint because that is the live ruleset check context. ci-gates.md also stops attributing the ECStore test run to the wrong job. |
||
|
Chris An
|
e5563e6004 |
feat(release): gate publication on candidate certification (R01) (#9090)
Add a fail-closed release certification path so incomplete or substituted evidence blocks every official publication exit. scripts/release/release_gate.py validates the versioned release-validation plan, the candidate identity and the G01 evidence-envelope set, then re-checks promoted bytes against the passed list. build/docker/package/helm publication jobs require that certification and stage artifacts before promoting identical bytes. |
||
|
Chris An
|
51275cb7d8 |
ci: move specialized lanes off the pull-request path (#9093)
Connect Feature Boundaries, real io_uring and the rio-v2, Swift, SFTP and HTTP/3 feature lanes no longer gate a merge. They run only in the scheduled and manual full-validation runs, which report a regression within a day; on the pull-request path they bought no earlier signal worth the runner occupancy (Connect alone measured 53.5 median minutes and occasionally set the wall). Workspace lint and tests, the short core contract, the debug build, E2E smoke and the S3 lanes stay required on PRs and merge groups. |
||
|
Chris An
|
3bf270b49f |
ci: record CI01 measurements, drill evidence and fix the OIDC live gate (#9091)
* ci: retain CI01 timing sample and fix the OIDC live gate tool Record the measured PR feedback and runner occupancy for the CI plan's CI01 criterion, including the honest gap between the experimental targets and the sampled medians, and replace the awscurl dependency in the live Keycloak gate with the AWS CLI so the security suite runs on PEP 668 managed fleets. * docs(testing): attach live run evidence to the event-drill index |
||
|
Chris An
|
d8d064bd18 | docs(testing): index event-class drills and retained CI evidence (#9088) | ||
|
Chris An
|
b8ee0301e9 |
feat(scanner): fair bucket service across restarts and budgeted first enumeration (#9083)
* feat(scanner): bound first-build enumeration with the cycle budget Charge raw directory entries and their name bytes, and let one bucket scan hold a child budget that stops its own token while charging the cycle. A count budget also bounds one directory listing, so a wide first build can no longer finish a full read_dir before the first object is charged. The new stop reason maps onto the persisted Directories variant so older readers keep decoding checkpoints. A directory with queued children keeps its index until they end, and every index on the active walk path is saved as an optional map field: losing either one would recharge its replay and stop a fixed budget from advancing. * feat(scanner): keep bucket service order across restarts Save each set's first-in-first-out bucket order as an optional map field of its usage cache, including after an incomplete cycle, which otherwise writes nothing. Durable age now outranks the dirty-first and leader-local cohort orders, which only break its ties, so a restart cannot hand the budget back to the same first buckets. Failed turns add visible debt without moving a bucket forward; a budget stop or a resource veto adds none. A bucket that spends its quantum and saves forward motion returns to the queue tail within the cycle. The order authorizes nothing: another movement epoch or a corrupt value starts a new one, and a recreated bucket joins as a new arrival. * test(scanner): cover quantum handoff and saved service order Run the real worker queue with injected bucket shares and verify a spent bucket yields to waiters, a share without durable motion ends its turn, and the saved order keeps certified coverage while rejecting older owners. * docs(scanner): describe budgeted first build and service order * test(scanner): pin index retention and requeue motion Bound the nested restart fixture tightly enough that dropping a directory index fails it, and cover a repeated share whose frontier never moves. * fix(scanner): keep service generations and owner stops intact Queue a new bucket behind waiters but ahead of buckets that already had a turn, matching the leader-local cohort, and keep the set cache untouched when the owner was cancelled so an interrupted cycle cannot advance its revision. The order holds one record per inventory bucket, like the set cache's entries, so a truncating bound that could starve the long-waiting buckets is removed. Publish the incomplete scope before saving so a contended lock cannot delay it. * ci(e2e): refresh the full selection digests * fix(scanner): correct a comment spelling * fix(scanner): separate service debt from vetoes and stale owners Keep a resource veto and a remote count-budget stop out of failure debt, settle the claim before a requeue exposes the bucket to another worker, and stamp the service order with its leader so an older owner cannot replace it after an incomplete save. The checkpoint fixture's budget now covers both service orders. * fix(ecstore): create parents before resource-scoped writes #9064 moved parent creation into the lazy writer, so a write that holds a resource unit reached the native open without its parent directory and failed with FileNotFound on nested paths. * fix(e2e): retry SlowDown codes during upgrade write readiness The rolling-upgrade readiness probe only retried 503 ServiceUnavailable, so a transient 503 SlowDown from a still-recovering node failed the four_node_rolling_upgrade test immediately. Accept SlowDown and SlowDownRead alongside ServiceUnavailable, matching the existing chaos test transient-error set. |
||
|
Chris An
|
0e5fe8c658 |
feat(storage): share physical admission across scanner and heal (#8911)
* feat(scanner): share node admission with background heal * fix(scanner): drain candidates after listing completion * fix(scanner): yield cache leases for shared node admission * test(scanner): attest adaptive admission in process receipts * fix(concurrency): declare Tokio time dependency * feat(concurrency): retain bounded physical resource subsets * fix(scanner): retain resource ownership in native filesystem jobs * feat(concurrency): acquire physical groups before node work * feat(scanner): gate background reads by physical device * feat(concurrency): combine preallocated resource selections * feat(scanner): share physical topology across disk plans * fix(storage): retain only native disk resource subsets * feat(storage): admit auxiliary physical resources without waiting * feat(heal): admit physical resources before object repair * fix(storage): retain resources during native format probes * fix(storage): retain resources through direct writer I/O * fix(storage): retain resources through buffered writer I/O * fix(storage): retain physical resources through streamed reads * fix(storage): retain resources through native filesystem jobs * fix(storage): retain physical ownership during bitrot reads * fix(ecstore): retain resources during positioned reads * fix(ecstore): retain staged metadata write ownership * fix(ecstore): retain native io_uring read ownership * fix(ecstore): enforce receiver-side background admission * fix(ecstore): retain resources through whole-file native I/O * fix(ecstore): retain physical ownership during durable rename sync * fix(storage): account for retained background stream buffers * fix(storage): retain memory charges for background read results * fix(storage): account for native io_uring read allocations * fix(storage): retain memory charges for direct-write buffers * fix(ecstore): account for heal and RPC reader buffers * test(scanner): verify samples and clean concurrent collectors * fix(storage): retain inline and HTTP writer memory reservations * perf(rio): reuse reserved HTTP writer storage * fix(rpc): reserve background scanner pipe storage * perf(rpc): reuse reserved response reader storage * feat(scanner): measure cache copy allocation requests * fix(rpc): budget namespace scanner responses * fix(scanner): retain the enabled supervisor task handle * fix(scanner): budget namespace frame read staging * fix(scanner): account namespace envelope decode storage * fix(heal): retain and join the MRF consumer on shutdown * test(heal): isolate process-global MRF consumer fixture * fix(heal): join manager workers during shutdown * test(heal): verify shutdown at native read boundaries * fix(heal): drain channel commands before stopping MRF * fix(heal): admit resume scans and retain replacement workers * build: pin profiling sources and trust fixed ownership dependencies * fix(heal): yield node admission while accepted native work drains * fix(startup): retain shutdown work and preserve deadline failures * fix(heal): admit inventory probes and retain replacement work * fix(shutdown): preserve maintenance and frontend join failures * fix(shutdown): retain HTTP child tasks through real retirement * fix(shutdown): join actual TLS reload work before retirement * test(multipart): preserve failures before fault barriers * fix: keep physical admission inside repository guards Route the multipart lock diagnostics filter through the e2e storage compatibility module, reuse the parent module's s3s body aliases in the namespace memory test, and share one Reed-Solomon verify error mapping across the reserved-buffer and default paths. * fix: reconcile physical admission with main's new call sites Main added a source-admission metadata write layer, a CAS write fence, fixture writers and scanner handoff tests against the old borrowed-slice and node-permit shapes. Pass owned Bytes through the new metadata layer, answer the scanner handoff with ScannerNodePermit, gate the test-only disk lookup re-export, and clear the lint findings that stable 1.99 reports for the shutdown and admission code without suppressions. * test(concurrency): name reversed attachment groups clearly * fix(ecstore): box disk operation futures below admission layers Every LocalDiskWrapper operation reaches the LocalDisk future through the health, admission and timeout layers (track_cleanup_mutation, the track_disk_health family, the io-permit and native-resource admission split, and track_admitted_disk_operation). Each layer embedded that future, and unoptimized builds copy it into every awaiting frame, so one local delete cost about 1.3 MiB of stack and 16 tests overflowed libtest's default 2 MiB stack. Box the operation future once where track_admitted_disk_operation creates it, through a non-async helper so the temporary never stays in a polled frame. The layers above now hold only a pointer; the cost is one allocation per local disk operation. * fix(heal): box object heal futures below admission The admission wrappers took the object heal future by value, so unoptimized builds copied it into three awaiting frames, about 650 KiB per heal. Create it through a non-async helper instead, leaving only a pointer in those frames. * test(ecstore): count the I/O backend's root FD owner |
||
|
Chris An
|
3fc610c8a2 |
fix(scanner): track heal generation and payload observations (#9008)
* fix(scanner): distinguish missing heal identity fields * test(scanner): simplify identity case types * fix(scanner): track heal generation and payload observations * fix(scanner): treat heal results as observations * test(scanner): route heal integration through storage boundary * test(scanner): scope the heal generation fixture * test(heal): fill observation fields in the retained outcome fixture |
||
|
Chris An
|
7d45685d70 |
feat: classify scanner metadata calls by diagnostic prefix (#9052)
* feat: classify scanner metadata calls by diagnostic prefix * fix: export zero scanner visit counts for diagnostic scope |
||
|
Chris An
|
76bab81aa6 |
fix: reject unverified scanner performance evidence (#9049)
* fix: reject unverified scanner performance evidence * fix: keep unverified G14 descriptors unpublished * test: release scanner fixtures without signaling stale pids |
||
|
Mathew Beane
|
7e536843ea |
fix(audit): emit a bounded audit entry for authorization denials (#9073)
A request refused by the authorization layer was logged as s3_authorization_denied but never reached the audit stream. Emit an audit entry from DenialContext::deny with the account, bucket, object, denied action and denial reason. Entries are bounded by a per-class token bucket (lower for anonymous callers) with a suppressed count carried on the next entry, quiet probe denials are skipped, DeleteObjects audits its refused keys as one aggregate entry, and the S3 response is unchanged. |
||
|
唐小鸭
|
7672867476 |
fix(replication): count force-delete outcomes in replication stats (#9072)
A replicated force-delete that a target rejected (for example a peer whose credentials lack s3:ForceDeleteObject) was only logged; the per-target replication counters never saw it, so `replication status` reported zero failures while the sites had already diverged. Build a ReplicatedTargetInfo for every target outcome of replicate_force_delete_to_targets (missing client, offline target, rejected remove, success) with a Pending previous status, and feed each through ReplicationStats::update so rejections count as failures and accepted force-deletes count as replicated. Document that the managed site-replicator-0 policy must name s3:ForceDeleteObject explicitly and that a bucket-replication target with its own credentials needs the same grant. |
||
|
Chris An
|
191c95a469 |
ci: enforce affected validation and rehearse bounded shards (#9070)
* ci: enforce affected validation and rehearse bounded shards * test(ci): keep regression output stable under forced color |
||
|
Chris An
|
dacb003856 |
ci: let failed-job reruns reuse the producer's test archives (#9071)
Since #9059 the shared test archives and debug binaries are named with the run attempt, and the archive identity binds the attempt. A "Re-run failed jobs" attempt does not rerun the successful producer, so its consumers look for <name>-2, fail with "Artifact not found", and every flaky test forces a full rerun. Give those artifacts one name per run, overwrite them when a producer does rerun, and bind the archive identity to the run instead of the attempt. Source, tree, lock, toolchain and environment stay bound, and an archive from another run is still rejected. |
||
|
Chris An
|
4fcbf60f10 |
ci: unify validation planning, artifacts and suite orchestration (#9059)
* ci: unify validation planning, artifacts and suite orchestration * fix(ci): preserve JSON diagnostics for release checks * fix(ci): keep manual attempts without plans visible * fix(ci): distinguish nextest setup scripts from test results * ci: reuse extracted archives and consolidate duplicate checks |
||
|
Chris An
|
b7ff09e357 | chore: strengthen agent delivery standards and validation (#9058) | ||
|
Hauser
|
e0444b519d |
feat(ecstore): add sampled batch CPU diagnostics (#9019)
* feat(ecstore): add sampled batch CPU diagnostics Keep timing and request correlation in a separate default-off feature. Preserve the existing positioned-read backend, byte and error contracts. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): route batch diagnostics through storage facade Keep the diagnostic implementation private to its owner and expose explicit feature-gated helpers through the existing storage boundary. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): retain diagnostic IDs across GET producer tasks Capture request context before spawning the legacy producer. Keep the existing producer, cancellation and read-lock lifetime intact. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): avoid unused batch read index Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(obs): isolate local batch diagnostic JSON events Omit span metadata and inferred parent IDs for diagnostic root events. Verify per-shard index order and document existing production metric timing. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): accelerate receipt fixture in CI Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(heal): allocate stack for bucket recovery fixture Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(s3): allocate stack for conditional GET fixture Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
cxymds
|
c90ba5922f |
fix(tiering): diagnose blocked tier deletion and retry cleanup (#9035)
* fix(tiering): diagnose blocked tier deletion and retry cleanup * fix(tiering): reuse admin errors for deletion diagnostics |
||
|
cxymds
|
4a63e53161 |
test(validation): run canonical cases and build pinned Heal tools (#8455)
* test(validation): add pinned auto-testing manual lane * Install pinned ILM client for canonical validation lane * Prepare Python client for pool validation cases * Install validation client for replication cases * test(validation): provide the Python client for Heal cases * test(validation): build the candidate-bound Heal inspector artifact * test(validation): expose tool artifacts through the existing manual lane * test(validation): verify pinned Heal tools before fleet execution * test(validation): verify Heal tools against released shard captures * ci: audit retained validation environments before finalization * test(validation): add optional native Heal regression tooling * test(validation): route native tools through the existing driver * Run validation tool resolution on hosted Linux without acquiring the fleet lock * test(validation): bind C07 to pinned native authority evidence * test(validation): build optional C10 fixture codec on hosted Linux * test(ci): bind C10 execution to the verified shared codec * feat(ci): expose read-only validation fleet inventory * docs(ci): describe supported validation recovery * test(ci): add isolated manager native validation builds * test(ci): expose manager regressions through tools-only dispatch * ci(validation): consume pinned manager native prerequisites * fix(heal): preserve typed bucket errors for bounded retries * fix(heal): preserve typed errors and SSE-C response headers * ci(validation): publish isolated immutable candidate packages * test(validation): enforce private candidate manifest schema * test(validation): verify isolated package publication failures * ci(validation): dispatch isolated packages through the driver * fix(heal): deeply verify read repair payload failures * test(ecstore): verify incarnation fence at repair commit * ci(validation): verify C11 native storage evidence * fix(ci): isolate validation package publication credentials * test(ci): archive scoped F07 directory fsync evidence * fix(ecstore): retain typed quorum errors for missing parts * chore(ecstore): shrink typed-error ratchet baseline * fix(ecstore): retain below-quorum data during no-remove heal * fix(ci): allow recovery of completed cancelled validation runs * fix(heal): retain exhausted admin batch repair receipts * fix(observability): retain typed object error diagnostics * fix(s3): map typed RPC unavailability to service refusal * fix(filemeta): classify supported inline framing damage as corrupt * test(ecstore): exercise heal during committed overwrite cleanup * ci(validation): admit source-bound G01 native evidence before fleet execution * test(ecstore): fence version races and bind cleanup slots * ci(validation): require complete race native runtime preflight * test(ecstore): fence marker and batch deletes during heal * ci(validation): require G04 native and runtime admission * test(ecstore): fence multipart continue complete and abort during heal * test(ecstore): cover default heal owners and copy metadata races * ci(validation): admit source-bound owner and copy race proofs * test(scanner): assert typed corruption after metadata truncation * fix(ci): require complete native admission for multipart races * test(validation): wire I05 queued-successor native prerequisites * fix(scanner): bind object heals to the scanning disk scope * test(scanner): reuse storage boundary for scope regression * style: format RPC digest precedence assertion * fix(heal): query automatic tasks on their owning node * test(admin): reuse error facade in heal query regression * fix(heal): retain cancellation after successful bucket storage I/O * fix(ci): retain validation evidence outside runner temp * fix(heal): retain resolved object identity on terminal failures * test(scanner): adapt scope regression to admission receipts * ci(validation): inspect shared runner without fleet access * ci(validation): prepare Python before creating client environments * ci(validation): isolate each client virtual environment * ci(validation): retain interrupted evidence without fleet recovery * ci(validation): bind recovery to retained runner evidence * ci(validation): wire persistence and lock source plans * fix(validation): align read-only dispatch defaults --------- Co-authored-by: Chris <anzhengchao@gmail.com> |
||
|
Hauser
|
a9d3022f37 |
fix(ecstore): prove multipart part sizes before cleanup (#9037)
* fix(ecstore): prove multipart part sizes before cleanup Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(ecstore): merge strict multipart observation passes Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test: respect lifecycle architecture boundaries Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test: fix multipart E2E clippy warnings Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
RustFS
|
7581a70412 |
feat(s3): serve Veeam SOSAPI system and capacity XML (#9007)
* feat(s3): serve Veeam SOSAPI system and capacity XML Authenticated GET and HEAD of the two well-known SOSAPI keys return generated XML, and read preconditions use that generated ETag and Last-Modified. Mutations of those keys are rejected while the feature is on. Co-authored-by: RustFS <hello@rustfs.com> * fix(s3): simplify Veeam quota metadata lookup Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(s3): harden Veeam SOSAPI capacity and reads Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(s3): bind Veeam capacity to bucket identity Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(s3): protect virtual objects and coalesce refreshes Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): refresh Veeam profile selections Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): bound pagination fixture writes Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: Hauser <housemecn@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Chris An
|
f6ad79dd58 |
feat(plugins): add minimal original-user runner integration (#9017)
* feat(plugins): add minimal original-user runner integration * test(plugins): use the main storage fixture * fix(plugins): reuse runner connections across requests * test(plugins): allocate stack for delegation fixture |
||
|
Chris An
|
8e88419f3a |
feat(scanner): share node admission with background heal (#8903)
* feat(scanner): share node admission with background heal * fix(scanner): drain candidates after listing completion * fix(scanner): yield cache leases for shared node admission * test(scanner): attest adaptive admission in process receipts * fix(concurrency): declare Tokio time dependency * fix(scanner): route test disk access through owner facade * test(scanner): fix lint warnings |
||
|
Chris An
|
7a3bf8cdbd |
fix(ecstore): resolve dangling latest versions in metadata updates (#9009)
* fix(ecstore): select the latest version at read quorum like MinIO GET and HEAD escalated latest-version selection to write quorum when enough metadata answered and skipped a newer version below write quorum on degraded reads. LIST and heal already pick the newer side of a split write at read quorum, so such keys listed but answered 503 until heal. Select with MinIO's getObjectFileInfo rule: the most common mod time that reaches the object's data blocks wins, ties go to the newer time, and one content identity at that time must reach the same quorum. Two identities tied at the selected time fail instead of resolving by hash order, and the selected identity's version-stack shape follows disk order. * fix(ecstore): resolve selection ties by disk order and guard writes Review of the read-quorum selection found three gaps: - Rejecting a same-time half split made explicit-version reads of a partially applied tagging update fail and blocked the retry that repairs it. Resolve ties to the identity on the lowest-index disk, for reads and heal alike, instead of rejecting or using map order. - Delete markers and empty versions can lower the common read quorum below the selected version's data blocks, so HEAD could report a version GET cannot decode. Require that many copies, as MinIO's getLatestFileInfo does. - A selection can now return exactly read-quorum disks, and partial metadata writes are not rolled back. Refuse an update before writing when its target disks cannot reach write quorum. * fix(ecstore): count identity copies in the data-blocks guard The first-candidate shortcut counts byte-identical copies, so copies that differ only in identity-neutral fields such as replication status were missing from the data-blocks check. When the shortcut's count falls short, count every copy of the selected content identity before refusing the version. * fix(ecstore): keep commit-proof selection for system-bucket reads Pool metadata's PREPARE/COMMIT recovery and identity epochs treat an ordinary read of pool.bin as proof that the record committed. Under the read-quorum rule, a COMMIT that reached only half of a 2+2 set was served as committed, and the pool_meta_native_physical_* integration tests failed. Route reads through select_fileinfo_for_read: S3 buckets keep MinIO's read-quorum rule, while is_meta_bucketname reads use the previous algorithm unchanged as select_committed_fileinfo, so a split stays unreadable until native recovery resolves it. * fix(ecstore): serve the latest readable version past dangling versions A failed versioned PUT whose rollback also failed can leave the new version on top of fewer drives than its data blocks. Reading only each drive's top version then left the committed version below it short of read quorum, so GET and HEAD of the latest version answered 503 although that version is intact in every stack. MinIO resolves the stacks in pickLatestQuorumFilesInfo. When every drive answered and an S3 latest read still cannot select a version, walk the newest version headers of each stack through exact-version reads: serve the readable version with the newest selected modification time, skip versions whose exact read proves them dangling (the NoSuchVersion an explicit GET reports) and queue heal for them, and keep the read-quorum error on any other outcome. Candidates are ordered by each version's newest modification time across drives, and at most eight exact reads are spent. Test PUTs now wait for every drive so tests that edit one drive's metadata no longer race the PUT's background tail writes. * fix(ecstore): resolve dangling latest versions in metadata updates A failed versioned PUT whose rollback also failed can leave the new version on fewer drives than its data blocks. GET and HEAD of the latest version already look past such versions, but read-before-write callers without a version id still answered 503: object tagging read each drive's top version with the quorum-error read, and put_object_metadata and the CopyObject source read selected from read_all_xl, whose merged latest version is missing on the other drives, so object_quorum_from_meta failed before selection. Object tagging now reads through a variant that runs the same walk past dangling versions as a latest GET while keeping the quorum error for a dangling key. put_object_metadata and the CopyObject source try the walk when their quorum or selection step fails with read quorum, before the existing dangling cleanup could report a key with a committed version absent. A version-only copy takes the served version's own copies and write quorum. The walk still applies only to S3 buckets, latest reads, and sets where every drive answered, and the update still targets only the served version's disks, so a version below write quorum is refused before writing. * fix(ecstore): refuse version-only copies below write quorum up front Review of the dangling-latest metadata updates found three gaps: - A version-only CopyObject wrote the new version and reverted on a write-quorum miss; refuse it before writing when the served version's disks cannot reach its write quorum, like update_object_meta_with_opts. - Replication status write-back, including the zero-write ValidateOnly mode, now keeps its read-quorum error and never walks past dangling versions, since the walk queues heal for the versions it skips. - The erasure-coding doc overstated that these callers never report a dangling key as absent; put_object_metadata and CopyObject keep their existing dangling cleanup when the walk finds no readable version. |
||
|
Hauser
|
b4026b43b2 |
feat(observability): attribute local batch shard reads (#8991)
Record local batch eligibility, blocking queue and execution time, shard outcomes, and actual read bytes behind the existing GET-stage metrics gate. Keep captured metric handles with the blocking worker after cancellation and use scoped low-latency histogram buckets for the new duration series. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Chris An
|
f879946ead | chore: remove redundant review skills (#9022) | ||
|
Chris An
|
d88f524d9f |
fix(ecstore): serve the latest readable version past dangling versions (#9004)
* fix(ecstore): select the latest version at read quorum like MinIO GET and HEAD escalated latest-version selection to write quorum when enough metadata answered and skipped a newer version below write quorum on degraded reads. LIST and heal already pick the newer side of a split write at read quorum, so such keys listed but answered 503 until heal. Select with MinIO's getObjectFileInfo rule: the most common mod time that reaches the object's data blocks wins, ties go to the newer time, and one content identity at that time must reach the same quorum. Two identities tied at the selected time fail instead of resolving by hash order, and the selected identity's version-stack shape follows disk order. * fix(ecstore): resolve selection ties by disk order and guard writes Review of the read-quorum selection found three gaps: - Rejecting a same-time half split made explicit-version reads of a partially applied tagging update fail and blocked the retry that repairs it. Resolve ties to the identity on the lowest-index disk, for reads and heal alike, instead of rejecting or using map order. - Delete markers and empty versions can lower the common read quorum below the selected version's data blocks, so HEAD could report a version GET cannot decode. Require that many copies, as MinIO's getLatestFileInfo does. - A selection can now return exactly read-quorum disks, and partial metadata writes are not rolled back. Refuse an update before writing when its target disks cannot reach write quorum. * fix(ecstore): count identity copies in the data-blocks guard The first-candidate shortcut counts byte-identical copies, so copies that differ only in identity-neutral fields such as replication status were missing from the data-blocks check. When the shortcut's count falls short, count every copy of the selected content identity before refusing the version. * fix(ecstore): keep commit-proof selection for system-bucket reads Pool metadata's PREPARE/COMMIT recovery and identity epochs treat an ordinary read of pool.bin as proof that the record committed. Under the read-quorum rule, a COMMIT that reached only half of a 2+2 set was served as committed, and the pool_meta_native_physical_* integration tests failed. Route reads through select_fileinfo_for_read: S3 buckets keep MinIO's read-quorum rule, while is_meta_bucketname reads use the previous algorithm unchanged as select_committed_fileinfo, so a split stays unreadable until native recovery resolves it. * fix(ecstore): serve the latest readable version past dangling versions A failed versioned PUT whose rollback also failed can leave the new version on top of fewer drives than its data blocks. Reading only each drive's top version then left the committed version below it short of read quorum, so GET and HEAD of the latest version answered 503 although that version is intact in every stack. MinIO resolves the stacks in pickLatestQuorumFilesInfo. When every drive answered and an S3 latest read still cannot select a version, walk the newest version headers of each stack through exact-version reads: serve the readable version with the newest selected modification time, skip versions whose exact read proves them dangling (the NoSuchVersion an explicit GET reports) and queue heal for them, and keep the read-quorum error on any other outcome. Candidates are ordered by each version's newest modification time across drives, and at most eight exact reads are spent. Test PUTs now wait for every drive so tests that edit one drive's metadata no longer race the PUT's background tail writes. * chore(ci): shrink the error(format!) ratchet baseline for rebalance The rebalance control and meta modules now have fewer Error::other(format!) call sites than the baseline records, which fails the shrink-only ratchet in Quick Checks. Record the smaller counts. * fix(ecstore): preserve canonical order in latest-version fallback |
||
|
Chris An
|
b5c253f3ed | docs(listing): describe degraded strict quorum limits (#8981) | ||
|
Chris An
|
f96c94f950 |
fix(listing): treat key markers as literal keys (#8978)
* fix(listing): treat version markers as literal keys * fix(listing): keep start-after and V1 markers literal * docs(listing): state cursor compatibility removal condition * fix(listing): resume protected table pages from raw keys * test(listing): keep protected pagination coverage in app layer |
||
|
Hauser
|
b9f01162b2 |
feat(tools): inspect BTF scalar member layouts (#9003)
Report selected integer offsets and widths with strict member and payload validation. Cross-check compiler fixtures with bpftool and independent C offsetof/sizeof results. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Chris An
|
5ab1d5191e | fix(heal): pace and parallelize replacement recovery (#9005) | ||
|
Chris An
|
acebee4ea3 |
fix(ecstore): select the latest version at read quorum like MinIO (#8982)
* fix(ecstore): select the latest version at read quorum like MinIO GET and HEAD escalated latest-version selection to write quorum when enough metadata answered and skipped a newer version below write quorum on degraded reads. LIST and heal already pick the newer side of a split write at read quorum, so such keys listed but answered 503 until heal. Select with MinIO's getObjectFileInfo rule: the most common mod time that reaches the object's data blocks wins, ties go to the newer time, and one content identity at that time must reach the same quorum. Two identities tied at the selected time fail instead of resolving by hash order, and the selected identity's version-stack shape follows disk order. * fix(ecstore): resolve selection ties by disk order and guard writes Review of the read-quorum selection found three gaps: - Rejecting a same-time half split made explicit-version reads of a partially applied tagging update fail and blocked the retry that repairs it. Resolve ties to the identity on the lowest-index disk, for reads and heal alike, instead of rejecting or using map order. - Delete markers and empty versions can lower the common read quorum below the selected version's data blocks, so HEAD could report a version GET cannot decode. Require that many copies, as MinIO's getLatestFileInfo does. - A selection can now return exactly read-quorum disks, and partial metadata writes are not rolled back. Refuse an update before writing when its target disks cannot reach write quorum. * fix(ecstore): count identity copies in the data-blocks guard The first-candidate shortcut counts byte-identical copies, so copies that differ only in identity-neutral fields such as replication status were missing from the data-blocks check. When the shortcut's count falls short, count every copy of the selected content identity before refusing the version. * fix(ecstore): keep commit-proof selection for system-bucket reads Pool metadata's PREPARE/COMMIT recovery and identity epochs treat an ordinary read of pool.bin as proof that the record committed. Under the read-quorum rule, a COMMIT that reached only half of a 2+2 set was served as committed, and the pool_meta_native_physical_* integration tests failed. Route reads through select_fileinfo_for_read: S3 buckets keep MinIO's read-quorum rule, while is_meta_bucketname reads use the previous algorithm unchanged as select_committed_fileinfo, so a split stays unreadable until native recovery resolves it. |
||
|
Chris An
|
e452d837dd | docs(scanner): clarify incomplete census diagnostics (#9002) | ||
|
Chris An
|
1451f3189b |
fix(listing): leave delete recovery out of list requests (#8980)
* fix(listing): leave delete recovery out of list requests * test(listing): expect residue preservation across walks |
||
|
Hauser
|
be485e6cab |
test(scanner): add memory acceptance runner and budget probes (#8988)
* test(scanner): add memory acceptance runner and budget probes Add owned S3 workload phases with process, endpoint, cgroup and binary identity checks for scanner memory and scanner-disabled request bursts. Keep scale, restart, heap-attribution and overall-acceptance limits explicit. Characterize healthy and seeded coverage-gap cache retention through the real folder walker and production cache codec without changing runtime code. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(scanner): bind acceptance capacity to effective storage Validate the serving process's complete local disk inventory before load writes and during runtime sampling. Resolve each disk inside the owned storage root, reject nested mounts and filesystem mismatches, and bind storage observations to phase identity for container and native runs. Cover the unused-mount bypass, CLI-overridden environment, native paths, symlink escapes, named volumes and stable multi-disk inventory ordering. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(scanner): fence status observations before workload writes Close each status API window with a second storage-bound process sample before Phase entry can authorize bucket creation or PUT requests. Add a fail-before regression for a process change during the initial status request, asserting no ownership or workload writes occur. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser
|
fbd94e129d |
test(e2e): validate late multipart part transaction packets (#8989)
Retain original prepare and commit-settlement RPCs beyond failed replacement and Abort cleanup, with and without bucket recreation. Require typed refusal or a physically verified idempotent result, unchanged fresh-upload bytes and complete object reads. Extend scoped proxy observations without rewriting signed frames or trailers, and preserve registered test membership and existing budgets. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |