mirror of
https://github.com/rustfs/rustfs.git
synced 2026-11-08 19:43:05 +00:00
fbc7af7df05ecd16c4fce845fcf66cb096491c21
917 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris An
|
d6210ac17b |
fix(s3,ecstore,table-catalog): bound work driven by client input (#9172)
* fix(ecstore,heal,select): bound storage-layer work driven by client input Scan findings in the erasure store, heal, file metadata and S3 Select layers, grouped because they share the disk walk, the multipart staging area, delete recovery and the select snapshot lease: - Multipart: upload ids are canonicalized before they name a staging path or a lock key (backlog#3027); trailing-slash keys are encoded like every other handler (backlog#3055); unlisted parts are removed at completion (backlog#3106); per-bucket uploads are found through a signed per-disk index instead of a scan of the shared staging area (backlog#3121); encrypted part ETags carry a per-upload secret so a write-only user cannot confirm plaintext guesses (backlog#3044). - Delete and recovery: recovery walks are bounded per request (backlog#3038); a delete marker on a key whose xl.meta path is a directory, and sidecar names used as child keys, no longer wedge the journal (backlog#3035, backlog#3105); over-quota PUTs no longer list the bucket under the ledger lock (backlog#3128). - Listing: scan_dir sends metadata without inline data (backlog#3052), counts delete markers against max-keys (backlog#3108), and rejects a marker that carries the cursor tag (backlog#3059). - Heal: bucket metadata and heal records live in separate namespaces so DeleteBucket cannot erase a drive rebuild (backlog#3039), a deleted bucket is skipped instead of failing the rebuild (backlog#3040), and directory objects are encoded on readback (backlog#3061). - Restore and tiering: RestoreObject preserves tags and the version's modification time (backlog#3073, backlog#3074); Snowball transport headers are not forwarded to the tier (backlog#3090); an epoch modification time round-trips through xl.meta (backlog#3063). - Locks: PutObject takes its bucket locks after the body is staged (backlog#3131); SelectObjectContent releases its namespace locks once the snapshot is taken (backlog#3132). - RPC and peers: the v3 signature check consumes the v2 nonce (backlog#3129); a bucket-metadata reload timeout no longer marks the peer offline (backlog#3124); replication target requests carry an operation deadline (backlog#3102); the scanner resumes its interrupted wait after a lifecycle write (backlog#3118). - S3 Select: bounded schema inference, column cap and budgeted CSV batches, Parquet dictionary bounds and pre-planner projection checks (backlog#3075, backlog#3076, backlog#3077, backlog#3078). * test(e2e): seed legacy duplicate bucket tags offline for the OPA contract * fix(s3): authorize what handlers store and bound request-driven work Scan findings in the S3 request layer (access hooks, object use cases, HTTP server layers), grouped because they share the authorization condition builder, the PutObject admission path and the response layers. Builds on the storage-layer batch for the bounded recovery and multipart APIs it calls. Authorization and policy conditions: - copy source is percent-decoded and version-canonical before the s3:x-amz-copy-source condition (backlog#3135); CopyObject authorizes the tag set it copies (backlog#3034); HeadObject with the replication check header still authorizes s3:GetObject (backlog#3047); the x-rustfs-source-version-id header feeds s3:versionid (backlog#3072); recursive force-delete with a REPLICA header authorizes every descendant as a delete (backlog#3070); include-deleted listings require s3:ListBucketVersions (backlog#3099); the metadata listing route keeps table isolation and uses the same prefix it authorizes (backlog#3053, backlog#3054); If-Match probing needs read permission (backlog#3097); Object Lambda checks s3:GetObjectVersion (backlog#3058); replication proxy fallback keeps ExistingObjectTag denies (backlog#3089); HeadObject returns lock and tag headers only with permission (backlog#3113); IPv4 clients on the dual-stack listener match IPv4 SourceIp conditions (backlog#3049); bucket policies are parsed once and cached (backlog#3032); keystone: access keys no longer verify with an empty secret (backlog#3043). Admission and resource bounds: - GetObject releases disk-read permits when the client stops reading (backlog#3023); PutObject takes buffer and foreground admission only after body bytes arrive and bounds the wait (backlog#3031, backlog#3084); xl.meta size is bounded on the write path (backlog#3100); XML compat layers stream instead of buffering whole bodies (backlog#3094); tier GETs stream the remote body (backlog#3046); on-demand migration pulls release permits on unread GETs (backlog#3110); SSE-KMS key policy covers UploadPart and completion (backlog#3130); MinIO admin decryption runs Argon2id off the async worker (backlog#3024). Input validation and leaks: - replication headers are validated (source ETag shape, original size) (backlog#3092, backlog#3093); control characters in keys and lifecycle rule IDs are rejected (backlog#3101, backlog#3117); Snowball member mtime needs s3:ReplicateObject (backlog#3083); DeleteObjects checks Content-MD5 and admin/STS reject SigV2 replays with altered query or body (backlog#3037, backlog#3026); request logs redact presigned queries and handler spans skip credentials (backlog#3068, backlog#3107); SSE-C listings and completions withhold plaintext metadata and checksums (backlog#3120, backlog#3050); CORS resolves the bucket like s3s does (backlog#3104); PutBucketTagging rejects duplicate keys (backlog#3126); notifications fire after a client disconnect (backlog#3127); ListenNotification streams re-check access (backlog#3119). * fix(api): drop imports left unused by the base merge * test(s3): move the KMS multipart denial test to the app layer The test drives app-layer multipart use cases, so it belongs with them; living in the SSE module it imported app code from the infra layer and failed the layer dependency guard. The SSE test helpers it borrows are exposed through the storage and app facades. * fix(s3): keep the SSE-C key fingerprint on keyless multipart completion The backlog#3050 fix withheld both the plaintext checksums and the customer-key MD5 header from a CompleteMultipartUpload that did not present the key. The checksums are a digest of the plaintext and stay hidden; the key MD5 identifies the key only, and the S3 conformance suite (test_copy_part_enc) expects it on a keyless completion, so it is echoed again. * fix(tier): stream GET bodies and document replication budgets * fix(ecstore,s3): keep the multipart index and cold fill off disk lifetimes Two lifecycle regressions surfaced once the storage and request batches ran together against main: - The multipart index owner cloned the disk's pinned publication root and bootstrapped on a task of its own. That kept the mount alive past the last LocalDisk owner (the disk lifecycle allows exactly three holders of the root FD) and left a stray task behind, which the local-disk and read-repair tests detect. The owner now keeps only the root path and reopens it for identity checks, and the bootstrap scan rides the disk's existing cleanup task, where stopping the disk abandons it and a reconnect resumes it. - The disk-read admission wrapper defers releasing its permit until an in-flight read completes, so a cancelled client cannot widen disk concurrency. The cold-fill producer, however, drops a read it has already classified as stalled when its deadline fires, and that permit stayed pinned for the lifetime of the stuck read. The producer now abandons the admission explicitly on timeout. * test(ecstore): pass the index owner to the Linux cleanup-loop test * fix(s3): release cold-fill admission after watchdog handoff * fix(s3,ecstore): close the round-1 adversarial review findings Adversarial review of the combined request- and storage-layer batch found fixes that fell short of their issues, two behaviour regressions for legitimate clients and several unrequested costs: - backlog#3038: a bulk DeleteObjects reserved the whole per-disk inspection allowance up front for every key in flight and exhausted the 16 Mi request budget on any set wider than eight drives, failing healthy keys with SlowDown. Calls are now admitted while budget remains and charged for the entries they actually visited. Object-scoped discovery charges plain child keys at one entry per 64 names instead of one each, so deleting a missing key that names a folder prefix completes instead of answering 503 forever while a hostile prefix stays bounded. A peer still on the previous binary is reported as an unsupported participant, the repository's rolling-upgrade contract, instead of as a budget verdict. - backlog#3037/#3026: a SigV2 configuration PUT without Content-MD5 was still replayable with another body; it is refused with MissingContentMD5 while SigV4 keeps the optional-MD5 contract. - backlog#3072: DeleteObject now resolves x-rustfs-source-version-id before authorization like PUT and COPY, so a replica delete cannot remove a version the policy never saw. - backlog#3121: the multipart index no longer fsyncs a marker and four directories per multipart metadata write (it is rebuilt on every start), and ListMultipartUploads discovers roots concurrently. - backlog#3077: Parquet SelectObjectContent decodes budget-sized batches instead of one row per batch. - backlog#3031: a full upload buffer pool hands out an unpooled buffer instead of refusing the upload; the progress floor already bounds stalls. - backlog#3023: the GET disk-read idle cut-off is configurable (RUSTFS_GET_DISK_READ_IDLE_TIMEOUT_SECS, default 120 s) so a paused client is not truncated after 30 s. - backlog#3075: JSON Lines schema inference keeps the sample collected at the 4 MiB bound instead of failing the query. - backlog#3024: compat admin decryption queues briefly for a slot instead of refusing the third concurrent request. - backlog#3113: HEAD skips its permission probes for the bucket owner. - backlog#3117: control characters in a lifecycle rule ID are rejected on PutBucketLifecycle as well. The lock-crate publication FIFO serves backlog#3111 and moves to the table-catalog PR. Also fixed: a lossy admin error message, an unused s3-client function, two unused test-util features, a duplicated key validation, the s3s footprint baseline, and a stale log event name in the reverse-proxy runbook. The ODM multipart write-back test runs on the large test stack like its sibling. * fix(table-catalog): bound maintenance work and isolate table namespaces (#9174) * fix(table-catalog): bound maintenance work and isolate table namespaces Scan findings in the Iceberg table catalog: - Metadata maintenance deleted every unreferenced *.metadata.json in a table's directory without checking whether a renamed table in another namespace still owned those files (backlog#3060); it now resolves the owning table through the directory index before deleting. - Manifest lists and manifests were decoded with the embedded Avro writer schema and no recursion depth limit; a self-referencing record schema overflowed the stack (backlog#3080). Schemas are validated for depth before decoding. - Compaction decoded every input Parquet file into Arrow batches held in memory; a decompression bomb OOM-killed the server (backlog#3085). Row groups are streamed through a byte budget and oversized inputs are refused. - An S3 request for a key outside an indexed table prefix listed the bucket's whole namespaces/ catalog prefix and failed once it held more than 4096 objects (backlog#3086); a per-bucket directory index answers index misses without the listing. - The maintenance planner read whole orphan files to obtain their modification time (backlog#3087); it reads metadata only. - A client disconnect between persisting active_rename_id and clearing it left the bucket's catalog entry locked at 503 (backlog#3088); the rename runs detached from the request future and clears the flag on every exit. - Table commits held the bucket publication fence until publication, starving every object write (backlog#3111); commits now hold it under a deadline and object writes are admitted between commits. The directory index is managed through three new admin routes under {warehouse}/catalog/directory, registered in the admin route policy. The durable-strong store is left as is (its snapshot partitioning for backlog#3109 and the strong-path deadline are not ported): that backing is being retired separately, so the new tests cover the object backing. * fix(table-catalog): close the review gaps in the scan-fix batch Adversarial review of the table-catalog batch found the fixes narrower than the issues they close and two cost regressions: - backlog#3085: the compaction footer check bounded nothing a page header could not override; parquet sizes its page buffers from the header. Walk every column chunk's page headers (Thrift compact, no decompression) and hold each page and the chunk total to the budget before decoding. The stored-input budget is per rewrite group, so a multi-group job no longer fails after writing earlier outputs. - backlog#3086: the default index-miss scan still counted maintenance reports toward its 4096-object limit; only table entries count now, and the listing itself is bounded to 64 pages. The directory index remains an optional accelerator. - backlog#3111: a create wrote its metadata file under the fence as the first durable mutation, which disarmed the deadline before the unbounded registration inventory. The write precedes publication now. The deadline gets its own knob (RUSTFS_TABLE_CATALOG_PUBLICATION_PREPARATION_TIMEOUT_SECS, default 30 s, never below the lock-acquire timeout) instead of reusing the 5 s lock timeout, and the ownership inventory reads entries concurrently. - Directory candidates are written only while a directory exists; the backfill covers earlier entries. A directory on the migration-fence path is reported as a conflict instead of a parse error. - Dead test hooks, a stale comment and the support-matrix text that did not match the code are removed or corrected. * fix(select): preserve complete samples and Parquet batch limits * fix(s3,ecstore): close the round-2 adversarial review findings DeleteObject resolved x-rustfs-source-version-id for authorization on every request while storage consumes it only for a REPLICA request, so a non-replica caller with s3:DeleteObjectVersion could trigger a plain delete of the current version. The hook now resolves the header under the same predicate as storage, and the regression test covers the non-replica case. HEAD no longer skips its permission probes for the bucket owner; an explicit bucket-policy Deny on the metadata actions binds the owner too. Multipart index pages verify each root against the staging area and prune markers whose staging root is gone, under the marker's own lease with a re-check, so the per-drive index stays proportional to live uploads instead of every key ever multipart-initiated on the drive. SigV2 configuration PUTs that declare an empty body need no Content-MD5: there is no body to replay. The minimum-progress floor stays independent of RUSTFS_HTTP_REQUEST_BODY_READ_TIMEOUT; the knob's documentation now says so instead of claiming 0 disables it. The quota bootstrap window (503 until the background scan publishes the ledger) is kept, because ledger admission must not come from scanner reporting; the compatibility matrix now documents it. The S3 compatibility matrix gains a section for the client-visible validation bounds, and the drive-timeout tuning guide tabulates RUSTFS_GET_DISK_READ_IDLE_TIMEOUT_SECS. * fix(lock): queue table-bucket publication writers fairly Table commits take the bucket publication fence as a write lock; under a stream of commits the lock's try-and-retry acquisition let a new commit jump ahead of object writers that had been waiting longer, which is part of what starved them (backlog#3111). Requests for the publication lock path now take a FIFO ticket per shard: a waiter is admitted only when it is at the head of the queue, cancelled waiters release their turn, and batch acquisitions join the same queue. Other lock paths are unchanged. This moved here from the request-layer batch, where no cited issue covered it; it serves the table-catalog issue this PR closes. * fix(table-catalog): close the round-2 adversarial review findings Parquet page-header walker: read the size fields by field id as parquet does, accepting I16/I32/I64 wire kinds and rejecting any other kind, so a header encoding uncompressed_page_size as I64 can no longer bypass the budget while parquet allocates it. Bound every list, set and map count by the bytes remaining in the chunk, and reject bool collection elements (parquet skips them with zero width, so either width would desynchronize the two parsers); this removes the 2^63-iteration spin on LIST<BOOL>. Start each column-chunk walk at the dictionary page offset whenever the footer declares one, exactly as parquet's byte_range does, instead of min(dictionary, data). Footer pre-walk: before parquet decodes a footer, walk FileMetaData with the hardened cursor and charge the declared row_groups count times the RowGroupMetaData size to the decode budget, because parquet 59 reserves that Vec before reading one element. Preparation deadline: clamp the RUSTFS_TABLE_CATALOG_PUBLICATION_PREPARATION_TIMEOUT_SECS knob to 86,400 seconds so Instant + Duration cannot overflow. Index-miss scan: after each listing page, jump the cursor past the maintenance subtree the page ended in, and allow one extra page per table entry already listed, so job-report history scales with the catalog while a report flood under one table stays bounded. Migration fence: read the fence-path object once and decide from its stored kind whether it is directory state or a fence. Expired or failed create: remove the initial metadata file when the registration fails before any catalog write may have begun; nothing else references it. Correct the comment and support matrix that claimed maintenance reclaims it. Tests cover each walker divergence, the footer row-group reservation, the deadline ceiling, the paging budget for honest catalogs and for a report flood, and the removal of the unpublished metadata file. * test(admin): register the catalog directory routes in the route matrix The table-catalog batch added GET, POST and DELETE {warehouse}/catalog/directory on both the Iceberg and the compat admin prefixes without listing them in the admin route matrix, so test_admin_route_matrix_matches_registered_routes failed on CI. * fix(s3,ecstore): close the round-3 adversarial review findings Multipart index pages now treat the cursor as the last examined marker rather than the last root, so a page that pruned a stale marker still validates and ListMultipartUploads keeps using the index. A marker is written before its staging root is published, so pruning spares markers younger than a five-minute grace period and an existing marker is touched on every new upload; the prune re-checks both under the marker lease. Bounded delete-recovery inspection rides the durable-mutation lane like the unbounded path instead of the one-worker foreground lane. The on-demand-migration inline body budget scales with the object size (at least 64 KiB/s) instead of a flat 120 s. The control-body digest service honors RUSTFS_HTTP_REQUEST_BODY_READ_TIMEOUT=0 as "no inactivity timeout" like every other reader of that knob. Quota reconciliation throttle objects are deleted together with a retired generation's ledger, so bucket turnover leaves no residue. The PUT guard test shim is gone; tests call the shared guard. Bounded DiskAPI defaults answer MethodNotAllowed like their siblings. Stale budget docs, an unreachable CSV arm, an always-Some limit parameter, commented-out constants and a misplaced doc comment are cleaned up; weak test assertions name the expected error; lifecycle rule-ID control characters gain a test. The compatibility matrix records the CopyObject tag-copy permission, the exact SigV2 empty-body predicate, replica and migration writes of control-character keys, and the SigV2 DeleteObjects digest rule. * fix(table-catalog,heal): close the round-3 review findings A. Pack compaction rewrite groups to min(max-rewrite-bytes-per-job, TABLE_COMPACTION_MAX_STORED_BYTES), reject an over-ceiling group before any output is written, and name the ceiling in the commit-path errors; document the effective cap in the support matrix and scheduler runbook. B. Keep TaskCancelled/TaskTimeout classification on replacement passes: the namespace verification runs only for completed or storage-failed passes, in the erasure healer and the task bucket prepass. C. Compare the replacement bucket pass per bucket: a recorded identity that is gone or replaced still errors or restarts the pass, buckets created mid-pass are appended to the identities, pending_buckets and a persisted positional order without rewinding progress or the checkpoint; verification ignores additions. D. Read the migration-fence object once in the object-backed write permit and dispatch on its kind. E. Drop the strong store's copy of maintenance_cleanup_roots. F. Remove the unused test hooks fail_put_prefix, corrupt_put_prefix and pause_after_metadata; make compact_parquet_data_files and PublicationTicket.state private. G. Document the body-revision identity, the legacy revalidation budget and its HTTP 503 in the support matrix. H. Define HEAL_REPLACEMENT_RECOVERY_DIR once in ecstore and consume it from heal through the storage_api facade. * fix(ecstore): cap the index pages one disk may answer Pruned multipart index pages may carry few roots, so the page cursor no longer bounds how many pages a peer can make a listing fetch. Cut a disk off after 4096 pages (about one million examined markers) so it contributes no coverage and the listing falls back to the staging scan. * fix(ecstore,table-catalog): close the round-4 adversarial review findings Multipart index markers dated in the future (a backwards clock step between recording a marker and publishing its staging root) were judged old and could be pruned while their upload was in flight; a future-dated marker is now kept. The marker touch on EEXIST is best effort, so a filesystem that refuses to set times no longer fails the upload commit. Pruning runs on the durable-mutation lane instead of the one-worker foreground lane. Quota reconciliation keeps one throttle object per bucket incarnation (a quota revision change replaces it instead of orphaning the previous revision's object), the retirement cleanup deletes it ahead of the ledger and also when the ledger is already gone, and a scan the retirement already refused writes nothing back. Bounded DiskAPI defaults answer UnsupportedDisk, the verdict the aggregate gives a peer without the protocol. The compaction ceiling pre-check runs over every rewrite group before any output is written, and the worker's ceiling error no longer names a placeholder group. Docs: the inline on-demand-migration client budget, the protocol-2 downgrade conflict of replacement passes, the exact footprint numbers. |
||
|
Hauser Zhi
|
babfeaed00 |
fix(io-core): preserve pooled capacity and accounting (#9171)
* fix(io-core): preserve pooled capacity and accounting Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(io-core): preserve pool allocation admission order Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * docs(io-core): clarify pool return locking Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser Zhi
|
fe3ada0689 |
chore(deps): refresh Cargo dependencies (#9156)
* chore(deps): refresh Cargo dependencies Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * chore(deps): update rustfs kafka to 1.4.1 Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Chris An
|
7f5fa4b8f6 |
fix(protocols,kms,keystone): enforce client CA, protect the KMS probe (#9168)
Scan findings in the alternate-protocol front ends and the KMS and Keystone integrations: - WebDAV and FTPS read RUSTFS_WEBDAV_CA_FILE / RUSTFS_FTPS_CA_FILE, checked that the file existed, and then built their TLS listeners with with_no_client_auth(), so mTLS was never enforced. A shared protocols::common::tls::server_config builds the rustls config with a client-certificate verifier from the configured CA, startup fails on an invalid CA, and a CA cannot be combined with cleartext (backlog#3069). - The KMS readiness probe used a fixed key id that the key-lifecycle admin routes accepted like any customer key, so kms:DisableKey on it failed readiness on every node. The probe key is reserved from key management and a disabled probe key no longer counts as a backend failure (backlog#3115). - The Keystone middleware answered anonymous requests carrying a junk X-Auth-Token with the configured Keystone URL in the body while Keystone was unreachable; failures no longer disclose upstream locations, and the login cache is bypassed when a subscription token refresh is required (backlog#3116). |
||
|
Chris An
|
a97c65895c |
fix(proxies,targets,obs): bound untrusted input in leaf crates (#9166)
Scan findings in crates whose behavior is driven by untrusted input and that stand alone from the storage core: - trusted-proxies: the simple implementation takes the client address from the right-most untrusted X-Forwarded-For hop instead of the left-most (backlog#3082); the legacy validator never falls back to the proxy identity when the chain is rejected and reads X-Forwarded-For unless RFC 7239 is explicitly selected (backlog#3051). - targets: webhook and similar outbound clients drain responses up to a byte limit instead of buffering arbitrary bodies (backlog#3062); the on-disk notification queue has a shared byte budget in addition to the entry count, configurable via RUSTFS_TARGET_STORE_MAX_BYTES (backlog#3122); AMQP and Redis manifests declare the broker url as a secret and bind credentials to their connection fields, and the plugin-instance PUT refuses to restore a redacted secret when the connection fields changed (backlog#3065, backlog#3066). - obs: the debug-level s3s response dump is filtered so STS and service-account credentials never reach the log (backlog#3029). |
||
|
Hauser Zhi
|
00a81b4048 |
chore(deps): refresh crate requirements and lockfile (#9140)
Run cargo update and cargo upgrade twice to bump compatible requirements (jiff, zip, dial9) and refresh transitive lockfile entries. |
||
|
Boyi (Elliot) Li
|
cc08e0d11b |
feat(fips): route internode RPC HMAC through aws-lc-rs FIPS (#9137)
* feat(fips): route internode RPC HMAC through aws-lc-rs FIPS * Revise KMS cryptographic compliance details Updated the KMS cryptographic compliance documentation to reflect changes in HMAC authentication and added details about the internode RPC HMAC authentication. Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Hauser Zhi <housemecn@gmail.com> --------- Signed-off-by: Hauser Zhi <housemecn@gmail.com> Co-authored-by: Hauser Zhi <housemecn@gmail.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
dependabot[bot]
|
12c58d25b5 |
chore(deps): bump the dependencies group across 1 directory with 5 updates (#9113)
* chore(deps): bump the dependencies group across 1 directory with 5 updates Bumps the dependencies group with 5 updates in the / directory: | Package | From | To | | --- | --- | --- | | [pulsar](https://github.com/streamnative/pulsar-rs) | `6.9.0` | `6.9.1` | | [arrow-array](https://github.com/apache/arrow-rs) | `59.3.2` | `60.1.0` | | [arrow-ipc](https://github.com/apache/arrow-rs) | `59.3.2` | `60.1.0` | | [arrow-schema](https://github.com/apache/arrow-rs) | `59.3.2` | `60.1.0` | | [parquet](https://github.com/apache/arrow-rs) | `59.3.2` | `60.1.0` | Updates `pulsar` from 6.9.0 to 6.9.1 - [Release notes](https://github.com/streamnative/pulsar-rs/releases) - [Commits](https://github.com/streamnative/pulsar-rs/compare/v6.9.0...v6.9.1) Updates `arrow-array` from 59.3.2 to 60.1.0 - [Release notes](https://github.com/apache/arrow-rs/releases) - [Changelog](https://github.com/apache/arrow-rs/blob/main/CHANGELOG.md) - [Commits](https://github.com/apache/arrow-rs/compare/59.3.2...60.1.0) Updates `arrow-ipc` from 59.3.2 to 60.1.0 - [Release notes](https://github.com/apache/arrow-rs/releases) - [Changelog](https://github.com/apache/arrow-rs/blob/main/CHANGELOG.md) - [Commits](https://github.com/apache/arrow-rs/compare/59.3.2...60.1.0) Updates `arrow-schema` from 59.3.2 to 60.1.0 - [Release notes](https://github.com/apache/arrow-rs/releases) - [Changelog](https://github.com/apache/arrow-rs/blob/main/CHANGELOG.md) - [Commits](https://github.com/apache/arrow-rs/compare/59.3.2...60.1.0) Updates `parquet` from 59.3.2 to 60.1.0 - [Release notes](https://github.com/apache/arrow-rs/releases) - [Changelog](https://github.com/apache/arrow-rs/blob/main/CHANGELOG.md) - [Commits](https://github.com/apache/arrow-rs/compare/59.3.2...60.1.0) --- updated-dependencies: - dependency-name: arrow-array dependency-version: 60.1.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: dependencies - dependency-name: arrow-ipc dependency-version: 60.1.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: dependencies - dependency-name: arrow-schema dependency-version: 60.1.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: dependencies - dependency-name: parquet dependency-version: 60.1.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: dependencies - dependency-name: pulsar dependency-version: 6.9.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: dependencies ... Signed-off-by: dependabot[bot] <support@github.com> * chore(deps): update dependencies and pin pulsar 6.9.0 Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * chore(deps): ignore pulsar 6.9 updates Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Hauser <housemecn@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser Zhi
|
ed4099ebe1 |
chore(deps): update crates and fix storage CI regressions (#9111)
* chore(deps): refresh crates and adopt openidconnect 4.1 Use published OIDC/OAuth2 releases and reject malformed HMAC signature lengths before upstream verification. Refresh workspace and fuzz locks, and advance the local zstd compatibility patch to 0.13.4 while retaining the fixed 0.14.3 codec backend. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * chore(deps): refresh compatible lockfile resolutions Resolve Prost build and derive dependencies with itertools 0.14 and prefer windows-sys 0.61.2 within the existing supported version ranges. Retain all package versions, sources and checksums. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(connect): separate cleanup checks from short probe windows Use the functional measurement budget for the cleanup HTTP fixture so setup and measured I/O can finish before checking delete status handling. Exercise both normal scheduling and an in-flight PUT clock advance, and include target classification in unexpected-failure diagnostics. Retain the cleanup deadline, outer timeout and delete-stage assertions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): isolate stalled multipart disk timeouts Apply the short injected timeout only to the deliberately stalled disk. Healthy disks retain their normal operation budgets, while logical lock, physical owner, immediate CopyObject and final shard assertions remain. Keep overrides scoped across cloned and reconnected handles, preserve shorter caller deadlines and explicit zero timeouts, and report drive health when the real CopyObject assertion fails. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(rebalance): retry metadata write lock quorum admission Recognize typed write-lock quorum failures for rebalance.bin within the existing bounded metadata admission retry policy. Each attempt must acquire quorum before loading or saving metadata; data writes are not replayed by this retry. Cover recovered and persistent quorum failures, cancellation after a failed attempt, and rejection of unrelated resource or read-lock errors. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Boyi (Elliot) Li
|
3a1fac16e3 |
feat(tls): add opt-in fips feature for aws-lc-rs and rustls (#9084)
Add an opt-in `fips` Cargo feature to `rustfs` and `rustfs-tls-runtime` that enables `rustls/fips` and `tokio-rustls/fips` (activating `aws-lc-rs/fips` / `aws-lc-fips-sys` and `rustls-webpki/aws-lc-rs-fips`). When compiled with `--features fips`, `startup_runtime_hooks` installs `rustls::crypto::default_fips_provider()` as the process-wide default crypto provider and asserts `provider.fips()` at startup. Also lock `aws-lc-fips-sys v0.14.4` in `Cargo.lock`, allow the `OpenSSL` license on `aws-lc-fips-sys` in `deny.toml`, and update `docs/operations/kms-cryptographic-compliance.md` to document the scope of `-p rustfs --features fips` vs. `rustfs-crypto`. Co-authored-by: hector <42570491+majinghe@users.noreply.github.com> |
||
|
Hauser Zhi
|
e5653ca055 |
chore(deps): refresh workspace dependencies (#9080)
Run cargo update --verbose and cargo upgrade --verbose twice from the current main baseline. Upgrade the Tonic family to 0.14.9, Redis to 1.8.1, sysinfo to 0.40.1, and hotpath to 0.30.2 with compatible transitive updates. Synchronize the standalone fuzz lockfile and retain its PKCS#1 rc.4 resolution for compatibility with pinned RSA rc.18. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> Co-authored-by: Chris An <anzhengchao@gmail.com> |
||
|
Chris An
|
0e5fe8c658 |
feat(storage): share physical admission across scanner and heal (#8911)
* feat(scanner): share node admission with background heal * fix(scanner): drain candidates after listing completion * fix(scanner): yield cache leases for shared node admission * test(scanner): attest adaptive admission in process receipts * fix(concurrency): declare Tokio time dependency * feat(concurrency): retain bounded physical resource subsets * fix(scanner): retain resource ownership in native filesystem jobs * feat(concurrency): acquire physical groups before node work * feat(scanner): gate background reads by physical device * feat(concurrency): combine preallocated resource selections * feat(scanner): share physical topology across disk plans * fix(storage): retain only native disk resource subsets * feat(storage): admit auxiliary physical resources without waiting * feat(heal): admit physical resources before object repair * fix(storage): retain resources during native format probes * fix(storage): retain resources through direct writer I/O * fix(storage): retain resources through buffered writer I/O * fix(storage): retain physical resources through streamed reads * fix(storage): retain resources through native filesystem jobs * fix(storage): retain physical ownership during bitrot reads * fix(ecstore): retain resources during positioned reads * fix(ecstore): retain staged metadata write ownership * fix(ecstore): retain native io_uring read ownership * fix(ecstore): enforce receiver-side background admission * fix(ecstore): retain resources through whole-file native I/O * fix(ecstore): retain physical ownership during durable rename sync * fix(storage): account for retained background stream buffers * fix(storage): retain memory charges for background read results * fix(storage): account for native io_uring read allocations * fix(storage): retain memory charges for direct-write buffers * fix(ecstore): account for heal and RPC reader buffers * test(scanner): verify samples and clean concurrent collectors * fix(storage): retain inline and HTTP writer memory reservations * perf(rio): reuse reserved HTTP writer storage * fix(rpc): reserve background scanner pipe storage * perf(rpc): reuse reserved response reader storage * feat(scanner): measure cache copy allocation requests * fix(rpc): budget namespace scanner responses * fix(scanner): retain the enabled supervisor task handle * fix(scanner): budget namespace frame read staging * fix(scanner): account namespace envelope decode storage * fix(heal): retain and join the MRF consumer on shutdown * test(heal): isolate process-global MRF consumer fixture * fix(heal): join manager workers during shutdown * test(heal): verify shutdown at native read boundaries * fix(heal): drain channel commands before stopping MRF * fix(heal): admit resume scans and retain replacement workers * build: pin profiling sources and trust fixed ownership dependencies * fix(heal): yield node admission while accepted native work drains * fix(startup): retain shutdown work and preserve deadline failures * fix(heal): admit inventory probes and retain replacement work * fix(shutdown): preserve maintenance and frontend join failures * fix(shutdown): retain HTTP child tasks through real retirement * fix(shutdown): join actual TLS reload work before retirement * test(multipart): preserve failures before fault barriers * fix: keep physical admission inside repository guards Route the multipart lock diagnostics filter through the e2e storage compatibility module, reuse the parent module's s3s body aliases in the namespace memory test, and share one Reed-Solomon verify error mapping across the reserved-buffer and default paths. * fix: reconcile physical admission with main's new call sites Main added a source-admission metadata write layer, a CAS write fence, fixture writers and scanner handoff tests against the old borrowed-slice and node-permit shapes. Pass owned Bytes through the new metadata layer, answer the scanner handoff with ScannerNodePermit, gate the test-only disk lookup re-export, and clear the lint findings that stable 1.99 reports for the shutdown and admission code without suppressions. * test(concurrency): name reversed attachment groups clearly * fix(ecstore): box disk operation futures below admission layers Every LocalDiskWrapper operation reaches the LocalDisk future through the health, admission and timeout layers (track_cleanup_mutation, the track_disk_health family, the io-permit and native-resource admission split, and track_admitted_disk_operation). Each layer embedded that future, and unoptimized builds copy it into every awaiting frame, so one local delete cost about 1.3 MiB of stack and 16 tests overflowed libtest's default 2 MiB stack. Box the operation future once where track_admitted_disk_operation creates it, through a non-async helper so the temporary never stays in a polled frame. The layers above now hold only a pointer; the cost is one allocation per local disk operation. * fix(heal): box object heal futures below admission The admission wrappers took the object heal future by value, so unoptimized builds copied it into three awaiting frames, about 650 KiB per heal. Create it through a non-async helper instead, leaving only a pointer in those frames. * test(ecstore): count the I/O backend's root FD owner |
||
|
Chris An
|
7d45685d70 |
feat: classify scanner metadata calls by diagnostic prefix (#9052)
* feat: classify scanner metadata calls by diagnostic prefix * fix: export zero scanner visit counts for diagnostic scope |
||
|
Hauser
|
84e24e7673 |
chore(deps): refresh crates and adopt released aws-sigv4 (#9047)
* chore(deps): refresh crates and adopt released aws-sigv4 Run two update/upgrade rounds for the main and standalone fuzz workspaces. Replace the temporary SigV4a Git override with aws-sigv4 1.6.4 from the registry and remove its obsolete source permission. Pin OpenID Connect to e20ddbb01ca68356f2979bc125f5613ab45b4e7e. Upgrade async-nats to 0.51.0, hotpath to 0.30.0, libunftp to 0.24.0, and the TOML test dependency to 1.1.8. Retain Arrow/Parquet 59.3.2 for DataFusion 55.2.0 and keep the fuzz PKCS#1 version compatible with pinned RSA rc.18. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(deps): retain safe openidconnect pin against HMAC panic The e20ddbb pin verifies HMAC signatures with mac.verify(signature.into()), which panics in generic-array on malformed length instead of returning InvalidIdentityToken. Retain the previous reviewed pin that uses verify_slice and add a malformed-signature regression covering HS256, HS384, and HS512. * chore(deps): upgrade sysinfo to 0.40.0 Update the main and standalone fuzz lockfiles. Adapt fallible system queries, CPU usage accessors, and OS-native network interface names. Preserve process sampling fallbacks and report unavailable diagnostic sources without panicking. Cover missing host and process collectors. Validation: 167 focused tests passed; production and all five fuzz targets passed cargo check. Formatting, logging guardrails, and the existing dependency policy passed. Linux and Windows runtime checks were not run. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ci): use the admin error facade for sysinfo failures Keep the sysinfo initialization error behind the existing admin S3 facade. The facade builds the same error code and message without expanding the direct S3 error-macro surface guarded by the migration ratchet. Validation: S3 footprint guard, formatting, whitespace, and production cargo check with sftp, hotpath, and connect_top_net passed. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * chore(deps): refresh hotpath and ctutils (#9055) Run two cargo update/upgrade rounds in both the main and fuzz workspaces on top of PR #9047. Upgrade the hotpath family to 0.30.1 and ctutils to 0.4.7. Retain fuzz pkcs1 0.8.0-rc.4 for the pinned RSA 0.10.0-rc.18; rc.5 fails to compile against that RSA version. Validation: 1096 library tests passed with 19 skipped; upstream ctutils and hotpath-drain property/concurrency suites passed. Production and all five fuzz targets passed cargo check. Formatting, S3 footprint, and the existing dependency policy passed. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser
|
d4d416ed47 |
fix(deps): upgrade russh and replace vulnerable zstd codecs (#8969)
* chore(deps): upgrade russh to 0.65.0 and refresh crates Run two cargo update and cargo upgrade rounds from origin/main. Adapt SFTP public-key callbacks to PublicKeyOffer and VerifiedPublicKey, preserving explicit rejection and password-only authentication. Cover the rejection policy with a pinned-host-key SSH handshake test. Refresh Arrow, Parquet, Brotli, ZIP, and compatible dependencies. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(deps): route legacy zstd consumers to fixed codecs Replace the old registry implementation with a private version-constraint adapter backed only by zstd 0.14.3 and zstd-safe 8.2.0. Preserve legacy features and Avro, Arrow IPC, Parquet, and Kafka formats. Cover abandoned contexts, legacy decode errors, seekable bounds, stored frames, consumer interoperability, and resolved implementation versions. Record the adapter removal condition in the compatibility register. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser
|
5e863badc2 |
chore(deps): update crypto-bigint to 0.7.10 (#8962)
Replace the yanked crypto-bigint 0.7.7 release and raise ctutils to its required 0.4.6 minimum. Preserve all other package resolutions, features and dependency policy. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser
|
b78e2ad6e7 |
test(e2e): avoid rescanning publication lock logs (#8959)
* test(e2e): read publication logs incrementally Retain the complete matching lock history while reading only appended bytes after an initial scan before remote DELETE is held. Keep the original field projection, file ordering, polling cadence, lock deadlines and same-attempt assertions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): remove redundant publication fixture clone Serialize the existing root by reference through json! without cloning the Value. Keep all compatibility rows and assertions unchanged. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser
|
3bcba78ccb |
chore(deps): refresh allocator, profiling, and archive crates (#8910)
Run two cargo update and cargo upgrade rounds from origin/main. Update mimalloc to 0.7.2, hotpath to 0.29.1, and ZIP to 9.1.0, with compatible crypto, compression, and smallvec updates. Retain the generic-array version required by crypto-common 0.1.7. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser
|
d2c1434a02 |
fix: isolate internode reads and migrate crypto dependencies (#8872)
* fix(rio): isolate internode streams with pooled HTTP/1.1 Default the internode HTTP data plane to reusable HTTP/1.1 connections so paused shard bodies cannot consume shared HTTP/2 receive credit. Retain explicit auto negotiation and the existing TLS, pooling, and recovery paths. Refs rustfs/rustfs#8843 and rustfs/backlog#2934. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(rpc): frame bounded shard reads for connection reuse Preserve the bounded read cap and stream-to-EOF sentinel while making connection completion independent of a chunked terminator. Cover exact reads without EOF polling, pooled TLS reuse, and partial-body truncation. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(deps): migrate signer and OIDC to patched crypto Pin the requested signer fix and upstream OIDC/OAuth2 crypto migrations to immutable commits. Preserve registry SDK type identity, remove the old bigint and elliptic-curve chains, and verify independent login state, nonce, and PKCE values. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Chris
|
b39fcb2e85 |
test(heal): cover rc.5 upgrades and refresh ctutils (#8847)
* test(e2e): use the resumed decommission completion budget * test(heal): exercise checkpoints written by rc.5 * test(heal): route upgrade fixtures through the storage facade * fix(deps): update ctutils to address constant-time advisory * test(lifecycle): gate recovery I/O while advancing test time |
||
|
Hauser
|
7d6172e666 |
chore(deps): refresh crates and upgrade hotpath to 0.29.0 (#8852)
Run two cargo update and cargo upgrade rounds from origin/main. Upgrade the hotpath crate family to 0.29.0 and refresh compatible AWS, Redis, profiling, crypto, UUID, and certificate dependencies. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser
|
5778384b68 |
chore(deps): refresh ICU and profiling dependencies (#8820)
Update Pyroscope to 2.1.4 and compatible ICU, yoke, and zerovec crates. Cover normalized Unicode hostnames and loopback rejection in egress tests. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser
|
bea4be68a2 |
chore(deps): upgrade workspace parking_lot to 0.13 (#8801)
* chore(deps): upgrade workspace parking_lot to 0.13 Move all six direct workspace consumers to parking_lot 0.13 while retaining the 0.12 dependency line required by upstream crates. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(connect): stabilize client failure classification Use an ephemeral loopback listener that explicitly rejects proxy CONNECT and direct TLS connection establishment instead of assuming fixed ports will promptly refuse connections. Give classification its own bounded transport and measurement budget while retaining the timeout tests. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser
|
5749752217 |
chore(deps): refresh cloud SDKs, runtime and profiling (#8787)
* chore(deps): refresh GCS, profiling and allocator dependencies Update published GCS, Pyroscope, mimalloc, and PPMd crate versions. Align the pinned erasure-fixture test recursion limit with ECStore to compile GCS-enabled test targets. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * chore(deps): update AWS SDKs, Tokio and memmap2 Refresh compatible AWS service and Smithy dependencies, Tokio timers, and memory mapping support with the regenerated lockfile. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser
|
1ac35917c2 |
chore(deps): refresh dependencies and patch certificate trust (#8761)
* chore(deps): refresh dependencies and patch certificate trust Update published dependencies and certificate verification/root data, and upgrade rustfs-mimalloc to 0.7.0. Retain the known webpki-roots 0.26.11 compatibility-wrapper advisory without adding an audit exception. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * remove hyper dependency * remove hyper dependency --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Chris
|
6f582b0548 |
fix(heal): decide object absence from storage evidence, not source (#8697)
* test(heal): add failing regressions for a source-independent absent verdict Add the regression tests for backlog#2900 that fail on the current code, where the same object-level not-found is treated as deleted on the plain heal path but as recreate_failed on the recreate path, and where the erasure-set healer, the task predicates and the error helpers disagree on wrapped not-found and transient inputs. Also add the per-object disposition cache module with its unit tests, the manager admission tests for it, the compat register entry for the display boundary, and the admission-invariant notes. The module and the manager tests are not declared yet, so nothing is wired into the build. * fix(heal): decide object absence from storage evidence, not request source Replace the five heal error classifiers (erasure-set healer outcome mapping, task-level not-found and lock predicates, error.rs recovery helpers and the storage existence-check matcher) with one typed classify_heal_error. An object-level not-found from the locked storage read is Absent for every source and every recreate_missing value; the recreate path, the usage-observation and scanner object-dir whitelists are gone, and display-text matching survives only in the marked classify_error_display boundary. Add a bounded, expiring per-object disposition memory consulted before queue admission so a best-effort producer cannot requeue an absent or terminally failed object on every tick; durable MRF, admin, forced, high-priority and dry-run requests bypass it. The read-repair producer no longer forces recreate_missing. * fix(heal): keep rendered grace windows out of task retries A grace window rendered into a task message belongs to an attempt that already finished; its owner schedules the next one. Only typed grace errors replay the task, as before the classifier merge. Update the ecstore read-repair black-box test: the producer no longer forces recreate_missing. * fix(heal): classify ambiguous legacy delete residue as permanent explicitly * fix(heal): scope disposition memory and bound absence parsing |
||
|
cxymds
|
508caab281 |
fix(lifecycle): skip empty delete replication tasks (#8628)
* fix(lifecycle): skip empty delete replication tasks * test(lifecycle): route replication hooks through the sink * test(e2e): make upgrade write readiness checks deterministic * fix(deps): replace yanked inout 0.2.3 with 0.2.4 |
||
|
Hauser
|
9d85d21ee6 |
chore(deps): refresh workspace dependencies (#8597)
* chore(deps): refresh workspace dependencies Run cargo update and cargo upgrade twice from origin/main. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(deps): pin Pulsar 6.9.0 until recovery is bounded Keep Pulsar on 6.9.0 after reproducing background recovery and lookup activity after delivery timeout and target destruction in 6.9.1. Use an exact requirement to prevent a later cargo update from selecting the affected release. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser
|
7462c25319 |
feat(diagnostics): port read attribution to main (#8579)
* feat(diagnostics): attribute storage reads to request origins Port the diagnostics instrumentation from its experiment base onto main without carrying the base branch's unrelated changes. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(diagnostics): consolidate read context scopes Keep caller and entry provenance in one task-local context, combine the object-read scope, and classify source paths without allocating. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser
|
84f7ab1f5c |
chore(deps): refresh Rustls and TAR parser test crates (#8547)
* chore(deps): update Rustls patch release Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * chore(deps): update TAR parser test crates Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(heal): avoid collecting requests in test Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: zhi22915 <qiuzgang@gmail.com> Co-authored-by: hector <42570491+majinghe@users.noreply.github.com> |
||
|
Chris
|
9408b90f13 |
chore(release): prepare 1.0.2 (#8549)
Co-authored-by: Hauser <housemecn@gmail.com> |
||
|
Hauser
|
20fc897036 |
chore(deps): refresh crates and use Pyroscope 2.1.2 (#8540)
* chore(deps): refresh crates and use Pyroscope 2.1.2 Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * chore(deps): refresh lockfile Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser
|
4502a1f9b0 |
fix(ci): repair profiling and compact transition test gates (#8526)
* fix(deps): patch profiling dependency for RUSTSEC-2026-0344 Vendor the published Pyroscope package with an object 0.40 dependency and lock ruzstd 0.9.1. Preserve runtime sources and profiling features, add bounded decoder regressions, and run the patched package in audit CI. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> Signed-off-by: Hauser <housemecn@gmail.com> * fix(deps): correct vendored pyroscope documentation typos Correct copied comments and changelog spelling for the repository typo check, and describe these comment-only changes in the vendor provenance. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> Signed-off-by: Hauser <housemecn@gmail.com> * fix(ecstore): refresh compact transition proof per test case Install the short-lived homogeneous fleet proof immediately before each kill-point transaction. Earlier cancellation and recovery work must not age the next case's writer capability fixture. Report an early transition result instead of waiting for an unreachable kill point. Preserve cancellation, remote-owner and payload assertions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> Signed-off-by: Hauser <housemecn@gmail.com> --------- Signed-off-by: Hauser <housemecn@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser
|
34803e8165 |
perf(ecstore): isolate cleanup and preserve metadata authority (#8289)
* chore: pin s3s to upstream git revision Use the upstream s3s git repository at 50c94aeb1e5ea9ef8e7393ef6bbc87b72f6891c6 for both s3s and s3s-sigv4. Allow the official s3s git source in cargo-deny so source checks continue to pass. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(ecstore): instrument put object hot path Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(ecstore): split cleanup worker wait metrics Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * upgrade version * chore(metrics): add PUT post-store stage probes Add lightweight PUT post-store stage probes for issue 2615 validation. * fix cargo deny check warn * upgrade version * perf(ecstore): bound and isolate cleanup execution (#8151) * perf(ecstore): bound and isolate cleanup execution * fix(ecstore): preserve cleanup cancellation and lease release * feat(ecstore): default isolated cleanup to CPUs 1-2 * test(ecstore): decouple budget tests from CPU affinity * upgrade version * perf(ecstore): reuse pinned directory handles for object I/O * feat(observability): add sampled local GetObject stage summaries Add an opt-in GetObject stage aggregation path that works without the OTLP metrics export. When RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_ENABLED is set, io-metrics keeps per-(path, stage, object class, size bucket) sampled counts and duration totals in a bounded in-process map, and a startup task flushes them periodically to /var/log/rustfs/get-stage-summary.log, tunable via the sample rate and interval env keys. * build(deps): bump workspace dependency releases Patch releases: async-compression 0.4.49, tokio-rustls 0.26.6, xxhash-rust 0.8.19, suppaftp 12.1.1, russh-sftp 3.0.1. Minor release: hotpath 0.27.0. * fix(ecstore): fence distributed metadata cache mutations Add authenticated cluster-wide mutation fences for the opt-in DistErasure GetObject metadata cache. Guard recursive prefix and batch deletes, fail closed on configuration mismatch or uncertain quorum, and cover replay, restart, peer loss, and mixed-configuration behavior. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(metrics): consolidate PUT observers on improve/base (#8039) * perf(object): instrument storage hot paths Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(metrics): consolidate PUT stage observation Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: zhi22915 <qiuzgang@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com> * build(deps): bump workspace dependency releases * ci: allowlist rustix OFlags WRONLY in typos config * refactor(storage): route ecstore symbols through storage_api boundaries * perf(ecstore): cache quorum-selected early-stop metadata (#8308) Publish eligible latest-version GET metadata after a validated read quorum when early-stop cancels the remaining fanout. Keep metadata errors, selection quorum, online read quorum, and mutation-generation checks as publication gates. Add slow-peer repeat-hit and concurrent mutation-fence regression tests. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * chore: merge latest main into improve/base (#8332) * refactor(storage): bind rename rollback to its commit context (#8324) * refactor(storage): separate metadata scheduling from quorum decisions (#8327) * test(storage): verify rejected rollback inspection on real disks (#8331) * test(storage): make commit fault checkpoints independent (#8326) * test(storage): make commit fault checkpoints independent * test(storage): simplify commit checkpoint lookup * fix(listing): count logical prefixes before pagination (#8328) * fix(listing): count logical prefixes before pagination * test(listing): normalize the fixed corpus seed --------- Co-authored-by: Chris <anzhengchao@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * perf(ecstore): borrow metadata cache lookup key (#8333) Avoid request-scoped Arc<str> allocations on eligible cache lookups while preserving full key identity. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * chore: sync improve/base with latest main (#8334) * refactor(storage): bind rename rollback to its commit context (#8324) * refactor(storage): separate metadata scheduling from quorum decisions (#8327) * test(storage): verify rejected rollback inspection on real disks (#8331) * test(storage): make commit fault checkpoints independent (#8326) * test(storage): make commit fault checkpoints independent * test(storage): simplify commit checkpoint lookup * fix(listing): count logical prefixes before pagination (#8328) * fix(listing): count logical prefixes before pagination * test(listing): normalize the fixed corpus seed * fix(scanner): isolate and coalesce dirty usage journals (#8325) * fix(scanner): isolate and coalesce dirty usage journals Bind dirty replay records to a stable node owner so peers cannot overwrite or delete another node's pending journal. Replace the unbounded command queue with bounded coalesced state and revision-aware background saves. Keep legacy records isolated and enforce replay byte limits with conservative whole-bucket or unverified fallback. Add publication-stage diagnostics and storage regressions for ownership, clearing, retry, and restart behavior. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(scanner): collapse confirmed journal clear condition Use a let-chain to retain the generation fence before removing a confirmed bucket and updating its scope byte accounting. Satisfy collapsible_if without changing journal clearing behavior. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: Chris <anzhengchao@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(obs): distinguish metrics by process identity (#8343) Attach process.pid to the OpenTelemetry resource and promote it for Prometheus so counter series from restarted processes remain distinct. Preserve the stable RustFS node and service instance attributes. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * merge: sync origin/main into improve/base (#8345) * refactor(storage): bind rename rollback to its commit context (#8324) * refactor(storage): separate metadata scheduling from quorum decisions (#8327) * test(storage): verify rejected rollback inspection on real disks (#8331) * test(storage): make commit fault checkpoints independent (#8326) * test(storage): make commit fault checkpoints independent * test(storage): simplify commit checkpoint lookup * fix(listing): count logical prefixes before pagination (#8328) * fix(listing): count logical prefixes before pagination * test(listing): normalize the fixed corpus seed * fix(scanner): isolate and coalesce dirty usage journals (#8325) * fix(scanner): isolate and coalesce dirty usage journals Bind dirty replay records to a stable node owner so peers cannot overwrite or delete another node's pending journal. Replace the unbounded command queue with bounded coalesced state and revision-aware background saves. Keep legacy records isolated and enforce replay byte limits with conservative whole-bucket or unverified fallback. Add publication-stage diagnostics and storage regressions for ownership, clearing, retry, and restart behavior. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(scanner): collapse confirmed journal clear condition Use a let-chain to retain the generation fence before removing a confirmed bucket and updating its scope byte accounting. Satisfy collapsible_if without changing journal clearing behavior. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(storage): retain typed owners across borrowed writes (#8329) * fix(storage): retain typed owners across borrowed writes * fix(storage): preserve Copy write completion policy * refactor(storage): remove redundant write owner references * fix(ecstore): hide rollback backup residue in prefix listings * test(connect): isolate short HTTP fixtures in nextest * fix(protocols): enforce S3 policies for file protocol operations (#8262) * test(ecstore): enforce main core regression gates (#8338) * test(ecstore): enforce main core regression gates * test(ecstore): pin external shards and completion regressions * fix(test): restore authenticated cluster fault scenarios (#8340) * fix(test): restore authenticated cluster fault scenarios * fix(test): use the E2E storage API boundary * ci: coerce the pool timeout input to a number timeout-minutes requires a numeric value, but inputs.pool_timeout_minutes is a string, so '${{ inputs.pool_timeout_minutes || 240 }}' evaluated to the string '240' and the runner rejected the template: 'An error occurred when attempting to determine the step timeout ... Unexpected value 240' (chain run 36505625402 job at 02:25:52, standalone run 36393453114 at 07:46:37). Both pool runs since #8155 merged have therefore executed without the intended step budget. Wrap the expression in fromJSON and update the policy test that pinned the broken literal. * ci: coerce the pool timeout input to a number timeout-minutes requires a numeric value, but inputs.pool_timeout_minutes is a string, so '${{ inputs.pool_timeout_minutes || 240 }}' evaluated to the string '240' and the runner rejected the template: 'An error occurred when attempting to determine the step timeout ... Unexpected value 240' (chain run 36505625402 job at 02:25:52, standalone run 36393453114 at 07:46:37). Both pool runs since #8155 merged have therefore executed without the intended step budget. Wrap the expression in fromJSON and update the policy test that pinned the broken literal. * fix(ecstore): own listing workers through cancellation (#8342) Consolidate the three page collectors, preserve completed worker failures, and abort nested merges when their owner is dropped. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): route commit sync through guarded helper Use the Linux-capable guarded directory sync entry point for both commit sync sites. This keeps group commit and file-sync admission behavior while ensuring the already-open commit directory remains the sync target. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): expose rename directory guard in production The commit path now acquires the directory guard in production builds, not only tests, so destination metadata reads and durable sync share the guarded context. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: Chris <anzhengchao@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> Co-authored-by: GatewayJ <835269233@qq.com> Co-authored-by: hector <42570491+majinghe@users.noreply.github.com> * fix(ecstore): remove unsafe metadata cache admission paths Keep distributed GET metadata on quorum reads and quota admission on authoritative storage. Preserve local cache invalidation, remove the experimental RPC surface, and gate diagnostic work. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): cover cold degraded mutation admission Check first PUT and DELETE with one peer absent and no SDK retries, then verify that losing write quorum still rejects writes. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): retain guarded Linux destination metadata reads Use the existing commit directory guard when reading old xl.meta on Linux. Cover symlinked destination metadata for inline and non-inline renames without changing the split commit structure. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * chore(test): remove retired cache RPC import Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * merge: sync latest main into improve/base (#8349) * test: allow cleanup fixture CI headroom (#8336) * fix(s3): encode multipart listing response keys (#8347) --------- Co-authored-by: cxymds <cxymds@gmail.com> Co-authored-by: Chris <anzhengchao@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * merge: sync main auth and scanner updates (#8352) * test: allow cleanup fixture CI headroom (#8336) * fix(s3): encode multipart listing response keys (#8347) * perf(scanner): reduce scan and checkpoint peak allocations (#8346) * perf(scanner): reduce scan and checkpoint peak allocations * test(scanner): isolate partial commit recovery scenarios * fix(auth): return ExpiredToken for retained STS credentials (#8344) * fix(auth): return ExpiredToken for retained STS credentials Validate session identity independently of credential expiry in both IAM rejection and claims validation, and preserve typed S3 errors at the caller. Keep token binding, disabled-account precedence, and service-account behavior. Add exact expiry, IAM, and HTTP/XML regression coverage. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): synchronize the metadata slowtail hedge regression Park the slow metadata task at a test-only fault checkpoint and release healthy reads after the fault is active. Assert the spare response and cancelled-task drain without imposing a 300 ms disk/scheduler deadline. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ilm): synchronize copy compensation transition stages Observe automatic compensation reaching the mock tier PUT checkpoint before waiting for the final metadata commit. Assert forced enqueue failure, automatic backfill, one completed tier upload, and payload integrity. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: cxymds <cxymds@gmail.com> Co-authored-by: Chris <anzhengchao@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): reject overlapping cleanup schedulers Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): reject overlapping cleanup schedulers Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * docs(ecstore): explain cleanup scheduler conflict Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(metrics): separate GET request and timeout outcomes (#8354) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * docs: refresh security advisory lessons (#8350) * fix(heal): bound durable MRF rescheduling (#8351) Reuse stable live-owner proof anchors to avoid duplicate in-flight admission. Pace unverified completions and unavailable attempts with bounded jitter while retaining checkpoints, exact proofs, and generation isolation. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(rustfs): route recovery errors through S3 gateway Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * chore(deps): update workspace dependencies Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(io-metrics): use is_multiple_of for sampling Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): satisfy strict Clippy pagination checks Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(heal): avoid cloning Copy purge identity Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * add ignore .mimosa * fix(zip): adapt archive readers to zip 9 APIs Upgrade zip and its required dependencies alongside fallible archive-name handling in log ingestion, report bundles, bucket metadata import, and archive tests. Preserve existing skip and error propagation behavior. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): reuse quorum metadata for body GETs Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ci): align storage test stacks and peer reset classification Apply the existing bounded ECStore stack setup to the affected cross-crate storage fixtures. Validate conditional-only metadata publication separately from full body-layout cache reuse. Classify typed h2 shutdown resets as unreachable while retaining application, authentication, cancellation, and deadline distinctions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * chore(deps): update workspace dependencies Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): preserve omitted-disk GET recovery Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * chore: integrate published workspace dependency updates Preserve the concurrent dependency refresh and retain the typed h2 reset classification and bounded storage-test stack setup. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): preserve snapshot size budget after fallback move * fix(test): use one barrier for compensation COPY Let the forced-enqueue helper own the PUT barrier and compensation assertions. Waiting on an outer barrier overwritten by the helper always timed out after the upload had completed. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * chore(ecstore): apply current lint fixes Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * chore(deps): update sha2 to 0.11.1 Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: zhi22915 <qiuzgang@gmail.com> Co-authored-by: GatewayJ <835269233@qq.com> Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: Chris <anzhengchao@gmail.com> Co-authored-by: hector <42570491+majinghe@users.noreply.github.com> Co-authored-by: cxymds <cxymds@gmail.com> |
||
|
dependabot[bot]
|
7ad4fa439e |
build(deps): bump the dependencies group across 1 directory with 3 updates (#8481)
Bumps the dependencies group with 3 updates in the / directory: [datafusion](https://github.com/apache/datafusion), rust-embed and [hotpath](https://github.com/pawurb/hotpath-rs). Updates `datafusion` from 55.1.0 to 55.2.0 - [Changelog](https://github.com/apache/datafusion/blob/main/CHANGELOG.md) - [Commits](https://github.com/apache/datafusion/compare/55.1.0...55.2.0) Updates `rust-embed` from 8.12.0 to 8.13.0 Updates `hotpath` from 0.28.5 to 0.28.6 - [Release notes](https://github.com/pawurb/hotpath-rs/releases) - [Changelog](https://github.com/pawurb/hotpath-rs/blob/main/CHANGELOG.md) - [Commits](https://github.com/pawurb/hotpath-rs/compare/v0.28.5...v0.28.6) --- updated-dependencies: - dependency-name: datafusion dependency-version: 55.2.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: dependencies - dependency-name: hotpath dependency-version: 0.28.6 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: dependencies - dependency-name: rust-embed dependency-version: 8.13.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: dependencies ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Hauser
|
1266d4c270 |
feat(tls): reload and publish external certificate generations (#8467)
* feat(tls): reload external certificate snapshots coherently Pin projected certificate generations, validate complete TLS material, coordinate server consumers, and invalidate outbound RPC caches coherently. Add externally managed TLS Secret support and rotation guidance. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * feat(tls): publish server certificates through a fenced admin API Add opt-in server-leaf publication with dedicated IAM actions, actual TLS transport checks, immutable generations, durable captured-token CAS and idempotent receipts. Preserve independent trust and client identity sources and distinguish visible publication state from the full serving chain. Validate 64 TLS tests, 41 TLS-runtime tests, the focused IAM policy test, and eight live local API scenarios including least privilege and restart. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(admin): route certificate handler types through S3 facade Use the existing admin S3 facade to keep direct s3s imports within the footprint ratchet. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(admin): include TLS certificate routes in route matrix Account for the registered certificate-state GET and certificate publication POST without changing route behavior or weakening matrix assertions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(tls): reconcile startup trust before skipping reloads Compare the initialized listener bundle with coherently published outbound material, retain startup trust and client identity when sources disappear, and cover independent startup rotations through real reload passes. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(tls): reuse pinned generations and prepared listener state Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(tls): preserve startup trust source requirements --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> Co-authored-by: overtrue <anzhengchao@gmail.com> |
||
|
GatewayJ
|
a424d10fa0 |
refactor(iam): construct OIDC runtime directly (#8436)
* refactor(iam): remove legacy OIDC federation paths * refactor(iam): construct OIDC runtime directly |
||
|
Chris
|
b77bd26874 | Merge commit from fork | ||
|
Hauser
|
25e73d3bd6 |
fix(zip): statically link liblzma on macOS (#8434)
* fix(zip): statically link liblzma on macOS Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(zip): pin macOS liblzma minimum to 0.4.8 Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(admin): route bucket recovery errors through facade Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): group version pagination markers Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(heal): avoid cloning copy purge identity Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser
|
b18c5afc8d |
feat(targets): support Kafka SASL without TLS (#8398)
* feat(targets): support Kafka SASL without TLS Propagate the explicit Kafka transport mode through notify and audit targets while retaining TLS as the default. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * chore(deps): update RustFS Kafka crates to 1.4.0 Use the published Kafka clients after the SASL plaintext transport support release. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Lindroos Hsu
|
db2abcf0a6 |
feat(http3): add opt-in QUIC transport (#7903)
* build(http3): add optional s3s-http3 dependency Signed-off-by: tommady <tommady@users.noreply.github.com> * feat(server): add opt-in HTTP/3 transport Signed-off-by: tommady <tommady@users.noreply.github.com> * test(server): cover HTTP/3 TLS configuration * feat(server): add opt-in HTTP/3 support Serve S3 requests over QUIC with the existing middleware stack. Signed-off-by: tommady <tommady@users.noreply.github.com> * fix(http3): cancel pending body reads without panicking * test(http3): keep transport independent of s3s --------- Signed-off-by: tommady <tommady@users.noreply.github.com> Co-authored-by: overtrue <anzhengchao@gmail.com> |
||
|
Hauser
|
14eb28dcd6 |
chore(deps): refresh workspace dependencies and hotpath profiling (#8380)
* chore(deps): update workspace dependencies Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix: allow hotpath label marker attributes Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix: refine hotpath profiling attribution Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test: guard object hotpath attribution Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * Add ProfileReasonCode to CPU profile capture Signed-off-by: Hauser <housemecn@gmail.com> * fix: restore CPU profile test import Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Signed-off-by: Hauser <housemecn@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> Co-authored-by: Chris <anzhengchao@gmail.com> |
||
|
GatewayJ
|
051a54f129 |
fix(table-catalog): verify Iceberg metadata path signatures (#8291)
* fix(table-catalog): verify Iceberg metadata path signatures * fix(deps): allow pinned s3s Git source |
||
|
dependabot[bot]
|
c71e4f72ea |
build(deps): bump the dependencies group with 4 updates (#8359)
Bumps the dependencies group with 4 updates: [aws-sdk-kms](https://github.com/awslabs/aws-sdk-rust), [aws-sdk-s3](https://github.com/awslabs/aws-sdk-rust), [aws-sdk-sts](https://github.com/awslabs/aws-sdk-rust) and [hotpath](https://github.com/pawurb/hotpath-rs). Updates `aws-sdk-kms` from 1.122.0 to 1.123.0 - [Release notes](https://github.com/awslabs/aws-sdk-rust/releases) - [Commits](https://github.com/awslabs/aws-sdk-rust/commits) Updates `aws-sdk-s3` from 1.151.0 to 1.152.0 - [Release notes](https://github.com/awslabs/aws-sdk-rust/releases) - [Commits](https://github.com/awslabs/aws-sdk-rust/commits) Updates `aws-sdk-sts` from 1.118.0 to 1.119.0 - [Release notes](https://github.com/awslabs/aws-sdk-rust/releases) - [Commits](https://github.com/awslabs/aws-sdk-rust/commits) Updates `hotpath` from 0.28.0 to 0.28.2 - [Release notes](https://github.com/pawurb/hotpath-rs/releases) - [Changelog](https://github.com/pawurb/hotpath-rs/blob/main/CHANGELOG.md) - [Commits](https://github.com/pawurb/hotpath-rs/compare/v0.28.0...v0.28.2) --- updated-dependencies: - dependency-name: aws-sdk-kms dependency-version: 1.123.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: dependencies - dependency-name: aws-sdk-s3 dependency-version: 1.152.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: dependencies - dependency-name: aws-sdk-sts dependency-version: 1.119.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: dependencies - dependency-name: hotpath dependency-version: 0.28.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: dependencies ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Hiroaki KAWAI
|
94105a1044 |
fix(heal): skip removed usage observations during read repair (#8135)
* fix(scanner): skip absent usage observations during heal admission * fix: skip usage observation metadata recreation * style: apply rustfmt to scanner fix * Revert "style: apply rustfmt to scanner fix" This reverts commit |
||
|
Chris
|
641c4b3493 |
chore(release): merge 1.0.1 back into main and refresh installation references (#8314)
* fix(ci): include pagination regression in full E2E selection * fix(deps): replace yanked yoke-derive release * fix(scanner): expose pause backlog replica diagnostics (#8258) * fix(scanner): expose pause backlog replica diagnostics Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(connect): stabilize runtime profile lease cancellation Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(connect): tolerate delayed schedule startup in CI Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): retry quota reads during usage warmup Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): avoid meta-bucket incarnation self-deadlock Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(test): use persisted incarnation in heal fixture Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(usage): reconcile stale counters after lifecycle expiration (#8108) * fix(usage): reconcile stale counters after lifecycle expiration * fix(usage): account lifecycle expiry during continuous writes * test(usage): run lifecycle usage scenarios on one scanner store * test(usage): use a Windows-representable pre-mutation offset * fix(usage): harden expiry accounting recovery and quota checks Borrow expiry receipt bucket names and avoid allocating a map key on cache hits. Cover cancelled receipts, durable snapshot recovery, and legacy quota admission after scanner confirmation. Use representable timestamp offsets in the quota regression. Refs rustfs/backlog#2689 Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(usage): recover stale persisted counts through the scanner Seed incorrect complete usage for empty and retained-object buckets, then run the real scanner and publication consumer without further object mutations. Verify durable and admin usage over two cycles instead of writing a corrected snapshot in the test. Refs rustfs/backlog#2689 Refs rustfs/backlog#2691 Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): bound delimiter pagination fixture concurrency The 120-second smoke timeout expired after 1018 of 1200 serial fixture PUTs, before LIST ran. Prepare the same objects with at most eight concurrent requests and await every PUT. Retain the timeout and strengthen exact prefix, KeyCount, empty Contents, and continuation-token assertions with phase diagnostics. Refs rustfs/backlog#2689 Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(upgrade): establish a persisted previous-release baseline Seed the pinned previous-release cluster and restart it once with its data intact before replacing any node. Require every old writer to pass the strict readiness probe and preserve the seed through both mixed phases and the final current cluster. Keep InternalError fail-fast behavior and all existing compatibility deadlines and assertions. Refs rustfs/backlog#2689 Refs rustfs/backlog#2384 Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): bound cancellation metadata persistence waits Use the system-bucket incarnation boundary now supplied by main PR #8268. Bound the three cancellation waits that previously hung during pool.bin persistence, retaining their remote-generation, target-cohort, and durable-state assertions. Refs rustfs/backlog#2697 Refs rustfs/backlog#2689 Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: Chris <anzhengchao@gmail.com> Co-authored-by: Hauser <housemecn@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(e2e): retain startup and shutdown failure diagnostics * fix(test): supply CPU workload for sampler regression * fix(ci): locate security chain scripts in the workspace * fix(usage): recover historical counters with generation fencing (#8273) * fix(usage): recover historical counters during continued writes Use newer converged scanner snapshots to reconcile stale absolute usage baselines while preserving concurrent mutation and expiry receipt fences. Cover durable publication, admin and quota reads, and legacy generations. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(usage): fence snapshots and move preserved cache entries Apply the cached scanner generation floor before every reconciliation path and retain it even when an older snapshot happens to match core counts. Move preserved usage entries instead of cloning their histogram maps under the cache lock, retaining expiry receipt identity and cancellation fences. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): verify checkpoint takeover and repair dispatch (#8275) * test(scanner): cover checkpoint handoff and repair dispatch Drive runtime budget expiry, partial-cycle persistence, leadership claims, and stale checkpoint rejection between real disk-backed fixture scans. Verify that a metadata repair beyond the first bounded prefix is saved in the scanner ledger and dispatched by the MRF consumer. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * ci: isolate scanner fixtures and refresh full e2e membership Reserve nextest capacity for the real-disk scanner publication and MRF admission fixtures. Bind both platform membership checks to the reviewed pagination deadline test added on main. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(scanner): consolidate checkpoint fixture lifecycle Keep one durable control store across timeout and leadership transitions, and inject generation advancement into the shared checkpoint scenario. Check the actual saved metadata path so late-write rejection also proves that existing checkpoint bytes remain intact. Centralize MRF fixture isolation and reuse nextest process isolation when the startup environment already satisfies the test contract. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(obs): distinguish allocator counters from live memory (#8274) * fix(obs): distinguish allocator counters from live memory Preserve count/counter semantics and mark requested-byte attribution unavailable when live statistics or sampling are missing. Document sustained multipart memory diagnosis. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(obs): parse allocator statistics from one node Resolve each statistic before interpreting its shape, avoiding unsupported-field tree scans and mixing data across wrapper scopes. Preserve unavailable-statistics policy and add precedence regressions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): isolate late parity recovery from metadata hedges Use the existing object-scoped hedge timer barrier in exact-count recovery fixtures. Preserve payload and total-read assertions and verify that the omitted parity disk is read only during late refresh. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(usage): combine identical snapshot retention branches * fix(test): await HTTP sender readiness in Top RPC fixture * [release/1.0.1] Gate multipart copy through write admission (#8284) Gate multipart copy through write admission Make UploadPartCopy acquire the shared foreground write admission permit before lifecycle locks or source readers so server-side multipart copy cannot bypass the same backpressure used by UploadPart. Document the shared queue semantics and add focused coverage for saturation, cancellation, lock ordering, and disabled admission. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * Gate multipart copy through write admission (#8283) Make UploadPartCopy acquire the shared foreground write admission permit before lifecycle locks or source readers so server-side multipart copy cannot bypass the same backpressure used by UploadPart. Document the shared queue semantics and add focused coverage for saturation, cancellation, lock ordering, and disabled admission. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix: add UploadPart OOM validation guardrails (#8287) * docs(release): validate candidates on release branch * fix(scanner): validate checkpoints against global cycle fence (#8278) ## Related Issues Related to rustfs/backlog#2701. ## Summary of Changes Route scanner checkpoint cycle and leader validation through the global store while retaining the owning set for cache persistence, CAS revisions and publication admission. ## Verification Two independent final-diff source reviews found no issues across correctness, concurrency and durability, test coverage, compatibility, performance and simplicity on head `8b8fe51d092090b053f552ae283960e2e306be33`. Root approval `5373624714` is bound to that exact head. Regression tests cover real two-pool routing, stale fences, post-save rejection and CAS conflicts; their reported local execution belongs to the PR author, not this merge operation. Current required CI remains pending, and this authorized admin squash does not establish CI or runtime acceptance. ## Impact Restores checkpoint progress when global cycle and leader state differ from a set-scoped view. No format, retry, timeout, assertion or scanner-policy changes are introduced by this diff. The three prior main scanner failures remain unproved repaired. ## Additional Notes Full validation must run on the resulting exact main revision. Reverting this patch restores the earlier set-scoped fence lookup and its checkpoint rejection behavior. * fix(ci): restore E2E membership and pagination timeouts (#8281) * ci: locate the auto-testing checkout for lanes that run evidence from a subdirectory (#8279) ## Related Issues Follow-up to #8229. ## Summary of Changes Locate the private auto-testing checkout from the lane root or the workspace root so the nested security checkout can record functional-chain evidence. ## Verification The exact PR head |
||
|
Chris
|
5851d9eb54 | fix: filter CPU profile samples to reviewed executable symbols (#8248) | ||
|
dependabot[bot]
|
a2bdcf497b | build(deps): bump the dependencies group with 14 updates (#8176) | ||
|
Chris
|
5564932f38 | fix(heal): retry cancelled internode RPC operations (#8205) | ||
|
Chris
|
6d2b1c629e | Generate CPU symbol catalogs from release executables (#8165) |