mirror of
https://github.com/rustfs/rustfs.git
synced 2026-11-10 04:14:46 +00:00
e3ed2cf2f41e6ec7e44656d555c653f705ca499a
7599 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris An
|
e3ed2cf2f4 | fix(decommission): reserve owned source usage during activation (#9220) | ||
|
Chris An
|
bf03b1117a | fix(swift): preserve bucket metadata with Swift tag values (#9218) | ||
|
Boyi (Elliot) Li
|
c3fe0f9649 |
test(rio,ecstore): add negative-branch durability unit tests (#9216)
Expand positive and negative branch unit test coverage across durability-critical modules: - crates/rio/src/checksum.rs: Checksum::matches (unset hasher, multipart want_parts mismatch, digest mismatch), new_with_type, as_map, obj_type, from_header, add_part (CRC32/CRC32C/CRC64_NVME combine + error branches), and get_content_checksum header validation. - crates/rio/src/hash_reader.rs: HashReader::new wrapping invariants (already-read inner reader, conflicting MD5 checksum, conflicting positive size, unbounded size propagation) and checksum helper error paths. - crates/ecstore/src/set_disk/ops/bitrot_self_verify.rs: verify_written_bitrot_shards missing inline data, missing disk shard, and under-quorum rejection. - crates/ecstore/src/erasure/coding/encode.rs: read_full_buf_or_eof immediate read error, InvalidData preservation after partial fill, and UnexpectedEof normalization. - crates/ecstore/src/io_support/shard_integrity.rs: ShardVerifier out-of-bounds coding_index, empty proof sources fail-closed, and spilled (>1 MiB) index disk write + verify_index_file + restore_proof_replicas round-trip and all-corrupt fail-closed boundary. - crates/ecstore/src/services/integrity/tests.rs: JobRequest::validate boundary/format checks with exact error messages and model::protection / model::unsupported classifications. |
||
|
唐小鸭
|
6e31f6a183 | fix(ecstore): complete and abort native multipart uploads on Windows (#9191) | ||
|
Hauser Zhi
|
d66d144309 |
feat(observability): record unary read RPC durations (#9202)
* feat(observability): record unary read RPC durations Record complete server-side durations for ReadMultiple and BatchReadVersion, including error and cancellation paths.\n\nCo-Authored-By: heihutu <heihutu@gmail.com>\nCo-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(observability): cover unary duration handlers Exercise ReadMultiple and BatchReadVersion missing-disk paths and assert operation duration histograms.\n\nCo-Authored-By: heihutu <heihutu@gmail.com>\nCo-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(observability): cache unary read RPC duration handles Reuse cached duration histograms for ReadMultiple and BatchReadVersion once the stable server label is available. Preserve the existing ReadVersion handle and the fallback for other operations and backends. Add an integration regression for handle reuse, startup fallback, labels, and recorded values using the production cache path. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(observability): separate server duration from client RTT Record unary ReadMultiple and BatchReadVersion handler lifetimes in a separate server duration histogram while preserving existing client RTT series and allocation-free cached handles for both measurements. Cover client/server sample separation, cache reuse, startup and backend fallbacks, and missing-disk handlers. Document the new server metric. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> Co-authored-by: Chris An <anzhengchao@gmail.com> |
||
|
cxymds
|
150836d707 |
feat(tier): support native Azure Blob endpoints (#9200)
* feat(tier): support native Azure Blob endpoints * fix(tier): guard Azure version cleanup and prefix limits * fix(tier): preserve Azure quoted headers and document version gate * fix(tier): handle Azure forbidden current-version deletion * test(tier): control transition commit lock failure timing |
||
|
Henry Guo
|
e4057b6746 |
fix(table-catalog): reject storage credentials for catalog access (#9185)
* fix(table-catalog): reject storage credentials for catalog access * fix(table-catalog): avoid growing the s3s error macro footprint Use the existing S3Error constructor for the storage-credential rejection while preserving its AccessDenied code and message. Keep the merged PR within the current s3s footprint baseline without relaxing the guard. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: Hauser Zhi <housemecn@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> Co-authored-by: Chris An <anzhengchao@gmail.com> |
||
|
Hauser Zhi
|
cfdd705b62 |
fix(cors): preserve readiness and complete cache selectors (#9207)
Use a direct response future for originless requests and forward through the polled-ready service. Nominate every retained CORS header input and transfer evaluated headers without rebuilding the map. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser Zhi
|
a1d0fa41d0 |
fix: preserve usage generations and unify request tracing (#9204)
* fix(ecstore): prevent cached usage snapshot rollback Order refreshed snapshots by their authoritative lineage before replacing cached usage. Share the rule with early bucket refreshes while retaining quota baselines, invalidation fencing, legitimate deletions, and clock recovery. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(server): skip trace propagation for disabled RPC spans Avoid extracting and assigning a remote parent when RPC tracing is disabled after authentication and admission. Cover disabled logging and the production HTTP span propagation path. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): preserve usage ordering across refresh recovery Retain the last good snapshot independently of negative-cache status. Order observations within the same authoritative baseline by their own coordinated epoch and cycle before wall time, including clock rollback and recovery. Capture persisted baseline identity without cloning its bucket maps. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(server): centralize request trace propagation Let the HTTP span factory own parent extraction and assignment. Use actual span filtering, preserve span-only subscribers, and remove repeated tracing work from RPC authentication. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Chris An
|
7d4982f206 | fix(storage): fence bounded deletes during rolling upgrades (#9214) | ||
|
Chris An
|
1a7fb2461d | fix(webdav): join connection and reload tasks on shutdown (#9213) | ||
|
Chris An
|
e1b9d1cb94 | fix(ecstore): preserve ownership across multi-pool recovery (#9212) | ||
|
Chris An
|
85313fc06f | fix(runtime): release retained replay and allocator memory (#9211) | ||
|
cxymds
|
b5f3207378 |
fix(lock): preserve acquisition progress during multipart uploads (#9209)
* fix(lock): retain slow responses within the quorum deadline * fix(lock): retain partial quorum timeout diagnostics * fix(lock): avoid guard registry scheduling convoys * chore(ecstore): trace metadata transaction lock purposes * fix(scanner): keep capability probes on read fences * fix(lock): preserve renewal and contention progress |
||
|
Hauser Zhi
|
5490e493ab |
fix(s3): retain protected multipart ETags in object listings (#9208)
* fix(s3): retain protected multipart ETags in object listings Record encrypted part ETag provenance, aggregate it at read quorum, and attest completed multipart ETags for keyless listings. Preserve legacy fingerprint protection and test versioned SSE-KMS and SSE-C upload, listing, and replay paths. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(s3): stream multipart ETag aggregation Restore the preserved ETag fast path when selected parts lack protected provenance. Stream native part digests through a fixed stack buffer, preserve legacy hex-or-raw semantics, and strengthen exact-output and override regression coverage. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(s3): preserve encrypted ETag proof during data movement Capture quorum-backed source proof before multipart completion metadata updates, require exact preserved ETag and version identity, and retain proof only when publication keeps the source version. Cover real two-pool rebalance, null versions, legacy sources, and injected or rewritten identities. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(s3): retain encrypted multipart ETag proof on restore --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> Co-authored-by: overtrue <anzhengchao@gmail.com> |
||
|
Hauser Zhi
|
134820c1e0 |
test(storage): reproduce HTTPS rejoin readiness in Docker (#9206)
* test(storage): reproduce HTTPS rejoin readiness in Docker Compare cluster-only and node-ready/write gates on immutable native binaries. Preserve first-failure and latest-acknowledged-content evidence across recovery, capture lock diagnostics before restart, and limit cleanup to verified owned resources. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(testing): unify Docker repro ownership and controls Use typed resource absence and one verified container inventory for staging, partial setup and retained nodes. Reject inherited experiment overrides before creating resources and remove duplicated state. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(testing): guard staging volumes and interrupted capture Verify volume ownership before privileged staging and disable implicit image-content copying. Preserve failed evidence and continue owned finalization when capture or cleanup is interrupted. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(testing): capture live rejoin logs and verify trace coverage Read live tmpfs logs inside the verified container and use one explicit stdout sink for diagnostics. Record bounded capture provenance and fail diagnostic coverage when acquisition, guard-drop or physical-release records are missing, without masking HTTP failures. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(testing): reject inherited aliases and finalize on SIGTERM Reject inherited MINIO compatibility settings before Docker resource creation. Route SIGTERM through interrupt-aware finalization and restore the previous handler after each run. Cover real process signals and failure preservation. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(testing): normalize legacy controls and termination Reject inherited MINIO compatibility aliases in neutral runtime images. Bridge run-mode SIGTERM into the existing interruption finalization and restore the caller signal handler on return or exception. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Chris An
|
2b9217b6f9 | test(e2e): fix quota and conditional write fixtures (#9210) | ||
|
Hauser Zhi
|
70a46d0e8c |
fix(webdav): support configured browser CORS access (#9205)
* fix(webdav): support configured browser CORS access Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> Signed-off-by: Hauser Zhi <housemecn@gmail.com> * perf(webdav): streamline CORS policy and preflight parsing Borrow connection-owned policy state and clone the backend only for actual requests. Validate borrowed header names and normalize HTTP lists when required. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> Signed-off-by: Hauser Zhi <housemecn@gmail.com> --------- Signed-off-by: Hauser Zhi <housemecn@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser Zhi
|
a910ef0dd0 |
fix(cors): isolate cache variants and streamline policy evaluation (#9203)
Apply cache variation at the shared response boundary, preserve existing Vary selectors, and reuse parsed global policy and compact CORS inputs. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Chris An
|
430ade191f |
fix(scanner): bound wide bucket-root fanout memory (#9193)
* fix(scanner): bound wide bucket-root fanout memory (#8961)
Root prefixes were excluded from ordinary compaction and force_compact only
flattened the root above a hardcoded 250k child ceiling, so buckets with
tens of thousands of top-level prefixes kept one cache entry each after
#8971. Compact wide roots outside checkpointed forward sweeps, and honor
force_compact's limit for root fanout.
* fix(scanner): keep root fanout on interrupted force_compact
Interrupted sweeps seal a resume frontier after force_compact. Applying
the ordinary 10k root limit there flattened frontier-named children and
made the next cycle replay completed prefixes. Use the emergency FORCE
ceiling on the interrupted path; successful completions still honor the
ordinary limit. Drop clippy assertions_on_constants in the new tests.
Signed-off-by: Chris An <anzhengchao@gmail.com>
* test(scanner): seal reloaded frontier after interrupted compaction
Const assertions keep the wide-root preconditions without tripping
clippy::assertions_on_constants. The interrupted and finished sweep
paths share one limit helper, and a cache wider than the ordinary
10k limit still seals its frontier child after reload.
Co-authored-by: Chris An <overtrue@users.noreply.github.com>
* fix(scanner): isolate unreadable children during root compaction
* fix(scanner): isolate compacted namespace child errors
* fix(scanner): preserve compacted scan accounting and coverage
Commit isolated namespace child usage only after successful traversal, while retaining completed accounting on cancellation and resource deferral. Synchronize reduced root totals before the cancellation-prone flush boundary.
Treat skipped namespaces as incomplete coverage and invalidate legacy sampling baselines until a fresh verification pass succeeds. Keep provisional usage out of progress updates and retain the upstream symlink regression.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): preserve legacy retry coverage
* test(scanner): regress cancel overwrite after wide-root reduce
houseme's
|
||
|
Hauser Zhi
|
c5b15b7279 |
fix(container): unify directory setup and startup diagnostics (#9201)
* fix(container): explain log directory initialization failures Report failed log directory creation with ownership guidance and a Compose quote hint while preserving configured values. Add a corrected example and real entrypoint regression coverage. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(container): unify startup directory preparation Share directory creation, diagnostics, and best-effort ownership across data and log paths. Skip delimiter and regex subprocesses for ordinary volume paths without introducing a second parser. Expand real-entrypoint coverage for ownership, failures, delimiters, distributed ranges, and setup command counts. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
cxymds
|
2a6da8208b |
fix(lock): converge cleanup after lost release acknowledgements (#9199)
* fix(lock): converge cleanup after lost release acknowledgements * fix(log-analyzer): track updated unlock cleanup logs * test(lock): align clocks in unlock cleanup regression |
||
|
cxymds
|
93b704420f | fix(ecstore): preserve multipart shards within write budgets (#9197) | ||
|
Ramakrishna Chilaka
|
dfd4d3e213 | perf(ecstore): fuse overwrite commits and optimize delete recovery (#9194) | ||
|
Hauser Zhi
|
1c1baa6b08 | fix(observability): avoid duplicate process metric labels (#9182) | ||
|
Chris An
|
0147b8671d |
ci: bump overtrue/cla-bot to v0.1.1 (#9189)
Co-authored-by: cursoragent <cursoragent@users.noreply.github.com> |
||
|
Hauser Zhi
|
8cd751f9d1 |
feat(ecstore): record unary gRPC operation duration (#9187)
Record end-to-end durations for ReadAll, WriteAll, ReadMultiple, and BatchReadVersion alongside existing request, error, and payload metrics. This makes direct unary payload ABBA results observable through the internode operation duration family. Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser Zhi
|
5ea2991850 |
fix(ecstore): retry failed internode channel prewarm (#9186)
Release the per-peer prewarm deduplication marker when the best-effort dial fails, allowing a later RemoteDisk construction to retry the connection warmup without changing lazy connect fallback behavior. Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Chris An
|
1556e05b8a |
fix(sftp): put directory objects as trailing-slash keys (#9180)
SFTP subdirectory mkdir and rmdir sent the on-disk __XLDIR__ marker through the public S3 API. After the suffix was reserved, those calls were rejected as invalid client keys, so ordinary directory operations failed while bucket mkdir still succeeded. Send the logical trailing-slash key and let storage encode the marker. Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Chris An <overtrue@users.noreply.github.com> |
||
|
Hauser Zhi
|
32238a179c |
fix(server): bound rejoin waits and preserve recovery ownership (#9181)
* test(server): cover recovered-node admission boundaries Document backend readiness and write-availability checks during rejoin. Extend the existing regression through the production middleware order without changing runtime admission or cluster-quorum semantics. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(server): bound rejoin waits and preserve recovery ownership Enforce startup and unreachable-grace bounds while consuming peer replies. Release admission when every set has its required peer majority and cancel surplus read-only probes without carrying votes between rounds. Bound local readiness observations and polling sleeps. Retain initialized IAM across preparation failures, install the context once, and retry only publication after readiness timeouts. Share preparation with inline startup. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(server): make peer admission folding incremental Compile peer-to-set memberships once and retain compact per-edge votes. Consume each reply once, discard its payload, and update only affected set counters instead of rescanning all sets and retained responses. Recheck the captured deadline after polling and folding a reply so late work cannot admit the node or clear an expired unreachable grace period. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Chris An
|
6b70584dde |
chore(ci): bump overtrue/cla-bot to v0.1.0 (#9188)
Pin CLA Check to f054da99 (v0.1.0) so built-in bot allowlist skips cursoragent, Dependabot, GitHub Actions, Renovate, and Copilot without requiring a CLA signature. Co-authored-by: cursoragent <cursoragent@users.noreply.github.com> |
||
|
Chris An
|
d6210ac17b |
fix(s3,ecstore,table-catalog): bound work driven by client input (#9172)
* fix(ecstore,heal,select): bound storage-layer work driven by client input Scan findings in the erasure store, heal, file metadata and S3 Select layers, grouped because they share the disk walk, the multipart staging area, delete recovery and the select snapshot lease: - Multipart: upload ids are canonicalized before they name a staging path or a lock key (backlog#3027); trailing-slash keys are encoded like every other handler (backlog#3055); unlisted parts are removed at completion (backlog#3106); per-bucket uploads are found through a signed per-disk index instead of a scan of the shared staging area (backlog#3121); encrypted part ETags carry a per-upload secret so a write-only user cannot confirm plaintext guesses (backlog#3044). - Delete and recovery: recovery walks are bounded per request (backlog#3038); a delete marker on a key whose xl.meta path is a directory, and sidecar names used as child keys, no longer wedge the journal (backlog#3035, backlog#3105); over-quota PUTs no longer list the bucket under the ledger lock (backlog#3128). - Listing: scan_dir sends metadata without inline data (backlog#3052), counts delete markers against max-keys (backlog#3108), and rejects a marker that carries the cursor tag (backlog#3059). - Heal: bucket metadata and heal records live in separate namespaces so DeleteBucket cannot erase a drive rebuild (backlog#3039), a deleted bucket is skipped instead of failing the rebuild (backlog#3040), and directory objects are encoded on readback (backlog#3061). - Restore and tiering: RestoreObject preserves tags and the version's modification time (backlog#3073, backlog#3074); Snowball transport headers are not forwarded to the tier (backlog#3090); an epoch modification time round-trips through xl.meta (backlog#3063). - Locks: PutObject takes its bucket locks after the body is staged (backlog#3131); SelectObjectContent releases its namespace locks once the snapshot is taken (backlog#3132). - RPC and peers: the v3 signature check consumes the v2 nonce (backlog#3129); a bucket-metadata reload timeout no longer marks the peer offline (backlog#3124); replication target requests carry an operation deadline (backlog#3102); the scanner resumes its interrupted wait after a lifecycle write (backlog#3118). - S3 Select: bounded schema inference, column cap and budgeted CSV batches, Parquet dictionary bounds and pre-planner projection checks (backlog#3075, backlog#3076, backlog#3077, backlog#3078). * test(e2e): seed legacy duplicate bucket tags offline for the OPA contract * fix(s3): authorize what handlers store and bound request-driven work Scan findings in the S3 request layer (access hooks, object use cases, HTTP server layers), grouped because they share the authorization condition builder, the PutObject admission path and the response layers. Builds on the storage-layer batch for the bounded recovery and multipart APIs it calls. Authorization and policy conditions: - copy source is percent-decoded and version-canonical before the s3:x-amz-copy-source condition (backlog#3135); CopyObject authorizes the tag set it copies (backlog#3034); HeadObject with the replication check header still authorizes s3:GetObject (backlog#3047); the x-rustfs-source-version-id header feeds s3:versionid (backlog#3072); recursive force-delete with a REPLICA header authorizes every descendant as a delete (backlog#3070); include-deleted listings require s3:ListBucketVersions (backlog#3099); the metadata listing route keeps table isolation and uses the same prefix it authorizes (backlog#3053, backlog#3054); If-Match probing needs read permission (backlog#3097); Object Lambda checks s3:GetObjectVersion (backlog#3058); replication proxy fallback keeps ExistingObjectTag denies (backlog#3089); HeadObject returns lock and tag headers only with permission (backlog#3113); IPv4 clients on the dual-stack listener match IPv4 SourceIp conditions (backlog#3049); bucket policies are parsed once and cached (backlog#3032); keystone: access keys no longer verify with an empty secret (backlog#3043). Admission and resource bounds: - GetObject releases disk-read permits when the client stops reading (backlog#3023); PutObject takes buffer and foreground admission only after body bytes arrive and bounds the wait (backlog#3031, backlog#3084); xl.meta size is bounded on the write path (backlog#3100); XML compat layers stream instead of buffering whole bodies (backlog#3094); tier GETs stream the remote body (backlog#3046); on-demand migration pulls release permits on unread GETs (backlog#3110); SSE-KMS key policy covers UploadPart and completion (backlog#3130); MinIO admin decryption runs Argon2id off the async worker (backlog#3024). Input validation and leaks: - replication headers are validated (source ETag shape, original size) (backlog#3092, backlog#3093); control characters in keys and lifecycle rule IDs are rejected (backlog#3101, backlog#3117); Snowball member mtime needs s3:ReplicateObject (backlog#3083); DeleteObjects checks Content-MD5 and admin/STS reject SigV2 replays with altered query or body (backlog#3037, backlog#3026); request logs redact presigned queries and handler spans skip credentials (backlog#3068, backlog#3107); SSE-C listings and completions withhold plaintext metadata and checksums (backlog#3120, backlog#3050); CORS resolves the bucket like s3s does (backlog#3104); PutBucketTagging rejects duplicate keys (backlog#3126); notifications fire after a client disconnect (backlog#3127); ListenNotification streams re-check access (backlog#3119). * fix(api): drop imports left unused by the base merge * test(s3): move the KMS multipart denial test to the app layer The test drives app-layer multipart use cases, so it belongs with them; living in the SSE module it imported app code from the infra layer and failed the layer dependency guard. The SSE test helpers it borrows are exposed through the storage and app facades. * fix(s3): keep the SSE-C key fingerprint on keyless multipart completion The backlog#3050 fix withheld both the plaintext checksums and the customer-key MD5 header from a CompleteMultipartUpload that did not present the key. The checksums are a digest of the plaintext and stay hidden; the key MD5 identifies the key only, and the S3 conformance suite (test_copy_part_enc) expects it on a keyless completion, so it is echoed again. * fix(tier): stream GET bodies and document replication budgets * fix(ecstore,s3): keep the multipart index and cold fill off disk lifetimes Two lifecycle regressions surfaced once the storage and request batches ran together against main: - The multipart index owner cloned the disk's pinned publication root and bootstrapped on a task of its own. That kept the mount alive past the last LocalDisk owner (the disk lifecycle allows exactly three holders of the root FD) and left a stray task behind, which the local-disk and read-repair tests detect. The owner now keeps only the root path and reopens it for identity checks, and the bootstrap scan rides the disk's existing cleanup task, where stopping the disk abandons it and a reconnect resumes it. - The disk-read admission wrapper defers releasing its permit until an in-flight read completes, so a cancelled client cannot widen disk concurrency. The cold-fill producer, however, drops a read it has already classified as stalled when its deadline fires, and that permit stayed pinned for the lifetime of the stuck read. The producer now abandons the admission explicitly on timeout. * test(ecstore): pass the index owner to the Linux cleanup-loop test * fix(s3): release cold-fill admission after watchdog handoff * fix(s3,ecstore): close the round-1 adversarial review findings Adversarial review of the combined request- and storage-layer batch found fixes that fell short of their issues, two behaviour regressions for legitimate clients and several unrequested costs: - backlog#3038: a bulk DeleteObjects reserved the whole per-disk inspection allowance up front for every key in flight and exhausted the 16 Mi request budget on any set wider than eight drives, failing healthy keys with SlowDown. Calls are now admitted while budget remains and charged for the entries they actually visited. Object-scoped discovery charges plain child keys at one entry per 64 names instead of one each, so deleting a missing key that names a folder prefix completes instead of answering 503 forever while a hostile prefix stays bounded. A peer still on the previous binary is reported as an unsupported participant, the repository's rolling-upgrade contract, instead of as a budget verdict. - backlog#3037/#3026: a SigV2 configuration PUT without Content-MD5 was still replayable with another body; it is refused with MissingContentMD5 while SigV4 keeps the optional-MD5 contract. - backlog#3072: DeleteObject now resolves x-rustfs-source-version-id before authorization like PUT and COPY, so a replica delete cannot remove a version the policy never saw. - backlog#3121: the multipart index no longer fsyncs a marker and four directories per multipart metadata write (it is rebuilt on every start), and ListMultipartUploads discovers roots concurrently. - backlog#3077: Parquet SelectObjectContent decodes budget-sized batches instead of one row per batch. - backlog#3031: a full upload buffer pool hands out an unpooled buffer instead of refusing the upload; the progress floor already bounds stalls. - backlog#3023: the GET disk-read idle cut-off is configurable (RUSTFS_GET_DISK_READ_IDLE_TIMEOUT_SECS, default 120 s) so a paused client is not truncated after 30 s. - backlog#3075: JSON Lines schema inference keeps the sample collected at the 4 MiB bound instead of failing the query. - backlog#3024: compat admin decryption queues briefly for a slot instead of refusing the third concurrent request. - backlog#3113: HEAD skips its permission probes for the bucket owner. - backlog#3117: control characters in a lifecycle rule ID are rejected on PutBucketLifecycle as well. The lock-crate publication FIFO serves backlog#3111 and moves to the table-catalog PR. Also fixed: a lossy admin error message, an unused s3-client function, two unused test-util features, a duplicated key validation, the s3s footprint baseline, and a stale log event name in the reverse-proxy runbook. The ODM multipart write-back test runs on the large test stack like its sibling. * fix(table-catalog): bound maintenance work and isolate table namespaces (#9174) * fix(table-catalog): bound maintenance work and isolate table namespaces Scan findings in the Iceberg table catalog: - Metadata maintenance deleted every unreferenced *.metadata.json in a table's directory without checking whether a renamed table in another namespace still owned those files (backlog#3060); it now resolves the owning table through the directory index before deleting. - Manifest lists and manifests were decoded with the embedded Avro writer schema and no recursion depth limit; a self-referencing record schema overflowed the stack (backlog#3080). Schemas are validated for depth before decoding. - Compaction decoded every input Parquet file into Arrow batches held in memory; a decompression bomb OOM-killed the server (backlog#3085). Row groups are streamed through a byte budget and oversized inputs are refused. - An S3 request for a key outside an indexed table prefix listed the bucket's whole namespaces/ catalog prefix and failed once it held more than 4096 objects (backlog#3086); a per-bucket directory index answers index misses without the listing. - The maintenance planner read whole orphan files to obtain their modification time (backlog#3087); it reads metadata only. - A client disconnect between persisting active_rename_id and clearing it left the bucket's catalog entry locked at 503 (backlog#3088); the rename runs detached from the request future and clears the flag on every exit. - Table commits held the bucket publication fence until publication, starving every object write (backlog#3111); commits now hold it under a deadline and object writes are admitted between commits. The directory index is managed through three new admin routes under {warehouse}/catalog/directory, registered in the admin route policy. The durable-strong store is left as is (its snapshot partitioning for backlog#3109 and the strong-path deadline are not ported): that backing is being retired separately, so the new tests cover the object backing. * fix(table-catalog): close the review gaps in the scan-fix batch Adversarial review of the table-catalog batch found the fixes narrower than the issues they close and two cost regressions: - backlog#3085: the compaction footer check bounded nothing a page header could not override; parquet sizes its page buffers from the header. Walk every column chunk's page headers (Thrift compact, no decompression) and hold each page and the chunk total to the budget before decoding. The stored-input budget is per rewrite group, so a multi-group job no longer fails after writing earlier outputs. - backlog#3086: the default index-miss scan still counted maintenance reports toward its 4096-object limit; only table entries count now, and the listing itself is bounded to 64 pages. The directory index remains an optional accelerator. - backlog#3111: a create wrote its metadata file under the fence as the first durable mutation, which disarmed the deadline before the unbounded registration inventory. The write precedes publication now. The deadline gets its own knob (RUSTFS_TABLE_CATALOG_PUBLICATION_PREPARATION_TIMEOUT_SECS, default 30 s, never below the lock-acquire timeout) instead of reusing the 5 s lock timeout, and the ownership inventory reads entries concurrently. - Directory candidates are written only while a directory exists; the backfill covers earlier entries. A directory on the migration-fence path is reported as a conflict instead of a parse error. - Dead test hooks, a stale comment and the support-matrix text that did not match the code are removed or corrected. * fix(select): preserve complete samples and Parquet batch limits * fix(s3,ecstore): close the round-2 adversarial review findings DeleteObject resolved x-rustfs-source-version-id for authorization on every request while storage consumes it only for a REPLICA request, so a non-replica caller with s3:DeleteObjectVersion could trigger a plain delete of the current version. The hook now resolves the header under the same predicate as storage, and the regression test covers the non-replica case. HEAD no longer skips its permission probes for the bucket owner; an explicit bucket-policy Deny on the metadata actions binds the owner too. Multipart index pages verify each root against the staging area and prune markers whose staging root is gone, under the marker's own lease with a re-check, so the per-drive index stays proportional to live uploads instead of every key ever multipart-initiated on the drive. SigV2 configuration PUTs that declare an empty body need no Content-MD5: there is no body to replay. The minimum-progress floor stays independent of RUSTFS_HTTP_REQUEST_BODY_READ_TIMEOUT; the knob's documentation now says so instead of claiming 0 disables it. The quota bootstrap window (503 until the background scan publishes the ledger) is kept, because ledger admission must not come from scanner reporting; the compatibility matrix now documents it. The S3 compatibility matrix gains a section for the client-visible validation bounds, and the drive-timeout tuning guide tabulates RUSTFS_GET_DISK_READ_IDLE_TIMEOUT_SECS. * fix(lock): queue table-bucket publication writers fairly Table commits take the bucket publication fence as a write lock; under a stream of commits the lock's try-and-retry acquisition let a new commit jump ahead of object writers that had been waiting longer, which is part of what starved them (backlog#3111). Requests for the publication lock path now take a FIFO ticket per shard: a waiter is admitted only when it is at the head of the queue, cancelled waiters release their turn, and batch acquisitions join the same queue. Other lock paths are unchanged. This moved here from the request-layer batch, where no cited issue covered it; it serves the table-catalog issue this PR closes. * fix(table-catalog): close the round-2 adversarial review findings Parquet page-header walker: read the size fields by field id as parquet does, accepting I16/I32/I64 wire kinds and rejecting any other kind, so a header encoding uncompressed_page_size as I64 can no longer bypass the budget while parquet allocates it. Bound every list, set and map count by the bytes remaining in the chunk, and reject bool collection elements (parquet skips them with zero width, so either width would desynchronize the two parsers); this removes the 2^63-iteration spin on LIST<BOOL>. Start each column-chunk walk at the dictionary page offset whenever the footer declares one, exactly as parquet's byte_range does, instead of min(dictionary, data). Footer pre-walk: before parquet decodes a footer, walk FileMetaData with the hardened cursor and charge the declared row_groups count times the RowGroupMetaData size to the decode budget, because parquet 59 reserves that Vec before reading one element. Preparation deadline: clamp the RUSTFS_TABLE_CATALOG_PUBLICATION_PREPARATION_TIMEOUT_SECS knob to 86,400 seconds so Instant + Duration cannot overflow. Index-miss scan: after each listing page, jump the cursor past the maintenance subtree the page ended in, and allow one extra page per table entry already listed, so job-report history scales with the catalog while a report flood under one table stays bounded. Migration fence: read the fence-path object once and decide from its stored kind whether it is directory state or a fence. Expired or failed create: remove the initial metadata file when the registration fails before any catalog write may have begun; nothing else references it. Correct the comment and support matrix that claimed maintenance reclaims it. Tests cover each walker divergence, the footer row-group reservation, the deadline ceiling, the paging budget for honest catalogs and for a report flood, and the removal of the unpublished metadata file. * test(admin): register the catalog directory routes in the route matrix The table-catalog batch added GET, POST and DELETE {warehouse}/catalog/directory on both the Iceberg and the compat admin prefixes without listing them in the admin route matrix, so test_admin_route_matrix_matches_registered_routes failed on CI. * fix(s3,ecstore): close the round-3 adversarial review findings Multipart index pages now treat the cursor as the last examined marker rather than the last root, so a page that pruned a stale marker still validates and ListMultipartUploads keeps using the index. A marker is written before its staging root is published, so pruning spares markers younger than a five-minute grace period and an existing marker is touched on every new upload; the prune re-checks both under the marker lease. Bounded delete-recovery inspection rides the durable-mutation lane like the unbounded path instead of the one-worker foreground lane. The on-demand-migration inline body budget scales with the object size (at least 64 KiB/s) instead of a flat 120 s. The control-body digest service honors RUSTFS_HTTP_REQUEST_BODY_READ_TIMEOUT=0 as "no inactivity timeout" like every other reader of that knob. Quota reconciliation throttle objects are deleted together with a retired generation's ledger, so bucket turnover leaves no residue. The PUT guard test shim is gone; tests call the shared guard. Bounded DiskAPI defaults answer MethodNotAllowed like their siblings. Stale budget docs, an unreachable CSV arm, an always-Some limit parameter, commented-out constants and a misplaced doc comment are cleaned up; weak test assertions name the expected error; lifecycle rule-ID control characters gain a test. The compatibility matrix records the CopyObject tag-copy permission, the exact SigV2 empty-body predicate, replica and migration writes of control-character keys, and the SigV2 DeleteObjects digest rule. * fix(table-catalog,heal): close the round-3 review findings A. Pack compaction rewrite groups to min(max-rewrite-bytes-per-job, TABLE_COMPACTION_MAX_STORED_BYTES), reject an over-ceiling group before any output is written, and name the ceiling in the commit-path errors; document the effective cap in the support matrix and scheduler runbook. B. Keep TaskCancelled/TaskTimeout classification on replacement passes: the namespace verification runs only for completed or storage-failed passes, in the erasure healer and the task bucket prepass. C. Compare the replacement bucket pass per bucket: a recorded identity that is gone or replaced still errors or restarts the pass, buckets created mid-pass are appended to the identities, pending_buckets and a persisted positional order without rewinding progress or the checkpoint; verification ignores additions. D. Read the migration-fence object once in the object-backed write permit and dispatch on its kind. E. Drop the strong store's copy of maintenance_cleanup_roots. F. Remove the unused test hooks fail_put_prefix, corrupt_put_prefix and pause_after_metadata; make compact_parquet_data_files and PublicationTicket.state private. G. Document the body-revision identity, the legacy revalidation budget and its HTTP 503 in the support matrix. H. Define HEAL_REPLACEMENT_RECOVERY_DIR once in ecstore and consume it from heal through the storage_api facade. * fix(ecstore): cap the index pages one disk may answer Pruned multipart index pages may carry few roots, so the page cursor no longer bounds how many pages a peer can make a listing fetch. Cut a disk off after 4096 pages (about one million examined markers) so it contributes no coverage and the listing falls back to the staging scan. * fix(ecstore,table-catalog): close the round-4 adversarial review findings Multipart index markers dated in the future (a backwards clock step between recording a marker and publishing its staging root) were judged old and could be pruned while their upload was in flight; a future-dated marker is now kept. The marker touch on EEXIST is best effort, so a filesystem that refuses to set times no longer fails the upload commit. Pruning runs on the durable-mutation lane instead of the one-worker foreground lane. Quota reconciliation keeps one throttle object per bucket incarnation (a quota revision change replaces it instead of orphaning the previous revision's object), the retirement cleanup deletes it ahead of the ledger and also when the ledger is already gone, and a scan the retirement already refused writes nothing back. Bounded DiskAPI defaults answer UnsupportedDisk, the verdict the aggregate gives a peer without the protocol. The compaction ceiling pre-check runs over every rewrite group before any output is written, and the worker's ceiling error no longer names a placeholder group. Docs: the inline on-demand-migration client budget, the protocol-2 downgrade conflict of replacement passes, the exact footprint numbers. |
||
|
Chris An
|
158aa276e9 |
fix(storage): bound heal retries and recover pool metadata (#9164)
* fix(ecstore): retry metadata writes rejected before staging * fix(ecstore): reuse registered disks during startup * fix(ecstore): retain remote health monitors during reconnect * test(heal): release the commit barrier during graceful shutdown * test(e2e): register graceful shutdown barrier regressions * fix(ecstore): roll back interrupted pool metadata commits * test(upgrade): honor the 1.0.4 delete recovery protocol * fix(heal): bound admission during object rate pressure * test(e2e): handle rebalance completion before stop * test(e2e): observe rebalance activity before the start reply * test(rebalance): hold the first run until stop cancels admission |
||
|
Hauser Zhi
|
3362421240 |
fix(obs): add fine-grained object lock histograms (#9177)
Use explicit buckets that preserve millisecond lock acquisitions while retaining visibility into long-held delete and recovery locks. Add an exporter-backed regression test for both histogram instruments. Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
GatewayJ
|
57df9181d0 | fix(audit): preserve OIDC identity in S3 request audits (#9178) | ||
|
Hauser Zhi
|
371f5e4ffc |
fix(ecstore): streamline lock RPC and listing ownership (#9176)
* fix(ecstore): preserve lock cleanup and listing trace context Own remote lock admission and dispatched RPC cleanup across caller cancellation. Reuse bounded late-result cleanup and retain the listing request span in metacache tasks. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(ecstore): streamline lock RPC and listing fanout Poll RPCs in callers and transfer pending work before channel eviction. Share walk options across producers and borrow cached peer keys. Keep deadline classification deterministic. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
cxymds
|
8a5a2c9c93 |
fix(s3): treat duplicate CreateBucket as owned by the requester (#9167)
* fix(s3): treat duplicate CreateBucket as owned by the requester
The duplicate-bucket response was decided from the requester's root-owner
flag, so every IAM user recreating a bucket received 409
BucketAlreadyExists, including a user recreating a bucket it created.
BucketAlreadyExists means the bucket belongs to another account; a
deployment owns a single namespace, so the answer is purely region-based:
the default region returns the legacy 200 OK, any other region returns
BucketAlreadyOwnedByYou.
Both the existing-bucket fast path and the BucketExists path now share one
helper. Authorization still runs first, so a requester without
s3:CreateBucket gets AccessDenied and cannot probe bucket existence.
Refs #8493
* test(ci): refresh e2e selections for duplicate CreateBucket
The two new create_bucket_region cases join the e2e-smoke and e2e-full
selections, which main also extended with multipart_upload_id tests.
smoke is measured from a local nextest listing: removing the two new cases
reproduces main's committed digest exactly. e2e-full is derived the same way
from main's listing artifact (run 41933823399,
|
||
|
Hauser Zhi
|
babfeaed00 |
fix(io-core): preserve pooled capacity and accounting (#9171)
* fix(io-core): preserve pooled capacity and accounting Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(io-core): preserve pool allocation admission order Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * docs(io-core): clarify pool return locking Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Jose Meira
|
736bf777b4 |
feat(container): add declarative bucket bootstrap (#9162)
* feat(container): add declarative bucket bootstrap * fix(container): handle false bucket versioning setting * Update Rust base image to version 1.99.1-trixie Signed-off-by: Hauser Zhi <housemecn@gmail.com> * Update Rust base image version to 1.99.1 Signed-off-by: Hauser Zhi <housemecn@gmail.com> --------- Signed-off-by: Hauser Zhi <housemecn@gmail.com> Co-authored-by: Hauser Zhi <housemecn@gmail.com> |
||
|
KarlEmm
|
7babc7d621 |
fix(s3): report malformed multipart upload IDs as NoSuchUpload (#9147)
UploadPart, UploadPartCopy, ListParts and CompleteMultipartUpload returned 400 InvalidArgument for a malformed upload ID, while AbortMultipartUpload returned 404 NoSuchUpload. AWS S3 documents NoSuchUpload for an upload ID that might be invalid. Report MalformedUploadID as NoSuchUpload through one helper used by all operations that address an existing upload, replacing the Abort-only special case. ListMultipartUploads keeps InvalidArgument for a malformed upload-id-marker. Add the multipart_upload_id_test e2e module to the e2e-smoke profile. It asserts 404 NoSuchUpload from every operation that addresses an existing upload, for a malformed upload ID and for an aborted one, and updates the e2e-smoke and Linux e2e-full selection digests. Co-authored-by: Karl Bastarache <bastarachek@google.com> Co-authored-by: Hauser Zhi <housemecn@gmail.com> |
||
|
Hauser Zhi
|
4d46bd25f8 |
perf(ecstore): parallelize bulk-delete lock and metadata fan-out (#9155)
* perf(ecstore): probe bulk-delete write locks in parallel waves Bulk delete acquired one distributed write lock per key in serialized order, paying a full quorum round-trip per key. Probe the unique keys in bounded parallel waves with a short timeout instead; any contention drops the whole wave and the canonical serial order resolves it as before, so two overlapping batches can never deadlock. * perf(ecstore): probe bulk-delete transaction locks in parallel The set-layer delete-transaction locks were also acquired one quorum round-trip per key after the raw write locks. Apply the same optimistic all-or-nothing probe wave with a canonical serial fallback, and keep the per-key error reporting the fallback already provided. * perf(ecstore): pre-fetch bulk-delete source metadata in parallel The admission loop ran one metadata quorum fan-out per key in sequence on top of the per-key lock round-trips. Pre-fetch the source reads the loop needs with bounded parallelism under the locks already held, and derive the per-key read inputs from one shared helper so the pre-fetch and the loop cannot disagree. * test(ecstore): drop unused import in batch-delete probe test * refactor(ecstore,lock): await wave releases and unify probe waves Adversarial review of the bulk-delete lock waves found three real issues. Distributed guard release is fire-and-forget on drop, so the serial fallback could re-acquire keys while unlock RPCs were still in flight and block on locks it already owned: guards are now released with an awaited unlock before the fallback runs. The wave also probed every key after the first failure, adding up to one probe timeout per chunk of pure overhead under contention: probing now stops at the first failing chunk. Finally the two copy-pasted wave implementations are replaced by one shared probe_lock_wave, and the per-key delete read inputs are resolved once instead of being recomputed. Tests now assert the wave/fallback counters so they prove which path ran, and an overlapping-batch test pins the no-deadlock property. * refactor(ecstore,lock): batch bulk-delete lock rounds per key Replace the optimistic probe waves with the bulk lock primitive the lock layer already supports on the wire: one lock_batch RPC per lock client for a whole DeleteObjects key set, per-key quorum guards with the usual TTL heartbeat, and per-key outcomes. The batched round is deadlock-free by construction (every lock server orders concurrent batches by the same global (shard, key) order), so the probe timeout, the wave release-and-wait dance, and the canonical serial fallback all go away. Per-key lock outcomes are now reported per key end to end: a locked-out key fails alone in the response while the rest of the batch proceeds, matching the S3 DeleteObjects contract and the set layer's existing behavior. The set layer's raw-write-lock and delete-transaction-lock waves fold into the same primitive via one shared resource-folding helper, and the fast-lock batch fast path now counts acquisitions so shard metrics keep their meaning. * perf(ecstore): move journal publish off the one-worker namespace lane A bulk delete publishes one recovery transaction per key, and every journal inspection and settle-plan closure ran on the per-disk foreground namespace lane (one worker), serializing the whole publish phase per drive behind every other mutation. Route those closures onto the durable-mutation lane the fused delete prepare already uses (eight workers per disk, fsyncs coalesce in the filesystem journal), and keep 32 journal publications in flight so a 16-drive set can feed those lanes. * test(ecstore): flatten lock-error iteration in overlap test * fix(ecstore,lock): refresh duplicate keys after recovery, bound unlock Review findings on the batched lock rounds. A batch entry whose recovery rolls back an interrupted prepared delete can bring a retained version back; a later duplicate of the same key still held its pre-fetched source-missing read, skipped the object-lock check, and deleted the restored version. Recovery now evicts the pre-fetched reads of every other entry naming the same object, so those entries take the loop's live re-read and see the restored version. The batch rollback release awaited the full unlock retry schedule, including the deferred background pass, parking the request behind an unreachable peer. The synchronous phase is now a single release pass with the deferred retries spawned in the background. * fix(ecstore): release local batch locks before deferred recovery A set-layer batch delete that runs deferred key recovery re-acquires a write lock on the same keys through recover_delete_object, but the raw local batch guards were only ever released at function exit — the distributed path releases its batch ids before the deferred loop and the local path missed the mirror release, so the recovery deadlocked against the batch's own locks and surfaced as a spurious Lock(Timeout) for every deferred key. A regression test plants the interrupted prepared delete this needs and pins the end-to-end contract: a batch naming a GOVERNANCE-retained version twice must deny both entries after the first entry's recovery rolls the prepared delete back, and the restored version must survive. * test(lock): cover batch acquire partial-grant rollback The distributed acquire_guard_batch rollback path — a key granted by fewer than quorum clients must fail alone and release its partial grants before the call returns — was only reachable in distributed mode and had no coverage. A scripted per-object LockClient pins both halves: the below-quorum key reports QuorumNotReached with its partial vote, and the orphaned grant is released synchronously while the quorate key's guard stays held. |
||
|
Ramakrishna Chilaka
|
ff5f491c5a |
perf(ecstore): skip unmatched FdCache scans and maintenance wakeups (#9165)
Track active descriptor volumes, exact paths, and component prefixes in FdCacheTracker so unmatched invalidate_under, invalidate_volume, and invalidate_exact calls return without scanning Moka or waking the background maintenance task. |
||
|
Hauser Zhi
|
a8be876fd9 |
fix(s3): honor presigned x-amz-meta-* query parameters (#9146)
* fix(s3): honor presigned x-amz-meta-* query parameters The AWS SDK presigners hoist signable x-amz-* headers into the query string (smithy moveHeadersToQuery), and AWS S3 accepts the hoisted form as the header it came from. RustFS read user metadata from headers only, so a presigned PutObject carrying ?x-amz-meta-user-id=u1 returned 200 and stored an object with empty metadata (rustfs/rustfs#9142). Add one shared extractor for user-metadata query parameters and wire it into every metadata-accepting write path (PutObject including snowball members, CreateMultipartUpload, CopyObject REPLACE) plus the snowball extract trigger, instead of normalizing per operation the way the presigned UploadPart checksum fix did. The stored key form is produced by the same helper as the header path, so the hoisted and header forms land on disk identically. Only a signature-covered query is honored: SigV4 signs the canonical query string, while SigV2 signs headers plus a fixed subresource list only, so honoring its query would let a presigned-URL holder add object properties the presigner never authorized. Keys and values that could not have arrived as a header are rejected rather than stored, since the read path turns them into header fields and a control character would fail every Head/Get of that object. * refactor(s3): hoist signed x-amz query params at the request boundary Replace the per-write-path query metadata extractor with one boundary mechanism: restore the `x-amz-*` request properties the SigV4 presigners hoist into the query (`@smithy/signature-v4` `moveHeadersToQuery`) as request headers before s3s routes and deserializes the request. One hook now covers every reader at once — s3s input parsing, `extract_metadata_from_mime`, the snowball extract trigger and its transport-key filter, tagging, storage class, object lock and copy source — so PutObject, CreateMultipartUpload and CopyObject need no changes at all. It also replaces repeated query parsing along the write path with a single pass at the boundary. An existing header always wins, so the hoist is purely additive and the unsigned-header guard keeps rejecting anything the signer did not sign; that guard now also accepts a header whose single value equals a same-named query parameter, which is what the hoist produces and what the signature covers. `query_string_is_signed` rules out SigV2 before it trusts any SigV4 marker. SigV2's string-to-sign covers headers plus a fixed subresource list rather than arbitrary query parameters, so a SigV2 presigned URL stays valid after `X-Amz-Algorithm`/`X-Amz-Signature` markers are appended to it; without this rule a URL holder could turn on `x-amz-meta-snowball-auto-extract` and have a plain upload untarred (review of #9146). * fix(s3): hoist signed query params after s3s signature verification The request-boundary hoist ran before s3s, so the restored `x-amz-*` headers reached s3s's own verifier, which rejects any `x-amz-*` header outside `X-Amz-SignedHeaders` (s3s `ops/signature.rs`, `reject_unsigned_amz_headers`). A valid presigned PUT with `SignedHeaders=host` and `?x-amz-meta-user-id=u1` therefore came back with 403 AccessDenied before any handler ran (review of #9146). Move the hoist to `authorize_request`, RustFS's shared authorization entry: s3s has already verified the signature there, and policy evaluation plus the handler read request properties after it, so the restored headers reach every operation without ever appearing unsigned to the verifier. The mechanism is unchanged — one gated, validated pass over the signature-covered query. CopyObject's replacement metadata now also reads the headers, because s3s parsed `input.metadata` before the hoist ran. The hoist itself and its tests stay in `server::layer`; the tower layer is gone. * perf(s3): single-pass signature-aware query hoist in auth Move the query hoist into `auth.rs`, next to the unsigned-header rules it is part of, so `storage` no longer reaches into `server` for it. Its documented contract — which request properties the signature covers, and why SigV2 must never be widened — now lives with the rule it protects. Make the hot path allocation-free and single-pass. The caller no longer clones the query just to satisfy the borrow checker, the function no longer clones it again, and one `form_urlencoded::parse` pass both recognizes the presigned-SigV4 marker set and stages the hoistable parameters. Nothing is validated or applied until the gate passes, so a request whose query is not signature-covered is never rejected here. A single ASCII scan skips the whole thing when the query names no `x-amz-` parameter at all, which is every ordinary `?versionId=`/`?uploadId=` request and every queryless one. The duplicate-parameter and header-value checks stay strict; only when they moved, not how they behave. * fix(s3): keep verifying a query-carried payload hash after the hoist The hoist restores `?x-amz-content-sha256=<digest>` as a header, but it runs after s3s has already built the request body. s3s reads its payload-hash input from the headers only, so it installs no hash validator for a digest that arrived in the query, and the restored header then made `get_content_sha256_with_query` conclude the auth layer was enforcing it. Neither layer verified the body, so a presigned PUT whose digest exists only in the query accepted a modified payload (review of #9146). The question is not whether a digest header is present but whether the digest reached the request as a query parameter: that one is always the storage layer's to verify. A digest sent as a header is still left to the auth layer, which wraps the body in a hash-validating reader for it, so no payload is hashed twice. * fix(auth): limit late query hoisting to user metadata * refactor(s3): drop the stale payload-hash compensation, scope the pre-filter Limiting the hoist to `x-amz-meta-*` user metadata means it never manufactures an `x-amz-content-sha256` header, so the payload-hash predicate's original signal — a digest header present at handler time is one s3s is enforcing — is exact again. Restore it and drop the query-aware form: that change existed only to undo a header the hoist is no longer allowed to create, and it would have hashed a payload twice for a presigned request that declared a real digest both in the query and as a header. Match the pre-filter to the same scope so a query carrying only a typed property (`x-amz-tagging`, `x-amz-storage-class`, `versionId`) skips the parse entirely instead of scanning past it. * test(access): use explicit imports in the signed-query lock regression The architecture migration rule requires RustFS storage-owner modules to name their imports instead of pulling the parent module in wholesale, and `signed_query_lock_permission_regression` was still on `use super::*`. Import the two symbols it actually uses and drop the unused one. * test(access): reference FS through the parent import The layer dependency guard tracks `use crate::...` edges, and naming `crate::storage::ecfs::FS` from the signed-query regression looked like a new `infra -> interface` reverse dependency. `access.rs` already reaches the same type through `use super::ecfs::FS`, so reuse that import instead of introducing a second, tracked spelling of an edge the module has always had. No baseline entry, no new edge. --------- Co-authored-by: overtrue <anzhengchao@gmail.com> |
||
|
Hauser Zhi
|
fe3ada0689 |
chore(deps): refresh Cargo dependencies (#9156)
* chore(deps): refresh Cargo dependencies Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * chore(deps): update rustfs kafka to 1.4.1 Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser Zhi
|
926355ae1f |
fix(api): keep SlowDownRead off write-path quorum failures (#9149)
* fix(api): keep SlowDownRead off write-path quorum failures * refactor(api): write-safe quorum default, GET opts into SlowDownRead * fix(api): keep object-read mapping behind error.rs architecture boundary * refactor(api): drop redundant object-read mapper wrapper |
||
|
Chris An
|
7f5fa4b8f6 |
fix(protocols,kms,keystone): enforce client CA, protect the KMS probe (#9168)
Scan findings in the alternate-protocol front ends and the KMS and Keystone integrations: - WebDAV and FTPS read RUSTFS_WEBDAV_CA_FILE / RUSTFS_FTPS_CA_FILE, checked that the file existed, and then built their TLS listeners with with_no_client_auth(), so mTLS was never enforced. A shared protocols::common::tls::server_config builds the rustls config with a client-certificate verifier from the configured CA, startup fails on an invalid CA, and a CA cannot be combined with cleartext (backlog#3069). - The KMS readiness probe used a fixed key id that the key-lifecycle admin routes accepted like any customer key, so kms:DisableKey on it failed readiness on every node. The probe key is reserved from key management and a disabled probe key no longer counts as a backend failure (backlog#3115). - The Keystone middleware answered anonymous requests carrying a junk X-Auth-Token with the configured Keystone URL in the body while Keystone was unreachable; failures no longer disclose upstream locations, and the login cache is bypassed when a subscription token refresh is required (backlog#3116). |
||
|
Chris An
|
a97c65895c |
fix(proxies,targets,obs): bound untrusted input in leaf crates (#9166)
Scan findings in crates whose behavior is driven by untrusted input and that stand alone from the storage core: - trusted-proxies: the simple implementation takes the client address from the right-most untrusted X-Forwarded-For hop instead of the left-most (backlog#3082); the legacy validator never falls back to the proxy identity when the chain is rejected and reads X-Forwarded-For unless RFC 7239 is explicitly selected (backlog#3051). - targets: webhook and similar outbound clients drain responses up to a byte limit instead of buffering arbitrary bodies (backlog#3062); the on-disk notification queue has a shared byte budget in addition to the entry count, configurable via RUSTFS_TARGET_STORE_MAX_BYTES (backlog#3122); AMQP and Redis manifests declare the broker url as a secret and bind credentials to their connection fields, and the plugin-instance PUT refuses to restore a redacted secret when the connection fields changed (backlog#3065, backlog#3066). - obs: the debug-level s3s response dump is filtered so STS and service-account credentials never reach the log (backlog#3029). |
||
|
Chris An
|
73e7afa01a |
fix(iam): close identity, session and admin-authorization gaps (#9163)
Scan findings in the IAM crate and the admin identity handlers, grouped because they share the IamCache mutation lock, the STS session sweep and the admin client-address plumbing: - Admin and STS authorization used the socket peer as aws:SourceIp behind a trusted proxy; they now read the verified client address the S3 path already uses (backlog#3025), and the admin usage route passes the bucket named in the path to the s3:ListBucket check (backlog#3033). - AssumeRoleWithWebIdentity accepted HS256 ID tokens MACed with the OIDC client secret; only provider-signed tokens verify (backlog#3028), and JWKS refresh is single-flight with a failure cool-down (backlog#3103). - A disabled group emptied the member's whole policy list, dropping every Deny; direct and other-group policies are preserved (backlog#3042). - Identity rewrites (secret change, add-user, status) ran without a lock and without re-reading the record, so a user's password change could undo an admin disable or reset (backlog#3048). add-user on an existing user and in-flight AssumeRole both left sessions minted under the old secret valid (backlog#3071, backlog#3079); rotation now revokes under the per-key mutation lock and derived-credential publication is ordered with it. - aws:groups never carried a built-in user's IAM group membership and jwt:iss was always the local marker, so group- and issuer-scoped Denies never matched; both now come from live IAM state (backlog#3095, backlog#3096). - OPA mode skipped the account-status gates for service accounts and STS sessions of a disabled user (backlog#3057). - The periodic IAM reload published nothing when any cache write raced it and kept stale partitions when a prefix listed empty, so a missed revocation never applied (backlog#3064, backlog#3114). - update-group-members never evaluated admin:RemoveUserFromGroup (backlog#3098); ExportIAM compared the site-replicator account against a name it never has (backlog#3045); add-user accepted the access key of a live STS session or service account (backlog#3134); a service account could create or extend a lifetime beyond its own (backlog#3112); sts:DurationSeconds and svc:DurationSeconds were never populated (backlog#3136); temporary-account-info returned service accounts (backlog#3137); an access key the object layer refuses no longer holds the service-account lock through the write retries (backlog#3133). |
||
|
Chris An
|
e0f26a540d |
fix(ecstore): gate null-version overwrites under Object Lock (#9157)
A put-like write without a version id to a key that is not versioning-Enabled replaces the key's null version in place. The commit-time Object Lock gate in put_object_with_old_current_size_inner ran only when the write carried an explicit version id, and the request-layer pre-check reads the key's latest version. When the retained null version was not the latest entry (a UUID version or a delete marker sat above it), a user holding only s3:PutObject could overwrite it together with its COMPLIANCE retention or legal hold, and CompleteMultipartUpload had no commit-time gate at all. Name the version a commit replaces (the explicit id, or the nil id when the write is not versioned and the bucket's Object Lock snapshot is not ConfirmedAbsent) and run the existing WORM gate against it, tolerating a null delete marker. complete_multipart_upload gets the same gate at its commit point, and ECStore::handle_complete_multipart_upload attaches the bucket's Object Lock snapshot the way a single PUT already does. Refs rustfs/backlog#3019 (ANT-2026-EH17ZP59). |
||
|
Chris An
|
3b3cc018aa |
fix(policy): close condition, action, principal and resource gaps (#9160)
Eight scan findings in the shared policy engine, each a case where a Deny did not apply or an Allow was broader than written: - A bare `*` Action was parsed as `s3:*`, so `Deny *` never matched admin:, kms: or sts: actions. `Action::All` now denies every family while the historical S3-only Allow is preserved (backlog#3041). - A bare IPv6 address in aws:SourceIp got `/32` appended, matching a 2^96-address network; a bare address now means one host, and IPv4 clients on a dual-stack listener are matched through their mapped form (backlog#3030). - Date conditions on aws:EpochTime parsed the epoch seconds as RFC 3339 and never matched, so a time-bounded Deny failed open (backlog#3036). - NumericNotEquals and DateNotEquals evaluated to false when the key was absent; negated operators now follow the AWS rule and match (backlog#3123). - Every trailing IfExists on a condition operator added a level of recursion without bound; the chain is rejected before parsing (backlog#3091). - Policy variable substitution re-expanded its own output for up to ten passes, so an access key holding k placeholders grew to k^10; values are now substituted once as literal data (backlog#3067). - A bucket-policy Service principal was matched against the signing access key, so a service account named after the service received its grants; Service principals round-trip but never match (backlog#3081). - Resource matching cleaned the trailing slash off a directory object, so a Deny on `prefix/*` missed `prefix/` itself (backlog#3125). Also deduplicate large Resource sets through a HashSet instead of a quadratic scan. |