mirror of
https://github.com/rustfs/rustfs.git
synced 2026-11-20 15:07:34 +00:00
38aa318fb204dc77fbbb901c27a9452c19e33613
1071 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
6e417a458e | test(upgrade): exercise the official 1.0.5 gateway baseline | ||
|
|
5b703b61f9 | test(ci): align storage guard fixtures with the surviving helper rule | ||
|
|
31e0798637 |
refactor(storage): retire the legacy S3 storage edge (#9527)
* test(storage): restore POST form metadata policy fixture * test(storage): follow the current PUT path in contract checks * fix(storage): verify archive transport EOF before parsing the final frame * test(rustfs): restore main verification after gateway migration Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(storage): retire the legacy S3 storage edge * fix(storage): bound the audited multipart copy future and trace LIST quorum causes * fix(deps): align bindgen and prettyplease on syn 3 for FIPS builds * fix(storage): expose the typed legacy cause on owned errors * test(storage): preserve stream lengths and request facts on retired seams --------- Co-authored-by: Hauser Zhi <housemecn@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
|
a52b51b764 | refactor(app): drop s3s entrypoints and move test seams to owned types | ||
|
|
9c0bc265c3 | ci: drop RUSTFS_S3_STACK from reference parity, nextest notes and the connect boundary guard | ||
|
|
74bf3ca5fa | ci(nightly): run the dual-stack nightly on the gateway stack and archive the both-stack scripts | ||
|
|
31c4f4a7ff | ci(acceptance): judge one server and drop the legacy reference run and the stack diff | ||
|
|
3fb8d861f0 | ci(e2e): run the distributed, replication and migration-interop lanes on one stack | ||
|
|
23a2593aac | ci: fold the gateway-stack lanes into the main lint, test and e2e jobs | ||
|
|
c794df64b8 |
refactor(storage): rehome live helpers out of the legacy edge
Rename storage/ecfs_extend.rs to storage/s3_support.rs, point its request-marker imports at storage::request_markers, and move parse_object_version_id and ListObjectUnorderedQuery out of storage/ecfs.rs into it. The module no longer depends on FS, S3Access or storage::access. Behaviour is unchanged. |
||
|
|
9e57f83885 |
feat(storage): add opt-in EC layouts and frame-aware local reads (#9255)
* perf(ecstore): reduce allocations for bounded bitrot frames * test(s3): cover dynamic block layouts over HTTP * perf: measure quantized range read shape * chore: checkpoint quantized read metrics draft Save work in progress for collaboration. Counts omit retired readers and transient or cancelled hedges. Frame semantics, async recorder coverage and disabled-metrics overhead remain unresolved; these counters are not ready for performance conclusions. Validation: rustfmt and ecstore check passed; io-metrics 120, quantized 4 and hashed-read 3 tests passed. No new live or performance validation. * perf(ecstore): reduce dynamic layout read overhead * fix(ecstore): avoid prefetching conservative short range tails * perf(ecstore): compact deferred shard reader state * perf(ecstore): include shard checksums in file buffer cap * fix(ecstore): preserve bounded quantum frames during heal * perf(ecstore): reuse owned quantized frame batches * perf(ecstore): batch q reads and retain shared copy framing * fix(ecstore): preserve typed quantized read errors * perf(ecstore): avoid zero-fill and serial copies in local batch reads The local batch reader zero-initialized every window at a nonzero offset before reading it, and copied every shard serially inside one blocking task. Two-segment q-window range reads (cross-shard, cross-block) pay both costs twice per request. Read each window with a positioned read into uninitialized Vec capacity, and dispatch each shard separately when every window is at least 512 KiB, where copies dominate blocking-pool round trips. Smaller windows keep one coalesced dispatch. * perf(ecstore): keep readahead pipelined for local batch windows A single window-sized pread on cold pages waits on each kernel readahead chunk in turn. Seek to the window start and fill the buffer with read_to_end over a length-limited reader instead: it still reads into uninitialized spare capacity, but its growing sequential reads keep readahead pipelined. Warm windows cost the same. * Revert "perf(ecstore): keep readahead pipelined for local batch windows" This reverts commit |
||
|
|
975e232a63 | chore(deps): refresh dependabot dependency set (#9492) | ||
|
|
d921f3cb24 | perf(ecstore): batch remote rename fanout (#9489) | ||
|
|
75b3dce2e0 | fix(test): preserve parity parameter identifiers | ||
|
|
dc83d478f8 | fix(test): preserve colons in pytest parameter IDs | ||
|
|
f2003af73d |
fix(ecstore): recover deletes on full disks (#9465)
* fix(ecstore): reserve emergency space for delete journals Keep bounded physical metadata headroom for recoverable deletion on Unix without bypassing quorum or rollback. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(ecstore): recover delete metadata space on demand Share a pinned reserve owner per physical disk and keep healthy deletion free of reserve filesystem probes. Retry only disposable staging and idempotent metadata primitives after actual ENOSPC, preserving transaction publication and quorum boundaries. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(ecstore): retry concurrent full-disk deletes Serialize ENOSPC waiters per physical disk and give each safe metadata primitive one bounded retry against shared released capacity. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ci): account for delete reserves in receipt fixture Reject undersized tmpfs mounts after including the eight per-disk delete reserves and keep a self-test for the capacity boundary. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(heal): refresh resume admission fixture lease Refresh the existing node admission sample immediately before starting resume GC so the cancellation test reaches the native-open ownership boundary. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): inject receipt ENOSPC on hardlink path Keep the receipt fault at the atomic allocation boundary after the stage-two hardlink fast path landed, so reserve recovery remains covered. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
|
cbe5b16758 | test(rpc): verify timeout boundary evidence (#9482) | ||
|
|
a33fa8a226 |
fix(rpc): retain cancelled streams and repair CI validation (#9439)
* fix(build): disable cuobject defaults at the workspace level * fix(rpc): retain cancelled streams during metadata read bursts * fix(deps): use published NIXL bindings with scoped advisory policy * fix(test): size multipart refusal fixture threads for real storage * fix(release): wait for uploaded artifact metadata visibility --------- Co-authored-by: Hauser Zhi <housemecn@gmail.com> |
||
|
|
571d100a27 |
perf(ecstore): align inline placement and batch deletion codecs (#9449)
* feat(ecstore): align inline budgets and binary batch decoding Use a fixed 128 KiB per-shard inline budget with versioned admission and preserve historical configuration markers and persisted object layouts. Decode binary delete batches independently of the legacy JSON list while retaining JSON-only requests and dual-write senders. Reject malformed binary entries before disk mutation and preallocate the selected batch. Cover budget boundaries, layout coexistence, MinIO metadata fixtures, binary-only batches and JSON compatibility. Document geometry and rollback boundaries. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): add isolated Docker inline geometry validation Build clean-source Linux validation images from a frozen Git bundle. Pin running image identities and binary hashes, and inspect persisted object geometry through the existing filemeta decoder. Provide an owned four-node Docker adapter for the existing ABBA runner. Guard resource ownership, isolate loopback access, serialize state mutations, and verify HEAD, full GET and Range contents. Keep PUT content probes available through an isolated Linux mc adapter when a host client cannot initialize. Cover ownership and failure paths with script tests wired into the existing runner. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): reuse bounded verified inline bodies Share the materialization contract between metadata pre-verification and buffered inline GETs. Bound responses to 2 MiB and a limited frame count, and reuse an already verified body within the selected snapshot. Route proof-protected parts through the readers that bind PartProofReader. Cover larger single-block bodies, allocation bounds and proof validation after a shard's bitrot checksum has been recomputed. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * feat(ecstore): support bounded multi-block inline objects Keep the default EC block and accumulated shard budget while bounding inline frame storage and verified-body reuse. Preserve transformed and protected write admission and existing metadata formats. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): isolate inline transport comparison modes Pin and verify both MessagePack-only gates for each Docker cell, preserve an explicit JSON compatibility cell and probe exact accumulated shard boundaries. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): cover inline rollback and mixed-reader fleets Add reusable compatibility fixtures for null and versioned objects, cross-block ranges, unfinished multipart uploads and batch version deletion. Record coordinator identity and per-version geometry, preserve retained-data manifests and archive manifests on dataset resets. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(ecstore): unify bounded inline read and write paths Share checked frame decoding across metadata preverification and GET, resolve write admission once, and use one checked physical shard length for inline and disk writers. Preserve proof-bound readers and erasure recovery for damaged or missing shards. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(testing): persist inline compatibility mutation receipts Save successful multipart and delete receipts before fallible content and geometry checks. Resume completed mutations through verification without repeating writes, and cover read failures, geometry failures, and command retries. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(gateway): isolate capability probes from multipart listings Give successful capability probes their own buckets so CreateMultipartUpload does not change the exact two-upload pagination fixtures. Preserve all listing-boundary assertions on both HTTP stacks. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(rpc): honor fleet-confirmed codecs for batch deletion Build DeleteVersions requests with one codec policy for the complete batch. Emit JSON compatibility fields by default and binary-only batches after both existing fleet gates pass. Cover request fields, canonical binding and actual remote sender, handler and disk deletion in both modes. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(ecstore): verify inline shards through borrowed frames Share borrowed frame verification across buffered GET, bitrot readers and deep inline healing. Validate exact physical framing before scanning memory slices. Cover short prefixes, legacy padded frames, damage and trailing data, and provide a reproducible manual ABBA kernel comparison. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ci): account for manual inline verification benchmark Raise the ECStore ignored-test ceiling from 2 to 3 for the opt-in ABBA comparison of borrowed and streaming bitrot verification. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): reject inexact inline shard framing Validate the complete encoded length of every selected inline data shard before decoding or caching a verified object body. Return to the erasure reader on missing, truncated, or trailing payload bytes. Extend decoder fixtures across single and multi-block geometry, legacy padding, unhashed payloads, and explicit bitrot verification bypass. Cover metadata early-stop rejection and GET recovery for trailing shard bytes. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(ecstore): share inline shard reader construction Build owned-metadata and borrowed-metadata readers through one Send iterator-based factory. Preserve shard ordering, empty slots, bitrot settings and caller-bound integrity verification. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(rpc): move endpoint buffers into batch delete requests Transfer the prepared endpoint String into the protobuf request and assert allocation identity across every batch codec mode. This removes the intermediate endpoint copy during request construction. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(gateway): remove superseded pagination isolation Use the upstream capability probes that clean up their own multipart sessions. Run probes against the seeded pagination bucket while keeping all exact page and cursor assertions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(table-catalog): recognize borrowed multipart guard calls Count owned and borrowed bucket/key guard forms while requiring all four production calls. Cover mixed split-file calls and reject wrong resources, missing error propagation and test-only mentions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): size heal cleanup fixtures above inline budgets Use an environment-independent per-pool fixture policy and payload sizes above the aggregate inline budget. Assert external placement before pausing old-data cleanup, rebuilding the committed generation and checking physical shards and the final payload. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(table-catalog): count borrowed object metadata guards Recognize the borrowed bucket/key guards in migrated legal-hold and retention entrypoints. Validate two object and two key guards and cover incorrect resources, missing propagation and test-only mentions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
|
fb9f19c6d2 |
fix(rpc): scope canary errors to grpc failures (#9453)
* fix(rpc): scope canary errors to grpc failures Exclude expected read-version remote responses without hiding transport, gRPC status, response-decode, or other gRPC operation failures. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(rpc): verify canary revisions from image history Use immutable image history only when recognized config labels are absent, keep mismatches fail-closed, and record the provenance source in verifier output. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(audit): restore bucket config audits and metadata guards Audit accelerate, request-payment, and website configuration mutations through the shared owned request scope, including failures and the website deletion status. Validate the replication mutation delegation before checking peer reload notifications. Count borrowed reserved-prefix guard calls and cover missing production calls, test-only mentions, audit completion, and stale bucket incarnations. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: zhi22915 <qiuzgang@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
|
f4266a4576 |
refactor(protocols): use owned object reads and deletion
Route both GetObject calls, HeadObject and DeleteObject to the owned use cases. Keep the request facts and millisecond timestamp behavior. PutObject stays unchanged; 12 operations and 13 trait calls remain. The client edge is an interface in the layer guard. Its public re-export is covered too, and app-to-client imports are rejected by fixtures. Runtime wiring: +177/-101 non-test lines. The new matrix has 18 negative and 13 positive operation cases, plus the extended identity-copy control. Validation: 28 protocol tests passed in both default and gateway-stack modes; both all-target Clippy modes passed. Script tests, layer controls and focused architecture/footprint/test guards passed. Five source faults failed at the intended claims, range, Head ETag, deletion-witness and version assertions. The precision oracles failed before the millisecond projection was added. Runtime and Clippy ran on e9da1d309; initial controls and faults ran on |
||
|
|
d0fd6425e4 |
feat(rpc): guard stale generations before dispatch (#9435)
* perf(grpc): attribute rebuilds and bound bulk responses Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(grpc): attribute rebuild success after publication Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * feat(obs): add exact HTTP/2 ping timeout diagnostics Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(multipart): run refusal matrix on large stack Run the real-store completion refusal matrix through the shared 32 MiB test harness so its deep async storage future does not overflow libtest's default stack. Keep the original test name and all refusal-order assertions intact. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * chore(deps): pin nixl-sys workspace source Manage the nixl-sys Git revision from the workspace dependency table and point the optional Linux transport dependency at houseme/nixl commit cc636c83f13d692e85b67e55aed092413d16fb41. Allow the reviewed source in cargo-deny and refresh the lockfile, replacing the unmaintained bincode dependency with the fork's wincode graph. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * feat(rpc): authenticate msgpack peer capability Add an optional boot-epoch-bound HMAC capability proof and cache the verified per-peer observation without enabling any msgpack send policy. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * feat(grpc): bind peer capability to connection generations (#9433) * feat(grpc): track physical connection generations Observe every tonic physical reconnect through a stable custom connector while preserving existing TCP, TLS, HTTP/2, cache, and public channel behavior. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * feat(rpc): bind msgpack capability to connections Scope authenticated msgpack observations to channel identity and physical connection generation so transparent reconnects require a fresh proof without enabling the send policy. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: zhi22915 <qiuzgang@gmail.com> * feat(rpc): guard stale generations before dispatch Reject generation-bound requests before inner transport call or first payload frame when tonic readiness observes a different physical connection, while leaving ordinary mutations unguarded until the send-policy follow-up. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * feat(rpc): add per-peer msgpack rollout gate Add the default-off peer-capability environment gate, cached evaluation, fixed-cardinality send-decision metrics, and rollout constraints for stable peer-affine connections. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * feat(rpc): apply per-peer msgpack send policy Capture one capability decision per eligible RPC, bind peer-mode requests to boot and connection generations, and rebuild at most one dual-write request only after a locally proven pre-dispatch rejection. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(rpc): verify msgpack peer canary evidence Add a strict three-phase four-node evidence verifier with raw Prometheus and workload receipts, adversarial tests, script-test wiring, and the operational manifest contract. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(rpc): align batch read manifest encoder Match the BatchReadVersion compatibility manifest to the request-owned encoder expression used by the production send path. Restore the owning ECStore source guard while preserving wire and runtime behavior. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(rpc): orchestrate msgpack peer canary Run the fixed mixed/all-new/rollback four-node matrix with prebuilt images, real workload and readback, raw Prometheus capture, strict manifest generation, and adversarial fake integration tests. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(rpc): verify canary image and telemetry identities * test(rpc): bind canary evidence provenance Scope canary metrics to one injected cluster identity and bind candidate, baseline, and per-phase running containers to immutable image IDs and exact build revisions. Reject foreign-cluster metrics, stale revision labels, same-image tags, and phase image mismatches. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(table-catalog): accept borrowed multipart guard calls Count the exact borrowed bucket/key guard call introduced by UploadPart target ownership while preserving the four-production-call contract. Cover the borrowed spelling in both the positive fixture and the test-only false-positive regression. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> Co-authored-by: overtrue <anzhengchao@gmail.com> |
||
|
|
3cee9c154b |
perf(grpc): attribute rebuilds and bound bulk responses (#9420)
* perf(grpc): attribute rebuilds and bound bulk responses Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(grpc): attribute rebuild success after publication Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * feat(obs): add exact HTTP/2 ping timeout diagnostics Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(multipart): run refusal matrix on large stack Run the real-store completion refusal matrix through the shared 32 MiB test harness so its deep async storage future does not overflow libtest's default stack. Keep the original test name and all refusal-order assertions intact. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * chore(deps): pin nixl-sys workspace source Manage the nixl-sys Git revision from the workspace dependency table and point the optional Linux transport dependency at houseme/nixl commit cc636c83f13d692e85b67e55aed092413d16fb41. Allow the reviewed source in cargo-deny and refresh the lockfile, replacing the unmaintained bincode dependency with the fork's wincode graph. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * feat(rpc): authenticate msgpack peer capability Add an optional boot-epoch-bound HMAC capability proof and cache the verified per-peer observation without enabling any msgpack send policy. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * feat(grpc): bind peer capability to connection generations (#9433) * feat(grpc): track physical connection generations Observe every tonic physical reconnect through a stable custom connector while preserving existing TCP, TLS, HTTP/2, cache, and public channel behavior. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * feat(rpc): bind msgpack capability to connections Scope authenticated msgpack observations to channel identity and physical connection generation so transparent reconnects require a fresh proof without enabling the send policy. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
|
ba9e8342f8 |
test(storage): add fail-closed quiescence gate (#9436)
* test(storage): add fail-closed quiescence gate Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(storage): gate benchmark rounds on quiescence Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(bench): bound quiescence Docker calls Start the storage-quiescence deadline before Docker discovery, pass the remaining budget to every subprocess, and reject samples completed after the deadline. Cover delayed initial lookup and delayed exec through the supported Docker binary override. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: zhi22915 <qiuzgang@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
|
a41019802d |
test(storage): expose direct-write validation controls (#9430)
Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
|
0567078f5b |
feat(gateway): serve GetObject through the owned read path
Register GetObject and share its authorization and native response facts
with the legacy edge. Preserve conditional headers, range completion,
stream errors, body ownership, SSE reads and request accounting. Pin the
gateway 304 response-override fix without unrelated dependency changes.
Keep the incoming native Put path and update its profiling guard source.
Validation on
|
||
|
|
fd8dc41f0c |
feat(ecstore): add file-sync durability tier (#9422)
* refactor(ecstore): split durability sync capabilities Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(ecstore): classify durability barriers Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * feat(ecstore): add file-sync durability tier Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(storage): harden durability docker validation Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(storage): isolate durability PUT benchmarks Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(storage): expose file sync grouping controls Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
|
925e5d0c0c |
feat(gateway): serve HeadObject through native response facts
Share HEAD authorization and its use case across both request edges. Keep
stored and proxy ETag parsing distinct, preserve conditional headers and
version bytes, and finish auditing before terminal header encoding. Pin
gateway to 1afff6a for the measured 304 response profile.
Verified 785 affected and embedded tests, then 12 tests after test-only
module/import cleanup. Both all-target Clippy configurations, script tests,
wiring and prepush checks passed. Tests cover malformed stored tags,
separate replication denial, conditional status conflicts and ordinary
success controls; the proxy oracle checks 779 spellings.
Observed 39 effective isolated faults: authorization/delegation, execution,
metadata exposure, stored/legacy/proxy tags, raw header escaping/repetition,
framing conflicts, length/counting, range boundaries, checksum and storage
class, ordinary completion, terminal code/status/default resolution, and
each of the seven 304 fields. An ineffective fallback mutation was retained
as a failed experiment and replaced with an explicit-status-only fault.
The 785-test base was c13232996; final module tests and Clippy ran on
|
||
|
|
0eec25e4c0 |
fix(ci): compile gateway features in nightly matrix legs
Select gateway-stack for replication and protocol gateway legs and use matching build and run features. Legacy feature selections remain unchanged.
Verified on
|
||
|
|
f2a43376de |
test: adapt owned entrypoint guard fixtures
Follow DeleteObject's typed and legacy entrypoints through the shared fields adapter into its owned implementation. Preserve the existing mutation guards and key/prefix/store ordering checks.
Let the TRACE mutation probe locate either ListObjectsV2 parameter shape exactly once, then require the real logging guard to reject DEBUG.
Validation on
|
||
|
|
0fde0f14a0 |
Revert "fix(ecstore): isolate unavailable local drives during startup (#9353)"
This reverts commit |
||
|
|
f6d64bc1fb |
fix(ecstore): isolate unavailable local drives during startup (#9353)
* fix(ecstore): isolate unavailable local drives during startup Keep missing local endpoints in their configured slots and initialize healthy local drives before applying the existing format and service quorums. Protect unformatted Linux URL endpoints from writes to the system filesystem, validate legacy admission, and retry missing local slots during format waits. Refs rustfs/rustfs#9350 Refs rustfs/backlog#3174 Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(ecstore): prepare local drives before registry publication Prepare local disks with bounded concurrency outside inventory locks and publish validated batches under per-instance writer coordination. Preserve existing registrations on failed or cancelled full initialization, reuse current-root handles, and withdraw stale reopened slots. Unify initial-format and reconnect publication, retain raw and explicit-context constructor contracts, and move filesystem queries into owned native jobs. Centralize format validation and reject special format files before startup writes. Refs rustfs/rustfs#9350 Refs rustfs/backlog#3174 Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * chore(ecstore): tighten formatted-error ratchet after refactor Drop the three removed runtime registration call sites from the shrink-only baseline. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test: align drive fixtures with startup admission Expect the disk-scoped missing-root error and enable the existing directory test mode for the synthetic URL-based receiver fixture. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(ecstore): unify local drive admission and publication Share bounded preparation and pre-I/O membership snapshots across local startup and retry paths, with coherent registry and topology publication. Coalesce native startup preflight and preserve root and buffer ownership. Keep reconnect ID caches instance-scoped and retain admitted health monitors. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): pin multipart index queue aliases to owner lifetime Derive the Linux index alias from the existing identity descriptor so a retired caller descriptor cannot redirect rebuild or reconnect queue lookup. Retain the original descriptor assertion and cover cross-disk alias reuse. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
|
c43e415b09 |
feat(server): serve ListBuckets and HeadBucket on the gateway stack (#9296)
The walking skeleton of the gateway edge, behind the gateway-stack cargo feature and RUSTFS_S3_STACK=gateway. rustfs-gateway inside the RustFS binary serves ListBuckets and HeadBucket end to end and answers every other S3 operation 501 NotImplemented. A default build carries neither the stack selection nor the wrapper: its request path is unchanged. - One per-operation unit: impl Served for dto::Op with a metric label, an authorize without a default body, and an execute. The only impl Handler is the blanket one over Served (access phase, authorize, count, execute, one error mapping). authorize calls the function the legacy access hook calls. - One table laid out per family (ten family files named after the migration batches); it registers the handlers and is the whole early gate. - A shared access phase used by both edges, fed on the gateway edge from the gateway's verdict. - Stored bucket names are materialised under the gateway floor only, so names the AWS grammar refuses are listed as before; HeadBucket's legacy edge builds its input the same way until the gateway re-pin (rustfs/gateway#1610). - With the gateway selected the legacy service receives exactly what the legacy router claims (extension routes, object lambda, control plane). - A start-up line names the stack the mounted service answers with. - The release-boundary guard reads feature closures from cargo metadata. Squashed from gw/T2.5-gateway-stack at |
||
|
|
ef79dcbe19 |
ci: conformance and client-matrix lanes against both S3 stacks
Acceptance lanes T3.4 (rustfs/backlog#2775) and T3.5 (rustfs/backlog#2776): .github/workflows/gateway-acceptance.yml runs the rustfs/gateway conformance corpus (--profile rustfs) and client matrix (ci/compat/run_matrix.sh --external) against the candidate binary on RUSTFS_S3_STACK=legacy and =gateway, one server per stack. The suites are checked out at the gateway rev Cargo.toml pins. Nightly schedule and workflow_dispatch only. - Each stack must be observed: the start step writes `ready` only when /health/ready answers while its own process is alive and the server's server_starting event names the requested stack (scripts/dual_stack.py stack); otherwise `did not start: ...`. Summaries print the observed stack. - Verdicts come from files the run leaves on disk (startup record, stack evidence, liveness after the corpus, exit codes, report.json, matrix.json), never from an earlier step's exit status. - Conformance: the rustfs profile describes the gateway edge, so only the gateway stack is judged (scripts/check_conformance_rulings.py against .config/conformance-rulings.toml). Red: an unruled failure, a ruled case failing differently from its ruling's `failure`, a stale ruling, and a skip on the server's own answer (endpoint not answering, refused fixture). The legacy run is an unjudged reference: it must serve, stay up and complete. - Client matrix (scripts/compat_matrix_report.py): per stack, red on a `fail` or `sut-unregistered` cell, a client that could not be installed, or an unfinished matrix; legacy is the baseline and every legacy pass must pass on gateway. - The gateway leg and the stack comparison are continue-on-error, report-only until T2.10. Not built: no re-pin (write cases need a gateway rev with --allow-external-writes plus that flag on the lane's run line); no legacy rulings; no use of the runner's own --rulings ledger (stale rulings do not block there); no call to the gateway's check_compat_matrix.sh (it does not fail `fail` cells or clients that never ran); no JUnit (the pinned runner writes a raw U+0001); no install cache; toolchain versions copied from the gateway's client-matrix.yml at 7915d069. Tests: --self-test of both checkers in make script-tests, 40 tests, 31 negative. Mutations, each red: unruled failure ignored (test_an_unruled_failure_is_red); stale only when passed (test_a_ruling_on_a_skipped_or_absent_case_is_stale); did-not-start green (test_a_server_that_did_not_start_is_red_and_says_why); swallowed runner exit (test_a_swallowed_exit_cannot_hide_a_failed_case); exit 3 accepted (test_an_environment_exit_is_red); legacy judged or tallied (test_the_legacy_reference_is_counted_never_judged); dead server green (test_a_server_that_died_during_the_corpus_is_red); endpoint and fixture skips not red (test_a_skip_on_the_endpoint_not_answering_is_red, test_a_skip_on_a_refused_fixture_is_red); ruling on id only (test_a_ruled_case_failing_differently_is_red); unknown verdict (test_an_unknown_verdict_is_refused); stack not compared (test_a_server_of_the_other_stack_is_red); no stack evidence (test_ready_without_stack_evidence_is_red); failed cell, skipped client, swallowed matrix exit (test_a_failed_cell_is_red, test_a_client_that_could_not_be_installed_is_skipped_never_passed, test_a_missing_manifest_is_red_even_when_the_exit_was_swallowed); superset gate (test_a_legacy_pass_lost_or_absent_on_gateway_is_red, test_a_gateway_superset_of_the_baseline_is_clean_and_its_extra_passes_are_listed, test_diff_without_both_manifests_is_red); diffs (test_diff_lists_changed_and_absent_cases_only, test_diff_without_both_reports_compares_nothing). Trial runs on rustfs/rustfs#9368: 43264289490, 43325101234. Gateway stack did not start (gateway stack is not assembled yet, T2.5); legacy reference completed, 140 of 890 cases executable against an external endpoint; legacy matrix 13 x 14: pass 121, fail 0, client-unsupported 61, sut-unregistered 0. Not verified: the stack observation and liveness record in CI (exercised locally under bash -eo pipefail with a stand-in server and main's dual_stack.py, both directions); a scheduled run; a diff with gateway content. |
||
|
|
5a26862e2e |
ci: build macOS and Windows on a weekly schedule (#9286)
* ci: build macOS and Windows on a weekly schedule Keep the 01:13 UTC cron on Linux and add a Sunday 16:00 UTC cron for macOS and Windows. Select that subset from the firing cron so a delayed runner cannot guess the weekday. Co-authored-by: hector <majinghe@users.noreply.github.com> * ci: move the weekly desktop cron off minute zero --------- Co-authored-by: hector <majinghe@users.noreply.github.com> Co-authored-by: overtrue <anzhengchao@gmail.com> |
||
|
|
b8efcb8dfa |
feat(release): package manual Linux builds from a git tag (#9332)
* feat(release): package manual Linux builds from a git tag Let operators name an existing tag, branch, or commit. An empty ref still packages the tip of origin/main, and a tag can supply the package version by dropping one leading v. Co-authored-by: hector <majinghe@users.noreply.github.com> * fix(release): confirm historical tags with the tooling helper The confirm step was calling the script from the tagged commit, which exits 127 when that commit predates the script and cannot confirm when it only has an older copy. Use a helper that implements confirm-source. Co-authored-by: hector <majinghe@users.noreply.github.com> --------- Co-authored-by: hector <majinghe@users.noreply.github.com> Co-authored-by: overtrue <anzhengchao@gmail.com> |
||
|
|
704b984c1f |
ci: both-stack e2e diff in the source workflows and a strict mint stack gate
T3.1/T3.3 (rustfs/backlog#2772, #2774). The existing nightly lanes get a stack dimension and their legacy leg is the legacy side: e2e-replication-nightly (repl, cluster, protocols), e2e-distributed and the ODM minio-source job. A gateway leg runs scripts/dual_stack.py preflight after its build and stops there when no gateway start is proven; the summary says 'skipped: gateway stack refuses startup: <why>'. One summary job per workflow diffs the legs of the same run with junit_diff.py. dual-stack-nightly.yml covers only e2e-smoke and e2e-full: one build, both stacks from that binary, gateway legs only when the preflight proved a start. junit_diff.py reports a missing, empty or case-less side as 'no result for <label>'. mint.yml stays weekly, builds the image once and runs it per stack; mint_stack_diff.py gates that every case legacy passes is present and passing on gateway with no new failure (missing, NA, unknown status and truncated lines never pass). Gateway legs and the compare job are report-only until T2.10. Not built: a shared-binary preflight job in the three source workflows (one preflight step per gateway leg); harness-only filtering for their five profiles; mc/Vault/iptables in the e2e-full leg; an s3s-e2e both-stack run; rustfs_gateway=debug. Tests: dual_stack 30 (preflight with fake binaries), mint_stack_diff 16, junit_diff 27; negatives outnumber positives. Mutations (red test named) are in the PR 9367 description. Not verified: the workflows have not run in CI in this shape (composites, stack matrices, image artifact round trip, needs result of a red continue-on-error leg). |
||
|
|
cb75405d8c |
ci: log the selected S3 stack at start-up and prove it from server logs
The both-stack lanes need the stack a server actually selected, not the variable a workflow set. The start-up line gains the s3_stack field (below the stack match, so a refused gateway request never logs it), and scripts/dual_stack.py stack reads that line from server logs: it fails on a server of the other stack, on no server of the requested stack, and on no server output; older-release servers count as unstamped. Not built: the preflight and summary subcommands, the workflows and the mint gate. Tests: 10 in scripts/test_dual_stack.py (7 negative, 3 positive), in make script-tests. Mutation: ignoring foreign-stack starts turns test_a_server_of_the_other_stack_fails_the_proof and test_the_other_direction_fails_too red. Not verified: clippy with --features gateway-stack; rustfs nextest. |
||
|
|
1d47ad5b61 |
ci: let a push pass when the s3s footprint is below its baseline (#9387)
Without --base, a footprint counter below its baseline line passes and prints the lowered line; a counter above it still fails. Pull requests and merge groups are still held to their base revision, so the footprint cannot rise. |
||
|
|
10103c824f |
ci: trim the merge-group proof and keep a proven plan honest (#9381)
Cleanups from the review of the merge-group proof change: the proof lookup no longer special-cases the event in the workflow, a proven plan carries connect=false and full_validation=false and the plan validator requires it, two redundant input guards are removed, plan contract 7. |
||
|
|
9e3d4cbcbb |
chore(app): localize use-case imports and add slice seams (#9372)
Preparation for the parallel operation slices of the s3s to gateway migration. No behaviour change. - Use-case modules import the s3s names they use instead of reaching them through a glob from the parent. - Per-slice test retirement tables and scripts/check_test_retirements.py. - Per-slice extension points in the dual-stack differential script. - The s3s footprint check compares a pull request with its base revision; the baseline file is a ceiling there. |
||
|
|
b42f444166 |
ci: stop rebuilding what the cache already built (#9371)
* ci: stop rebuilding what the cache already built cache-warm.yml built `--workspace --all-targets`, a selection no ci-dev reader builds. Feature resolution differs per selection, so the test archive recompiled 123 external units and the E2E archive about 290 on every exact cache hit. warm-ci-dev now runs the readers' own builds: `ci_test_artifact.py cargo warm` (the Cargo build of `nextest archive`) for both archive configurations, the hooks server and the lint command. Rust-cache never saves over an exact key match, so the warm jobs skip their builds after one; ci_gate.py --check-workflow pins both rules. The manual shard rehearsal ran only after a full execution, saved nothing, and could not run since #9071 renamed the archive it downloads. It is removed with its plan field, jobs, script and tests; the plan contract moves to 5. * ci: key ci-dev by its warm commands and close the warm guard's gaps The ci-dev key now ends in a digest of warm-ci-dev's build steps, so the reader selections this branch warms are written under a new key instead of being skipped as an exact hit of the old one. ci_gate.py --check-workflow derives the digest, requires the writer and every reader in ci.yml and reference-parity.yml to name that key, and prints the key a change needs. The presence check now reads only the steps that run whenever the key is written, so a selection moved into the on-request --timings step no longer counts. Configuration names are compared as sets in both directions and accept either option spelling. The plan contract moves to 6: #9369 already used 5 for a different plan. |
||
|
|
70525a9698 |
feat(ecstore): expose fsync worker policy experiments (#9357)
* feat(ecstore): expose fsync worker policy experiments Add independent idle-spin controls and gated worker diagnostics for controlled performance comparisons. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(perf): forward Docker telemetry in ABBA runs Validate node mappings and record per-leg container resources for isolated fsync policy experiments. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(perf): reject baseline drift in either direction Apply absolute delta budgets to A2 baseline checks while preserving directional candidate checks and their exemptions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(perf): stop ABBA after a load driver failure Preserve the load driver exit status through command substitution and stop before later deployment legs while retaining cleanup traps. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
|
6b27fa4756 |
ci: accept the proving merge-group run as release evidence (#9376)
* ci: accept the proving merge-group run as release evidence A commit that lands through the merge queue gets a push run of ci.yml in mode proven: Quick Checks and Test and Lint succeed, and every validation job, End-to-End Tests (full merge gate) included, is skipped. source_ci refused such a commit, so no release candidate could be cut from it. source_ci now accepts that push run when Quick Checks and Test and Lint succeeded and a successful merge_group run of ci.yml on that branch's queue has the same head tree and passed all three required jobs itself; that run is recorded as the candidate's exact-source CI identity. The match is ci_merge_group_proof.proves, the same predicate the push plan uses. Also: a test that a full merge_group run requires every job a full integration push requires; the sibling queue-branch case (gh-readonly-queue/mainline/...); select_plan's proof is keyword-only; the release, CI gate and workflow docs describe proven pushes. * test(release): pin the proving run's position, attempt and branch The release test now lists the matching merge-group run among a run of another tree and a run of another branch's queue, in both orders; gives it attempt 3 and reads jobs by run and attempt; and accepts a release branch's own queue run while refusing a main queue run for it. |
||
|
|
4a34e512cd |
ci: skip the push run for a tree the merge queue already proved (#9369)
A merge-queue landing fast-forwards main to the group's commit, so the push run re-validates a tree a successful merge_group run of ci.yml has just passed. Select CI scope now looks that run up with the job's read-only Actions token (scripts/ci_merge_group_proof.py) and plans mode `proven`: Select CI scope, Typos and Quick Checks run, every validation job is skipped, and Test and Lint requires exactly that. The plan names the proving run, its not-applicable reasons and the aggregate's summary say the jobs were skipped as proven. Any doubt runs everything: no token, an API or network error, a rate limit, an unexpected payload, no successful queue run of the same tree, or a proof on another event. A direct push has no such run. The plan contract moves to 5. |
||
|
|
5f460c2383 |
perf(grpc): optimize peer pools and unary RPC observability (#9355)
* perf(protos): balance peer channels and synchronize TLS eviction Use bounded per-peer bulk cursors and shared TLS generation reads. Keep eviction and generation removal coherent with concurrent publication, and expose opt-in bulk transport tuning with checked capacities. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * feat(metrics): observe internode unary request lifetimes Record client attempts through errors and cancellation, preserve the published server-duration family, and add gated activity and completion metrics. Cache enabled instruments after recorder publication while retaining startup duration recording. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(perf): make internode gRPC experiments reproducible Reset phase-owned settings before applying overrides, separate lifecycle experiments, and provide a bounded loopback workload with independently reported control and bulk latency. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(perf): isolate transport knobs and redact dry runs Add single-parameter P0 presets with separate artifacts while preserving the legacy combined default. Mask credential arguments in dry-run output without changing actual dispatch, and document fleet stop barriers and measurement boundaries. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * feat(observability): qualify gRPC windows and trace shutdown Reject stale, incomplete or maintenance-active measurement evidence using producer timestamps and independent process identities. Preserve raw inputs, bind evidence hashes, and exercise failure paths through the script entry point. Record HTTP/2 graceful, child-join and TLS reloader retirement phases without changing shared deadlines, join ownership or unclean-shutdown marker policy. Document RPC coverage, fixed-window behavior and separate maintenance workloads. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(perf): add authenticated ReadAll and fixed-rate Ping probe Exercise production remote-disk and signed control clients against task-owned temporary fixtures. Preserve all offered control arrivals, dispatch delay, payload validation and bounded raw request samples, with optional external measurement markers. Register probe regressions with the default package test discovery and document its telemetry, retry and storage-acceptance boundaries. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(perf): resolve RPC probe disks by explicit UUID Allow a complete non-nil disk UUID mapping to be installed through the existing remote-disk client API before fixture preflight. This supports test services that register local paths instead of the probe's transport URLs, without modifying server metadata. Validate mapping shape and assert the actual wire target in the loopback regression. Keep preflight diagnostics limited to the numeric disk-error classification. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * docs(perf): reseed temporary RPC fixtures after restart Separate startup-cleaned RPC temporary files from confirmed S3 durability fixtures. Revalidate readiness and disk identity before reseeding the probe after recovery. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(server): cancel unfinished TLS handshakes during shutdown Cancel pre-request TLS acceptance when the listener retires so silent peers cannot retain graceful watchers beyond the shared drain budget. Keep established TLS/H2 response draining and original child joins intact. Add real TCP/TLS shutdown regressions and document fresh native OTLP observation fences and exec-confirmed diagnostic test units. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(grpc): reuse bulk keys and fix IPv6 TLS verification Reuse bounded per-peer slot keys and retain the existing cache identities for larger pools. Delegate custom-CA certificate names to tonic's URI parser and cover real IPv6 control/bulk TLS connections with a wrong-name rejection regression. Classify ReadVersion failure boundaries while preserving error totals, raise the bounded raw probe budget, and align the Linux-only rename test helper with its caller's platform condition. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(grpc): keep eviction race behind cache helpers Pause eviction on the owned TLS-generation lock and inspect the channel through the common cache API. Assert lock ordering while both mutations are queued, then retain the replacement-generation assertion after they complete. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(grpc): load native trust and preserve mTLS client identity Select explicit private trust or platform roots, then attach the published client identity for either mode. Build the TLS connector on the blocking pool under the existing acquisition deadline. Cover real control/bulk TLS connections, required client authentication, private-root scope and queued configuration timeout without late connection establishment. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
|
55d9bf08a2 |
ci: run workspace clippy beside the test binary build (#9345)
Clippy ran in front of the test-binary archive inside Workspace Lint and Build, so every workspace test waited 13 to 17 minutes for a lint that shares no compiler output with the archive. Move the unchanged Clippy step into its own workspace-clippy job with the same condition, needs, runner and cache key. Workspace Lint and Build keeps its ID and name as the archive producer. The aggregate and the scheduled alert depend on the new job, and the plan contract moves to 4 so a base planner that predates the job set falls back to the full matrix. ci_gate.py now rejects a workflow where the lint is conditional, duplicated, outside the aggregate, scoped differently from the archive producer, or back on the archive consumers' dependency chain, and where the workflow's plan-contract marker differs from the planner's. |
||
|
|
cf3e359794 |
refactor(storage): split impl S3 for FS bodies into legacy_edge files (#9319)
* feat(server): add the RUSTFS_S3_STACK selector and dual-stack e2e lane (#8411) * refactor(server): group the s3s-compat tower layers under legacy_compat (#8410) Move the ten tower layers that patch s3s behaviour out of rustfs/src/server/layer.rs into rustfs/src/server/legacy_compat/, one file per layer with its tests. Each file header names the s3s behaviour it patches, the gateway mechanism that replaces it (Replaced by:), and the task that removes it (Removed by: T2.12, rustfs/backlog#2771). The layers keep their code, their order in external_service_stack!, and their behaviour; no layer is deleted. Add server::http::tests::stack_census, which renders the ordered service chain of the real external_service_stack! expansion and compares it with a committed constant, so a reorder, insertion or removal is a visible diff. Route the two s3s error types the moved layers use through a storage_api::server::legacy_compat shim so the split adds no s3s-importing file, and lower the s3s footprint baselines to the measured values (files 209 to 208, s3_error! lines 1586 to 1585). Keep the server runtime-source bypass guard covering the moved code. Refs rustfs/backlog#2787, rustfs/backlog#2734 * feat(s3-types): add the RustFS-owned S3 error type and s3_error! macro (#8406) `rustfs-s3-types` now provides `S3ErrorCode` (generated by build.rs from src/codes.txt, the grep-derived call surface, plus `Custom`), `S3Error`, `S3Result`, `StdError` and the `s3_error` macro with the same call syntax as s3s, so the T1.2 codemod is a `use` swap. Statuses are hand-written and pinned to the s3s behaviour RustFS ships today by an oracle test that uses s3s as a dev-dependency only. The `From` conversions for the legacy edge live behind the off-by-default `compat-s3s` feature (the orphan rule forbids them in rustfs/src/error.rs); `rustfs` enables it. Refs rustfs/backlog#2742, rustfs/backlog#2734 * ci: add the per-crate test-count ratchet (#8404) Quick Checks now fails any PR that lowers the number of test attributes in a workspace crate or adds #[ignore], measured per crate against the exact rows of .config/test-count-baseline.txt (rustfs/backlog#2734 section 1 rule 3). The counter widens the spec's literal #[(tokio::)?test] to the tokio attributes that carry arguments and both spellings of #[test_case(...)], and anchors at line start so commented-out attributes do not count. The initial baseline covers 52 crates, 21077 tests and 164 ignored. Refs rustfs/backlog#2738 Refs rustfs/backlog#2734 * test(ecstore): pin the persisted bucket-config XML bytes for 13 families (#8408) Before any DTO or codec is replaced, pin the exact bytes that deserialize followed by serialize produces for every persisted bucket-configuration XML family in its three source shapes (AWS documentation, MinIO-written, RustFS-written), and the bytes RustFS writes today for a typed value. Byte equality only: a structural comparison would let a codec replacement change the persisted encoding unnoticed. The generator script refuses to overwrite existing goldens without --force, refuses on dirty codec paths, cannot report success from its write mode, and records the generating commit in the fixtures README. Refs rustfs/backlog#2744, rustfs/backlog#2734 * ci: raise the test-count baseline to the integration tree (#8426) * ci: make the s3s footprint baseline exact and lower-only (#8403) * ci: make the s3s footprint baseline exact and lower-only The s3s footprint guard compared three counters against ceilings held in the script itself, so a ceiling could sit above the tree and a drift was fixable by raising it (rustfs/rustfs#5964 raised the s3_error! ceiling to go green). The three values now live in .config/s3s-footprint-baseline.txt as files, s3_error_lines and ecstore_files, the script reads only that file, and a measurement that differs from its baseline in either direction fails: growth prints "s3s footprint grew: <measured> > <baseline>", shrinkage prints "lower the baseline to <measured> in this PR". The counting method is unchanged; the baseline is the current measurement of gateway/integration (209 / 1585 / 36). The header records the end state: once all three reach 0, T4.2 switches the script to failing on any occurrence. Refs rustfs/backlog#2739, rustfs/backlog#2734 * ci: add the s3s edge allowlist mode to the footprint guard (#8405) `scripts/check_s3s_footprint.sh --mode allowlist` fails on every Rust file that references s3s (`s3s::` or `s3s_sigv4`) and is not matched by an entry of the new `.config/s3s-edge-allowlist.txt`; `--dry-run` lists those files and exits 0. The allowlist carries the initial edge from the spec, and the script rejects an entry that names or matches anything under crates/ other than crates/e2e_test. The baseline mode and the argument-less CI invocation are unchanged. Refs rustfs/backlog#2741, rustfs/backlog#2734 * test(ecstore): reach the bucket-config codec through the test facade (#8428) * refactor(s3-types): own the HTTP Body and drop s3s from three crates (#8431) Add rustfs_s3_types::Body, a BoxBody<Bytes, StdError> wrapper with empty() and From<Bytes | Vec<u8> | String | &'static str>, and the legacy From conversions to and from s3s::Body behind compat-s3s. The signer is now generic over the request body: it only reads and writes the request head, so it needs no body type and its callers keep theirs. s3-client sends its requests with the owned Body, madmin signs a bodiless head, and targets drops its two S3Request/S3Response header wrappers in favour of extract_params_header. rustfs-signer, rustfs-madmin and rustfs-targets no longer depend on s3s; the s3s footprint baseline drops from 209 to 202 files. Refs rustfs/backlog#2747, rustfs/backlog#2734 * refactor(ecstore): move S3ErrorCode to rustfs-s3-types by codemod (#8432) * test(ecstore): pin the InvalidRange responses of new_getobjectreader Both error branches of new_getobjectreader build an ErrorResponse with S3ErrorCode::InvalidRange, and no test observed either code. Pin the code, its wire name and the location fields of each branch before the code type moves to rustfs-s3-types (rustfs/backlog#2743), so a changed code on either branch fails a test. The ecstore test-count row rises by the two tests. * chore(scripts): add the s3s-to-rustfs-s3-types error codemod Task T1.2 of rustfs/backlog#2734 (rustfs/backlog#2743) moves S3Error, S3ErrorCode, S3Result and s3_error! from s3s to rustfs-s3-types by script, one crate closure per PR. The codemod rewrites use trees (splitting mixed s3s imports) and qualified code paths, never comments or literals, adds the rustfs-s3-types dependency, drops s3s only from a crate that no longer names it, refuses every file of the s3s edge allowlist, takes --hold for a file whose type is fixed outside the closure, and prints a per-crate census. A second run changes nothing. Its self-test runs under make script-tests. * refactor(ecstore): take S3ErrorCode from rustfs-s3-types Output of scripts/codemods/s3_error_to_s3_types.py over crates/ecstore, crates/lifecycle, crates/scanner and crates/replication (rustfs/backlog#2743 batches 1 and 2). The ErrorResponse code field, the two InvalidRange constructions in new_getobjectreader and an unused tier import now use rustfs_s3_types::S3ErrorCode; the variant names and wire names are unchanged. lifecycle, scanner and replication reference none of the four error names. warm_backend.rs and warm_backend_s3.rs are held: they compare against rustfs_s3_client::ErrorResponse.code, which stays the s3s type until s3-client moves. * ci(footprint): count only s3_error! lines still on s3s's macro s3_error_lines counted every line holding the text `s3_error!`, which also matches the RustFS-owned `rustfs_s3_types::s3_error!`. A codemod that moves call sites onto rustfs-s3-types therefore could not lower it (rustfs/backlog#2743, decision 3; rustfs/rustfs#8432, gotcha 2). The counter now applies a textual, per-file and per-line rule that errs on the side of counting (full rule in the script header): - occurrences spelled `rustfs_s3_types::s3_error!` never count; - a line holding `s3s::s3_error!` always counts; - any other `s3_error!` line counts unless its file is cleared, that is, it imports the macro from rustfs_s3_types by name on a line starting with the use item (or defines it with `macro_rules! s3_error`) and does not also bring s3s's macro in (`use s3s::s3_error`, a `use s3s::{..}` tree naming it or a crate-root `*`, `use s3s::*`, or `#[macro_use] extern crate s3s`); - fail closed: a file matching neither form (`use super::*`, a crate-local re-export, a renamed or glob rustfs_s3_types import, a commented-out import, a comment-only mention) counts. Census of the 1585 lines on |
||
|
|
da6058e37b |
feat(storage): support per-instance internode data transports (#9257)
* feat(storage): support per-instance internode data transports * perf(rpc): avoid staging and client cache overhead on TCP writes |
||
|
|
06eca12064 |
fix(docker): preserve local pool arguments during expansion (#9352)
* fix(docker): preserve local pool arguments during expansion * refactor(docker): use one path expansion for volume arguments |