mirror of
https://github.com/rustfs/rustfs.git
synced 2026-10-28 06:47:21 +00:00
gateway/integration
666 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 overtrue
|
67fbafa13e |
chore(gateway): merge main into integration
Source: main |
||
|
Hauser
|
fea5a584c3 |
fix(storage): fence multipart and metadata recovery (#8919)
* perf(ilm): back off idle tier-delete journal recovery Use bounded idle backoff only after complete empty journal and dispatch scans, while preserving immediate notifications and the retry cadence for observed work. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): reconcile interrupted pool metadata publication Bind pool metadata CAS and Undo to exact native predecessors. Recover an interrupted physical publication only from an ordinary committed authority and complete verified identity, current and backup evidence, then require normal V3 CAS and quorum readback. Preserve known-conflict gates, startup and heal lock ordering, targeted admission and the original kill-recovery deadline. Wake proof-ready recovery immediately and bound parallel capacity sampling without removing mutation fences. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(ecstore): reuse proven pool metadata durability Publish native rollback backups directly in their owned UUID directories. Reuse a one-shot data-durability receipt only for an exact hardlink to the pinned, fully synchronized head and root. Preserve new-head and directory synchronization, native predecessor CAS, conditional Undo and uncertainty gates. Keep unknown, copied, raw-parent, cancelled and failed operations cold. Add bounded diagnostics, sync-call regressions and path-isolated test hooks. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(multipart): exercise selective RPC delay and peer loss Add authenticated method-entry barriers for metadata and lifecycle locks under e2e-test-hooks. Verify real dispatch, write quorum, prior part payloads, lock drain and incarnation isolation without changing production budgets. Record cancellation separately from normal exits so late filesystem completion cannot overwrite Drop evidence. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * docs: define multipart delay gates and receipt invariants Document RPC entry evidence, peer-loss and cancellation oracles, remaining incident attribution limits, and immutable head inodes required for durability receipts. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(readiness): expose missing runtime drive slots Report configured slots without a runtime handle separately from unavailable inventoried drives. Preserve quorum and probe status semantics. Wait for every coordinator to regain its complete live inventory before multipart RPC fault windows and single-attempt recovery oracles. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): retry committed promotion convergence safely Require peer reload convergence on every worker attempt, including unchanged retries. Recheck durable identity, replicas, namespace fence, generation and cancellation before allowing any data movement. Keep known committed promotions active on a verified propagation pause. Release precise owners, retain queued successors and reuse the existing watchdog. Preserve uncertainty gates and terminal operation behavior. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): guard native metadata mutation aliases Reject protected RenamePart source and destination paths before I/O. Classify the protected namespace through the native path resolver, and require canonical conditional RenameData and Undo keys so aliases cannot create separate lease domains. Preserve ordinary multipart and canonical CAS behavior, warm durability receipts and zero-allocation canonical fast paths. Cover real alias, byte identity, receipt and normal mutation behavior. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): register Darwin multipart RPC delay gates Update distributed membership from the actual Darwin listing. Preserve the full-profile digest and leave Linux unchanged until its own listing is produced, as required by the contributor guide. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(multipart): fence native publication and recovery Publish multipart data through private hardlinks and atomically consume the prepared source head. Bind exact compensation capabilities and native recovery to the physical cohort, source generation and durable publication state. Require fresh terminal confirmation for native replay and preserve indeterminate data when no safe recovery witness exists. Release logical guards only after validated irreversible quorum success while retaining physical mutation leases, accounting and resource ownership through actual callbacks. Preserve local instance proof authority and historical Copy metadata without granting healed donors fresh publication authority. Keep general movement identity strict while preserving directory capacity compatibility in a separate checked comparison. Cover native fault and cancellation boundaries, zero-byte objects, real Copy, heal and migration, exact physical readback and distributed RPC delay. Retain all quorum and durability requirements and document rollout limitations. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): fix ratchet scan and Windows fault hooks Move native pool metadata inventory tests after production items so the error-format ratchet sees the existing call site. Preserve the baseline instead of accepting a false count decrease. Make the directory-barrier failure hook available on every test platform and exercise the parent-barrier failure/retry cases in Windows CI. Keep production synchronization behavior and all existing assertions intact. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(protos): remove redundant recovery request clone Move the baseline into its final struct update while retaining the earlier clone needed to construct the first request variant. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): align RPC delay gates with native abort * fix(storage): validate native source recovery and CI gates Preserve wire layouts while bounding native response sizes and fix the last-use recovery request clone. Avoid destination replay reads only for fully observed initial native sources; keep upgraded and recovered sources conservative. Cover real legacy publication after a cold restart. Align upgrade acceptance with the native mixed-version mutation pause and use exact upload-scoped test completion evidence before advancing time. Keep all original quorum, ownership, byte and deadline assertions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): verify native abort under partial admission Delay native cleanup at the exact upload root after all-participant closure. Verify quorum-only admissions retain every source byte on refused Abort and complete the same upload without creating a missing original participant. Keep sub-quorum orphans and their discoverable identities isolated while a fresh all-participant upload proves service recovery. Route shared constants through the existing E2E storage facade. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): avoid formatting a fixed invalid upload path Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): align native MPU fault recovery assertions Verify real source participants, preserved orphan bytes and same-upload completion after quorum-only admission. Require full physical inventories so an expected read-quorum error cannot hide unrelated leakage. Preserve ordinary quorum and recovery budgets, and document listing availability through the affected set. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): keep MPU tail deadlines on the wall clock Global Tokio time advancement can expire drive-health probes while real filesystem callbacks are pending. Use a short real drive deadline with the existing physical-entry and cleanup barriers, retaining the byte, lock, resource-permit and final-shard assertions. Fixes #8937 Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(storage): align native fixtures with instance ownership Open independent disks for the simulated capacity-loss instance, scope four coordinator journal disks without rebinding source authority, and probe configured namespace keys while retaining physical I/O hooks. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): isolate native multipart future construction Construct optional native publication futures behind synchronous heap boundaries so ordinary rename callers do not carry their state machines. Preserve native guards and ownership without adding allocations to ordinary PUT, copy or heal branches. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(heal): assert coordinator journal scope after rejoin Check the exact original four-member journal map after every rejoin and manager stop while preserving all sixteen native data slots. Reconnect expansion fails the test rather than being filtered away. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(heal): bound object repair future construction Construct the normal and FileCorrupt deep-retry state machines behind a synchronous heap boundary. Preserve guards and cancellation while bounding the shared rename and deletion call stacks without increasing test stacks. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): qualify fixture disk identity checks Call the same DiskAPI method explicitly so the independently reopened disk identity assertion compiles under every target and feature combination. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): isolate native inspection from mutation state Select a dedicated read-only inspection future behind the existing single Box allocation. Preserve validation, source ownership and the canonical inspection body while keeping mutation state out of inline read callers. Reject misrouted operations with typed errors and retain source evidence. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): pause abort time after native cleanup Wait for durable source deletion on the wall clock before advancing the capacity lease clock at the post-delete barrier. Paused time can otherwise expire the barrier while native filesystem cleanup is still pending. Retain lease-loss injection, durable inflight-byte accounting and restart reconciliation assertions. Document the test-util feature required to select the native decommission fixtures in package-scoped regressions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): reclaim data from owned multipart undo Retain exact runtime-only Undo ownership through deferred snapshot cleanup. Revalidate every target metadata version and the retirement anchor, keep the delete claim through the physical callback, and remove the anchor after the payload barrier. Preserve generic cleanup refusal when metadata is missing. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): preserve unformatted bootstrap routes Keep explicitly unformatted local paths reachable by an in-flight elected formatter while quarantining their service slots and disk-ID membership. Own bounded legacy-read handles so the actual asynchronous creator future remains Send. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(s3): bound CopyObject tracing poll frames Construct the business future behind an outlined single-Box boundary and lazily instrument its handle. Preserve Send, Unpin, span timing and cancellation without embedding concrete business-state construction in the tracing poll. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(mpu): observe recovery after direct legacy bootstrap Seed all sixteen real previous-release sources before attaching the same-port method observers. Require exact per-peer disk-route Unimplemented responses with zero publication calls, retain byte and cleanup assertions, and remove superseded raw transport. Register the two new regressions from actual Linux and macOS listings. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(mpu): prepare the homogeneous previous-release fleet Reuse the existing release baseline before creating the genuine legacy multipart sources. Restart old processes with their normal budgets and require every drive through the reduced-redundancy probes before installing the mixed-version observers and current fault budgets. Keep standard EC:4, single-attempt calls, all mixed-version byte and reclamation assertions, and their original deadlines unchanged. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(mpu): isolate release compatibility from fault profiles Keep normal component configuration for the observed mixed-release rollout, which injects no RPC delay or loss. Retain the short component profile in actual method-fault scenarios and preserve the single-attempt 45-second probe and every typed error, routing, source byte and reclamation assertion. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(mpu): use normal shutdown for planned release replacements Use the existing bounded SIGTERM path during homogeneous release preparation and the observed mixed-version rollout. Preserve every multipart inventory, routing, quorum and byte assertion, and keep actual crash recovery on its original hard-kill path. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * revert: retain the verified release compatibility fixture Restore the published fixture after the local configuration and lifecycle experiments failed to resolve the macOS startup failures. Keep their diagnostic records separate from the delivered implementation. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(copy): bound inherent handler polling stack Build the inherent CopyObject future through a synchronous noinline factory at the existing trait await. This keeps its large state out of the trait wrapper's active polling frame during historical self-copy. Preserve cancellation, tracing and post-write notifications, with one additional allocation restricted to COPY. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): validate authoritative quota and native pool heals Require actual authoritative quota usage in the existing readiness loop instead of treating HTTP success or unknown usage as a numeric baseline. Keep the original quota assertions and convergence budgets. Allow the owning pool metadata CAS to republish into a fresh valid data directory while retaining exact version, layout, completeness and shard fingerprints. Require every node's real metadata/read/write readiness before accepting recovery. Keep ordinary object manifests and the original 180-second recovery deadline unchanged, with negative oracle regressions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): distinguish certified native Abort recovery Require native participants with a persisted full closure certificate to retire their heads and reclaim every surviving source on retry. Keep uncertified legacy authority below identity quorum conservative, while retaining actual partial-delete and committed-object byte checks. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> Co-authored-by: overtrue <anzhengchao@gmail.com> |
||
|
Hauser
|
c610d11e8f |
fix(storage): guard MPU recovery and startup RPC admission (#8934)
* fix(ecstore): preserve uncertainty after partial multipart abort Keep recovery callers from reporting NoSuchUpload while minority primary metadata remains. Retain all quorum and deletion checks. Exercise actual partial Abort and explicitly retained late RPC packets after caller retirement, staging cleanup and bucket recreation. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): prevent retired multipart staging resurrection Require the upload root to exist before preparing recovery publication. Preserve rooted directory walks for nested bases on non-Linux Unix and cover missing-source native work, configured aliases and pinned mixed-version coordinator recovery. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): use IPv4 peer aliases for pinned mixed recovery Keep direct IPv6 S3 listeners behind IPv4 RPC proxies for the pinned previous release, whose peer health resolver rejects bracketed IPv6 hosts. Preserve signed authorities, ports and strict recovery assertions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): separate pinned startup from MPU fault budgets Use the old fleet defaults during bootstrap, then apply the original short fault budgets to the current replacement. Keep direct S3 URLs aligned with IPv6 listeners and preserve all quorum and physical cleanup assertions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): seed legacy MPU through transparent transport Retire raw proxy connections before enabling method faults so legacy streaming setup stays byte-transparent and every later scoped mutation must traverse the fault barrier. Verify raw and adapted protocol authority, bodies and trailers. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): retain direct listeners for legacy MPU peers Advertise same-port localhost peers for the pinned resolver while retaining direct IPv4 S3 listeners. Exact scoped RPC hit counts still reject bypassed injection and all physical authority assertions remain unchanged. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(server): keep startup gRPC on the authenticated router Use the hybrid protocol predicate for startup RPC admission so early control responses retain production authentication and owner validation. Keep REST FullReady gating and pinned replay capability rejection intact. Add deterministic regressions, scoped diagnostics and bounded legacy fixture resources. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ci): verify MPU recovery across upgrade source versions Run the exact sixteen-drive mixed MPU case for every checksum-verified upgrade source, independently of generic compatibility failures. Require one listed test and retain per-version logs while preserving the historical distributed source pin and every recovery assertion. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ci): trigger upgrade MPU checks for fixture changes Include distributed fixtures and the RPC fault proxy in the upgrade workflow paths so test-only changes schedule the strict recovery gate. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(storage): verify resend owners across MPU finalization Hold real part-transaction RPCs while another coordinator completes or aborts the upload, with both live and cancelled client response waiters. Require full object bytes, terminal upload state, and all staging roots to remain consistent after the part owner drains. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
overtrue
|
8cf6187ff4 | chore(gateway): merge main into integration | ||
|
Chris
|
b39fcb2e85 |
test(heal): cover rc.5 upgrades and refresh ctutils (#8847)
* test(e2e): use the resumed decommission completion budget * test(heal): exercise checkpoints written by rc.5 * test(heal): route upgrade fixtures through the storage facade * fix(deps): update ctutils to address constant-time advisory * test(lifecycle): gate recovery I/O while advancing test time |
||
|
overtrue
|
62b6f5f3fc |
Merge main into gateway/integration
The scheduled sync has stopped on conflicts since 2026-10-19. Resolved by keeping both sides: main's restructured delete-rollback path in disk/local.rs with the integration-only native-publication record, main's per-instance region with the rustfs-s3-types Region, main's new lifecycle, rebalance and admin code ported to the persisted lifecycle types and rustfs-s3-types errors, both sides' new scanner tests, and the union of the access_consumer imports. The e2e smoke, full (darwin) and repl-nightly selections and the test-count baseline are regenerated from the merged tree. |
||
|
Chris
|
b06923beaf |
test(e2e): add scanner/heal topology protection matrix (#8705)
* test(e2e): add scanner/heal topology protection matrix Add the S00 protection baseline for the Scanner/Heal program (rustfs/backlog#2873, program rustfs/backlog#2872): seven scanner_heal_topology_ cases in the distributed e2e lane that pin pool expansion, rebalance and decommission behavior before the runtime changes. The fixture seeds a deterministic inventory (empty, inline, external, multipart, special and nested keys; a versioned bucket with an explicit null version, three history versions and a delete marker per key; an Object Lock bucket with legal hold and COMPLIANCE retention), records every version with its requested and listed version id, length and SHA-256, verifies the manifest through every node, reads physical placement from xl.meta on the pool drives, and reads the strict durability mode back from the admin API and the node logs. Fault cases hold the mover at proven windows through file barriers that compile only with the e2e-test-hooks feature: before the multipart target commit and before source cleanup (shared by decommission and rebalance), plus the existing pool-metadata before_publish cutpoint. An unreached window fails the case. DistCluster gains capture_logs_under so a case can read node startup events without a global environment variable. docs/testing/scanner-heal-topology-gates.md maps the gates to the tests and records the timeout basis. * test(e2e): keep the barrier failure message static The error(format!) ratchet rejects new formatted io::Error::other call sites in ecstore. The reached observation already records the stage and object, so the injected failure needs no per-call detail. * test(e2e): require e2e-test-hooks for topology fault cases The four scanner_heal_topology_ fault cases inject at server barriers that exist only with the e2e-test-hooks feature. Against a default server they waited for the barrier timeout and reported it as not reached. Check the verified binary's features first so the missing prerequisite fails immediately and is named. * ci(e2e): build the distributed lane with e2e-test-hooks Every other e2e lane builds and runs the fault-test hooks server. The distributed lane alone used the default server, so the topology fault cases could never reach their barriers there. Build and run with the same feature so the receipt matches, and update the documented local commands. The upgrade cases spawn the pinned previous release directly and do not compare feature sets. * test(e2e): keep topology artifacts under RUSTFS_E2E_LOG_DIR The topology fixture captured node logs, barrier observations and evidence in a temp dir removed on drop, so the distributed workflow uploaded nothing for these cases. Fall back to RUSTFS_E2E_LOG_DIR, which the workflow uploads, when no explicit evidence directory is set. * test(e2e): remove retired pools before restart; race the pre-ACK start The queue case restarted with decommissioned pools 0 and 1 still on the command line, which the server rightly refuses. Add remove_single_node_pools and restart without them, then expect the remaining pools renumbered to 0 and 1. The pre-ACK case now races the start request against the barrier. A start that returns first, or stays in flight past the documented 60 s budget, is reported as such instead of as an unreached barrier, and the evidence records the reach latency. capture_logs_under now overrides a log path assigned from RUSTFS_E2E_LOG_DIR, so the seed node logs into the case directory too. * test(e2e): bound fault-window waits by the movement budget On Linux CI a busy target capacity gate delayed the first fault object by about two minutes, past the 120 s wait borrowed from the server's release poll. Wait for a movement window within the 240 s movement budget instead, and fail at once when the decommission or rebalance ends without entering the window, so an uninjected fault is never counted. * test(e2e): bound status probes while waiting for a fault window The admin decommission status request can wait on a mover held at a test barrier until the barrier's own 120 s release timeout. The window wait issued that request each loop, so a reached barrier went unnoticed until the server timed it out and failed the decommission, which surfaced as a false resume failure. Bound each status probe to 5 s and treat a slow probe as still running. * test(e2e): retry a busy decommission start A decommission start that meets a busy movement gate can answer a retryable 503 instead of waiting. The harness treated any non-proof failure as fatal. Share one start helper that retries 503 within the case's movement budget and the fleet-proof 500 for 45 s, and keeps every other status fatal. The pre-ACK case retries 503 within its window budget. |
||
|
hector
|
023f2e7258 |
ci(pool): finalize decommission removal and verify surviving pools (#8749)
ci(pool): finalize decommission removal and verify surviving pools (rustfs#8726) Select suite step 10 together with step 9 and pass --finalize-decommission so the nightly exercises the post-removal survivor path: pool ids must stay unique, bucket metadata writes must keep succeeding across a restart, and no node may log 'pool metadata layout changed'. Step 10 self-skips when decommission is disabled. Co-authored-by: Hauser <housemecn@gmail.com> |
||
|
Chris
|
83fb90cb6e |
ci: move API-only and summary jobs to hosted runners (#8707)
Apply the placement rule in .github/workflows/README.md to the 22 jobs still on sm-standard-2 that only call APIs, read workflow metadata, print summaries or scan published images, including cla.yml, the busiest workflow. Moving the ODM cloud-source lane also stops it restoring the ci-odm-interop key across images, since its writer already runs on ubuntu-latest; targets-live gets a -hosted cache key for the same reason. |
||
|
Chris
|
473cddd7dc |
ci: gate pull requests on Clippy and warm its check artifacts (#8708)
Clippy ran only after merge, so lint regressions merged freely and then failed every main run until fixed (10 of 12 main runs on 2026-10-17/18). Run the workspace Clippy step on every event, warm its check-mode artifacts in cache-warm with the identical invocation so pull requests re-check only what they touch, and pin the unconditional step in the ci_gate contract and its self-test. |
||
|
Chris
|
20d9277ec3 |
ci: return pull-request build jobs to 2 after an IO-bound timeout at 3 (#8698)
The first pull-request run at CARGO_BUILD_JOBS=3 that met a contended node (39193827281) hit the 75-minute nextest timeout before any test started. Its sampler shows cargo spinning for 70 minutes with the pod's io.pressure "full" at 86-88% and cpu.pressure at zero, with node-level IO pressure already at 36% when the step began: three concurrent workspace-scale compile units saturate the pod's shared storage, the mechanism the original #5394 mitigation named. Keep 3 on main pushes and dispatches, where three runs succeeded, and 2 on pull requests, in both the nextest and the serial ILM step; pin the ILM budget guard back to that expression and record the evidence in the step comment. |
||
|
Chris
|
90749fbb06 | ci: run feature lanes nightly and warm caches on a schedule (#8683) | ||
|
Chris
|
1ec620ac9c |
fix(ci): restore the release-profile short-credential guard (#8691)
#8682 replaced the cold `cargo check -p rustfs --release` with a `cargo rustc ... -C debug-assertions=off` on the rustfs lib alone. That leaves the dependencies on the dev-profile cfg, so the lib also fails on items gated on one side of cfg(debug_assertions) only (E0599 on Filenames::Embedded), and the step's "fails solely at the compile_error" assertion can never hold; the lane has been red on main and every pull request since. Restore the whole-graph release check, drop the two-job limit that only a linking step needed, and record why the single-crate variant is unsound. |
||
|
Chris
|
392ccadb78 | ci: build development packages nightly instead of on every main push (#8684) | ||
|
Chris
|
4d7cd23f8e |
ci: reuse the dev cache in Connect boundary lanes and raise build jobs (#8682)
Raise CARGO_BUILD_JOBS to 3 on every event in the workspace test lane and record the closed backlog#1601 experiment: the test-profile compile took 28m21s at 2 on a pull request and 16-23 min at 3 across three successful main runs. Merge the two Connect boundary jobs into connect-feature-boundaries. The offline enrollment gate no longer exports a fresh CARGO_TARGET_DIR, so it rebuilds only the rustfs crate on top of the restored ci-dev cache; the E2E root is isolated by the feature gate and the dedicated binary name, not by the target directory. The short-credential release guard is now `cargo rustc -p rustfs --lib --features connect-e2e-short-credentials -- -C debug-assertions=off`, which trips the same cfg(not(debug_assertions)) compile_error against cached dev-profile dependencies instead of a cold release check of the whole graph. Every assertion of both lanes is kept. Update ci_gate.py and the ci-gates.md PR matrix for the merged job. |
||
|
Chris
|
fca83c72e4 |
ci: move compile-free and prebuilt-binary lanes to hosted runners (#8679)
* ci: move compile-free and prebuilt-binary lanes to hosted runners Standard GitHub-hosted runners are free for this public repository, while every sm-standard minute is paid self-hosted capacity. Move the jobs that never compile (quick checks, typos, scope selection, the required-checks aggregate, audit, closed-PR cancellation) and the two s3-tests lanes that only run the prebuilt debug binary to ubuntu-latest, drop the wget install the custom image needed for typos, and move the upgrade-compatibility workflow, whose build shape already runs on hosted runners in e2e-distributed, with renamed cache keys so the hosted and self-hosted images never share a rust-cache entry. Gate the package summary job on resolve having run, so development builds stop starting a pod to report a skipped job. * docs(ci): document runner billing and placement rules Record in .github/workflows/README.md which runner labels GitHub bills (none for this public repository), which ones RustFS pays for, the rule for when a job may leave ubuntu-latest, the rust-cache single-writer and same-image constraints, the patterns that have leaked self-hosted minutes, and a checklist for new or moved jobs. Point to it from .github/AGENTS.md, docs/testing/ci-gates.md and the ci.yml placement comment so the rule has a single home. |
||
|
唐小鸭
|
b40f1e5fab |
fix(utils): harden Win32 volume probes and flag Windows network drives (#8631)
* fix(utils): NUL-terminate and extend paths passed to Win32 volume APIs `get_volume_name` returned the volume root truncated at its NUL and the result was then passed to `GetVolumeInformationW` as a PCWSTR, an out-of-bounds read. The disk-space and volume APIs also received the caller's raw path, so they failed beyond MAX_PATH, and a UNC share root lacked the trailing backslash these APIs require. - Wrap every Win32 path in a NUL-terminated `WidePath` that rejects interior NULs; volume roots returned by `GetVolumePathNameW` keep their terminator. - Convert inputs to the extended-length `\\?\` / `\\?\UNC\` form after `GetFullPathNameW` normalization, and give directory queries a trailing backslash. - Query `GetDiskFreeSpaceW` and `GetVolumeInformationW` on the volume root as documented; size buffers to the documented MAX_PATH + 1 or the input length. - Report Win32 error codes instead of raw HRESULTs so `ErrorKind` matches `std::fs`. - Keep volume ids in their conventional `C:\` / `\\server\share\` form. Add Windows-only `os::` tests for long directories, UNC share roots and non-empty fstype, and run them in the Windows filesystem workflow. * fix(utils): report Windows network volumes to the fs guard Windows names the server's filesystem (usually NTFS) for SMB shares, so the startup unsupported-filesystem guard, which matches fstype names such as cifs or smb2, never fired for UNC paths or mapped drives. - Add `DiskInfo::remote`. On Windows it is set when the directory handle answers `FileRemoteProtocolInfo` (only network redirectors do, and the handle follows symlinks and junctions to their target) or the volume root's `GetDriveTypeW` is `DRIVE_REMOTE`. Unix keeps it false because the client fstype already names network mounts. - Treat remote volumes as unsupported in the startup guard and label them "<fstype> on network share". - Cover UNC shares, paths beyond MAX_PATH below a share, a directory symlink to a share and a `net use` mapped drive with Windows-only `os::` tests that skip when `\\localhost\<drive>$` is unreachable; the Windows filesystem workflow maps the drive before running them. * chore(deps): replace yanked inout 0.2.3 with 0.2.4 cargo-deny's advisories check rejects the yanked inout 0.2.3 that main's lockfile pins, failing every open PR. Lockfile-only patch bump. * fix(utils): detect network targets from the final NT path On the Windows runner, a directory symlink to \\localhost\C$ was reported as local: FileRemoteProtocolInfo did not identify it, and the volume root of the link is a fixed drive. Also treat a handle whose final NT path (GetFinalPathNameByHandleW with VOLUME_NAME_NT) lies under \Device\Mup\ as remote; the Multiple UNC Provider fronts every UNC and mapped-drive target. The tests now assert the handle probe on its own for local and share paths and report both signals when it disagrees. --------- Co-authored-by: overtrue <anzhengchao@gmail.com> |
||
|
唐小鸭
|
276cb24e26 |
fix(ecstore): share cache and lock identity across Windows key aliases (#8635)
* fix(ecstore): share metadata cache identity across Windows key aliases Windows disks treat `\` as a path separator and `path_join_buf` drops the empty segments that leaves, so `a\b`, `a\\b`, `\a\b` and `a\b\` all address the xl.meta of `a/b`. The GetObject metadata cache keyed and hashed the raw spelling, so DELETE `a\b` retired nothing cached for `a/b`: GET/HEAD `a/b` kept returning the deleted object for up to the 2s cache TTL. Fold the object name into the same segment form the disk resolves to before computing the cache hash, key and lookup key. Off Windows the name stays verbatim and borrowed, so `\` remains an ordinary key byte. * fix(ecstore): serialize Windows backslash key aliases on one namespace lock Windows disks treat `\` as a path separator and drop the empty segments it leaves, so `a\b`, `a\\b`, `\a\b` and `a\b\` all address the xl.meta of `a/b`. Namespace locks keyed the raw spelling, so writes, deletes and heals through different spellings ran concurrently on one xl.meta and a read-modify-write in rename_data could drop a version. Fold the object name into its disk identity (rustfs_utils::path:: object_name_disk_identity, a no-op off Windows) wherever a namespace lock key is formed: - SetDisks::new_ns_lock, which every object lock path goes through; - the batch delete lock request and its dedup/lookups, which build ObjectKeys directly. Two consumers compared lock identities by spelling and follow along: - WriteCommitGuard records the folded key from the lock resource, so matching a borrowed guard now folds the target name too; - CopyObject holds the source read guard until EOF on streaming reads. `a\b\` -> `a/b` is not a same-object copy by path_join_buf, but shares one lock, so it now takes the self-copy lock instead of waiting on itself until the lock timeout. Distributed lock keys are folded by the requesting node only; lock servers compare them verbatim. * refactor(ecstore): key the metadata cache by the shared disk identity The metadata cache fix carried its own copy of the Windows backslash folding rule. Use rustfs_utils::path::object_name_disk_identity, which namespace locks already use, so cache and lock identities cannot drift. * ci(windows): run filesystem tests when key identity rules change The Windows backslash folding for object names lives in utils path.rs and is consumed by the namespace lock and write commit guard; run the windows_ tests when any of them changes. |
||
|
唐小鸭
|
776cdc517b |
fix(ecstore): keep Windows disk roots out of verbatim form (#8633)
* fix(ecstore): keep Windows disk roots out of verbatim form The disk root comes from GetFinalPathNameByHandleW, which returns a verbatim path. `simplified` only strips `\\?\C:` below MAX_PATH, so a data directory past 260 characters (or one reached through a UNC share) kept the prefix. Disk IO extends the root by joining `/`-separated relative paths as strings (`read_raw` builds `<object>/xl.meta` with `path_join`), and inside a verbatim path `/` is not a separator: the first metadata read, pool.bin.identity, failed with ERROR_INVALID_NAME until startup gave up. Add `rustfs_utils::non_verbatim`, which strips the disk and UNC verbatim prefixes without a length limit while still keeping paths whose components Win32 would rewrite, and use it for the publication root. std re-adds the extended-length prefix for long paths. Cover a root past MAX_PATH with a disk-level read/write test and run all `dunce::` tests in the Windows filesystem workflow. * fix(ecstore): query Win32 volume APIs with extended long disk roots A disk root past the legacy limit used to reach `rustfs_utils::os` in verbatim form, which the Win32 volume APIs accept. Now that the root stays non-verbatim, GetDiskFreeSpaceExW rejected it with ERROR_PATH_NOT_FOUND and LocalDisk::new failed in the Windows filesystem workflow. Pass long roots to `get_info` and `get_physical_device_ids` in extended-length form; shorter roots stay unchanged so reported volume identities keep their `C:\` form. |
||
|
唐小鸭
|
353e67bbf3 |
fix(ecstore): resolve Windows drive paths in URL endpoints (#8629)
On Windows, URL endpoints such as http://host:9000/C:/data{1...4} (the only form multi-node and pool-expansion layouts accept) produced the disk path `/C:/data1`: `try_from` stripped the synthetic slash, but `Url::set_path` put it back, and `get_file_path` only removed it for `file` endpoints. Windows resolves `/C:/data1` as an invalid `C:` component under the current drive, so disk validation and `resolve_local_disk_root` failed. - `get_file_path` now restores `/X:/...` to `X:/...` for every scheme on Windows; URL paths without a drive (`/export1`, `/C:foo`) are unchanged. - UNC shares (`\\nas\share\d1`, `//nas/share/d1`, `\\?\UNC\...`) are rejected with an explicit error instead of becoming a file URL whose host is the share server and whose path drops the share name. - A drive root (`http://host:9000/D:/`) is accepted in URL form, matching the path-style `D:\` endpoint that was already accepted. - The "distinct physical disks" error explains on Windows that directories on one volume share a device identity, alongside the existing RUSTFS_UNSAFE_BYPASS_DISK_CHECK hint. - The Windows filesystem lane now runs on layout changes and runs the layout::endpoint:: tests; new Windows tests carry the windows_ prefix. - Document Windows endpoint forms, the same-volume check, and the SNSD expansion limit in cluster-lifecycle-operations.md. |
||
|
唐小鸭
|
78ea655d87 |
fix(startup): drive startup on an explicitly sized thread (#8630)
`run_process` called `block_on(async_main())` on the native main thread, so the whole startup chain ran on whatever stack the platform hands the main thread: 8 MiB from `ulimit -s` on Linux and macOS, but the 1 MiB PE default on Windows. A Windows debug build overflowed before it listened. `RUST_MIN_STACK` does not reach the native main thread. Run the runtime on a `rustfs-main` thread with an explicit 8 MiB stack, keeping the budget Linux and macOS already had and making it independent of the linker, toolchain (msvc/gnu) and `ulimit`. Panics are re-raised on the main thread so the exit status is unchanged. Measured on macOS arm64 by lowering the main-thread stack with `ulimit -s` before this change: debug startup overflows at 1792 KiB and reaches /health/ready at 2048 KiB; release overflows at 512 KiB and is ready at 768 KiB. After it, debug starts with a 256 KiB main thread. Add a Windows `Startup Smoke` job that builds the debug server, runs `--version`, and waits for /health/ready on a single volume. |
||
|
唐小鸭
|
5efae5baa6 |
fix(ecstore): unify object key admission for Windows hosts (#8632)
* fix(ecstore): admit CopyObject destinations like PutObject Object-key platform checks were split across call sites with gaps: CopyObject only ran check_bucket_and_object_names on its destination and then reached the pools' put_object directly, so it admitted keys PutObject rejects (over 1024 bytes, leading slash, and on Windows reserved characters or unaddressable segments, the "writes but cannot be read" case of #3449). - rustfs-utils: expose check_windows_file_name as the single per-segment Win32 rule set; dunce's simplified() reuses it. Reserved device names now include COM/LPT with superscript digits and CONIN$/CONOUT$. - ecstore: replace the duplicated reserved-name table with that function; Windows also rejects control characters 0x01-0x1F (NUL keeps its platform-independent ObjectNameInvalid answer). - ecstore: add check_new_object_args as the shared admission entry for PutObject, cross-key CopyObject destinations and CreateMultipartUpload. In-place copies (metadata replace, restore) keep the copy-only check so existing keys stay rewritable. CreateMultipartUpload now also rejects over-NAME_MAX segments up front instead of failing at completion. - CI: run bucket::utils and dunce tests in the Windows filesystem job. * fix(ecstore): reject new backslash keys on Windows hosts A Windows disk treats '\' as a path separator, so a new key `a\b` lands on the same xl.meta as `a/b` and the two keys overwrite each other. Reject '\' only where a key enters the namespace (check_new_object_args: PutObject, cross-key CopyObject destinations, CreateMultipartUpload) and only on Windows. Reads, deletes, in-flight multipart operations, in-place copies, data movement and internal meta buckets keep accepting it, so keys already stored with '\' stay readable, deletable and movable. Pending owner ruling: whether new '\' keys should be rejected at all, and whether reads/deletes should keep accepting them. * ci(windows): quote filtered test runs and drop duplicate path |
||
|
唐小鸭
|
62b6b6149c |
fix(ecstore): publish delete rollback backups past Windows MAX_PATH (#8606)
tempfile's Windows `persist` passes the raw paths to MoveFileExW and SetFileAttributesW, bypassing std's extended-length handling. Once `<object>/<rollback>/xl.meta.bkp` reached 260 UTF-16 units every delete of that key failed with 500, while PUT and GET kept working. Add `rustfs_utils::extended_length_path`, which normalizes a path with GetFullPathNameW and prefixes it with `\\?\` (or `\\?\UNC\`), and use it for the staging directory and destination handed to tempfile. Cover the single and batch delete paths with a deep-key regression test and run the helper's Windows assertions in the Windows filesystem workflow. Fixes #8545 |
||
|
Hauser
|
8c8c74cbc3 |
fix(ecstore): unify bounded delete recovery across upgrades (#8594)
* fix(ecstore): complete delete recovery across upgrades and restarts Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): expect typed quorum error after delete rollback Assert the exact InsufficientWriteQuorum bucket and object instead of the removed Io wrapper, retaining the rollback and cleanup ownership checks. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
overtrue
|
72da0e21c4 |
chore: merge main into gateway/integration
Keep the migrated compatibility adapters in their active modules while bringing in upstream fixes. Preserve native publication history on the new create, rename and borrowed-open paths. Validation: - `cargo nextest run --locked -p rustfs-ecstore --lib -E 'test(disk::os::native_creation_history_tests::) | test(disk::local::test::borrowed_open_cancellation_retains_started_history_before_execution) | test(disk::local::test::owned_direct_child_open_attempt_consumes_native_history)' --no-fail-fast`: 9/9 passed after restoring all 12 mutation cases at cea78f2590645eb0d303cd7cf56f4dd0d9c00e87; these source blobs are unchanged in this merge. - `cargo nextest run --locked -p rustfs-s3select-query --lib -E 'test(dispatcher::json_document_schema::tests::)' --no-fail-fast`: 5/5 passed at 5421ba682d1329cc7ebeaac7c6a863da2cea1996, with the same tree. - `cargo fmt --all --check`: passed. - `./scripts/check_test_count_ratchet.sh --update`: 52 crates, 22007 tests, 165 ignored. - Profile baselines use seven actual Darwin listings and seven actual Linux listings. - Earlier conflict snapshot f5e0cdf0c58aed1ebe53b9243cf52f94832ab049: `cargo nextest run --locked -p rustfs-ecstore -p rustfs --lib -E 'test(disk::os::) | test(bucket::metadata_sys::) | test(server::legacy_compat::) | test(server::layer::tests::request_logging_watchdog) | test(server::http::tests::conditional_object_http::) | test(storage::helper::tests::operation_helper)' --no-fail-fast` passed 218/218 before the borrowed-open fix. This is historical evidence. |
||
|
overtrue
|
f0b584d61a | fix(ci): stop PRs for sync conflicts | ||
|
Hauser
|
20fc897036 |
chore(deps): refresh crates and use Pyroscope 2.1.2 (#8540)
* chore(deps): refresh crates and use Pyroscope 2.1.2 Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * chore(deps): refresh lockfile Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser
|
a86117590c |
fix(ci): prepare pinned Compose for Connect acceptance (#8531)
* fix(ci): prepare pinned Compose for Connect acceptance Install the verified Compose CLI through one local action shared by all 11 connected acceptance entry points. Publish only checksum-verified binaries and keep daemon, provenance and timeout gates unchanged. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ci): use daemon-visible Connect temporary inputs Match the existing official Connect acceptance TMPDIR contract in all 11 real E2E invocations so daemon bind mounts can read generated inputs. Preserve key modes, read-only mounts and cleanup checks. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser
|
ceb53fef85 |
fix(ci): select Python 3.14 for Connect acceptance (#8530)
The shared connected preflight uses uuid.uuid7, which is unavailable in older runner Python environments. Select Python 3.14 with the existing pinned setup action in all eleven acceptance callers. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser
|
c7848df9a0 |
fix(build): enforce GNU ABI floor and canonical target metadata (#8524)
* fix(build): pin gnu glibc baseline to 2.31 and report compile target in version metadata gnu release binaries were built natively on Ubuntu 24.04, embedding GLIBC_2.38/2.39 symbol references while the RPM/DEB metadata promised glibc >= 2.31, so Rocky/RHEL 9 (glibc 2.34) could not run them (#8062). The two linux-gnu legs now build through cargo-zigbuild pinned to the 2.31 baseline (same-arch; zig acts as the linker), with post-build guards asserting both the embedded version metadata and the highest referenced GLIBC symbol version, keeping the package declarations honest. Version metadata itself now comes from cargo's TARGET cfg instead of shadow-rs' host stamping, so builds whose host differs from their target (cross compilation, musl targets on a glibc-hosted toolchain) report the platform they were compiled for (#8048 follow-up). The cross toolchain (zig 0.14.x, cargo-zigbuild 0.23.4) is pinned in the setup action now that it is load-bearing for releases, the build job gets an explicit matrix-independent name so the connect-* acceptance workflows keep matching it when matrix fields change, and the preview-release workflow contract is updated for the TARGET_ARG build commands. Fixes #8062 Follow-up on #8048 * fix(build): enforce GNU ABI floor and canonical target metadata Pin Zig 0.17.0 and cargo-zigbuild, isolate cross-toolchain caches, and build GNU releases against the declared glibc 2.31 floor. Validate both release ELF binaries using their loader version requirements. Generate target metadata once through shadow-rs and check the actual release version output without adding a parallel metadata facade. Refs: rustfs/backlog#2824 Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(build): reuse GNU release profile across binaries Switching the CLI back to the default strip profile restarted 419 compilation units after the server build. Keep the profiling build profile for both binaries, then strip only the CLI artifact. Retain serial links, server profiling symbols, and post-strip ABI gates. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(build): limit release artifacts to the rustfs server Build only rustfs in the release workflow and remove the CLI build, strip, and packaging paths. Preserve GNU profiling symbols and catalog sidecars, and align strict acceptance archive readers with the new set. Exercise all six build targets, Linux package checks, and all four strict acceptance extractors using their actual workflow scripts. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: majinghe <42570491+majinghe@users.noreply.github.com> Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser
|
4502a1f9b0 |
fix(ci): repair profiling and compact transition test gates (#8526)
* fix(deps): patch profiling dependency for RUSTSEC-2026-0344 Vendor the published Pyroscope package with an object 0.40 dependency and lock ruzstd 0.9.1. Preserve runtime sources and profiling features, add bounded decoder regressions, and run the patched package in audit CI. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> Signed-off-by: Hauser <housemecn@gmail.com> * fix(deps): correct vendored pyroscope documentation typos Correct copied comments and changelog spelling for the repository typo check, and describe these comment-only changes in the vendor provenance. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> Signed-off-by: Hauser <housemecn@gmail.com> * fix(ecstore): refresh compact transition proof per test case Install the short-lived homogeneous fleet proof immediately before each kill-point transaction. Earlier cancellation and recovery work must not age the next case's writer capability fixture. Report an early transition result instead of waiting for an unreachable kill point. Preserve cancellation, remote-owner and payload assertions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> Signed-off-by: Hauser <housemecn@gmail.com> --------- Signed-off-by: Hauser <housemecn@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
overtrue
|
cbe38a120f |
Merge main into gateway/integration
Merge frozen main |
||
|
GatewayJ
|
0018bbf5d7 |
ci: guard OIDC architecture boundaries (#8447)
* ci: guard OIDC architecture boundaries * fix(ci): guard OIDC session ownership * fix(ci): ignore OIDC authorization destructuring * fix(ci): accept typed authorization destructuring --------- Co-authored-by: overtrue <anzhengchao@gmail.com> |
||
|
Chris
|
ba2047b184 | ci: reuse workspace test builds for serial ILM checks (#8465) | ||
|
GatewayJ
|
da42266bfb | refactor(iam): organize OIDC runtime modules (#8445) | ||
|
Chris
|
367f13f3f9 | ci: reduce repeated PR runner work (#8463) | ||
|
Hauser
|
25e73d3bd6 |
fix(zip): statically link liblzma on macOS (#8434)
* fix(zip): statically link liblzma on macOS Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(zip): pin macOS liblzma minimum to 0.4.8 Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(admin): route bucket recovery errors through facade Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): group version pagination markers Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(heal): avoid cloning copy purge identity Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
rustfs-cla-assistant[bot]
|
c47432e353 | Merge remote-tracking branch 'origin/main' into gateway/integration | ||
|
Chris
|
1a450bf228 |
ci(e2e): require fresh distributed JUnit reports (#8384)
* ci(e2e): require fresh distributed JUnit reports * test(e2e): emit JUnit from successful workflow mocks |
||
|
Chris
|
f84bbfe28f | feat(server): add the RUSTFS_S3_STACK selector and dual-stack e2e lane (#8411) | ||
|
Chris
|
681c95f359 |
ci: add daily main sync for the gateway integration branch (#8401)
Merge main into gateway/integration every weekday with a --no-ff merge commit; abort a conflicting merge and open a sync/<source>-<date> pull request listing the conflicting files instead. Run ci.yml on pull requests that target gateway/integration so the branch ruleset's required checks can report, and document both in the CI gate matrix. Refs rustfs/backlog#2736, rustfs/backlog#2734 |
||
|
Lindroos Hsu
|
db2abcf0a6 |
feat(http3): add opt-in QUIC transport (#7903)
* build(http3): add optional s3s-http3 dependency Signed-off-by: tommady <tommady@users.noreply.github.com> * feat(server): add opt-in HTTP/3 transport Signed-off-by: tommady <tommady@users.noreply.github.com> * test(server): cover HTTP/3 TLS configuration * feat(server): add opt-in HTTP/3 support Serve S3 requests over QUIC with the existing middleware stack. Signed-off-by: tommady <tommady@users.noreply.github.com> * fix(http3): cancel pending body reads without panicking * test(http3): keep transport independent of s3s --------- Signed-off-by: tommady <tommady@users.noreply.github.com> Co-authored-by: overtrue <anzhengchao@gmail.com> |
||
|
yihong
|
1c5bbc0c47 |
ci: do not run some actions on fork (#8318)
Signed-off-by: yihong0618 <zouzou0208@gmail.com> |
||
|
hector
|
45f97bccf9 |
ci: coerce the pool timeout input to a number
timeout-minutes requires a numeric value, but inputs.pool_timeout_minutes
is a string, so '${{ inputs.pool_timeout_minutes || 240 }}' evaluated to
the string '240' and the runner rejected the template: 'An error occurred
when attempting to determine the step timeout ... Unexpected value 240'
(chain run 36505625402 job at 02:25:52, standalone run 36393453114 at
07:46:37). Both pool runs since #8155 merged have therefore executed
without the intended step budget. Wrap the expression in fromJSON and
update the policy test that pinned the broken literal.
|
||
|
Chris
|
641c4b3493 |
chore(release): merge 1.0.1 back into main and refresh installation references (#8314)
* fix(ci): include pagination regression in full E2E selection * fix(deps): replace yanked yoke-derive release * fix(scanner): expose pause backlog replica diagnostics (#8258) * fix(scanner): expose pause backlog replica diagnostics Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(connect): stabilize runtime profile lease cancellation Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(connect): tolerate delayed schedule startup in CI Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): retry quota reads during usage warmup Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): avoid meta-bucket incarnation self-deadlock Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(test): use persisted incarnation in heal fixture Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(usage): reconcile stale counters after lifecycle expiration (#8108) * fix(usage): reconcile stale counters after lifecycle expiration * fix(usage): account lifecycle expiry during continuous writes * test(usage): run lifecycle usage scenarios on one scanner store * test(usage): use a Windows-representable pre-mutation offset * fix(usage): harden expiry accounting recovery and quota checks Borrow expiry receipt bucket names and avoid allocating a map key on cache hits. Cover cancelled receipts, durable snapshot recovery, and legacy quota admission after scanner confirmation. Use representable timestamp offsets in the quota regression. Refs rustfs/backlog#2689 Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(usage): recover stale persisted counts through the scanner Seed incorrect complete usage for empty and retained-object buckets, then run the real scanner and publication consumer without further object mutations. Verify durable and admin usage over two cycles instead of writing a corrected snapshot in the test. Refs rustfs/backlog#2689 Refs rustfs/backlog#2691 Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): bound delimiter pagination fixture concurrency The 120-second smoke timeout expired after 1018 of 1200 serial fixture PUTs, before LIST ran. Prepare the same objects with at most eight concurrent requests and await every PUT. Retain the timeout and strengthen exact prefix, KeyCount, empty Contents, and continuation-token assertions with phase diagnostics. Refs rustfs/backlog#2689 Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(upgrade): establish a persisted previous-release baseline Seed the pinned previous-release cluster and restart it once with its data intact before replacing any node. Require every old writer to pass the strict readiness probe and preserve the seed through both mixed phases and the final current cluster. Keep InternalError fail-fast behavior and all existing compatibility deadlines and assertions. Refs rustfs/backlog#2689 Refs rustfs/backlog#2384 Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): bound cancellation metadata persistence waits Use the system-bucket incarnation boundary now supplied by main PR #8268. Bound the three cancellation waits that previously hung during pool.bin persistence, retaining their remote-generation, target-cohort, and durable-state assertions. Refs rustfs/backlog#2697 Refs rustfs/backlog#2689 Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: Chris <anzhengchao@gmail.com> Co-authored-by: Hauser <housemecn@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(e2e): retain startup and shutdown failure diagnostics * fix(test): supply CPU workload for sampler regression * fix(ci): locate security chain scripts in the workspace * fix(usage): recover historical counters with generation fencing (#8273) * fix(usage): recover historical counters during continued writes Use newer converged scanner snapshots to reconcile stale absolute usage baselines while preserving concurrent mutation and expiry receipt fences. Cover durable publication, admin and quota reads, and legacy generations. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(usage): fence snapshots and move preserved cache entries Apply the cached scanner generation floor before every reconciliation path and retain it even when an older snapshot happens to match core counts. Move preserved usage entries instead of cloning their histogram maps under the cache lock, retaining expiry receipt identity and cancellation fences. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): verify checkpoint takeover and repair dispatch (#8275) * test(scanner): cover checkpoint handoff and repair dispatch Drive runtime budget expiry, partial-cycle persistence, leadership claims, and stale checkpoint rejection between real disk-backed fixture scans. Verify that a metadata repair beyond the first bounded prefix is saved in the scanner ledger and dispatched by the MRF consumer. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * ci: isolate scanner fixtures and refresh full e2e membership Reserve nextest capacity for the real-disk scanner publication and MRF admission fixtures. Bind both platform membership checks to the reviewed pagination deadline test added on main. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(scanner): consolidate checkpoint fixture lifecycle Keep one durable control store across timeout and leadership transitions, and inject generation advancement into the shared checkpoint scenario. Check the actual saved metadata path so late-write rejection also proves that existing checkpoint bytes remain intact. Centralize MRF fixture isolation and reuse nextest process isolation when the startup environment already satisfies the test contract. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(obs): distinguish allocator counters from live memory (#8274) * fix(obs): distinguish allocator counters from live memory Preserve count/counter semantics and mark requested-byte attribution unavailable when live statistics or sampling are missing. Document sustained multipart memory diagnosis. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(obs): parse allocator statistics from one node Resolve each statistic before interpreting its shape, avoiding unsupported-field tree scans and mixing data across wrapper scopes. Preserve unavailable-statistics policy and add precedence regressions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): isolate late parity recovery from metadata hedges Use the existing object-scoped hedge timer barrier in exact-count recovery fixtures. Preserve payload and total-read assertions and verify that the omitted parity disk is read only during late refresh. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(usage): combine identical snapshot retention branches * fix(test): await HTTP sender readiness in Top RPC fixture * [release/1.0.1] Gate multipart copy through write admission (#8284) Gate multipart copy through write admission Make UploadPartCopy acquire the shared foreground write admission permit before lifecycle locks or source readers so server-side multipart copy cannot bypass the same backpressure used by UploadPart. Document the shared queue semantics and add focused coverage for saturation, cancellation, lock ordering, and disabled admission. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * Gate multipart copy through write admission (#8283) Make UploadPartCopy acquire the shared foreground write admission permit before lifecycle locks or source readers so server-side multipart copy cannot bypass the same backpressure used by UploadPart. Document the shared queue semantics and add focused coverage for saturation, cancellation, lock ordering, and disabled admission. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix: add UploadPart OOM validation guardrails (#8287) * docs(release): validate candidates on release branch * fix(scanner): validate checkpoints against global cycle fence (#8278) ## Related Issues Related to rustfs/backlog#2701. ## Summary of Changes Route scanner checkpoint cycle and leader validation through the global store while retaining the owning set for cache persistence, CAS revisions and publication admission. ## Verification Two independent final-diff source reviews found no issues across correctness, concurrency and durability, test coverage, compatibility, performance and simplicity on head `8b8fe51d092090b053f552ae283960e2e306be33`. Root approval `5373624714` is bound to that exact head. Regression tests cover real two-pool routing, stale fences, post-save rejection and CAS conflicts; their reported local execution belongs to the PR author, not this merge operation. Current required CI remains pending, and this authorized admin squash does not establish CI or runtime acceptance. ## Impact Restores checkpoint progress when global cycle and leader state differ from a set-scoped view. No format, retry, timeout, assertion or scanner-policy changes are introduced by this diff. The three prior main scanner failures remain unproved repaired. ## Additional Notes Full validation must run on the resulting exact main revision. Reverting this patch restores the earlier set-scoped fence lookup and its checkpoint rejection behavior. * fix(ci): restore E2E membership and pagination timeouts (#8281) * ci: locate the auto-testing checkout for lanes that run evidence from a subdirectory (#8279) ## Related Issues Follow-up to #8229. ## Summary of Changes Locate the private auto-testing checkout from the lane root or the workspace root so the nested security checkout can record functional-chain evidence. ## Verification The exact PR head |
||
|
Chris
|
87c1c851c2 | ci: remove Docker Hub description sync (#8303) | ||
|
Chris
|
2806a80c91 |
fix: prevent metadata lock reentry and stabilize CI fixtures (#8257)
* fix(test): establish a writable previous-release upgrade baseline * fix(ci): reserve capacity for durable admin fixtures * fix(ci): group durable IAM state fixtures by resource needs * ci: run E2E doctests with the E2E dependency graph * fix(ci): separate fixture startup from transport deadlines * fix(ci): bound pagination after seeding and revisit restored copies * fix(ci): make filesystem fixture timing deterministic * fix(ecstore): avoid metadata lock reentry during internal mutations * test: align recovery fixtures with durable ownership contracts |
||
|
KarlEmm
|
e573fb087b |
ci(helm): sync README to root of helm repository on release (#8263)
Include helm/rustfs/README.md in the uploaded helm-package artifact so publish-helm-package updates the root README.md in rustfs/helm. The build-helm-package job copies helm/README.md into helm/rustfs/ before running helm package, but upload-artifact previously uploaded only helm/rustfs/*.tgz. Because both files share the helm/rustfs/ parent directory, actions/upload-artifact places both the chart archive and README.md at the artifact root, and download-artifact extracts README.md into the root of rustfs/helm alongside the tarball. |
||
|
Chris
|
9e33d54269 | fix(release): package stable preview tags without updating channels (#8256) | ||
|
Chris
|
a88225d208 |
fix(ci): reduce duplicate work and preserve reliable test failures (#8233)
* fix(ci): reduce duplicate work and preserve reliable test failures * fix(ci): retain protocol evidence and repair stale test fixtures * test(connect): honor parent deadline during API fixture readiness * fix(ci): reserve IO capacity for state writer proofs * test(connect): align RPC fixtures with service capture contracts * test(connect): cover pinned service capture failures |