mirror of
https://github.com/rustfs/rustfs.git
synced 2026-11-18 14:05:27 +00:00
10cb203c76290c8b95cd90c94004c19bbfd5fd15
149 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
c43e415b09 |
feat(server): serve ListBuckets and HeadBucket on the gateway stack (#9296)
The walking skeleton of the gateway edge, behind the gateway-stack cargo feature and RUSTFS_S3_STACK=gateway. rustfs-gateway inside the RustFS binary serves ListBuckets and HeadBucket end to end and answers every other S3 operation 501 NotImplemented. A default build carries neither the stack selection nor the wrapper: its request path is unchanged. - One per-operation unit: impl Served for dto::Op with a metric label, an authorize without a default body, and an execute. The only impl Handler is the blanket one over Served (access phase, authorize, count, execute, one error mapping). authorize calls the function the legacy access hook calls. - One table laid out per family (ten family files named after the migration batches); it registers the handlers and is the whole early gate. - A shared access phase used by both edges, fed on the gateway edge from the gateway's verdict. - Stored bucket names are materialised under the gateway floor only, so names the AWS grammar refuses are listed as before; HeadBucket's legacy edge builds its input the same way until the gateway re-pin (rustfs/gateway#1610). - With the gateway selected the legacy service receives exactly what the legacy router claims (extension routes, object lambda, control plane). - A start-up line names the stack the mounted service answers with. - The release-boundary guard reads feature closures from cargo metadata. Squashed from gw/T2.5-gateway-stack at |
||
|
|
ef79dcbe19 |
ci: conformance and client-matrix lanes against both S3 stacks
Acceptance lanes T3.4 (rustfs/backlog#2775) and T3.5 (rustfs/backlog#2776): .github/workflows/gateway-acceptance.yml runs the rustfs/gateway conformance corpus (--profile rustfs) and client matrix (ci/compat/run_matrix.sh --external) against the candidate binary on RUSTFS_S3_STACK=legacy and =gateway, one server per stack. The suites are checked out at the gateway rev Cargo.toml pins. Nightly schedule and workflow_dispatch only. - Each stack must be observed: the start step writes `ready` only when /health/ready answers while its own process is alive and the server's server_starting event names the requested stack (scripts/dual_stack.py stack); otherwise `did not start: ...`. Summaries print the observed stack. - Verdicts come from files the run leaves on disk (startup record, stack evidence, liveness after the corpus, exit codes, report.json, matrix.json), never from an earlier step's exit status. - Conformance: the rustfs profile describes the gateway edge, so only the gateway stack is judged (scripts/check_conformance_rulings.py against .config/conformance-rulings.toml). Red: an unruled failure, a ruled case failing differently from its ruling's `failure`, a stale ruling, and a skip on the server's own answer (endpoint not answering, refused fixture). The legacy run is an unjudged reference: it must serve, stay up and complete. - Client matrix (scripts/compat_matrix_report.py): per stack, red on a `fail` or `sut-unregistered` cell, a client that could not be installed, or an unfinished matrix; legacy is the baseline and every legacy pass must pass on gateway. - The gateway leg and the stack comparison are continue-on-error, report-only until T2.10. Not built: no re-pin (write cases need a gateway rev with --allow-external-writes plus that flag on the lane's run line); no legacy rulings; no use of the runner's own --rulings ledger (stale rulings do not block there); no call to the gateway's check_compat_matrix.sh (it does not fail `fail` cells or clients that never ran); no JUnit (the pinned runner writes a raw U+0001); no install cache; toolchain versions copied from the gateway's client-matrix.yml at 7915d069. Tests: --self-test of both checkers in make script-tests, 40 tests, 31 negative. Mutations, each red: unruled failure ignored (test_an_unruled_failure_is_red); stale only when passed (test_a_ruling_on_a_skipped_or_absent_case_is_stale); did-not-start green (test_a_server_that_did_not_start_is_red_and_says_why); swallowed runner exit (test_a_swallowed_exit_cannot_hide_a_failed_case); exit 3 accepted (test_an_environment_exit_is_red); legacy judged or tallied (test_the_legacy_reference_is_counted_never_judged); dead server green (test_a_server_that_died_during_the_corpus_is_red); endpoint and fixture skips not red (test_a_skip_on_the_endpoint_not_answering_is_red, test_a_skip_on_a_refused_fixture_is_red); ruling on id only (test_a_ruled_case_failing_differently_is_red); unknown verdict (test_an_unknown_verdict_is_refused); stack not compared (test_a_server_of_the_other_stack_is_red); no stack evidence (test_ready_without_stack_evidence_is_red); failed cell, skipped client, swallowed matrix exit (test_a_failed_cell_is_red, test_a_client_that_could_not_be_installed_is_skipped_never_passed, test_a_missing_manifest_is_red_even_when_the_exit_was_swallowed); superset gate (test_a_legacy_pass_lost_or_absent_on_gateway_is_red, test_a_gateway_superset_of_the_baseline_is_clean_and_its_extra_passes_are_listed, test_diff_without_both_manifests_is_red); diffs (test_diff_lists_changed_and_absent_cases_only, test_diff_without_both_reports_compares_nothing). Trial runs on rustfs/rustfs#9368: 43264289490, 43325101234. Gateway stack did not start (gateway stack is not assembled yet, T2.5); legacy reference completed, 140 of 890 cases executable against an external endpoint; legacy matrix 13 x 14: pass 121, fail 0, client-unsupported 61, sut-unregistered 0. Not verified: the stack observation and liveness record in CI (exercised locally under bash -eo pipefail with a stand-in server and main's dual_stack.py, both directions); a scheduled run; a diff with gateway content. |
||
|
|
5a26862e2e |
ci: build macOS and Windows on a weekly schedule (#9286)
* ci: build macOS and Windows on a weekly schedule Keep the 01:13 UTC cron on Linux and add a Sunday 16:00 UTC cron for macOS and Windows. Select that subset from the firing cron so a delayed runner cannot guess the weekday. Co-authored-by: hector <majinghe@users.noreply.github.com> * ci: move the weekly desktop cron off minute zero --------- Co-authored-by: hector <majinghe@users.noreply.github.com> Co-authored-by: overtrue <anzhengchao@gmail.com> |
||
|
|
704b984c1f |
ci: both-stack e2e diff in the source workflows and a strict mint stack gate
T3.1/T3.3 (rustfs/backlog#2772, #2774). The existing nightly lanes get a stack dimension and their legacy leg is the legacy side: e2e-replication-nightly (repl, cluster, protocols), e2e-distributed and the ODM minio-source job. A gateway leg runs scripts/dual_stack.py preflight after its build and stops there when no gateway start is proven; the summary says 'skipped: gateway stack refuses startup: <why>'. One summary job per workflow diffs the legs of the same run with junit_diff.py. dual-stack-nightly.yml covers only e2e-smoke and e2e-full: one build, both stacks from that binary, gateway legs only when the preflight proved a start. junit_diff.py reports a missing, empty or case-less side as 'no result for <label>'. mint.yml stays weekly, builds the image once and runs it per stack; mint_stack_diff.py gates that every case legacy passes is present and passing on gateway with no new failure (missing, NA, unknown status and truncated lines never pass). Gateway legs and the compare job are report-only until T2.10. Not built: a shared-binary preflight job in the three source workflows (one preflight step per gateway leg); harness-only filtering for their five profiles; mc/Vault/iptables in the e2e-full leg; an s3s-e2e both-stack run; rustfs_gateway=debug. Tests: dual_stack 30 (preflight with fake binaries), mint_stack_diff 16, junit_diff 27; negatives outnumber positives. Mutations (red test named) are in the PR 9367 description. Not verified: the workflows have not run in CI in this shape (composites, stack matrices, image artifact round trip, needs result of a red continue-on-error leg). |
||
|
|
b42f444166 |
ci: stop rebuilding what the cache already built (#9371)
* ci: stop rebuilding what the cache already built cache-warm.yml built `--workspace --all-targets`, a selection no ci-dev reader builds. Feature resolution differs per selection, so the test archive recompiled 123 external units and the E2E archive about 290 on every exact cache hit. warm-ci-dev now runs the readers' own builds: `ci_test_artifact.py cargo warm` (the Cargo build of `nextest archive`) for both archive configurations, the hooks server and the lint command. Rust-cache never saves over an exact key match, so the warm jobs skip their builds after one; ci_gate.py --check-workflow pins both rules. The manual shard rehearsal ran only after a full execution, saved nothing, and could not run since #9071 renamed the archive it downloads. It is removed with its plan field, jobs, script and tests; the plan contract moves to 5. * ci: key ci-dev by its warm commands and close the warm guard's gaps The ci-dev key now ends in a digest of warm-ci-dev's build steps, so the reader selections this branch warms are written under a new key instead of being skipped as an exact hit of the old one. ci_gate.py --check-workflow derives the digest, requires the writer and every reader in ci.yml and reference-parity.yml to name that key, and prints the key a change needs. The presence check now reads only the steps that run whenever the key is written, so a selection moved into the on-request --timings step no longer counts. Configuration names are compared as sets in both directions and accept either option spelling. The plan contract moves to 6: #9369 already used 5 for a different plan. |
||
|
|
6b27fa4756 |
ci: accept the proving merge-group run as release evidence (#9376)
* ci: accept the proving merge-group run as release evidence A commit that lands through the merge queue gets a push run of ci.yml in mode proven: Quick Checks and Test and Lint succeed, and every validation job, End-to-End Tests (full merge gate) included, is skipped. source_ci refused such a commit, so no release candidate could be cut from it. source_ci now accepts that push run when Quick Checks and Test and Lint succeeded and a successful merge_group run of ci.yml on that branch's queue has the same head tree and passed all three required jobs itself; that run is recorded as the candidate's exact-source CI identity. The match is ci_merge_group_proof.proves, the same predicate the push plan uses. Also: a test that a full merge_group run requires every job a full integration push requires; the sibling queue-branch case (gh-readonly-queue/mainline/...); select_plan's proof is keyword-only; the release, CI gate and workflow docs describe proven pushes. * test(release): pin the proving run's position, attempt and branch The release test now lists the matching merge-group run among a run of another tree and a run of another branch's queue, in both orders; gives it attempt 3 and reads jobs by run and attempt; and accepts a release branch's own queue run while refusing a main queue run for it. |
||
|
|
4a34e512cd |
ci: skip the push run for a tree the merge queue already proved (#9369)
A merge-queue landing fast-forwards main to the group's commit, so the push run re-validates a tree a successful merge_group run of ci.yml has just passed. Select CI scope now looks that run up with the job's read-only Actions token (scripts/ci_merge_group_proof.py) and plans mode `proven`: Select CI scope, Typos and Quick Checks run, every validation job is skipped, and Test and Lint requires exactly that. The plan names the proving run, its not-applicable reasons and the aggregate's summary say the jobs were skipped as proven. Any doubt runs everything: no token, an API or network error, a rate limit, an unexpected payload, no successful queue run of the same tree, or a proof on another event. A direct push has no such run. The plan contract moves to 5. |
||
|
|
55d9bf08a2 |
ci: run workspace clippy beside the test binary build (#9345)
Clippy ran in front of the test-binary archive inside Workspace Lint and Build, so every workspace test waited 13 to 17 minutes for a lint that shares no compiler output with the archive. Move the unchanged Clippy step into its own workspace-clippy job with the same condition, needs, runner and cache key. Workspace Lint and Build keeps its ID and name as the archive producer. The aggregate and the scheduled alert depend on the new job, and the plan contract moves to 4 so a base planner that predates the job set falls back to the full matrix. ci_gate.py now rejects a workflow where the lint is conditional, duplicated, outside the aggregate, scoped differently from the archive producer, or back on the archive consumers' dependency chain, and where the workflow's plan-contract marker differs from the planner's. |
||
|
|
83ce509ff3 |
ci: run the gateway smoke lane only where the gateway stack is touched (#9344)
* ci: run the gateway smoke lane only where the gateway stack is touched The report-only `e2e-smoke (gateway, report-only until T2.10)` lane ran on every pull request and failed on every one of them: the gateway S3 stack does not serve the whole e2e-smoke suite before T2.10. A pull request that touches none of the lane's inputs can neither cause nor fix that failure, yet it was shown a red check and occupied an sm-standard-2 runner. Select CI scope now publishes one more output, `gateway_smoke`, computed by `scripts/ci_gate.py select` from the change set: true when a path in GATEWAY_SMOKE_PATHS changed or a `rustfs-gateway*` pin in Cargo.toml or Cargo.lock differs, and on every event that is not a pull request. The lane is skipped only on a pull request whose selection is the literal `false`; a missing or refused selection runs it. Job-level continue-on-error stays, and no step may carry it, so a failed suite is never rendered green. `ci_gate.py --check-workflow` pins the condition by truth table: the lane must run wherever End-to-End Tests runs except on unselected pull requests, must need classify-changes, and the output must be published. * ci: keep the e2e suite's own files out of the gateway smoke selection Feature pull requests add e2e tests and touch the smoke selection all the time, and until T2.10 the gateway lane cannot pass on any of them, so selecting on `crates/e2e_test/*`, `.config/nextest.toml` and `.config/e2e-smoke-selection.txt` kept producing the red check this change exists to remove. What the suite contains carries no signal while the stack cannot serve it. The gateway code paths, the pin comparison, the workflow, the test-runtime action and the two scripts stay. An e2e-only pull request being skipped is now a pinned self-test case. At T2.10 the selection is deleted, not extended. |
||
|
|
31e1e42ad7 |
refactor: land the s3s decoupling groundwork on main (#9276)
Lands gateway/integration on main: the guard rails and the domain decoupling for replacing s3s with rustfs/gateway (rustfs/backlog#2734, Phase 0 and Phase 1). The S3 edge is still s3s and no wire behaviour change is intended. - Guard rails: lower-only s3s footprint baseline and edge allow-list, per-crate test-count floor, 39 byte-level goldens for the persisted bucket-config XML families, a two-server differential harness over all 73 S3 operations, a JUnit differ, and a reference-parity lane against the sha256-pinned 1.0.5 binary. - Owned types: rustfs-s3-types (S3Error, S3ErrorCode, Body), RequestEnvelope, AuthzFacts. - Engine crates off s3s: ecstore, lifecycle, replication, scanner, protocols, s3select, s3-client, rpc. - Admin: error types moved; 300 of 314 Operation impls take the owned AdminRequest. - Server: the s3s-compat tower layers grouped under server/legacy_compat; RUSTFS_S3_STACK=legacy|gateway selector, where gateway refuses to start until the gateway edge exists. Fixes made on the branch that change ecstore and heal behaviour are listed in the pull request description. The per-commit history stays on gateway/integration. |
||
|
|
95e128336d |
ci: narrow checks for stable installation metadata (#9269)
* ci: narrow checks for stable installation metadata * fix: reject release metadata marker collisions |
||
|
|
134820c1e0 |
test(storage): reproduce HTTPS rejoin readiness in Docker (#9206)
* test(storage): reproduce HTTPS rejoin readiness in Docker Compare cluster-only and node-ready/write gates on immutable native binaries. Preserve first-failure and latest-acknowledged-content evidence across recovery, capture lock diagnostics before restart, and limit cleanup to verified owned resources. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(testing): unify Docker repro ownership and controls Use typed resource absence and one verified container inventory for staging, partial setup and retained nodes. Reject inherited experiment overrides before creating resources and remove duplicated state. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(testing): guard staging volumes and interrupted capture Verify volume ownership before privileged staging and disable implicit image-content copying. Preserve failed evidence and continue owned finalization when capture or cleanup is interrupted. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(testing): capture live rejoin logs and verify trace coverage Read live tmpfs logs inside the verified container and use one explicit stdout sink for diagnostics. Record bounded capture provenance and fail diagnostic coverage when acquisition, guard-drop or physical-release records are missing, without masking HTTP failures. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(testing): reject inherited aliases and finalize on SIGTERM Reject inherited MINIO compatibility settings before Docker resource creation. Route SIGTERM through interrupt-aware finalization and restore the previous handler after each run. Cover real process signals and failure preservation. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(testing): normalize legacy controls and termination Reject inherited MINIO compatibility aliases in neutral runtime images. Bridge run-mode SIGTERM into the existing interruption finalization and restore the caller signal handler on return or exception. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
|
158aa276e9 |
fix(storage): bound heal retries and recover pool metadata (#9164)
* fix(ecstore): retry metadata writes rejected before staging * fix(ecstore): reuse registered disks during startup * fix(ecstore): retain remote health monitors during reconnect * test(heal): release the commit barrier during graceful shutdown * test(e2e): register graceful shutdown barrier regressions * fix(ecstore): roll back interrupted pool metadata commits * test(upgrade): honor the 1.0.4 delete recovery protocol * fix(heal): bound admission during object rate pressure * test(e2e): handle rebalance completion before stop * test(e2e): observe rebalance activity before the start reply * test(rebalance): hold the first run until stop cancels admission |
||
|
|
965d805a4f |
ci: run distributed and upgrade e2e on schedule only (#9126)
Remove pull_request from the distributed and upgrade workflows. Distributed stays on the nightly cron, and upgrade stays on the Monday cron plus version tags. Both remain manual-dispatch lanes and stay outside the required merge checks. Co-authored-by: majinghe <majinghe@users.noreply.github.com> |
||
|
|
1cb4c0ebe2 |
ci: stamp the plan contract and rename the workspace lint-and-build key (#9095)
* ci: gate the plan on a contract marker A base planner predating the NIGHTLY_JOBS split still emits the old required set, and the strict verifier would reject it. Select now stamps plan_contract=2 and the workflow refuses any plan without the current marker, falling back to the full matrix, so a stale base planner can neither authorize the new narrower selection nor force a job the new workflow no longer schedules. * ci: rename the workspace lint-and-build job key and evidence path The job key test-and-lint named a job that runs no tests: it does Clippy, the test-binary archive, doctests and the log-rule anchors, while the tests live in workspace-tests. Anyone reading needs: [test-and-lint] would conclude tests had already gated that dependency. The key becomes workspace-lint-and-build to match its display name, and the shared evidence directory becomes artifacts/workspace so the path stops asserting the same thing. The aggregate's display name stays Test and Lint because that is the live ruleset check context. ci-gates.md also stops attributing the ECStore test run to the wrong job. |
||
|
|
e5563e6004 |
feat(release): gate publication on candidate certification (R01) (#9090)
Add a fail-closed release certification path so incomplete or substituted evidence blocks every official publication exit. scripts/release/release_gate.py validates the versioned release-validation plan, the candidate identity and the G01 evidence-envelope set, then re-checks promoted bytes against the passed list. build/docker/package/helm publication jobs require that certification and stage artifacts before promoting identical bytes. |
||
|
|
51275cb7d8 |
ci: move specialized lanes off the pull-request path (#9093)
Connect Feature Boundaries, real io_uring and the rio-v2, Swift, SFTP and HTTP/3 feature lanes no longer gate a merge. They run only in the scheduled and manual full-validation runs, which report a regression within a day; on the pull-request path they bought no earlier signal worth the runner occupancy (Connect alone measured 53.5 median minutes and occasionally set the wall). Workspace lint and tests, the short core contract, the debug build, E2E smoke and the S3 lanes stay required on PRs and merge groups. |
||
|
|
3bf270b49f |
ci: record CI01 measurements, drill evidence and fix the OIDC live gate (#9091)
* ci: retain CI01 timing sample and fix the OIDC live gate tool Record the measured PR feedback and runner occupancy for the CI plan's CI01 criterion, including the honest gap between the experimental targets and the sampled medians, and replace the awscurl dependency in the live Keycloak gate with the AWS CLI so the security suite runs on PEP 668 managed fleets. * docs(testing): attach live run evidence to the event-drill index |
||
|
|
d8d064bd18 | docs(testing): index event-class drills and retained CI evidence (#9088) | ||
|
|
b8ee0301e9 |
feat(scanner): fair bucket service across restarts and budgeted first enumeration (#9083)
* feat(scanner): bound first-build enumeration with the cycle budget Charge raw directory entries and their name bytes, and let one bucket scan hold a child budget that stops its own token while charging the cycle. A count budget also bounds one directory listing, so a wide first build can no longer finish a full read_dir before the first object is charged. The new stop reason maps onto the persisted Directories variant so older readers keep decoding checkpoints. A directory with queued children keeps its index until they end, and every index on the active walk path is saved as an optional map field: losing either one would recharge its replay and stop a fixed budget from advancing. * feat(scanner): keep bucket service order across restarts Save each set's first-in-first-out bucket order as an optional map field of its usage cache, including after an incomplete cycle, which otherwise writes nothing. Durable age now outranks the dirty-first and leader-local cohort orders, which only break its ties, so a restart cannot hand the budget back to the same first buckets. Failed turns add visible debt without moving a bucket forward; a budget stop or a resource veto adds none. A bucket that spends its quantum and saves forward motion returns to the queue tail within the cycle. The order authorizes nothing: another movement epoch or a corrupt value starts a new one, and a recreated bucket joins as a new arrival. * test(scanner): cover quantum handoff and saved service order Run the real worker queue with injected bucket shares and verify a spent bucket yields to waiters, a share without durable motion ends its turn, and the saved order keeps certified coverage while rejecting older owners. * docs(scanner): describe budgeted first build and service order * test(scanner): pin index retention and requeue motion Bound the nested restart fixture tightly enough that dropping a directory index fails it, and cover a repeated share whose frontier never moves. * fix(scanner): keep service generations and owner stops intact Queue a new bucket behind waiters but ahead of buckets that already had a turn, matching the leader-local cohort, and keep the set cache untouched when the owner was cancelled so an interrupted cycle cannot advance its revision. The order holds one record per inventory bucket, like the set cache's entries, so a truncating bound that could starve the long-waiting buckets is removed. Publish the incomplete scope before saving so a contended lock cannot delay it. * ci(e2e): refresh the full selection digests * fix(scanner): correct a comment spelling * fix(scanner): separate service debt from vetoes and stale owners Keep a resource veto and a remote count-budget stop out of failure debt, settle the claim before a requeue exposes the bucket to another worker, and stamp the service order with its leader so an older owner cannot replace it after an incomplete save. The checkpoint fixture's budget now covers both service orders. * fix(ecstore): create parents before resource-scoped writes #9064 moved parent creation into the lazy writer, so a write that holds a resource unit reached the native open without its parent directory and failed with FileNotFound on nested paths. * fix(e2e): retry SlowDown codes during upgrade write readiness The rolling-upgrade readiness probe only retried 503 ServiceUnavailable, so a transient 503 SlowDown from a still-recovering node failed the four_node_rolling_upgrade test immediately. Accept SlowDown and SlowDownRead alongside ServiceUnavailable, matching the existing chaos test transient-error set. |
||
|
|
191c95a469 |
ci: enforce affected validation and rehearse bounded shards (#9070)
* ci: enforce affected validation and rehearse bounded shards * test(ci): keep regression output stable under forced color |
||
|
|
dacb003856 |
ci: let failed-job reruns reuse the producer's test archives (#9071)
Since #9059 the shared test archives and debug binaries are named with the run attempt, and the archive identity binds the attempt. A "Re-run failed jobs" attempt does not rerun the successful producer, so its consumers look for <name>-2, fail with "Artifact not found", and every flaky test forces a full rerun. Give those artifacts one name per run, overwrite them when a producer does rerun, and bind the archive identity to the run instead of the attempt. Source, tree, lock, toolchain and environment stay bound, and an archive from another run is still rejected. |
||
|
|
4fcbf60f10 |
ci: unify validation planning, artifacts and suite orchestration (#9059)
* ci: unify validation planning, artifacts and suite orchestration * fix(ci): preserve JSON diagnostics for release checks * fix(ci): keep manual attempts without plans visible * fix(ci): distinguish nextest setup scripts from test results * ci: reuse extracted archives and consolidate duplicate checks |
||
|
|
b7ff09e357 | chore: strengthen agent delivery standards and validation (#9058) | ||
|
|
4a63e53161 |
test(validation): run canonical cases and build pinned Heal tools (#8455)
* test(validation): add pinned auto-testing manual lane * Install pinned ILM client for canonical validation lane * Prepare Python client for pool validation cases * Install validation client for replication cases * test(validation): provide the Python client for Heal cases * test(validation): build the candidate-bound Heal inspector artifact * test(validation): expose tool artifacts through the existing manual lane * test(validation): verify pinned Heal tools before fleet execution * test(validation): verify Heal tools against released shard captures * ci: audit retained validation environments before finalization * test(validation): add optional native Heal regression tooling * test(validation): route native tools through the existing driver * Run validation tool resolution on hosted Linux without acquiring the fleet lock * test(validation): bind C07 to pinned native authority evidence * test(validation): build optional C10 fixture codec on hosted Linux * test(ci): bind C10 execution to the verified shared codec * feat(ci): expose read-only validation fleet inventory * docs(ci): describe supported validation recovery * test(ci): add isolated manager native validation builds * test(ci): expose manager regressions through tools-only dispatch * ci(validation): consume pinned manager native prerequisites * fix(heal): preserve typed bucket errors for bounded retries * fix(heal): preserve typed errors and SSE-C response headers * ci(validation): publish isolated immutable candidate packages * test(validation): enforce private candidate manifest schema * test(validation): verify isolated package publication failures * ci(validation): dispatch isolated packages through the driver * fix(heal): deeply verify read repair payload failures * test(ecstore): verify incarnation fence at repair commit * ci(validation): verify C11 native storage evidence * fix(ci): isolate validation package publication credentials * test(ci): archive scoped F07 directory fsync evidence * fix(ecstore): retain typed quorum errors for missing parts * chore(ecstore): shrink typed-error ratchet baseline * fix(ecstore): retain below-quorum data during no-remove heal * fix(ci): allow recovery of completed cancelled validation runs * fix(heal): retain exhausted admin batch repair receipts * fix(observability): retain typed object error diagnostics * fix(s3): map typed RPC unavailability to service refusal * fix(filemeta): classify supported inline framing damage as corrupt * test(ecstore): exercise heal during committed overwrite cleanup * ci(validation): admit source-bound G01 native evidence before fleet execution * test(ecstore): fence version races and bind cleanup slots * ci(validation): require complete race native runtime preflight * test(ecstore): fence marker and batch deletes during heal * ci(validation): require G04 native and runtime admission * test(ecstore): fence multipart continue complete and abort during heal * test(ecstore): cover default heal owners and copy metadata races * ci(validation): admit source-bound owner and copy race proofs * test(scanner): assert typed corruption after metadata truncation * fix(ci): require complete native admission for multipart races * test(validation): wire I05 queued-successor native prerequisites * fix(scanner): bind object heals to the scanning disk scope * test(scanner): reuse storage boundary for scope regression * style: format RPC digest precedence assertion * fix(heal): query automatic tasks on their owning node * test(admin): reuse error facade in heal query regression * fix(heal): retain cancellation after successful bucket storage I/O * fix(ci): retain validation evidence outside runner temp * fix(heal): retain resolved object identity on terminal failures * test(scanner): adapt scope regression to admission receipts * ci(validation): inspect shared runner without fleet access * ci(validation): prepare Python before creating client environments * ci(validation): isolate each client virtual environment * ci(validation): retain interrupted evidence without fleet recovery * ci(validation): bind recovery to retained runner evidence * ci(validation): wire persistence and lock source plans * fix(validation): align read-only dispatch defaults --------- Co-authored-by: Chris <anzhengchao@gmail.com> |
||
|
|
a9d3022f37 |
fix(ecstore): prove multipart part sizes before cleanup (#9037)
* fix(ecstore): prove multipart part sizes before cleanup Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(ecstore): merge strict multipart observation passes Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test: respect lifecycle architecture boundaries Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test: fix multipart E2E clippy warnings Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
|
be485e6cab |
test(scanner): add memory acceptance runner and budget probes (#8988)
* test(scanner): add memory acceptance runner and budget probes Add owned S3 workload phases with process, endpoint, cgroup and binary identity checks for scanner memory and scanner-disabled request bursts. Keep scale, restart, heap-attribution and overall-acceptance limits explicit. Characterize healthy and seeded coverage-gap cache retention through the real folder walker and production cache codec without changing runtime code. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(scanner): bind acceptance capacity to effective storage Validate the serving process's complete local disk inventory before load writes and during runtime sampling. Resolve each disk inside the owned storage root, reject nested mounts and filesystem mismatches, and bind storage observations to phase identity for container and native runs. Cover the unused-mount bypass, CLI-overridden environment, native paths, symlink escapes, named volumes and stable multi-disk inventory ordering. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(scanner): fence status observations before workload writes Close each status API window with a second storage-bound process sample before Phase entry can authorize bucket creation or PUT requests. Add a fail-before regression for a process change during the initial status request, asserting no ownership or workload writes occur. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
|
fbd94e129d |
test(e2e): validate late multipart part transaction packets (#8989)
Retain original prepare and commit-settlement RPCs beyond failed replacement and Abort cleanup, with and without bucket recreation. Require typed refusal or a physically verified idempotent result, unchanged fresh-upload bytes and complete object reads. Extend scoped proxy observations without rewriting signed frames or trailers, and preserve registered test membership and existing budgets. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
|
aba9e2e9e6 |
test(e2e): cover partial EC:4 node recovery (#8983)
Verify a sixteen-drive set rejects reads and writes after two nodes stop, then resumes complete S3 reads and new writes when only one node returns. Keep the fourth node stopped throughout checks from all three live coordinators and verify the original 12+4 shard geometry on disk. Let the existing GET recovery probe retry the compatible SlowDownRead response only with HTTP 503, retaining fatal status and byte checks. Register the case in both platform selections and document the boundary between process recovery and physical disk reattachment acceptance. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
|
26d55bb04e | ci: remove never-green Performance A/B workflow (#8979) | ||
|
|
c9f7e48a44 |
fix(observability): correlate physical namespace lock releases (#8974)
* fix(observability): correlate physical namespace lock releases Map logical namespace guards to their actual granted peer lock IDs and local physical guard identities. Record physical completion only after successful manager release and registry removal, and record validated positive remote unlock responses without changing release behavior. Exercise real distributed ownership, asynchronous cleanup, duplicate and disabled guards, stale-owner retirement, and explicit release followed by Drop. Document the required diagnostic filters and the remaining filesystem-tail boundaries. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): remove unused lock manager test import The diagnostic regression uses inherent manager methods and does not need the LockManager trait import. Remove it to pass strict warning checks. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
|
1880bdca91 |
refactor(listing): list latest versions with MinIO's listing quorum (#8965)
* refactor(listing): list latest versions with MinIO's listing quorum Latest-version SetDisks::list_path now follows MinIO's er.listPath: ceil(asked/2) is the listing quorum for min_disks, directories and objects, agreed entries are emitted as read, and a split sample goes through MetaCacheEntries::resolve. One object can no longer fail a page. A set with fewer than half of its drives online still fails the page with ErasureReadQuorum. Remove the write-quorum layer this replaces: ListingSupplement, ListingReconciler and the LIST object read lock, the latest_listing_* quorum helpers, resolve_agreed_entry, resolve_with_write_quorum and the enforce_write_quorum / write_quorum_slack merge branches, and the per-version listing quorum helpers in filemeta. Make merge_file_meta_borrowed_streams pick a version id's most common header regardless of reader order, so an interrupted null-version overwrite no longer drops the committed generation from the merge. * test(listing): do not assert sampled optimal listings are deterministic optimal asks a random half of the drives, so an EC:8 object committed on nine of sixteen drives can miss the four-copy listing quorum in roughly one sample in six, exactly as in MinIO. Keep the deterministic assertion for strict and for optimal when every online drive is asked. |
||
|
|
fea5a584c3 |
fix(storage): fence multipart and metadata recovery (#8919)
* perf(ilm): back off idle tier-delete journal recovery Use bounded idle backoff only after complete empty journal and dispatch scans, while preserving immediate notifications and the retry cadence for observed work. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): reconcile interrupted pool metadata publication Bind pool metadata CAS and Undo to exact native predecessors. Recover an interrupted physical publication only from an ordinary committed authority and complete verified identity, current and backup evidence, then require normal V3 CAS and quorum readback. Preserve known-conflict gates, startup and heal lock ordering, targeted admission and the original kill-recovery deadline. Wake proof-ready recovery immediately and bound parallel capacity sampling without removing mutation fences. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(ecstore): reuse proven pool metadata durability Publish native rollback backups directly in their owned UUID directories. Reuse a one-shot data-durability receipt only for an exact hardlink to the pinned, fully synchronized head and root. Preserve new-head and directory synchronization, native predecessor CAS, conditional Undo and uncertainty gates. Keep unknown, copied, raw-parent, cancelled and failed operations cold. Add bounded diagnostics, sync-call regressions and path-isolated test hooks. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(multipart): exercise selective RPC delay and peer loss Add authenticated method-entry barriers for metadata and lifecycle locks under e2e-test-hooks. Verify real dispatch, write quorum, prior part payloads, lock drain and incarnation isolation without changing production budgets. Record cancellation separately from normal exits so late filesystem completion cannot overwrite Drop evidence. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * docs: define multipart delay gates and receipt invariants Document RPC entry evidence, peer-loss and cancellation oracles, remaining incident attribution limits, and immutable head inodes required for durability receipts. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(readiness): expose missing runtime drive slots Report configured slots without a runtime handle separately from unavailable inventoried drives. Preserve quorum and probe status semantics. Wait for every coordinator to regain its complete live inventory before multipart RPC fault windows and single-attempt recovery oracles. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): retry committed promotion convergence safely Require peer reload convergence on every worker attempt, including unchanged retries. Recheck durable identity, replicas, namespace fence, generation and cancellation before allowing any data movement. Keep known committed promotions active on a verified propagation pause. Release precise owners, retain queued successors and reuse the existing watchdog. Preserve uncertainty gates and terminal operation behavior. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): guard native metadata mutation aliases Reject protected RenamePart source and destination paths before I/O. Classify the protected namespace through the native path resolver, and require canonical conditional RenameData and Undo keys so aliases cannot create separate lease domains. Preserve ordinary multipart and canonical CAS behavior, warm durability receipts and zero-allocation canonical fast paths. Cover real alias, byte identity, receipt and normal mutation behavior. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): register Darwin multipart RPC delay gates Update distributed membership from the actual Darwin listing. Preserve the full-profile digest and leave Linux unchanged until its own listing is produced, as required by the contributor guide. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(multipart): fence native publication and recovery Publish multipart data through private hardlinks and atomically consume the prepared source head. Bind exact compensation capabilities and native recovery to the physical cohort, source generation and durable publication state. Require fresh terminal confirmation for native replay and preserve indeterminate data when no safe recovery witness exists. Release logical guards only after validated irreversible quorum success while retaining physical mutation leases, accounting and resource ownership through actual callbacks. Preserve local instance proof authority and historical Copy metadata without granting healed donors fresh publication authority. Keep general movement identity strict while preserving directory capacity compatibility in a separate checked comparison. Cover native fault and cancellation boundaries, zero-byte objects, real Copy, heal and migration, exact physical readback and distributed RPC delay. Retain all quorum and durability requirements and document rollout limitations. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): fix ratchet scan and Windows fault hooks Move native pool metadata inventory tests after production items so the error-format ratchet sees the existing call site. Preserve the baseline instead of accepting a false count decrease. Make the directory-barrier failure hook available on every test platform and exercise the parent-barrier failure/retry cases in Windows CI. Keep production synchronization behavior and all existing assertions intact. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(protos): remove redundant recovery request clone Move the baseline into its final struct update while retaining the earlier clone needed to construct the first request variant. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): align RPC delay gates with native abort * fix(storage): validate native source recovery and CI gates Preserve wire layouts while bounding native response sizes and fix the last-use recovery request clone. Avoid destination replay reads only for fully observed initial native sources; keep upgraded and recovered sources conservative. Cover real legacy publication after a cold restart. Align upgrade acceptance with the native mixed-version mutation pause and use exact upload-scoped test completion evidence before advancing time. Keep all original quorum, ownership, byte and deadline assertions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): verify native abort under partial admission Delay native cleanup at the exact upload root after all-participant closure. Verify quorum-only admissions retain every source byte on refused Abort and complete the same upload without creating a missing original participant. Keep sub-quorum orphans and their discoverable identities isolated while a fresh all-participant upload proves service recovery. Route shared constants through the existing E2E storage facade. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): avoid formatting a fixed invalid upload path Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): align native MPU fault recovery assertions Verify real source participants, preserved orphan bytes and same-upload completion after quorum-only admission. Require full physical inventories so an expected read-quorum error cannot hide unrelated leakage. Preserve ordinary quorum and recovery budgets, and document listing availability through the affected set. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): keep MPU tail deadlines on the wall clock Global Tokio time advancement can expire drive-health probes while real filesystem callbacks are pending. Use a short real drive deadline with the existing physical-entry and cleanup barriers, retaining the byte, lock, resource-permit and final-shard assertions. Fixes #8937 Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(storage): align native fixtures with instance ownership Open independent disks for the simulated capacity-loss instance, scope four coordinator journal disks without rebinding source authority, and probe configured namespace keys while retaining physical I/O hooks. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): isolate native multipart future construction Construct optional native publication futures behind synchronous heap boundaries so ordinary rename callers do not carry their state machines. Preserve native guards and ownership without adding allocations to ordinary PUT, copy or heal branches. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(heal): assert coordinator journal scope after rejoin Check the exact original four-member journal map after every rejoin and manager stop while preserving all sixteen native data slots. Reconnect expansion fails the test rather than being filtered away. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(heal): bound object repair future construction Construct the normal and FileCorrupt deep-retry state machines behind a synchronous heap boundary. Preserve guards and cancellation while bounding the shared rename and deletion call stacks without increasing test stacks. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): qualify fixture disk identity checks Call the same DiskAPI method explicitly so the independently reopened disk identity assertion compiles under every target and feature combination. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): isolate native inspection from mutation state Select a dedicated read-only inspection future behind the existing single Box allocation. Preserve validation, source ownership and the canonical inspection body while keeping mutation state out of inline read callers. Reject misrouted operations with typed errors and retain source evidence. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): pause abort time after native cleanup Wait for durable source deletion on the wall clock before advancing the capacity lease clock at the post-delete barrier. Paused time can otherwise expire the barrier while native filesystem cleanup is still pending. Retain lease-loss injection, durable inflight-byte accounting and restart reconciliation assertions. Document the test-util feature required to select the native decommission fixtures in package-scoped regressions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): reclaim data from owned multipart undo Retain exact runtime-only Undo ownership through deferred snapshot cleanup. Revalidate every target metadata version and the retirement anchor, keep the delete claim through the physical callback, and remove the anchor after the payload barrier. Preserve generic cleanup refusal when metadata is missing. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): preserve unformatted bootstrap routes Keep explicitly unformatted local paths reachable by an in-flight elected formatter while quarantining their service slots and disk-ID membership. Own bounded legacy-read handles so the actual asynchronous creator future remains Send. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(s3): bound CopyObject tracing poll frames Construct the business future behind an outlined single-Box boundary and lazily instrument its handle. Preserve Send, Unpin, span timing and cancellation without embedding concrete business-state construction in the tracing poll. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(mpu): observe recovery after direct legacy bootstrap Seed all sixteen real previous-release sources before attaching the same-port method observers. Require exact per-peer disk-route Unimplemented responses with zero publication calls, retain byte and cleanup assertions, and remove superseded raw transport. Register the two new regressions from actual Linux and macOS listings. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(mpu): prepare the homogeneous previous-release fleet Reuse the existing release baseline before creating the genuine legacy multipart sources. Restart old processes with their normal budgets and require every drive through the reduced-redundancy probes before installing the mixed-version observers and current fault budgets. Keep standard EC:4, single-attempt calls, all mixed-version byte and reclamation assertions, and their original deadlines unchanged. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(mpu): isolate release compatibility from fault profiles Keep normal component configuration for the observed mixed-release rollout, which injects no RPC delay or loss. Retain the short component profile in actual method-fault scenarios and preserve the single-attempt 45-second probe and every typed error, routing, source byte and reclamation assertion. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(mpu): use normal shutdown for planned release replacements Use the existing bounded SIGTERM path during homogeneous release preparation and the observed mixed-version rollout. Preserve every multipart inventory, routing, quorum and byte assertion, and keep actual crash recovery on its original hard-kill path. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * revert: retain the verified release compatibility fixture Restore the published fixture after the local configuration and lifecycle experiments failed to resolve the macOS startup failures. Keep their diagnostic records separate from the delivered implementation. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(copy): bound inherent handler polling stack Build the inherent CopyObject future through a synchronous noinline factory at the existing trait await. This keeps its large state out of the trait wrapper's active polling frame during historical self-copy. Preserve cancellation, tracing and post-write notifications, with one additional allocation restricted to COPY. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): validate authoritative quota and native pool heals Require actual authoritative quota usage in the existing readiness loop instead of treating HTTP success or unknown usage as a numeric baseline. Keep the original quota assertions and convergence budgets. Allow the owning pool metadata CAS to republish into a fresh valid data directory while retaining exact version, layout, completeness and shard fingerprints. Require every node's real metadata/read/write readiness before accepting recovery. Keep ordinary object manifests and the original 180-second recovery deadline unchanged, with negative oracle regressions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): distinguish certified native Abort recovery Require native participants with a persisted full closure certificate to retire their heads and reclaim every surviving source on retry. Keep uncertified legacy authority below identity quorum conservative, while retaining actual partial-delete and committed-object byte checks. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> Co-authored-by: overtrue <anzhengchao@gmail.com> |
||
|
|
4a96f06983 |
refactor(listing): remove env-gated ListObjects index provider (#8948)
* refactor(listing): keep paginated listing on ECStore only `Sets` and `SetDisks` carried their own copies of `inner_list_objects_v2`, `list_objects_generic`, `inner_list_object_versions`, `list_path` and `list_merged`, plus a third copy of the task that converts merged raw entries into the walk result stream. Only `ECStore` can answer a paginated listing, since a page merges every set of every pool; the copies drifted (the `Sets` copy still rejected `//` prefixes that #8605 made literal on `ECStore`) and were reachable only from tests and one decommission helper. Remove the `ListOperations` impls for `Sets` and `SetDisks` together with the duplicated pagination code. `SetDisks::walk` stays as an inherent method for set-scoped callers and now honors `versions_sort` like the store-wide walk; `SetDisks::list_path_result` stays for the recursive delete lister. The walk entry conversion moves into one `spawn_walk_entry_forwarder` shared by both walks. The decommission receipt lister walks its pool's sets instead of paginating through the removed `Sets` listing. Tests that exercised the removed layers now exercise `ECStore` for pagination or `SetDisks::walk` for per-pool version inspection. The architecture and logging guard rows that pinned the removed `SetDisks`/`Sets` listing entry points are dropped, and the error-other ratchet baseline shrinks by one. * fix(listing): exclude synthetic directories from receipt walks * refactor(listing): remove env-gated ListObjects index provider Delete the experimental, default-off key-only / verified-page / metadata-fast index provider from `ECStore::list_objects_generic` together with everything only it drove: the `RUSTFS_LIST_OBJECTS_INDEX_*`, `RUSTFS_LIST_OBJECTS_NAMESPACE_JOURNAL_*` and `RUSTFS_LIST_OBJECTS_METADATA_FAST_*` readers, the persistent key-only index and its on-disk format, the namespace mutation journal (state, chaos injector, system/local backends, degraded set, global lock), the index lifecycle/health/source-selection types, the verified-index candidate pager, the `list-chaos` cargo feature, the `rustfs_s3_list_objects_index_*` metrics and the archived bench script. `list_objects_generic` now always walks. The write-path hooks `observe_list_objects_mutation(s)` are gone; their callers bump the scanner namespace generation directly through `observe_scanner_namespace_mutations`, which is the sub-step the old hook performed before taking the journal lock, so the generation observed by the scanner, the node RPC service and the data-movement lease checks is unchanged while PUT/DELETE no longer take a global tokio RwLock or read environment variables. `ListSourceMode::{metadata_authority, requires_live_metadata_verification, can_satisfy_strong_listing}` and `ListMetadataAuthority` were only reachable from the provider and are removed as well; the enum, its variants, the cursor encoding and `ListContinuationV2` are left untouched. * docs(listing): define index retirement upgrade prerequisites * fix(ci): retire the removed listing index test requirement |
||
|
|
fab4bb0586 |
test(e2e): cover simultaneous multipart terminal contenders (#8943)
* test(e2e): assert Abort refusal below identity quorum Keep exactly-quorum success and recoverable cleanup assertions. For minority staging authority, require typed refusal, zero further Delete RPCs, unchanged per-disk metadata and the previous object body. This aligns the phase matrix with the existing conservative recovery contract. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): exercise simultaneous multipart terminal contenders Retain the four pairwise resend cases and add live/cancelled three-way Complete, Abort and resend races. Observe distinct forwarded lock owners and verify terminal bytes and physical cleanup. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): freeze contender evidence before multipart cleanup Retire the passive lock observer before terminal ListParts checks so the reported count identifies only the two contenders at the preparation barrier. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): require both multipart terminal race outcomes Retain the free races and select both terminal winners with independently released, owner-scoped lock gates. Verify real contender admission, unchanged signed packets and exact terminal bytes for live and cancelled part waiters. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): isolate losing coordinator lock votes Hold every remote vote for the losing coordinator while the same gate observes the winner. Prevent partial lock grants from delaying the requested terminal order and retain ordinary RPC caller lifetime. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
|
c610d11e8f |
fix(storage): guard MPU recovery and startup RPC admission (#8934)
* fix(ecstore): preserve uncertainty after partial multipart abort Keep recovery callers from reporting NoSuchUpload while minority primary metadata remains. Retain all quorum and deletion checks. Exercise actual partial Abort and explicitly retained late RPC packets after caller retirement, staging cleanup and bucket recreation. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): prevent retired multipart staging resurrection Require the upload root to exist before preparing recovery publication. Preserve rooted directory walks for nested bases on non-Linux Unix and cover missing-source native work, configured aliases and pinned mixed-version coordinator recovery. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): use IPv4 peer aliases for pinned mixed recovery Keep direct IPv6 S3 listeners behind IPv4 RPC proxies for the pinned previous release, whose peer health resolver rejects bracketed IPv6 hosts. Preserve signed authorities, ports and strict recovery assertions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): separate pinned startup from MPU fault budgets Use the old fleet defaults during bootstrap, then apply the original short fault budgets to the current replacement. Keep direct S3 URLs aligned with IPv6 listeners and preserve all quorum and physical cleanup assertions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): seed legacy MPU through transparent transport Retire raw proxy connections before enabling method faults so legacy streaming setup stays byte-transparent and every later scoped mutation must traverse the fault barrier. Verify raw and adapted protocol authority, bodies and trailers. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): retain direct listeners for legacy MPU peers Advertise same-port localhost peers for the pinned resolver while retaining direct IPv4 S3 listeners. Exact scoped RPC hit counts still reject bypassed injection and all physical authority assertions remain unchanged. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(server): keep startup gRPC on the authenticated router Use the hybrid protocol predicate for startup RPC admission so early control responses retain production authentication and owner validation. Keep REST FullReady gating and pinned replay capability rejection intact. Add deterministic regressions, scoped diagnostics and bounded legacy fixture resources. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ci): verify MPU recovery across upgrade source versions Run the exact sixteen-drive mixed MPU case for every checksum-verified upgrade source, independently of generic compatibility failures. Require one listed test and retain per-version logs while preserving the historical distributed source pin and every recovery assertion. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ci): trigger upgrade MPU checks for fixture changes Include distributed fixtures and the RPC fault proxy in the upgrade workflow paths so test-only changes schedule the strict recovery gate. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(storage): verify resend owners across MPU finalization Hold real part-transaction RPCs while another coordinator completes or aborts the upload, with both live and cancelled client response waiters. Require full object bytes, terminal upload state, and all staging roots to remain consistent after the part owner drains. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
|
017b85a5bf |
fix(ecstore): preserve inactive multipart abort authority (#8920)
Retain per-disk recovery metadata through partial publication and require write-quorum proof for completion replay. Keep completed uploads terminal and strengthen RPC recovery acceptance. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
|
0f24511535 |
test(e2e): harden multipart fault and TLS acceptance (#8907)
* test(e2e): tighten scoped multipart fault acceptance Target Abort root deletion instead of child transaction cleanup, require actual metadata peer hits, and keep faulted APIs single-attempt. Validate bounded ordinary recovery and safe quorum refusal for partial mutations, with the recovery enhancement tracked in backlog#2940. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(e2e): drive concurrent TLS reads independently Give each coordinator its own supervised task and cooperative polling budget while retaining the existing workload, deadline and byte/hash assertions. Preserve node logs after fixture teardown and report the operation, response phase and received bytes when a transfer times out. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
|
882217221e |
fix(lock): reach quorum behind silent voters (#8895)
* test(e2e): add MPU RPC quorum and recovery acceptance Exercise separate storage and lock RPC delays across multipart phases, including quorum boundaries, cancellation, bucket recreation, and bounded eight-node load. Honor TMPDIR for isolated cluster drive directories. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(lock): probe healthy voters behind silent peers Spread additional probes across the existing acquisition attempt budget when the initial quorum-plus-spare wave stalls. Preserve quorum thresholds, fast-path fan-out, and existing ownership and late-response cleanup. Cover every four-node read-voter ordering, a stalled eight-node write quorum, and normal fan-out. Extend MPU acceptance to EC12+4 at twelve and eleven metadata acknowledgements. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
|
631cd0ceca | fix(ecstore): preserve migrated reads and retry busy pool heals (#8821) | ||
|
|
d9ab8760ee |
test(e2e): accept retried rebalance deferrals; retry busy decom preflight (#8813)
test(e2e): accept retried rebalance deferrals and retry busy decommission preflight |
||
|
|
b06923beaf |
test(e2e): add scanner/heal topology protection matrix (#8705)
* test(e2e): add scanner/heal topology protection matrix Add the S00 protection baseline for the Scanner/Heal program (rustfs/backlog#2873, program rustfs/backlog#2872): seven scanner_heal_topology_ cases in the distributed e2e lane that pin pool expansion, rebalance and decommission behavior before the runtime changes. The fixture seeds a deterministic inventory (empty, inline, external, multipart, special and nested keys; a versioned bucket with an explicit null version, three history versions and a delete marker per key; an Object Lock bucket with legal hold and COMPLIANCE retention), records every version with its requested and listed version id, length and SHA-256, verifies the manifest through every node, reads physical placement from xl.meta on the pool drives, and reads the strict durability mode back from the admin API and the node logs. Fault cases hold the mover at proven windows through file barriers that compile only with the e2e-test-hooks feature: before the multipart target commit and before source cleanup (shared by decommission and rebalance), plus the existing pool-metadata before_publish cutpoint. An unreached window fails the case. DistCluster gains capture_logs_under so a case can read node startup events without a global environment variable. docs/testing/scanner-heal-topology-gates.md maps the gates to the tests and records the timeout basis. * test(e2e): keep the barrier failure message static The error(format!) ratchet rejects new formatted io::Error::other call sites in ecstore. The reached observation already records the stage and object, so the injected failure needs no per-call detail. * test(e2e): require e2e-test-hooks for topology fault cases The four scanner_heal_topology_ fault cases inject at server barriers that exist only with the e2e-test-hooks feature. Against a default server they waited for the barrier timeout and reported it as not reached. Check the verified binary's features first so the missing prerequisite fails immediately and is named. * ci(e2e): build the distributed lane with e2e-test-hooks Every other e2e lane builds and runs the fault-test hooks server. The distributed lane alone used the default server, so the topology fault cases could never reach their barriers there. Build and run with the same feature so the receipt matches, and update the documented local commands. The upgrade cases spawn the pinned previous release directly and do not compare feature sets. * test(e2e): keep topology artifacts under RUSTFS_E2E_LOG_DIR The topology fixture captured node logs, barrier observations and evidence in a temp dir removed on drop, so the distributed workflow uploaded nothing for these cases. Fall back to RUSTFS_E2E_LOG_DIR, which the workflow uploads, when no explicit evidence directory is set. * test(e2e): remove retired pools before restart; race the pre-ACK start The queue case restarted with decommissioned pools 0 and 1 still on the command line, which the server rightly refuses. Add remove_single_node_pools and restart without them, then expect the remaining pools renumbered to 0 and 1. The pre-ACK case now races the start request against the barrier. A start that returns first, or stays in flight past the documented 60 s budget, is reported as such instead of as an unreached barrier, and the evidence records the reach latency. capture_logs_under now overrides a log path assigned from RUSTFS_E2E_LOG_DIR, so the seed node logs into the case directory too. * test(e2e): bound fault-window waits by the movement budget On Linux CI a busy target capacity gate delayed the first fault object by about two minutes, past the 120 s wait borrowed from the server's release poll. Wait for a movement window within the 240 s movement budget instead, and fail at once when the decommission or rebalance ends without entering the window, so an uninjected fault is never counted. * test(e2e): bound status probes while waiting for a fault window The admin decommission status request can wait on a mover held at a test barrier until the barrier's own 120 s release timeout. The window wait issued that request each loop, so a reached barrier went unnoticed until the server timed it out and failed the decommission, which surfaced as a false resume failure. Bound each status probe to 5 s and treat a slow probe as still running. * test(e2e): retry a busy decommission start A decommission start that meets a busy movement gate can answer a retryable 503 instead of waiting. The harness treated any non-proof failure as fatal. Share one start helper that retries 503 within the case's movement budget and the fleet-proof 500 for 45 s, and keeps every other status fatal. The pre-ACK case retries 503 within its window budget. |
||
|
|
473cddd7dc |
ci: gate pull requests on Clippy and warm its check artifacts (#8708)
Clippy ran only after merge, so lint regressions merged freely and then failed every main run until fixed (10 of 12 main runs on 2026-10-17/18). Run the workspace Clippy step on every event, warm its check-mode artifacts in cache-warm with the identical invocation so pull requests re-check only what they touch, and pin the unconditional step in the ci_gate contract and its self-test. |
||
|
|
90749fbb06 | ci: run feature lanes nightly and warm caches on a schedule (#8683) | ||
|
|
1ec620ac9c |
fix(ci): restore the release-profile short-credential guard (#8691)
#8682 replaced the cold `cargo check -p rustfs --release` with a `cargo rustc ... -C debug-assertions=off` on the rustfs lib alone. That leaves the dependencies on the dev-profile cfg, so the lib also fails on items gated on one side of cfg(debug_assertions) only (E0599 on Filenames::Embedded), and the step's "fails solely at the compile_error" assertion can never hold; the lane has been red on main and every pull request since. Restore the whole-graph release check, drop the two-job limit that only a linking step needed, and record why the single-crate variant is unsound. |
||
|
|
392ccadb78 | ci: build development packages nightly instead of on every main push (#8684) | ||
|
|
4d7cd23f8e |
ci: reuse the dev cache in Connect boundary lanes and raise build jobs (#8682)
Raise CARGO_BUILD_JOBS to 3 on every event in the workspace test lane and record the closed backlog#1601 experiment: the test-profile compile took 28m21s at 2 on a pull request and 16-23 min at 3 across three successful main runs. Merge the two Connect boundary jobs into connect-feature-boundaries. The offline enrollment gate no longer exports a fresh CARGO_TARGET_DIR, so it rebuilds only the rustfs crate on top of the restored ci-dev cache; the E2E root is isolated by the feature gate and the dedicated binary name, not by the target directory. The short-credential release guard is now `cargo rustc -p rustfs --lib --features connect-e2e-short-credentials -- -C debug-assertions=off`, which trips the same cfg(not(debug_assertions)) compile_error against cached dev-profile dependencies instead of a cold release check of the whole graph. Every assertion of both lanes is kept. Update ci_gate.py and the ci-gates.md PR matrix for the merged job. |
||
|
|
ff70c06cef | fix(auth): restrict root-derived accounts and credential logs (#8674) | ||
|
|
fca83c72e4 |
ci: move compile-free and prebuilt-binary lanes to hosted runners (#8679)
* ci: move compile-free and prebuilt-binary lanes to hosted runners Standard GitHub-hosted runners are free for this public repository, while every sm-standard minute is paid self-hosted capacity. Move the jobs that never compile (quick checks, typos, scope selection, the required-checks aggregate, audit, closed-PR cancellation) and the two s3-tests lanes that only run the prebuilt debug binary to ubuntu-latest, drop the wget install the custom image needed for typos, and move the upgrade-compatibility workflow, whose build shape already runs on hosted runners in e2e-distributed, with renamed cache keys so the hosted and self-hosted images never share a rust-cache entry. Gate the package summary job on resolve having run, so development builds stop starting a pod to report a skipped job. * docs(ci): document runner billing and placement rules Record in .github/workflows/README.md which runner labels GitHub bills (none for this public repository), which ones RustFS pays for, the rule for when a job may leave ubuntu-latest, the rust-cache single-writer and same-image constraints, the patterns that have leaked self-hosted minutes, and a checklist for new or moved jobs. Point to it from .github/AGENTS.md, docs/testing/ci-gates.md and the ci.yml placement comment so the rule has a single home. |
||
|
|
78ea655d87 |
fix(startup): drive startup on an explicitly sized thread (#8630)
`run_process` called `block_on(async_main())` on the native main thread, so the whole startup chain ran on whatever stack the platform hands the main thread: 8 MiB from `ulimit -s` on Linux and macOS, but the 1 MiB PE default on Windows. A Windows debug build overflowed before it listened. `RUST_MIN_STACK` does not reach the native main thread. Run the runtime on a `rustfs-main` thread with an explicit 8 MiB stack, keeping the budget Linux and macOS already had and making it independent of the linker, toolchain (msvc/gnu) and `ulimit`. Panics are re-raised on the main thread so the exit status is unchanged. Measured on macOS arm64 by lowering the main-thread stack with `ulimit -s` before this change: debug startup overflows at 1792 KiB and reaches /health/ready at 2048 KiB; release overflows at 512 KiB and is ready at 768 KiB. After it, debug starts with a 256 KiB main thread. Add a Windows `Startup Smoke` job that builds the debug server, runs `--version`, and waits for /health/ready on a single volume. |